FelixYBW opened a new issue, #12659: URL: https://github.com/apache/gluten/issues/12659
### Backend VL (Velox) ### Bug description Noted as a known issue here. The issue happens when you set **large creationSideThreshold, maxNumBits, and maxNumItems with CBO enabled**. When CBO (Cost-Based Optimizer) is disabled, Spark creates a Bloom filter subquery based on creationSideThreshold, falling back to default values for estimatedNumItems and numBits. The query successfully runs with partial_bloom_filter_agg(expr, estimatedNumItems, numBits, 0, 0). However, when CBO is enabled, Spark estimates estimatedNumItems and numBits dynamically. If these CBO-estimated values become exceptionally large, partial_velox_bloom_filter_agg will consume significantly more memory and generate a much larger Bloom filter compared to native Spark under the exact same configuration. This discrepancy leads to the following cascading failures: Gluten throws a spark.rpc.message.maxSize error due to the massive payload size. Even if spark.rpc.message.maxSize is significantly increased, the subsequent velox_bloom_filter_agg operator eventually throws an OOM (Out of Memory) error. e.g. if we test TPCDS SF1T, the ```partial_bloom_filter_agg(xxhash64(wr_order_number#65L, 42), 72002305, 739372223, 0, 0)``` can pass on Spark but OOM on gluten. The shuffle data size on Spark is only **69.4M**, while on Gluten it's **9.3 GiB**! cc @jinchengchenghh ### Gluten version _No response_ ### Spark version None ### Spark configurations _No response_ ### System information _No response_ ### Relevant logs ```bash ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
