luomh0910 opened a new issue, #13596: URL: https://github.com/apache/gluten/issues/13596
### Backend VL (Velox) ### Bug description ColumnarBuildSideRelation.buildHashTable deserializes all broadcast batches before starting native hash-table construction. For large broadcast relations, this creates significant memory pressure because the following allocations can coexist during the build: - The serialized broadcast data. - All deserialized build-side batches. - The hash table being constructed. The deserialized batches can occupy substantially more memory than their serialized representation, especially for wide schemas and variable-length columns. Retaining the entire deserialized input alongside the growing hash table increases peak memory usage and can lead to out-of-memory failures, even when the final hash table alone could fit in memory. ### Expected Behavior Support consuming broadcast batches incrementally: deserialize a batch, feed it into the hash-table builder, and release the input batch when it is no longer needed. This issue was written with the assistance of AI (Codex with GPT-6). ### Gluten version _No response_ ### Spark version None ### Spark configurations _No response_ ### System information _No response_ ### Relevant logs ```bash ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
