luomh0910 opened a new pull request, #13616: URL: https://github.com/apache/gluten/pull/13616
## What changes are proposed in this pull request? This PR adds an optional streaming input path for Velox broadcast hash-table construction. Instead of deserializing and retaining all input batches before building the hash table, the new path consumes batches on demand and feeds them incrementally to the native hash-table builder. - Add spark.gluten.velox.broadcastBuild.inputStreaming.enabled, disabled by default. - Pass broadcast build input through ColumnarBatchInIterator and a native batch callback. - Force single-threaded construction when streaming is enabled, including a native-side fallback when the requested thread count differs. - Support both on-heap and off-heap broadcast relations, with driver-side and executor-side hash-table construction. - Keep the materialized input path and existing serial/parallel build behavior when streaming is disabled. The change avoids retaining all deserialized input batches simultaneously; it does not change the final join-table preparation. ## How was this patch tested? - Added native HashTableBuilderTest coverage for incremental input consumption, batch lifetime, empty input, null-aware anti-join early termination, error handling, and join semantics. - Extended VeloxHashJoinSuite to cover streaming enabled/disabled, on-heap/off-heap relations, driver-side/executor-side builds, residual join conditions, broadcast reuse, and dump/replay. - Added VeloxBroadcastBuildThreadSuite to verify thread selection with streaming enabled and disabled. - Local validation passed: Scala Spotless, C++ formatting with clang-format 15.0.7 ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex <GPT-6-Astra> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
