andygrove opened a new pull request, #7646: URL: https://github.com/apache/datafusion-comet/pull/7646
## Which issue does this PR close? Backport of #7193 to `branch-1.2`. Closes #7192 on this branch. ## Rationale for this change `coalesceBroadcastBatches` appends the whole broadcast into one `VectorSchemaRoot`. Arrow Java's `VectorAppender` grows a string or binary data buffer by doubling it and cannot go past 1 GiB, so a broadcast join fails with `OversizedAllocationException` once one such column of the build side holds more than that, where Spark runs the same query. `branch-1.2` has the same unbounded coalescing in `Utils.coalesceBroadcastBatches`. See #7193 and #7192 for the full analysis and repro. ## What changes are included in this PR? Cherry-pick of 4e54f1c461 (`git cherry-pick -x`) with no conflicts and no adaptations. `coalesceBroadcastBatches` starts a new coalesced batch when the next input batch would take a string or binary vector (nested ones included) past 2^30 bytes of values, and an `OversizedAllocationException` from the appender now leaves the broadcast uncoalesced. A broadcast below that size is still one batch. It includes the source PR's `UtilsSuite` test. Other release branches: `branch-1.2` is the newest release branch, so there is no newer one to port to. Whether `branch-1.1` needs a backport is not part of this PR. ## How are these changes tested? On this branch, with JDK 17 and the default Spark profile (4.1): `./mvnw test-compile` passes, and `UtilsSuite` passes (15 tests, 0 failures) including the new test. I did not rerun the 1 GiB repro from #7192. The test cannot fail on `branch-1.2` without the fix in the usual way, because it passes the new `maxValueBytes` parameter, so with `spark/src/main` reverted to `branch-1.2`'s version the test does not compile. I ran the default Spark profile only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
