andygrove opened a new pull request, #7646:
URL: https://github.com/apache/datafusion-comet/pull/7646

   ## Which issue does this PR close?
   
   Backport of #7193 to `branch-1.2`. Closes #7192 on this branch.
   
   ## Rationale for this change
   
   `coalesceBroadcastBatches` appends the whole broadcast into one 
`VectorSchemaRoot`. Arrow Java's `VectorAppender` grows a string or binary data 
buffer by doubling it and cannot go past 1 GiB, so a broadcast join fails with 
`OversizedAllocationException` once one such column of the build side holds 
more than that, where Spark runs the same query. `branch-1.2` has the same 
unbounded coalescing in `Utils.coalesceBroadcastBatches`. See #7193 and #7192 
for the full analysis and repro.
   
   ## What changes are included in this PR?
   
   Cherry-pick of 4e54f1c461 (`git cherry-pick -x`) with no conflicts and no 
adaptations. `coalesceBroadcastBatches` starts a new coalesced batch when the 
next input batch would take a string or binary vector (nested ones included) 
past 2^30 bytes of values, and an `OversizedAllocationException` from the 
appender now leaves the broadcast uncoalesced. A broadcast below that size is 
still one batch. It includes the source PR's `UtilsSuite` test.
   
   Other release branches: `branch-1.2` is the newest release branch, so there 
is no newer one to port to. Whether `branch-1.1` needs a backport is not part 
of this PR.
   
   ## How are these changes tested?
   
   On this branch, with JDK 17 and the default Spark profile (4.1): `./mvnw 
test-compile` passes, and `UtilsSuite` passes (15 tests, 0 failures) including 
the new test. I did not rerun the 1 GiB repro from #7192. The test cannot fail 
on `branch-1.2` without the fix in the usual way, because it passes the new 
`maxValueBytes` parameter, so with `spark/src/main` reverted to `branch-1.2`'s 
version the test does not compile. I ran the default Spark profile only.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to