ted-jenks opened a new pull request, #7537: URL: https://github.com/apache/datafusion-comet/pull/7537
## Which issue does this PR close? Fixes the default shuffle read path in #7496. I'll follow up on the other paths listed there. ## Rationale for this change A small dictionary-encoded shuffle block can expand past 2 GiB, which made ShuffleScanExec panic. branch-1.2 has the same panic. ## What changes are included in this PR? ShuffleScanExec now emits such a block as several batches that each fit, and returns an error instead of panicking. Operators that merge batches can still overflow until #7271. ScanExec, the JNI remote decode and the JVM decode still fail with an error on such blocks. ## How are these changes tested? Rust unit tests, including the shape from the issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
