viirya opened a new pull request, #59937: URL: https://github.com/apache/spark/pull/59937
### What changes were proposed in this pull request? This is a backport of #59786 to branch-4.3. It fixes how `SliceBytesArrowOutputProcessorImpl`, used when the internal config `spark.sql.execution.arrow.maxBytesPerOutputBatch` is set, measures an output batch and compares it with the limit: the size is computed as Long so a batch over 2 GiB no longer measures negative, both checks use `<=`, the shrink search no longer restarts from all the remaining rows, a negative limit no longer wraps around to a positive one, and a batch that Arrow cannot slice is output whole using `BaseSliceArrowOutputProcessor.canSlice` from SPARK-60627, which is already on branch-4.3. See #59786 for the details. ### Why are the changes needed? With the config set, a batch over 2 GiB could be output without slicing, a batch whose size equals the limit was sliced although the doc allows it, slicing one batch copied O(R^2/c) offsets, and a negative value could become a positive limit. See #59786. ### Does this PR introduce _any_ user-facing change? Only for the internal config `spark.sql.execution.arrow.maxBytesPerOutputBatch`, which is disabled by default: batches over 2 GiB are now sliced unless Arrow cannot slice them, a batch whose size equals the limit is no longer sliced, and values below `Int.MinValue` mean no limit. ### How was this patch tested? The tests from #59786 in `PythonArrowOutputSuite`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
