viirya opened a new pull request, #59769: URL: https://github.com/apache/spark/pull/59769
### What changes were proposed in this pull request? This is a backport of #59616 to branch-4.2. It fixes the sliced Arrow output processors of the Python runners, which are used when the internal config `spark.sql.execution.arrow.maxRecordsPerOutputBatch` or `spark.sql.execution.arrow.maxBytesPerOutputBatch` is set, and disables the slicing for `ApplyInPandasWithStatePythonRunner`, `ArrowPythonUDTFRunner` and `ColumnarArrowPythonWithNamedArgumentRunner`, whose consumers need whole batches. See #59616 for the details. Differences from the master commit: - `ApplyInPandasWithStatePythonRunner`: the conflict with the `envVars` override (SPARK-59233), which branch-4.2 does not have, is resolved by keeping only the two new overrides. - `test_arrow_python_udf_cached.py` does not exist on branch-4.2, so its new test is dropped. The override in `ColumnarArrowPythonWithNamedArgumentRunner` is kept, since branch-4.2 has the columnar input path. ### Why are the changes needed? With either config set, Python UDF output could contain duplicated rows or wrong values, or fail with `IndexOutOfBoundsException`, and `applyInPandasWithState`, Arrow Python UDTFs and the columnar input path of Arrow Python UDFs could fail or return wrong results. See #59616. ### Does this PR introduce _any_ user-facing change? Yes, only for users who set one of the internal configs above (both are disabled by default): the affected operators no longer return duplicated rows or wrong values, or fail. ### How was this patch tested? The tests from #59616, except the one in `test_arrow_python_udf_cached.py`: `PythonArrowOutputSuite`, and the new tests in `test_arrow_map.py`, `test_arrow_python_udf.py`, `test_udtf.py` and `test_pandas_grouped_map_with_state.py`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
