viirya opened a new pull request, #57419: URL: https://github.com/apache/spark/pull/57419
### What changes were proposed in this pull request? Backport of the Scala fix from #57403 (master commit e6d66abda0a) to branch-4.2. `ColumnarArrowPythonWithNamedArgumentRunner` still wrote `schema.json` as a UTF string at the head of the UDF command section for `SQL_ARROW_BATCHED_UDF`. SPARK-56340 -- which is in branch-4.2 -- moved that schema into `evalConf` under the `"input_type"` key (branch-4.2's worker reads it from there, and branch-4.2's row-based `ArrowPythonWithNamedArgumentRunner` already writes it there), so the worker parses the command section as a UDF count followed by UDF entries. The stray schema string desyncs the parse and the worker blocks waiting for bytes that never arrive, hanging the task forever. This aligns the columnar runner with the row-based one: schema via `evalConf`, `writeUDF` writes only the UDF list. Scope differences from the master PR, both verified against branch-4.2: - The Python-side fail-fast hardening is not applicable: the message-receiver files (`pyspark/messages/`, `worker_message.py`) do not exist on branch-4.2 (the message-based protocol refactor is 4.3+). - The end-to-end test is not portable: it drives the columnar runner through the Arrow-cached scan (SPARK-57268), which is 4.3+; branch-4.2 has no in-tree producer of Arrow-backed columnar input, so the bug there is reachable only via Arrow-backed DSv2 connectors. ### Why are the changes needed? Both origins of the bug are in branch-4.2: SPARK-56340 (protocol migration, reader side) and SPARK-56350 (the columnar runner with the legacy write). Any Python UDF whose input plan produces Arrow-backed `ColumnarBatch` (e.g. an Arrow-backed DSv2 connector) deadlocks on 4.2 exactly as on master. ### Does this PR introduce _any_ user-facing change? No. It fixes a hang; the wire format now matches what the branch-4.2 worker expects. ### How was this patch tested? The fix mirrors the `evalConf` pattern already shipping in branch-4.2's row-based runner. Compile and scalastyle pass on branch-4.2, and `pyspark.sql.tests.arrow.test_arrow_python_udf` passes against the patched build. The scenario-level end-to-end coverage lives in the master PR (#57403), where the Arrow-cached scan provides an in-tree columnar Arrow input; branch-4.2 has no in-tree way to construct that input. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code This pull request and its description were written by Claude Code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
