sunchao opened a new pull request, #5561: URL: https://github.com/apache/datafusion-comet/pull/5561
## Which issue does this PR close? Extracted from #5557 while addressing #5555. ## Rationale for this change Spark 4.2 changed the Python worker command protocol. `BasePythonRunner` now writes `runnerConf` and `evalConf` before `writeCommand`, but Comet still writes its legacy configuration map inside `writeCommand`. The Python worker interprets that extra map size as the number of UDFs, so accelerated `mapInArrow` / `mapInPandas` execution fails before it can read the command. The problem is independent of large Arrow offsets and was previously missed because the real-worker PyArrow workflow covered Spark 4.0 and 4.1 but not Spark 4.2. ## What changes are included in this PR? The shared runner extracts its legacy configuration serialization into a hook. Spark 4.0 and 4.1 keep writing that frame in the command exactly as before. The Spark 4.2 subclass instead contributes Comet's worker settings through Spark's native `runnerConf` slot and suppresses the legacy frame. The PyArrow workflow now watches all three version-specific runner files and runs the existing real-worker suite on released PySpark 4.2.0 in addition to Spark 4.0 and 4.1. This makes protocol drift observable at the worker boundary rather than through compilation alone. ## How are these changes tested? Native code was built before the JVM checks; this PR does not change native code. - Spark 4.0 / Scala 2.13 root-reactor package build: **14/14 focused JVM tests passed**, verifying that the shared hook preserves the 4.0 command path. - The Spark 4.2 subclass compiled against the released Spark 4.2 artifacts, and Spark 4.2 Spotless checked all 447 Scala files. - Workflow YAML parsing, matrix assertions for Spark 4.0 / 4.1 / 4.2, Scala formatting/style, and `git diff --check` passed. - The same three-file protocol change passed the Spark 4.2 real-worker job on #5557. This PR's new matrix job will validate the isolated head again. The configured local Maven mirror stalled while resolving `jackson-bom:2.21.2` for a full Spark 4.2 reactor package, so that local scope remains unverified. The mirror configuration was left unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
