HyukjinKwon opened a new pull request, #56646:
URL: https://github.com/apache/spark/pull/56646
### What changes were proposed in this pull request?
`TestDirectWorkerHelpers` (the test dispatcher shared by the UDF-worker
tests)
creates its private socket directory with `Files.createTempDirectory(...)`,
which resolves under `java.io.tmpdir`. In the Maven CI build `java.io.tmpdir`
points at `<module>/target/tmp` (a deep path, e.g.
`/home/runner/work/spark/spark/sql/core/target/tmp`). Combined with the
generated socket leaf `spark-udf-worker<random>/w-<16hex>.sock`, the
resulting
Unix-domain socket path exceeds the platform `sun_path` limit (108 bytes on
Linux, 104 on macOS).
This PR anchors the socket temp directory under a short base directory
(`/tmp`
when it is a writable directory, otherwise the configured `java.io.tmpdir`),
keeping the full UDS path well within the limit. Only the test helper is
affected.
### Why are the changes needed?
The scheduled Maven builds fail in `PythonUDFWorkerSpecificationSuite` with:
```
- PythonUDFWorkerSpecification.fromPythonFunction produces a spec that
spawns a Python worker *** FAILED ***
org.apache.spark.udf.worker.core.direct.DirectWorkerException: Worker
exited with code 1 before creating socket at
/home/runner/work/spark/spark/sql/core/target/tmp/spark-udf-worker.../w-....sock
...
OSError: AF_UNIX path too long
```
This reproduces deterministically across the JDK 21, JDK 25, ARM and macOS
Maven builds because they all set a deep `java.io.tmpdir`. The fix is
platform-independent and resolves it for all of them.
Failing scheduled runs (examples):
- Build / Maven (Scala 2.13, JDK 21):
https://github.com/apache/spark/actions/runs/27907992223
- Build / Maven (Scala 2.13, JDK 25):
https://github.com/apache/spark/actions/runs/27907953084
- Build / Maven (Scala 2.13, JDK 21, ARM):
https://github.com/apache/spark/actions/runs/27909251524
- Build / Maven (Scala 2.13, JDK 21, MacOS-26):
https://github.com/apache/spark/actions/runs/27916516947
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
Existing `PythonUDFWorkerSpecificationSuite` / `DirectWorkerDispatcherSuite`.
Verified on a fork by triggering the scheduled "Build / Maven (Scala 2.13,
JDK
21)" build against this change (on a separate branch the matrix was
temporarily
reduced to the `sql#core` lane that runs the affected suite, and the workflow
guard/trigger were flipped to run on the fork):
Verification build:
https://github.com/HyukjinKwon/spark/actions/runs/27923741891
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
