This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 3eb19a8e3e4e [SPARK-57602][CORE][TEST] Use a short base directory for
UDF worker test sockets to avoid AF_UNIX path-too-long
3eb19a8e3e4e is described below
commit 3eb19a8e3e4ee2a8f331698a6c5b444622962198
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Tue Jun 23 12:40:47 2026 +0900
[SPARK-57602][CORE][TEST] Use a short base directory for UDF worker test
sockets to avoid AF_UNIX path-too-long
### What changes were proposed in this pull request?
`TestDirectWorkerHelpers` (the test dispatcher shared by the UDF-worker
tests)
creates its private socket directory with `Files.createTempDirectory(...)`,
which resolves under `java.io.tmpdir`. In the Maven CI build
`java.io.tmpdir`
points at `<module>/target/tmp` (a deep path, e.g.
`/home/runner/work/spark/spark/sql/core/target/tmp`). Combined with the
generated socket leaf `spark-udf-worker<random>/w-<16hex>.sock`, the
resulting
Unix-domain socket path exceeds the platform `sun_path` limit (108 bytes on
Linux, 104 on macOS).
This PR anchors the socket temp directory under a short base directory
(`/tmp`
when it is a writable directory, otherwise the configured `java.io.tmpdir`),
keeping the full UDS path well within the limit. Only the test helper is
affected.
### Why are the changes needed?
The scheduled Maven builds fail in `PythonUDFWorkerSpecificationSuite` with:
```
- PythonUDFWorkerSpecification.fromPythonFunction produces a spec that
spawns a Python worker *** FAILED ***
org.apache.spark.udf.worker.core.direct.DirectWorkerException: Worker
exited with code 1 before creating socket at
/home/runner/work/spark/spark/sql/core/target/tmp/spark-udf-worker.../w-....sock
...
OSError: AF_UNIX path too long
```
This reproduces deterministically across the JDK 21, JDK 25, ARM and macOS
Maven builds because they all set a deep `java.io.tmpdir`. The fix is
platform-independent and resolves it for all of them.
Failing scheduled runs (examples):
- Build / Maven (Scala 2.13, JDK 21):
https://github.com/apache/spark/actions/runs/27907992223
- Build / Maven (Scala 2.13, JDK 25):
https://github.com/apache/spark/actions/runs/27907953084
- Build / Maven (Scala 2.13, JDK 21, ARM):
https://github.com/apache/spark/actions/runs/27909251524
- Build / Maven (Scala 2.13, JDK 21, MacOS-26):
https://github.com/apache/spark/actions/runs/27916516947
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
Existing `PythonUDFWorkerSpecificationSuite` /
`DirectWorkerDispatcherSuite`.
Verified on a fork by triggering the scheduled "Build / Maven (Scala 2.13,
JDK
21)" build against this change (on a separate branch the matrix was
temporarily
reduced to the `sql#core` lane that runs the affected suite, and the
workflow
guard/trigger were flipped to run on the fork). The build is **green** and
`PythonUDFWorkerSpecificationSuite` now passes (previously
`OSError: AF_UNIX path too long`):
- Green verification build:
https://github.com/HyukjinKwon/spark/actions/runs/27923741891
- `sql#core` lane (runs the suite):
https://github.com/HyukjinKwon/spark/actions/runs/27923741891/job/82625724873
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #56646 from HyukjinKwon/fix-udf-worker-uds-socket-path.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit 065c9f618f2abdc83988e27645563684e26e63a6)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../udf/worker/core/TestDirectWorkerHelpers.scala | 36 ++++++++++++++++++++--
1 file changed, 33 insertions(+), 3 deletions(-)
diff --git
a/udf/worker/core/src/test/scala/org/apache/spark/udf/worker/core/TestDirectWorkerHelpers.scala
b/udf/worker/core/src/test/scala/org/apache/spark/udf/worker/core/TestDirectWorkerHelpers.scala
index ca7c86a82a26..85b4afc5ed07 100644
---
a/udf/worker/core/src/test/scala/org/apache/spark/udf/worker/core/TestDirectWorkerHelpers.scala
+++
b/udf/worker/core/src/test/scala/org/apache/spark/udf/worker/core/TestDirectWorkerHelpers.scala
@@ -17,7 +17,7 @@
package org.apache.spark.udf.worker.core
import java.io.{File, IOException}
-import java.nio.file.{Files, FileVisitResult, Path, SimpleFileVisitor}
+import java.nio.file.{Files, FileVisitResult, Path, Paths, SimpleFileVisitor}
import java.nio.file.attribute.{BasicFileAttributes, PosixFilePermissions}
import org.apache.spark.udf.worker.{Cancel, DataRequest, DataResponse, Finish,
FinishResponse,
@@ -213,15 +213,23 @@ class TestDirectWorkerDispatcher(
* Creates a private (owner-only, 0700) temp directory for worker sockets.
* On POSIX filesystems the permissions are applied atomically at creation;
* the non-POSIX branch tightens best-effort after creation.
+ *
+ * The directory is anchored under a short base path (see [[shortTempBase]])
+ * rather than the configured `java.io.tmpdir`. Builds point the latter at a
+ * deep location (e.g. `<module>/target/tmp`), which combined with the
+ * generated socket leaf would push the worker's Unix-domain socket path past
+ * the platform `sun_path` limit (108 bytes on Linux, 104 on macOS) and fail
+ * with "AF_UNIX path too long".
*/
private def createPrivateTempDirectory(): Path = {
val attr = PosixFilePermissions.asFileAttribute(
PosixFilePermissions.fromString("rwx------"))
+ val base = shortTempBase()
try {
- Files.createTempDirectory("spark-udf-worker", attr)
+ Files.createTempDirectory(base, "spark-udf-worker", attr)
} catch {
case _: UnsupportedOperationException =>
- val dir = Files.createTempDirectory("spark-udf-worker")
+ val dir = Files.createTempDirectory(base, "spark-udf-worker")
val f = dir.toFile
// Bit-wise AND (NOT &&): all six setters must run even if an earlier
// one returns false, so the final permission state matches owner-only.
@@ -238,4 +246,26 @@ class TestDirectWorkerDispatcher(
dir
}
}
+
+ /**
+ * Picks the shortest usable base directory for the socket temp dir so the
+ * resulting Unix-domain socket path stays within the platform `sun_path`
+ * limit (108 bytes on Linux, 104 on macOS).
+ *
+ * This mirrors how PySpark already chooses its UDS directory in
+ * `python/run-tests.py` (`spark.python.unix.domain.socket.dir`): prefer the
+ * OS temp-dir env vars (`TMPDIR`/`TEMP`/`TMP`) and fall back to `/tmp`.
Builds
+ * point `java.io.tmpdir` at a deep `<module>/target/tmp`, which combined
with
+ * the generated socket leaf would overflow `sun_path`; the OS temp dir is
+ * short (e.g. a per-user `/tmp/...` on Linux runners, `/var/folders/...` via
+ * `$TMPDIR` on macOS). `java.io.tmpdir` remains the last-resort fallback.
+ */
+ private def shortTempBase(): Path = {
+ val candidates =
+ Seq("TMPDIR", "TEMP", "TMP").flatMap(sys.env.get).filter(_.nonEmpty) :+
"/tmp"
+ candidates
+ .map(Paths.get(_))
+ .find(p => Files.isDirectory(p) && Files.isWritable(p))
+ .getOrElse(Paths.get(System.getProperty("java.io.tmpdir")))
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
