This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 226ea5671ea4 [SPARK-57949][PYTHON][TEST] Use shortest temp base for
direct-worker UDS path to fix macOS AF_UNIX path too long
226ea5671ea4 is described below
commit 226ea5671ea4571ba36bb961b2e0b29ba1e30c14
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Jul 6 17:25:58 2026 +0900
[SPARK-57949][PYTHON][TEST] Use shortest temp base for direct-worker UDS
path to fix macOS AF_UNIX path too long
### What changes were proposed in this pull request?
`TestDirectWorkerDispatcher.shortTempBase` (in
`TestDirectWorkerHelpers.scala`) chooses a
base directory for the direct-worker Unix-domain socket by taking the
**first** writable
entry among `$TMPDIR`, `$TEMP`, `$TMP`, `/tmp`. This PR changes it to pick
the **shortest**
usable base, measured by its real (symlink-resolved) path.
### Why are the changes needed?
On macOS, `$TMPDIR` is a long per-user path like
`/var/folders/k8/j7r3p6cx43xdqhzy2rmp6tqr0000gn/T/` (~49 chars). Combined
with the
`createTempDirectory` random suffix and the `w-<id>.sock` leaf, the bound
socket path
exceeds the **104-byte macOS `sun_path` limit**, so
`PythonUDFWorkerSpecificationSuite`
fails with:
```
OSError: AF_UNIX path too long
at
...TestDirectWorkerDispatcher.throwWorkerExitedBeforeSocket(TestDirectWorkerHelpers.scala:207)
```
This keeps the scheduled `Build / Maven (Scala 2.13, JDK 21, MacOS-26)`
lane red. Picking the
shortest base makes macOS use `/tmp` (which resolves to the short
`/private/tmp`) instead of
the long `$TMPDIR`, keeping the socket path well within the limit. Linux is
unaffected — its
per-user temp dirs are already short — and the choice is now deterministic
(shortest wins)
rather than order-dependent.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
Ran the affected suites on both `macos-14` and `macos-15` GitHub Actions
runners (arm64, the
same short-`$TMPDIR` layout as `macos-26`), where `$TMPDIR` was confirmed
to be the long
`/var/folders/...` path (`len=49`):
- `udf-worker-core`: `DirectWorkerDispatcherSuite` + `WorkerSessionSuite` —
**48 tests, 0 failures**
- `sql/core`: `PythonUDFWorkerSpecificationSuite` — **1 test, 0 failures**
(previously
`AF_UNIX path too long`)
Both runners green:
https://github.com/HyukjinKwon/spark/actions/runs/28765870386
Before this change the same suite fails on `macos-26`:
https://github.com/apache/spark/actions/runs/28753698265
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
Closes #57028 from HyukjinKwon/ci-fix/agent4-macos-uds-path-pr.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../spark/udf/worker/core/TestDirectWorkerHelpers.scala | 15 +++++++++++++--
1 file changed, 13 insertions(+), 2 deletions(-)
diff --git
a/udf/worker/core/src/test/scala/org/apache/spark/udf/worker/core/TestDirectWorkerHelpers.scala
b/udf/worker/core/src/test/scala/org/apache/spark/udf/worker/core/TestDirectWorkerHelpers.scala
index 85b4afc5ed07..7ffe72f91757 100644
---
a/udf/worker/core/src/test/scala/org/apache/spark/udf/worker/core/TestDirectWorkerHelpers.scala
+++
b/udf/worker/core/src/test/scala/org/apache/spark/udf/worker/core/TestDirectWorkerHelpers.scala
@@ -263,9 +263,20 @@ class TestDirectWorkerDispatcher(
private def shortTempBase(): Path = {
val candidates =
Seq("TMPDIR", "TEMP", "TMP").flatMap(sys.env.get).filter(_.nonEmpty) :+
"/tmp"
- candidates
+ val usable = candidates
.map(Paths.get(_))
- .find(p => Files.isDirectory(p) && Files.isWritable(p))
+ .filter(p => Files.isDirectory(p) && Files.isWritable(p))
+ // Pick the SHORTEST usable base, measured by its *real*
(symlink-resolved) path,
+ // because that is what actually counts against the `sun_path` limit when
the socket
+ // is bound. On macOS `$TMPDIR` is a long `/var/folders/<...>/T` path
while `/tmp`
+ // resolves to the short `/private/tmp`; picking the first writable
candidate (the old
+ // behavior) chose the long `$TMPDIR` and overflowed the 104-byte macOS
limit
+ // ("AF_UNIX path too long"). Choosing the shortest real path avoids that.
+ def realLen(p: Path): Int =
+ try p.toRealPath().toString.length catch { case _: IOException =>
p.toString.length }
+ usable
+ .sortBy(realLen)
+ .headOption
.getOrElse(Paths.get(System.getProperty("java.io.tmpdir")))
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
