Hyukjin Kwon created SPARK-57948:
------------------------------------
Summary: Fix pipelined Python UDF data transfer over Unix domain
sockets
Key: SPARK-57948
URL: https://issues.apache.org/jira/browse/SPARK-57948
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
SPARK-56642 added pipelined JVM->Python UDF data transfer via
BasePythonRunner.createPipelinedDataIn, which reads the worker through the
legacy socket API (worker.channel.socket().setSoTimeout / getInputStream). For
a Unix domain SocketChannel, SocketChannel.socket() throws
java.lang.UnsupportedOperationException, so with
spark.python.unix.domain.socket.enabled=true and
spark.python.udf.pipelined.enabled=true every pipelined Arrow/Python UDF fails.
This breaks the scheduled Build / Unix Domain Socket (build_uds) master lane.
Fix: in UDS mode read from the channel via Channels.newInputStream and skip the
SO_TIMEOUT-based idle timeout, matching the existing sync-mode UDS handling in
PythonRunner.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
