Jungtaek Lim created SPARK-61959:
------------------------------------
Summary: Python UDFs and Python data sources in a streaming query
can't see the session's artifacts
Key: SPARK-61959
URL: https://issues.apache.org/jira/browse/SPARK-61959
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
A streaming query can't use the files and Python files that the session added
with {{addArtifacts}}. The same code works in a batch query.
This happens when artifact isolation is on, which is the default in Spark
Connect. It affects:
* Python UDFs, with and without Arrow. {{SparkFiles.get}} returns a path in the
shared {{userFiles-...}} folder, not the session's folder. A Python file added
with {{pyfile=True}} can't be imported.
* Python data sources with a stream reader ({{DataSourceStreamReader}}). Their
{{read}} has the same problem.
Reproduction with Spark Connect:
{code:python}
import os, tempfile
from pyspark.sql.functions import udf
d = tempfile.mkdtemp()
with open(os.path.join(d, "probe_file.txt"), "w") as f:
f.write("hello")
spark.addArtifacts(os.path.join(d, "probe_file.txt"), file=True)
@udf("string")
def read_file(i):
from pyspark import SparkFiles
with open(SparkFiles.get("probe_file.txt")) as f:
return f.read()
# Batch: works and returns "hello".
spark.range(1).select(read_file("id")).collect()
# Streaming: fails.
stream_dir = tempfile.mkdtemp()
with open(os.path.join(stream_dir, "input.json"), "w") as f:
f.write('{"id": 0}\n')
q = (spark.readStream.schema("id
BIGINT").json(stream_dir).select(read_file("id"))
.writeStream.format("memory").queryName("probe").trigger(availableNow=True).start())
q.awaitTermination()
# StreamingQueryException: ... FileNotFoundError: [Errno 2] No such file or
directory:
# '.../userFiles-<uuid>/probe_file.txt'
# The file is in '.../userFiles-<uuid>/<session uuid>/probe_file.txt'.
{code}
Cause:
* A streaming query runs its batches in a clone of the session
({{StreamExecution.sparkSessionForStream}}). Since SPARK-49854 the clone has
its own UUID and its own copy of the artifacts.
* SPARK-49249 made {{SQLExecution.withNewExecutionId}} apply the session's
artifact state while it runs the jobs. It also removed the wrapper that set the
caller's artifact state on the stream thread (added by SPARK-44794). That
wrapper set the parent session's state, not the clone's.
* A micro-batch is planned before {{SQLExecution.withNewExecutionId}} starts
({{MicroBatchExecution.runBatch}}). The Python plan nodes (for example
{{BatchEvalPythonExec}} and {{ArrowEvalPythonExec}}) keep the artifact state
that is active when they are built. On the stream thread there is none, so the
Python workers use the shared root folder.
* {{PythonMicroBatchStream.createReaderFactory}} always passes no artifact
state, so streaming Python data source reads use the shared root folder too.
Spark 3.5 is not affected: it still has the SPARK-44794 wrapper, and its
session clone shares the parent's artifacts.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]