Jungtaek Lim created SPARK-61959:
------------------------------------

             Summary: Python UDFs and Python data sources in a streaming query 
can't see the session's artifacts
                 Key: SPARK-61959
                 URL: https://issues.apache.org/jira/browse/SPARK-61959
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


A streaming query can't use the files and Python files that the session added 
with {{addArtifacts}}. The same code works in a batch query.

This happens when artifact isolation is on, which is the default in Spark 
Connect. It affects:

* Python UDFs, with and without Arrow. {{SparkFiles.get}} returns a path in the 
shared {{userFiles-...}} folder, not the session's folder. A Python file added 
with {{pyfile=True}} can't be imported.
* Python data sources with a stream reader ({{DataSourceStreamReader}}). Their 
{{read}} has the same problem.

Reproduction with Spark Connect:

{code:python}
import os, tempfile
from pyspark.sql.functions import udf

d = tempfile.mkdtemp()
with open(os.path.join(d, "probe_file.txt"), "w") as f:
    f.write("hello")
spark.addArtifacts(os.path.join(d, "probe_file.txt"), file=True)

@udf("string")
def read_file(i):
    from pyspark import SparkFiles
    with open(SparkFiles.get("probe_file.txt")) as f:
        return f.read()

# Batch: works and returns "hello".
spark.range(1).select(read_file("id")).collect()

# Streaming: fails.
stream_dir = tempfile.mkdtemp()
with open(os.path.join(stream_dir, "input.json"), "w") as f:
    f.write('{"id": 0}\n')
q = (spark.readStream.schema("id 
BIGINT").json(stream_dir).select(read_file("id"))
     
.writeStream.format("memory").queryName("probe").trigger(availableNow=True).start())
q.awaitTermination()
# StreamingQueryException: ... FileNotFoundError: [Errno 2] No such file or 
directory:
#   '.../userFiles-<uuid>/probe_file.txt'
# The file is in '.../userFiles-<uuid>/<session uuid>/probe_file.txt'.
{code}

Cause:

* A streaming query runs its batches in a clone of the session 
({{StreamExecution.sparkSessionForStream}}). Since SPARK-49854 the clone has 
its own UUID and its own copy of the artifacts.
* SPARK-49249 made {{SQLExecution.withNewExecutionId}} apply the session's 
artifact state while it runs the jobs. It also removed the wrapper that set the 
caller's artifact state on the stream thread (added by SPARK-44794). That 
wrapper set the parent session's state, not the clone's.
* A micro-batch is planned before {{SQLExecution.withNewExecutionId}} starts 
({{MicroBatchExecution.runBatch}}). The Python plan nodes (for example 
{{BatchEvalPythonExec}} and {{ArrowEvalPythonExec}}) keep the artifact state 
that is active when they are built. On the stream thread there is none, so the 
Python workers use the shared root folder.
* {{PythonMicroBatchStream.createReaderFactory}} always passes no artifact 
state, so streaming Python data source reads use the shared root folder too.

Spark 3.5 is not affected: it still has the SPARK-44794 wrapper, and its 
session clone shares the parent's artifacts.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to