Jungtaek Lim created SPARK-61997:
------------------------------------

             Summary: Spark Connect Python foreach writer can't see the 
session's artifacts
                 Key: SPARK-61997
                 URL: https://issues.apache.org/jira/browse/SPARK-61997
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming, PySpark, Connect
    Affects Versions: 4.0.5, 4.1.4, 4.2.1, 4.3.0, 4.4.0, 5.0.0
            Reporter: Jungtaek Lim


A Python {{foreach}} writer in a Spark Connect streaming query can't read files 
or import Python files that the session added with {{addArtifacts}}, when the 
executors run in their own JVMs.

Cause:

* {{SparkConnectPlanner}} builds {{PythonForeachWriter}} while it handles the 
{{writeStream}} request, under the artifact state of the parent session. The 
writer keeps that UUID ({{PythonForeachWriter.scala:88}}) and passes it to its 
{{PythonRunner}}.
* The batch's tasks run under the artifact state of the stream's session (a 
clone with its own UUID), and the executor fetches the session's files into 
{{<executor root>/<stream session UUID>}}.
* So the writer's Python worker uses {{<executor root>/<parent session UUID>}} 
as its files root and working directory, which has the files only if a task of 
the parent session ran on that executor before.

Reproduction on a Spark Connect server with {{local-cluster[2,2,512]}}:

{code:python}
import os, tempfile
d = tempfile.mkdtemp()
with open(os.path.join(d, "probe_file.txt"), "w") as f:
    f.write("hello")
spark.addArtifacts(os.path.join(d, "probe_file.txt"), file=True)

input_dir = tempfile.mkdtemp()
with open(os.path.join(input_dir, "input.json"), "w") as f:
    f.write('{"id": 0}\n')

def process(row):
    from pyspark import SparkFiles
    with open(SparkFiles.get("probe_file.txt")) as f:
        f.read()

q = (spark.readStream.schema("id BIGINT").json(input_dir)
     .writeStream.foreach(process).trigger(availableNow=True).start())
q.awaitTermination()
# FileNotFoundError: ... '<executor root>/<parent session UUID>/probe_file.txt'
{code}

Local mode hides the problem, because the driver already fetched the files into 
the parent's folder.

Found in the review of SPARK-61959 
(https://github.com/apache/spark/pull/60657), which fixes the same problem for 
Python UDFs and Python data sources.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to