Jungtaek Lim created SPARK-61997:
------------------------------------
Summary: Spark Connect Python foreach writer can't see the
session's artifacts
Key: SPARK-61997
URL: https://issues.apache.org/jira/browse/SPARK-61997
Project: Spark
Issue Type: Bug
Components: Structured Streaming, PySpark, Connect
Affects Versions: 4.0.5, 4.1.4, 4.2.1, 4.3.0, 4.4.0, 5.0.0
Reporter: Jungtaek Lim
A Python {{foreach}} writer in a Spark Connect streaming query can't read files
or import Python files that the session added with {{addArtifacts}}, when the
executors run in their own JVMs.
Cause:
* {{SparkConnectPlanner}} builds {{PythonForeachWriter}} while it handles the
{{writeStream}} request, under the artifact state of the parent session. The
writer keeps that UUID ({{PythonForeachWriter.scala:88}}) and passes it to its
{{PythonRunner}}.
* The batch's tasks run under the artifact state of the stream's session (a
clone with its own UUID), and the executor fetches the session's files into
{{<executor root>/<stream session UUID>}}.
* So the writer's Python worker uses {{<executor root>/<parent session UUID>}}
as its files root and working directory, which has the files only if a task of
the parent session ran on that executor before.
Reproduction on a Spark Connect server with {{local-cluster[2,2,512]}}:
{code:python}
import os, tempfile
d = tempfile.mkdtemp()
with open(os.path.join(d, "probe_file.txt"), "w") as f:
f.write("hello")
spark.addArtifacts(os.path.join(d, "probe_file.txt"), file=True)
input_dir = tempfile.mkdtemp()
with open(os.path.join(input_dir, "input.json"), "w") as f:
f.write('{"id": 0}\n')
def process(row):
from pyspark import SparkFiles
with open(SparkFiles.get("probe_file.txt")) as f:
f.read()
q = (spark.readStream.schema("id BIGINT").json(input_dir)
.writeStream.foreach(process).trigger(availableNow=True).start())
q.awaitTermination()
# FileNotFoundError: ... '<executor root>/<parent session UUID>/probe_file.txt'
{code}
Local mode hides the problem, because the driver already fetched the files into
the parent's folder.
Found in the review of SPARK-61959
(https://github.com/apache/spark/pull/60657), which fixes the same problem for
Python UDFs and Python data sources.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]