Jungtaek Lim created SPARK-62000:
------------------------------------
Summary: Python workers of an isolated session can't import Python
files given with --py-files or spark.submit.pyFiles
Key: SPARK-62000
URL: https://issues.apache.org/jira/browse/SPARK-62000
Project: Spark
Issue Type: Bug
Components: Spark Core, PySpark, Connect
Affects Versions: 4.0.5, 4.1.4, 4.2.1, 4.3.0, 4.4.0, 5.0.0
Reporter: Jungtaek Lim
With artifact isolation on (the default in Spark Connect), the Python workers
of a session can't import the Python files that the application was started
with through {{--py-files}} or {{spark.submit.pyFiles}}. This affects batch and
streaming queries. On YARN the files are importable, because YARN puts them on
the executors' {{PYTHONPATH}}; on local, standalone and Kubernetes client mode
they are not.
Reproduction, with a Spark Connect server started with a Python file:
{code}
echo 'value = "from py-files"' > /tmp/pyfiles/pyfmod.py
bin/spark-submit --class
org.apache.spark.sql.connect.service.SparkConnectServer \
--master "local-cluster[1,1,1024]" --py-files /tmp/pyfiles/pyfmod.py
spark-internal
{code}
{code:python}
from pyspark.sql import SparkSession
from pyspark.sql.functions import udf
spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
@udf("string")
def f(x):
import pyfmod
return pyfmod.value
spark.range(1).select(f("id")).collect()
# ModuleNotFoundError: No module named 'pyfmod'
{code}
The same happens with {{--conf spark.submit.pyFiles=...}} and with {{--master
local[2]}}.
Cause:
* In client mode on a cluster manager other than YARN, spark-submit ships
{{--py-files}} through {{spark.files}} ("Non-PySpark applications can need
Python dependencies"), so they land in the default files directory,
{{SparkFiles.getRootDirectory()}}.
* A Python worker of an isolated session uses {{<root>/<session uuid>}} as its
files directory and working directory, and takes its includes only from the
session's artifact manager ({{PythonWorkerUtils.writeSparkFiles}},
{{PythonWorkerFactory}}). So it never sees the default directory.
* On YARN, {{Client}} adds {{--py-files}} to the executors' {{PYTHONPATH}},
which {{PythonWorkerFactory}} passes to every worker, so isolated sessions can
import them there.
The JVM side already shares the application's global dependencies with isolated
sessions: SPARK-51537 makes an isolated session's class loader on the executors
inherit the default session's class loader, so that the global {{--jars}} stay
visible. Python files given at submit time have no such path.
A possible fix: give the Python workers of an isolated session the default
context's Python files too (the entries of {{spark.submit.pyFiles}}, in the
default files directory), after the session's own, so that a session's files
still take precedence.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]