Jungtaek Lim created SPARK-62000:
------------------------------------

             Summary: Python workers of an isolated session can't import Python 
files given with --py-files or spark.submit.pyFiles
                 Key: SPARK-62000
                 URL: https://issues.apache.org/jira/browse/SPARK-62000
             Project: Spark
          Issue Type: Bug
          Components: Spark Core, PySpark, Connect
    Affects Versions: 4.0.5, 4.1.4, 4.2.1, 4.3.0, 4.4.0, 5.0.0
            Reporter: Jungtaek Lim


With artifact isolation on (the default in Spark Connect), the Python workers 
of a session can't import the Python files that the application was started 
with through {{--py-files}} or {{spark.submit.pyFiles}}. This affects batch and 
streaming queries. On YARN the files are importable, because YARN puts them on 
the executors' {{PYTHONPATH}}; on local, standalone and Kubernetes client mode 
they are not.

Reproduction, with a Spark Connect server started with a Python file:

{code}
echo 'value = "from py-files"' > /tmp/pyfiles/pyfmod.py
bin/spark-submit --class 
org.apache.spark.sql.connect.service.SparkConnectServer \
  --master "local-cluster[1,1,1024]" --py-files /tmp/pyfiles/pyfmod.py 
spark-internal
{code}

{code:python}
from pyspark.sql import SparkSession
from pyspark.sql.functions import udf

spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()

@udf("string")
def f(x):
    import pyfmod
    return pyfmod.value

spark.range(1).select(f("id")).collect()
# ModuleNotFoundError: No module named 'pyfmod'
{code}

The same happens with {{--conf spark.submit.pyFiles=...}} and with {{--master 
local[2]}}.

Cause:
* In client mode on a cluster manager other than YARN, spark-submit ships 
{{--py-files}} through {{spark.files}} ("Non-PySpark applications can need 
Python dependencies"), so they land in the default files directory, 
{{SparkFiles.getRootDirectory()}}.
* A Python worker of an isolated session uses {{<root>/<session uuid>}} as its 
files directory and working directory, and takes its includes only from the 
session's artifact manager ({{PythonWorkerUtils.writeSparkFiles}}, 
{{PythonWorkerFactory}}). So it never sees the default directory.
* On YARN, {{Client}} adds {{--py-files}} to the executors' {{PYTHONPATH}}, 
which {{PythonWorkerFactory}} passes to every worker, so isolated sessions can 
import them there.

The JVM side already shares the application's global dependencies with isolated 
sessions: SPARK-51537 makes an isolated session's class loader on the executors 
inherit the default session's class loader, so that the global {{--jars}} stay 
visible. Python files given at submit time have no such path.

A possible fix: give the Python workers of an isolated session the default 
context's Python files too (the entries of {{spark.submit.pyFiles}}, in the 
default files directory), after the session's own, so that a session's files 
still take precedence.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to