[ 
https://issues.apache.org/jira/browse/SPARK-62000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62000:
-----------------------------------
    Labels: pull-request-available  (was: )

> Python workers of an isolated session can't import Python files given with 
> --py-files or spark.submit.pyFiles
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62000
>                 URL: https://issues.apache.org/jira/browse/SPARK-62000
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect, PySpark, Spark Core
>    Affects Versions: 4.3.0, 5.0.0, 4.2.1, 4.0.5, 4.1.4, 4.4.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>              Labels: pull-request-available
>
> With artifact isolation on (the default in Spark Connect), the Python workers 
> of a session can't import the Python files that the application was started 
> with through {{--py-files}} or {{spark.submit.pyFiles}}. This affects batch 
> and streaming queries. On YARN the files are importable, because YARN puts 
> them on the executors' {{PYTHONPATH}}; on local, standalone and Kubernetes 
> client mode they are not.
> Reproduction, with a Spark Connect server started with a Python file:
> {code}
> echo 'value = "from py-files"' > /tmp/pyfiles/pyfmod.py
> bin/spark-submit --class 
> org.apache.spark.sql.connect.service.SparkConnectServer \
>   --master "local-cluster[1,1,1024]" --py-files /tmp/pyfiles/pyfmod.py 
> spark-internal
> {code}
> {code:python}
> from pyspark.sql import SparkSession
> from pyspark.sql.functions import udf
> spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
> @udf("string")
> def f(x):
>     import pyfmod
>     return pyfmod.value
> spark.range(1).select(f("id")).collect()
> # ModuleNotFoundError: No module named 'pyfmod'
> {code}
> The same happens with {{--conf spark.submit.pyFiles=...}} and with {{--master 
> local[2]}}.
> Cause:
> * In client mode on a cluster manager other than YARN, spark-submit ships 
> {{--py-files}} through {{spark.files}} ("Non-PySpark applications can need 
> Python dependencies"), so they land in the default files directory, 
> {{SparkFiles.getRootDirectory()}}.
> * A Python worker of an isolated session uses {{<root>/<session uuid>}} as 
> its files directory and working directory, and takes its includes only from 
> the session's artifact manager ({{PythonWorkerUtils.writeSparkFiles}}, 
> {{PythonWorkerFactory}}). So it never sees the default directory.
> * On YARN, {{Client}} adds {{--py-files}} to the executors' {{PYTHONPATH}}, 
> which {{PythonWorkerFactory}} passes to every worker, so isolated sessions 
> can import them there.
> The JVM side already shares the application's global dependencies with 
> isolated sessions: SPARK-51537 makes an isolated session's class loader on 
> the executors inherit the default session's class loader, so that the global 
> {{--jars}} stay visible. Python files given at submit time have no such path.
> A possible fix: give the Python workers of an isolated session the default 
> context's Python files too (the entries of {{spark.submit.pyFiles}}, in the 
> default files directory), after the session's own, so that a session's files 
> still take precedence.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to