[
https://issues.apache.org/jira/browse/SPARK-62000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-62000:
-----------------------------------
Labels: pull-request-available (was: )
> Python workers of an isolated session can't import Python files given with
> --py-files or spark.submit.pyFiles
> -------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62000
> URL: https://issues.apache.org/jira/browse/SPARK-62000
> Project: Spark
> Issue Type: Bug
> Components: Connect, PySpark, Spark Core
> Affects Versions: 4.3.0, 5.0.0, 4.2.1, 4.0.5, 4.1.4, 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
> Labels: pull-request-available
>
> With artifact isolation on (the default in Spark Connect), the Python workers
> of a session can't import the Python files that the application was started
> with through {{--py-files}} or {{spark.submit.pyFiles}}. This affects batch
> and streaming queries. On YARN the files are importable, because YARN puts
> them on the executors' {{PYTHONPATH}}; on local, standalone and Kubernetes
> client mode they are not.
> Reproduction, with a Spark Connect server started with a Python file:
> {code}
> echo 'value = "from py-files"' > /tmp/pyfiles/pyfmod.py
> bin/spark-submit --class
> org.apache.spark.sql.connect.service.SparkConnectServer \
> --master "local-cluster[1,1,1024]" --py-files /tmp/pyfiles/pyfmod.py
> spark-internal
> {code}
> {code:python}
> from pyspark.sql import SparkSession
> from pyspark.sql.functions import udf
> spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
> @udf("string")
> def f(x):
> import pyfmod
> return pyfmod.value
> spark.range(1).select(f("id")).collect()
> # ModuleNotFoundError: No module named 'pyfmod'
> {code}
> The same happens with {{--conf spark.submit.pyFiles=...}} and with {{--master
> local[2]}}.
> Cause:
> * In client mode on a cluster manager other than YARN, spark-submit ships
> {{--py-files}} through {{spark.files}} ("Non-PySpark applications can need
> Python dependencies"), so they land in the default files directory,
> {{SparkFiles.getRootDirectory()}}.
> * A Python worker of an isolated session uses {{<root>/<session uuid>}} as
> its files directory and working directory, and takes its includes only from
> the session's artifact manager ({{PythonWorkerUtils.writeSparkFiles}},
> {{PythonWorkerFactory}}). So it never sees the default directory.
> * On YARN, {{Client}} adds {{--py-files}} to the executors' {{PYTHONPATH}},
> which {{PythonWorkerFactory}} passes to every worker, so isolated sessions
> can import them there.
> The JVM side already shares the application's global dependencies with
> isolated sessions: SPARK-51537 makes an isolated session's class loader on
> the executors inherit the default session's class loader, so that the global
> {{--jars}} stay visible. Python files given at submit time have no such path.
> A possible fix: give the Python workers of an isolated session the default
> context's Python files too (the entries of {{spark.submit.pyFiles}}, in the
> default files directory), after the session's own, so that a session's files
> still take precedence.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]