Tian Gao created SPARK-60855:
--------------------------------
Summary: Remove the Spark JAR from the Python worker PYTHONPATH
Key: SPARK-60855
URL: https://issues.apache.org/jira/browse/SPARK-60855
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 5.0.0
Reporter: Tian Gao
PythonUtils.sparkPythonPath adds SparkContext.jarOfObject(this), i.e. the
spark-core JAR, to the PYTHONPATH of every Python process Spark launches
(Python workers, PythonRunner, Python UDF workers).
That entry dates from SPARK-1549 (Spark 1.0), when SPARK-1004 packaged
pyspark/*.py and py4j into the core/assembly JAR so PySpark on YARN could
import them from the JAR. SPARK-6869 started shipping pyspark.zip and the py4j
zip to YARN separately, and SPARK-7485 (Spark 1.4) removed the Python files
from the JAR, but the PYTHONPATH entry was never removed. No Spark build has
put Python sources into a JAR since 1.4 (the spark-core 4.0.0 JAR has 5,261
entries and no .py/.pyc files).
The entry is not free. Python creates a zipimporter for it and reads the JAR's
central directory, and every top-level import that misses pyspark.zip and py4j
probes it. Python workers also call importlib.invalidate_caches() when they add
Spark files to sys.path; that re-reads the JAR directory once per new worker,
on Python 3.11-3.12 directly and on 3.13+ lazily through the importlib.metadata
import done by invalidate_caches(). In setup_spark_files this costs about 7-15
ms per new worker (e.g. 13.5 ms -> 4.9 ms on Python 3.13).
Remove the entry.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
