[
https://issues.apache.org/jira/browse/SPARK-60893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tian Gao updated SPARK-60893:
-----------------------------
Description:
Python workers call importlib.invalidate_caches() in
worker_util.setup_spark_files when they add Spark files to sys.path, which
happens on the first task of every new worker process (and on every task when
spark.python.worker.reuse is disabled).
importlib.invalidate_caches() calls invalidate_caches() on every finder in
sys.path_importer_cache. Each package imported from a zip archive has its own
zipimporter there (pyspark.zip has about 15 after importing pyspark.worker). Up
to Python 3.12, zipimporter.invalidate_caches() re-reads the archive's whole
central directory, so pyspark.zip is re-read once per package imported from it.
On 3.13+ the listing is dropped and re-read once per archive on the next
lookup, which happens right away because invalidate_caches() imports
importlib.metadata.
Spark's own archives, the zips the worker's pyspark and py4j were loaded from
($SPARK_HOME/python/lib/pyspark.zip and the py4j zip, or the copies Spark ships
to YARN containers), are not replaced while an application runs, so their
listings never need to be refreshed. Keep the zipimporters of those archives
when invalidating, and keep invalidating everything else, including user
archives on PYTHONPATH (which may be rebuilt while the Python daemon lives) and
zips added through python_includes (addPyFile, --py-files, Spark Connect
artifacts).
With the spark-core JAR removed from the worker path (SPARK-60855),
setup_spark_files goes from about 40-42 ms to about 5 ms on Python 3.11/3.12,
and from about 5 ms to about 2.3 ms on 3.13/3.14.
was:
Python workers call importlib.invalidate_caches() in
worker_util.setup_spark_files when they add Spark files to sys.path, which
happens on the first task of every new worker process (and on every task when
spark.python.worker.reuse is disabled).
importlib.invalidate_caches() calls invalidate_caches() on every finder in
sys.path_importer_cache. Each package imported from a zip archive has its own
zipimporter there (pyspark.zip has about 15 after importing pyspark.worker). Up
to Python 3.12, zipimporter.invalidate_caches() re-reads the archive's whole
central directory, so pyspark.zip is re-read once per package imported from it.
On 3.13+ the listing is dropped and re-read once per archive on the next
lookup, which happens right away because invalidate_caches() imports
importlib.metadata.
The zip archives that are on sys.path when the worker starts (pyspark.zip, the
py4j zip, and the rest of PYTHONPATH such as YARN-localized --py-files) do not
change while the worker runs, so their listings never need to be refreshed.
Keep the zipimporters of those archives when invalidating, and keep
invalidating everything else, including zips added later through
python_includes (addPyFile, --py-files, Spark Connect artifacts).
setup_spark_files goes from about 40-57 ms to about 5 ms on Python 3.11/3.12,
and from about 13 ms to about 2.3 ms on 3.13/3.14.
Summary: Keep the import caches of Spark's own zip archives when Python
workers invalidate import caches (was: Skip zip archives present at Python
worker startup when invalidating import caches)
> Keep the import caches of Spark's own zip archives when Python workers
> invalidate import caches
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-60893
> URL: https://issues.apache.org/jira/browse/SPARK-60893
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 5.0.0
> Reporter: Tian Gao
> Priority: Major
> Labels: pull-request-available
>
> Python workers call importlib.invalidate_caches() in
> worker_util.setup_spark_files when they add Spark files to sys.path, which
> happens on the first task of every new worker process (and on every task when
> spark.python.worker.reuse is disabled).
> importlib.invalidate_caches() calls invalidate_caches() on every finder in
> sys.path_importer_cache. Each package imported from a zip archive has its own
> zipimporter there (pyspark.zip has about 15 after importing pyspark.worker).
> Up to Python 3.12, zipimporter.invalidate_caches() re-reads the archive's
> whole central directory, so pyspark.zip is re-read once per package imported
> from it. On 3.13+ the listing is dropped and re-read once per archive on the
> next lookup, which happens right away because invalidate_caches() imports
> importlib.metadata.
> Spark's own archives, the zips the worker's pyspark and py4j were loaded from
> ($SPARK_HOME/python/lib/pyspark.zip and the py4j zip, or the copies Spark
> ships to YARN containers), are not replaced while an application runs, so
> their listings never need to be refreshed. Keep the zipimporters of those
> archives when invalidating, and keep invalidating everything else, including
> user archives on PYTHONPATH (which may be rebuilt while the Python daemon
> lives) and zips added through python_includes (addPyFile, --py-files, Spark
> Connect artifacts).
> With the spark-core JAR removed from the worker path (SPARK-60855),
> setup_spark_files goes from about 40-42 ms to about 5 ms on Python 3.11/3.12,
> and from about 5 ms to about 2.3 ms on 3.13/3.14.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
