[
https://issues.apache.org/jira/browse/SPARK-60893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60893:
-----------------------------------
Labels: pull-request-available (was: )
> Skip zip archives present at Python worker startup when invalidating import
> caches
> ----------------------------------------------------------------------------------
>
> Key: SPARK-60893
> URL: https://issues.apache.org/jira/browse/SPARK-60893
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 5.0.0
> Reporter: Tian Gao
> Priority: Major
> Labels: pull-request-available
>
> Python workers call importlib.invalidate_caches() in
> worker_util.setup_spark_files when they add Spark files to sys.path, which
> happens on the first task of every new worker process (and on every task when
> spark.python.worker.reuse is disabled).
> importlib.invalidate_caches() calls invalidate_caches() on every finder in
> sys.path_importer_cache. Each package imported from a zip archive has its own
> zipimporter there (pyspark.zip has about 15 after importing pyspark.worker).
> Up to Python 3.12, zipimporter.invalidate_caches() re-reads the archive's
> whole central directory, so pyspark.zip is re-read once per package imported
> from it. On 3.13+ the listing is dropped and re-read once per archive on the
> next lookup, which happens right away because invalidate_caches() imports
> importlib.metadata.
> The zip archives that are on sys.path when the worker starts (pyspark.zip,
> the py4j zip, and the rest of PYTHONPATH such as YARN-localized --py-files)
> do not change while the worker runs, so their listings never need to be
> refreshed. Keep the zipimporters of those archives when invalidating, and
> keep invalidating everything else, including zips added later through
> python_includes (addPyFile, --py-files, Spark Connect artifacts).
> setup_spark_files goes from about 40-57 ms to about 5 ms on Python 3.11/3.12,
> and from about 13 ms to about 2.3 ms on 3.13/3.14.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
