[
https://issues.apache.org/jira/browse/SPARK-60893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tian Gao updated SPARK-60893:
-----------------------------
Affects Version/s: 4.4.0
(was: 5.0.0)
> Keep the import caches of Spark's own zip archives when Python workers
> invalidate import caches
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-60893
> URL: https://issues.apache.org/jira/browse/SPARK-60893
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.4.0
> Reporter: Tian Gao
> Assignee: Tian Gao
> Priority: Major
> Labels: pull-request-available
>
> Python workers call importlib.invalidate_caches() in
> worker_util.setup_spark_files when they add Spark files to sys.path, which
> happens on the first task of every new worker process (and on every task when
> spark.python.worker.reuse is disabled).
> importlib.invalidate_caches() calls invalidate_caches() on every finder in
> sys.path_importer_cache. Each package imported from a zip archive has its own
> zipimporter there (pyspark.zip has about 15 after importing pyspark.worker).
> Up to Python 3.12, zipimporter.invalidate_caches() re-reads the archive's
> whole central directory, so pyspark.zip is re-read once per package imported
> from it. On 3.13+ the listing is dropped and re-read once per archive on the
> next lookup, which happens right away because invalidate_caches() imports
> importlib.metadata.
> Spark's own archives, the zips the worker's pyspark and py4j were loaded from
> ($SPARK_HOME/python/lib/pyspark.zip and the py4j zip, or the copies Spark
> ships to YARN containers), are not replaced while an application runs, so
> their listings never need to be refreshed. Keep the zipimporters of those
> archives when invalidating, and keep invalidating everything else, including
> user archives on PYTHONPATH (which may be rebuilt while the Python daemon
> lives) and zips added through python_includes (addPyFile, --py-files, Spark
> Connect artifacts).
> With the spark-core JAR removed from the worker path (SPARK-60855),
> setup_spark_files goes from about 40-42 ms to about 5 ms on Python 3.11/3.12,
> and from about 5 ms to about 2.3 ms on 3.13/3.14.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]