Qianlan Chen created SPARK-60012:
------------------------------------
Summary: Avoid copying Ivy caches when cloning ArtifactManager
Key: SPARK-60012
URL: https://issues.apache.org/jira/browse/SPARK-60012
Project: Spark
Issue Type: Bug
Components: Connect, SQL
Affects Versions: 4.4.0
Reporter: Qianlan Chen
SPARK-59553 added per-session Maven resolution to ArtifactManager. The resolver
stores its Ivy cache at \{{artifactPath/.ivy-cache}}.
\{{ArtifactManager.clone()}} recursively copies the entire parent
\{{artifactPath}}. After Maven resolution, this copies both:
* The disposable Ivy cache, including downloaded and retrieved jars.
* The installed session artifacts under \{{jars/}}, \{{classes/}}, and other
artifact directories.
For a large transitive dependency tree, every initialized cloned session can
therefore incur unnecessary disk usage and clone latency.
h2. Proposed change
Update \{{ArtifactManager.clone()}} to exclude the top-level \{{.ivy-cache}}
directory while continuing to copy installed session artifacts.
The cloned session should lazily create a fresh \{{.ivy-cache}} if it later
performs Maven resolution.
h2. Acceptance criteria
* Installed jars, classes, and Python artifacts remain available in the
cloned session.
* The parent's \{{.ivy-cache}} is not copied.
* Maven resolution in the cloned session still succeeds and creates a new
cache when needed.
* Add a regression test covering these behaviors.
h2. Related work
* SPARK-59553
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
