[
https://issues.apache.org/jira/browse/SPARK-60012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60012:
-----------------------------------
Labels: pull-request-available (was: )
> Avoid copying Ivy caches when cloning ArtifactManager
> -----------------------------------------------------
>
> Key: SPARK-60012
> URL: https://issues.apache.org/jira/browse/SPARK-60012
> Project: Spark
> Issue Type: Bug
> Components: Connect, SQL
> Affects Versions: 4.4.0
> Reporter: Qianlan Chen
> Priority: Major
> Labels: pull-request-available
>
> SPARK-59553 added per-session Maven resolution to ArtifactManager. The
> resolver stores its Ivy cache at {{{}artifactPath/.ivy-cache{}}}.
> {{ArtifactManager.clone()}} recursively copies the entire parent
> {{{}artifactPath{}}}. After Maven resolution, this copies both:
> * The disposable Ivy cache, including downloaded and retrieved jars.
> * The installed session artifacts under {{{}jars/{}}}, {{{}classes/{}}},
> and other artifact directories.
> For a large transitive dependency tree, every initialized cloned session can
> therefore incur unnecessary disk usage and clone latency.
> h2. Proposed change
> Update {{ArtifactManager.clone()}} to exclude the top-level {{.ivy-cache}}
> directory while continuing to copy installed session artifacts.
> The cloned session should lazily create a fresh {{.ivy-cache}} if it later
> performs Maven resolution.
> h2. Acceptance criteria
> * Installed jars, classes, and Python artifacts remain available in the
> cloned session.
> * The parent's {{.ivy-cache}} is not copied.
> * Maven resolution in the cloned session still succeeds and creates a new
> cache when needed.
> * Add a regression test covering these behaviors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
