Samuel Shpuntoff created SPARK-60584:
----------------------------------------
Summary: unpersist() waits for unrelated cache materializations to
complete
Key: SPARK-60584
URL: https://issues.apache.org/jira/browse/SPARK-60584
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.4, 4.1.3, 4.2.0, 3.5.8
Environment: Spark 4.2 with Java 25 on ARM64
Reporter: Samuel Shpuntoff
{{Dataset.unpersist()}} and {{Catalog.dropTempView()}} wait until every
in-progress cache materialization on the driver completes, including those
unrelated to the plan being uncached. A single long-running materialization
therefore delays cache cleanup for all other jobs on the driver.
h3. Repro
spark-shell, {{{}local[8]{}}}:
import org.apache.spark.sql.functions._val small = spark.range(10).cache();
small.count()val sleepy = udf((x: Long) => { Thread.sleep(60000); x })val slow
= spark.range(0, 2, 1, 2).select(sleepy(col("id"))).repartition(4).cache()new
Thread(() => slow.count()).start(); Thread.sleep(5000)
small.unpersist() // 4.0.0: ~9 ms. 4.2.0: ~55 s
h3. Cause
{{CacheManager.recacheByCondition}} evaluates {{isCachedColumnBuffersLoaded}}
on every cache entry before checking whether the entry depends on the uncached
plan. Since SPARK-53435, this method acquires the builder's monitor, which is
held for the duration of an in-progress materialization. For unrelated entries
the result is not used.
Affects 4.0.0+ and branch-3.5
([#52199|https://github.com/apache/spark/pull/52199]).
h3. Proposed fix
Evaluate the plan match first:
cd.plan.exists(isMatchedPlan) &&
!cd.cachedRepresentation.cacheBuilder.isCachedColumnBuffersLoaded
With this change the repro completes in ~10 ms and the existing cache test
suites pass. A regression test is included.
A PR will be added as a comment shortly.
----
Generated-by: Claude Code (Claude Opus 5.5)
Github Issue:
[https://github.com/apache/spark/issues/59803]
(This is my first issue, so apologies for any mistakes on the form template)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
