sshpuntoff opened a new issue, #59803:
URL: https://github.com/apache/spark/issues/59803
`Dataset.unpersist()` and `Catalog.dropTempView()` wait until every
in-progress cache materialization on the driver completes, including those
unrelated to the plan being uncached. A single long-running materialization
therefore delays cache cleanup for all other jobs on the driver.
### Repro
spark-shell, `local[8]`:
```scala
import org.apache.spark.sql.functions._
val small = spark.range(10).cache(); small.count()
val sleepy = udf((x: Long) => { Thread.sleep(60000); x })
val slow = spark.range(0, 2, 1,
2).select(sleepy(col("id"))).repartition(4).cache()
new Thread(() => slow.count()).start(); Thread.sleep(5000)
small.unpersist() // 4.0.0: ~9 ms. 4.2.0: ~55 s
```
### Cause
`CacheManager.recacheByCondition` evaluates `isCachedColumnBuffersLoaded` on
every cache entry before checking whether the entry depends on the uncached
plan. Since SPARK-53435, this method acquires the builder's monitor, which is
held for the duration of an in-progress materialization. For unrelated entries
the result is not used.
Affects 4.0.1+ and branch-3.5 (#52199).
### Proposed fix
Evaluate the plan match first:
```scala
cd.plan.exists(isMatchedPlan) &&
!cd.cachedRepresentation.cacheBuilder.isCachedColumnBuffersLoaded
```
With this change the repro completes in ~10 ms and the existing cache test
suites pass. A regression test is included.
A PR will be added as a comment shortly.
---
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
