linhongliu-db commented on code in PR #59890:
URL: https://github.com/apache/spark/pull/59890#discussion_r4472569474
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala:
##########
@@ -634,26 +636,45 @@ class CacheManager extends Logging with
AdaptiveSparkPlanHelper {
fs: FileSystem,
includeTimeTravel: Boolean): Unit = {
val qualifiedPath = fs.makeQualified(resourcePath)
+ // recacheByCondition eagerly filters all entries before rebuilding any of
them, so an
+ // index shared across entries needs refreshing only once during this
call. Use object
+ // identity: equal root paths can have independent file-listing caches.
+ val refreshedIndexes = Collections.newSetFromMap(
+ new IdentityHashMap[FileIndex, java.lang.Boolean]())
recacheByCondition(
spark,
- _.plan.exists(lookupAndRefresh(_, fs, qualifiedPath, includeTimeTravel)))
+ cached => {
+ var refreshed = false
+ // lookupAndRefresh has side effects: exists would stop after the
first matching read.
+ // Refresh matching indexes in the operator tree before rebuilding a
multi-read cache.
+ // As before, this traversal does not enter plans held by subquery
expressions.
+ cached.plan.foreach { plan =>
Review Comment:
Thanks for identifying this remaining gap. This PR fixes the
short-circuiting traversal of the cached plan's operator tree. Reads held by
subquery expressions have the same behavior on the base branch, and the PR
description and source comment explicitly retain that scope, following the
option discussed in the previous review. Traversing subqueries would also
select additional cache entries for refreshByPath and write-driven re-caching,
so it expands invalidation behavior and needs its own regression coverage. I
suggest addressing that in a separate follow-up and keeping this PR focused on
refreshing every matching index reached by the existing traversal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
