joelrobin18 opened a new pull request, #59857: URL: https://github.com/apache/spark/pull/59857
### What changes were proposed in this pull request? Follow-up to #58907 (SPARK-59631). `CacheManager.recacheTableOrView` and `CacheManager.recacheByPath` now match cache entries through subquery expressions as well as plan children, using a small `existsWithSubqueries` helper built on `collectFirstWithSubqueries`. ### Why are the changes needed? SPARK-59631 made write-driven invalidation (`includeTimeTravel = false`) skip V1 relations whose `FileIndex.isTimeTravel` is true. Both routes test each cache entry with `plan.exists`, which only walks plan children, so a relation read inside an `IN` / `EXISTS` / scalar subquery is never visited. Before SPARK-59631, a pinned relation of the same table matched by name, and that match also covered a live read of the table inside a subquery. Now the pinned relation is skipped and the live read is never reached. For example, a cached ```sql SELECT * FROM t VERSION AS OF 5 WHERE id NOT IN (SELECT id FROM t) ``` is no longer invalidated by a write to `t`, and keeps returning results computed from the old live data without any error. The path route has the same gap. ### Does this PR introduce _any_ user-facing change? Yes, a bug fix. After a write, cached plans that read the written table inside a subquery are rebuilt. This fixes stale cached results introduced by SPARK-59631 for plans that also read the table at a pinned version. It also covers entries that read the table only inside a subquery, which previously were not invalidated either. SPARK-59631 is on master, branch-4.x and branch-4.3, so this follow-up should go to the same branches. ### How was this patch tested? Added two tests to `V1TimeTravelCacheSuite`, one per route (`recacheTableOrView` and `recacheByPath`). Each caches a query that reads the table at a pinned version and live only inside an `IN` subquery, then checks that write-driven invalidation rebuilds the cache; the path test also checks that only the live index is refreshed. Both tests fail without this change (the cache stays loaded). Also ran, all passing: - `V1TimeTravelCacheSuite`, `CachedTableSuite`, `DatasetCacheSuite`, `MergeIntoDataFrameSuite`, `DataSourceV2SQLSuiteV1Filter`, `DataSourceV2SQLSuiteV2Filter` (683 tests) - `org.apache.spark.sql.hive.CachedTableSuite` (24 tests) - `sql/scalastyle` and `sql/Test/scalastyle` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Isaac (Claude Opus 5.5) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
