linhongliu-db opened a new pull request, #59890: URL: https://github.com/apache/spark/pull/59890
### What changes were proposed in this pull request? Refresh every matching file index in a cached plan before rebuilding the cache. Deduplicate shared indexes by object identity within each cached entry, while preserving the existing path and time-travel checks. ### Why are the changes needed? `recacheByPath` calls a file-index refresh inside a short-circuiting `plan.exists` traversal. For a cached union or join with multiple matching reads, the traversal stops after refreshing the first index. Other reads can retain stale file listings when the cache is rebuilt. For example, after files are added externally under two directories read by one cached union, refreshing their parent directory should make both branches see the new rows. Previously, only the first branch's listing was refreshed. Object identity matters: distinct indexes with equal root paths have independent listings and must each be refreshed. An index shared by several plan nodes needs only one refresh per cached entry. ### Does this PR introduce _any_ user-facing change? Yes. `refreshByPath` updates all matching reads in a cached query, preventing mixed stale and refreshed results. It performs the additional index refreshes needed for those reads; unrelated paths and existing time-travel exclusions retain their behavior. ### How was this patch tested? Added regressions cover one, two, and three matching reads; shared and distinct equal indexes; unrelated paths; repeated refreshes; and multiple cached entries. They verify both complete query results and index refresh counts. Files are added outside Spark's writer so automatic destination refresh cannot hide the defect. All seven `CachedTableSuite` tests matching `refreshByPath` passed with SBT using the ScalaTest runner. Before the production change, the regressions for two reads, three reads, and distinct equal indexes failed with stale results. SQL production and test Scala style checks passed. The suite emitted a possible-thread-leak warning and completed successfully. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex CLI 0.164.0 (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
