viirya opened a new pull request, #59448: URL: https://github.com/apache/spark/pull/59448
### What changes were proposed in this pull request? SPARK-54004 changed `CacheManager.isMatchedTableOrView` to match on `EliminateSubqueryAliases(plan)`, which rewrites the whole tree. The check is evaluated on every node of every cached plan (via `plan.exists`) when uncaching or recaching by table name, so each scan rewrites every subtree of every cached plan. This PR strips only the chain of `SubqueryAlias` nodes at the root, since all matched cases inspect only the root node. `tryRefreshPlan` and `findCachedRelations`, which also only look at the root, use the same helper. The matching result is unchanged. ### Why are the changes needed? The by-name scan runs on every DSv2 write (`refreshCache` -> `recacheTableOrView`) and on DDL such as DROP / CREATE OR REPLACE / ALTER VIEW. With deep cached plans, the cost grows super-linearly. Measured locally for a by-name scan that matches no entry, with plans built from chained filters: | plan depth x entries | before | after | |---|---|---| | 100 x 10 | 26 ms | 0.2 ms | | 300 x 10 | 720 ms | 0.1 ms | | 1000 x 10 | ~40 s | 0.4 ms | `findCachedRelations` is also on the DSv2 table resolution path, where it previously rewrote each cached plan once per lookup. ### Does this PR introduce _any_ user-facing change? No, other than the performance improvement. ### How was this patch tested? New test in `CachedTableSuite` that counts plan copies during by-name cache lookups (fails before this change with 185 copies, passes with 0), plus existing `CachedTableSuite`, `DataSourceV2DataFrameSuite` and `DatasetCacheSuite`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5.5) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
