L. C. Hsieh created SPARK-60228:
-----------------------------------
Summary: Avoid quadratic cache lookups by table name in
CacheManager
Key: SPARK-60228
URL: https://issues.apache.org/jira/browse/SPARK-60228
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
SPARK-54004 changed CacheManager.isMatchedTableOrView to match on
EliminateSubqueryAliases(plan), which rewrites the whole tree. The check is
evaluated on every node of every cached plan (via plan.exists) when uncaching
or recaching by table name, so each scan rewrites every subtree of every cached
plan. The cost grows super-linearly with the cached plan depth, times the
number of cache entries.
The by-name scan runs on every DSv2 write (refreshCache -> recacheTableOrView)
and on DDL such as DROP / CREATE OR REPLACE / ALTER VIEW. Measured locally for
a by-name scan matching no entry, with 10 cached plans of 1000 chained filters
each, a single scan takes about 40 seconds.
All matched cases only inspect the root node, so only the chain of
SubqueryAlias nodes at the root needs to be stripped.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
