[
https://issues.apache.org/jira/browse/SPARK-60228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh resolved SPARK-60228.
---------------------------------
Fix Version/s: 4.3.1
4.4.0
Resolution: Fixed
Issue resolved by pull request 59448
[https://github.com/apache/spark/pull/59448]
> Avoid quadratic cache lookups by table name in CacheManager
> -----------------------------------------------------------
>
> Key: SPARK-60228
> URL: https://issues.apache.org/jira/browse/SPARK-60228
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0, 4.3.0, 4.1.3, 4.4.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.1, 4.4.0
>
>
> SPARK-54004 changed CacheManager.isMatchedTableOrView to match on
> EliminateSubqueryAliases(plan), which rewrites the whole tree. The check is
> evaluated on every node of every cached plan (via plan.exists) when uncaching
> or recaching by table name, so each scan rewrites every subtree of every
> cached plan. The cost grows super-linearly with the cached plan depth, times
> the number of cache entries.
> The by-name scan runs on every DSv2 write (refreshCache ->
> recacheTableOrView) and on DDL such as DROP / CREATE OR REPLACE / ALTER VIEW.
> Measured locally for a by-name scan matching no entry, with 10 cached plans
> of 1000 chained filters each, a single scan takes about 40 seconds.
> All matched cases only inspect the root node, so only the chain of
> SubqueryAlias nodes at the root needs to be stripped.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
