Ben Hollis created SPARK-59501:
----------------------------------
Summary: Unnecessary allocations in unchanged TreeNode traversals
Key: SPARK-59501
URL: https://issues.apache.org/jira/browse/SPARK-59501
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: Ben Hollis
Catalyst rules repeatedly traverse large, immutable plan and expression trees
that they mostly leave unchanged. Even when a node and its subtree are not
modified, the generic TreeNode traversal still allocates on every node.
This cost is paid by every applicable analyzer and optimizer rule, on every
pass, and scales with tree size, so queries whose analyzed plan expands into
hundreds of branches amplify it. On a synthetic 500-branch compatibility view
(4,001 analyzed plan nodes, ~378K expression occurrences), the unchanged-tree
allocation overhead dominates optimizer time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
