[
https://issues.apache.org/jira/browse/SPARK-43883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Chester Jenks resolved SPARK-43883.
---------------------------------------
Resolution: Won't Fix
> CTAS Command Nodes Prevent Some Optimizer Rules From Running
> ------------------------------------------------------------
>
> Key: SPARK-43883
> URL: https://issues.apache.org/jira/browse/SPARK-43883
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.4.0, 3.4.1
> Reporter: Ted Chester Jenks
> Priority: Major
> Attachments: Not Working - Create Table.png, Working - 3.2.0.png,
> Working - No Create Table.png
>
>
> The changes introduced to resolve SPARK-41713 in
> [https://github.com/apache/spark/pull/39220] modified the CTAS commands from
> having a `DataWritingCommand` trait to a `LeafRunnableCommand` trait. The
> `DataWritingCommand` trait extends `UnaryCommand`, and has children set to
> the value of query in the CTAS command. This means that when `transform` is
> called to traverse the tree with the CTAS command at the root, the entire
> query is traversed. `LeafRunnableCommand` has a `LeafLike` trait which
> explicitly sets the value of children to `Nil`. This means that when
> `transform` is called on the command, no children are found and the query is
> unaffected by the rule.
> In practice, this means that optimizer rules that rely on `transform` (such
> as `BooleanSimplification`) to traverse the tree do not work with a CTAS.
> This can be demonstrated with a simple query in spark-shell. Without the CTAS
> we can run a command with an easily simplified boolean expression (`id == 9
> && id == 9`) and see it gets optimized out:
> !Working - No Create Table.png|width=883,height=342!
> With a CTAS, the optimisation does not get applied (as we can see from the
> `AND` still present in the optimized and physical plans):
> !Not Working - Create Table.png|width=885,height=524!
> This works in 3.2.0 which had the old CTAS implementation:
> !Working - 3.2.0.png|width=885,height=345!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
