Vladimir Golubev created SPARK-50559:
----------------------------------------
Summary: Store Union, Except and Intersect operator's output as
lazy val
Key: SPARK-50559
URL: https://issues.apache.org/jira/browse/SPARK-50559
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.0.0
Reporter: Vladimir Golubev
Currently `Union`'s (same is for `Except` and `Intersect`) `output` is a `def`.
This creates performance issues for queries with large number of stacked
`UNION`s because of rules like `WidenSetOperationTypes` that traverse the
logical plan and call `output` on each `Union` node. This has quadratic
complexity: O(number_of_unions * (1 + 2 + 3 + ... + number_of_unions)).
The proposed solution is to use a `lazy val`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
