[
https://issues.apache.org/jira/browse/SPARK-61667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh updated SPARK-61667:
--------------------------------
Affects Version/s: 4.4.0
(was: 5.0.0)
> Use Expression.isTotal to fold the size of CreateArray in ConstantFolding
> -------------------------------------------------------------------------
>
> Key: SPARK-61667
> URL: https://issues.apache.org/jira/browse/SPARK-61667
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: L. C. Hsieh
> Priority: Major
>
> ConstantFolding folds size(array(e1, ..., en)) to n without evaluating the
> elements when none of them can fail. It decides that with its own check,
> hasNoSideEffect, which above attributes, literals and some casts only accepts
> CreateArray and CreateNamedStruct.
> Since SPARK-61663, every expression declares whether it is total with
> Expression.isTotal, and the ones that are always total mix in NoThrow
> (comparisons, And/Or/Not, IsNull, Coalesce, Length, Substring, ...).
> SPARK-61663 kept ConstantFolding as it was by naming CreateArray and
> CreateNamedStruct there, so e.g. size(array(i > 1)) is still evaluated
> although its element cannot fail.
> This is to revisit hasNoSideEffect so that it relies on the declaration
> (NoThrow, or Expression.isTotal for the expressions that are total in some
> eval modes only) instead of the two names. It changes which plans are folded,
> so it needs its own review: the rule skips the evaluation of the elements,
> and a wrong declaration would hide an error (see SPARK-60820 and SPARK-60868).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]