[ 
https://issues.apache.org/jira/browse/SPARK-61667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

L. C. Hsieh reassigned SPARK-61667:
-----------------------------------

    Assignee: L. C. Hsieh

> Use Expression.isTotal to fold the size of CreateArray in ConstantFolding
> -------------------------------------------------------------------------
>
>                 Key: SPARK-61667
>                 URL: https://issues.apache.org/jira/browse/SPARK-61667
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.4.0
>            Reporter: L. C. Hsieh
>            Assignee: L. C. Hsieh
>            Priority: Major
>
> ConstantFolding folds size(array(e1, ..., en)) to n without evaluating the 
> elements when none of them can fail. It decides that with its own check, 
> hasNoSideEffect, which above attributes, literals and some casts only accepts 
> CreateArray and CreateNamedStruct.
> Since SPARK-61663, every expression declares whether it is total with 
> Expression.isTotal, and the ones that are always total mix in NoThrow 
> (comparisons, And/Or/Not, IsNull, Coalesce, Length, Substring, ...). 
> SPARK-61663 kept ConstantFolding as it was by naming CreateArray and 
> CreateNamedStruct there, so e.g. size(array(i > 1)) is still evaluated 
> although its element cannot fail.
> This is to revisit hasNoSideEffect so that it relies on the declaration 
> (NoThrow, or Expression.isTotal for the expressions that are total in some 
> eval modes only) instead of the two names. It changes which plans are folded, 
> so it needs its own review: the rule skips the evaluation of the elements, 
> and a wrong declaration would hide an error (see SPARK-60820 and SPARK-60868).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to