L. C. Hsieh created SPARK-61667:
-----------------------------------

             Summary: Use Expression.isTotal to fold the size of CreateArray in 
ConstantFolding
                 Key: SPARK-61667
                 URL: https://issues.apache.org/jira/browse/SPARK-61667
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: L. C. Hsieh


ConstantFolding folds size(array(e1, ..., en)) to n without evaluating the 
elements when none of them can fail. It decides that with its own check, 
hasNoSideEffect, which above attributes, literals and some casts only accepts 
CreateArray and CreateNamedStruct.

Since SPARK-61663, every expression declares whether it is total with 
Expression.isTotal, and the ones that are always total mix in NoThrow 
(comparisons, And/Or/Not, IsNull, Coalesce, Length, Substring, ...). 
SPARK-61663 kept ConstantFolding as it was by naming CreateArray and 
CreateNamedStruct there, so e.g. size(array(i > 1)) is still evaluated although 
its element cannot fail.

This is to revisit hasNoSideEffect so that it relies on the declaration 
(NoThrow, or Expression.isTotal for the expressions that are total in some eval 
modes only) instead of the two names. It changes which plans are folded, so it 
needs its own review: the rule skips the evaluation of the elements, and a 
wrong declaration would hide an error (see SPARK-60820 and SPARK-60868).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to