Yang Jie created SPARK-62133:
--------------------------------

             Summary: V2Aggregator ignores BoundFunction.isDeterministic()
                 Key: SPARK-62133
                 URL: https://issues.apache.org/jira/browse/SPARK-62133
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Yang Jie


{{V2Aggregator}}, which evaluates an {{AggregateFunction}} from a 
{{FunctionCatalog}}, does not override {{deterministic}}. So Spark treats such 
an aggregate as deterministic even when its bound function returns {{false}} 
from {{isDeterministic()}}. SPARK-37957 fixed this for scalar functions in 
{{ApplyFunctionExpression}}, and {{ScalaUDAF}} and {{ScalaAggregator}} respect 
their own flag.

For an aggregate function {{f}} whose bound function is not deterministic, 
{{SELECT f(i), f(i) FROM t}} evaluates {{f}} once and returns the same value in 
both columns, since {{PhysicalAggregation}} merges equal deterministic 
aggregate expressions. A non-deterministic Spark UDAF is evaluated twice there.

Plans also differ from a non-deterministic Spark UDAF in other rules that read 
{{deterministic}}. For example, in {{SELECT * FROM (SELECT g, f(i) AS v FROM t 
GROUP BY g) WHERE g > 0}}, {{g > 0}} is pushed below the {{Aggregate}}, while 
for such a UDAF it stays above.

{{V2Aggregator}} could override {{deterministic}} the way 
{{ApplyFunctionExpression}} does. apache/spark#60672 (SPARK-61870) checks 
{{isDeterministic()}} directly where it decides whether a scan with a pushed 
aggregate reads the same rows again, and has a TODO to drop that check once 
this is fixed.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to