Yang Jie created SPARK-62133:
--------------------------------
Summary: V2Aggregator ignores BoundFunction.isDeterministic()
Key: SPARK-62133
URL: https://issues.apache.org/jira/browse/SPARK-62133
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Yang Jie
{{V2Aggregator}}, which evaluates an {{AggregateFunction}} from a
{{FunctionCatalog}}, does not override {{deterministic}}. So Spark treats such
an aggregate as deterministic even when its bound function returns {{false}}
from {{isDeterministic()}}. SPARK-37957 fixed this for scalar functions in
{{ApplyFunctionExpression}}, and {{ScalaUDAF}} and {{ScalaAggregator}} respect
their own flag.
For an aggregate function {{f}} whose bound function is not deterministic,
{{SELECT f(i), f(i) FROM t}} evaluates {{f}} once and returns the same value in
both columns, since {{PhysicalAggregation}} merges equal deterministic
aggregate expressions. A non-deterministic Spark UDAF is evaluated twice there.
Plans also differ from a non-deterministic Spark UDAF in other rules that read
{{deterministic}}. For example, in {{SELECT * FROM (SELECT g, f(i) AS v FROM t
GROUP BY g) WHERE g > 0}}, {{g > 0}} is pushed below the {{Aggregate}}, while
for such a UDAF it stays above.
{{V2Aggregator}} could override {{deterministic}} the way
{{ApplyFunctionExpression}} does. apache/spark#60672 (SPARK-61870) checks
{{isDeterministic()}} directly where it decides whether a scan with a pushed
aggregate reads the same rows again, and has a TODO to drop that check once
this is fixed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]