Ankur Kotwal. created FLINK-41227:
-------------------------------------
Summary: AVG on TINYINT, SMALLINT and INT overflows when
SplitAggregateRule splits distinct aggregates
Key: FLINK-41227
URL: https://issues.apache.org/jira/browse/FLINK-41227
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Affects Versions: 2.3.0
Reporter: Ankur Kotwal.
With table.optimizer.distinct-agg.split.enabled = true, SplitAggregateRule
rewrites AVG(x) into SUM0(x) and COUNT(x), which are computed in the partial
and final aggregates. SUM0 keeps the type of its argument, so for TINYINT,
SMALLINT and INT columns the partial and final sums overflow, and AVG returns a
wrong value. The AVG function itself accumulates these types as BIGINT, so
without the split the result is correct.
Reproduction (streaming, split enabled):
{code:sql}
-- x = 2147483647, 2147483647, 1
SELECT COUNT(DISTINCT k), AVG(x) FROM MaxIntTable
{code}
Expected: 3, 1431655765
Actual: 3, 0
This is a sibling of FLINK-40919, which fixes the same overflow in the AVG to
SUM / COUNT reduction (AggregateReduceFunctionsRule). That fix doesn't cover
SplitAggregateRule, which builds its own SUM0.
Proposed fix: before splitting, project the argument of such AVG calls as
BIGINT. The final result is cast back to the AVG result type as before. I have
a patch with a regression test in SplitAggregateITCase, and I'll open a PR once
FLINK-40919's approach is settled, since the two should be consistent.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
