[
https://issues.apache.org/jira/browse/SPARK-61623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61623:
-----------------------------------
Labels: correctness pull-request-available (was: correctness)
> Segment-tree window SUM fails with ARITHMETIC_OVERFLOW although no frame
> overflows
> ----------------------------------------------------------------------------------
>
> Key: SPARK-61623
> URL: https://issues.apache.org/jira/browse/SPARK-61623
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: sepuri sai krishna
> Priority: Major
> Labels: correctness, pull-request-available
>
> With {{spark.sql.window.segmentTree.enabled=true}}, a moving-frame {{sum}}
> over a BIGINT column fails with {{ARITHMETIC_OVERFLOW}} under ANSI mode
> although the sum of every frame is in range. The query succeeds when the flag
> is off.
> {code:sql}
> CREATE TEMPORARY VIEW t AS
> SELECT id AS i, CAST(1000000000000000 AS BIGINT) AS v FROM range(0, 100000);
> SET spark.sql.window.segmentTree.enabled=false;
> SELECT max(x) FROM (
> SELECT sum(v) OVER (ORDER BY i ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS
> x FROM t);
> -- 2000000000000000
> SET spark.sql.window.segmentTree.enabled=true;
> SELECT max(x) FROM (
> SELECT sum(v) OVER (ORDER BY i ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS
> x FROM t);
> -- [ARITHMETIC_OVERFLOW] overflow. Use 'try_add' to tolerate overflow and
> return NULL instead. ...
> {code}
> Every frame holds two rows, so its sum is 2e15, far below the BIGINT maximum
> of about 9.2e18. Sums of large BIGINT values over short frames are realistic,
> for example over epoch times in nanoseconds (about 1.8e18 each, so that the
> sum of any six rows overflows).
> *Cause*
> {{WindowSegmentTree}} aggregates rows that no frame covers together:
> * {{computeBlockAggregates}} computes, when the tree is built, the aggregate
> of every block of {{spark.sql.window.segmentTree.blockSize}} (65536) rows. In
> the example one block sums to 6.5e19.
> * {{buildBlockLevels}} computes the aggregate of every internal node of a
> block (16, 256, 4096, ... rows).
> For an integral {{Sum}} under ANSI, the buffer is a BIGINT that is added with
> an exact addition, so such an aggregate can overflow and fail the query, even
> though no frame reads it. {{try_sum}}, {{sum}} over DECIMAL (overflow is
> checked at the end) and over DOUBLE, and non-ANSI mode are not affected.
> *Fix*
> Do not fail on an arithmetic error of a block root or of an internal node:
> keep the node as unavailable, and let a query that covers it merge its
> children instead, down to the rows. A frame whose own sum overflows still
> fails, as its values are merged into the result.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]