sepuri sai krishna created SPARK-61623:
------------------------------------------

             Summary: Segment-tree window SUM fails with ARITHMETIC_OVERFLOW 
although no frame overflows
                 Key: SPARK-61623
                 URL: https://issues.apache.org/jira/browse/SPARK-61623
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.2.0
            Reporter: sepuri sai krishna


With {{spark.sql.window.segmentTree.enabled=true}}, a moving-frame {{sum}} over 
a BIGINT column fails with {{ARITHMETIC_OVERFLOW}} under ANSI mode although the 
sum of every frame is in range. The query succeeds when the flag is off.

{code:sql}
CREATE TEMPORARY VIEW t AS
  SELECT id AS i, CAST(1000000000000000 AS BIGINT) AS v FROM range(0, 100000);

SET spark.sql.window.segmentTree.enabled=false;
SELECT max(x) FROM (
  SELECT sum(v) OVER (ORDER BY i ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS x 
FROM t);
-- 2000000000000000

SET spark.sql.window.segmentTree.enabled=true;
SELECT max(x) FROM (
  SELECT sum(v) OVER (ORDER BY i ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS x 
FROM t);
-- [ARITHMETIC_OVERFLOW] overflow. Use 'try_add' to tolerate overflow and 
return NULL instead. ...
{code}

Every frame holds two rows, so its sum is 2e15, far below the BIGINT maximum of 
about 9.2e18. Sums of large BIGINT values over short frames are realistic, for 
example over epoch times in nanoseconds (about 1.8e18 each, so that the sum of 
any six rows overflows).

*Cause*

{{WindowSegmentTree}} aggregates rows that no frame covers together:

* {{computeBlockAggregates}} computes, when the tree is built, the aggregate of 
every block of {{spark.sql.window.segmentTree.blockSize}} (65536) rows. In the 
example one block sums to 6.5e19.
* {{buildBlockLevels}} computes the aggregate of every internal node of a block 
(16, 256, 4096, ... rows).

For an integral {{Sum}} under ANSI, the buffer is a BIGINT that is added with 
an exact addition, so such an aggregate can overflow and fail the query, even 
though no frame reads it. {{try_sum}}, {{sum}} over DECIMAL (overflow is 
checked at the end) and over DOUBLE, and non-ANSI mode are not affected.

*Fix*

Do not fail on an arithmetic error of a block root or of an internal node: keep 
the node as unavailable, and let a query that covers it merge its children 
instead, down to the rows. A frame whose own sum overflows still fails, as its 
values are merged into the result.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to