Udonburo commented on issue #26223:
URL: https://github.com/apache/datafusion/issues/26223#issuecomment-6138011019
I understand the point about reassociation in the original example. I
checked a finite-input overflow case on `6663e4348` (Rust 1.99.0) using the
sqllogictest runner.
From `datafusion/sqllogictest`, create a headerless
`test_files/_sum_overflow_input.csv` containing:
```csv
-1e308
-1e308
```
Then run:
```sql
SET datafusion.execution.target_partitions = 1;
SET datafusion.execution.batch_size = 8192;
CREATE EXTERNAL TABLE overflow_input (x DOUBLE)
STORED AS CSV LOCATION 'test_files/_sum_overflow_input.csv'
OPTIONS ('format.has_header' 'false');
SELECT SUM(x + 1e308) FROM overflow_input;
-- 0
SELECT SUM(x + 1e308), SUM(x + 5e307) FROM overflow_input;
-- NaN, -Infinity
EXPLAIN SELECT SUM(x + 1e308), SUM(x + 5e307) FROM overflow_input;
```
The paired query's plan scans the CSV, aggregates `SUM(x)` and `COUNT(x)`,
then projects `SUM(x) + c * CAST(COUNT(x) AS Float64)`. I confirmed that `x` is
`Float64` and each original `x + 1e308` is exactly zero, so that SUM is zero
independently of reduction order. In the rewritten form, `SUM(x)` is
`-Infinity` and `1e308 * COUNT(x)` is `Infinity`, producing `NaN`.
With the same CSV and paired query:
- Removing the **whole `SimplifyExpressions` optimizer rule** through
`with_optimizer_rules`, retaining the other optimizer rules, gives `0, -1e308`.
- A local guard that skips this rewrite for floating-point literals also
gives `0, -1e308`.
These results were the same for `(target_partitions, batch_size) = (1,
8192)` and `(4, 1)`. The latter plan uses `RoundRobinBatch(4)` and
Partial/Final aggregation.
Is a rewrite-created NaN also an allowed consequence of reassociation when
the original per-row expression and its SUM remain finite regardless of
aggregation order?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
