yashmayya opened a new pull request, #19790: URL: https://github.com/apache/pinot/pull/19790
## Problem The multi-stage engine maps `DECIMAL(p, 0)` to `INT` when `p <= 10` and to `LONG` when `p <= 38`. `INT` cannot hold every 10-digit number, and `LONG` cannot hold every number with 19 to 38 digits. `RexExpressionUtils.fromRexLiteralValue` then calls `intValue()` / `longValue()`, which wrap the value without an error. Calcite gives the type `DECIMAL(p, 0)` to integer literals beyond the `BIGINT` range and to `CAST(... AS DECIMAL(p, 0))`. Results on master, with both planners: | Query | Result on master | |---|---| | `WHERE longCol = 99999999999999999999` | Matches rows with `7766279631452241919` | | `WHERE longCol < 9223372036854775808` | Fails with `Invalid range` | | `SELECT intCol + 99999999999999999999` | About `7.77E18` | | `WHERE intCol = CAST(3000000000 AS DECIMAL(10, 0))` | Matches rows with `-1294967296` | | `SUM(CAST(intCol AS DECIMAL(10, 0)))` | Stops at `2147483647` | | `SELECT 99999999999999999999 + 1` (folded during planning) | About `7.77E18` | ## Fix Map `DECIMAL(p, 0)` to `INT` only when `p <= 9`, to `LONG` only when `p <= 18`, and to `BIG_DECIMAL` otherwise. The change is in `RelToPlanNodeConverter` and in its copy for the physical optimizer, `PRelToPlanNodeConverter`. ## Compatibility - Result types change for some queries. `DECIMAL(10, 0)` is now `LONG` (was `INT`). `DECIMAL(19..38, 0)` is now `BIG_DECIMAL` (was `LONG`), so the broker returns it as a string, like other `BIG_DECIMAL` values. `SUM`, `MIN`, `MAX` and `AVG` over these values change the same way. A plain `CAST(x AS DECIMAL)` is `DECIMAL(2000, 1000)`, so it does not change. - Arrays of `DECIMAL(19..38, 0)` are now `DOUBLE_ARRAY` (was `LONG_ARRAY`). This is the same as arrays with `p > 38` today (see the `BIG_DECIMAL_ARRAY` TODO). - Mixed versions: the broker sets all plan types. Servers already accept `BIG_DECIMAL` literals in plans (#13344), so no new type goes over the wire. ## Not in this PR - `SUM`, `MAX` and `MOD` compute in `double`, so values above 2^53 lose precision. This also happens on master for plain `LONG` columns. - The single-stage engine has the same wrap in `RequestUtils.getLiteral`. ## Tests - `NumericTypes.json`: 16 queries compared with H2, in the leaf stage, in intermediate stages and during planning. 14 fail on master with both planners. The other 2 make sure that a `BIGINT` promoted to `DECIMAL(19, 0)` keeps its exact value. - Unit tests for the new limits in `RelToPlanNodeConverterTest` and `PRelToPlanNodeConverterTest`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
