yashmayya opened a new pull request, #19791: URL: https://github.com/apache/pinot/pull/19791
## Problem `isDistinctFrom` and `isNotDistinctFrom` (`ObjectFunctions`) compared their arguments with `equals`. `Integer.equals(Long)` is always false, so equal numbers of different types were "distinct". `BigDecimal.equals` also checks the scale, and `byte[].equals` compares references. This scalar runs above the leaf stage in the multi-stage engine, and on the single-stage broker (literal folding, post-aggregation, `HAVING`). The leaf stage and single-stage servers use `DistinctFromTransformFunction`, which already compares by value. So the same call gave different answers depending on where it ran. | Query | Master | This PR | |---|---|---| | MSE: `SELECT isDistinctFrom(t.intCol, CAST(5 AS BIGINT)) FROM (SELECT intCol FROM tbl GROUP BY intCol) t`, `intCol = 5` | `true` | `false` | | MSE (above leaf): `isNotDistinctFrom(bigDecimalCol, CAST(-1.5 AS DECIMAL(38, 3)))`, value `-1.5` | `false` | `true` | | MSE (above leaf): `isDistinctFrom(SQRT(doubleCol), CAST(SQRT(doubleCol) AS REAL))`, NaN | `true` | `false` | | MSE (above leaf): `isDistinctFrom(bytesCol1, bytesCol2)`, equal bytes | `true` | `false` | | SSE: `SELECT isDistinctFrom(MAX(intCol), 5)`, max is 5 | `true` | `false` | | SSE: `GROUP BY intCol HAVING isNotDistinctFrom(MAX(longCol), 5)` | no rows | matching rows | | SSE: `SELECT isDistinctFrom(5, 5.0)` (folded) | `true` | `false` | The SQL form `a IS DISTINCT FROM b` is not affected. Calcite rewrites it to `<>` and null checks. ## Change Numbers (`Integer`, `Long`, `Float`, `Double`, `BigDecimal`) are compared by value, with the rules of `BinaryOperatorTransformFunction`: - Two integers: as `long`. - Two floating point numbers: `Double.compare`. A `float` is widened, so `1.1f` is distinct from `1.1`. NaN equals NaN, and `-0.0` is distinct from `0.0`, the same as `Double.equals` and the leaf stage. - An integer and a floating point number: exact. - Any `BigDecimal`: `compareTo`, so `1.5` equals `1.50`. A double is converted with `BigDecimal.valueOf`, so `0.1` equals the `BigDecimal` `0.1`. Two edge cases differ from `BinaryOperatorTransformFunction`, which is wrong there: - A `long` above 2^53 is compared exactly with a double. The transform function compares the double's decimal string, so it says `2^60` is distinct from `2^60` as a double. - NaN or infinity compared with a `BigDecimal` is distinct. The transform function throws `NumberFormatException`. Arrays (`BYTES` and multi-value values) are compared by content. Other values still use `equals`. Behavior change: the queries above now return the correct result instead of "distinct". ## Tests - `ObjectFunctionsTest` (pinot-common): mixed types, BigDecimal scale, NaN, `-0.0`, infinity, values above 2^53, arrays, nulls, both argument orders. - `Comparisons.json` (`ResourceBasedQueriesTest`): calls above the leaf stage, compared with H2. - `PostAggregationFunctionTest`: the single-stage post-aggregation path. All new cases fail on master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
