cool9850311 opened a new pull request, #27474:
URL: https://github.com/apache/datafusion/pull/27474

   ## Which issue does this PR close?
   
   - Closes #27424.
   - Closes #27095.
   
   #27095 already has #27124 open against it, which fixes it with the same 
variant-selection
   idea this takes (credit to @namanjain24-sudo). Whether this PR should close 
#27095, or whether
   that issue should stay with #27124, is a call for the maintainers: say which 
you prefer and I
   will edit the description. If #27124 lands first, I am happy to rebase this 
onto it and keep
   only the part that removes the `?`.
   
   ## Rationale for this change
   
   An integer compared with or added to a `Decimal32` or `Decimal64` value 
returns wrong
   rows instead of an error. A `Decimal32(9, 2)` holding `5.50` compares equal 
to `5`, is
   not greater than `5`, and `5.50 + 1` returns `6` as an `Int64`. The same 
values as
   `Decimal128` give the right answers.
   
   How many digits an integer needs is known exactly, so the common type of an 
integer and
   a decimal has to keep room for them. It did not, in two ways:
   
   - `coerce_numeric_type_to_decimal32` had no arm for `Int32` or wider and
     `coerce_numeric_type_to_decimal64` none for `Int64`, since those integers 
do not fit
     the variant. Both callers treated that `None` as "not a decimal case" and 
fell through
     to the integer rules, which coerce the decimal to an integer and drop its 
fraction.
   - When the required precision passed the variant's maximum, it was capped 
while the
     scale was kept, leaving a type the integer could not be cast into at all
     (`Failed to cast field 'a' from Int32 to Decimal128(38, 38)`, #27095).
   
   Both are the same hole, so one rule closes both: an integer's precision and 
scale are
   now computed independently of any variant, and the common type is the 
narrowest variant
   that holds the required precision and is no narrower than the decimal's own.
   `Decimal32(9, 2)` with `Int64` becomes `Decimal128(22, 2)`; `Decimal128(38, 
38)` with
   `Int32` becomes `Decimal256(48, 38)`. Where no variant holds the required 
precision --
   only a `Decimal256` whose scale is large in either direction can reach that 
-- the
   precision is still capped, so the comparison stays in decimals.
   
   A float keeps the previous behaviour. The scale a float coerces to is an 
approximation of
   its own (`Float64` becomes `Decimal128(30, 15)`), and widening on that basis 
would move
   ordinary comparisons like `WHERE price = 4.00` onto a wider variant.
   
   ## What changes are included in this PR?
   
   In `datafusion/expr-common/src/type_coercion/binary.rs`:
   
   - `integer_decimal_precision` gives an integer's precision and scale without 
choosing a
     variant; `narrowest_decimal_variant` picks the narrowest variant that 
holds a precision
     and is no narrower than a given one.
   - `get_common_decimal_type` uses them for an integer, and the four
     `coerce_numeric_type_to_decimal*` helpers are now float-only
     (`coerce_float_type_to_decimal*`), since an integer no longer goes through 
them.
   - `math_decimal_coercion`'s eight integer arms become two guarded on
     `is_decimal() && is_integer()`: both operands move to the variant that 
holds them, each
     keeping its own precision and scale, so the result keeps the precision 
arrow derives
     from them.
   - `decimal_precision_and_scale` replaces the two inline matches in
     `get_wider_decimal_type_cross_variant`.
   
   ## What is the testing strategy for this PR?
   
   Unit tests for the coerced types, and sqllogictest for what a query returns:
   
   - `type_coercion/binary/tests/comparison.rs`: the common type of 
`Decimal32`/`Decimal64`
     against each integer width, of each integer width against a wide-scale 
`Decimal128`, a
     scale of 9 and a scale of -2 on a `Decimal32`, the `Decimal256` whose 
precision is
     capped, the cases that already fitted (which must not move), and a float 
against both.
   - `type_coercion/binary/tests/arithmetic.rs`: the operand pair for each 
arithmetic case in
     both operand orders, plus `integer_decimal_precision` itself;
     `test_decimal_mathematics_op_type` now covers the float-only helpers.
   - `interval_arithmetic.rs`: `contains_value` of a `Decimal32` interval 
against an integer,
     whose bounds used to be coerced to the integer and lose their fraction, 
which put `1`
     inside `[1.50, 2.50]`.
   - `decimal.slt`: the five queries from #27424 with `arrow_typeof` pinned, 
the #27095
     queries, which return rows instead of failing, an `IN` list, 
`greatest`/`least` and a
     `UNION ALL`, and a `Decimal256(76, 70)` compared with integers.
   - `asof_join.slt`: a `Decimal32` ASOF equality key against an `Int64` one, 
which used to
     make `1.50` and `1` the same key, and a match condition across the two 
types.
   
   Each of those tests was run against this branch and against its base: every 
one fails on
   the base for the value it pins. The `To Reproduce` queries of both issues 
were also run by
   hand on both builds, and on the base they reproduce the reported output word 
for word.
   
   ## Are there any user-facing changes?
   
   Comparing or combining an integer with a `Decimal32` or `Decimal64` value 
now stays in
   decimal arithmetic instead of truncating the decimal, and comparing an 
integer with a
   wide-scale decimal returns rows instead of failing. Everything that asks for 
a common type
   sees this, not just a comparison in `WHERE`:
   
   - an `IN` list: `d32 IN (5, 6)` with `d32` at `5.50` returned `true`, now 
returns `false`;
   - `greatest`/`least`: `greatest(d32, 5)` returned `5` as an `Int64`, now 
`5.50`;
   - `UNION ALL` of such a column with an integer: the fraction was dropped 
silently;
   - an ASOF join key of the two types: `1.50` and `1` were the same key;
   - interval and statistics containment, which read the same common type;
   - `gcd`/`lcm`: `gcd(<Decimal32>, 4)` used to be a plan error.
   
   No API changes.
   
   ## Notes for reviewers
   
   - A product's precision is still capped at the variant's maximum, so
     `Decimal32(9, 2) * Int32` has `Decimal64` operands and a `Decimal64(18, 
2)` result. That
     is what every decimal pair already gets: `Decimal32 + Decimal32` caps the 
same way, and
     `Decimal128 * Int64` caps at 38 rather than widening to `Decimal256`. 
Widening a variant
     to fit a result would be a change of policy for all decimal arithmetic, so 
it is not
     here.
   - `narrowest_decimal_variant` checks the precision only. A variant's maximum 
scale equals
     its maximum precision and the required precision already leaves room for 
the scale, so a
     scale check would never change the answer.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to