cool9850311 opened a new pull request, #27474:
URL: https://github.com/apache/datafusion/pull/27474
## Which issue does this PR close?
- Closes #27424.
- Closes #27095.
#27095 already has #27124 open against it, which fixes it with the same
variant-selection
idea this takes (credit to @namanjain24-sudo). Whether this PR should close
#27095, or whether
that issue should stay with #27124, is a call for the maintainers: say which
you prefer and I
will edit the description. If #27124 lands first, I am happy to rebase this
onto it and keep
only the part that removes the `?`.
## Rationale for this change
An integer compared with or added to a `Decimal32` or `Decimal64` value
returns wrong
rows instead of an error. A `Decimal32(9, 2)` holding `5.50` compares equal
to `5`, is
not greater than `5`, and `5.50 + 1` returns `6` as an `Int64`. The same
values as
`Decimal128` give the right answers.
How many digits an integer needs is known exactly, so the common type of an
integer and
a decimal has to keep room for them. It did not, in two ways:
- `coerce_numeric_type_to_decimal32` had no arm for `Int32` or wider and
`coerce_numeric_type_to_decimal64` none for `Int64`, since those integers
do not fit
the variant. Both callers treated that `None` as "not a decimal case" and
fell through
to the integer rules, which coerce the decimal to an integer and drop its
fraction.
- When the required precision passed the variant's maximum, it was capped
while the
scale was kept, leaving a type the integer could not be cast into at all
(`Failed to cast field 'a' from Int32 to Decimal128(38, 38)`, #27095).
Both are the same hole, so one rule closes both: an integer's precision and
scale are
now computed independently of any variant, and the common type is the
narrowest variant
that holds the required precision and is no narrower than the decimal's own.
`Decimal32(9, 2)` with `Int64` becomes `Decimal128(22, 2)`; `Decimal128(38,
38)` with
`Int32` becomes `Decimal256(48, 38)`. Where no variant holds the required
precision --
only a `Decimal256` whose scale is large in either direction can reach that
-- the
precision is still capped, so the comparison stays in decimals.
A float keeps the previous behaviour. The scale a float coerces to is an
approximation of
its own (`Float64` becomes `Decimal128(30, 15)`), and widening on that basis
would move
ordinary comparisons like `WHERE price = 4.00` onto a wider variant.
## What changes are included in this PR?
In `datafusion/expr-common/src/type_coercion/binary.rs`:
- `integer_decimal_precision` gives an integer's precision and scale without
choosing a
variant; `narrowest_decimal_variant` picks the narrowest variant that
holds a precision
and is no narrower than a given one.
- `get_common_decimal_type` uses them for an integer, and the four
`coerce_numeric_type_to_decimal*` helpers are now float-only
(`coerce_float_type_to_decimal*`), since an integer no longer goes through
them.
- `math_decimal_coercion`'s eight integer arms become two guarded on
`is_decimal() && is_integer()`: both operands move to the variant that
holds them, each
keeping its own precision and scale, so the result keeps the precision
arrow derives
from them.
- `decimal_precision_and_scale` replaces the two inline matches in
`get_wider_decimal_type_cross_variant`.
## What is the testing strategy for this PR?
Unit tests for the coerced types, and sqllogictest for what a query returns:
- `type_coercion/binary/tests/comparison.rs`: the common type of
`Decimal32`/`Decimal64`
against each integer width, of each integer width against a wide-scale
`Decimal128`, a
scale of 9 and a scale of -2 on a `Decimal32`, the `Decimal256` whose
precision is
capped, the cases that already fitted (which must not move), and a float
against both.
- `type_coercion/binary/tests/arithmetic.rs`: the operand pair for each
arithmetic case in
both operand orders, plus `integer_decimal_precision` itself;
`test_decimal_mathematics_op_type` now covers the float-only helpers.
- `interval_arithmetic.rs`: `contains_value` of a `Decimal32` interval
against an integer,
whose bounds used to be coerced to the integer and lose their fraction,
which put `1`
inside `[1.50, 2.50]`.
- `decimal.slt`: the five queries from #27424 with `arrow_typeof` pinned,
the #27095
queries, which return rows instead of failing, an `IN` list,
`greatest`/`least` and a
`UNION ALL`, and a `Decimal256(76, 70)` compared with integers.
- `asof_join.slt`: a `Decimal32` ASOF equality key against an `Int64` one,
which used to
make `1.50` and `1` the same key, and a match condition across the two
types.
Each of those tests was run against this branch and against its base: every
one fails on
the base for the value it pins. The `To Reproduce` queries of both issues
were also run by
hand on both builds, and on the base they reproduce the reported output word
for word.
## Are there any user-facing changes?
Comparing or combining an integer with a `Decimal32` or `Decimal64` value
now stays in
decimal arithmetic instead of truncating the decimal, and comparing an
integer with a
wide-scale decimal returns rows instead of failing. Everything that asks for
a common type
sees this, not just a comparison in `WHERE`:
- an `IN` list: `d32 IN (5, 6)` with `d32` at `5.50` returned `true`, now
returns `false`;
- `greatest`/`least`: `greatest(d32, 5)` returned `5` as an `Int64`, now
`5.50`;
- `UNION ALL` of such a column with an integer: the fraction was dropped
silently;
- an ASOF join key of the two types: `1.50` and `1` were the same key;
- interval and statistics containment, which read the same common type;
- `gcd`/`lcm`: `gcd(<Decimal32>, 4)` used to be a plan error.
No API changes.
## Notes for reviewers
- A product's precision is still capped at the variant's maximum, so
`Decimal32(9, 2) * Int32` has `Decimal64` operands and a `Decimal64(18,
2)` result. That
is what every decimal pair already gets: `Decimal32 + Decimal32` caps the
same way, and
`Decimal128 * Int64` caps at 38 rather than widening to `Decimal256`.
Widening a variant
to fit a result would be a change of policy for all decimal arithmetic, so
it is not
here.
- `narrowest_decimal_variant` checks the precision only. A variant's maximum
scale equals
its maximum precision and the required precision already leaves room for
the scale, so a
scale check would never change the answer.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]