zhuxiangyi opened a new pull request, #9809:
URL: https://github.com/apache/paimon/pull/9809
### Purpose
Extracting a variant number into a narrower integral type goes through the
generic `CastExecutors` rules (`Number::intValue`, `(int)
DecimalUtils.castToIntegral(...)`), which wrap or saturate a value that does
not fit. With `spark.sql.variant.pushVariantIntoScan=true` on Spark 4.1 the
cast runs inside the Paimon reader, so the same query returns different numbers
depending on the session flag:
```sql
CREATE TABLE T (id INT, v VARIANT);
INSERT INTO T VALUES (1, parse_json('{"n":99999999999,"d":1e30}')), (2,
parse_json('{"n":7,"d":1.5}'));
SELECT id, try_variant_get(v, '$.n', 'int'), try_variant_get(v, '$.d',
'bigint') FROM T ORDER BY id;
-- pushVariantIntoScan=true : (1, 1215752191, 9223372036854775807), (2, 7, 1)
-- pushVariantIntoScan=false: (1, NULL, NULL), (2, 7, 1) <- Spark's TRY
cast
```
The shredded `typed_value` path (`BaseVariantReader.ScalarReader`) shares
the problem through the same cast rules, e.g. a `BIGINT` typed value read as
`INT`.
This PR checks the truncated value against the target range before
delegating to the cast rules, in a helper shared by `VariantGet` and the scalar
reader, so an out-of-range number becomes an invalid cast (`NULL` for
`try_variant_get`, an error for `variant_get`), matching Spark. A cast rule
returning `null` (e.g. a `BIGINT` that overflows a `DECIMAL(p, s)` target) is
now also reported as an invalid cast instead of being returned as a silent
`NULL` under `failOnError=true`.
In-range behaviour is unchanged: fractional parts are still truncated (`1.5`
as `int` → `1`) and widening casts still succeed.
### Tests
- `GenericVariantTest#testVariantGetIntegralOverflow`: LONG / DOUBLE /
DECIMAL inputs into TINYINT..BIGINT, try mode returns `NULL`, strict mode
throws; boundary values still cast.
- `VariantShreddingReadTest#testReadIntegralOverflowAsInvalidCast`:
parameterised over a plain file and a file shredded with `n BIGINT, d DOUBLE`,
covering both `VariantGet` and the shredded scalar reader.
- `VariantTestBase`: `try_variant_get` out-of-range extraction returns
`NULL`; runs under all four Spark 4.x configurations (with/without pushdown,
with/without inferred shredding). Verified on Spark 4.1.2: 116 tests pass.
### API and Format
No.
### Documentation
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
