zhuxiangyi opened a new pull request, #9811:
URL: https://github.com/apache/paimon/pull/9811
### Purpose
**This is a bug fix.** With `variant.inferShreddingSchema=true`, writing a
Variant that contains a decimal with trailing zeros fails the whole file:
```sql
CREATE TABLE T (id INT, v VARIANT) TBLPROPERTIES
('variant.inferShreddingSchema' = 'true');
INSERT INTO T VALUES (1, parse_json('{"price":10.0}')), (2,
parse_json('{"price":20.5}'));
-- Job aborted ... java.lang.IllegalArgumentException: Decimal scale must be
between 0 and the precision 1 (both inclusive).
-- at org.apache.paimon.types.DataTypes.DECIMAL(DataTypes.java:115)
-- at
org.apache.paimon.data.variant.InferVariantShreddingSchema.schemaOf(InferVariantShreddingSchema.java:367)
```
`InferVariantShreddingSchema.schemaOf` derives the shredded decimal type
from `getDecimal()`, which strips trailing zeros: `10.0` arrives as `1E+1` and
`100.00` as `1E+2`, both with a negative scale that `DecimalType` rejects. The
existing guard only covered `precision < scale` (values below `0.1`). Integers
beyond the `long` range that end in zeros (`100000000000000000000` → `1E+20`)
hit the same path. Any engine that writes through the inference path (Spark,
Flink) is affected, and `10.0`-style values are common in real data.
The fix folds a negative scale back into the digits (`setScale(0)`) before
building the type, the same normalisation `VariantGet` gained in #9672, and
keeps the precision at least the scale. `10.0` now infers as `DECIMAL(2, 0)`,
merges with `20.5` into `DECIMAL(18, 1)`, and a lone `100.00` widens to
`BIGINT` as an integer-like decimal, which the shredding writer accepts because
it compares against the original scale and allows exact rescaling.
### Tests
The new tests reproduce the bug: on `master` they fail with the
`IllegalArgumentException` above (both the unit test and the two
inferred-shredding Spark suites), and pass with this change.
- `InferVariantShreddingSchemaTest#testInferSchemaWithDecimalTrailingZeros`:
`10.0` / `100.00` / `0.05` / `100000000000000000000` infer to `DECIMAL(18, 1)`
/ `BIGINT` / `DECIMAL(18, 3)` / `DECIMAL(38, 0)`.
- `InferVariantShreddingWriteTest#testInferSchemaWithDecimalTrailingZeros`:
writes such rows with inference on, checks the physical Parquet schema, the
reconstructed Variant and typed extraction of `$.price` / `$.whole`.
- `VariantTestBase`: inserts `10.0` / `100.00` and reads them back; runs
under all four Spark 4.x configurations. Verified on Spark 4.1.2: 116 tests
pass.
### API and Format
No.
### Documentation
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
