zhuxiangyi opened a new pull request, #9818:
URL: https://github.com/apache/paimon/pull/9818
### Purpose
**This is a bug fix.** Extracting a decimal as a string returns different
text depending on whether the file is shredded:
```sql
CREATE TABLE T (id INT, v VARIANT);
INSERT INTO T VALUES (1, parse_json('{"price":1.50}')), (2,
parse_json('{"price":0.05}'));
SELECT variant_get(v, '$.price', 'string') FROM T;
-- plain file, or Spark evaluating itself : "1.5", "0.05"
-- file shredded with price DECIMAL(18, 2): "1.50", "0.05"
```
The unshredded leg (`VariantGet.cast`) casts from `getDecimal()`, which
strips trailing zeros, so it matches Spark's `VariantGet`. The shredded leg
(`BaseVariantReader.ScalarReader`) reads the `typed_value` with the scale of
the file schema and its string cast keeps that scale. Only decimal → string is
affected; numeric targets agree on both legs.
This matters because the docs promise that "the SQL and result are the same
for plain, shredded, and mixed-layout files", `'string'` is the one target type
a strict `variant_get` is pushed into the scan for, and with
`variant.inferShreddingSchema` a table routinely mixes plain and shredded
files, so a single column yields `1.5` for some rows and `1.50` for others.
The fix extracts the normalization already used by the unshredded leg (strip
trailing zeros, fold a negative scale, keep precision at least the scale) into
`VariantGet.normalizedDecimal` and applies it to a `typed_value` decimal before
it is cast, so both legs cast from the same value.
### Tests
The new unit and format tests reproduce the bug: on `master` they fail with
`expected: +I(10,1.5,0.05,0,...) but was: +I(10.0,1.50,0.05,0.00,...)`, and
pass with this change.
- `PaimonShreddingUtilsTest#testShreddedDecimalCastsLikeUnshredded`: the
same extractions through a shredded and an unshredded `VariantSchema` yield
identical strings, and decimal / double / bigint targets are unchanged.
- `VariantShreddingReadTest#testReadDecimalAsStringConsistently`:
parameterised over a plain Parquet file and one shredded with `price
DECIMAL(18, 1), amount DECIMAL(18, 2)`.
- `VariantTestBase`: `variant_get(..., 'string')` on values whose inferred
scale keeps trailing zeros; runs under all four Spark 4.x configurations
(with/without pushdown, with/without inferred shredding). Verified on Spark
4.1.2: 116 tests pass.
### API and Format
No.
### Documentation
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
