jackylee-ch opened a new pull request, #9672:
URL: https://github.com/apache/paimon/pull/9672
### Purpose
`VariantGet` took the precision and scale straight off the `BigDecimal` that
`getDecimal()` returns. `precision()` counts the unscaled digits, so it is
below the scale for anything under 0.1, and the trailing-zero stripping in
`getDecimal()` turns `100.00` into `1E+2`, a negative scale. `DecimalType`
rejects both, so extraction threw `Decimal scale must be between 0 and the
precision 1`. Reachable on Spark 4.1+ pushdown, which accepts a `string` target
and any `try_variant_get`.
A negative scale is rescaled to zero and the precision widened to the scale,
matching Spark's `Decimal.set(BigDecimal)`. `checkDecimal` caps both at 38 on
read, so this stays in range.
Two things it does not close, both owned by open PRs:
- `InferVariantShreddingSchema` widens precision but never clamps a negative
scale, so `{"round": 100.00}` still throws under `variant.inferShreddingSchema`
(#9532).- `DECIMAL` to `STRING` now yields `100` unshredded and `100.00`
shredded, since the shredded leaf does not strip. Spark covers that with
`castDecimalToString`; Paimon's belongs in `BaseVariantReader` (#9616).
### Tests
`GenericVariantTest` for both shapes and the scale-38 boundary,
`PaimonShreddingUtilsTest` for the pushdown path.
Written with Claude Code; verification is mine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
