qlong opened a new pull request, #59689: URL: https://github.com/apache/spark/pull/59689
### What changes were proposed in this pull request? Variant decoding rejected a decimal as `MALFORMED_VARIANT` when its scale was larger than the max precision of its physical type (9 for decimal4, 18 for decimal8). This PR bounds the scale by `MAX_DECIMAL16_PRECISION` (38) for every decimal type, in both `VariantUtil.checkDecimal` (JVM) and `VariantUtils._check_decimal` (PySpark). The per-type precision bound is unchanged. Scale bytes above 38, including a negative scale read as an unsigned byte, are still `MALFORMED_VARIANT`. Spark's writers (`VariantBuilder` on the JVM and in PySpark) are unchanged. They still pick the decimal type by both scale and precision, so Spark releases without this fix can still read what Spark writes. ### Why are the changes needed? The [Variant spec](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md) only ties precision to the physical type. Every decimal type stores a "1 byte scale in range [0, 38]", and the physical type only sets the size of the unscaled integer. Spark rejected valid values, and accepted or rejected the same value depending on its encoding: Encoding Value bytes (hex) Value Before After decimal8, scale 13 24 0D D2 04 00 00 00 00 00 00 1.234E-10 0.0000000001234 0.0000000001234 decimal4, scale 13 20 0D D2 04 00 00 1.234E-10 MALFORMED_VARIANT 0.0000000001234 decimal8, scale 25 24 19 D2 04 00 00 00 00 00 00 1.234E-22 MALFORMED_VARIANT 0.0000000000000000000001234 parquet-java 1.17.0+ ([apache/parquet-java#3335](https://github.com/apache/parquet-java/pull/3335)) picks the decimal type by precision only, so it writes values like the decimal4 row above. Spark fails on any expression that decodes such a field, e.g. `to_json`, `variant_get`, `cast` and `schema_of_variant`. The per-type scale check came from the SPARK-47681 follow-up (#46338), which assumed the spec bounds scale per type. ### Does this PR introduce _any_ user-facing change? Yes. Variant decimals with a scale above 9 (decimal4) or 18 (decimal8), up to 38, now decode instead of failing with `MALFORMED_VARIANT`. This affects only data produced by other writers. Variants written by Spark are not affected. ### How was this patch tested? New tests: - `VariantExpressionSuite`: decimal4/8/16 with scale above the per-type precision, including negative values and zero, decode correctly through `to_json`, `getDecimal`, `schema_of_variant` and `variant_get`. The same value encoded as decimal4, decimal8 and decimal16 decodes identically. Scale 39 and a negative scale byte still fail. - `VariantWriteShreddingSuite`: a decimal4 with scale 13 shreds into a typed decimal column exactly like Spark's own encoding of the same value (decimal8). - `test_types.py`: the same decode cases for `VariantVal.toPython`. One existing assertion, which expected decimal4 with scale 10 to raise, now expects the decoded value. build/sbt 'catalyst/testOnly *VariantExpressionSuite' 'sql/testOnly *VariantWriteShreddingSuite' ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
