Jack Leitch created SPARK-60443:
-----------------------------------
Summary: VariantUtil.checkDecimal rejects spec-conformant decimal
values when scale > type precision
Key: SPARK-60443
URL: https://issues.apache.org/jira/browse/SPARK-60443
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.1, 4.0.0, 4.1.0
Reporter: Jack Leitch
VariantUtil.checkDecimal rejects spec-conformant Variant decimal values when
the scale is greater than the precision of the chosen decimal type.
Repro:
// write v into a parquet file as a VARIANT column, then read:
spark.read.parquet(path).selectExpr("cast(payload as string)").collect();
// throws MALFORMED_VARIANT
The value 0.00042626262 is encoded by parquet-variant as decimal4 with scale
11. The raw value bytes are: 20 0b d6 6c 8a 02 - header 0x20 (decimal4), scale
0x0b = 11, little-endian int32 42626262.
Why this is a Spark bug and not a parquet-variant bug:
The Variant encoding spec
([https://github.com/apache/parquet-format/blob/master/VariantEncoding.md])
defines scale as [0, 38] across decimal4, decimal8, and decimal16:
decimal4: 1-byte scale in range [0, 38], followed by little-endian unscaled
value
decimal8: 1-byte scale in range [0, 38], followed by little-endian unscaled
value
decimal16: 1-byte scale in range [0, 38], followed by little-endian unscaled
value
The per-type restriction is on precision, not scale. parquet-java PR #3335
([https://github.com/apache/parquet-java/pull/3335]) explicitly removed scale
from decimal width selection on that basis.
VariantUtil.checkDecimal conflates the two:
private static void checkDecimal(BigDecimal d, int maxPrecision) {
if (d.precision() > maxPrecision || d.scale() > maxPrecision)
{ throw malformedVariant(); }
}
For decimal4, maxPrecision = 9, so scale = 11 throws. DuckDB, Snowflake, and
parquet-variant's own Variant.getDecimal() all read these bytes without
complaint. Spark is the strictest reader, which is why it alone surfaces the
issue.
Fix:
Bound d.scale() against MAX_DECIMAL16_PRECISION (the universal 38 ceiling)
rather than the per-type maxPrecision. Precision continues to be bounded by the
type width.
PR to follow.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
