JingsongLi commented on PR #10196:
URL: https://github.com/apache/paimon/pull/10196#issuecomment-5950997990
[P2] Handle decimals outside BIGINT range in the new z-order path
The new branch at `SparkZOrderUDF.java:368-369` accepts every `DecimalType`,
but casting it to `LongType` makes the support depend on both the value and
Spark's ANSI setting. Legal `DECIMAL(38,2)` values such as
`9223372036854775808.00` and `-9223372036854775809.00` raise `CAST_OVERFLOW`
when ANSI is enabled. This is also the default setting in Spark 4.
I reproduced this through an actual persisted Paimon table, rather than
invoking the UDF alone:
```sql
SET spark.sql.ansi.enabled=true;
CREATE TABLE T (id INT, amount DECIMAL(38,2)) TBLPROPERTIES ('bucket'='-1');
INSERT INTO T VALUES (1, 9223372036854775808.00), (2,
-9223372036854775809.00);
CALL paimon.sys.compact(table => 'test.T', order_strategy => 'zorder',
order_by => 'amount');
```
The compact job aborts in the decimal-to-BIGINT cast. The old snapshot
remains readable and no replacement snapshot is published, so I did not observe
data loss, but the advertised decimal compaction cannot complete for these
valid values. Ordinary positive/negative fractional values and nulls do compact
and retain their original data.
Please use an encoding that covers the decimal domain without an
ANSI-sensitive narrowing cast, and add a procedure-level regression test with
out-of-BIGINT values. The existing decimal encoding in `ZIndexer` is a useful
reference. Mirroring Hilbert's current cast preserves its older limitation; it
does not cover the full domain here.
Validation: all three existing `SparkZOrderUDFTest` cases pass; an
additional actual-table SQL probe verifies successful in-range compaction and
reproduces the out-of-range failure on Spark 3.5.8 and Spark 4.1.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
