mbutrovich opened a new issue, #7442: URL: https://github.com/apache/datafusion-comet/issues/7442
## Describe the bug For an integer `truncate[W]` with `W > 2^30`, Comet computes two different values for the same row. - The `system.truncate` kernel reproduces Iceberg Java's `TruncateUtil` ([`truncate.rs`](https://github.com/apache/datafusion-comet/blob/79b5ddd50ba2e3f605399d641e4c0b3be84c20ef/native/spark-expr/src/iceberg_funcs/truncate.rs#L34-L41)). In Java, `(v % W) + W` overflows `int` for these widths. `tinyint` and `smallint` inputs go through the same function. - The native writer gets partition values from iceberg-rust's `PartitionSplitter` ([`iceberg_write.rs`](https://github.com/apache/datafusion-comet/blob/79b5ddd50ba2e3f605399d641e4c0b3be84c20ef/native/core/src/execution/operators/iceberg_write.rs#L1054)). At the pinned revision, iceberg-rust computes [`v - v.rem_euclid(width)`](https://github.com/apache/iceberg-rust/blob/b154603a710d730d2f43f8af4db257cc137237ef/crates/iceberg/src/transform/truncate.rs#L52-L54), which never overflows. For `W = 2_000_000_000` and `v = 1_500_000_000`, the kernel returns `-2_000_000_000` and iceberg-rust returns `0`. apache/iceberg-rust#3141 reports this mismatch as the clustered writer's "The input is not sorted!" error. The iceberg-rust value is the one the spec requires. The [truncate transform details](https://github.com/apache/iceberg/blob/f5704ee94b62839d79d9caf38045a3275fd3b4f0/format/spec.md?plain=1#L653-L661) define integer truncation as `v - (v % W)` with a positive remainder. Java's overflow makes the remainder negative. Iceberg Java treats this as a bug in apache/iceberg#18111, with a fix in apache/iceberg#18112 that isn't merged yet. The iceberg-rust discussion is in apache/iceberg-rust#3141 and apache/iceberg-rust#3573. The comment on [`truncate_agrees_with_iceberg_rust`](https://github.com/apache/datafusion-comet/blob/79b5ddd50ba2e3f605399d641e4c0b3be84c20ef/native/core/src/execution/operators/iceberg_write.rs#L6143-L6148) calls the large-width case an iceberg-rust bug. Per the spec it is a Java bug. Only the `i32::MIN` and `i64::MIN` cases are iceberg-rust bugs: there the result doesn't fit in the type, the spec is silent, Java wraps, and iceberg-rust panics in debug builds. The [user guide](https://github.com/apache/datafusion-comet/blob/79b5ddd50ba2e3f605399d641e4c0b3be84c20ef/docs/source/user-guide/latest/iceberg.md?plain=1#L272-L279) describes the kernels as reproducing Java's semantics and doesn't mention the spec. ## Steps to reproduce Add a large-width case to `truncate_agrees_with_iceberg_rust` in `native/core/src/execution/operators/iceberg_write.rs`, for example `Int32Array::from(vec![Some(1_500_000_000)])` with width `2_000_000_000`. The kernel and iceberg-rust's transform disagree (`-2000000000` against `0`). ## Expected behavior Comet should match Spark, and Spark's result here depends on the Iceberg jar the user runs, not on the Spark version. Every Iceberg release through 1.12.0 overflows. A release with apache/iceberg#18112 will return the spec value. A Spark-version shim can't pick the right behavior, because users choose their Iceberg runtime independently of the Spark profile. Comet should fall back to Spark in two cases: - An integer `truncate` call with `W > 2^30`, so the result is whatever the user's Iceberg jar computes. - A native Iceberg write whose partition spec has an integer `truncate[W]` with `W > 2^30`. Native writes produce the spec value, while Spark on Iceberg 1.12.0 and earlier writes the overflowed value into the same table. Comet doesn't control which Iceberg jar users bring, so this fallback stays after Iceberg ships the fix. If these widths ever need to run natively, Comet could probe the jar at plan time instead: `TruncateUtil.truncateInt(Integer.MAX_VALUE, Integer.MAX_VALUE - 1)` returns `0` when the jar follows the spec. Probing the behavior catches backports and vendor builds that parsing `IcebergBuild.version()` would miss. The kernel would then take a flag for spec or overflow semantics, and native writes would run only when the jar follows the spec. The test comment and the user guide should also cite the spec. The large-width overflow is a Java bug. The wrap at the integer minimum is Java's choice where the spec is silent. ## Additional context Widths above 2^30 are rare in practice. The fallback doesn't affect other widths, because every width up to 2^30 gives the same value under both formulas. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
