coca-alex commented on issue #14216: URL: https://github.com/apache/iceberg/issues/14216#issuecomment-6062144864
Hi all, I'd also like to add a concrete cross-implementation consequence of this issue. Tables written by one engine and read by another are increasingly common, and in that setup the signed/unsigned mismatch leads not just to inconsistent results but to **silent data loss in Java engines**. For example, iceberg-rust compares UUIDs via `uuid::Uuid`, whose ordering is derived from its `[u8; 16]` bytes, i.e. unsigned byte-wise (big-endian): - `Datum::partial_cmp` for `PrimitiveType::Uuid`: [datum.rs#L254-L259](https://github.com/apache/iceberg-rust/blob/ab047b5e2f6e2ba6b999731653a3f7722b84e6f7/crates/iceberg/src/spec/values/datum.rs#L254-L259) - bounds are serialized with `to_be_bytes`: [datum.rs#L453](https://github.com/apache/iceberg-rust/blob/ab047b5e2f6e2ba6b999731653a3f7722b84e6f7/crates/iceberg/src/spec/values/datum.rs#L453) The same comparison is used both when merging row-group stats into data-file `lower_bounds`/`upper_bounds` ([parquet_writer.rs#L334-L355](https://github.com/apache/iceberg-rust/blob/ab047b5e2f6e2ba6b999731653a3f7722b84e6f7/crates/iceberg/src/writer/file_writer/parquet_writer.rs#L334-L355), `update_state_min/max`) and when building manifest-list partition summaries ([manifest/writer.rs#L535-L558](https://github.com/apache/iceberg-rust/blob/ab047b5e2f6e2ba6b999731653a3f7722b84e6f7/crates/iceberg/src/spec/manifest/writer.rs#L535-L558)). So these bounds are correct per RFC 9562 and the [Parquet UUID sort order](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#uuid). On the read side, engines that plan scans through the Java library (Spark, Flink, and others built on iceberg-core) compare UUID literals with `Comparator.naturalOrder()`, i.e. `java.util.UUID.compareTo`, which compares two signed longs ([JDK-7025832](https://bugs.openjdk.org/browse/JDK-7025832)): - [Comparators.java#L49](https://github.com/apache/iceberg/blob/4d5811f2c701b15be4379686b17144503ed1e685/api/src/main/java/org/apache/iceberg/types/Comparators.java#L49) Bounds written in RFC order are then evaluated in signed order: - data files: `InclusiveEvalVisitor.eq` (used by `InclusiveMetricsEvaluator`), [InclusiveEvalVisitor.java#L253-L279](https://github.com/apache/iceberg/blob/4d5811f2c701b15be4379686b17144503ed1e685/api/src/main/java/org/apache/iceberg/expressions/InclusiveEvalVisitor.java#L253-L279) - manifests: `ManifestEvaluator.eq`, [ManifestEvaluator.java#L245-L265](https://github.com/apache/iceberg/blob/4d5811f2c701b15be4379686b17144503ed1e685/api/src/main/java/org/apache/iceberg/expressions/ManifestEvaluator.java#L245-L265) When a file or manifest contains UUIDs from both halves of the range (first hex digit `0-7` and `8-f`), the RFC-ordered `upper_bound` (e.g. `9…`) has its top bit set, so it is **negative** as a signed long. Java therefore sees `upper < literal` for any literal in the `0-7` half and returns `ROWS_CANNOT_MATCH`. #### Example A data file written by iceberg-rust contains: 40000000-0000-0000-0000-000000000000 50000000-0000-0000-0000-000000000000 90000000-0000-0000-0000-000000000000 Its bounds are `lower = 40000000-…`, `upper = 90000000-…` (unsigned order). In Spark: SELECT * FROM t WHERE id = '50000000-0000-0000-0000-000000000000' - `lower (0x40…) > lit (0x50…)`? No, so evaluation continues. - `upper (0x90…, negative as signed) < lit (0x50…, positive)`? **Yes**, so the result is `ROWS_CANNOT_MATCH`. The file is pruned, and the row is **silently missing** from the result: no error, no warning. The same happens in `ManifestEvaluator` for identity-partitioned UUID columns, where **entire manifests** are skipped. This is a correctness issue, not a performance one. In my opinion, it argues for prioritizing #14500. Its dual-comparator approach (prune only if both the signed and the unsigned evaluations agree) is what makes these mixed-writer tables safe to read without breaking tables written by Java. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
