DGShimiru commented on issue #14216: URL: https://github.com/apache/iceberg/issues/14216#issuecomment-6056638608
Hello, We found that Spark can silently lose data on queries with `WHERE uuid = '...'` because it compares UUID statistics in signed order while Parquet writes them in unsigned order. Setup: - Spark 3.5.1, Iceberg 1.11.0 - Parquet files written by Spark (`rewrite_data_files`) - `uuid` column with metrics mode `counts`, so only Parquet row group min/max are used - rows not sorted by `uuid` When a row group min starts with 0x00–0x7f and its max with 0x80–0xff, the signed comparison sees min > max, the row group is skipped and the query returns no rows. Trino returns these rows only because it skips UUID statistics entirely, both in manifests and in Parquet footers, so UUID predicates get no pruning there. With `truncate(16)` the same happens in Spark at file level, using the bounds in manifests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
