sdaberdaku opened a new issue, #18937: URL: https://github.com/apache/iceberg/issues/18937
### Apache Iceberg version 1.12.0 (latest release) ### Query engine Other ### Please describe the bug 🐞 **What happens.** After partition spec evolution, the incremental computation of partition statistics writes a file that drops partitions and undercounts others. This is `PartitionStatsHandler.computeAndWriteStatsFile` when an ancestor snapshot already has a partition statistics file, which is also what Spark's `compute_partition_stats` runs. A full computation of the same snapshot is correct. Incremental computation was added in 1.10.0 (#12629), so 1.10.0 through 1.12.0 and main are affected. **Why.** `computeAndMergeStatsIncremental` puts the rows of the previous statistics file into a `PartitionMap` keyed by their partition, which is in the unified partition type ([L190](https://github.com/apache/iceberg/blob/d1a422988cbde266f10c9e174ddf7c36a08de51a/core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java#L190)). `PartitionMap` compares the partitions of a spec in that spec's own partition type ([L181](https://github.com/apache/iceberg/blob/d1a422988cbde266f10c9e174ddf7c36a08de51a/core/src/main/java/org/apache/iceberg/util/PartitionMap.java#L181)). When a spec's fields are not a prefix of the unified type, for example after removing a partition field that is not the last, the previous rows of that spec are read at the wrong positions. They collide, so only the last one is kept, and the newly computed statistics, keyed in the spec's type, never match them. **Reproduction** (in-memory catalog, format version 2 or 3): ```java Schema schema = new Schema(optional(1, "id", IntegerType.get()), optional(2, "part", IntegerType.get())); Table table = catalog.createTable(TableIdentifier.of("db", "t"), schema, PartitionSpec.builderFor(schema).identity("part").build(), Map.of("format-version", "2")); table.newAppend().appendFile(file(table.spec(), "part=1", 10)).appendFile(file(table.spec(), "part=2", 20)).commit(); computeAndCommit(table); // full table.updateSpec().removeField("part").addField(Expressions.bucket("id", 4)).commit(); table.newAppend() .appendFile(file(table.spec(), "id_bucket_4=0", 5)) .appendFile(file(table.spec(), "id_bucket_4=1", 7)) .appendFile(file(table.spec(), "id_bucket_4=2", 9)) .commit(); computeAndCommit(table); // incremental table.newAppend().appendFile(file(table.spec(), "id_bucket_4=0", 3)).commit(); computeAndCommit(table); // incremental ``` where `computeAndCommit` runs `PartitionStatsHandler.computeAndWriteStatsFile(table)` and registers the result, and `file` builds a `DataFile` with the given partition path and record count. | Partition of spec 1 | Incremental result | Full computation of the same snapshot | |---|---|---| | `id_bucket_4=0` | 3 records, 1 file | 8 records, 2 files | | `id_bucket_4=1` | missing | 7 records, 1 file | | `id_bucket_4=2` | 9 records, 1 file | 9 records, 1 file | **Impact.** Every later incremental computation starts from the wrong file, so the error persists. Tables that changed their partitioning in this way and use incrementally computed statistics files, for example from Spark's `compute_partition_stats`, may already have wrong files. Workaround: remove the table's partition statistics files and compute again, which is a full computation. Fix: #18936 ### Willingness to contribute - [x] I can contribute a fix for this bug independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
