sdaberdaku opened a new issue, #18937:
URL: https://github.com/apache/iceberg/issues/18937

   ### Apache Iceberg version
   
   1.12.0 (latest release)
   
   ### Query engine
   
   Other
   
   ### Please describe the bug 🐞
   
   **What happens.** After partition spec evolution, the incremental 
computation of partition statistics writes a file that drops partitions and 
undercounts others. This is `PartitionStatsHandler.computeAndWriteStatsFile` 
when an ancestor snapshot already has a partition statistics file, which is 
also what Spark's `compute_partition_stats` runs. A full computation of the 
same snapshot is correct. Incremental computation was added in 1.10.0 (#12629), 
so 1.10.0 through 1.12.0 and main are affected.
   
   **Why.** `computeAndMergeStatsIncremental` puts the rows of the previous 
statistics file into a `PartitionMap` keyed by their partition, which is in the 
unified partition type
   
([L190](https://github.com/apache/iceberg/blob/d1a422988cbde266f10c9e174ddf7c36a08de51a/core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java#L190)).
   `PartitionMap` compares the partitions of a spec in that spec's own 
partition type
   
([L181](https://github.com/apache/iceberg/blob/d1a422988cbde266f10c9e174ddf7c36a08de51a/core/src/main/java/org/apache/iceberg/util/PartitionMap.java#L181)).
   When a spec's fields are not a prefix of the unified type, for example after 
removing a partition field that is not the last, the previous rows of that spec 
are read at the wrong positions. They collide, so only the last one is kept, 
and the newly computed statistics, keyed in the spec's type, never match them.
   
   **Reproduction** (in-memory catalog, format version 2 or 3):
   
   ```java
   Schema schema = new Schema(optional(1, "id", IntegerType.get()), optional(2, 
"part", IntegerType.get()));
   Table table = catalog.createTable(TableIdentifier.of("db", "t"), schema,
       PartitionSpec.builderFor(schema).identity("part").build(), 
Map.of("format-version", "2"));
   
   table.newAppend().appendFile(file(table.spec(), "part=1", 
10)).appendFile(file(table.spec(), "part=2", 20)).commit();
   computeAndCommit(table); // full
   
   table.updateSpec().removeField("part").addField(Expressions.bucket("id", 
4)).commit();
   table.newAppend()
       .appendFile(file(table.spec(), "id_bucket_4=0", 5))
       .appendFile(file(table.spec(), "id_bucket_4=1", 7))
       .appendFile(file(table.spec(), "id_bucket_4=2", 9))
       .commit();
   computeAndCommit(table); // incremental
   table.newAppend().appendFile(file(table.spec(), "id_bucket_4=0", 
3)).commit();
   computeAndCommit(table); // incremental
   ```
   
   where `computeAndCommit` runs 
`PartitionStatsHandler.computeAndWriteStatsFile(table)` and registers the 
result, and `file` builds a `DataFile` with the given partition path and record 
count.
   
   | Partition of spec 1 | Incremental result | Full computation of the same 
snapshot |
   |---|---|---|
   | `id_bucket_4=0` | 3 records, 1 file | 8 records, 2 files |
   | `id_bucket_4=1` | missing | 7 records, 1 file |
   | `id_bucket_4=2` | 9 records, 1 file | 9 records, 1 file |
   
   **Impact.** Every later incremental computation starts from the wrong file, 
so the error persists. Tables that changed their partitioning in this way and 
use incrementally computed statistics files, for example from Spark's 
`compute_partition_stats`, may already have wrong files. Workaround: remove the 
table's partition statistics files and compute again, which is a full 
computation.
   
   Fix: #18936
   
   ### Willingness to contribute
   
   - [x] I can contribute a fix for this bug independently
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to