sdaberdaku opened a new pull request, #18936: URL: https://github.com/apache/iceberg/pull/18936
Incremental computation of partition statistics keyed the rows of the previous statistics file by their partition in the unified partition type, while `PartitionMap` compares the partitions of a spec in that spec's own type. For a spec whose fields are not a prefix of the unified type, for example after removing a partition field that is not the last, previous rows collided and new statistics never matched them, so the file lost partitions and undercounted others. The previous rows are now keyed by their partition projected to the type of their spec. A spec whose source column was dropped cannot be projected and falls back to a full computation. Test plan: `incrementalComputeAfterRemovingPartitionField` removes the first partition field, computes incrementally twice and compares the result with a full computation of the same snapshot, for format versions 2 and above, in Avro and Parquet. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
