sdaberdaku opened a new pull request, #18936:
URL: https://github.com/apache/iceberg/pull/18936

   Incremental computation of partition statistics keyed the rows of the 
previous statistics file by their partition in the unified partition type, 
while `PartitionMap` compares the partitions of a spec in that spec's own type. 
For a spec whose fields are not a prefix of the unified type, for example after 
removing a partition field that is not the last, previous rows collided and new 
statistics never matched them, so the file lost partitions and undercounted 
others. The previous rows are now keyed by their partition projected to the 
type of their spec. A spec whose source column was dropped cannot be projected 
and falls back to a full computation.
   
   Test plan: `incrementalComputeAfterRemovingPartitionField` removes the first 
partition field, computes incrementally twice and compares the result with a 
full computation of the same snapshot, for format versions 2 and above, in Avro 
and Parquet.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to