szehon-ho commented on code in PR #6661:
URL: https://github.com/apache/iceberg/pull/6661#discussion_r1093546826
##########
core/src/main/java/org/apache/iceberg/PartitionsTable.java:
##########
@@ -236,5 +273,17 @@ void update(DataFile file) {
this.fileCount += 1;
this.specId = file.specId();
}
+
+ void update(DeleteFile file) {
+ if (file.content() == FileContent.POSITION_DELETES) {
+ this.posDeleteRecordCount += file.recordCount();
Review Comment:
Yea I think this has the same problem I mentioned in the earlier pr :
https://github.com/apache/iceberg/pull/4005 (which was doing the same thing).
Summing will not give a unique delete file count.
I think one option as I mentioned is to use the new PositionDeletes scan :
https://github.com/apache/iceberg/pull/6365 , which you can then use to stream
and update partitions.
It's probably less memory intensive than trying to cache every existing
delete file in a set, which would be the other solution for de-duping.
Would like to see what @RussellSpitzer thinks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
