amogh-jahagirdar commented on code in PR #15154:
URL: https://github.com/apache/iceberg/pull/15154#discussion_r2743729254
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/ExpireSnapshotsSparkAction.java:
##########
@@ -174,14 +182,71 @@ public Dataset<FileInfo> expireFiles() {
// fetch valid files after expiration
TableMetadata updatedMetadata = ops.refresh();
- Dataset<FileInfo> validFileDS = fileDS(updatedMetadata);
- // fetch files referenced by expired snapshots
+ // find IDs of expired snapshots
Set<Long> deletedSnapshotIds = findExpiredSnapshotIds(originalMetadata,
updatedMetadata);
- Dataset<FileInfo> deleteCandidateFileDS = fileDS(originalMetadata,
deletedSnapshotIds);
Review Comment:
Hm, a lot of the cases called out in the PR description should be implicitly
handled when you look at how fileDS works? e.g. we're creating the set of files
from the set of manifests, and if there are no manifests it's already an empty
set that we're doing the anti-join against.
Do we have any particular cases that we see improve after this change (if
there are numbers that would be helpful)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
