amogh-jahagirdar commented on code in PR #4578:
URL: https://github.com/apache/iceberg/pull/4578#discussion_r885147961
##########
core/src/main/java/org/apache/iceberg/RemoveSnapshots.java:
##########
@@ -161,21 +174,89 @@ public List<Snapshot> apply() {
private TableMetadata internalApply() {
this.base = ops.refresh();
+ if (base.snapshots().isEmpty()) {
+ return base;
+ }
Set<Long> idsToRetain = Sets.newHashSet();
- List<Long> ancestorIds = SnapshotUtil.ancestorIds(base.currentSnapshot(),
base::snapshot);
- if (minNumSnapshots >= ancestorIds.size()) {
- idsToRetain.addAll(ancestorIds);
- } else {
- idsToRetain.addAll(ancestorIds.subList(0, minNumSnapshots));
+ // Identify refs that should be removed
+ Map<String, SnapshotRef> retainedRefs = computeRetainedRefs(base.refs());
+ Map<Long, List<String>> retainedIdToRefs = Maps.newHashMap();
+ for (Map.Entry<String, SnapshotRef> retainedRefEntry :
retainedRefs.entrySet()) {
+ long snapshotId = retainedRefEntry.getValue().snapshotId();
+ retainedIdToRefs.putIfAbsent(snapshotId, Lists.newArrayList());
+ retainedIdToRefs.get(snapshotId).add(retainedRefEntry.getKey());
+ }
+
+ for (long idToRemove : idsToRemove) {
+ List<String> refsForId = retainedIdToRefs.get(idToRemove);
+ Preconditions.checkArgument(refsForId == null,
+ "Cannot expire %s. Still referenced by refs: %s", idToRemove,
refsForId);
}
- TableMetadata updateMeta = base.removeSnapshotsIf(snapshot ->
- idsToRemove.contains(snapshot.snapshotId()) ||
- (snapshot.timestampMillis() < expireOlderThan &&
!idsToRetain.contains(snapshot.snapshotId())));
- List<Snapshot> updateSnapshots = updateMeta.snapshots();
- List<Snapshot> baseSnapshots = base.snapshots();
- return updateSnapshots.size() != baseSnapshots.size() ? updateMeta : base;
+ idsToRetain.addAll(retainedIdToRefs.keySet());
+ Set<Long> branchSnapshotsToRetain =
computeAllBranchSnapshotsToRetain(retainedRefs);
+ idsToRetain.addAll(branchSnapshotsToRetain);
+ TableMetadata.Builder updatedMetaBuilder = TableMetadata.buildFrom(base);
+ idsToRemove.addAll(base.snapshots().stream().map(Snapshot::snapshotId)
+ .filter(snapshot ->
!idsToRetain.contains(snapshot)).collect(Collectors.toSet()));
+ base.refs().keySet().stream().filter(ref ->
!retainedRefs.containsKey(ref)).forEach(updatedMetaBuilder::removeRef);
+ updatedMetaBuilder.removeSnapshots(idsToRemove);
+ return updatedMetaBuilder.build();
+ }
+
+ private Map<String, SnapshotRef> computeRetainedRefs(Map<String,
SnapshotRef> refs) {
+ Map<String, SnapshotRef> retainedRefs = Maps.newHashMap();
+ retainedRefs.put(SnapshotRef.MAIN_BRANCH,
refs.get(SnapshotRef.MAIN_BRANCH));
Review Comment:
The issue of the case where there is no main branch yet has come up in a few
cases, I'm thinking it makes more sense that when parsing the metadata in case
there is no main branch, we populate it with the current snapshot id. That way
much of the API/core classes can operate under the assumption that main exists
(which it always does, even for older metadata files main was just the
existing, single lineage). Thoughts @rdblue?
I'll raise another PR for that, and we can discuss there if that makes sense
(if it doesn't we can close it).
But regardless of that PR, the check in the loop does look cleaner and also
handles the case where it doesn't exist so I'll just do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
