aokolnychyi commented on a change in pull request #675: Inherit snapshot ids
for manifest entries
URL: https://github.com/apache/incubator-iceberg/pull/675#discussion_r370757797
##########
File path: core/src/main/java/org/apache/iceberg/BaseRewriteManifests.java
##########
@@ -295,7 +305,10 @@ private WriterWrapper getWriter(Object key) {
@Override
protected void cleanUncommitted(Set<ManifestFile> committed) {
cleanUncommitted(newManifests, committed);
- cleanUncommitted(addedManifests, committed);
+ if (!snapshotIdInheritanceEnabled) {
Review comment:
If inheritance is enabled, `addedManifests` contains original manifests that
should not be removed no matter what the operation outcome is. If the commit
fails, the caller can retry. If the commit succeeds, the manifests are part of
the metadata now.
If inheritance is not enabled, `addedManifests` will contain manifest
copies. Those must be always cleaned up as the caller doesn't have access to
them.
I tried to summarize that in the description to `AppendFiles`.
> By default, the manifest will be rewritten to assign all entries this
update's snapshot ID.
> In that case, it is always the responsibility of the caller to manage the
lifecycle of
> the original manifest.
>
> If manifest entries are allowed to inherit the snapshot ID assigned on
commit, the manifest
> should never be deleted manually if the commit succeeds as it will become
part of the table
> metadata and will be cleaned up on expiry. If the manifest gets merged
with others while
> preparing a new snapshot, it will be deleted automatically if this
operation is successful.
> If the commit fails, the manifest will never be deleted and it is up to
the caller whether
> to delete or reuse it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
