aokolnychyi commented on a change in pull request #2328:
URL: https://github.com/apache/iceberg/pull/2328#discussion_r594664462
##########
File path: core/src/main/java/org/apache/iceberg/SnapshotProducer.java
##########
@@ -293,8 +293,15 @@ public void commit() {
taskOps.commit(base, updated.withUUID());
});
+ } catch (CommitFailedException commitFailedException) {
+ // We have an acknowledged failure from the Catalog. We are confident
that the commit has not been applied
+ Exceptions.suppressAndThrow(commitFailedException, this::cleanAll);
} catch (RuntimeException e) {
- Exceptions.suppressAndThrow(e, this::cleanAll);
+ LOG.error("Cannot determine whether the commit was successful or not,
the underlying data files may or " +
Review comment:
I agree it is a valid concern when clients don't handle
`UnknownStateExceptions` and blindly retry. However, we can try to address it
in another way. For example, if users have idempotent requirements, they either
should make their retry mechanisms smarter and be aware of
`UnknownCommitStateException` or use something that guarantees the same batch
won't be committed twice. We already handle such cases in Structured Streaming.
I hope we do the same in Flink jobs. We can generalize this approach and make
Iceberg handle such cases in batch writes too.
I feel the current approach is extremely dangerous. If we delete a file that
was actually committed, we corrupt the table and require a manual intervention.
Unfortunately, I've seen this quite frequently. Also, getting a reply from the
metastore or checking the commit status may take a substantial amount of time
allowing a concurrent operation to succeed (if the lock expires). That's the
worst what can happen as we will silently corrupt the table and will detect it
only while querying a specific portion of the table. I had to fix such a case
and we were simply lucky we found this out only after a couple of days while
rewriting manifests. Duplicating data is less critical than loosing it, in my
view.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
