mxm commented on code in PR #14810:
URL: https://github.com/apache/iceberg/pull/14810#discussion_r2664830343
##########
flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicCommitter.java:
##########
@@ -126,9 +125,14 @@ public void
commit(Collection<CommitRequest<DynamicCommittable>> commitRequests)
return;
}
- // For every table and every checkpoint, we store the list of
to-be-committed
- // DynamicCommittable.
- // There may be DynamicCommittable from previous checkpoints which have
not been committed yet.
+ /*
+ Each (table, branch, checkpoint) triplet must have only one commit
request.
+ There may be commit requests from previous checkpoints which have not
been committed yet.
+
+ We currently keep a List of commit requests per checkpoint instead of a
single CommitRequest<DynamicCommittable>
+ to process the Flink state from previous releases, which had multiple
commit requests due to a bug in the upstream
+ DynamicWriteResultAggregator. We should replace this with a single
commit request in the next major release.
Review Comment:
I think it is acceptable to advise users to upgrade their applications to
1.11.0 via a stop-with-savepoint. This note would have to go into an official
migration guide.
>Could we change the uid for the committer somehow? If we could do it, then
we could enforce using a news state for the new committer.
This wouldn't really improve the state migration process because there will
still be unmapped old state during migration.
>If we simplify the code and remove lists in this change, the committer
state would still be compatible and have the same operator ID
That would be nice if it doesn't introduce too much additional complexity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
