kumarpritam863 commented on code in PR #18583:
URL: https://github.com/apache/iceberg/pull/18583#discussion_r4315886519
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/TableUpdater.java:
##########
@@ -101,6 +102,13 @@ private void findOrCreateTable(
}
private void findOrCreateBranch(TableIdentifier identifier, String branch) {
+ if (SnapshotRef.MAIN_BRANCH.equals(branch)) {
+ // The main branch is created by the first commit to the table, so there
is nothing to
+ // create. Creating it here would commit an empty snapshot to a table
that has none, and
+ // every writer subtask that gets here before the first data commit
would add another.
+ return;
+ }
Review Comment:
Actually for other branches also check create an empty snapshot if there is
nothing but skipping for other branches is not safe as that snapshot currently
also gives the branch its own history: without it, the first commit to the
branch is validated by DynamicCommitter's MaxCommittedCheckpointIdValidator
against the ancestry that SnapshotUtil.latestSnapshot(metadata, branch) returns
for a branch that does not exist, which is the head of main. If main has
already committed the same job and operator for that checkpoint, the commit to
the new branch is treated as a duplicate and skipped. I reproduced this by also
skipping creation for other branches:
TestDynamicIcebergSink#testMultipleBranches then lost the commit to branch1 in
about one run in four. For main a missing ref has no ancestry to fall back to,
so the early return is safe.
I am still checking if we can make some generic checks but I was thinking to
take that as a seperate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
