JingsongLi commented on code in PR #10625:
URL: https://github.com/apache/paimon/pull/10625#discussion_r4374819956
##########
paimon-core/src/main/java/org/apache/paimon/utils/TagManager.java:
##########
@@ -132,11 +172,204 @@ public void createTag(
List<TagCallback> callbacks,
boolean ignoreIfExists) {
checkArgument(!StringUtils.isNullOrWhitespaceOnly(tagName), "Tag name
shouldn't be blank.");
- if (tagExists(tagName)) {
- checkArgument(ignoreIfExists, "Tag '%s' already exists.", tagName);
+ // The existence check and metadata write must not race with snapshot
expiration, which
+ // deletes files based on a tag set collected under the same lock.
+ ReentrantLock coordinationLock = coordinationLock();
+ coordinationLock.lock();
+ try {
+ if (tagExists(tagName)) {
+ checkArgument(ignoreIfExists, "Tag '%s' already exists.",
tagName);
+ if (awaitSettledTag(tagName)) {
+ return;
+ }
+ }
+ Runnable beforeCommit = beforeTagCommit;
+ if (beforeCommit != null) {
+ beforeCommit.run();
+ }
+ ensureSnapshotRetained(snapshot, tagName, false);
+ String owner = UUID.randomUUID().toString();
+ createPendingTagMarker(tagName, owner);
+ try {
+ createOrReplaceTag(snapshot, tagName, timeRetained, callbacks,
true);
+ } finally {
+ fileIO.deleteQuietly(pendingTagMarkerPath(tagName, owner));
Review Comment:
[P1] Keep the pending fence until publication is validated or rollback is
confirmed
This finally clears the pending marker even when the tag remains
unvalidated. `overwriteFileUtf8` can persist the complete tag and then throw on
close; its existing catch leaves that file because the outcome is unknown, so
post-write retention validation never runs. Alternatively, validation can fail
while `deleteQuietly(tagPath)` fails to remove the file. In both cases this
finally removes the fence, and another JVM's `createTag(...,
ignoreIfExists=true)` sees the tag with no pending marker and returns success.
I reproduced both cases while actual expiration was paused after its final
tag scan: the retry exited successfully, then expiration completed with the tag
still present but its snapshot and delta-manifest list deleted. The first case
throws only after a real successful tag write; the second returns false only
for the tag's deletion. Keep the pending state until publication has passed
validation or rollback is confirmed. An unresolved write/delete outcome should
remain fenced (as the documented crashed-creator case does), or be resolved
safely before acknowledging retries.
##########
paimon-core/src/main/java/org/apache/paimon/utils/TagManager.java:
##########
@@ -181,6 +416,15 @@ private void createOrReplaceTag(
e);
}
+ if (rollbackIfSnapshotExpired) {
+ try {
+ ensureSnapshotRetained(snapshot, tagName, true);
+ } catch (RuntimeException e) {
+ fileIO.deleteQuietly(tagPath);
Review Comment:
[P1] Roll back only the publication owned by this creator
Two creator JVMs can both pass the initial absence check, so the per-name
pending markers do not make this deletion owned. With expiration paused after
its final scan, let B pass the absence check and wait; A publishes `same` for
fenced snapshot 2 and pauses after the write. B then publishes `same` for
retained snapshot 4, passes validation and returns success. When A resumes, its
snapshot-2 validation fails and this deletion removes B's already acknowledged
snapshot-4 tag.
An actual-expirer/two-creator JDK 8 reproduction confirms the tag points to
snapshot 4 after B succeeds, then disappears when A rolls back, although
snapshot 4 remains valid. Removing only this new deletion in a scratch control
preserves B's tag under the identical schedule, so this is additional damage
beyond the baseline's overwrite race. Please give same-name publication
cross-process ownership and make rollback conditional on that ownership; a
separate read-before-delete check still races with another publication. Add the
overlapping-creator case alongside the existing waiter test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]