mvanhorn commented on code in PR #10625:
URL: https://github.com/apache/paimon/pull/10625#discussion_r4371517772
##########
paimon-core/src/main/java/org/apache/paimon/utils/TagManager.java:
##########
@@ -132,11 +161,127 @@ public void createTag(
List<TagCallback> callbacks,
boolean ignoreIfExists) {
checkArgument(!StringUtils.isNullOrWhitespaceOnly(tagName), "Tag name
shouldn't be blank.");
- if (tagExists(tagName)) {
- checkArgument(ignoreIfExists, "Tag '%s' already exists.", tagName);
+ // The existence check and metadata write must not race with snapshot
expiration, which
+ // deletes files based on a tag set collected under the same lock.
+ ReentrantLock coordinationLock = coordinationLock();
+ coordinationLock.lock();
+ try {
+ if (tagExists(tagName)) {
+ checkArgument(ignoreIfExists, "Tag '%s' already exists.",
tagName);
+ return;
Review Comment:
Fixed in 7ba4304. createTag now writes a pending marker under
tag-pending/<name>/<owner> before it publishes the tag file, and deletes it
only after its post-write validation has passed or rolled the tag back. The
ignoreIfExists branch no longer returns on a visible file: it waits until no
pending marker is left for that name, then returns only if the tag is still
there. If A rolled it back, B creates the tag itself and goes through the same
validation, so in your interleaving B fails with the snapshot #2 error instead
of reporting success.
New tests:
- testIgnoreIfExistsWaitsForAnotherCreatorsProvisionalTag runs your
scenario: expiration paused after its final tag scan, JVM A publishes tag-same
and pauses before validation, JVM B calls createTag("same",
ignoreIfExists=true). B is still waiting 2 seconds later. After A is released,
both exit non-zero, the tag is gone, and expiration finishes.
- testIgnoreIfExistsFailsWhileAnotherCreatorStaysPending covers a pending
marker that never clears.
With the ignoreIfExists wait taken out, both tests fail.
ExpireSnapshotsTest, TagManagerTest, TagTest and TagAutoManagerTest pass on JDK
11 (71 tests), and spotless and checkstyle pass on paimon-core.
One limit, the same kind as the expire markers: a creator that crashes
between publishing and cleanup leaves its pending marker behind. A later
ignoreIfExists call for that name then fails after 60 seconds, and the error
names the directory to remove. It does not acknowledge the tag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
