JingsongLi commented on code in PR #11302:
URL: https://github.com/apache/paimon/pull/11302#discussion_r4657851338


##########
paimon-core/src/main/java/org/apache/paimon/tag/TagAutoCreation.java:
##########
@@ -132,30 +152,127 @@ public boolean forceCreatingSnapshot() {
         return false;
     }
 
+    /** Validate a newly supplied watermark before it can advance the snapshot 
watermark. */
+    public static void validateWatermark(
+            CoreOptions options,
+            long watermark,
+            SnapshotManager snapshotManager,
+            @Nullable Snapshot latestSnapshot,
+            @Nullable TagAutoCreation creation) {
+        if (watermark == Long.MIN_VALUE
+                || options.tagCreationMode() != 
CoreOptions.TagCreationMode.WATERMARK) {
+            return;
+        }
+        ZoneId zone = ZoneId.of(options.sinkWatermarkTimeZone());
+        LocalDateTime time = 
LocalDateTime.ofInstant(Instant.ofEpochMilli(watermark), zone);
+        TagPeriodHandler tagPeriodHandler = TagPeriodHandler.create(options);
+        Duration delay = options.tagCreationDelay();
+        UnaryOperator<LocalDateTime> normalize = 
tagPeriodHandler::normalizeForWatermarkValidation;
+        LocalDateTime firstTag = normalize.apply(time);
+        LocalDateTime delayedTag =
+                latestSnapshot == null || delay.isZero()
+                        ? firstTag
+                        : normalize.apply(time.minus(delay));
+        try {
+            tagPeriodHandler.timeToTag(firstTag);
+            if (!firstTag.equals(delayedTag)) {
+                tagPeriodHandler.timeToTag(delayedTag);
+            }

Review Comment:
   **[P2] Validate the completion tag due after a restart.** With UTC, 
`tag.creation-period-duration=70 s`, and `tag.automatic-completion=true`, I 
committed watermark `253402300830000` and got tag `999912312359`. I then 
reopened the committer and committed `253402300880000`. The watermark-derived 
boundary still falls in year 9999, so validation passes and snapshot 2 is 
published. Restoring progress from the minute-precision tag name makes 
`+10000-01-01T00:00:10` the next tag, and that tag is due at snapshot 2. 
Maintenance throws a `DateTimeException` and remains stuck on that snapshot. 
Please validate the tag due under the restored completion progress before 
publishing the snapshot.



##########
paimon-core/src/main/java/org/apache/paimon/tag/TagAutoCreation.java:
##########
@@ -132,30 +152,127 @@ public boolean forceCreatingSnapshot() {
         return false;
     }
 
+    /** Validate a newly supplied watermark before it can advance the snapshot 
watermark. */
+    public static void validateWatermark(
+            CoreOptions options,
+            long watermark,
+            SnapshotManager snapshotManager,
+            @Nullable Snapshot latestSnapshot,
+            @Nullable TagAutoCreation creation) {
+        if (watermark == Long.MIN_VALUE
+                || options.tagCreationMode() != 
CoreOptions.TagCreationMode.WATERMARK) {
+            return;
+        }
+        ZoneId zone = ZoneId.of(options.sinkWatermarkTimeZone());
+        LocalDateTime time = 
LocalDateTime.ofInstant(Instant.ofEpochMilli(watermark), zone);
+        TagPeriodHandler tagPeriodHandler = TagPeriodHandler.create(options);
+        Duration delay = options.tagCreationDelay();
+        UnaryOperator<LocalDateTime> normalize = 
tagPeriodHandler::normalizeForWatermarkValidation;
+        LocalDateTime firstTag = normalize.apply(time);
+        LocalDateTime delayedTag =
+                latestSnapshot == null || delay.isZero()
+                        ? firstTag
+                        : normalize.apply(time.minus(delay));
+        try {
+            tagPeriodHandler.timeToTag(firstTag);
+            if (!firstTag.equals(delayedTag)) {
+                tagPeriodHandler.timeToTag(delayedTag);
+            }
+        } catch (DateTimeException e) {
+            DateTimeException cause = e;
+            // If even the delayed boundary is too far ahead, completing tags 
one at a time
+            // or using a separate tag creator cannot make that boundary 
printable.
+            boolean exceedsTagRange =
+                    firstTag.getYear() > MAX_TAG_YEAR && delayedTag.getYear() 
> MAX_TAG_YEAR;
+            if (latestSnapshot != null && !exceedsTagRange) {
+                if (creation == null) {
+                    if (options.writeOnly()) {
+                        // A dedicated creator can remember tags that have 
already expired.
+                        return;

Review Comment:
   **[P2] Validate the first usable watermark for write-only writers.** With 
hourly UTC tags, a 15-minute creation delay, and `write-only=true`, I committed 
`Long.MIN_VALUE` and then `253402304700000` to a table with no tags. The 
delayed boundary is printable, so this branch accepts the write. The separate 
maintainer creates its first tag from the immediate boundary and throws a 
`DateTimeException` for year 10000. A later write with a valid 2024 watermark 
still inherits the larger value. This branch can also let an extreme negative 
watermark wrap into a bogus future tag. Please require evidence of existing tag 
progress before allowing this fallback. When that progress is unknown, validate 
the immediate first-tag boundary.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to