JingsongLi commented on code in PR #11302:
URL: https://github.com/apache/paimon/pull/11302#discussion_r4642758898


##########
paimon-core/src/main/java/org/apache/paimon/tag/TagAutoCreation.java:
##########
@@ -132,30 +150,118 @@ public boolean forceCreatingSnapshot() {
         return false;
     }
 
+    /** Validate a newly supplied watermark before it can advance the snapshot 
watermark. */
+    public static void validateWatermark(
+            CoreOptions options,
+            long watermark,
+            SnapshotManager snapshotManager,
+            @Nullable Snapshot latestSnapshot,
+            @Nullable TagAutoCreation creation) {
+        if (watermark == Long.MIN_VALUE
+                || options.tagCreationMode() != 
CoreOptions.TagCreationMode.WATERMARK) {
+            return;
+        }
+        ZoneId zone = ZoneId.of(options.sinkWatermarkTimeZone());
+        LocalDateTime time = 
LocalDateTime.ofInstant(Instant.ofEpochMilli(watermark), zone);
+        TagPeriodHandler tagPeriodHandler = TagPeriodHandler.create(options);
+        Duration delay = options.tagCreationDelay();
+        try {
+            
tagPeriodHandler.timeToTag(tagPeriodHandler.normalizeToPreviousTag(time));
+            if (latestSnapshot != null && !delay.isZero()) {
+                tagPeriodHandler.timeToTag(
+                        
tagPeriodHandler.normalizeToPreviousTag(time.minus(delay)));
+            }
+        } catch (DateTimeException e) {
+            DateTimeException cause = e;
+            if (latestSnapshot != null) {
+                if (creation == null) {
+                    if (options.writeOnly()) {
+                        // A dedicated creator can remember tags that have 
already expired.
+                        return;
+                    }
+                    TagManager tagManager =
+                            new TagManager(
+                                    snapshotManager.fileIO(),
+                                    snapshotManager.tablePath(),
+                                    snapshotManager.branch());
+                    creation =
+                            create(options, snapshotManager, tagManager, 
Collections.emptyList());
+                    checkState(creation != null, "Expected a tag creator for 
watermark mode.");
+                }
+                try {
+                    LocalDateTime nextTag =
+                            creation.processSnapshotsUpTo(latestSnapshot.id(), 
false);
+                    LocalDateTime tag = creation.tagTime(time, nextTag);

Review Comment:
   **[P1] Validate the watermark that drives automatic tag completion.** With 
hourly tags, UTC, and `tag.automatic-completion=true`, I ran three Flink SQL 
inserts with watermarks `1726000000000`, `1726000000000000`, and 
`1726000060000`. All three committed. The last snapshot still had the 
mis-scaled watermark, and the table had tags `2024-09-10 19`, `20`, and `21` 
even though the last valid input was 20:27 UTC. The fallback validates only the 
next completion tag, which is printable, and lets future inserts inherit the 
bad maximum. Validate the eventual delayed tag boundary for forward watermarks 
when completion is enabled, and test a valid write after the rejected input.



##########
paimon-core/src/main/java/org/apache/paimon/tag/TagAutoCreation.java:
##########
@@ -132,30 +150,118 @@ public boolean forceCreatingSnapshot() {
         return false;
     }
 
+    /** Validate a newly supplied watermark before it can advance the snapshot 
watermark. */
+    public static void validateWatermark(
+            CoreOptions options,
+            long watermark,
+            SnapshotManager snapshotManager,
+            @Nullable Snapshot latestSnapshot,
+            @Nullable TagAutoCreation creation) {
+        if (watermark == Long.MIN_VALUE
+                || options.tagCreationMode() != 
CoreOptions.TagCreationMode.WATERMARK) {
+            return;
+        }
+        ZoneId zone = ZoneId.of(options.sinkWatermarkTimeZone());
+        LocalDateTime time = 
LocalDateTime.ofInstant(Instant.ofEpochMilli(watermark), zone);
+        TagPeriodHandler tagPeriodHandler = TagPeriodHandler.create(options);
+        Duration delay = options.tagCreationDelay();
+        try {
+            
tagPeriodHandler.timeToTag(tagPeriodHandler.normalizeToPreviousTag(time));
+            if (latestSnapshot != null && !delay.isZero()) {
+                tagPeriodHandler.timeToTag(
+                        
tagPeriodHandler.normalizeToPreviousTag(time.minus(delay)));
+            }
+        } catch (DateTimeException e) {
+            DateTimeException cause = e;
+            if (latestSnapshot != null) {
+                if (creation == null) {
+                    if (options.writeOnly()) {
+                        // A dedicated creator can remember tags that have 
already expired.
+                        return;

Review Comment:
   **[P2] Reject a far-future watermark with a separate tag maintainer.** I 
tested hourly tags with completion disabled and Flink SQL writes using 
`write-only=true`. The first write at `1726000000000` allowed a separate 
`TagAutoManager` to create `2024-09-10 19`. The next write at 
`1726000000000000` and a subsequent write at `1726003600000` both committed. 
The latest snapshot kept `1726000000000000`, and the separate maintainer failed 
while formatting year 56664. The older-watermark case with an expired tag in 
the new test can still be allowed, but the writer can reject a forward 
watermark when both the immediate and delayed tag times exceed the formatter's 
year range.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to