JingsongLi commented on code in PR #11302:
URL: https://github.com/apache/paimon/pull/11302#discussion_r4657851338
##########
paimon-core/src/main/java/org/apache/paimon/tag/TagAutoCreation.java:
##########
@@ -132,30 +152,127 @@ public boolean forceCreatingSnapshot() {
return false;
}
+ /** Validate a newly supplied watermark before it can advance the snapshot
watermark. */
+ public static void validateWatermark(
+ CoreOptions options,
+ long watermark,
+ SnapshotManager snapshotManager,
+ @Nullable Snapshot latestSnapshot,
+ @Nullable TagAutoCreation creation) {
+ if (watermark == Long.MIN_VALUE
+ || options.tagCreationMode() !=
CoreOptions.TagCreationMode.WATERMARK) {
+ return;
+ }
+ ZoneId zone = ZoneId.of(options.sinkWatermarkTimeZone());
+ LocalDateTime time =
LocalDateTime.ofInstant(Instant.ofEpochMilli(watermark), zone);
+ TagPeriodHandler tagPeriodHandler = TagPeriodHandler.create(options);
+ Duration delay = options.tagCreationDelay();
+ UnaryOperator<LocalDateTime> normalize =
tagPeriodHandler::normalizeForWatermarkValidation;
+ LocalDateTime firstTag = normalize.apply(time);
+ LocalDateTime delayedTag =
+ latestSnapshot == null || delay.isZero()
+ ? firstTag
+ : normalize.apply(time.minus(delay));
+ try {
+ tagPeriodHandler.timeToTag(firstTag);
+ if (!firstTag.equals(delayedTag)) {
+ tagPeriodHandler.timeToTag(delayedTag);
+ }
Review Comment:
**[P2] Validate the completion tag due after a restart.** With UTC,
`tag.creation-period-duration=70 s`, and `tag.automatic-completion=true`, I
committed watermark `253402300830000` and got tag `999912312359`. I then
reopened the committer and committed `253402300880000`. The watermark-derived
boundary still falls in year 9999, so validation passes and snapshot 2 is
published. Restoring progress from the minute-precision tag name makes
`+10000-01-01T00:00:10` the next tag, and that tag is due at snapshot 2.
Maintenance throws a `DateTimeException` and remains stuck on that snapshot.
Please validate the tag due under the restored completion progress before
publishing the snapshot.
##########
paimon-core/src/main/java/org/apache/paimon/tag/TagAutoCreation.java:
##########
@@ -132,30 +152,127 @@ public boolean forceCreatingSnapshot() {
return false;
}
+ /** Validate a newly supplied watermark before it can advance the snapshot
watermark. */
+ public static void validateWatermark(
+ CoreOptions options,
+ long watermark,
+ SnapshotManager snapshotManager,
+ @Nullable Snapshot latestSnapshot,
+ @Nullable TagAutoCreation creation) {
+ if (watermark == Long.MIN_VALUE
+ || options.tagCreationMode() !=
CoreOptions.TagCreationMode.WATERMARK) {
+ return;
+ }
+ ZoneId zone = ZoneId.of(options.sinkWatermarkTimeZone());
+ LocalDateTime time =
LocalDateTime.ofInstant(Instant.ofEpochMilli(watermark), zone);
+ TagPeriodHandler tagPeriodHandler = TagPeriodHandler.create(options);
+ Duration delay = options.tagCreationDelay();
+ UnaryOperator<LocalDateTime> normalize =
tagPeriodHandler::normalizeForWatermarkValidation;
+ LocalDateTime firstTag = normalize.apply(time);
+ LocalDateTime delayedTag =
+ latestSnapshot == null || delay.isZero()
+ ? firstTag
+ : normalize.apply(time.minus(delay));
+ try {
+ tagPeriodHandler.timeToTag(firstTag);
+ if (!firstTag.equals(delayedTag)) {
+ tagPeriodHandler.timeToTag(delayedTag);
+ }
+ } catch (DateTimeException e) {
+ DateTimeException cause = e;
+ // If even the delayed boundary is too far ahead, completing tags
one at a time
+ // or using a separate tag creator cannot make that boundary
printable.
+ boolean exceedsTagRange =
+ firstTag.getYear() > MAX_TAG_YEAR && delayedTag.getYear()
> MAX_TAG_YEAR;
+ if (latestSnapshot != null && !exceedsTagRange) {
+ if (creation == null) {
+ if (options.writeOnly()) {
+ // A dedicated creator can remember tags that have
already expired.
+ return;
Review Comment:
**[P2] Validate the first usable watermark for write-only writers.** With
hourly UTC tags, a 15-minute creation delay, and `write-only=true`, I committed
`Long.MIN_VALUE` and then `253402304700000` to a table with no tags. The
delayed boundary is printable, so this branch accepts the write. The separate
maintainer creates its first tag from the immediate boundary and throws a
`DateTimeException` for year 10000. A later write with a valid 2024 watermark
still inherits the larger value. This branch can also let an extreme negative
watermark wrap into a bogus future tag. Please require evidence of existing tag
progress before allowing this fallback. When that progress is unknown, validate
the immediate first-tag boundary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]