JingsongLi commented on code in PR #11302:
URL: https://github.com/apache/paimon/pull/11302#discussion_r4642758898
##########
paimon-core/src/main/java/org/apache/paimon/tag/TagAutoCreation.java:
##########
@@ -132,30 +150,118 @@ public boolean forceCreatingSnapshot() {
return false;
}
+ /** Validate a newly supplied watermark before it can advance the snapshot
watermark. */
+ public static void validateWatermark(
+ CoreOptions options,
+ long watermark,
+ SnapshotManager snapshotManager,
+ @Nullable Snapshot latestSnapshot,
+ @Nullable TagAutoCreation creation) {
+ if (watermark == Long.MIN_VALUE
+ || options.tagCreationMode() !=
CoreOptions.TagCreationMode.WATERMARK) {
+ return;
+ }
+ ZoneId zone = ZoneId.of(options.sinkWatermarkTimeZone());
+ LocalDateTime time =
LocalDateTime.ofInstant(Instant.ofEpochMilli(watermark), zone);
+ TagPeriodHandler tagPeriodHandler = TagPeriodHandler.create(options);
+ Duration delay = options.tagCreationDelay();
+ try {
+
tagPeriodHandler.timeToTag(tagPeriodHandler.normalizeToPreviousTag(time));
+ if (latestSnapshot != null && !delay.isZero()) {
+ tagPeriodHandler.timeToTag(
+
tagPeriodHandler.normalizeToPreviousTag(time.minus(delay)));
+ }
+ } catch (DateTimeException e) {
+ DateTimeException cause = e;
+ if (latestSnapshot != null) {
+ if (creation == null) {
+ if (options.writeOnly()) {
+ // A dedicated creator can remember tags that have
already expired.
+ return;
+ }
+ TagManager tagManager =
+ new TagManager(
+ snapshotManager.fileIO(),
+ snapshotManager.tablePath(),
+ snapshotManager.branch());
+ creation =
+ create(options, snapshotManager, tagManager,
Collections.emptyList());
+ checkState(creation != null, "Expected a tag creator for
watermark mode.");
+ }
+ try {
+ LocalDateTime nextTag =
+ creation.processSnapshotsUpTo(latestSnapshot.id(),
false);
+ LocalDateTime tag = creation.tagTime(time, nextTag);
Review Comment:
**[P1] Validate the watermark that drives automatic tag completion.** With
hourly tags, UTC, and `tag.automatic-completion=true`, I ran three Flink SQL
inserts with watermarks `1726000000000`, `1726000000000000`, and
`1726000060000`. All three committed. The last snapshot still had the
mis-scaled watermark, and the table had tags `2024-09-10 19`, `20`, and `21`
even though the last valid input was 20:27 UTC. The fallback validates only the
next completion tag, which is printable, and lets future inserts inherit the
bad maximum. Validate the eventual delayed tag boundary for forward watermarks
when completion is enabled, and test a valid write after the rejected input.
##########
paimon-core/src/main/java/org/apache/paimon/tag/TagAutoCreation.java:
##########
@@ -132,30 +150,118 @@ public boolean forceCreatingSnapshot() {
return false;
}
+ /** Validate a newly supplied watermark before it can advance the snapshot
watermark. */
+ public static void validateWatermark(
+ CoreOptions options,
+ long watermark,
+ SnapshotManager snapshotManager,
+ @Nullable Snapshot latestSnapshot,
+ @Nullable TagAutoCreation creation) {
+ if (watermark == Long.MIN_VALUE
+ || options.tagCreationMode() !=
CoreOptions.TagCreationMode.WATERMARK) {
+ return;
+ }
+ ZoneId zone = ZoneId.of(options.sinkWatermarkTimeZone());
+ LocalDateTime time =
LocalDateTime.ofInstant(Instant.ofEpochMilli(watermark), zone);
+ TagPeriodHandler tagPeriodHandler = TagPeriodHandler.create(options);
+ Duration delay = options.tagCreationDelay();
+ try {
+
tagPeriodHandler.timeToTag(tagPeriodHandler.normalizeToPreviousTag(time));
+ if (latestSnapshot != null && !delay.isZero()) {
+ tagPeriodHandler.timeToTag(
+
tagPeriodHandler.normalizeToPreviousTag(time.minus(delay)));
+ }
+ } catch (DateTimeException e) {
+ DateTimeException cause = e;
+ if (latestSnapshot != null) {
+ if (creation == null) {
+ if (options.writeOnly()) {
+ // A dedicated creator can remember tags that have
already expired.
+ return;
Review Comment:
**[P2] Reject a far-future watermark with a separate tag maintainer.** I
tested hourly tags with completion disabled and Flink SQL writes using
`write-only=true`. The first write at `1726000000000` allowed a separate
`TagAutoManager` to create `2024-09-10 19`. The next write at
`1726000000000000` and a subsequent write at `1726003600000` both committed.
The latest snapshot kept `1726000000000000`, and the separate maintainer failed
while formatting year 56664. The older-watermark case with an expired tag in
the new test can still be allowed, but the writer can reject a forward
watermark when both the immediate and delayed tag times exceed the formatter's
year range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]