taoran92 opened a new pull request, #11302: URL: https://github.com/apache/paimon/pull/11302
Fixes #11301. ### Purpose With watermark-based automatic tags enabled, a Flink batch `INSERT` can publish a watermark that the tag formatter cannot represent. For example, Paimon interprets `end-input.watermark=1726000000000000` as epoch milliseconds, yielding the year 56664. Tag creation fails because the formatter requires a four-digit year. Subsequent writes inherit the larger watermark, so tag creation still fails after a later `INSERT` supplies a valid millisecond value. When a supplied value sets or advances the snapshot watermark on a table with `tag.automatic-creation=watermark`, the Java commit path checks whether it can format any tag name due for that watermark before publishing the snapshot. The check uses the configured tag period and time zone. The first tag is due immediately, and subsequent tags follow the configured delay. The Flink option description now specifies epoch milliseconds. On a write-only table, the first commit validates the watermark before publishing the snapshot. Once a snapshot exists, a write-only commit can publish the watermark even if the separate tag creator later fails to format the tag name. ### Tests - Added watermark regression tests in `TagAutoManagerTest` covering rejection before publication and recovery on the next write, a 15-minute delay across the year boundary, existing and expired tags, and pending snapshots. The write-only case checks that a watermark rejected on an empty table can be committed after a separate creator has advanced past it and its tag has expired. - A process-time regression test verifies that an empty checkpoint does not force another snapshot while the old tag's deletion callback is still running. - Verified with Flink 1.20.3 batch SQL and `tag.creation-period=hourly`. The rejected `INSERT` left the snapshot and tag tables empty. The next `INSERT`, with watermark `1726003600000`, created snapshot 1 and the UTC tag `2024-09-10 20`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
