taoran92 opened a new pull request, #11302:
URL: https://github.com/apache/paimon/pull/11302

   Fixes #11301.
   
   ### Purpose
   
   With watermark-based automatic tags enabled, a Flink batch `INSERT` can 
publish a watermark that the tag formatter cannot represent. For example, 
Paimon interprets `end-input.watermark=1726000000000000` as epoch milliseconds, 
yielding the year 56664. Tag creation fails because the formatter requires a 
four-digit year. Subsequent writes inherit the larger watermark, so tag 
creation still fails after a later `INSERT` supplies a valid millisecond value.
   
   When a supplied value sets or advances the snapshot watermark on a table 
with `tag.automatic-creation=watermark`, the Java commit path checks whether it 
can format any tag name due for that watermark before publishing the snapshot. 
The check uses the configured tag period and time zone. The first tag is due 
immediately, and subsequent tags follow the configured delay. The Flink option 
description now specifies epoch milliseconds.
   
   On a write-only table, the first commit validates the watermark before 
publishing the snapshot. Once a snapshot exists, a write-only commit can 
publish the watermark even if the separate tag creator later fails to format 
the tag name.
   
   ### Tests
   
   - Added watermark regression tests in `TagAutoManagerTest` covering 
rejection before publication and recovery on the next write, a 15-minute delay 
across the year boundary, existing and expired tags, and pending snapshots. The 
write-only case checks that a watermark rejected on an empty table can be 
committed after a separate creator has advanced past it and its tag has expired.
   - A process-time regression test verifies that an empty checkpoint does not 
force another snapshot while the old tag's deletion callback is still running.
   - Verified with Flink 1.20.3 batch SQL and `tag.creation-period=hourly`. The 
rejected `INSERT` left the snapshot and tag tables empty. The next `INSERT`, 
with watermark `1726003600000`, created snapshot 1 and the UTC tag `2024-09-10 
20`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to