Shawn-Hx commented on PR #3420: URL: https://github.com/apache/fluss/pull/3420#issuecomment-4795342138
> Thanks for working on this. > > I have one high-level question about the motivation/use case: if Fluss implements #3314, would that satisfy the requirement this PR is trying to address? > > My understanding is that #3314 would provide a partition-level "mark done" / readiness signal for Paimon lake tiering, while this PR reports an event-time watermark into Paimon snapshots. If the main goal is to let downstream batch jobs know when a partition is ready to process, #3314 may be the more direct mechanism and this PR might not be necessary. But if the requirement is specifically to expose event-time progress through Paimon snapshot watermarks, then this PR still seems to cover a different need. > > Could you clarify the intended downstream use case here? Yes, the main purpose of this PR is to determine whether a partition is done based on the watermark. Paimon’s PartitionMarkDone mechanism supports the watermark-based mode. I’m not sure whether the solution proposed in #3314 takes the watermark into consideration. If we need to determine whether a partition is done based on the watermark, the tiering job must be able to get the watermark. So I think this PR is a prerequisite for #3314. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
