luoyuxia commented on PR #3420: URL: https://github.com/apache/fluss/pull/3420#issuecomment-4797391268
> > Thanks for working on this. > > I have one high-level question about the motivation/use case: if Fluss implements #3314, would that satisfy the requirement this PR is trying to address? > > My understanding is that #3314 would provide a partition-level "mark done" / readiness signal for Paimon lake tiering, while this PR reports an event-time watermark into Paimon snapshots. If the main goal is to let downstream batch jobs know when a partition is ready to process, #3314 may be the more direct mechanism and this PR might not be necessary. But if the requirement is specifically to expose event-time progress through Paimon snapshot watermarks, then this PR still seems to cover a different need. > > Could you clarify the intended downstream use case here? > > Yes, the main purpose of this PR is to determine whether a partition is done based on the watermark. Paimon’s PartitionMarkDone mechanism supports the watermark-based mode. I’m not sure whether the solution proposed in #3314 takes the watermark into consideration. If we need to determine whether a partition is done based on the watermark, the tiering job must be able to get the watermark. So I think this PR is a prerequisite for #3314. Thanks. The issue is expected to take the watermark into consideration. And I agree on the pr or some similar thing is a prerequisite for #3314. But I think may be we need to limit the scope of the pr to avoid over implementation since seems write watermark is not needed. @beryllw Could you please have to look to determine which part for watermark is needed in you design for #3314 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
