SteNicholas commented on PR #8753: URL: https://github.com/apache/hudi/pull/8753#issuecomment-1552823258
@danny0405, in the data warehouse incremental scenario, whether partition completes to write is depended on the progress of writing, which is used to commit partition for trigger downstream task to read the committed partition data. The watermark could identify the current progress of writing data and infer the committed partition. In BiliBili, incremental ETL uses the watermark to identify the partition committed and downstream ETL tasks could only query the committed partition data. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
