SteNicholas opened a new pull request, #8753: URL: https://github.com/apache/hudi/pull/8753
### Change Logs `AbstractStreamWriteFunction` could process watermark from `StreamWriteOperator` and send `WriteMetadataEvent` with the watermark of write function to `StreamWriteOperatorCoordinator`. `StreamWriteOperatorCoordinator` stores the min watermark of `WriteMetadataEvent`s from subtasks into the extra metadata of `HoodieCommitMetadata`. Meanwhile, `StreamWriteOperatorCoordinator` advances the min watermark for subtasks in which no data has written, and does not advance watermark for commit on empty batch. ### Impact `StreamWriteOperatorCoordinator` stores the min watermark of write function into the extra metadata of `HoodieCommitMetadata` to identify the current progress of writing. ### Risk level (write none, low medium or high below) None. ### Documentation Update None. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Change Logs and Impact were stated clearly - [x] Adequate tests were added if applicable - [x] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
