cshuo opened a new issue, #19116: URL: https://github.com/apache/hudi/issues/19116
## Background In Flink streaming writes, the `StreamWriteOperatorCoordinator` buffers `WriteMetadataEvent`s from write tasks per checkpoint. Once all events for a checkpoint are received (the snapshot/data write of that batch has succeeded on the task side), the corresponding instant becomes a *completed but not-yet-committed* instant — it is waiting for the coordinator to commit it on checkpoint completion. Today there is no easy way to observe how many such instants are pending commit at the coordinator. When commits are slow, stuck, or backing up (e.g. due to slow metadata/table services, lock contention, or checkpoint issues), operators have no direct signal and must infer the problem from logs or the timeline. ## Proposal Expose a coordinator-level metric `pendingCommitInstantCount` that reports the number of instants whose data writes have completed (snapshot succeeded) but which have not yet been committed by the coordinator. This makes it straightforward to monitor, at a glance, how many completed-but-uncommitted instants are currently accumulating, helping diagnose commit backpressure and stalls in the streaming write pipeline. ## Notes - Implemented as a Flink gauge registered by the coordinator on start. - Empty data-write buffers are excluded from the count so they are not reported as pending commits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
