Shri-Vardhan opened a new issue, #38587: URL: https://github.com/apache/beam/issues/38587
### What would you like to happen? Currently in `KafkaCommitOffset.CommitOffsetDoFn` there is no visibility into how often offset commits are failing or how many retries are being attempted. If commits are silently failing in production you have no way to know without digging through logs. Adding Beam metrics here would make it easy to monitor commit health through existing monitoring dashboards. Proposed approach Add a counter for total commit failures Add a counter for retry attempts Add a counter for commits that exhausted all retries Use Beam's built in `Metrics.counter()` so it works with any runner **Why this is useful** If you're running a high throughput pipeline and offset commits are struggling, right now you'd have no idea until you see duplicate processing. This gives you an early warning signal. ### Issue Priority Priority: 3 (nice-to-have improvement) ### Issue Components - [ ] Component: Python SDK - [x] Component: Java SDK - [ ] Component: Go SDK - [ ] Component: Typescript SDK - [x] Component: IO connector - [ ] Component: Beam YAML - [ ] Component: Beam examples - [ ] Component: Beam playground - [ ] Component: Beam katas - [ ] Component: Website - [ ] Component: Infrastructure - [ ] Component: Spark Runner - [ ] Component: Flink Runner - [ ] Component: Prism Runner - [ ] Component: Twister2 Runner - [ ] Component: Hazelcast Jet Runner - [ ] Component: Google Cloud Dataflow Runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
