Hi all,
I'd like to start a discussion on adding JMX metrics to the iceberg-kafka-connect sink connector. Implementation is up as PR #17025 and has been through review; the reviewers asked that the metric surface itself get wider visibility before it merges, since JMX attribute names become a public operational contract with no deprecation path once released. Proposal: https://github.com/apache/iceberg/issues/18007 Briefly: the connector is close to unobservable today. There are no metrics for coordinator commit latency, no way to distinguish a timeout-driven partial commit from a healthy full commit, and no signal for growth of the coordinator's in-memory commit buffers — which is the leading indicator of the degradation described in issue #16389. The PR adds timers for the commit path and control-message handling, counters for emitted control-topic events, and gauges for the coordinator's buffers, registered under an `iceberg.kafka.connect` JMX domain. I'd particularly like input on the naming and tagging decisions, which are the parts we cannot change later without breaking operators' dashboards: 1. The JMX domain (`iceberg.kafka.connect`) and the `type=worker-metrics` / `type=coordinator-metrics` values. 2. Whether data files and delete files should be counted separately (`data-files-written` + `delete-files-written`, matching the `added-data-files` / `added-delete-files` split in snapshot summaries) or together under a `content-files-written` name. 3. `start-commit-total` counts *successfully sent* START_COMMIT events rather than commit initiations — whether the name should say so. 4. Whether the `commit-mode=full|partial` tag should apply to all commit-related metrics rather than only the commit timer. 5. Whether durations should be published as cumulative total+count pairs (the proposal's choice — Kafka's sampled Avg/Max read NaN for most of a 300s commit interval) or some other representation. Details and the alternatives considered for each are in the proposal. Feedback welcome, on the mailing list or on the proposal issue. Thanks, Henry
