imavishek opened a new pull request, #23463: URL: https://github.com/apache/kafka/pull/23463
## Summary Operators running tiered storage currently have no per-consumer visibility into remote (tiered) storage fetch activity. Topic-level `RemoteFetchBytesPerSec` and `RemoteFetchRequestsPerSec` already exist on `BrokerTopicMetrics` (KIP-963), but they cannot tell which `client-id` is driving remote fetches. This adds an opt-in, per-`client-id` view under a new `RemoteFetchMetrics` group, tagged with `client-id`: - `RemoteFetchBytesPerSec` — bytes read from remote storage per second - `RemoteFetchRequestsPerSec` — remote fetch request rate Example MBean: ``` kafka.server:type=RemoteFetchMetrics,name=RemoteFetchBytesPerSec,client-id=<id> ``` ## Changes - New broker config `remote.log.metrics.client.id.enabled` (default `false`). - `RemoteStorageFetchInfo` carries an optional `clientId`, populated from the fetch request's client metadata in `ReplicaManager`. A back-compat constructor keeps existing call sites unchanged. - `RemoteLogReader` records the request on submit and the bytes on a successful fetch, at the same points it already updates the topic-level metrics. Failed fetches do not record bytes. - Per-client sensors are created lazily and expire automatically once a client is idle, so no explicit cache eviction or MBean bookkeeping is needed. - Recording is a no-op when the config is disabled or the fetch carried no `client-id`. ## Cardinality In practice remote fetches are rare — most consumers read recent data served from local storage and only fall back to remote storage occasionally — so the set of `client-id`s that ever trigger a remote fetch is small, and idle sensors expire. The added metric cardinality is therefore low, and the feature is disabled by default. ## Testing - `RemoteFetchClientMetricsTest` — metric registry, lazy creation, client-id tagging, sensor re-creation after expiry. - `RemoteStorageFetchInfoTest` — `clientId` propagation contract (default-empty back-compat constructor). - `RemoteLogReaderTest` — verifies the record calls (request always, bytes only on success). - `RemoteLogManagerTest` — config gating (disabled by default; recorded when enabled) and empty-client-id no-op. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
