[
https://issues.apache.org/jira/browse/KAFKA-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steven Schlansker updated KAFKA-21078:
--------------------------------------
Description:
{{Sensor.add}} registers every metric with the registry and thus with every
{{{}MetricsReporter{}}}, regardless of whether the sensor's {{RecordingLevel}}
is enabled by {{{}metrics.recording.level{}}}. Only {{Sensor.record}} checks
{{{}shouldRecord{}}}. As a result, an application at INFO still exposes many
DEBUG-level metrics with a value of 0 (a {{Rate}} with no samples measures 0,
not NaN).
{{KafkaMetricsCollector.collectMetric}} has two filters: the subscription
prefix ({{{}MetricsEmitter.shouldEmitMetric{}}}) and "value is a Number". It
has no notion of recording level and does not drop zero values. So the KIP-714
push carries every disabled metric with its full label set on every interval,
and in delta mode it emits a fresh zero delta for each disabled sum.
For Kafka Streams (KIP-1076) the disabled DEBUG metrics are most of the
payload: per-store per-operation rates ({{{}stream-state-metrics{}}}), per-node
{{{}process-rate{}}}/{{{}process-total{}}}, and task-level {{process-\*}}. In
one production topology at INFO, {{org.apache.kafka.stream.state.\*.rate}}
alone was several thousand series per stream thread, all zero. This inflates
the push toward {{telemetry.max.bytes}} (see the linked bug) and the
cardinality of the receiving backend.
{{KafkaMetric}} does not know its sensor's level
({{{}MetricConfig.recordLevel(){}}} is the configured threshold, not the
sensor's own level), so the collector cannot filter today. Options:
* {{Sensor.add}} skips {{registry.registerMetric}} when {{shouldRecord()}} is
false. This could be fragile if a user reconfigures the metric level, but also
cleans up JMX.
* {{Metrics}} exposes the sensor level for a metric, and
{{KafkaMetricsCollector}} skips metrics whose sensor is disabled.
The first option changes what JMX shows and needs a KIP note. The second is
local to the telemetry reporter.
was:
{{Sensor.add}} registers every metric with the registry and thus with every
{{{}MetricsReporter{}}}, regardless of whether the sensor's {{RecordingLevel}}
is enabled by {{{}metrics.recording.level{}}}. Only {{Sensor.record}} checks
{{{}shouldRecord{}}}. As a result, an application at INFO still exposes many
DEBUG-level metrics with a value of 0 (a {{Rate}} with no samples measures 0,
not NaN).
{{KafkaMetricsCollector.collectMetric}} has two filters: the subscription
prefix ({{{}MetricsEmitter.shouldEmitMetric{}}}) and "value is a Number". It
has no notion of recording level and does not drop zero values. So the KIP-714
push carries every disabled metric with its full label set on every interval,
and in delta mode it emits a fresh zero delta for each disabled sum.
For Kafka Streams (KIP-1076) the disabled DEBUG metrics are most of the
payload: per-store per-operation rates ({{{}stream-state-metrics{}}}), per-node
{{{}process-rate{}}}/{{{}process-total{}}}, and task-level {{{}process-*{}}}.
In one production topology at INFO, {{org.apache.kafka.stream.state.*.rate}}
alone was several thousand series per stream thread, all zero. This inflates
the push toward {{telemetry.max.bytes}} (see the linked bug) and the
cardinality of the receiving backend.
{{KafkaMetric}} does not know its sensor's level
({{{}MetricConfig.recordLevel(){}}} is the configured threshold, not the
sensor's own level), so the collector cannot filter today. Options:
* {{Sensor.add}} skips {{registry.registerMetric}} when {{shouldRecord()}} is
false. This could be fragile if a user reconfigures the metric level, but also
cleans up JMX.
* {{Metrics}} exposes the sensor level for a metric, and
{{KafkaMetricsCollector}} skips metrics whose sensor is disabled.
The first option changes what JMX shows and needs a KIP note. The second is
local to the telemetry reporter.
> Telemetry reporter pushes metrics whose sensors are disabled by
> metrics.recording.level
> ---------------------------------------------------------------------------------------
>
> Key: KAFKA-21078
> URL: https://issues.apache.org/jira/browse/KAFKA-21078
> Project: Kafka
> Issue Type: Improvement
> Components: clients, streams
> Affects Versions: 4.3.1
> Reporter: Steven Schlansker
> Priority: Minor
>
> {{Sensor.add}} registers every metric with the registry and thus with every
> {{{}MetricsReporter{}}}, regardless of whether the sensor's
> {{RecordingLevel}} is enabled by {{{}metrics.recording.level{}}}. Only
> {{Sensor.record}} checks {{{}shouldRecord{}}}. As a result, an application at
> INFO still exposes many DEBUG-level metrics with a value of 0 (a {{Rate}}
> with no samples measures 0, not NaN).
> {{KafkaMetricsCollector.collectMetric}} has two filters: the subscription
> prefix ({{{}MetricsEmitter.shouldEmitMetric{}}}) and "value is a Number". It
> has no notion of recording level and does not drop zero values. So the
> KIP-714 push carries every disabled metric with its full label set on every
> interval, and in delta mode it emits a fresh zero delta for each disabled sum.
> For Kafka Streams (KIP-1076) the disabled DEBUG metrics are most of the
> payload: per-store per-operation rates ({{{}stream-state-metrics{}}}),
> per-node {{{}process-rate{}}}/{{{}process-total{}}}, and task-level
> {{process-\*}}. In one production topology at INFO,
> {{org.apache.kafka.stream.state.\*.rate}} alone was several thousand series
> per stream thread, all zero. This inflates the push toward
> {{telemetry.max.bytes}} (see the linked bug) and the cardinality of the
> receiving backend.
> {{KafkaMetric}} does not know its sensor's level
> ({{{}MetricConfig.recordLevel(){}}} is the configured threshold, not the
> sensor's own level), so the collector cannot filter today. Options:
> * {{Sensor.add}} skips {{registry.registerMetric}} when {{shouldRecord()}}
> is false. This could be fragile if a user reconfigures the metric level, but
> also cleans up JMX.
> * {{Metrics}} exposes the sensor level for a metric, and
> {{KafkaMetricsCollector}} skips metrics whose sensor is disabled.
> The first option changes what JMX shows and needs a KIP note. The second is
> local to the telemetry reporter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
