shendongsd opened a new issue, #11304: URL: https://github.com/apache/rocketmq/issues/11304
### Before Creating the Bug Report - [x] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions). - [x] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate. - [x] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ. ### Runtime platform environment When using metricsExporterType=PROM (PrometheusHttpServer), if there are Pop consumers connected to the broker, the /metrics endpoint crashes with DuplicateLabelsException, causing all Master broker metrics to be lost. Slave brokers are not affected because Pop consumers only connect to Master. ### RocketMQ version Environment RocketMQ Version: V5_3_3 (issue also present in develop branch as of 2026-10-01) metricsExporterType: PROM Consume mode: Pop consumption ### JDK Version _No response_ ### Describe the Bug Root Cause In BrokerMetricsManager.initLagAndDlqMetrics(), three gauges (rocketmq_consumer_lag_messages, rocketmq_consumer_lag_latency, rocketmq_consumer_ready_messages) use a single measurement object shared between two calculators: `consumerLagLatency = brokerMeter.gaugeBuilder(GAUGE_CONSUMER_LAG_LATENCY) .buildWithCallback(measurement -> { consumerLagCalculator.calculateLag(lagResult -> measurement.record(lagResult.getLagLatency(), buildLagAttributes(lagResult))); // ① liteConsumerLagCalculator.calculateLiteLagLatency(lagResult -> measurement.record(lagResult.getLagLatency(), buildLagAttributes(lagResult))); // ② });` For Pop consumers, ConsumerLagCalculator.calculate() records two entries per ProcessGroupInfo — one with isRetry=false and one with isRetry=true. When the same (consumer_group, topic, is_retry, is_system) label combination is produced across ① and ②, MetricSnapshot.validateLabels() throws DuplicateLabelsException. Once triggered, the entire /metrics scrape fails — all metrics from the Master broker disappear. ### Steps to Reproduce Environment RocketMQ Version: V5_3_3 (issue also present in develop branch as of 2026-10-01) metricsExporterType: PROM Consume mode: Pop consumption ### What Did You Expect to See? # HELP rocketmq_broker_permission default view for gauge. # TYPE rocketmq_broker_permission gauge rocketmq_broker_permission{cluster="...",node_id="...",node_type="broker",otel_scope_name="broker-meter"} 6.0 # HELP rocketmq_consumer_connections default view for gauge. # TYPE rocketmq_consumer_connections gauge rocketmq_consumer_connections{cluster="...",consume_mode="push",consumer_group="...",is_system="false",language="java",node_id="...",node_type="broker",otel_scope_name="broker-meter",protocol_type="remoting",version="v5_3_3"} 2.0 # HELP rocketmq_consumer_group_number default view for gauge. # TYPE rocketmq_consumer_group_number gauge rocketmq_consumer_group_number{cluster="...",node_id="...",node_type="broker",otel_scope_name="broker-meter"} 113.0 # HELP rocketmq_consumer_inflight_messages default view for gauge. # TYPE rocketmq_consumer_inflight_messages gauge rocketmq_consumer_inflight_messages{cluster="...",consumer_group="...",is_retry="false",is_system="true",node_id="...",node_type="broker",otel_scope_name="broker-meter",topic="..."} 0.0 # HELP rocketmq_consumer_lag_latency_milliseconds default view for gauge. # TYPE rocketmq_consumer_lag_latency_milliseconds gauge rocketmq_consumer_lag_latency_milliseconds{cluster="...",consumer_group="...",is_retry="false",is_system="true",node_id="...",node_type="broker",otel_scope_name="broker-meter",topic="..."} 0.0 # HELP rocketmq_consumer_lag_messages default view for gauge. # TYPE rocketmq_consumer_lag_messages gauge rocketmq_consumer_lag_messages{cluster="...",consumer_group="...",is_retry="false",is_system="true",node_id="...",node_type="broker",otel_scope_name="broker-meter",topic="..."} 0.0 # HELP rocketmq_consumer_queueing_latency_milliseconds default view for gauge. # TYPE rocketmq_consumer_queueing_latency_milliseconds gauge rocketmq_consumer_queueing_latency_milliseconds{cluster="...",consumer_group="...",is_retry="false",is_system="true",node_id="...",node_type="broker",otel_scope_name="broker-meter",topic="..."} 9.7563873E7 # HELP rocketmq_consumer_ready_messages default view for gauge. # TYPE rocketmq_consumer_ready_messages gauge rocketmq_consumer_ready_messages{cluster="...",consumer_group="...",is_retry="false",is_system="true",node_id="...",node_type="broker",otel_scope_name="broker-meter",topic="..."} 7741731.0 # HELP rocketmq_delay_message_latency_seconds Timer message set latency distribution # TYPE rocketmq_delay_message_latency_seconds histogram rocketmq_delay_message_latency_seconds_bucket{cluster="...",node_id="...",node_type="broker",otel_scope_name="broker-meter",storage_medium="disk ### What Did You See Instead? An Exception occurred while scraping metrics: io.prometheus.metrics.model.snapshots.DuplicateLabelsException: Duplicate labels for metric "rocketmq_consumer_lag_latency_milliseconds": {cluster="...", consumer_group="...", is_retry="true", is_system="false", node_id="...", node_type="broker", otel_scope_name="broker-meter", topic="..."} at io.prometheus.metrics.model.snapshots.MetricSnapshot.validateLabels(MetricSnapshot.java:42) at io.prometheus.metrics.model.snapshots.MetricSnapshot.<init>(MetricSnapshot.java:27) at io.prometheus.metrics.model.snapshots.GaugeSnapshot.<init>(GaugeSnapshot.java:18) at io.opentelemetry.exporter.prometheus.Otel2PrometheusConverter.convertLongGauge(...) at io.opentelemetry.exporter.prometheus.Otel2PrometheusConverter.convert(...) at io.opentelemetry.exporter.prometheus.PrometheusMetricReader.collect(...) at io.prometheus.metrics.model.registry.PrometheusRegistry.scrape(...) ### Additional Context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
