ofek opened a new issue #13869: URL: https://github.com/apache/pulsar/issues/13869
**Describe the bug** As documented in the [official spec](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#histogram-1) and mentioned in [Pulsar's docs](https://pulsar.apache.org/docs/en/next/reference-metrics/), histogram buckets are suffixed by `_bucket` with an upper bound `le` label. Instead, the label and value is embedded in the metric name as a suffix: ``` # TYPE pulsar_storage_write_latency_le_0_5 gauge pulsar_storage_write_latency_le_0_5{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_write_latency_le_1 gauge pulsar_storage_write_latency_le_1{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_write_latency_le_5 gauge pulsar_storage_write_latency_le_5{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_write_latency_le_10 gauge pulsar_storage_write_latency_le_10{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_write_latency_le_20 gauge pulsar_storage_write_latency_le_20{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_write_latency_le_50 gauge pulsar_storage_write_latency_le_50{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_write_latency_le_100 gauge pulsar_storage_write_latency_le_100{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_write_latency_le_200 gauge pulsar_storage_write_latency_le_200{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_write_latency_le_1000 gauge pulsar_storage_write_latency_le_1000{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_write_latency_overflow gauge pulsar_storage_write_latency_overflow{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_write_latency_count gauge pulsar_storage_write_latency_count{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_write_latency_sum gauge pulsar_storage_write_latency_sum{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_ledger_write_latency_le_0_5 gauge pulsar_storage_ledger_write_latency_le_0_5{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_ledger_write_latency_le_1 gauge pulsar_storage_ledger_write_latency_le_1{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_ledger_write_latency_le_5 gauge pulsar_storage_ledger_write_latency_le_5{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_ledger_write_latency_le_10 gauge pulsar_storage_ledger_write_latency_le_10{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_ledger_write_latency_le_20 gauge pulsar_storage_ledger_write_latency_le_20{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_ledger_write_latency_le_50 gauge pulsar_storage_ledger_write_latency_le_50{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_ledger_write_latency_le_100 gauge pulsar_storage_ledger_write_latency_le_100{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_ledger_write_latency_le_200 gauge pulsar_storage_ledger_write_latency_le_200{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_ledger_write_latency_le_1000 gauge pulsar_storage_ledger_write_latency_le_1000{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_ledger_write_latency_overflow gauge pulsar_storage_ledger_write_latency_overflow{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_ledger_write_latency_count gauge pulsar_storage_ledger_write_latency_count{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_storage_ledger_write_latency_sum gauge pulsar_storage_ledger_write_latency_sum{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_entry_size_le_128 gauge pulsar_entry_size_le_128{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_entry_size_le_512 gauge pulsar_entry_size_le_512{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_entry_size_le_1_kb gauge pulsar_entry_size_le_1_kb{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_entry_size_le_2_kb gauge pulsar_entry_size_le_2_kb{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_entry_size_le_4_kb gauge pulsar_entry_size_le_4_kb{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_entry_size_le_16_kb gauge pulsar_entry_size_le_16_kb{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_entry_size_le_100_kb gauge pulsar_entry_size_le_100_kb{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_entry_size_le_1_mb gauge pulsar_entry_size_le_1_mb{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_entry_size_le_overflow gauge pulsar_entry_size_le_overflow{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_entry_size_count gauge pulsar_entry_size_count{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 # TYPE pulsar_entry_size_sum gauge pulsar_entry_size_sum{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/metadata",partition="-1"} 0.0 1642722619078 pulsar_storage_write_latency_le_0_5{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_le_1{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_le_5{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_le_10{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_le_20{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_le_50{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_le_100{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_le_200{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_le_1000{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_overflow{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_count{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_sum{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_le_0_5{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_le_1{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_le_5{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_le_10{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_le_20{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_le_50{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_le_100{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_le_200{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_le_1000{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_overflow{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_count{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_sum{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_entry_size_le_128{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_entry_size_le_512{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_entry_size_le_1_kb{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_entry_size_le_2_kb{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_entry_size_le_4_kb{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_entry_size_le_16_kb{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_entry_size_le_100_kb{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_entry_size_le_1_mb{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_entry_size_le_overflow{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_entry_size_count{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_entry_size_sum{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/coordinate",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_le_0_5{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_le_1{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_le_5{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_le_10{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_le_20{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_le_50{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_le_100{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_le_200{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_le_1000{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_overflow{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_count{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_write_latency_sum{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_le_0_5{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_le_1{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_le_5{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_le_10{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_le_20{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_le_50{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_le_100{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_le_200{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_le_1000{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_overflow{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_count{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_storage_ledger_write_latency_sum{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_entry_size_le_128{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_entry_size_le_512{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_entry_size_le_1_kb{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_entry_size_le_2_kb{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_entry_size_le_4_kb{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_entry_size_le_16_kb{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_entry_size_le_100_kb{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_entry_size_le_1_mb{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_entry_size_le_overflow{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_entry_size_count{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 pulsar_entry_size_sum{cluster="standalone",namespace="public/functions",topic="persistent://public/functions/assignments",partition="-1"} 0.0 1642722619079 ``` **To Reproduce** Steps to reproduce the behavior: ``` curl -L http://localhost:8080/metrics ``` ```yaml version: '3' services: pulsar: container_name: pulsar image: apachepulsar/pulsar:2.9.1 command: - bash - -c - > bin/apply-config-from-env-with-prefix.py BOOKKEEPER_ conf/bookkeeper.conf && bin/apply-config-from-env-with-prefix.py BROKER_ conf/broker.conf && bin/apply-config-from-env-with-prefix.py STANDALONE_ conf/standalone.conf && exec bin/pulsar standalone ports: - '6650:6650' - '8080:8080' environment: - BOOKKEEPER_enableStatistics=true - BOOKKEEPER_prometheusStatsHttpPort=8080 - BROKER_exposeTopicLevelMetricsInPrometheus=true - BROKER_exposeConsumerLevelMetricsInPrometheus=true - BROKER_exposeProducerLevelMetricsInPrometheus=true - BROKER_exposeManagedLedgerMetricsInPrometheus=true - BROKER_exposeManagedCursorMetricsInPrometheus=true - BROKER_exposePublisherStats=true - BROKER_exposePreciseBacklogInPrometheus=true - BROKER_splitTopicAndPartitionLabelInPrometheus=true - STANDALONE_exposeTopicLevelMetricsInPrometheus=true - STANDALONE_exposeConsumerLevelMetricsInPrometheus=true - STANDALONE_exposeProducerLevelMetricsInPrometheus=true - STANDALONE_exposeManagedLedgerMetricsInPrometheus=true - STANDALONE_exposeManagedCursorMetricsInPrometheus=true - STANDALONE_exposePublisherStats=true - STANDALONE_exposePreciseBacklogInPrometheus=true - STANDALONE_splitTopicAndPartitionLabelInPrometheus=true ``` **Desktop (please complete the following information):** - OS: Windows 10 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
