Richard-Yi opened a new issue #12617: URL: https://github.com/apache/pulsar/issues/12617
**Describe the bug** Prometheus collect the metric of one of brokers fail while the metric interface returns normal metrics and the other broker is successfully collected. I think there may be a problem with the content returned by the interface, causing prometheus fail to parse the metrics. Did anyone encounter the same problem？  Part of the metric content of the broker fail to collect . (Do I need upload the whole metric content of the broker cause it's too long？) ``` # TYPE pulsar_broker_load_manager_bundle_assigment summary pulsar_broker_load_manager_bundle_assigment{cluster="pulsar-cluster-zk-1",quantile="0.5"} NaN pulsar_broker_load_manager_bundle_assigment{cluster="pulsar-cluster-zk-1",quantile="0.99"} NaN pulsar_broker_load_manager_bundle_assigment{cluster="pulsar-cluster-zk-1",quantile="0.999"} NaN pulsar_broker_load_manager_bundle_assigment{cluster="pulsar-cluster-zk-1",quantile="1.0"} -Inf pulsar_broker_load_manager_bundle_assigment_count{cluster="pulsar-cluster-zk-1"} 15.0 pulsar_broker_load_manager_bundle_assigment_sum{cluster="pulsar-cluster-zk-1"} 139.0 # TYPE caffeine_cache_hit_total counter caffeine_cache_hit_total{cluster="pulsar-cluster-zk-1",cache="owned-bundles"} 12678.0 caffeine_cache_hit_total{cluster="pulsar-cluster-zk-1",cache="bookies-racks-exists"} 0.0 caffeine_cache_hit_total{cluster="pulsar-cluster-zk-1",cache="global-zk-children"} 0.0 caffeine_cache_hit_total{cluster="pulsar-cluster-zk-1",cache="bookies-racks-children"} 0.0 caffeine_cache_hit_total{cluster="pulsar-cluster-zk-1",cache="local-zk-exists"} 2762.0 caffeine_cache_hit_total{cluster="pulsar-cluster-zk-1",cache="local-zk-children"} 289.0 caffeine_cache_hit_total{cluster="pulsar-cluster-zk-1",cache="bundles"} 15162.0 caffeine_cache_hit_total{cluster="pulsar-cluster-zk-1",cache="global-zk-data"} 1.3430365E7 caffeine_cache_hit_total{cluster="pulsar-cluster-zk-1",cache="local-zk-data"} 5880.0 caffeine_cache_hit_total{cluster="pulsar-cluster-zk-1",cache="bookies-racks-data"} 0.0 caffeine_cache_hit_total{cluster="pulsar-cluster-zk-1",cache="global-zk-exists"} 0.0 # TYPE caffeine_cache_miss_total counter caffeine_cache_miss_total{cluster="pulsar-cluster-zk-1",cache="owned-bundles"} 5132.0 caffeine_cache_miss_total{cluster="pulsar-cluster-zk-1",cache="bookies-racks-exists"} 0.0 caffeine_cache_miss_total{cluster="pulsar-cluster-zk-1",cache="global-zk-children"} 0.0 caffeine_cache_miss_total{cluster="pulsar-cluster-zk-1",cache="bookies-racks-children"} 0.0 caffeine_cache_miss_total{cluster="pulsar-cluster-zk-1",cache="local-zk-exists"} 755.0 caffeine_cache_miss_total{cluster="pulsar-cluster-zk-1",cache="local-zk-children"} 317.0 caffeine_cache_miss_total{cluster="pulsar-cluster-zk-1",cache="bundles"} 9.0 caffeine_cache_miss_total{cluster="pulsar-cluster-zk-1",cache="global-zk-data"} 122.0 caffeine_cache_miss_total{cluster="pulsar-cluster-zk-1",cache="local-zk-data"} 4176.0 caffeine_cache_miss_total{cluster="pulsar-cluster-zk-1",cache="bookies-racks-data"} 0.0 caffeine_cache_miss_total{cluster="pulsar-cluster-zk-1",cache="global-zk-exists"} 0.0 # TYPE caffeine_cache_requests_total counter caffeine_cache_requests_total{cluster="pulsar-cluster-zk-1",cache="owned-bundles"} 17810.0 caffeine_cache_requests_total{cluster="pulsar-cluster-zk-1",cache="bookies-racks-exists"} 0.0 caffeine_cache_requests_total{cluster="pulsar-cluster-zk-1",cache="global-zk-children"} 0.0 caffeine_cache_requests_total{cluster="pulsar-cluster-zk-1",cache="bookies-racks-children"} 0.0 caffeine_cache_requests_total{cluster="pulsar-cluster-zk-1",cache="local-zk-exists"} 3517.0 caffeine_cache_requests_total{cluster="pulsar-cluster-zk-1",cache="local-zk-children"} 606.0 caffeine_cache_requests_total{cluster="pulsar-cluster-zk-1",cache="bundles"} 15171.0 caffeine_cache_requests_total{cluster="pulsar-cluster-zk-1",cache="global-zk-data"} 1.3430487E7 caffeine_cache_requests_total{cluster="pulsar-cluster-zk-1",cache="local-zk-data"} 10056.0 caffeine_cache_requests_total{cluster="pulsar-cluster-zk-1",cache="bookies-racks-data"} 0.0 caffeine_cache_requests_total{cluster="pulsar-cluster-zk-1",cache="global-zk-exists"} 0.0 # TYPE caffeine_cache_eviction_total counter caffeine_cache_eviction_total{cluster="pulsar-cluster-zk-1",cache="owned-bundles"} 0.0 caffeine_cache_eviction_total{cluster="pulsar-cluster-zk-1",cache="bookies-racks-exists"} 0.0 caffeine_cache_eviction_total{cluster="pulsar-cluster-zk-1",cache="global-zk-children"} 0.0 caffeine_cache_eviction_total{cluster="pulsar-cluster-zk-1",cache="bookies-racks-children"} 0.0 caffeine_cache_eviction_total{cluster="pulsar-cluster-zk-1",cache="local-zk-exists"} 673.0 caffeine_cache_eviction_total{cluster="pulsar-cluster-zk-1",cache="local-zk-children"} 47.0 caffeine_cache_eviction_total{cluster="pulsar-cluster-zk-1",cache="bundles"} 0.0 caffeine_cache_eviction_total{cluster="pulsar-cluster-zk-1",cache="global-zk-data"} 0.0 caffeine_cache_eviction_total{cluster="pulsar-cluster-zk-1",cache="local-zk-data"} 0.0 caffeine_cache_eviction_total{cluster="pulsar-cluster-zk-1",cache="bookies-racks-data"} 0.0 caffeine_cache_eviction_total{cluster="pulsar-cluster-zk-1",cache="global-zk-exists"} 0.0 # TYPE caffeine_cache_eviction_weight gauge caffeine_cache_eviction_weight{cluster="pulsar-cluster-zk-1",cache="owned-bundles"} 0.0 caffeine_cache_eviction_weight{cluster="pulsar-cluster-zk-1",cache="bookies-racks-exists"} 0.0 caffeine_cache_eviction_weight{cluster="pulsar-cluster-zk-1",cache="global-zk-children"} 0.0 caffeine_cache_eviction_weight{cluster="pulsar-cluster-zk-1",cache="bookies-racks-children"} 0.0 caffeine_cache_eviction_weight{cluster="pulsar-cluster-zk-1",cache="local-zk-exists"} 673.0 caffeine_cache_eviction_weight{cluster="pulsar-cluster-zk-1",cache="local-zk-children"} 47.0 caffeine_cache_eviction_weight{cluster="pulsar-cluster-zk-1",cache="bundles"} 0.0 caffeine_cache_eviction_weight{cluster="pulsar-cluster-zk-1",cache="global-zk-data"} 0.0 caffeine_cache_eviction_weight{cluster="pulsar-cluster-zk-1",cache="local-zk-data"} 0.0 caffeine_cache_eviction_weight{cluster="pulsar-cluster-zk-1",cache="bookies-racks-data"} 0.0 caffeine_cache_eviction_weight{cluster="pulsar-cluster-zk-1",cache="global-zk-exists"} 0.0 # TYPE caffeine_cache_load_failure_total counter caffeine_cache_load_failure_total{cluster="pulsar-cluster-zk-1",cache="owned-bundles"} 1.0 caffeine_cache_load_failure_total{cluster="pulsar-cluster-zk-1",cache="bookies-racks-exists"} 0.0 caffeine_cache_load_failure_total{cluster="pulsar-cluster-zk-1",cache="global-zk-children"} 0.0 caffeine_cache_load_failure_total{cluster="pulsar-cluster-zk-1",cache="bookies-racks-children"} 0.0 caffeine_cache_load_failure_total{cluster="pulsar-cluster-zk-1",cache="local-zk-exists"} 0.0 caffeine_cache_load_failure_total{cluster="pulsar-cluster-zk-1",cache="local-zk-children"} 0.0 caffeine_cache_load_failure_total{cluster="pulsar-cluster-zk-1",cache="bundles"} 0.0 caffeine_cache_load_failure_total{cluster="pulsar-cluster-zk-1",cache="global-zk-data"} 39.0 caffeine_cache_load_failure_total{cluster="pulsar-cluster-zk-1",cache="local-zk-data"} 2077.0 caffeine_cache_load_failure_total{cluster="pulsar-cluster-zk-1",cache="bookies-racks-data"} 0.0 caffeine_cache_load_failure_total{cluster="pulsar-cluster-zk-1",cache="global-zk-exists"} 0.0 # TYPE caffeine_cache_loads_total counter caffeine_cache_loads_total{cluster="pulsar-cluster-zk-1",cache="owned-bundles"} 14.0 caffeine_cache_loads_total{cluster="pulsar-cluster-zk-1",cache="bookies-racks-exists"} 0.0 caffeine_cache_loads_total{cluster="pulsar-cluster-zk-1",cache="global-zk-children"} 0.0 caffeine_cache_loads_total{cluster="pulsar-cluster-zk-1",cache="bookies-racks-children"} 0.0 caffeine_cache_loads_total{cluster="pulsar-cluster-zk-1",cache="local-zk-exists"} 755.0 caffeine_cache_loads_total{cluster="pulsar-cluster-zk-1",cache="local-zk-children"} 317.0 caffeine_cache_loads_total{cluster="pulsar-cluster-zk-1",cache="bundles"} 9.0 caffeine_cache_loads_total{cluster="pulsar-cluster-zk-1",cache="global-zk-data"} 237259.0 caffeine_cache_loads_total{cluster="pulsar-cluster-zk-1",cache="local-zk-data"} 2964.0 caffeine_cache_loads_total{cluster="pulsar-cluster-zk-1",cache="bookies-racks-data"} 0.0 caffeine_cache_loads_total{cluster="pulsar-cluster-zk-1",cache="global-zk-exists"} 0.0 # TYPE caffeine_cache_estimated_size gauge caffeine_cache_estimated_size{cluster="pulsar-cluster-zk-1",cache="owned-bundles"} 13.0 caffeine_cache_estimated_size{cluster="pulsar-cluster-zk-1",cache="bookies-racks-exists"} 0.0 caffeine_cache_estimated_size{cluster="pulsar-cluster-zk-1",cache="global-zk-children"} 0.0 caffeine_cache_estimated_size{cluster="pulsar-cluster-zk-1",cache="bookies-racks-children"} 0.0 caffeine_cache_estimated_size{cluster="pulsar-cluster-zk-1",cache="local-zk-exists"} 1.0 caffeine_cache_estimated_size{cluster="pulsar-cluster-zk-1",cache="local-zk-children"} 1.0 caffeine_cache_estimated_size{cluster="pulsar-cluster-zk-1",cache="bundles"} 9.0 caffeine_cache_estimated_size{cluster="pulsar-cluster-zk-1",cache="global-zk-data"} 8.0 caffeine_cache_estimated_size{cluster="pulsar-cluster-zk-1",cache="local-zk-data"} 11.0 caffeine_cache_estimated_size{cluster="pulsar-cluster-zk-1",cache="bookies-racks-data"} 0.0 caffeine_cache_estimated_size{cluster="pulsar-cluster-zk-1",cache="global-zk-exists"} 0.0 # TYPE caffeine_cache_load_duration_seconds summary caffeine_cache_load_duration_seconds_count{cluster="pulsar-cluster-zk-1",cache="owned-bundles"} 14.0 caffeine_cache_load_duration_seconds_sum{cluster="pulsar-cluster-zk-1",cache="owned-bundles"} 0.066185552 caffeine_cache_load_duration_seconds_count{cluster="pulsar-cluster-zk-1",cache="bookies-racks-exists"} 0.0 caffeine_cache_load_duration_seconds_sum{cluster="pulsar-cluster-zk-1",cache="bookies-racks-exists"} 0.0 caffeine_cache_load_duration_seconds_count{cluster="pulsar-cluster-zk-1",cache="global-zk-children"} 0.0 caffeine_cache_load_duration_seconds_sum{cluster="pulsar-cluster-zk-1",cache="global-zk-children"} 0.0 caffeine_cache_load_duration_seconds_count{cluster="pulsar-cluster-zk-1",cache="bookies-racks-children"} 0.0 caffeine_cache_load_duration_seconds_sum{cluster="pulsar-cluster-zk-1",cache="bookies-racks-children"} 0.0 caffeine_cache_load_duration_seconds_count{cluster="pulsar-cluster-zk-1",cache="local-zk-exists"} 755.0 caffeine_cache_load_duration_seconds_sum{cluster="pulsar-cluster-zk-1",cache="local-zk-exists"} 0.888731904 caffeine_cache_load_duration_seconds_count{cluster="pulsar-cluster-zk-1",cache="local-zk-children"} 317.0 caffeine_cache_load_duration_seconds_sum{cluster="pulsar-cluster-zk-1",cache="local-zk-children"} 0.422091904 caffeine_cache_load_duration_seconds_count{cluster="pulsar-cluster-zk-1",cache="bundles"} 9.0 caffeine_cache_load_duration_seconds_sum{cluster="pulsar-cluster-zk-1",cache="bundles"} 0.032046353 caffeine_cache_load_duration_seconds_count{cluster="pulsar-cluster-zk-1",cache="global-zk-data"} 237259.0 caffeine_cache_load_duration_seconds_sum{cluster="pulsar-cluster-zk-1",cache="global-zk-data"} 0.193615526 caffeine_cache_load_duration_seconds_count{cluster="pulsar-cluster-zk-1",cache="local-zk-data"} 2964.0 caffeine_cache_load_duration_seconds_sum{cluster="pulsar-cluster-zk-1",cache="local-zk-data"} 3.489291702 caffeine_cache_load_duration_seconds_count{cluster="pulsar-cluster-zk-1",cache="bookies-racks-data"} 0.0 caffeine_cache_load_duration_seconds_sum{cluster="pulsar-cluster-zk-1",cache="bookies-racks-data"} 0.0 caffeine_cache_load_duration_seconds_count{cluster="pulsar-cluster-zk-1",cache="global-zk-exists"} 0.0 caffeine_cache_load_duration_seconds_sum{cluster="pulsar-cluster-zk-1",cache="global-zk-exists"} 0.0 # TYPE log4j2_appender_total counter log4j2_appender_total{cluster="pulsar-cluster-zk-1",level="debug"} 0.0 log4j2_appender_total{cluster="pulsar-cluster-zk-1",level="warn"} 2185.0 log4j2_appender_total{cluster="pulsar-cluster-zk-1",level="trace"} 0.0 log4j2_appender_total{cluster="pulsar-cluster-zk-1",level="error"} 69.0 log4j2_appender_total{cluster="pulsar-cluster-zk-1",level="fatal"} 0.0 log4j2_appender_total{cluster="pulsar-cluster-zk-1",level="info"} 2478956.0 # TYPE jvm_classes_loaded gauge jvm_classes_loaded{cluster="pulsar-cluster-zk-1"} 12355.0 # TYPE jvm_classes_loaded_total counter jvm_classes_loaded_total{cluster="pulsar-cluster-zk-1"} 12355.0 # TYPE jvm_classes_unloaded_total counter jvm_classes_unloaded_total{cluster="pulsar-cluster-zk-1"} 0.0 # TYPE topic_load_times summary topic_load_times{cluster="pulsar-cluster-zk-1",quantile="0.5"} NaN topic_load_times{cluster="pulsar-cluster-zk-1",quantile="0.75"} NaN topic_load_times{cluster="pulsar-cluster-zk-1",quantile="0.95"} NaN topic_load_times{cluster="pulsar-cluster-zk-1",quantile="0.99"} NaN topic_load_times{cluster="pulsar-cluster-zk-1",quantile="0.999"} NaN topic_load_times{cluster="pulsar-cluster-zk-1",quantile="0.9999"} NaN topic_load_times_count{cluster="pulsar-cluster-zk-1"} 0.0 topic_load_times_sum{cluster="pulsar-cluster-zk-1"} 0.0 # TYPE pulsar_broker_lookup summary pulsar_broker_lookup{cluster="pulsar-cluster-zk-1",quantile="0.5"} NaN pulsar_broker_lookup{cluster="pulsar-cluster-zk-1",quantile="0.99"} NaN pulsar_broker_lookup{cluster="pulsar-cluster-zk-1",quantile="0.999"} NaN pulsar_broker_lookup{cluster="pulsar-cluster-zk-1",quantile="1.0"} -Inf pulsar_broker_lookup_count{cluster="pulsar-cluster-zk-1"} 2869.0 pulsar_broker_lookup_sum{cluster="pulsar-cluster-zk-1"} 191.0 # TYPE jvm_memory_bytes_used gauge jvm_memory_bytes_used{cluster="pulsar-cluster-zk-1",area="heap"} 4.34723952E8 jvm_memory_bytes_used{cluster="pulsar-cluster-zk-1",area="nonheap"} 1.41110336E8 # TYPE jvm_memory_bytes_committed gauge jvm_memory_bytes_committed{cluster="pulsar-cluster-zk-1",area="heap"} 2.147483648E9 jvm_memory_bytes_committed{cluster="pulsar-cluster-zk-1",area="nonheap"} 1.4958592E8 # TYPE jvm_memory_bytes_max gauge jvm_memory_bytes_max{cluster="pulsar-cluster-zk-1",area="heap"} 2.147483648E9 jvm_memory_bytes_max{cluster="pulsar-cluster-zk-1",area="nonheap"} -1.0 # TYPE jvm_memory_bytes_init gauge jvm_memory_bytes_init{cluster="pulsar-cluster-zk-1",area="heap"} 2.147483648E9 jvm_memory_bytes_init{cluster="pulsar-cluster-zk-1",area="nonheap"} 2555904.0 # TYPE jvm_memory_pool_bytes_used gauge jvm_memory_pool_bytes_used{cluster="pulsar-cluster-zk-1",pool="Code Cache"} 5.8152128E7 jvm_memory_pool_bytes_used{cluster="pulsar-cluster-zk-1",pool="Metaspace"} 7.440004E7 jvm_memory_pool_bytes_used{cluster="pulsar-cluster-zk-1",pool="Compressed Class Space"} 8558168.0 jvm_memory_pool_bytes_used{cluster="pulsar-cluster-zk-1",pool="G1 Eden Space"} 2.65289728E8 jvm_memory_pool_bytes_used{cluster="pulsar-cluster-zk-1",pool="G1 Survivor Space"} 2097152.0 jvm_memory_pool_bytes_used{cluster="pulsar-cluster-zk-1",pool="G1 Old Gen"} 1.67337072E8 # TYPE jvm_memory_pool_bytes_committed gauge jvm_memory_pool_bytes_committed{cluster="pulsar-cluster-zk-1",pool="Code Cache"} 5.8720256E7 jvm_memory_pool_bytes_committed{cluster="pulsar-cluster-zk-1",pool="Metaspace"} 8.1084416E7 jvm_memory_pool_bytes_committed{cluster="pulsar-cluster-zk-1",pool="Compressed Class Space"} 9781248.0 jvm_memory_pool_bytes_committed{cluster="pulsar-cluster-zk-1",pool="G1 Eden Space"} 1.126170624E9 jvm_memory_pool_bytes_committed{cluster="pulsar-cluster-zk-1",pool="G1 Survivor Space"} 2097152.0 jvm_memory_pool_bytes_committed{cluster="pulsar-cluster-zk-1",pool="G1 Old Gen"} 1.019215872E9 # TYPE jvm_memory_pool_bytes_max gauge jvm_memory_pool_bytes_max{cluster="pulsar-cluster-zk-1",pool="Code Cache"} 2.5165824E8 jvm_memory_pool_bytes_max{cluster="pulsar-cluster-zk-1",pool="Metaspace"} -1.0 jvm_memory_pool_bytes_max{cluster="pulsar-cluster-zk-1",pool="Compressed Class Space"} 1.073741824E9 jvm_memory_pool_bytes_max{cluster="pulsar-cluster-zk-1",pool="G1 Eden Space"} -1.0 jvm_memory_pool_bytes_max{cluster="pulsar-cluster-zk-1",pool="G1 Survivor Space"} -1.0 jvm_memory_pool_bytes_max{cluster="pulsar-cluster-zk-1",pool="G1 Old Gen"} 2.147483648E9 # TYPE jvm_memory_pool_bytes_init gauge jvm_memory_pool_bytes_init{cluster="pulsar-cluster-zk-1",pool="Code Cache"} 2555904.0 jvm_memory_pool_bytes_init{cluster="pulsar-cluster-zk-1",pool="Metaspace"} 0.0 jvm_memory_pool_bytes_init{cluster="pulsar-cluster-zk-1",pool="Compressed Class Space"} 0.0 jvm_memory_pool_bytes_init{cluster="pulsar-cluster-zk-1",pool="G1 Eden Space"} 1.128267776E9 jvm_memory_pool_bytes_init{cluster="pulsar-cluster-zk-1",pool="G1 Survivor Space"} 0.0 jvm_memory_pool_bytes_init{cluster="pulsar-cluster-zk-1",pool="G1 Old Gen"} 1.019215872E9 # TYPE jvm_gc_collection_seconds summary jvm_gc_collection_seconds_count{cluster="pulsar-cluster-zk-1",gc="G1 Young Generation"} 451.0 jvm_gc_collection_seconds_sum{cluster="pulsar-cluster-zk-1",gc="G1 Young Generation"} 51.331 jvm_gc_collection_seconds_count{cluster="pulsar-cluster-zk-1",gc="G1 Old Generation"} 0.0 jvm_gc_collection_seconds_sum{cluster="pulsar-cluster-zk-1",gc="G1 Old Generation"} 0.0 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
