divick opened a new issue, #13202:
URL: https://github.com/apache/druid/issues/13202
Please provide a detailed title (e.g. "Broker crashes when using TopN query
with Bound filter" instead of just "Broker crashes").
### Affected Version
0.23.0
### Description
The cluster is deployed with druid-operator in k8s cluster with 3 nodes and
broker, coordinator, router and historicals with each having 1, 1, 1, 2
replicas each. Note that even though I have deployed druid in k8s but the steps
and configuration should be same even for druid running on non k8s cluster.
- Cluster size -- 3 nodes
- Configurations in use
- Steps to reproduce the problem
Provision druid cluster using druid-operator in k8s with following
configurations:
```
druid.extensions.loadList=["prometheus-emitter",
"mysql-metadata-storage", ...]
druid.emitter=prometheus
druid.emitter.prometheus.strategy=exporter
druid.emitter.prometheus.port=9100
druid.emitter.prometheus.namespace=druid
druid.emitter.prometheus.addHostAsLabel=true
druid.emitter.prometheus.addServiceAsLabel=true
druid.emitter.logging.logLevel=debug
```
Metrics Dimensions json should be:
```
metricDimensions.json: |-
{
"query/time" : { "dimensions" : ["dataSource", "type"], "type" :
"timer"},
"query/bytes" : { "dimensions" : ["dataSource", "type"], "type" :
"count"},
"query/priority" : { "dimensions" : ["dataSource", "lane", "type"],
"type" : "gauge"},
"sqlQuery/bytes" : { "dimensions" : [], "type" : "count"},
"query/node/time" : { "dimensions" : ["server"], "type" : "timer"},
"query/node/ttfb" : { "dimensions" : ["server"], "type" : "timer"},
"query/node/bytes" : { "dimensions" : ["server"], "type" : "count"},
"query/node/backpressure": { "dimensions" : ["server"], "type" :
"timer"},
"query/intervalChunk/time" : { "dimensions" : [], "type" : "timer"},
"query/segment/time" : { "dimensions" : [], "type" : "timer"},
"query/wait/time" : { "dimensions" : [], "type" : "timer"},
"segment/scan/pending" : { "dimensions" : [], "type" : "gauge"},
"query/segmentAndCache/time" : { "dimensions" : [], "type" : "timer"
},
"query/cpu/time" : { "dimensions" : ["dataSource", "type"], "type" :
"timer" },
"query/count" : { "dimensions" : [], "type" : "count" },
"query/success/count" : { "dimensions" : [], "type" : "count" },
"query/failed/count" : { "dimensions" : [], "type" : "count" },
"query/interrupted/count" : { "dimensions" : [], "type" : "count" },
"query/timeout/count" : { "dimensions" : [], "type" : "count" },
"query/cache/delta/numEntries" : { "dimensions" : [], "type" : "count"
},
"query/cache/delta/sizeBytes" : { "dimensions" : [], "type" : "count"
},
"query/cache/delta/hits" : { "dimensions" : [], "type" : "count" },
"query/cache/delta/misses" : { "dimensions" : [], "type" : "count" },
"query/cache/delta/evictions" : { "dimensions" : [], "type" : "count"
},
"query/cache/delta/hitRate" : { "dimensions" : [], "type" : "count",
"convertRange" : true },
"query/cache/delta/averageBytes" : { "dimensions" : [], "type" :
"count" },
"query/cache/delta/timeouts" : { "dimensions" : [], "type" : "count"
},
"query/cache/delta/errors" : { "dimensions" : [], "type" : "count" },
"query/cache/total/numEntries" : { "dimensions" : [], "type" : "gauge"
},
"query/cache/total/sizeBytes" : { "dimensions" : [], "type" : "gauge"
},
"query/cache/total/hits" : { "dimensions" : [], "type" : "gauge" },
"query/cache/total/misses" : { "dimensions" : [], "type" : "gauge" },
"query/cache/total/evictions" : { "dimensions" : [], "type" : "gauge"
},
"query/cache/total/hitRate" : { "dimensions" : [], "type" : "gauge",
"convertRange" : true },
"query/cache/total/averageBytes" : { "dimensions" : [], "type" :
"gauge" },
"query/cache/total/timeouts" : { "dimensions" : [], "type" : "gauge"
},
"query/cache/total/errors" : { "dimensions" : [], "type" : "gauge" },
"sqlQuery/time" : { "dimensions" : ["dataSource"], "type" : "timer"},
"ingest/events/thrownAway" : { "dimensions" : ["dataSource"], "type" :
"count" },
"ingest/events/unparseable" : { "dimensions" : ["dataSource"], "type"
: "count" },
"ingest/events/duplicate" : { "dimensions" : ["dataSource"], "type" :
"count" },
"ingest/events/processed" : { "dimensions" : ["dataSource",
"taskType", "taskId"], "type" : "count" },
"ingest/events/messageGap" : { "dimensions" : ["dataSource"], "type" :
"gauge" },
"ingest/rows/output" : { "dimensions" : ["dataSource"], "type" :
"count" },
"ingest/persists/count" : { "dimensions" : ["dataSource"], "type" :
"count" },
"ingest/persists/time" : { "dimensions" : ["dataSource"], "type" :
"timer" },
"ingest/persists/cpu" : { "dimensions" : ["dataSource"], "type" :
"timer" },
"ingest/persists/backPressure" : { "dimensions" : ["dataSource"],
"type" : "gauge" },
"ingest/persists/failed" : { "dimensions" : ["dataSource"], "type" :
"count" },
"ingest/handoff/failed" : { "dimensions" : ["dataSource"], "type" :
"count" },
"ingest/merge/time" : { "dimensions" : ["dataSource"], "type" :
"timer" },
"ingest/merge/cpu" : { "dimensions" : ["dataSource"], "type" : "timer"
},
"ingest/kafka/lag" : { "dimensions" : ["dataSource"], "type" : "gauge"
},
"ingest/kafka/maxLag" : { "dimensions" : ["dataSource"], "type" :
"gauge" },
"ingest/kafka/avgLag" : { "dimensions" : ["dataSource"], "type" :
"gauge" },
"task/success/count" : { "dimensions" : ["dataSource"], "type" :
"count" },
"task/failed/count" : { "dimensions" : ["dataSource"], "type" :
"count" },
"task/running/count" : { "dimensions" : ["dataSource"], "type" :
"gauge" },
"task/pending/count" : { "dimensions" : ["dataSource"], "type" :
"gauge" },
"task/waiting/count" : { "dimensions" : ["dataSource"], "type" :
"gauge" },
"taskSlot/total/count" : { "dimensions" : [], "type" : "gauge" },
"taskSlot/idle/count" : { "dimensions" : [], "type" : "gauge" },
"taskSlot/busy/count" : { "dimensions" : [], "type" : "gauge" },
"taskSlot/lazy/count" : { "dimensions" : [], "type" : "gauge" },
"taskSlot/blacklisted/count" : { "dimensions" : [], "type" : "gauge"
},
"task/run/time" : { "dimensions" : ["dataSource", "taskType"], "type"
: "timer" },
"segment/added/bytes" : { "dimensions" : ["dataSource", "taskType"],
"type" : "count" },
"segment/moved/bytes" : { "dimensions" : ["dataSource", "taskType"],
"type" : "count" },
"segment/nuked/bytes" : { "dimensions" : ["dataSource", "taskType"],
"type" : "count" },
"segment/assigned/count" : { "dimensions" : ["tier"], "type" : "count" },
"segment/moved/count" : { "dimensions" : ["tier"], "type" : "count" },
"segment/dropped/count" : { "dimensions" : ["tier"], "type" : "count"
},
"segment/deleted/count" : { "dimensions" : ["tier"], "type" : "count"
},
"segment/unneeded/count" : { "dimensions" : ["tier"], "type" : "count"
},
"segment/unavailable/count" : { "dimensions" : ["dataSource"], "type"
: "gauge" },
"segment/underReplicated/count" : { "dimensions" : ["dataSource",
"tier"], "type" : "gauge" },
"segment/cost/raw" : { "dimensions" : ["tier"], "type" : "count" },
"segment/cost/normalization" : { "dimensions" : ["tier"], "type" :
"count" },
"segment/cost/normalized" : { "dimensions" : ["tier"], "type" :
"count" },
"segment/loadQueue/size" : { "dimensions" : ["server"], "type" :
"gauge" },
"segment/loadQueue/failed" : { "dimensions" : ["server"], "type" :
"gauge" },
"segment/loadQueue/count" : { "dimensions" : ["server"], "type" :
"gauge" },
"segment/dropQueue/count" : { "dimensions" : ["server"], "type" :
"gauge" },
"segment/size" : { "dimensions" : ["dataSource"], "type" : "gauge" },
"segment/overShadowed/count" : { "dimensions" : [], "type" : "gauge"
},
"segment/max" : { "dimensions" : [], "type" : "gauge"},
"segment/used" : { "dimensions" : ["dataSource", "tier", "priority"],
"type" : "gauge" },
"segment/usedPercent" : { "dimensions" : ["dataSource", "tier",
"priority"], "type" : "gauge", "convertRange" : true },
"segment/pendingDelete" : { "dimensions" : [], "type" : "gauge"},
"segment/count" : { "dimensions" : ["dataSource"], "type" : "gauge" },
"coordinator-segment/count" : { "dimensions" : ["dataSource"], "type"
: "gauge" },
"historical-segment/count" : { "dimensions" : ["dataSource", "tier",
"priority"], "type" : "gauge" },
"jvm/pool/committed" : { "dimensions" : ["poolKind", "poolName"],
"type" : "gauge" },
"jvm/pool/init" : { "dimensions" : ["poolKind", "poolName"], "type" :
"gauge" },
"jvm/pool/max" : { "dimensions" : ["poolKind", "poolName"], "type" :
"gauge" },
"jvm/pool/used" : { "dimensions" : ["poolKind", "poolName"], "type" :
"gauge" },
"jvm/bufferpool/count" : { "dimensions" : ["bufferpoolName"], "type" :
"gauge" },
"jvm/bufferpool/used" : { "dimensions" : ["bufferpoolName"], "type" :
"gauge" },
"jvm/bufferpool/capacity" : { "dimensions" : ["bufferpoolName"],
"type" : "gauge" },
"jvm/mem/init" : { "dimensions" : ["memKind"], "type" : "gauge" },
"jvm/mem/max" : { "dimensions" : ["memKind"], "type" : "gauge" },
"jvm/mem/used" : { "dimensions" : ["memKind"], "type" : "gauge" },
"jvm/mem/committed" : { "dimensions" : ["memKind"], "type" : "gauge"
},
"jvm/gc/count" : { "dimensions" : ["gcName", "gcGen"], "type" :
"count" },
"jvm/gc/cpu" : { "dimensions" : ["gcName", "gcGen"], "type" : "count"
},
"jvm/gc/mem/max" : { "dimensions" : [], "type" : "gauge" },
"jvm/gc/mem/capacity" : { "dimensions" : [], "type" : "gauge" },
"jvm/gc/mem/used" : { "dimensions" : [], "type" : "gauge" },
"jvm/cpu/user" : { "dimensions" : [], "type" : "gauge" },
"jvm/cpu/total" : { "dimensions" : [], "type" : "gauge" },
"jvm/threads/started" : { "dimensions" : [], "type" : "count" },
"jvm/threads/finished" : { "dimensions" : [], "type" : "count" },
"jvm/threads/live" : { "dimensions" : [], "type" : "count" },
"ingest/events/buffered" : { "dimensions" : ["serviceName",
"bufferCapacity"], "type" : "gauge"},
"ingest/shuffle/bytes" : { "dimensions" : ["supervisorTaskId"], "type"
: "gauge"},
"sys/swap/free" : { "dimensions" : [], "type" : "gauge"},
"sys/swap/max" : { "dimensions" : [], "type" : "gauge"},
"sys/swap/pageIn" : { "dimensions" : [], "type" : "gauge"},
"sys/swap/pageOut" : { "dimensions" : [], "type" : "gauge"},
"sys/disk/write/count" : { "dimensions" : ["fsDevName"], "type" :
"count"},
"sys/disk/read/count" : { "dimensions" : ["fsDevName"], "type" :
"count"},
"sys/disk/write/size" : { "dimensions" : ["fsDevName"], "type" :
"count"},
"sys/disk/read/size" : { "dimensions" : ["fsDevName"], "type" :
"count"},
"sys/net/write/size" : { "dimensions" : [], "type" : "count"},
"sys/net/read/size" : { "dimensions" : [], "type" : "count"},
"sys/fs/used" : { "dimensions" : ["fsDevName", "fsDirName",
"fsTypeName", "fsSysTypeName", "fsOptions"], "type" : "gauge"},
"sys/fs/max" : { "dimensions" : ["fsDevName", "fsDirName",
"fsTypeName", "fsSysTypeName", "fsOptions"], "type" : "gauge"},
"sys/mem/used" : { "dimensions" : [], "type" : "gauge"},
"sys/mem/max" : { "dimensions" : [], "type" : "gauge"},
"sys/storage/used" : { "dimensions" : ["fsDirName"], "type" :
"gauge"},
"sys/cpu" : { "dimensions" : ["cpuName", "cpuTime"], "type" :
"gauge"},
"jvm/cpu/percent" : { "dimensions" : ["dataSource"], "type" : "gauge"
},
"jetty/numOpenConnections" : { "dimensions" : [], "type" : "gauge" },
"query/cache/caffeine/total/requests" : { "dimensions" : [], "type" :
"gauge" },
"query/cache/caffeine/total/loadTime" : { "dimensions" : [], "type" :
"gauge" },
"query/cache/caffeine/total/evictionBytes" : { "dimensions" : [],
"type" : "gauge" },
"query/cache/memcached/total" : { "dimensions" : ["[MEM] Reconnecting
Nodes (ReconnectQueue)",
"[MEM] Request Rate: All",
"[MEM] Average Bytes written to OS per write",
"[MEM] Average Bytes read from OS per read",
"[MEM] Response Rate: All (Failure + Success + Retry)",
"[MEM] Response Rate: Retry",
"[MEM] Response Rate: Failure",
"[MEM] Response Rate: Success"],
"type" : "gauge" },
"query/cache/caffeine/delta/requests" : { "dimensions" : [], "type" :
"count" },
"query/cache/caffeine/delta/loadTime" : { "dimensions" : [], "type" :
"count" },
"query/cache/caffeine/delta/evictionBytes" : { "dimensions" : [],
"type" : "count" },
"query/cache/memcached/delta" : { "dimensions" : ["[MEM] Reconnecting
Nodes (ReconnectQueue)",
"[MEM] Request Rate: All",
"[MEM] Average Bytes written to OS per write",
"[MEM] Average Bytes read from OS per read",
"[MEM] Response Rate: All (Failure + Success + Retry)",
"[MEM] Response Rate: Retry",
"[MEM] Response Rate: Failure",
"[MEM] Response Rate: Success"],
"type" : "count" }
}
```
The cluster spec in druid-operator is:
```
nodes:
brokers:
kind: Deployment
nodeType: "broker"
druid.port: 8088
nodeConfigMountPath: "/opt/druid/conf/druid/cluster/query/broker"
replicas: 1
runtime.properties: |
druid.service=druid/broker
# HTTP server threads
druid.broker.http.numConnections=5
druid.server.http.numThreads=10
# Processing threads and buffers
druid.processing.buffer.sizeBytes=1
druid.processing.numMergeBuffers=1
druid.processing.numThreads=1
druid.sql.enable=true
druid.monitoring.monitors=["org.apache.druid.client.cache.CacheMonitor","org.apache.druid.java.util.metrics.JvmMonitor","org.apache.druid.java.util.metrics.JvmCpuMonitor","org.apache.druid.java.util.metrics.JvmThreadsMonitor","org.apache.druid.server.metrics.HistoricalMetricsMonitor",
"org.apache.druid.server.metrics.QueryCountStatsMonitor"]
coordinators:
# Optionally specify for running coordinator as Deployment
kind: Deployment
nodeType: "coordinator"
druid.port: 8088
nodeConfigMountPath:
"/opt/druid/conf/druid/cluster/master/coordinator-overlord"
replicas: 1
runtime.properties: |
druid.service=druid/coordinator
# HTTP server threads
druid.coordinator.startDelay=PT30S
druid.coordinator.period=PT30S
# Configure this coordinator to also run as Overlord
druid.coordinator.asOverlord.enabled=true
druid.coordinator.asOverlord.overlordService=druid/overlord
druid.indexer.queue.startDelay=PT30S
druid.indexer.runner.type=local
druid.monitoring.monitors=["org.apache.druid.client.cache.CacheMonitor","org.apache.druid.java.util.metrics.JvmMonitor","org.apache.druid.java.util.metrics.JvmCpuMonitor","org.apache.druid.java.util.metrics.JvmThreadsMonitor"]
extra.jvm.options: |-
-Xmx512M
-Xms512M
historicals:
nodeType: "historical"
druid.port: 8088
nodeConfigMountPath: "/opt/druid/conf/druid/cluster/data/historical"
replicas: 2
runtime.properties: |
druid.service=druid/historical
druid.server.http.numThreads=5
druid.processing.buffer.sizeBytes=536870912
druid.processing.numMergeBuffers=1
druid.processing.numThreads=1
# Segment storage
druid.segmentCache.locations=[{\"path\":\"/druid/data/segments\",\"maxSize\":10737418240}]
druid.server.maxSize=10737418240
druid.monitoring.monitors=["org.apache.druid.client.cache.CacheMonitor","org.apache.druid.java.util.metrics.JvmMonitor","org.apache.druid.java.util.metrics.JvmCpuMonitor","org.apache.druid.java.util.metrics.JvmThreadsMonitor","org.apache.druid.server.metrics.HistoricalMetricsMonitor",
"org.apache.druid.server.metrics.QueryCountStatsMonitor",
"org.apache.druid.indexing.worker.shuffle.ShuffleMonitor"]
extra.jvm.options: |-
-Xmx512M
-Xms512M
routers:
kind: Deployment
nodeType: "router"
druid.port: 8088
nodeConfigMountPath: "/opt/druid/conf/druid/cluster/query/router"
replicas: 1
runtime.properties: |
druid.service=druid/router
# HTTP proxy
druid.router.http.numConnections=10
druid.router.http.readTimeout=PT5M
druid.router.http.numMaxThreads=10
druid.server.http.numThreads=10
# Service discovery
druid.router.defaultBrokerServiceName=druid/broker
druid.router.coordinatorServiceName=druid/coordinator
# Management proxy to coordinator / overlord: required for unified
web console.
druid.router.managementProxy.enabled=true
druid.monitoring.monitors=["org.apache.druid.client.cache.CacheMonitor","org.apache.druid.java.util.metrics.JvmMonitor","org.apache.druid.java.util.metrics.JvmCpuMonitor","org.apache.druid.java.util.metrics.JvmThreadsMonitor","org.apache.druid.server.metrics.QueryCountStatsMonitor"]
extra.jvm.options: |-
-Xmx512M
-Xms512M
```
- The error message or stack traces encountered. Providing more context,
such as nearby log messages or even entire logs, can be helpful.
Output from broker (have removed the log prefixes to keep things simple:
```
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[query/timeout/count]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/threads/started]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/threads/finished]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/threads/live]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/threads/liveDaemon]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/threads/livePeak]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/gc/mem/max]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/gc/mem/capacity]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/gc/mem/used]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/gc/mem/init]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/gc/mem/max]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/gc/mem/capacity]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/gc/mem/used]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/gc/mem/init]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/gc/mem/max]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/gc/mem/capacity]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/gc/mem/used]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/gc/mem/init]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/gc/mem/max]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/gc/mem/capacity]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/gc/mem/used]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/gc/mem/init]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/heapAlloc/bytes]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[avatica/server/AvaticaProtobufHandler/Handler/RequestTimings]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[avatica/server/AvaticaJsonHandler/Handler/RequestTimings]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[avatica/remote/ProtobufHandler/Handler/Serialization]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[avatica/remote/JsonHandler/Handler/Serialization]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/cpu/user]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/cpu/total]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/cpu/sys]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jvm/cpu/percent]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[query/cache/delta/put/ok]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[query/cache/delta/put/error]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[query/cache/delta/put/oversized]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[query/cache/total/put/ok]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[query/cache/total/put/error]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[query/cache/total/put/oversized]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[query/cache/caffeine/delta/requests]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[query/cache/caffeine/total/requests]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[query/cache/caffeine/delta/loadTime]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[query/cache/caffeine/total/loadTime]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[query/cache/caffeine/delta/evictionBytes]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[query/cache/caffeine/total/evictionBytes]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jetty/numOpenConnections]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jetty/threadPool/total]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jetty/threadPool/idle]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jetty/threadPool/isLowOnThreads]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jetty/threadPool/min]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jetty/threadPool/max]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jetty/threadPool/queueSize]
[MonitorScheduler-0] org.a.d.e.p.PrometheusEmitter - Unmapped metric
[jetty/threadPool/busy]
```
Output from coordinator:
```
Unmapped metric [query/cache/delta/put/ok]
Unmapped metric [query/cache/delta/put/error]
Unmapped metric [query/cache/delta/put/oversized]
Unmapped metric [query/cache/total/put/ok]
Unmapped metric [query/cache/total/put/error]
Unmapped metric [query/cache/total/put/oversized]
Unmapped metric [query/cache/caffeine/delta/requests]
Unmapped metric [query/cache/caffeine/total/requests]
Unmapped metric [query/cache/caffeine/delta/loadTime]
Unmapped metric [query/cache/caffeine/total/loadTime]
Unmapped metric [query/cache/caffeine/delta/evictionBytes]
Unmapped metric [query/cache/caffeine/total/evictionBytes]
Unmapped metric [jvm/cpu/user]
Unmapped metric [jvm/cpu/total]
Unmapped metric [jvm/cpu/sys]
Unmapped metric [jvm/cpu/percent]
Unmapped metric [jvm/threads/started]
Unmapped metric [jvm/threads/finished]
Unmapped metric [jvm/threads/live]
Unmapped metric [jvm/threads/liveDaemon]
Unmapped metric [jvm/threads/livePeak]
Unmapped metric [jetty/numOpenConnections]
Unmapped metric [jetty/threadPool/total]
Unmapped metric [jetty/threadPool/idle]
Unmapped metric [jetty/threadPool/isLowOnThreads]
Unmapped metric [jetty/threadPool/min]
Unmapped metric [jetty/threadPool/max]
Unmapped metric [jetty/threadPool/queueSize]
Unmapped metric [jetty/threadPool/busy]
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/heapAlloc/bytes]
Unmapped metric [tier/required/capacity]
Unmapped metric [tier/total/capacity]
Unmapped metric [tier/replication/factor]
Unmapped metric [tier/historical/count]
Unmapped metric [segment/count]
Unmapped metric [coordinator/time]
Unmapped metric [coordinator/time]
Unmapped metric [coordinator/time]
Unmapped metric [coordinator/time]
Unmapped metric [coordinator/time]
Unmapped metric [coordinator/time]
Unmapped metric [coordinator/global/time]
```
Output from historical:
```
Unmapped metric [query/cache/delta/put/ok]
Unmapped metric [query/cache/delta/put/error]
Unmapped metric [query/cache/delta/put/oversized]
Unmapped metric [query/cache/total/put/ok]
Unmapped metric [query/cache/total/put/error]
Unmapped metric [query/cache/total/put/oversized]
Unmapped metric [query/cache/caffeine/delta/requests]
Unmapped metric [query/cache/caffeine/total/requests]
Unmapped metric [query/cache/caffeine/delta/loadTime]
Unmapped metric [query/cache/caffeine/total/loadTime]
Unmapped metric [query/cache/caffeine/delta/evictionBytes]
Unmapped metric [query/cache/caffeine/total/evictionBytes]
Unmapped metric [jvm/cpu/user]
Unmapped metric [jvm/cpu/total]
Unmapped metric [jvm/cpu/sys]
Unmapped metric [jvm/cpu/percent]
Unmapped metric [query/timeout/count]
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/heapAlloc/bytes]
Unmapped metric [jvm/threads/started]
Unmapped metric [jvm/threads/finished]
Unmapped metric [jvm/threads/live]
Unmapped metric [jvm/threads/liveDaemon]
Unmapped metric [jvm/threads/livePeak]
Unmapped metric [jetty/numOpenConnections]
Unmapped metric [jetty/threadPool/total]
Unmapped metric [jetty/threadPool/idle]
Unmapped metric [jetty/threadPool/isLowOnThreads]
Unmapped metric [jetty/threadPool/min]
Unmapped metric [jetty/threadPool/max]
Unmapped metric [jetty/threadPool/queueSize]
Unmapped metric [jetty/threadPool/busy]
Unmapped metric [segment/count]
```
Output from router:
```
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/gc/mem/max]
Unmapped metric [jvm/gc/mem/capacity]
Unmapped metric [jvm/gc/mem/used]
Unmapped metric [jvm/gc/mem/init]
Unmapped metric [jvm/heapAlloc/bytes]
Unmapped metric [jvm/cpu/user]
Unmapped metric [jvm/cpu/total]
Unmapped metric [jvm/cpu/sys]
Unmapped metric [jvm/cpu/percent]
Unmapped metric [jetty/numOpenConnections]
Unmapped metric [jetty/threadPool/total]
Unmapped metric [jetty/threadPool/idle]
Unmapped metric [jetty/threadPool/isLowOnThreads]
Unmapped metric [jetty/threadPool/min]
Unmapped metric [jetty/threadPool/max]
Unmapped metric [jetty/threadPool/queueSize]
Unmapped metric [jetty/threadPool/busy]
Unmapped metric [query/timeout/count]
Unmapped metric [jvm/threads/started]
Unmapped metric [jvm/threads/finished]
Unmapped metric [jvm/threads/live]
Unmapped metric [jvm/threads/liveDaemon]
Unmapped metric [jvm/threads/livePeak]
```
- Any debugging that you have already done
I have tried mapping the metrics which do not exist in metricDimensions.json
but it still complains. As an example, one could see that in my
metricsDimensions.json above,
```
"jvm/threads/started" : { "dimensions" : [], "type" : "count" },
"jvm/threads/finished" : { "dimensions" : [], "type" : "count" },
"jvm/threads/live" : { "dimensions" : [], "type" : "count" },
```
is included and the JvmThreadsMonitor is in the list of monitors for each
but I still see that it complains of unmapped metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
