JackieTien97 opened a new pull request, #18748:
URL: https://github.com/apache/iotdb/pull/18748
## Description
Changing `dn_metric_level` with `SET CONFIGURATION` + `LOAD CONFIGURATION`,
or calling `restartService()` of the metric service over JMX, runs
`MetricService.restartService()`: it drops all metrics, then unbinds and binds
every metric set again. Many metric sets could not survive this, so their
metric families disappeared until the node restarted, e.g. all `thread_pool_*`
series (#18681 fixed only `client_manager`). Some other metric series were
never exported or reported wrong values, because two owners registered the same
series, or the same metric name with different tag keys.
### Metrics lost after the metric service restarts
The metric sets followed three broken patterns:
- `unbindFrom` dropped the registrations that `bindTo` binds from, so
nothing was bound again. Now `unbindFrom` only removes the metrics and keeps
the registrations, which are removed when their owner deregisters:
`ThreadPoolMetrics`, the pipe metric sets on the DataNode and the ConfigNode,
`PipeEventCommitMetrics`, the subscription queue metrics, and the
IoTConsensusV2 sync lag, which is now released when the server stops instead of
when its metrics unbind. Thread pools are also unregistered by instance, so a
new pool with the same name is kept when the old one shuts down.
- Metric objects cached by their users were replaced by the restart. Ratis
creates its metrics once and keeps using them, so the Ratis metric registry
bridge now keeps its counters, timers and gauges, and points its proxies to the
new metrics when binding. The pipe sink compression timers are fetched on each
use.
- Metrics created only by one-shot paths are created again when binding,
with the recorded values: `TsFileMetrics` (`file_global_*`, `file_level_*`),
the per-table schema gauges, the `WritingMetrics` thresholds, WAL queue size
and active memtable / time partition counts, the load TsFile memory, the active
loading counters, the system disk metrics (also when the level was OFF before),
and the JVM GC pause timers. `JvmGcMetrics` also stops the GC listeners of the
previous binding, which kept running and reset the timers.
Besides, a metric set failing to rebind no longer stops the others,
`CacheMetrics` (partition cache hits) is registered at all, and
`RPCServiceThriftHandlerMetrics` no longer has two instances owning the same
series.
The ConfigNode starts the metric service after the consensus layer, so its
Ratis metrics (`ConfigRegion` ...) were missing from the start for the same
reason. They are exported now.
### Metric series with conflicting owners
- The metric manager requires the same tag keys for a metric name, and
silently returns a no-op metric otherwise.
- The per-table device number gauges added a `table` tag to
`schema_engine` / `schema_region`, so they were never exported. They now have
their own names `schema_engine_table` and `schema_region_table`, with the same
tags.
- The pipe-based and consensus-based subscription queues share three
metric names, but only the latter have the `region` tag (#18277), so whichever
registered later lost its metrics. The pipe-based queues now have the `region`
tag too, with an empty value as they are not bound to a region. Renaming the
metrics of the consensus-based queues instead would drop the dashboard
compatibility that #18277 kept on purpose.
- `mem{name="database_<db>"}` was registered by each data region of the
database, and `mem{name="chunkMetaData_<db>"}` by each TsFileProcessor, so the
gauge reported one of them and disappeared when any of them was closed. They
now sum up all data regions / processors of the database, and the last one
removes the gauge.
- The IoTConsensus `iot_send_log` stage timers of a region were created and
removed by the dispatcher thread of each peer: a peer leaving removed the
timers the others still used, and after a restart only one peer recorded into
the exported timers. The peers of a region now share the timers, and the last
one to unbind removes them. Making the timers per peer instead would change the
`name` tag.
- `WritingMetrics` recorded the active memtable count under `region="<N>"`,
while it creates and removes the counter under `region="DataRegion[<N>]"`, so
each region had two series and one of them leaked when the region was deleted.
- `JvmGcMonitorMetrics` (`jvm_gc_accumulated_time_percentage`) left the
latest sample out of the observation window, and kept the ring buffer indices
of the previous binding, which counted the samples before a restart again. As
the latest sample now counts, the first one is taken after a whole interval (3
s instead of 50 ms), so the percentage is never computed over a tiny window.
### Compatibility
- New metric names `schema_engine_table` and `schema_region_table`. These
gauges were never exported before.
- `subscription_uncommitted_event_count`, `subscription_current_commit_id`
and `subscription_event_transfer` of pipe-based queues get `region=""`.
Prometheus treats an empty label like a missing one; the IoTDB reporter paths
of these metrics get a `region=` node.
- `active_memtable_count` is only exported with `region="DataRegion[<N>]"`.
- `mem{name="database_<db>"}` and `mem{name="chunkMetaData_<db>"}` report
the totals of the database.
- The names and labels of the other metrics are unchanged.
### Validation
- 99 JUnit tests in 28 classes pass, 15 of the classes are new. They use a
real `MetricService`, as a mocked one cannot show the tag key conflicts.
- The new tests fail on the unmodified code, each with the symptom it
targets, e.g. thread pool, pipe, Ratis and file metrics missing after
`restartService()`, the table gauge not registered, 2 of 3 subscription queues
exported, 2 counters for one region, 1 of 2 timer records counted, 200 instead
of 300 for the database memory. `JvmGcMonitorMetricsTest` and the load TsFile
memory and cache cases of `DataNodeMetricsRestartTest` call methods added by
this PR, so they cannot run on the old code.
- The full English reactor build, the Chinese-locale test compilation and
Spotless pass.
- Manually on a 1C1D standalone (Windows) reporting to Prometheus at level
`ALL`, with tree model data, table model data and a pipe: scraped, set
`dn_metric_level` to `IMPORTANT` and back to `ALL` with `LOAD CONFIGURATION ON
LOCAL`, and scraped again, twice.
- DataNode: 0 of the 548 metric families of the first scrape were missing
afterwards (master: 234 of 545 after one round, 236 after two). Only series
created on first use were missing until their next use, e.g. per-statement
latencies.
- ConfigNode: 254 families, including 164 `ConfigRegion` Ratis series
(master: 112 families, no Ratis series).
- The per-table gauges were exported, and the sum of
`mem{name="database_*"}` equaled the sum of `data_region_mem_cost` (4255 =
4255; master: 2770 vs 4255).
- A temporary check logging every metric rejected for mismatched tag keys
reported none during the run.
- Not covered end to end: the subscription queue metrics need consumers, and
the IoTConsensus timers need replicas. Unit tests cover them.
```sh
mvn test -pl
iotdb-core/node-commons,iotdb-core/consensus,iotdb-core/confignode,iotdb-core/datanode
-am
-Dtest=ThreadPoolMetricsTest,MetricServiceRestartTest,JvmGcMonitorMetricsTest,PipeEventCommitMetricsTest,ClientManagerMetricsTest,MetricReporterSwitchTest,IoTDBThreadPoolFactoryTest,MetricManagerLifecycleTest,PrometheusReporterTest,RatisMetricSetTest,IoTConsensusV2ServerMetricsTest,RatisConsensusTest,LogDispatcherThreadMetricsTest,PipeConfigNodeMetricsRestartTest,PipeMetricsRestartTest,PipeSchemaRegionListenerMetricsTest,PipeSchemaRegionSinkMetricsTest,TsFileMetricsTest,DataNodeMetricsRestartTest,SchemaMemMetricTableTest,SubscriptionMetricsRestartTest,ConsensusSubscriptionPrefetchingQueueMetricsTest,WritingMetricsTest,MetricServiceTest,DatabaseMemMetricsTest,ActiveLoadListeningDirConfigTest,ActiveLoadDirScannerTest,ActiveLoadTsFileLoaderTest
-Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false
mvn test-compile -DskipTests
mvn test-compile -P with-zh-locale -DskipTests
```
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths.
- [x] been tested on a 1C1D standalone.
<hr>
##### Key changed/added classes in this PR
- `MetricService`, `ThreadPoolMetrics`, `PipeEventCommitMetrics`,
`JvmGcMonitorMetrics`, `Metric`
- `RatisMetricSet`, `MetricRegistryManager`, `IoTDBMetricRegistry`,
`CounterProxy`, `TimerProxy`
- `IoTConsensusV2ServerImpl`, `IoTConsensusV2ServerMetrics`,
`LogDispatcherThreadMetrics`
- The pipe metric sets under `org.apache.iotdb.db.pipe.metric` and
`org.apache.iotdb.confignode.manager.pipe.metric`
- `SubscriptionPrefetchingQueueMetrics`,
`ConsensusSubscriptionPrefetchingQueueMetrics`
- `TsFileMetrics`, `WritingMetrics`, `SchemaEngineMemMetric`,
`SchemaRegionMemMetric`, `DataRegionMetrics`, `TsFileProcessorInfoMetrics`
- `LoadTsFileMemMetricSet`, `ActiveLoadingFilesMetricsSet`,
`ActiveLoadingFilesNumberMetricsSet`
- `RPCServiceThriftHandlerMetrics`, `CacheMetrics`
- `SystemMetrics`, `JvmGcMetrics`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]