JackieTien97 opened a new pull request, #18748:
URL: https://github.com/apache/iotdb/pull/18748

   ## Description
   
   Changing `dn_metric_level` with `SET CONFIGURATION` + `LOAD CONFIGURATION`, 
or calling `restartService()` of the metric service over JMX, runs 
`MetricService.restartService()`: it drops all metrics, then unbinds and binds 
every metric set again. Many metric sets could not survive this, so their 
metric families disappeared until the node restarted, e.g. all `thread_pool_*` 
series (#18681 fixed only `client_manager`). Some other metric series were 
never exported or reported wrong values, because two owners registered the same 
series, or the same metric name with different tag keys.
   
   ### Metrics lost after the metric service restarts
   
   The metric sets followed three broken patterns:
   
   - `unbindFrom` dropped the registrations that `bindTo` binds from, so 
nothing was bound again. Now `unbindFrom` only removes the metrics and keeps 
the registrations, which are removed when their owner deregisters: 
`ThreadPoolMetrics`, the pipe metric sets on the DataNode and the ConfigNode, 
`PipeEventCommitMetrics`, the subscription queue metrics, and the 
IoTConsensusV2 sync lag, which is now released when the server stops instead of 
when its metrics unbind. Thread pools are also unregistered by instance, so a 
new pool with the same name is kept when the old one shuts down.
   - Metric objects cached by their users were replaced by the restart. Ratis 
creates its metrics once and keeps using them, so the Ratis metric registry 
bridge now keeps its counters, timers and gauges, and points its proxies to the 
new metrics when binding. The pipe sink compression timers are fetched on each 
use.
   - Metrics created only by one-shot paths are created again when binding, 
with the recorded values: `TsFileMetrics` (`file_global_*`, `file_level_*`), 
the per-table schema gauges, the `WritingMetrics` thresholds, WAL queue size 
and active memtable / time partition counts, the load TsFile memory, the active 
loading counters, the system disk metrics (also when the level was OFF before), 
and the JVM GC pause timers. `JvmGcMetrics` also stops the GC listeners of the 
previous binding, which kept running and reset the timers.
   
   Besides, a metric set failing to rebind no longer stops the others, 
`CacheMetrics` (partition cache hits) is registered at all, and 
`RPCServiceThriftHandlerMetrics` no longer has two instances owning the same 
series.
   
   The ConfigNode starts the metric service after the consensus layer, so its 
Ratis metrics (`ConfigRegion` ...) were missing from the start for the same 
reason. They are exported now.
   
   ### Metric series with conflicting owners
   
   - The metric manager requires the same tag keys for a metric name, and 
silently returns a no-op metric otherwise.
     - The per-table device number gauges added a `table` tag to 
`schema_engine` / `schema_region`, so they were never exported. They now have 
their own names `schema_engine_table` and `schema_region_table`, with the same 
tags.
     - The pipe-based and consensus-based subscription queues share three 
metric names, but only the latter have the `region` tag (#18277), so whichever 
registered later lost its metrics. The pipe-based queues now have the `region` 
tag too, with an empty value as they are not bound to a region. Renaming the 
metrics of the consensus-based queues instead would drop the dashboard 
compatibility that #18277 kept on purpose.
   - `mem{name="database_<db>"}` was registered by each data region of the 
database, and `mem{name="chunkMetaData_<db>"}` by each TsFileProcessor, so the 
gauge reported one of them and disappeared when any of them was closed. They 
now sum up all data regions / processors of the database, and the last one 
removes the gauge.
   - The IoTConsensus `iot_send_log` stage timers of a region were created and 
removed by the dispatcher thread of each peer: a peer leaving removed the 
timers the others still used, and after a restart only one peer recorded into 
the exported timers. The peers of a region now share the timers, and the last 
one to unbind removes them. Making the timers per peer instead would change the 
`name` tag.
   - `WritingMetrics` recorded the active memtable count under `region="<N>"`, 
while it creates and removes the counter under `region="DataRegion[<N>]"`, so 
each region had two series and one of them leaked when the region was deleted.
   - `JvmGcMonitorMetrics` (`jvm_gc_accumulated_time_percentage`) left the 
latest sample out of the observation window, and kept the ring buffer indices 
of the previous binding, which counted the samples before a restart again. As 
the latest sample now counts, the first one is taken after a whole interval (3 
s instead of 50 ms), so the percentage is never computed over a tiny window.
   
   ### Compatibility
   
   - New metric names `schema_engine_table` and `schema_region_table`. These 
gauges were never exported before.
   - `subscription_uncommitted_event_count`, `subscription_current_commit_id` 
and `subscription_event_transfer` of pipe-based queues get `region=""`. 
Prometheus treats an empty label like a missing one; the IoTDB reporter paths 
of these metrics get a `region=` node.
   - `active_memtable_count` is only exported with `region="DataRegion[<N>]"`.
   - `mem{name="database_<db>"}` and `mem{name="chunkMetaData_<db>"}` report 
the totals of the database.
   - The names and labels of the other metrics are unchanged.
   
   ### Validation
   
   - 99 JUnit tests in 28 classes pass, 15 of the classes are new. They use a 
real `MetricService`, as a mocked one cannot show the tag key conflicts.
   - The new tests fail on the unmodified code, each with the symptom it 
targets, e.g. thread pool, pipe, Ratis and file metrics missing after 
`restartService()`, the table gauge not registered, 2 of 3 subscription queues 
exported, 2 counters for one region, 1 of 2 timer records counted, 200 instead 
of 300 for the database memory. `JvmGcMonitorMetricsTest` and the load TsFile 
memory and cache cases of `DataNodeMetricsRestartTest` call methods added by 
this PR, so they cannot run on the old code.
   - The full English reactor build, the Chinese-locale test compilation and 
Spotless pass.
   - Manually on a 1C1D standalone (Windows) reporting to Prometheus at level 
`ALL`, with tree model data, table model data and a pipe: scraped, set 
`dn_metric_level` to `IMPORTANT` and back to `ALL` with `LOAD CONFIGURATION ON 
LOCAL`, and scraped again, twice.
     - DataNode: 0 of the 548 metric families of the first scrape were missing 
afterwards (master: 234 of 545 after one round, 236 after two). Only series 
created on first use were missing until their next use, e.g. per-statement 
latencies.
     - ConfigNode: 254 families, including 164 `ConfigRegion` Ratis series 
(master: 112 families, no Ratis series).
     - The per-table gauges were exported, and the sum of 
`mem{name="database_*"}` equaled the sum of `data_region_mem_cost` (4255 = 
4255; master: 2770 vs 4255).
     - A temporary check logging every metric rejected for mismatched tag keys 
reported none during the run.
   - Not covered end to end: the subscription queue metrics need consumers, and 
the IoTConsensus timers need replicas. Unit tests cover them.
   
   ```sh
   mvn test -pl 
iotdb-core/node-commons,iotdb-core/consensus,iotdb-core/confignode,iotdb-core/datanode
 -am 
-Dtest=ThreadPoolMetricsTest,MetricServiceRestartTest,JvmGcMonitorMetricsTest,PipeEventCommitMetricsTest,ClientManagerMetricsTest,MetricReporterSwitchTest,IoTDBThreadPoolFactoryTest,MetricManagerLifecycleTest,PrometheusReporterTest,RatisMetricSetTest,IoTConsensusV2ServerMetricsTest,RatisConsensusTest,LogDispatcherThreadMetricsTest,PipeConfigNodeMetricsRestartTest,PipeMetricsRestartTest,PipeSchemaRegionListenerMetricsTest,PipeSchemaRegionSinkMetricsTest,TsFileMetricsTest,DataNodeMetricsRestartTest,SchemaMemMetricTableTest,SubscriptionMetricsRestartTest,ConsensusSubscriptionPrefetchingQueueMetricsTest,WritingMetricsTest,MetricServiceTest,DatabaseMemMetricsTest,ActiveLoadListeningDirConfigTest,ActiveLoadDirScannerTest,ActiveLoadTsFileLoaderTest
 -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false
   mvn test-compile -DskipTests
   mvn test-compile -P with-zh-locale -DskipTests
   ```
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added comments explaining the "why" and the intent of the code 
wherever would not be obvious for an unfamiliar reader.
   - [x] added unit tests or modified existing tests to cover new code paths.
   - [x] been tested on a 1C1D standalone.
   
   <hr>
   
   ##### Key changed/added classes in this PR
   
   - `MetricService`, `ThreadPoolMetrics`, `PipeEventCommitMetrics`, 
`JvmGcMonitorMetrics`, `Metric`
   - `RatisMetricSet`, `MetricRegistryManager`, `IoTDBMetricRegistry`, 
`CounterProxy`, `TimerProxy`
   - `IoTConsensusV2ServerImpl`, `IoTConsensusV2ServerMetrics`, 
`LogDispatcherThreadMetrics`
   - The pipe metric sets under `org.apache.iotdb.db.pipe.metric` and 
`org.apache.iotdb.confignode.manager.pipe.metric`
   - `SubscriptionPrefetchingQueueMetrics`, 
`ConsensusSubscriptionPrefetchingQueueMetrics`
   - `TsFileMetrics`, `WritingMetrics`, `SchemaEngineMemMetric`, 
`SchemaRegionMemMetric`, `DataRegionMetrics`, `TsFileProcessorInfoMetrics`
   - `LoadTsFileMemMetricSet`, `ActiveLoadingFilesMetricsSet`, 
`ActiveLoadingFilesNumberMetricsSet`
   - `RPCServiceThriftHandlerMetrics`, `CacheMetrics`
   - `SystemMetrics`, `JvmGcMetrics`
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to