Caideyipi commented on code in PR #18562:
URL: https://github.com/apache/iotdb/pull/18562#discussion_r3910626991
##########
iotdb-core/metrics/interface/src/main/java/org/apache/iotdb/metrics/reporter/prometheus/PrometheusReporter.java:
##########
@@ -135,6 +173,66 @@ public boolean start() {
return true;
}
+ @SuppressWarnings("unsafeThreadSchedule")
+ private void startSnapshotUpdater() {
+ // Keep metric collection off Reactor HTTP threads and avoid overlapping
scrapes.
+ if (snapshotUpdateExecutor == null) {
+ snapshotUpdateExecutor =
+ Executors.newSingleThreadScheduledExecutor(
+ runnable -> {
+ Thread thread = new Thread(runnable,
"prometheus-reporter-snapshot-updater");
+ thread.setDaemon(true);
+ return thread;
+ });
+ }
+ // Delay the first background scrape until metric sets have been bound by
the metric service.
+ snapshotUpdateFuture =
+ snapshotUpdateExecutor.scheduleAtFixedRate(
+ this::updateSnapshot,
+ PROMETHEUS_DEFAULT_SCRAPE_INTERVAL_SECONDS,
+ PROMETHEUS_DEFAULT_SCRAPE_INTERVAL_SECONDS,
+ TimeUnit.SECONDS);
+ }
+
+ private void updateSnapshot() {
+ try {
+ String snapshot = scrape();
+ // Do not publish an empty scrape taken before metric sets are bound.
The request path will
+ // synchronously scrape until the first complete snapshot is available.
Empty snapshots are
+ // published after initialization so removed metrics are not kept in the
cache indefinitely.
+ if (!snapshot.isEmpty() || metricsSnapshot != null) {
+ metricsSnapshot = snapshot;
+ }
+ } catch (Throwable t) {
+ LOGGER.error(
+ MetricsMessages
+
.LOG_PROMETHEUSREPORTER_FAILED_TO_UPDATE_METRICS_SNAPSHOT_ASYNCHRONOUSLY_F19FE4E3,
+ t);
+ }
+ }
+
+ private String getMetricsSnapshot() {
+ String snapshot = metricsSnapshot;
+ if (snapshot == null) {
+ snapshot = scrape();
+ if (!snapshot.isEmpty()) {
+ metricsSnapshot = snapshot;
+ }
+ }
+ return snapshot;
+ }
+
+ private void stopSnapshotUpdater() {
+ if (snapshotUpdateFuture != null) {
+ snapshotUpdateFuture.cancel(false);
+ snapshotUpdateFuture = null;
+ }
+ if (snapshotUpdateExecutor != null) {
+ snapshotUpdateExecutor.shutdownNow();
Review Comment:
**[P2] Recreate the managed scheduler when the reporter restarts**
`stopSnapshotUpdater()` always shuts down and clears the executor supplied
by `MetricService`. If this reporter is started again through the public
`AbstractMetricService.stop/start(ReporterType)` APIs, or retried after
`start()` failed, `startSnapshotUpdater()` falls back to a raw daemon executor.
That replacement is not registered with JMX or `ThreadPoolMetrics`, and its
lowercase name does not match
`ThreadName.PROMETHEUS_REPORTER_SNAPSHOT_UPDATER`, so CPU metrics classify it
as `UNKNOWN`. Please retain an executor supplier/factory and create a new
IoTDB-managed executor on every start (or recreate the reporter) rather than
losing the managed executor after the first stop or failed start.
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/service/metric/MetricService.java:
##########
@@ -74,7 +76,15 @@ protected void loadReporter() {
hasJmxReporter = true;
break;
case PROMETHEUS:
- reporter = new PrometheusReporter(metricManager);
+ if (METRIC_CONFIG.isPrometheusReporterAsyncUpdate()) {
+ reporter =
+ new PrometheusReporter(
+ metricManager,
+ IoTDBThreadPoolFactory.newSingleThreadScheduledExecutor(
Review Comment:
**[P3] Avoid registering the scheduler before duplicate reporters are
rejected**
`setMetricReporterList()` preserves duplicates, while
`CompositeReporter.addReporter()` rejects them only after this constructor has
run. With `PROMETHEUS,PROMETHEUS`, the second `IoTDBThreadPoolFactory` executor
registers under the same pool name and overwrites the first entry in
`ThreadPoolMetrics`; the second reporter is then discarded. The active
updater's `thread_pool_*` gauges therefore point at the never-started executor.
Please deduplicate before constructing reporters, make `addReporter()` report
acceptance, or allocate the scheduler lazily in `start()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
