[
https://issues.apache.org/jira/browse/IOTDB-3961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17571168#comment-17571168
]
Bo Zhang commented on IOTDB-3961:
---------------------------------
root cause is : when report Metric to Prometheus, it triggers exception, which
casue CompactionWorker.run exists and can not recover.
=================================================================
2022-07-25 17:49:12,907 [pool-8-IoTDB-Compaction-3] ERROR
o.a.i.d.e.c.CompactionWorker:80 - run(), CompactionWorker exception.
java.lang.IllegalArgumentException: Prometheus requires that all meters with
the same name have the same set of tag keys. There is already an existing meter
named 'compaction_task_count' containing tag keys [name]. The meter you are
attempting to register has keys [name, type].
at
io.micrometer.prometheus.PrometheusMeterRegistry.lambda$throwExceptionOnRegistrationFailure$17(PrometheusMeterRegistry.java:444)
at
io.micrometer.core.instrument.MeterRegistry.meterRegistrationFailed(MeterRegistry.java:1028)
at
io.micrometer.prometheus.PrometheusMeterRegistry.lambda$applyToCollector$16(PrometheusMeterRegistry.java:419)
at
java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1877)
at
io.micrometer.prometheus.PrometheusMeterRegistry.applyToCollector(PrometheusMeterRegistry.java:406)
at
io.micrometer.prometheus.PrometheusMeterRegistry.newCounter(PrometheusMeterRegistry.java:109)
at
io.micrometer.core.instrument.MeterRegistry.lambda$registerMeterIfNecessary$5(MeterRegistry.java:561)
at
io.micrometer.core.instrument.MeterRegistry.getOrCreateMeter(MeterRegistry.java:614)
at
io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:568)
at
io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:561)
at
io.micrometer.core.instrument.MeterRegistry.counter(MeterRegistry.java:284)
at
io.micrometer.core.instrument.Counter$Builder.register(Counter.java:128)
at
io.micrometer.core.instrument.composite.CompositeCounter.registerNewMeter(CompositeCounter.java:50)
at
io.micrometer.core.instrument.composite.CompositeCounter.registerNewMeter(CompositeCounter.java:23)
at
io.micrometer.core.instrument.composite.AbstractCompositeMeter.add(AbstractCompositeMeter.java:66)
at java.lang.Iterable.forEach(Iterable.java:75)
at java.util.Collections$SetFromMap.forEach(Collections.java:5530)
at
io.micrometer.core.instrument.composite.CompositeMeterRegistry.lambda$new$0(CompositeMeterRegistry.java:65)
at
io.micrometer.core.instrument.composite.CompositeMeterRegistry.lock(CompositeMeterRegistry.java:184)
at
io.micrometer.core.instrument.composite.CompositeMeterRegistry.lambda$new$1(CompositeMeterRegistry.java:65)
at
io.micrometer.core.instrument.MeterRegistry.getOrCreateMeter(MeterRegistry.java:624)
at
io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:568)
at
io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:561)
at
io.micrometer.core.instrument.MeterRegistry.counter(MeterRegistry.java:284)
at
io.micrometer.core.instrument.Counter$Builder.register(Counter.java:128)
at
io.micrometer.core.instrument.MeterRegistry.counter(MeterRegistry.java:391)
at
io.micrometer.core.instrument.MeterRegistry.counter(MeterRegistry.java:402)
at
org.apache.iotdb.metrics.micrometer.MicrometerMetricManager.lambda$count$7(MicrometerMetricManager.java:218)
at
java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at
org.apache.iotdb.metrics.micrometer.MicrometerMetricManager.count(MicrometerMetricManager.java:217)
at
org.apache.iotdb.db.service.metrics.recorder.CompactionMetricsRecorder.recordTaskInfo(CompactionMetricsRecorder.java:151)
at
org.apache.iotdb.db.engine.compaction.CompactionTaskManager.removeRunningTaskFuture(CompactionTaskManager.java:286)
at
org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask.start(AbstractCompactionTask.java:81)
at
org.apache.iotdb.db.engine.compaction.CompactionWorker.run(CompactionWorker.java:75)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
> bug for Tsfile compact
> ----------------------
>
> Key: IOTDB-3961
> URL: https://issues.apache.org/jira/browse/IOTDB-3961
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Compaction
> Reporter: Bo Zhang
> Priority: Major
>
> find 1 critial bug during long term stabliity testing.
> After 4 hours stabliity testing, the tsfiles' number will become bigger and
> bigger.
>
> !https://aone.alibaba-inc.com/v2/api/workitem/adapter/file/url?fileIdentifier=workitem%2Falibaba%2Fdefault%2F1658719978006image.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)