Zihan Li created GOBBLIN-1598:
---------------------------------
Summary: Fix metrics already exist issue in dag manager
Key: GOBBLIN-1598
URL: https://issues.apache.org/jira/browse/GOBBLIN-1598
Project: Apache Gobblin
Issue Type: Bug
Reporter: Zihan Li
{code:java}
2021/06/12 07:11:32.067 ERROR [DagManager] [pool-19-thread-3]
[gobblin-service-war] [] Exception encountered in
org.apache.gobblin.service.modules.orchestration.DagManager$DagManagerThread
java.lang.IllegalArgumentException: A metric named
GobblinService.SN_CRMSYNC.DY-CON-579112402-461.RunningStatus already exists
at
org.apache.gobblin.metrics.InnerMetricContext.register(InnerMetricContext.java:266)
~[gobblin-service-fat-2.0.8-all.jar:?]
at org.apache.gobblin.metrics.MetricContext.register(MetricContext.java:409)
~[gobblin-service-fat-2.0.8-all.jar:?]
at
org.apache.gobblin.service.modules.orchestration.DagManager$DagManagerThread.initialize(DagManager.java:661)
~[gobblin-service-fat-2.0.8-all.jar:?]
at
org.apache.gobblin.service.modules.orchestration.DagManager$DagManagerThread.run(DagManager.java:493)
[gobblin-service-fat-2.0.8-all.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_172]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[?:1.8.0_172]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_172]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0_172]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_172]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_172]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
{code}
Likely root cause:
If we activate/deactivate DagManager several times, DagManagerThreads alongside
with their flowGauges maps will be recreated, but RootMetricContext will still
stay the same. So the newly created threads will incorrectly think that the
flow was not registered previously, and try to register it again.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
