[
https://issues.apache.org/jira/browse/GOBBLIN-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zihan Li updated GOBBLIN-1598:
------------------------------
Description:
{code:java}
2021/06/12 07:11:32.067 ERROR [DagManager] [pool-19-thread-3]
[gobblin-service-war] [] Exception encountered in
org.apache.gobblin.service.modules.orchestration.DagManager$DagManagerThread
java.lang.IllegalArgumentException: A metric named
GobblinService.testGroup.testFlow.RunningStatus already exists
at
org.apache.gobblin.metrics.InnerMetricContext.register(InnerMetricContext.java:266)
at org.apache.gobblin.metrics.MetricContext.register(MetricContext.java:409)
at
org.apache.gobblin.service.modules.orchestration.DagManager$DagManagerThread.initialize(DagManager.java:661)
at
org.apache.gobblin.service.modules.orchestration.DagManager$DagManagerThread.run(DagManager.java:493)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_172]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[?:1.8.0_172]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_172]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0_172]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_172]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_172]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
{code}
Likely root cause:
If we activate/deactivate DagManager several times, DagManagerThreads alongside
with their flowGauges maps will be recreated, but RootMetricContext will still
stay the same. So the newly created threads will incorrectly think that the
flow was not registered previously, and try to register it again.
was:
{code:java}
2021/06/12 07:11:32.067 ERROR [DagManager] [pool-19-thread-3]
[gobblin-service-war] [] Exception encountered in
org.apache.gobblin.service.modules.orchestration.DagManager$DagManagerThread
java.lang.IllegalArgumentException: A metric named
GobblinService.SN_CRMSYNC.DY-CON-579112402-461.RunningStatus already exists
at
org.apache.gobblin.metrics.InnerMetricContext.register(InnerMetricContext.java:266)
~[gobblin-service-fat-2.0.8-all.jar:?]
at org.apache.gobblin.metrics.MetricContext.register(MetricContext.java:409)
~[gobblin-service-fat-2.0.8-all.jar:?]
at
org.apache.gobblin.service.modules.orchestration.DagManager$DagManagerThread.initialize(DagManager.java:661)
~[gobblin-service-fat-2.0.8-all.jar:?]
at
org.apache.gobblin.service.modules.orchestration.DagManager$DagManagerThread.run(DagManager.java:493)
[gobblin-service-fat-2.0.8-all.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_172]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[?:1.8.0_172]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_172]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0_172]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_172]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_172]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
{code}
Likely root cause:
If we activate/deactivate DagManager several times, DagManagerThreads alongside
with their flowGauges maps will be recreated, but RootMetricContext will still
stay the same. So the newly created threads will incorrectly think that the
flow was not registered previously, and try to register it again.
> Fix metrics already exist issue in dag manager
> ----------------------------------------------
>
> Key: GOBBLIN-1598
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1598
> Project: Apache Gobblin
> Issue Type: Bug
> Reporter: Zihan Li
> Priority: Major
>
> {code:java}
> 2021/06/12 07:11:32.067 ERROR [DagManager] [pool-19-thread-3]
> [gobblin-service-war] [] Exception encountered in
> org.apache.gobblin.service.modules.orchestration.DagManager$DagManagerThread
> java.lang.IllegalArgumentException: A metric named
> GobblinService.testGroup.testFlow.RunningStatus already exists
> at
> org.apache.gobblin.metrics.InnerMetricContext.register(InnerMetricContext.java:266)
>
> at org.apache.gobblin.metrics.MetricContext.register(MetricContext.java:409)
> at
> org.apache.gobblin.service.modules.orchestration.DagManager$DagManagerThread.initialize(DagManager.java:661)
>
> at
> org.apache.gobblin.service.modules.orchestration.DagManager$DagManagerThread.run(DagManager.java:493)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_172]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [?:1.8.0_172]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [?:1.8.0_172]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [?:1.8.0_172]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_172]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_172]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
> {code}
> Likely root cause:
> If we activate/deactivate DagManager several times, DagManagerThreads
> alongside with their flowGauges maps will be recreated, but RootMetricContext
> will still stay the same. So the newly created threads will incorrectly think
> that the flow was not registered previously, and try to register it again.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
