[jira] [Work logged] (BEAM-6829) Duplicate metric warnings clutter log
[
https://issues.apache.org/jira/browse/BEAM-6829?focusedWorklogId=325450=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325450
]
ASF GitHub Bot logged work on BEAM-6829:
Author: ASF GitHub Bot
Created on: 09/Oct/19 02:43
Start Date: 09/Oct/19 02:43
Worklog Time Spent: 10m
Work Description: tweise commented on issue #8585: [BEAM-6829] Use
transform/pcollection name for metric namespace if none provided
URL: https://github.com/apache/beam/pull/8585#issuecomment-539796815
Manual test shows that duplicate metric warnings are gone.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 325450)
Time Spent: 3h 40m (was: 3.5h)
> Duplicate metric warnings clutter log
> -
>
> Key: BEAM-6829
> URL: https://issues.apache.org/jira/browse/BEAM-6829
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.11.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Labels: portability
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Logs fill up quickly with these warnings:
> {code:java}
> WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
> contains a Metric with the name ...{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-6829) Duplicate metric warnings clutter log
[
https://issues.apache.org/jira/browse/BEAM-6829?focusedWorklogId=325449=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325449
]
ASF GitHub Bot logged work on BEAM-6829:
Author: ASF GitHub Bot
Created on: 09/Oct/19 02:42
Start Date: 09/Oct/19 02:42
Worklog Time Spent: 10m
Work Description: tweise commented on pull request #8585: [BEAM-6829] Use
transform/pcollection name for metric namespace if none provided
URL: https://github.com/apache/beam/pull/8585
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 325449)
Time Spent: 3.5h (was: 3h 20m)
> Duplicate metric warnings clutter log
> -
>
> Key: BEAM-6829
> URL: https://issues.apache.org/jira/browse/BEAM-6829
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.11.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Labels: portability
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Logs fill up quickly with these warnings:
> {code:java}
> WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
> contains a Metric with the name ...{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-6829) Duplicate metric warnings clutter log
[
https://issues.apache.org/jira/browse/BEAM-6829?focusedWorklogId=325428=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325428
]
ASF GitHub Bot logged work on BEAM-6829:
Author: ASF GitHub Bot
Created on: 09/Oct/19 01:33
Start Date: 09/Oct/19 01:33
Worklog Time Spent: 10m
Work Description: tweise commented on pull request #8585: [BEAM-6829] Use
transform/pcollection name for metric namespace if none provided
URL: https://github.com/apache/beam/pull/8585#discussion_r332798020
##
File path:
runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricName.java
##
@@ -51,7 +51,14 @@ private MonitoringInfoMetricName(String urn, Map labels) {
@Override
public String getNamespace() {
if (labels.containsKey(MonitoringInfoConstants.Labels.NAMESPACE)) {
+ // User-generated metric
return labels.getOrDefault(MonitoringInfoConstants.Labels.NAMESPACE,
null);
+} else if (labels.containsKey(MonitoringInfoConstants.Labels.PCOLLECTION))
{
+ // System-generated metric, prepend with a colon
+ return ":" +
labels.getOrDefault(MonitoringInfoConstants.Labels.PCOLLECTION, null);
+} else if (labels.containsKey(MonitoringInfoConstants.Labels.PTRANSFORM)) {
+ // System-generated metric, prepend with a colon
+ return ":" +
labels.getOrDefault(MonitoringInfoConstants.Labels.PTRANSFORM, null);
Review comment:
Agreed, there should be no need for this prefix. I'm going to remove it. I
wouldn't be surprised though if we revisit the metric names as we test this
with a real world metric backend. The current names are quite verbose. Examples:
`36read/Read/Reshuffle/RemoveRandomKeys.None/beam:env:docker:v1:0:beam:metric:user_distribution
{NAMESPACE=__main__.WordExtractingDoFn,
PTRANSFORM=ref_AppliedPTransform_split_17, NAME=word_len_dist}:
DistributionResult{sum=114, count=29, min=0, max=11}`
(user metric created as `self.word_lengths_dist =
Metrics.distribution(self.__class__, 'word_len_dist')`
or
`17read/Read/Impulse.None/beam:env:docker:v1:0:beam:metric:sampled_byte_size:v1
{PCOLLECTION=ref_PCollection_PCollection_1}: DistributionResult{sum=13,
count=1, min=13, max=13},
6format.None/beam:env:docker:v1:0:beam:metric:sampled_byte_size:v1
{PCOLLECTION=ref_PCollection_PCollection_17}: DistributionResult{sum=395,
count=18, min=19, max=29}`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 325428)
Time Spent: 3h 20m (was: 3h 10m)
> Duplicate metric warnings clutter log
> -
>
> Key: BEAM-6829
> URL: https://issues.apache.org/jira/browse/BEAM-6829
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.11.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Labels: portability
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Logs fill up quickly with these warnings:
> {code:java}
> WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
> contains a Metric with the name ...{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-6829) Duplicate metric warnings clutter log
[
https://issues.apache.org/jira/browse/BEAM-6829?focusedWorklogId=325387=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-325387
]
ASF GitHub Bot logged work on BEAM-6829:
Author: ASF GitHub Bot
Created on: 08/Oct/19 23:53
Start Date: 08/Oct/19 23:53
Worklog Time Spent: 10m
Work Description: tweise commented on issue #8585: [BEAM-6829] Use
transform/pcollection name for metric namespace if none provided
URL: https://github.com/apache/beam/pull/8585#issuecomment-539751962
Run Java PreCommit
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 325387)
Time Spent: 3h 10m (was: 3h)
> Duplicate metric warnings clutter log
> -
>
> Key: BEAM-6829
> URL: https://issues.apache.org/jira/browse/BEAM-6829
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.11.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Labels: portability
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Logs fill up quickly with these warnings:
> {code:java}
> WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
> contains a Metric with the name ...{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-6829) Duplicate metric warnings clutter log
[
https://issues.apache.org/jira/browse/BEAM-6829?focusedWorklogId=322496=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322496
]
ASF GitHub Bot logged work on BEAM-6829:
Author: ASF GitHub Bot
Created on: 03/Oct/19 09:37
Start Date: 03/Oct/19 09:37
Worklog Time Spent: 10m
Work Description: mxm commented on issue #8585: [BEAM-6829] Use
transform/pcollection name for metric namespace if none provided
URL: https://github.com/apache/beam/pull/8585#issuecomment-537869268
>Sounds good, I will try it again with the latest changes.
Thanks, curious to see the results! :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 322496)
Time Spent: 3h (was: 2h 50m)
> Duplicate metric warnings clutter log
> -
>
> Key: BEAM-6829
> URL: https://issues.apache.org/jira/browse/BEAM-6829
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.11.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Labels: portability
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Logs fill up quickly with these warnings:
> {code:java}
> WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
> contains a Metric with the name ...{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-6829) Duplicate metric warnings clutter log
[
https://issues.apache.org/jira/browse/BEAM-6829?focusedWorklogId=322492=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322492
]
ASF GitHub Bot logged work on BEAM-6829:
Author: ASF GitHub Bot
Created on: 03/Oct/19 09:31
Start Date: 03/Oct/19 09:31
Worklog Time Spent: 10m
Work Description: mxm commented on issue #8585: [BEAM-6829] Use
transform/pcollection name for metric namespace if none provided
URL: https://github.com/apache/beam/pull/8585#issuecomment-537584412
@tweise I've revisited this problem and found two other important fixes in
addition to the first commit:
1. We also need to include PCollection-scoped metrics, e.g. num elements
2. The metric reporting was doubled in ExecutableDoFnOperator. Once by the
operator,
once by the wrapping metrics DoFnRunner.
In my tests I could not find any duplicate metrics anymore. Plus, the
scoping of the metrics is done correctly, whether it is user metrics, transform
metrics, or pcollection metrics.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 322492)
Time Spent: 2h 40m (was: 2.5h)
> Duplicate metric warnings clutter log
> -
>
> Key: BEAM-6829
> URL: https://issues.apache.org/jira/browse/BEAM-6829
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.11.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Labels: portability
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Logs fill up quickly with these warnings:
> {code:java}
> WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
> contains a Metric with the name ...{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-6829) Duplicate metric warnings clutter log
[
https://issues.apache.org/jira/browse/BEAM-6829?focusedWorklogId=322204=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322204
]
ASF GitHub Bot logged work on BEAM-6829:
Author: ASF GitHub Bot
Created on: 02/Oct/19 20:13
Start Date: 02/Oct/19 20:13
Worklog Time Spent: 10m
Work Description: tweise commented on issue #8585: [BEAM-6829] Use
transform/pcollection name for metric namespace if none provided
URL: https://github.com/apache/beam/pull/8585#issuecomment-537660439
Sounds good, I will try it again with the latest changes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 322204)
Time Spent: 2.5h (was: 2h 20m)
> Duplicate metric warnings clutter log
> -
>
> Key: BEAM-6829
> URL: https://issues.apache.org/jira/browse/BEAM-6829
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.11.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Labels: portability
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Logs fill up quickly with these warnings:
> {code:java}
> WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
> contains a Metric with the name ...{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-6829) Duplicate metric warnings clutter log
[
https://issues.apache.org/jira/browse/BEAM-6829?focusedWorklogId=322156=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322156
]
ASF GitHub Bot logged work on BEAM-6829:
Author: ASF GitHub Bot
Created on: 02/Oct/19 18:59
Start Date: 02/Oct/19 18:59
Worklog Time Spent: 10m
Work Description: mxm commented on issue #8585: [BEAM-6829] Use
transform/pcollection name for metric namespace if none provided
URL: https://github.com/apache/beam/pull/8585#issuecomment-537633043
Test failures are unrelated:
https://builds.apache.org/job/beam_PreCommit_Java_Commit/7949/
```
Test Result (2 failures / +2)
org.apache.beam.sdk.io.TFRecordIOTest.testReadInvalidRecord
org.apache.beam.sdk.transforms.ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 322156)
Time Spent: 2h 20m (was: 2h 10m)
> Duplicate metric warnings clutter log
> -
>
> Key: BEAM-6829
> URL: https://issues.apache.org/jira/browse/BEAM-6829
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.11.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Labels: portability
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Logs fill up quickly with these warnings:
> {code:java}
> WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
> contains a Metric with the name ...{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-6829) Duplicate metric warnings clutter log
[
https://issues.apache.org/jira/browse/BEAM-6829?focusedWorklogId=322052=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322052
]
ASF GitHub Bot logged work on BEAM-6829:
Author: ASF GitHub Bot
Created on: 02/Oct/19 16:55
Start Date: 02/Oct/19 16:55
Worklog Time Spent: 10m
Work Description: mxm commented on issue #8585: [BEAM-6829] Use transform
name for metric namespace if none provided
URL: https://github.com/apache/beam/pull/8585#issuecomment-537584412
@tweise I've revisited this problem and found two other important fixes in
addition to the first commit:
1. We also need to include PCollection-scoped metrics, e.g. num elements
2. The metric reporting was doubled in ExecutableDoFnOperator. Once by the
operator,
once by the wrapping metrics operator.
In my tests I could not find any duplicate metrics anymore. Plus, the
scoping of the metrics is done correctly, whether it is user metrics, transform
metrics, or pcollection metrics.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 322052)
Time Spent: 2h 10m (was: 2h)
> Duplicate metric warnings clutter log
> -
>
> Key: BEAM-6829
> URL: https://issues.apache.org/jira/browse/BEAM-6829
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.11.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Labels: portability
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Logs fill up quickly with these warnings:
> {code:java}
> WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
> contains a Metric with the name ...{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-6829) Duplicate metric warnings clutter log
[ https://issues.apache.org/jira/browse/BEAM-6829?focusedWorklogId=322046=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322046 ] ASF GitHub Bot logged work on BEAM-6829: Author: ASF GitHub Bot Created on: 02/Oct/19 16:47 Start Date: 02/Oct/19 16:47 Worklog Time Spent: 10m Work Description: mxm commented on pull request #8585: [BEAM-6829] Use transform name for metric namespace if none provided URL: https://github.com/apache/beam/pull/8585 In non-portable pipelines, transforms are always scoped by an operator. Metric names do not have to be unique across transforms. In portable pipelines, duplicate metric names inevitably occur if no unique namespace has been set, due to multiple transforms being fused together inside a single operator. This avoids duplicate metric names by using the transform name as the namespace, if none has been provided. It is still possible to provide a custom namespace. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | --- Pre-Commit Tests Status (on master branch) --- |Java | Python | Go | Website --- | --- | --- | --- | --- Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [![Build
[jira] [Work logged] (BEAM-6829) Duplicate metric warnings clutter log
[
https://issues.apache.org/jira/browse/BEAM-6829?focusedWorklogId=322047=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322047
]
ASF GitHub Bot logged work on BEAM-6829:
Author: ASF GitHub Bot
Created on: 02/Oct/19 16:47
Start Date: 02/Oct/19 16:47
Worklog Time Spent: 10m
Work Description: stale[bot] commented on issue #8585: [BEAM-6829] Use
transform name for metric namespace if none provided
URL: https://github.com/apache/beam/pull/8585#issuecomment-537581141
This pull request is no longer marked as stale.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 322047)
Time Spent: 2h (was: 1h 50m)
> Duplicate metric warnings clutter log
> -
>
> Key: BEAM-6829
> URL: https://issues.apache.org/jira/browse/BEAM-6829
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.11.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Labels: portability
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Logs fill up quickly with these warnings:
> {code:java}
> WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
> contains a Metric with the name ...{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-6829) Duplicate metric warnings clutter log
[
https://issues.apache.org/jira/browse/BEAM-6829?focusedWorklogId=292508=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292508
]
ASF GitHub Bot logged work on BEAM-6829:
Author: ASF GitHub Bot
Created on: 10/Aug/19 11:12
Start Date: 10/Aug/19 11:12
Worklog Time Spent: 10m
Work Description: stale[bot] commented on issue #8585: [BEAM-6829] Use
transform name for metric namespace if none provided
URL: https://github.com/apache/beam/pull/8585#issuecomment-520140584
This pull request has been closed due to lack of activity. If you think that
is incorrect, or the pull request requires review, you can revive the PR at any
time.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 292508)
Time Spent: 1.5h (was: 1h 20m)
> Duplicate metric warnings clutter log
> -
>
> Key: BEAM-6829
> URL: https://issues.apache.org/jira/browse/BEAM-6829
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.11.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Labels: portability
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Logs fill up quickly with these warnings:
> {code:java}
> WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
> contains a Metric with the name ...{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Work logged] (BEAM-6829) Duplicate metric warnings clutter log
[
https://issues.apache.org/jira/browse/BEAM-6829?focusedWorklogId=292509=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-292509
]
ASF GitHub Bot logged work on BEAM-6829:
Author: ASF GitHub Bot
Created on: 10/Aug/19 11:12
Start Date: 10/Aug/19 11:12
Worklog Time Spent: 10m
Work Description: stale[bot] commented on pull request #8585: [BEAM-6829]
Use transform name for metric namespace if none provided
URL: https://github.com/apache/beam/pull/8585
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 292509)
Time Spent: 1h 40m (was: 1.5h)
> Duplicate metric warnings clutter log
> -
>
> Key: BEAM-6829
> URL: https://issues.apache.org/jira/browse/BEAM-6829
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.11.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Labels: portability
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Logs fill up quickly with these warnings:
> {code:java}
> WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
> contains a Metric with the name ...{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Work logged] (BEAM-6829) Duplicate metric warnings clutter log
[
https://issues.apache.org/jira/browse/BEAM-6829?focusedWorklogId=288428=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-288428
]
ASF GitHub Bot logged work on BEAM-6829:
Author: ASF GitHub Bot
Created on: 03/Aug/19 10:23
Start Date: 03/Aug/19 10:23
Worklog Time Spent: 10m
Work Description: stale[bot] commented on issue #8585: [BEAM-6829] Use
transform name for metric namespace if none provided
URL: https://github.com/apache/beam/pull/8585#issuecomment-517914070
This pull request has been marked as stale due to 60 days of inactivity. It
will be closed in 1 week if no further activity occurs. If you think that’s
incorrect or this pull request requires a review, please simply write any
comment. If closed, you can revive the PR at any time and @mention a reviewer
or discuss it on the d...@beam.apache.org list. Thank you for your
contributions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 288428)
Time Spent: 1h 20m (was: 1h 10m)
> Duplicate metric warnings clutter log
> -
>
> Key: BEAM-6829
> URL: https://issues.apache.org/jira/browse/BEAM-6829
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.11.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Labels: portability
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Logs fill up quickly with these warnings:
> {code:java}
> WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
> contains a Metric with the name ...{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Work logged] (BEAM-6829) Duplicate metric warnings clutter log
[
https://issues.apache.org/jira/browse/BEAM-6829?focusedWorklogId=253657=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253657
]
ASF GitHub Bot logged work on BEAM-6829:
Author: ASF GitHub Bot
Created on: 04/Jun/19 10:01
Start Date: 04/Jun/19 10:01
Worklog Time Spent: 10m
Work Description: mxm commented on issue #8585: [BEAM-6829] Use transform
name for metric namespace if none provided
URL: https://github.com/apache/beam/pull/8585#issuecomment-498606551
@tweise You should be seeing only a fraction of the previous warnings. The
reason why you see some of the warnings is that there is no "PTRANSFORM" set
for the metric. In this case it looks like we do not have enough information to
de-duplicate metrics. This could also be an issue with how the Beam metric
label is populated.
> Instead, shouldn't the transform name be part of the scope?
The metric group is the best way to define the scope of a metric as part of
an ExecutableStage operator which serves metrics for multiple transforms
running inside an SDK Harness. We cannot make the transform name part of the
operator scope because it hosts multiple transforms with overlapping metric
names.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 253657)
Time Spent: 1h 10m (was: 1h)
> Duplicate metric warnings clutter log
> -
>
> Key: BEAM-6829
> URL: https://issues.apache.org/jira/browse/BEAM-6829
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.11.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Labels: portability
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Logs fill up quickly with these warnings:
> {code:java}
> WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
> contains a Metric with the name ...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6829) Duplicate metric warnings clutter log
[
https://issues.apache.org/jira/browse/BEAM-6829?focusedWorklogId=251796=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251796
]
ASF GitHub Bot logged work on BEAM-6829:
Author: ASF GitHub Bot
Created on: 31/May/19 14:06
Start Date: 31/May/19 14:06
Worklog Time Spent: 10m
Work Description: tweise commented on issue #8585: [BEAM-6829] Use
transform name for metric namespace if none provided
URL: https://github.com/apache/beam/pull/8585#issuecomment-497721088
Looks as if the change applies to some metrics (in the log snipped
`pardo_execution_time`, but not `element_count`). But I believe we need to keep
the metric name intact and augment the scope.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251796)
Time Spent: 1h (was: 50m)
> Duplicate metric warnings clutter log
> -
>
> Key: BEAM-6829
> URL: https://issues.apache.org/jira/browse/BEAM-6829
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.11.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Labels: portability
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Logs fill up quickly with these warnings:
> {code:java}
> WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
> contains a Metric with the name ...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6829) Duplicate metric warnings clutter log
[
https://issues.apache.org/jira/browse/BEAM-6829?focusedWorklogId=251792=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251792
]
ASF GitHub Bot logged work on BEAM-6829:
Author: ASF GitHub Bot
Created on: 31/May/19 13:57
Start Date: 31/May/19 13:57
Worklog Time Spent: 10m
Work Description: tweise commented on issue #8585: [BEAM-6829] Use
transform name for metric namespace if none provided
URL: https://github.com/apache/beam/pull/8585#issuecomment-497718117
With this change, I still see the duplicate metrics warning:
```
[Source: Impulse -> [2]write/Write/WriteImpl/DoOnce/{FlatMap(), Map(decode)} -> [1]write/Write/WriteImpl/InitializeWrite ->
(Map -> ToKeyedWorkItem, Map -> ToKeyedWorkItem, Map -> ToKeyedWorkItem) (1/1)]
WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
contains a Metric with the name 'beam.metric:element_count:v1'. Metric will not
be reported.[localhost, taskmanager, 0bd0adb5-1fba-49c7-811f-c8ae76f83bb8,
BeamApp-tweise-0531134309-499ce3fa,
[2]write/Write/WriteImpl/DoOnce/{FlatMap(),
Map(decode)}, 0]
```
But I also suspect that the proposed fix will have other side effects, since
it is changing the metric name (at least that's how it would appear in our
metric system).
The metrics as logged with the slf4j reporter before this change look like
this:
```
localhost.taskmanager.87150cf7-cc34-412c-9057-656edef0590c.BeamApp-tweise-0531054435-7c2c37e1.[1]write/Write/WriteImpl/FinalizeWrite.0.numRecordsIn:
1
localhost.taskmanager.87150cf7-cc34-412c-9057-656edef0590c.BeamApp-tweise-0531054435-7c2c37e1.[3]{read,
split, pair_with_one}.0.beam.metric:element_count:v1: 1
localhost.taskmanager.87150cf7-cc34-412c-9057-656edef0590c.BeamApp-tweise-0531054435-7c2c37e1.ToKeyedWorkItem.0.numRecordsOut:
0
localhost.taskmanager.87150cf7-cc34-412c-9057-656edef0590c.BeamApp-tweise-0531054435-7c2c37e1.[3]{read,
split,
pair_with_one}.0.ref_AppliedPTransform_read/Read/ReadSplits_16.metric:pardo_execution_time:finish_bundle_msecs:v1:
0
```
In our backend this translates to
`.operator.beam-metric-element_count-v1.gauge.sum`
If we replace `beam` with ``, then we would effectively
change the metric name. Instead, shouldn't the transform name be part of the
scope?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251792)
Time Spent: 50m (was: 40m)
> Duplicate metric warnings clutter log
> -
>
> Key: BEAM-6829
> URL: https://issues.apache.org/jira/browse/BEAM-6829
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.11.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Labels: portability
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Logs fill up quickly with these warnings:
> {code:java}
> WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
> contains a Metric with the name ...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6829) Duplicate metric warnings clutter log
[
https://issues.apache.org/jira/browse/BEAM-6829?focusedWorklogId=247608=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247608
]
ASF GitHub Bot logged work on BEAM-6829:
Author: ASF GitHub Bot
Created on: 23/May/19 18:22
Start Date: 23/May/19 18:22
Worklog Time Spent: 10m
Work Description: tweise commented on issue #8585: [BEAM-6829] Use
transform name for metric namespace if none provided
URL: https://github.com/apache/beam/pull/8585#issuecomment-495330368
Unfortunately not yet. This will take a bit more time to verify the correct
propagation of the metrics to our backend.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 247608)
Time Spent: 40m (was: 0.5h)
> Duplicate metric warnings clutter log
> -
>
> Key: BEAM-6829
> URL: https://issues.apache.org/jira/browse/BEAM-6829
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.11.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Labels: portability
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Logs fill up quickly with these warnings:
> {code:java}
> WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
> contains a Metric with the name ...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6829) Duplicate metric warnings clutter log
[
https://issues.apache.org/jira/browse/BEAM-6829?focusedWorklogId=247476=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-247476
]
ASF GitHub Bot logged work on BEAM-6829:
Author: ASF GitHub Bot
Created on: 23/May/19 15:31
Start Date: 23/May/19 15:31
Worklog Time Spent: 10m
Work Description: mxm commented on issue #8585: [BEAM-6829] Use transform
name for metric namespace if none provided
URL: https://github.com/apache/beam/pull/8585#issuecomment-495268808
@tweise Have you had a chance to take a look?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 247476)
Time Spent: 0.5h (was: 20m)
> Duplicate metric warnings clutter log
> -
>
> Key: BEAM-6829
> URL: https://issues.apache.org/jira/browse/BEAM-6829
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.11.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Labels: portability
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Logs fill up quickly with these warnings:
> {code:java}
> WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
> contains a Metric with the name ...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6829) Duplicate metric warnings clutter log
[
https://issues.apache.org/jira/browse/BEAM-6829?focusedWorklogId=243203=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243203
]
ASF GitHub Bot logged work on BEAM-6829:
Author: ASF GitHub Bot
Created on: 16/May/19 09:09
Start Date: 16/May/19 09:09
Worklog Time Spent: 10m
Work Description: mxm commented on issue #8585: [BEAM-6829] Use transform
name for metric namespace if none provided
URL: https://github.com/apache/beam/pull/8585#issuecomment-492985582
@tweise This gets rid of these warnings:
```
WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
contains a Metric with the name ...
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243203)
Time Spent: 20m (was: 10m)
> Duplicate metric warnings clutter log
> -
>
> Key: BEAM-6829
> URL: https://issues.apache.org/jira/browse/BEAM-6829
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.11.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Labels: portability
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Logs fill up quickly with these warnings:
> {code:java}
> WARN org.apache.flink.metrics.MetricGroup - Name collision: Group already
> contains a Metric with the name ...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6829) Duplicate metric warnings clutter log
[ https://issues.apache.org/jira/browse/BEAM-6829?focusedWorklogId=242615=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-242615 ] ASF GitHub Bot logged work on BEAM-6829: Author: ASF GitHub Bot Created on: 15/May/19 16:01 Start Date: 15/May/19 16:01 Worklog Time Spent: 10m Work Description: mxm commented on pull request #8585: [BEAM-6829] Use transform name for metric namespace if none provided URL: https://github.com/apache/beam/pull/8585 In non-portable pipelines, transforms are always scoped by an operator. Metric names do not have to be unique across transforms. In portable pipelines, duplicate metric names inevitably occur if no unique namespace has been set, due to multiple transforms being fused together inside a single operator. This avoids duplicate metric names by using the transform name as the namespace, if none has been provided. It is still possible to provide a custom namespace. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | --- Pre-Commit Tests Status (on master branch) --- |Java | Python | Go | Website --- | --- | --- | --- | --- Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [