[
https://issues.apache.org/jira/browse/BEAM-9488?focusedWorklogId=426843&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-426843
]
ASF GitHub Bot logged work on BEAM-9488:
----------------------------------------
Author: ASF GitHub Bot
Created on: 24/Apr/20 03:28
Start Date: 24/Apr/20 03:28
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #11514:
URL: https://github.com/apache/beam/pull/11514#discussion_r414265186
##########
File path: sdks/python/apache_beam/runners/worker/bundle_processor.py
##########
@@ -1035,59 +1035,12 @@ def monitoring_infos(self):
# Construct a new dict first to remove duplicates.
all_monitoring_infos_dict = {}
for transform_id, op in self.ops.items():
- for mi in op.monitoring_infos(transform_id).values():
- fixed_mi = self._fix_output_tags_monitoring_info(transform_id, mi)
- all_monitoring_infos_dict[monitoring_infos.to_key(fixed_mi)] = fixed_mi
-
- infos_list = list(all_monitoring_infos_dict.values())
-
- def inject_pcollection(monitoring_info):
- """
- If provided metric is element count metric:
- Finds relevant transform output info in current process_bundle_descriptor
- and adds tag with PCOLLECTION_LABEL and pcollection_id into monitoring
- info.
- """
- if monitoring_info.urn in URNS_NEEDING_PCOLLECTIONS:
- if not monitoring_infos.PTRANSFORM_LABEL in monitoring_info.labels:
- return
- ptransform_label = monitoring_info.labels[
- monitoring_infos.PTRANSFORM_LABEL]
- if not monitoring_infos.TAG_LABEL in monitoring_info.labels:
- return
- tag_label = monitoring_info.labels[monitoring_infos.TAG_LABEL]
-
- if not ptransform_label in self.process_bundle_descriptor.transforms:
- return
Review comment:
yeah
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 426843)
Time Spent: 1h 10m (was: 1h)
> Python SDK sending unexpected MonitoringInfo
> --------------------------------------------
>
> Key: BEAM-9488
> URL: https://issues.apache.org/jira/browse/BEAM-9488
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Ruoyun Huang
> Assignee: Luke Cwik
> Priority: Minor
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> element_count metrics is supposed to be tied with pcollection ids, but by
> inspecting what is sent over by python sdk, we see there are monitoringInfo
> sent wit ptransforms in it.
> [Double checked the job graph, these seem to be redundant. i.e. the
> corresponding pcollection does have its own MonitoringInfo reported.]
> Likely a bug.
> Proof:
> urn: "beam:metric:element_count:v1"
> type: "beam:metrics:sum_int_64"
> metric {
> counter_data {
> int64_value: 1
> }
> }
> labels {
> key: "PTRANSFORM"
> value: "start/MaybeReshuffle/Reshuffle/RemoveRandomKeys-ptransform-85"
> }
> labels {
> key: "TAG"
> value: "None"
> }
> timestamp {
> seconds: 1583949073
> nanos: 842402935
> }
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
