[
https://issues.apache.org/jira/browse/BEAM-3310?focusedWorklogId=138339&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-138339
]
ASF GitHub Bot logged work on BEAM-3310:
----------------------------------------
Author: ASF GitHub Bot
Created on: 27/Aug/18 10:49
Start Date: 27/Aug/18 10:49
Worklog Time Spent: 10m
Work Description: JozoVilcek commented on issue #4548: [BEAM-3310]
Metrics pusher
URL: https://github.com/apache/beam/pull/4548#issuecomment-416188576
So, I am playing with flink `MetricReporter` (and also with UI) and observe
a bit strange thing. Job reading from kafka should report `elements_read` beam
counter. I can see it in UI if I select source task. But, there are many many
instances. I would expect to see the counter only for one `operator` and its
`subtasks` (I am guessing it relates to parallelism). But, what I observe is
like `elements_read` counter being defined for each operator within that task.
Can this be some sort of double counting or mirror or what is going on? I am
not sure how to deal with this in `MetricReporter` so I do not double count in
time-series database.
P.S.: I should probably be asking this somewhere else, right? :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 138339)
Time Spent: 18h 20m (was: 18h 10m)
> Push metrics to a backend in an runner agnostic way
> ---------------------------------------------------
>
> Key: BEAM-3310
> URL: https://issues.apache.org/jira/browse/BEAM-3310
> Project: Beam
> Issue Type: New Feature
> Components: runner-extensions-metrics, sdk-java-core
> Reporter: Etienne Chauchot
> Assignee: Etienne Chauchot
> Priority: Major
> Time Spent: 18h 20m
> Remaining Estimate: 0h
>
> The idea is to avoid relying on the runners to provide access to the metrics
> (either at the end of the pipeline or while it runs) because they don't have
> all the same capabilities towards metrics (e.g. spark runner configures sinks
> like csv, graphite or in memory sinks using the spark engine conf). The
> target is to push the metrics in the common runner code so that no matter the
> chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML:
> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
> And the design doc:
> https://s.apache.org/runner_independent_metrics_extraction
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
