[
https://issues.apache.org/jira/browse/BEAM-7605?focusedWorklogId=266620&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266620
]
ASF GitHub Bot logged work on BEAM-7605:
----------------------------------------
Author: ASF GitHub Bot
Created on: 25/Jun/19 13:02
Start Date: 25/Jun/19 13:02
Worklog Time Spent: 10m
Work Description: echauchot commented on issue #8913: [BEAM-7605] Allow
user-code to read counters from the dataflow worker
URL: https://github.com/apache/beam/pull/8913#issuecomment-505434594
Indeed the aim is to use MetricsPusher (see my comment on the ticket for the
reasons) thanks @lukecwik for orienting in the right direction ! @steveniemitz
you're right, for Spark MetricsPusher thread will be instantiated on the Driver
machine and Flink MetricsPusher thread will be instantiated on the JobManager
machine. Indeed it pushes aggregated user metrics every each x seconds to a
configured sink. Also, not related to metrics pusher, the runners update native
spark/flink metrics with beam user metrics. See the talk I did at the apachecon
last september
https://docs.google.com/presentation/d/1KM2M1Ddqwozw_1NFvhFFGqBWjpq9cYhaPcMDOUxCagY/edit?usp=sharing.
That being said, if what you want are non-aggregated system metrics,
MetricsPusher does not do that currently, it would need to be enhanced. But the
problem is that Dataflow Beam runner is a client nutshell that just delegates
the run of a serialized pipeline to the remote cloud hosted Dataflow engine
(see BEAM-3926), so it would require to code on the Dataflow engine side a
MetricsPusher-like service (which has not been done yet and this part of the DF
code is not opensource AFAIK see BEAM-3926). Such an impl could push whatever
metrics (system/user, aggregated/not aggregated) to a MetricsSink (see related
class in Beam).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 266620)
Time Spent: 1h 10m (was: 1h)
> Provide a way for user code to read dataflow runner stats
> ---------------------------------------------------------
>
> Key: BEAM-7605
> URL: https://issues.apache.org/jira/browse/BEAM-7605
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Reporter: Steve Niemitz
> Assignee: Steve Niemitz
> Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The dataflow runner collects (and publishes to the dataflow service) a large
> number of useful stats. While these can be polled from the dataflow service
> via its API, there are a few downsides to this:
> * it requires another process to poll and collect the stats
> * the stats are aggregated across all workers, so per-worker stats are lost
> It would be simple to provide a hook to allow users to receive stats updates
> as well, and then do whatever they want with them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
