[
https://issues.apache.org/jira/browse/BEAM-7605?focusedWorklogId=266779&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266779
]
ASF GitHub Bot logged work on BEAM-7605:
----------------------------------------
Author: ASF GitHub Bot
Created on: 25/Jun/19 15:16
Start Date: 25/Jun/19 15:16
Worklog Time Spent: 10m
Work Description: lukecwik commented on issue #8913: [BEAM-7605] Allow
user-code to read counters from the dataflow worker
URL: https://github.com/apache/beam/pull/8913#issuecomment-505491076
On Tue, Jun 25, 2019 at 7:25 AM Steven Niemitz <[email protected]>
wrote:
> Thanks for the reply!
>
> you're right, for Spark MetricsPusher thread will be instantiated on the
> Driver machine and Flink MetricsPusher thread will be instantiated on the
> JobManager machine. Indeed it pushes aggregated user metrics every each x
> seconds to a configured sink
>
> Looking at it, it doesn't seem like it'd be a large amount of work to have
> it run inside of the worker process, rather than on the submitter.
>
> That being said, if what you want are non-aggregated system metrics,
> MetricsPusher does not do that currently, it would need to be enhanced.
>
> Agreed. It doesn't seem like a large amount of work though. Even if it is
> aggregating, it'll only be aggregating a single worker anyways, so its
> essentially non-aggregated.
>
> But the problem is that Dataflow Beam runner is a client nutshell that
> just delegates the run of a serialized pipeline to the remote cloud hosted
> Dataflow engine (see BEAM-3926), so it would require to code on the
> Dataflow engine side a MetricsPusher-like service.
>
> The Dataflow worker code was opensourced and lives here:
https://github.com/apache/beam/tree/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker
> That is what this PR is actually adding (kind of). CounterUpdateReceiver
> is called into from the dataflow worker with the counters it has collected
> over that period. This contains all system and user metrics that were
> collected, albeit in a different format. There are (probably?) metrics that
> the windmill service collects (and pushes to the dataflow metrics
> collection service) that this doesn't capture, but its good enough for now,
> and anything is better than nothing here imo.
>
> Here's what I could see going forward:
>
> - "somehow" adapt the dataflow metrics container to a
> MetricsContainerStepMap. It seems like implementing a
> MetricsContainerStepMap that wraps the dataflow metrics (
> pendingDeltaCounters and pendingCumulativeCounters) would be the way
> to go here.
> - figure out how to indicate "where" to push metrics from. It looks
> like dataflow doesn't support MetricsPusher at all, so there's no
backwards
> compatibility problem where, but it might be confusing that some runners
> push aggregated metrics from the submitter, and some non-aggregated from
> the worker. Maybe another flag that indicates aggregated vs
non-aggregated?
>
> Interested to hear your thoughts!
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
>
<https://github.com/apache/beam/pull/8913?email_source=notifications&email_token=ACM4V3DUJ4MBIEI2RNBWRILP4ITG5A5CNFSM4HZTTQW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYQNR6Q#issuecomment-505469178>,
> or mute the thread
>
<https://github.com/notifications/unsubscribe-auth/ACM4V3GN4ZYEZCFQ3XDI77DP4ITG5ANCNFSM4HZTTQWQ>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 266779)
Time Spent: 1h 40m (was: 1.5h)
> Provide a way for user code to read dataflow runner stats
> ---------------------------------------------------------
>
> Key: BEAM-7605
> URL: https://issues.apache.org/jira/browse/BEAM-7605
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Reporter: Steve Niemitz
> Assignee: Steve Niemitz
> Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The dataflow runner collects (and publishes to the dataflow service) a large
> number of useful stats. While these can be polled from the dataflow service
> via its API, there are a few downsides to this:
> * it requires another process to poll and collect the stats
> * the stats are aggregated across all workers, so per-worker stats are lost
> It would be simple to provide a hook to allow users to receive stats updates
> as well, and then do whatever they want with them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
