Lukasz Gajowy created BEAM-6100:
-----------------------------------
Summary: Runtime and total bytes metrics are not collected properly
Key: BEAM-6100
URL: https://issues.apache.org/jira/browse/BEAM-6100
Project: Beam
Issue Type: Sub-task
Components: testing
Reporter: Lukasz Gajowy
Assignee: Lukasz Gajowy
Currently, we collect time (distribution) and bytes (counter) metrics from one
ParDo (called MetricsMonitor) that is put in pipelines in one, arbitrarily
chosen place (usually "in the middle" of pipeline's graph. In some cases,
invalid time (or total bytes count) is registered.
Taking [this|https://github.com/apache/beam/pull/6987#discussion_r231976671]
discussion into account, ideally, we'd like to:
- collect runtime by recording time at the root and sink(s) of the pipeline
- collect total bytes in a separate ParDo that allows deciding what byte
amount do we actually want to collect (now it's coupled to the time-collecting
Monitor which is inconvenient).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
