True, metrics extraction vs collections does make a difference. Maybe only my understanding / interpretation of what is written in design docs is different.
I personally do not look for a single point to collect all global aggregated metrics as much as having a way of bull beam experience of generating and delivering metrics to a time-series database for monitoring. Aggregated or partly aggregated, I do would not mid. I would be fine with per task manager collector / reporter similar as flink natively does. Right now one must submit metrics in one API (beam) and report them via runner internal API and translate models. Also in flink reports much more dimensions on those metrics which are kind of blurred away by beam model but one have to handle this in reporting. By those I mean things like `operator_id`, `tm_id`, `task_attempt_num`, `subtask_index` etc. But full aggregate sounds good too. But sounds kind of internal to flink how to support job choosing and running single authority to report to [ Full content available at: https://github.com/apache/beam/pull/4548 ] This message was relayed via gitbox.apache.org for [email protected]
