@JozoVilcek I understand and I agree with the importance of detached job submission. Please know that the beam flink runner (and the other runners) translate beam code into native code but still relies on flink engine to run the native code (of course). For the metrics it relies on native flink accumulators to deal with parallelism and merge. A lot of features of flink are deactivated by flink engine itself in this mode, not only accumulators: In `DetachedEnvironment` class in Flink code (regular Flink, not Beam Flink runner) an exception is thrown when you try to access flink accumulators in detached mode. The message states this: `Job was submitted in detached mode. Results of job execution, such as accumulators, runtime, job id etc. are not available. Please make sure your program doesn't call an eager execution function [collect, print, printToErr, count].` So, AFAIK, there is nothing to bind to in native flink for the beam metrics in detached mode. BTW: the current PR was about agnostic metrics **extraction** not collection.
@aljoscha, @StephanEwen do you have any idea about how we could support beam metrics in Flink in detached mode (if it is possible in the current state of Flink architecture)? [ Full content available at: https://github.com/apache/beam/pull/4548 ] This message was relayed via gitbox.apache.org for [email protected]
