Hi Phil,No, spark does not support committed metrics either. Only dataflow
supports them, and only in batch mode. All
the other runners except Direct Runner and Dataflow use AccumulatedMetricResult
that throws an Exception if the user
requests committed metrics, see for ex
https://github.com/apache/beam/blob/236d0dd89c210e235ef45be9663ec77267d7b451/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerStepMap.java#L413
Also please take a look at the talk I gave at the ApacheCon in September if you
have more questions regarding metrics :
https://apachecon.dukecon.org/acna/2018/#/scheduledEvent/e22bd89bacbe03a36 .
There is a download link for the slides but
no audio/video link unfortunately.
Best,Etienne
Le vendredi 30 novembre 2018 à 10:10 -0600, Phil Franklin a écrit :
> Hi again, Etienne! You didn’t mention whether spark is reporting committed
> values now, but you also didn’t mention
> opening a ticket concerning the spark output. Am I right in inferring that
> spark does in fact report committed
> values?
> Thanks!-Phil
> On 2018/11/30 13:57:43, Etienne Chauchot <[email protected]> wrote:
> Hi Phil,> Thanks for using MetricsPusher and Beam in general ! > -
> MetricsHttpSink works that way: it filters out
> committed metrics from the json output when committed metrics are not>
> supported. I checked, Flink runner still does
> not support committed metrics. So there should be no committed metrics>
> values in the output json.There might be a
> bug. I'll open a ticket: thx for pointing out ! You tested on flink and
> spark> right? and both output committed
> metrics values right?> - there is no default mechanism to fallback committed
> metrics values on attempted ones> -
> Apache Flink does no make flink Accumulators available in detached mode, so
> indeed, metrics are not available in this>
> mode.> CCing dev list.> Etienne> Le lundi 26 novembre 2018 à 15:57 -0600,
> Phil Franklin a écrit :>
> All of the discussion I’ve seen says that Flink and Spark only provided
> attempted metric values, but when I use>
> MetricsHttpSink and look at the JSON it has both attempted and committed
> values (albeit, both the same for my simple>
> testing). Has the metrics processing been updated recently, and I’m just
> missing the change updates? Or are the>
> committed values being defaulted to the attempted values? >
>
> Also, I’ve seen it mentioned that Flink doesn’t report metrics when in
> detached mode. Is this still the case?>
>
> Thanks for your help!>
