Hi Maximilian,
I have used Beam's Metrics in our pipeline implementation but it's of no
use.
I just want to understand whether beam's metrics system automatically sends
these counters to flink's statsd implementation. FYI I do get flink's
general metrics like taskSlotsTotal, numRunningJobs, etc in my
telegraf->influx stack which means flink is able to communicate with statsd
just fine.
public class KafkaRecordToMessageDoFn extends DoFn<KafkaRecord<byte[],
byte[]>, KV<String, Message>> {
private Counter counter =
Metrics.counter(KafkaRecordToMessageDoFn.class, "my-counter");
@ProcessElement
public void processElement(ProcessContext c) throws Exception {
counter.inc();
}
}
Regards
Kush Sharma
On Fri, Nov 8, 2019 at 4:10 PM Maximilian Michels <m...@apache.org> wrote:
> Hi Kush,
>
> Beam has its own Metrics[1] which are reported via the Flink metric
> system. You may want to use those and utilize the Flink statsd reporter:
>
> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#statsd-orgapacheflinkmetricsstatsdstatsdreporter
>
> Cheers,
> Max
>
> [1]
>
> https://beam.apache.org/releases/javadoc/2.1.0/org/apache/beam/sdk/metrics/Metrics.html
>
> On 08.11.19 11:31, Kush Kumar Sharma wrote:
> > Hi Devs!
> >
> > I am trying to use a *Statsd* client in beam to export some runner
> > metrics. I am able to extract out metrics from the base application but
> > once the job is submitted to its runner(in this case I am using
> > *FlinkRunner*), statsd client stops working. This is a streaming job and
> > I need to extract out those metrics continuously(like how many files are
> > getting processed, etc).
> >
> > Are there any helper classes for this particular use case in beam? If
> > not, are there any flink specific classes that I can use here? I have
> > seen *MetricResults*(from *FlinkRunnerResult*) but it looks like I can
> > only use that once the job is done not when in progress.
> >
> > My current statsd client is
> https://github.com/tim-group/java-statsd-client
> >
> > Regards
> > Kush Sharma
>
