Hi Maximilian, I have used Beam's Metrics in our pipeline implementation but it's of no use. I just want to understand whether beam's metrics system automatically sends these counters to flink's statsd implementation. FYI I do get flink's general metrics like taskSlotsTotal, numRunningJobs, etc in my telegraf->influx stack which means flink is able to communicate with statsd just fine.
public class KafkaRecordToMessageDoFn extends DoFn<KafkaRecord<byte[], byte[]>, KV<String, Message>> { private Counter counter = Metrics.counter(KafkaRecordToMessageDoFn.class, "my-counter"); @ProcessElement public void processElement(ProcessContext c) throws Exception { counter.inc(); } } Regards Kush Sharma On Fri, Nov 8, 2019 at 4:10 PM Maximilian Michels <m...@apache.org> wrote: > Hi Kush, > > Beam has its own Metrics[1] which are reported via the Flink metric > system. You may want to use those and utilize the Flink statsd reporter: > > https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#statsd-orgapacheflinkmetricsstatsdstatsdreporter > > Cheers, > Max > > [1] > > https://beam.apache.org/releases/javadoc/2.1.0/org/apache/beam/sdk/metrics/Metrics.html > > On 08.11.19 11:31, Kush Kumar Sharma wrote: > > Hi Devs! > > > > I am trying to use a *Statsd* client in beam to export some runner > > metrics. I am able to extract out metrics from the base application but > > once the job is submitted to its runner(in this case I am using > > *FlinkRunner*), statsd client stops working. This is a streaming job and > > I need to extract out those metrics continuously(like how many files are > > getting processed, etc). > > > > Are there any helper classes for this particular use case in beam? If > > not, are there any flink specific classes that I can use here? I have > > seen *MetricResults*(from *FlinkRunnerResult*) but it looks like I can > > only use that once the job is done not when in progress. > > > > My current statsd client is > https://github.com/tim-group/java-statsd-client > > > > Regards > > Kush Sharma >