Hi Padarn for what it's worth I am using DataDog metrics on EMR with Flink
1.7.1 and this here my flink-conf configuration:
- Classification: flink-conf
ConfigurationProperties:
metrics.reporter.dghttp.class:
org.apache.flink.metrics.datadog.DatadogHttpReporter
metrics.reporter.dghttp.apikey: <your-datadog-key>
metrics.reporter.dghttp.tags: 'service:myservice,env:prod,region:us-east-1'
metrics.scope.jm: 'jobmanager'
metrics.scope.jm.job: 'jobmanager'
metrics.scope.operator: 'taskmanager.<job_name>.<operator_name>'
metrics.scope.task: 'taskmanager.<job_name>.<task_name>'
metrics.scope.tm: 'taskmanager'
metrics.scope.tm.job: 'taskmanager'
Configurations: []
On Thu, May 30, 2019 at 6:46 AM Yun Tang <myas...@live.com> wrote:
> Hi Padarn
>
> If you want to verify why no metrics sending out, how about using the
> built-in Slf4j reporter [1] which would record metrics in logs.
> If you could view the metrics after enabled slf4j-reporter, you could then
> compare the configurations.
>
> Best
> Yun Tang
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter
>
> ------------------------------
> *From:* Padarn Wilson <padarn.wil...@grab.com>
> *Sent:* Thursday, May 30, 2019 18:20
> *To:* user
> *Subject:* [External] Flink 1.7.1 on EMR metrics
>
> Hello all,
>
> I am trying to run Flink 1.7.1 on EMR and having some trouble with metric
> reporting.
>
> I was using the DataDogHttpReporter, but have also tried the
> StatsDReporter, but with both was seeing no metrics being collected.
>
> To debug this I implemented my own reporter (based on StatsDReporter) and
> logged the name of the metric being sent:
>
> private void send(final String name, final String value) {
> log.info("STATSD SENDING: ", name, value);
> try {
> String formatted = String.format("%s:%s|g", name, value);
> byte[] data = formatted.getBytes(StandardCharsets.UTF_8);
> socket.send(new DatagramPacket(data, data.length, this.address));
> }
> catch (IOException e) {
> LOG.error("unable to send packet to statsd at '{}:{}'",
> address.getHostName(), address.getPort());
> }
> }
>
>
> This code is certainly reached, because in my log I see a lot of this:
>
> 2019-05-30 10:18:40,351 INFO com.grab.statsd.StatsDReporter
> - STATSD SENDING:
> 2019-05-30 10:18:40,351 INFO com.grab.statsd.StatsDReporter
> - STATSD SENDING:
> 2019-05-30 10:18:40,351 INFO com.grab.statsd.StatsDReporter
> - STATSD SENDING:
> 2019-05-30 10:18:40,351 INFO com.grab.statsd.StatsDReporter
> - STATSD SENDING:
> 2019-05-30 10:18:40,351 INFO com.grab.statsd.StatsDReporter
> - STATSD SENDING:
> 2019-05-30 10:18:40,351 INFO com.grab.statsd.StatsDReporter
> - STATSD SENDING:
> 2019-05-30 10:18:40,352 INFO com.grab.statsd.StatsDReporter
> - STATSD SENDING:
>
> As you can see, the name and value for the metric being reported is empty.
>
>
> And the logs show everything initialized fine with no error:
>
> 2019-05-30 10:18:30,342 INFO
> org.apache.flink.runtime.metrics.MetricRegistryImpl - Configuring
> stsd with {port=8125, host=127.0.0.1, class=com.grab.statsd.StatsDReporter}.
> 2019-05-30 10:18:30,344 INFO com.grab.statsd.StatsDReporter
> - Configured StatsDReporter with {host:127.0.0.1, port:8125}
> 2019-05-30 10:18:30,344 INFO
> org.apache.flink.runtime.metrics.MetricRegistryImpl - Periodically
> reporting metrics in intervals of 10 SECONDS for reporter stsd of type
> com.grab.statsd.StatsDReporter.
>
>
>
>
> Has anyone else tried to work with Flink and metrics on EMR 1.7.1 (latest
> version on EMR). If so, any pointers as to what could be set up incorrectly?
>
>
>
>
> *Grab is hiring. Learn more at https://grab.careers
> <https://grab.careers/>*
>
> By communicating with Grab Inc and/or its subsidiaries, associate
> companies and jointly controlled entities (“Grab Group”), you are deemed to
> have consented to the processing of your personal data as set out in the
> Privacy Notice which can be viewed at https://grab.com/privacy/
>
> This email contains confidential information and is only for the intended
> recipient(s). If you are not the intended recipient(s), please do not
> disseminate, distribute or copy this email Please notify Grab Group
> immediately if you have received this by mistake and delete this email from
> your system. Email transmission cannot be guaranteed to be secure or
> error-free as any information therein could be intercepted, corrupted,
> lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do
> not accept liability for any errors or omissions in the contents of this
> email arises as a result of email transmission. All intellectual property
> rights in this email and attachments therein shall remain vested in Grab
> Group, unless otherwise provided by law.
>
