[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008336#comment-16008336 ] ASF GitHub Bot commented on FLINK-6013: --- Github user bowenli86 commented on the issue: https://github.com/apache/flink/pull/3736 Cool! 1.3 here we go! > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008169#comment-16008169 ] ASF GitHub Bot commented on FLINK-6013: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/3736 Merging to 1.3 now. > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16004701#comment-16004701 ] ASF GitHub Bot commented on FLINK-6013: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/3736 Well we already messed with the timings of the process by extending the feature freeze for a week, so I'm going with the next best thing which is the order of events. (Feature freeze -> Code freeze -> RC) But let's cut this short and ask the source of truth, aka the release manager. @rmetzger Is the code-freeze in effect? If not, when will it start? > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16004673#comment-16004673 ] ASF GitHub Bot commented on FLINK-6013: --- Github user greghogan commented on the issue: https://github.com/apache/flink/pull/3736 @zentol yes, and the release release 16 days from now. > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16004639#comment-16004639 ] ASF GitHub Bot commented on FLINK-6013: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/3736 According to https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases the release of an RC happens when code-freeze is in effect. > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16004629#comment-16004629 ] ASF GitHub Bot commented on FLINK-6013: --- Github user greghogan commented on the issue: https://github.com/apache/flink/pull/3736 @zentol, perhaps "feature freeze" is better named "major feature freeze" and next week's "code freeze" as "minor feature freeze". My understanding is that the first cutoff is to prevent half-merged features (with "major" described as requiring multiple PRs), though master has always been quasi-releasable. > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16004222#comment-16004222 ] ASF GitHub Bot commented on FLINK-6013: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/3736 @greghogan My simple answer is that its feature freeze. There has to be some point in time where new features, regardless of size, stability etc. can't make it into a release. My understanding is that we decided to make the feature freeze date this point in time. When something like this is put into place it naturally follows that some contributions that are ready around the time of the FF will not be merged. Given that the FF was already delayed by a week, implicitly giving every contribution a 1 week grace period, I don't see a reason to make an exception here. It wasn't merged before the FF; it wasn't merged before the delayed FF, so it's not going into the release. That's my take on the whole thing. PS: Me closing this PR was not meant to imply "and that's the end of it". I just amended the commit before this discussion started and didn't remove it. > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16004210#comment-16004210 ] Chesnay Schepler commented on FLINK-6013: - 1.4: 54ceec16c11655da4181c0816a3b12d1c4bab465 > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16004193#comment-16004193 ] ASF GitHub Bot commented on FLINK-6013: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3736 > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16003301#comment-16003301 ] ASF GitHub Bot commented on FLINK-6013: --- Github user greghogan commented on the issue: https://github.com/apache/flink/pull/3736 @zentol @StephanEwen why not also merge this into 1.3? There are literally no modified or removed lines of code. > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002606#comment-16002606 ] ASF GitHub Bot commented on FLINK-6013: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/3736 merging. > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001380#comment-16001380 ] ASF GitHub Bot commented on FLINK-6013: --- Github user bowenli86 commented on the issue: https://github.com/apache/flink/pull/3736 Sure. BTW, that reminds me of one thing - I need to update the project version in my code from 1.3-SNAPSHOT to 1.4-SNAPSHOT. > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001362#comment-16001362 ] ASF GitHub Bot commented on FLINK-6013: --- Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3736 @bowenli86 Th good thing is that should be able to use the reporter version 1.4-SNAPSHOT also with a Flink 1.3 release, because the reporter API is quite stable by now. > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001359#comment-16001359 ] ASF GitHub Bot commented on FLINK-6013: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/3736 I can merge it sometime this week, once i found a few more things to merge alongside it. > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001346#comment-16001346 ] ASF GitHub Bot commented on FLINK-6013: --- Github user bowenli86 commented on the issue: https://github.com/apache/flink/pull/3736 I see - by looking at the code, I saw Flink has switched to version 1.4-SNAPSHOT 12h ago. Getting it into 1.4 is fine, our company has a open ticket for getting this code into Flink, and I want to get it done. We haven't seen the issue you described on our side yet. I'll keep an eye on it if it comes up. What's next for this PR? Anyone from Data Artisan will take a look at this PR, and merge it in? > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001311#comment-16001311 ] ASF GitHub Bot commented on FLINK-6013: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/3736 This won't make it into 1.3. For one the feature freeze is in effect since today, so it is too late to merge it now. That said it may be possible to merge it for 1.4 and maintain a separate repository for a 1.3 version that we can also link in the docs. But I'm not entirely sure. I tried it out this morning, and it worked very well. There was an odd thing where the host field in the metric identifier wasn't shown for the taskmanager metrics but for the jobmanager, but that may hidden by Datadog automatically when the hosts match. (i.e. the first part of the metric identifier) > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000225#comment-16000225 ] ASF GitHub Bot commented on FLINK-6013: --- Github user bowenli86 commented on the issue: https://github.com/apache/flink/pull/3736 @zentol cool! Let me know how to get this into 1.3 :) > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1673#comment-1673 ] ASF GitHub Bot commented on FLINK-6013: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/3736 No, I think we addressed all issues :) > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997202#comment-15997202 ] ASF GitHub Bot commented on FLINK-6013: --- Github user bowenli86 commented on the issue: https://github.com/apache/flink/pull/3736 @zentol do you have any other concerns? > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995194#comment-15995194
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user bowenli86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r114595396
--- Diff:
flink-metrics/flink-metrics-datadog/src/test/java/org/apache/flink/metrics/datadog/DatadogHttpClientTest.java
---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.datadog;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Meter;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
+import org.junit.runner.RunWith;
+import org.powermock.api.mockito.PowerMockito;
+import org.powermock.core.classloader.annotations.PrepareForTest;
+import org.powermock.modules.junit4.PowerMockRunner;
+
+import java.util.Arrays;
+import java.util.List;
+
+import static org.junit.Assert.assertEquals;
+
+@RunWith(Enclosed.class)
+public class DatadogHttpClientTest {
+ public static class TestApiKey {
+ @Test(expected = IllegalArgumentException.class)
+ public void testValidateApiKey() {
+ new DatadogHttpClient("fake_key");
--- End diff --
Make sense. There're lots of uncertainty why testing integration with 3rd
party - you don't want to depend to much on it, but you have to test it
somehow. It's hard to find a balance somewhere in between.
I'll make changes as you recommended.
> Add Datadog HTTP metrics reporter
> -
>
> Key: FLINK-6013
> URL: https://issues.apache.org/jira/browse/FLINK-6013
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Critical
> Fix For: 1.3.0
>
>
> We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a
> lot other companies also do.
> Flink right now only has a StatsD metrics reporter, and users have to set up
> Datadog Agent in order to receive metrics from StatsD and transport them to
> Datadog. We don't like this approach.
> We prefer to have a Datadog metrics reporter directly contacting Datadog http
> endpoint.
> I'll take this ticket myself.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994524#comment-15994524
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r114501062
--- Diff:
flink-metrics/flink-metrics-datadog/src/test/java/org/apache/flink/metrics/datadog/DatadogHttpClientTest.java
---
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.datadog;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Meter;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
+import org.junit.runner.RunWith;
+import org.powermock.api.mockito.PowerMockito;
+import org.powermock.core.classloader.annotations.PrepareForTest;
+import org.powermock.modules.junit4.PowerMockRunner;
+
+import java.util.Arrays;
+import java.util.List;
+
+import static org.junit.Assert.assertEquals;
+
+@RunWith(Enclosed.class)
+public class DatadogHttpClientTest {
+ public static class TestApiKey {
+ @Test(expected = IllegalArgumentException.class)
+ public void testValidateApiKey() {
+ new DatadogHttpClient("fake_key");
--- End diff --
This always sends a request to the Datadog servers correct? I'm a bit
skeptical since this means if the servers are down we end up with a failing
test. In general though, throwing out requests at them even though we know they
will fail anyway doesn't seem ehhh nice?
Just checking that it fails when supplying null or an empty string is
enough.
> Add Datadog HTTP metrics reporter
> -
>
> Key: FLINK-6013
> URL: https://issues.apache.org/jira/browse/FLINK-6013
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Critical
> Fix For: 1.3.0
>
>
> We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a
> lot other companies also do.
> Flink right now only has a StatsD metrics reporter, and users have to set up
> Datadog Agent in order to receive metrics from StatsD and transport them to
> Datadog. We don't like this approach.
> We prefer to have a Datadog metrics reporter directly contacting Datadog http
> endpoint.
> I'll take this ticket myself.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989382#comment-15989382 ] ASF GitHub Bot commented on FLINK-6013: --- Github user bowenli86 commented on the issue: https://github.com/apache/flink/pull/3736 @StephanEwen @zentol I've made the following changes: 1) Datadog (DD) itself has a bug of being unstable when users filter metrics by 'host' in dashboards if 'host' is sent as tags, details in [FLINK-6013](https://issues.apache.org/jira/browse/FLINK-6013). Communicated with DD engineers, they fixed the issue, and now, as advised by them, I'm making 'host' a separate field in metric serialization. Tested the new approach successfully, and we are able to filter metrics by 'host' consistently correctly on DD dashboard. 2) Added more unit tests for the above change. Unit tests cover both cases when 'host' is present in MetricGroup and when it isn't 3) Found Maven can only recognize and run unit tests when file name ends with 'Test' rather than 'Tests'. So I renamed my test file to 'DatadogHttpClientTest' from 'DatadogHttpClientTests' 4) Upgraded okhttp and okio to newer versions 5) Found mocking system millisec will impact the new okhttp. So I separated unit tests to two enclosed test sets in 'DatadogHttpClientTest' Having successfully tested this across my company in the past couple days, I'm now pretty confident this is production ready. I agree on @zentol 's proposal if you guys are currently busy with other things, as long as we make sure this ends up in 1.3 :) Thanks, guys! > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989181#comment-15989181 ] Bowen Li commented on FLINK-6013: - [~StephanEwen] [~Zentol] While testing the reporter, I found Datadog is unstable in filtering metrics by host names. Contacted Datadog engineering team. They got back to me this morning saying they found the issue, and instead of putting host names as tags, Datadog opens a metric format with extra field of ‘host’. Example. Rather than "metric='my.series2', points=150, tags=["host:localhost, version:1"])", they changed to "metric='my.series', points=100, host="localhost", tags=["version:1"]" I'll make corresponding changes to my reporter > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15987100#comment-15987100 ] ASF GitHub Bot commented on FLINK-6013: --- Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3736 @bowenli86 Can you build and test this? Then I would be +1 to merge it to 1.3 It would not be committer tested, but contributor tested, and it is an optional component that is not part of Flink's core functionality, so less critical. > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15987062#comment-15987062 ] ASF GitHub Bot commented on FLINK-6013: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/3736 Sorry, I'm rather busy at the moment myself due to the feature freeze and may not be able to try this out in time. To be honest, given that if we merge it for 1.4-SNAPSHOT the jar can be easily ported to 1.3 (just replace the versions in the pom.xml and rebuild it) I'm prioritizing this less than other changes that don't have this luxury. > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15987010#comment-15987010 ] ASF GitHub Bot commented on FLINK-6013: --- Github user bowenli86 commented on the issue: https://github.com/apache/flink/pull/3736 @StephanEwen @zentol Hi guys, I just tested it on my machine and it works as expected. What's your plan on this PR? I found on email thread that there'll be a feature freeze of release 1.3 on May 1st, and I'd really desire to get this into 1.3 :) > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985247#comment-15985247
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r113519219
--- Diff: flink-metrics/flink-metrics-datadog/pom.xml ---
@@ -0,0 +1,109 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+ org.apache.flink
+ flink-metrics
+ 1.3-SNAPSHOT
+ ..
+
+
+ flink-metrics-datadog
+ flink-metrics-datadog
+
+
+
+ org.apache.flink
+ flink-metrics-core
+ ${project.version}
+ provided
+
+
+
+ com.fasterxml.jackson.core
+ jackson-databind
+ provided
+
+
+
+ com.squareup.okhttp3
+ okhttp
+ 3.6.0
+
+
+
+ com.squareup.okio
+ okio
+ 1.11.0
+
+
+
+
+
+
+ org.apache.flink
+ flink-runtime_2.10
+ ${project.version}
+ test
+
+
+
+ org.apache.flink
+ flink-test-utils-junit
+ ${project.version}
+ test
+
+
+
+
+
+
+ org.apache.maven.plugins
+ maven-shade-plugin
+
+
+ package
+
+ shade
+
+
+
jar-with-dependencies
--- End diff --
yes.
> Add Datadog HTTP metrics reporter
> -
>
> Key: FLINK-6013
> URL: https://issues.apache.org/jira/browse/FLINK-6013
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Critical
> Fix For: 1.3.0
>
>
> We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a
> lot other companies also do.
> Flink right now only has a StatsD metrics reporter, and users have to set up
> Datadog Agent in order to receive metrics from StatsD and transport them to
> Datadog. We don't like this approach.
> We prefer to have a Datadog metrics reporter directly contacting Datadog http
> endpoint.
> I'll take this ticket myself.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985190#comment-15985190
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user bowenli86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r113512176
--- Diff:
flink-metrics/flink-metrics-datadog/src/main/java/org/apache/flink/metrics/datadog/DatadogHttpReporter.java
---
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.datadog;
+
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.metrics.reporter.Scheduled;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+
+/**
+ * Metric Reporter for Datadog
+ *
+ * Variables in metrics scope will be sent to Datadog as tags
+ * */
+public class DatadogHttpReporter implements MetricReporter, Scheduled {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(DatadogHttpReporter.class);
+
+ // Both Flink's Gauge and Meter values are taken as gauge in Datadog
+ private final Map gauges = new ConcurrentHashMap<>();
+ private final Map counters = new
ConcurrentHashMap<>();
+ private final Map meters = new ConcurrentHashMap<>();
+
+ private DatadogHttpClient client;
+ private List configTags;
+
+ public static final String API_KEY = "apikey";
--- End diff --
I wouldn't worry too much about since 1) the current code works well, and
changing it doesn't provide extra readability 2) it requires dependency on
flink-core, which is completely unnecessary just because of a config
> Add Datadog HTTP metrics reporter
> -
>
> Key: FLINK-6013
> URL: https://issues.apache.org/jira/browse/FLINK-6013
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Critical
> Fix For: 1.3.0
>
>
> We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a
> lot other companies also do.
> Flink right now only has a StatsD metrics reporter, and users have to set up
> Datadog Agent in order to receive metrics from StatsD and transport them to
> Datadog. We don't like this approach.
> We prefer to have a Datadog metrics reporter directly contacting Datadog http
> endpoint.
> I'll take this ticket myself.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985159#comment-15985159
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user bowenli86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r113510125
--- Diff: flink-metrics/flink-metrics-datadog/pom.xml ---
@@ -0,0 +1,109 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+ org.apache.flink
+ flink-metrics
+ 1.3-SNAPSHOT
+ ..
+
+
+ flink-metrics-datadog
+ flink-metrics-datadog
+
+
+
+ org.apache.flink
+ flink-metrics-core
+ ${project.version}
+ provided
+
+
+
+ com.fasterxml.jackson.core
+ jackson-databind
+ provided
+
+
+
+ com.squareup.okhttp3
+ okhttp
+ 3.6.0
+
+
+
+ com.squareup.okio
+ okio
+ 1.11.0
+
+
+
+
+
+
+ org.apache.flink
+ flink-runtime_2.10
+ ${project.version}
+ test
+
+
+
+ org.apache.flink
+ flink-test-utils-junit
+ ${project.version}
+ test
+
+
+
+
+
+
+ org.apache.maven.plugins
+ maven-shade-plugin
+
+
+ package
+
+ shade
+
+
+
jar-with-dependencies
--- End diff --
After removing this line, the shaded jar will be named
'flink-metrics-datadog-1.3-SNAPSHOT-shaded.jar'. Is it what you want?
> Add Datadog HTTP metrics reporter
> -
>
> Key: FLINK-6013
> URL: https://issues.apache.org/jira/browse/FLINK-6013
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Critical
> Fix For: 1.3.0
>
>
> We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a
> lot other companies also do.
> Flink right now only has a StatsD metrics reporter, and users have to set up
> Datadog Agent in order to receive metrics from StatsD and transport them to
> Datadog. We don't like this approach.
> We prefer to have a Datadog metrics reporter directly contacting Datadog http
> endpoint.
> I'll take this ticket myself.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984447#comment-15984447
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r113405541
--- Diff:
flink-metrics/flink-metrics-datadog/src/main/java/org/apache/flink/metrics/datadog/DatadogHttpReporter.java
---
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.datadog;
+
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.metrics.reporter.Scheduled;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+
+/**
+ * Metric Reporter for Datadog
+ *
+ * Variables in metrics scope will be sent to Datadog as tags
+ * */
+public class DatadogHttpReporter implements MetricReporter, Scheduled {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(DatadogHttpReporter.class);
+
+ // Both Flink's Gauge and Meter values are taken as gauge in Datadog
+ private final Map gauges = new ConcurrentHashMap<>();
+ private final Map counters = new
ConcurrentHashMap<>();
+ private final Map meters = new ConcurrentHashMap<>();
+
+ private DatadogHttpClient client;
+ private List configTags;
+
+ public static final String API_KEY = "apikey";
--- End diff --
You could define these as a `ConfigOption` which would make it more obvious
what the default value is.
> Add Datadog HTTP metrics reporter
> -
>
> Key: FLINK-6013
> URL: https://issues.apache.org/jira/browse/FLINK-6013
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Critical
> Fix For: 1.3.0
>
>
> We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a
> lot other companies also do.
> Flink right now only has a StatsD metrics reporter, and users have to set up
> Datadog Agent in order to receive metrics from StatsD and transport them to
> Datadog. We don't like this approach.
> We prefer to have a Datadog metrics reporter directly contacting Datadog http
> endpoint.
> I'll take this ticket myself.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984448#comment-15984448
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r113405034
--- Diff: flink-metrics/flink-metrics-datadog/pom.xml ---
@@ -0,0 +1,109 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+ org.apache.flink
+ flink-metrics
+ 1.3-SNAPSHOT
+ ..
+
+
+ flink-metrics-datadog
+ flink-metrics-datadog
+
+
+
+ org.apache.flink
+ flink-metrics-core
+ ${project.version}
+ provided
+
+
+
+ com.fasterxml.jackson.core
+ jackson-databind
+ provided
+
+
+
+ com.squareup.okhttp3
+ okhttp
+ 3.6.0
+
+
+
+ com.squareup.okio
+ okio
+ 1.11.0
+
+
+
+
+
+
+ org.apache.flink
+ flink-runtime_2.10
+ ${project.version}
+ test
+
+
+
+ org.apache.flink
+ flink-test-utils-junit
+ ${project.version}
+ test
+
+
+
+
+
+
+ org.apache.maven.plugins
+ maven-shade-plugin
+
+
+ package
+
+ shade
+
+
+
jar-with-dependencies
--- End diff --
I would remove this and adjust the `opt.xml` again; this makes it more
consistent with other shaded jars that we produce.
> Add Datadog HTTP metrics reporter
> -
>
> Key: FLINK-6013
> URL: https://issues.apache.org/jira/browse/FLINK-6013
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Critical
> Fix For: 1.3.0
>
>
> We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a
> lot other companies also do.
> Flink right now only has a StatsD metrics reporter, and users have to set up
> Datadog Agent in order to receive metrics from StatsD and transport them to
> Datadog. We don't like this approach.
> We prefer to have a Datadog metrics reporter directly contacting Datadog http
> endpoint.
> I'll take this ticket myself.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984387#comment-15984387 ] ASF GitHub Bot commented on FLINK-6013: --- Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3736 +1 from my side @zentol any further comments/concerns? > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983655#comment-15983655 ] ASF GitHub Bot commented on FLINK-6013: --- Github user bowenli86 commented on the issue: https://github.com/apache/flink/pull/3736 @StephanEwen @zentol I shaded okhttp3 and okio from flink-metrics-datadog. I didn't use 'shade-flink' because I found it somehow prevents me from building a uber jar. Let me know if it's ok to shade them this way. ``` $ jar -tf target/flink-metrics-datadog-1.3-SNAPSHOT-jar-with-dependencies.jar META-INF/ META-INF/MANIFEST.MF META-INF/DEPENDENCIES META-INF/LICENSE META-INF/NOTICE org/ org/apache/ org/apache/flink/ org/apache/flink/metrics/ org/apache/flink/metrics/datadog/ org/apache/flink/metrics/datadog/DatadogHttpClient.class org/apache/flink/metrics/datadog/DatadogHttpReporter$DatadogHttpRequest.class org/apache/flink/metrics/datadog/DatadogHttpReporter.class org/apache/flink/metrics/datadog/DCounter.class org/apache/flink/metrics/datadog/DGauge.class org/apache/flink/metrics/datadog/DMeter.class org/apache/flink/metrics/datadog/DMetric.class org/apache/flink/metrics/datadog/DSeries.class org/apache/flink/metrics/datadog/MetricType.class META-INF/maven/ META-INF/maven/org.apache.flink/ META-INF/maven/org.apache.flink/flink-metrics-datadog/ META-INF/maven/org.apache.flink/flink-metrics-datadog/pom.xml META-INF/maven/org.apache.flink/flink-metrics-datadog/pom.properties META-INF/maven/org.apache.flink/force-shading/ META-INF/maven/org.apache.flink/force-shading/pom.xml META-INF/maven/org.apache.flink/force-shading/pom.properties org/apache/flink/shaded/ org/apache/flink/shaded/okhttp3/ org/apache/flink/shaded/okhttp3/Address.class . org/apache/flink/shaded/okhttp3/WebSocketListener.class META-INF/maven/com.squareup.okhttp3/ META-INF/maven/com.squareup.okhttp3/okhttp/ META-INF/maven/com.squareup.okhttp3/okhttp/pom.xml META-INF/maven/com.squareup.okhttp3/okhttp/pom.properties org/apache/flink/shaded/okio/ org/apache/flink/shaded/okio/AsyncTimeout$1.class ... org/apache/flink/shaded/okio/Util.class META-INF/maven/com.squareup.okio/ META-INF/maven/com.squareup.okio/okio/ META-INF/maven/com.squareup.okio/okio/pom.xml META-INF/maven/com.squareup.okio/okio/pom.properties ``` > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983370#comment-15983370 ] ASF GitHub Bot commented on FLINK-6013: --- Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3736 I think this is starting to look very good! Given that this introduces new libraries as dependencies (okhttp, okio), should we pro-actively shade those away to avoid dependency conflicts? Admittedly, it would only impact users that explicitly drop in the datadog reporter, but it might still be nice for those users. Given that we build a jr-with-dependencies anyways, the step to shading is small... > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983292#comment-15983292
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user bowenli86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r113259225
--- Diff: flink-dist/src/main/assemblies/opt.xml ---
@@ -95,5 +95,12 @@
flink-metrics-statsd-${project.version}.jar
0644
+
+
+
../flink-metrics/flink-metrics-datadog/target/flink-metrics-datadog-${project.version}.jar
--- End diff --
Thank you for clarifying!
> Add Datadog HTTP metrics reporter
> -
>
> Key: FLINK-6013
> URL: https://issues.apache.org/jira/browse/FLINK-6013
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Critical
> Fix For: 1.3.0
>
>
> We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a
> lot other companies also do.
> Flink right now only has a StatsD metrics reporter, and users have to set up
> Datadog Agent in order to receive metrics from StatsD and transport them to
> Datadog. We don't like this approach.
> We prefer to have a Datadog metrics reporter directly contacting Datadog http
> endpoint.
> I'll take this ticket myself.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982722#comment-15982722
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r113167440
--- Diff: flink-dist/src/main/assemblies/opt.xml ---
@@ -95,5 +95,12 @@
flink-metrics-statsd-${project.version}.jar
0644
+
+
+
../flink-metrics/flink-metrics-datadog/target/flink-metrics-datadog-${project.version}.jar
--- End diff --
The `maven-jar-plugin` creates 2 jars. For the datadog reporter these are,
right now,
1. `flink-metrics-datadog-1.3-SNAPSHOT.jar`
2. `flink-metrics-datadog-1.3-SNAPSHOT-jav-with-dependencies.jar`
The first jar doesn't contain any dependencies, whereas the second one
contains all dependencies declared in the pom.xml, as long as the aren't set to
`provided`. We want the latter to be placed in `/opt`, as such the ``
tag must point to that jar.
Basically replace the existing `` configuration with
`../flink-metrics/flink-metrics-datadog/target/flink-metrics-datadog-${project.version}-jar-with-dependencies.jar`
> Add Datadog HTTP metrics reporter
> -
>
> Key: FLINK-6013
> URL: https://issues.apache.org/jira/browse/FLINK-6013
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Critical
> Fix For: 1.3.0
>
>
> We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a
> lot other companies also do.
> Flink right now only has a StatsD metrics reporter, and users have to set up
> Datadog Agent in order to receive metrics from StatsD and transport them to
> Datadog. We don't like this approach.
> We prefer to have a Datadog metrics reporter directly contacting Datadog http
> endpoint.
> I'll take this ticket myself.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981512#comment-15981512 ] ASF GitHub Bot commented on FLINK-6013: --- Github user bowenli86 commented on the issue: https://github.com/apache/flink/pull/3736 @zentol any further details and suggestions on the maven-jar-plugin part? > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981404#comment-15981404
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user bowenli86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r112985930
--- Diff: flink-dist/src/main/assemblies/opt.xml ---
@@ -95,5 +95,12 @@
flink-metrics-statsd-${project.version}.jar
0644
+
+
+
../flink-metrics/flink-metrics-datadog/target/flink-metrics-datadog-${project.version}.jar
--- End diff --
@zentol any details?
> Add Datadog HTTP metrics reporter
> -
>
> Key: FLINK-6013
> URL: https://issues.apache.org/jira/browse/FLINK-6013
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Critical
> Fix For: 1.3.0
>
>
> We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a
> lot other companies also do.
> Flink right now only has a StatsD metrics reporter, and users have to set up
> Datadog Agent in order to receive metrics from StatsD and transport them to
> Datadog. We don't like this approach.
> We prefer to have a Datadog metrics reporter directly contacting Datadog http
> endpoint.
> I'll take this ticket myself.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979038#comment-15979038
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user bowenli86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r112729232
--- Diff:
flink-metrics/flink-metrics-datadog/src/main/java/org/apache/flink/metrics/datadog/DatadogHttpClient.java
---
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.datadog;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import okhttp3.MediaType;
+import okhttp3.OkHttpClient;
+import okhttp3.Request;
+import okhttp3.Response;
+import okhttp3.RequestBody;
+
+import java.io.IOException;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Http client talking to Datadog
+ * */
+public class DatadogHttpClient{
+ private static final String SERIES_URL_FORMAT =
"https://app.datadoghq.com/api/v1/series?api_key=%s;;
+ private static final String VALIDATE_URL_FORMAT =
"https://app.datadoghq.com/api/v1/validate?api_key=%s;;
+ private static final MediaType MEDIA_TYPE =
MediaType.parse("application/json; charset=utf-8");
+ private static final int TIMEOUT = 3;
+ private static final ObjectMapper MAPPER = new ObjectMapper();
+
+ private final String seriesUrl;
+ private final String validateUrl;
+ private final OkHttpClient client;
+ private final String apiKey;
+
+ public DatadogHttpClient(String dgApiKey) {
+ if(dgApiKey == null || dgApiKey.isEmpty()) {
--- End diff --
done
> Add Datadog HTTP metrics reporter
> -
>
> Key: FLINK-6013
> URL: https://issues.apache.org/jira/browse/FLINK-6013
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Critical
> Fix For: 1.3.0
>
>
> We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a
> lot other companies also do.
> Flink right now only has a StatsD metrics reporter, and users have to set up
> Datadog Agent in order to receive metrics from StatsD and transport them to
> Datadog. We don't like this approach.
> We prefer to have a Datadog metrics reporter directly contacting Datadog http
> endpoint.
> I'll take this ticket myself.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979034#comment-15979034
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user bowenli86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r112729036
--- Diff: flink-dist/src/main/assemblies/opt.xml ---
@@ -95,5 +95,12 @@
flink-metrics-statsd-${project.version}.jar
0644
+
+
+
../flink-metrics/flink-metrics-datadog/target/flink-metrics-datadog-${project.version}.jar
--- End diff --
Can you elaborate more, and give me more guidance on how to do it
correctly?
I copied this piece of code, and saw it exists in
flink-metrics-dropwizard/pom.xml, flink-metrics-ganglia/pom.xml, and
flink-metrics-graphite/pom.xml.
> Add Datadog HTTP metrics reporter
> -
>
> Key: FLINK-6013
> URL: https://issues.apache.org/jira/browse/FLINK-6013
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Critical
> Fix For: 1.3.0
>
>
> We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a
> lot other companies also do.
> Flink right now only has a StatsD metrics reporter, and users have to set up
> Datadog Agent in order to receive metrics from StatsD and transport them to
> Datadog. We don't like this approach.
> We prefer to have a Datadog metrics reporter directly contacting Datadog http
> endpoint.
> I'll take this ticket myself.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979022#comment-15979022
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user bowenli86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r112728271
--- Diff:
flink-metrics/flink-metrics-datadog/src/main/java/org/apache/flink/metrics/datadog/DMetric.java
---
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.datadog;
+
+import org.apache.flink.metrics.datadog.utils.TimestampUtils;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Abstract metric of Datadog for serialization
+ * */
+public abstract class DMetric {
+ private final String metric; // Metric name
+ private final MetricType type;
+ private final List tags;
+
+ public DMetric(MetricType metricType, String metric, List tags)
{
+ this.type = metricType;
+ this.metric = metric;
+ this.tags = tags;
+ }
+
+ public MetricType getType() {
+ return type;
--- End diff --
In DGauge and DCounter, their fields 'gauge' and 'counter' don't impact the
json serialization.
I commented on DCounter#getMetricValue() and DGauge#getMetricValue()
> Add Datadog HTTP metrics reporter
> -
>
> Key: FLINK-6013
> URL: https://issues.apache.org/jira/browse/FLINK-6013
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Critical
> Fix For: 1.3.0
>
>
> We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a
> lot other companies also do.
> Flink right now only has a StatsD metrics reporter, and users have to set up
> Datadog Agent in order to receive metrics from StatsD and transport them to
> Datadog. We don't like this approach.
> We prefer to have a Datadog metrics reporter directly contacting Datadog http
> endpoint.
> I'll take this ticket myself.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979013#comment-15979013
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user bowenli86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r112727162
--- Diff:
flink-metrics/flink-metrics-datadog/src/main/java/org/apache/flink/metrics/datadog/DatadogHttpReporter.java
---
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.datadog;
+
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.metrics.reporter.Scheduled;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+
+/**
+ * Metric Reporter for Datadog
+ *
+ * Variables in metrics scope will be sent to Datadog as tags
+ * */
+public class DatadogHttpReporter implements MetricReporter, Scheduled {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(DatadogHttpReporter.class);
+
+ // Both Flink's Gauge and Meter values are taken as gauge in Datadog
+ private final Map gauges = new ConcurrentHashMap<>();
+ private final Map counters = new
ConcurrentHashMap<>();
+ private final Map meters = new ConcurrentHashMap<>();
+
+ private DatadogHttpClient client;
+ private List configTags;
+
+ public static final String API_KEY = "apikey";
+ public static final String TAGS = "tags";
+
+ @Override
+ public void notifyOfAddedMetric(Metric metric, String metricName,
MetricGroup group) {
+ final String name = group.getMetricIdentifier(metricName);
+
+ List tags = new ArrayList<>(configTags);
+ tags.addAll(getTagsFromMetricGroup(group));
+
+ if (metric instanceof Counter) {
+ Counter c = (Counter) metric;
+ counters.put(c, new DCounter(c, name, tags));
+ } else if (metric instanceof Gauge) {
+ Gauge g = (Gauge) metric;
+ gauges.put(g, new DGauge(g, name, tags));
+ } else if(metric instanceof Meter) {
+ Meter m = (Meter) metric;
+ // Only consider rate
+ meters.put(m, new DMeter(m, name, tags));
+ } else if (metric instanceof Histogram) {
+ LOGGER.warn("Cannot add {} because Datadog HTTP API
doesn't support Histogram", metricName);
+ } else {
+ LOGGER.warn("Cannot add unknown metric type {}. This
indicates that the reporter " +
+ "does not support this metric type.",
metric.getClass().getName());
+ }
+ }
+
+ @Override
+ public void notifyOfRemovedMetric(Metric metric, String metricName,
MetricGroup group) {
+ if (metric instanceof Counter) {
+ counters.remove(metric);
+ } else if (metric instanceof Gauge) {
+ gauges.remove(metric);
+ } else if (metric instanceof Meter) {
+ meters.remove(metric);
+ } else if (metric instanceof Histogram) {
+ // No Histogram is registered
+ } else {
+ LOGGER.warn("Cannot remove unknown metric type {}. This
indicates that the reporter " +
+
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979012#comment-15979012
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user bowenli86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r112727145
--- Diff:
flink-metrics/flink-metrics-datadog/src/main/java/org/apache/flink/metrics/datadog/DatadogHttpReporter.java
---
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.datadog;
+
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.metrics.reporter.Scheduled;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+
+/**
+ * Metric Reporter for Datadog
+ *
+ * Variables in metrics scope will be sent to Datadog as tags
+ * */
+public class DatadogHttpReporter implements MetricReporter, Scheduled {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(DatadogHttpReporter.class);
+
+ // Both Flink's Gauge and Meter values are taken as gauge in Datadog
+ private final Map gauges = new ConcurrentHashMap<>();
+ private final Map counters = new
ConcurrentHashMap<>();
+ private final Map meters = new ConcurrentHashMap<>();
+
+ private DatadogHttpClient client;
+ private List configTags;
+
+ public static final String API_KEY = "apikey";
+ public static final String TAGS = "tags";
+
+ @Override
+ public void notifyOfAddedMetric(Metric metric, String metricName,
MetricGroup group) {
+ final String name = group.getMetricIdentifier(metricName);
+
+ List tags = new ArrayList<>(configTags);
+ tags.addAll(getTagsFromMetricGroup(group));
+
+ if (metric instanceof Counter) {
+ Counter c = (Counter) metric;
+ counters.put(c, new DCounter(c, name, tags));
+ } else if (metric instanceof Gauge) {
+ Gauge g = (Gauge) metric;
+ gauges.put(g, new DGauge(g, name, tags));
+ } else if(metric instanceof Meter) {
--- End diff --
done
> Add Datadog HTTP metrics reporter
> -
>
> Key: FLINK-6013
> URL: https://issues.apache.org/jira/browse/FLINK-6013
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Critical
> Fix For: 1.3.0
>
>
> We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a
> lot other companies also do.
> Flink right now only has a StatsD metrics reporter, and users have to set up
> Datadog Agent in order to receive metrics from StatsD and transport them to
> Datadog. We don't like this approach.
> We prefer to have a Datadog metrics reporter directly contacting Datadog http
> endpoint.
> I'll take this ticket myself.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979011#comment-15979011
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user bowenli86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r112727098
--- Diff:
flink-metrics/flink-metrics-datadog/src/main/java/org/apache/flink/metrics/datadog/DatadogHttpClient.java
---
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.datadog;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import okhttp3.MediaType;
+import okhttp3.OkHttpClient;
+import okhttp3.Request;
+import okhttp3.Response;
+import okhttp3.RequestBody;
+
+import java.io.IOException;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Http client talking to Datadog
+ * */
+public class DatadogHttpClient{
+ private static final String SERIES_URL_FORMAT =
"https://app.datadoghq.com/api/v1/series?api_key=%s;;
+ private static final String VALIDATE_URL_FORMAT =
"https://app.datadoghq.com/api/v1/validate?api_key=%s;;
+ private static final MediaType MEDIA_TYPE =
MediaType.parse("application/json; charset=utf-8");
+ private static final int TIMEOUT = 3;
+ private static final ObjectMapper MAPPER = new ObjectMapper();
+
+ private final String seriesUrl;
+ private final String validateUrl;
+ private final OkHttpClient client;
+ private final String apiKey;
+
+ public DatadogHttpClient(String dgApiKey) {
+ if(dgApiKey == null || dgApiKey.isEmpty()) {
+ throw new IllegalArgumentException(
--- End diff --
done
> Add Datadog HTTP metrics reporter
> -
>
> Key: FLINK-6013
> URL: https://issues.apache.org/jira/browse/FLINK-6013
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Critical
> Fix For: 1.3.0
>
>
> We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a
> lot other companies also do.
> Flink right now only has a StatsD metrics reporter, and users have to set up
> Datadog Agent in order to receive metrics from StatsD and transport them to
> Datadog. We don't like this approach.
> We prefer to have a Datadog metrics reporter directly contacting Datadog http
> endpoint.
> I'll take this ticket myself.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978120#comment-15978120
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r112613622
--- Diff:
flink-metrics/flink-metrics-datadog/src/main/java/org/apache/flink/metrics/datadog/DatadogHttpClient.java
---
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.datadog;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import okhttp3.MediaType;
+import okhttp3.OkHttpClient;
+import okhttp3.Request;
+import okhttp3.Response;
+import okhttp3.RequestBody;
+
+import java.io.IOException;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Http client talking to Datadog
+ * */
+public class DatadogHttpClient{
+ private static final String SERIES_URL_FORMAT =
"https://app.datadoghq.com/api/v1/series?api_key=%s;;
+ private static final String VALIDATE_URL_FORMAT =
"https://app.datadoghq.com/api/v1/validate?api_key=%s;;
+ private static final MediaType MEDIA_TYPE =
MediaType.parse("application/json; charset=utf-8");
+ private static final int TIMEOUT = 3;
+ private static final ObjectMapper MAPPER = new ObjectMapper();
+
+ private final String seriesUrl;
+ private final String validateUrl;
+ private final OkHttpClient client;
+ private final String apiKey;
+
+ public DatadogHttpClient(String dgApiKey) {
+ if(dgApiKey == null || dgApiKey.isEmpty()) {
--- End diff --
missing space after if
> Add Datadog HTTP metrics reporter
> -
>
> Key: FLINK-6013
> URL: https://issues.apache.org/jira/browse/FLINK-6013
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Critical
> Fix For: 1.3.0
>
>
> We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a
> lot other companies also do.
> Flink right now only has a StatsD metrics reporter, and users have to set up
> Datadog Agent in order to receive metrics from StatsD and transport them to
> Datadog. We don't like this approach.
> We prefer to have a Datadog metrics reporter directly contacting Datadog http
> endpoint.
> I'll take this ticket myself.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978117#comment-15978117
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r112613661
--- Diff:
flink-metrics/flink-metrics-datadog/src/main/java/org/apache/flink/metrics/datadog/DatadogHttpClient.java
---
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.datadog;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import okhttp3.MediaType;
+import okhttp3.OkHttpClient;
+import okhttp3.Request;
+import okhttp3.Response;
+import okhttp3.RequestBody;
+
+import java.io.IOException;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Http client talking to Datadog
+ * */
+public class DatadogHttpClient{
+ private static final String SERIES_URL_FORMAT =
"https://app.datadoghq.com/api/v1/series?api_key=%s;;
+ private static final String VALIDATE_URL_FORMAT =
"https://app.datadoghq.com/api/v1/validate?api_key=%s;;
+ private static final MediaType MEDIA_TYPE =
MediaType.parse("application/json; charset=utf-8");
+ private static final int TIMEOUT = 3;
+ private static final ObjectMapper MAPPER = new ObjectMapper();
+
+ private final String seriesUrl;
+ private final String validateUrl;
+ private final OkHttpClient client;
+ private final String apiKey;
+
+ public DatadogHttpClient(String dgApiKey) {
+ if(dgApiKey == null || dgApiKey.isEmpty()) {
+ throw new IllegalArgumentException(
--- End diff --
please remove the line-break here
> Add Datadog HTTP metrics reporter
> -
>
> Key: FLINK-6013
> URL: https://issues.apache.org/jira/browse/FLINK-6013
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Critical
> Fix For: 1.3.0
>
>
> We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a
> lot other companies also do.
> Flink right now only has a StatsD metrics reporter, and users have to set up
> Datadog Agent in order to receive metrics from StatsD and transport them to
> Datadog. We don't like this approach.
> We prefer to have a Datadog metrics reporter directly contacting Datadog http
> endpoint.
> I'll take this ticket myself.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978118#comment-15978118
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r112614175
--- Diff:
flink-metrics/flink-metrics-datadog/src/main/java/org/apache/flink/metrics/datadog/DatadogHttpReporter.java
---
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.datadog;
+
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.metrics.reporter.Scheduled;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+
+/**
+ * Metric Reporter for Datadog
+ *
+ * Variables in metrics scope will be sent to Datadog as tags
+ * */
+public class DatadogHttpReporter implements MetricReporter, Scheduled {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(DatadogHttpReporter.class);
+
+ // Both Flink's Gauge and Meter values are taken as gauge in Datadog
+ private final Map gauges = new ConcurrentHashMap<>();
+ private final Map counters = new
ConcurrentHashMap<>();
+ private final Map meters = new ConcurrentHashMap<>();
+
+ private DatadogHttpClient client;
+ private List configTags;
+
+ public static final String API_KEY = "apikey";
+ public static final String TAGS = "tags";
+
+ @Override
+ public void notifyOfAddedMetric(Metric metric, String metricName,
MetricGroup group) {
+ final String name = group.getMetricIdentifier(metricName);
+
+ List tags = new ArrayList<>(configTags);
+ tags.addAll(getTagsFromMetricGroup(group));
+
+ if (metric instanceof Counter) {
+ Counter c = (Counter) metric;
+ counters.put(c, new DCounter(c, name, tags));
+ } else if (metric instanceof Gauge) {
+ Gauge g = (Gauge) metric;
+ gauges.put(g, new DGauge(g, name, tags));
+ } else if(metric instanceof Meter) {
--- End diff --
missing space after if
> Add Datadog HTTP metrics reporter
> -
>
> Key: FLINK-6013
> URL: https://issues.apache.org/jira/browse/FLINK-6013
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Critical
> Fix For: 1.3.0
>
>
> We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a
> lot other companies also do.
> Flink right now only has a StatsD metrics reporter, and users have to set up
> Datadog Agent in order to receive metrics from StatsD and transport them to
> Datadog. We don't like this approach.
> We prefer to have a Datadog metrics reporter directly contacting Datadog http
> endpoint.
> I'll take this ticket myself.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978122#comment-15978122
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r112613595
--- Diff:
flink-metrics/flink-metrics-datadog/src/main/java/org/apache/flink/metrics/datadog/DMetric.java
---
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.datadog;
+
+import org.apache.flink.metrics.datadog.utils.TimestampUtils;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Abstract metric of Datadog for serialization
+ * */
+public abstract class DMetric {
+ private final String metric; // Metric name
+ private final MetricType type;
+ private final List tags;
+
+ public DMetric(MetricType metricType, String metric, List tags)
{
+ this.type = metricType;
+ this.metric = metric;
+ this.tags = tags;
+ }
+
+ public MetricType getType() {
+ return type;
--- End diff --
you missed adjusting the DGauge, DCounter and DSeries javadocs.
> Add Datadog HTTP metrics reporter
> -
>
> Key: FLINK-6013
> URL: https://issues.apache.org/jira/browse/FLINK-6013
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Critical
> Fix For: 1.3.0
>
>
> We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a
> lot other companies also do.
> Flink right now only has a StatsD metrics reporter, and users have to set up
> Datadog Agent in order to receive metrics from StatsD and transport them to
> Datadog. We don't like this approach.
> We prefer to have a Datadog metrics reporter directly contacting Datadog http
> endpoint.
> I'll take this ticket myself.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978121#comment-15978121
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r112613477
--- Diff: flink-dist/src/main/assemblies/opt.xml ---
@@ -95,5 +95,12 @@
flink-metrics-statsd-${project.version}.jar
0644
+
+
+
../flink-metrics/flink-metrics-datadog/target/flink-metrics-datadog-${project.version}.jar
--- End diff --
Since you're using the maven-jar-plugin the jar containing the dependencies
has the "-jar-with-dependencies" suffix, like
`flink-metrics-graphite-1.2-SNAPSHOT-jar-with-dependencies.jar`.
> Add Datadog HTTP metrics reporter
> -
>
> Key: FLINK-6013
> URL: https://issues.apache.org/jira/browse/FLINK-6013
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Critical
> Fix For: 1.3.0
>
>
> We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a
> lot other companies also do.
> Flink right now only has a StatsD metrics reporter, and users have to set up
> Datadog Agent in order to receive metrics from StatsD and transport them to
> Datadog. We don't like this approach.
> We prefer to have a Datadog metrics reporter directly contacting Datadog http
> endpoint.
> I'll take this ticket myself.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978119#comment-15978119
]
ASF GitHub Bot commented on FLINK-6013:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3736#discussion_r112613733
--- Diff:
flink-metrics/flink-metrics-datadog/src/main/java/org/apache/flink/metrics/datadog/DatadogHttpReporter.java
---
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.datadog;
+
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.metrics.reporter.Scheduled;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+
+/**
+ * Metric Reporter for Datadog
+ *
+ * Variables in metrics scope will be sent to Datadog as tags
+ * */
+public class DatadogHttpReporter implements MetricReporter, Scheduled {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(DatadogHttpReporter.class);
+
+ // Both Flink's Gauge and Meter values are taken as gauge in Datadog
+ private final Map gauges = new ConcurrentHashMap<>();
+ private final Map counters = new
ConcurrentHashMap<>();
+ private final Map meters = new ConcurrentHashMap<>();
+
+ private DatadogHttpClient client;
+ private List configTags;
+
+ public static final String API_KEY = "apikey";
+ public static final String TAGS = "tags";
+
+ @Override
+ public void notifyOfAddedMetric(Metric metric, String metricName,
MetricGroup group) {
+ final String name = group.getMetricIdentifier(metricName);
+
+ List tags = new ArrayList<>(configTags);
+ tags.addAll(getTagsFromMetricGroup(group));
+
+ if (metric instanceof Counter) {
+ Counter c = (Counter) metric;
+ counters.put(c, new DCounter(c, name, tags));
+ } else if (metric instanceof Gauge) {
+ Gauge g = (Gauge) metric;
+ gauges.put(g, new DGauge(g, name, tags));
+ } else if(metric instanceof Meter) {
+ Meter m = (Meter) metric;
+ // Only consider rate
+ meters.put(m, new DMeter(m, name, tags));
+ } else if (metric instanceof Histogram) {
+ LOGGER.warn("Cannot add {} because Datadog HTTP API
doesn't support Histogram", metricName);
+ } else {
+ LOGGER.warn("Cannot add unknown metric type {}. This
indicates that the reporter " +
+ "does not support this metric type.",
metric.getClass().getName());
+ }
+ }
+
+ @Override
+ public void notifyOfRemovedMetric(Metric metric, String metricName,
MetricGroup group) {
+ if (metric instanceof Counter) {
+ counters.remove(metric);
+ } else if (metric instanceof Gauge) {
+ gauges.remove(metric);
+ } else if (metric instanceof Meter) {
+ meters.remove(metric);
+ } else if (metric instanceof Histogram) {
+ // No Histogram is registered
+ } else {
+ LOGGER.warn("Cannot remove unknown metric type {}. This
indicates that the reporter " +
+
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977755#comment-15977755 ] ASF GitHub Bot commented on FLINK-6013: --- Github user bowenli86 commented on the issue: https://github.com/apache/flink/pull/3736 Green build! @zentol > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977188#comment-15977188 ] ASF GitHub Bot commented on FLINK-6013: --- Github user bowenli86 commented on the issue: https://github.com/apache/flink/pull/3736 Aha! Updated! Apparently Chrome search is case insensitive > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977179#comment-15977179 ] ASF GitHub Bot commented on FLINK-6013: --- Github user bowenli86 commented on the issue: https://github.com/apache/flink/pull/3736 Where is the title that you're referring to? This PR's title at the very top is already "[FLINK-6013][metrics] Add Datadog HTTP metrics reporter" (without double quote) > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[ https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977162#comment-15977162 ] ASF GitHub Bot commented on FLINK-6013: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/3736 Could you change the PR title to "[FLINK-6013][metrics] Add Datadog HTTP metrics reporter"? The comments aren't being mirrored to JIRA. > Add Datadog HTTP metrics reporter > - > > Key: FLINK-6013 > URL: https://issues.apache.org/jira/browse/FLINK-6013 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.3.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Critical > Fix For: 1.3.0 > > > We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a > lot other companies also do. > Flink right now only has a StatsD metrics reporter, and users have to set up > Datadog Agent in order to receive metrics from StatsD and transport them to > Datadog. We don't like this approach. > We prefer to have a Datadog metrics reporter directly contacting Datadog http > endpoint. > I'll take this ticket myself. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6013) Add Datadog HTTP metrics reporter
[
https://issues.apache.org/jira/browse/FLINK-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973388#comment-15973388
]
ASF GitHub Bot commented on FLINK-6013:
---
GitHub user bowenli86 opened a pull request:
https://github.com/apache/flink/pull/3736
[Flink-6013][metrics] Add Datadog HTTP metrics reporter
I'm adding a DatadogHttpReporter for Flink metrics system.
The implementation, including making parameters in Flink's metrics as
Datadog tags, is a best practice based on our long time working experience and
understanding of Datadog. It might be a bit different than how other metrics
reporters work, but it truly helps developers to find and filter metrics
quickly, better categorize metrics, and visualize them on Datadog dashboards,
especially when users (like OfferUp) have a dozen individual Flink clusters.
--
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [x] General
- The pull request references the related JIRA issue ("[FLINK-6013] Add
Datadog HTTP metrics reporter")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [x] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [x] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bowenli86/flink FLINK-6013
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3736.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3736
commit 72037bde640258bde618ddc10b8bd10645bbaf8d
Author: Bowen Li
Date: 2017-04-18T17:27:17Z
[FLINK-6013][metrics] Add Datadog HTTP metrics reporter
commit e8ced6d03eac47150648401566afce6f12ea03d0
Author: Bowen Li
Date: 2017-04-18T17:27:54Z
Merge branch 'master' into FLINK-6013
commit 27ae0584eb79bd1339934c06d4a4266be9264fb2
Author: Bowen Li
Date: 2017-04-18T18:23:10Z
move okhttp dependencies to flink-metrics
commit 4b48f4d32a8b122f35dfd6322174e469ff0a5a89
Author: Bowen Li
Date: 2017-04-18T18:57:11Z
add Apache License file header
commit a9ca61a92e4f0beb37652da361acfdcd50d11523
Author: Bowen Li
Date: 2017-04-18T19:12:24Z
add more code comments
commit cfe2fdf8d7456d657bd35e937e0f7618086af024
Author: Bowen Li
Date: 2017-04-18T19:47:15Z
remove okhttp from flink-metrics
commit 76b54b8ab7fc6eaad8c2bd7d54e79791110d9690
Author: Bowen Li
Date: 2017-04-18T19:53:14Z
add more doc
> Add Datadog HTTP metrics reporter
> -
>
> Key: FLINK-6013
> URL: https://issues.apache.org/jira/browse/FLINK-6013
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.2.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Critical
> Fix For: 1.3.0, 1.2.1
>
>
> We at OfferUp use Datadog a lot for metrics and dashboards, and I believe a
> lot other companies also do.
> Flink right now only has a StatsD metrics reporter, and users have to set up
> Datadog Agent in order to receive metrics from StatsD and transport them to
> Datadog. We don't like this approach.
> We prefer to have a Datadog metrics reporter directly contacting Datadog http
> endpoint.
> I'll take this ticket myself.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
