[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-05-12 Thread bowenli86
Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/3736
  
Cool! 1.3 here we go!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-05-12 Thread zentol
Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3736
  
Merging to 1.3 now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-05-10 Thread zentol
Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3736
  
Well we already messed with the timings of the process by extending the 
feature freeze for a week, so I'm going with the next best thing which is the 
order of events. (Feature freeze -> Code freeze -> RC)

But let's cut this short and ask the source of truth, aka the release 
manager.

@rmetzger Is the code-freeze in effect? If not, when will it start?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-05-10 Thread greghogan
Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/3736
  
@zentol yes, and the release release 16 days from now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-05-10 Thread zentol
Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3736
  
According to 
https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases the 
release of an RC happens when code-freeze is in effect.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-05-10 Thread greghogan
Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/3736
  
@zentol, perhaps "feature freeze" is better named "major feature freeze" 
and next week's "code freeze" as "minor feature freeze". My understanding is 
that the first cutoff is to prevent half-merged features (with "major" 
described as requiring multiple PRs), though master has always been 
quasi-releasable.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-05-10 Thread zentol
Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3736
  
@greghogan My simple answer is that its feature freeze. There has to be 
some point in time where new features, regardless of size, stability etc. can't 
make it into a release. My understanding is that we decided to make the feature 
freeze date this point in time. When something like this is put into place it 
naturally follows that some contributions that are ready around the time of the 
FF will not be merged. Given that the FF was already delayed by a week, 
implicitly giving every contribution a 1 week grace period, I don't see a 
reason to make an exception here. It wasn't merged before the FF; it wasn't 
merged before the delayed FF, so it's not going into the release.

That's my take on the whole thing.

PS: Me closing this PR was not meant to imply "and that's the end of it". I 
just amended the commit before this discussion started and didn't remove it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-05-09 Thread greghogan
Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/3736
  
@zentol @StephanEwen why not also merge this into 1.3? There are literally 
no modified or removed lines of code.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-05-09 Thread zentol
Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3736
  
merging.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-05-08 Thread bowenli86
Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/3736
  
Sure.

BTW, that reminds me of one thing - I need to update the project version in 
my code from 1.3-SNAPSHOT to 1.4-SNAPSHOT.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-05-08 Thread StephanEwen
Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/3736
  
@bowenli86 Th good thing is that should be able to use the reporter version 
1.4-SNAPSHOT also with a Flink 1.3 release, because the reporter API is quite 
stable by now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-05-08 Thread zentol
Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3736
  
I can merge it sometime this week, once i found a few more things to merge 
alongside it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-05-08 Thread bowenli86
Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/3736
  
I see - by looking at the code, I saw Flink has switched to version 
1.4-SNAPSHOT 12h ago. Getting it into 1.4 is fine, our company has a open 
ticket for getting this code into Flink, and I want to get it done. 

We haven't seen the issue you described on our side yet. I'll keep an eye 
on it if it comes up.

What's next for this PR? Anyone from Data Artisan will take a look at this 
PR, and merge it in?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-05-08 Thread zentol
Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3736
  
This won't make it into 1.3. For one the feature freeze is in effect since 
today, so it is too late to merge it now.

That said it may be possible to merge it for 1.4 and maintain a separate 
repository for a 1.3 version that we can also link in the docs. But I'm not 
entirely sure.

I tried it out this morning, and it worked very well. There was an odd 
thing where the host field in the metric identifier wasn't shown for the 
taskmanager metrics but for the jobmanager, but that may hidden by Datadog 
automatically when the hosts match. (i.e. the first part of the metric 
identifier)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-05-07 Thread bowenli86
Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/3736
  
@zentol cool! Let me know how to get this into 1.3 :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-05-07 Thread zentol
Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3736
  
No, I think we addressed all issues :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-05-04 Thread bowenli86
Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/3736
  
@zentol do you have any other concerns?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-04-28 Thread bowenli86
Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/3736
  
@StephanEwen @zentol  I've made the following changes:

1) Datadog (DD) itself has a bug of being unstable when users filter 
metrics by 'host' in dashboards if 'host' is sent as tags, details in 
[FLINK-6013](https://issues.apache.org/jira/browse/FLINK-6013). Communicated 
with DD engineers, they fixed the issue, and now, as advised by them,  I'm 
making 'host' a separate field in metric serialization. Tested the new approach 
successfully, and we are able to filter metrics by 'host' consistently 
correctly on DD dashboard.
2) Added more unit tests for the above change. Unit tests cover both cases 
when 'host' is present in MetricGroup and when it isn't
3) Found Maven can only recognize and run unit tests when file name ends 
with 'Test' rather than 'Tests'. So I renamed my test file to 
'DatadogHttpClientTest' from 'DatadogHttpClientTests'
4) Upgraded okhttp and okio to newer versions
5) Found mocking system millisec will impact the new okhttp. So I separated 
unit tests to two enclosed test sets in 'DatadogHttpClientTest' 

Having successfully tested this across my company in the past couple days, 
I'm now pretty confident this is production ready. 

I agree on @zentol 's proposal if you guys are currently busy with other 
things, as long as we make sure this ends up in 1.3 :)  Thanks, guys!



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-04-27 Thread StephanEwen
Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/3736
  
@bowenli86 Can you build and test this? Then I would be +1 to merge it to 
1.3
It would not be committer tested, but contributor tested, and it is an 
optional component that is not part of Flink's core functionality, so less 
critical.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-04-27 Thread zentol
Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3736
  
Sorry, I'm rather busy at the moment myself due to the feature freeze and 
may not be able to try this out in time.

To be honest, given that if we merge it for 1.4-SNAPSHOT the jar can be 
easily ported to 1.3 (just replace the versions in the pom.xml and rebuild it) 
I'm prioritizing this less than other changes that don't have this luxury.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-04-27 Thread bowenli86
Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/3736
  
@StephanEwen @zentol 
Hi guys, I just tested it on my machine and it works as expected. What's 
your plan on this PR? I found on email thread that there'll be a feature freeze 
of release 1.3 on May 1st, and I'd really desire to get this into 1.3 :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-04-26 Thread StephanEwen
Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/3736
  
+1 from my side
@zentol any further comments/concerns?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-04-25 Thread bowenli86
Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/3736
  
@StephanEwen @zentol I shaded okhttp3 and okio from flink-metrics-datadog. 
I didn't use 'shade-flink' because I found it somehow prevents me from 
building a uber jar. Let me know if it's ok to shade them this way.


```
$ jar -tf 
target/flink-metrics-datadog-1.3-SNAPSHOT-jar-with-dependencies.jar

META-INF/
META-INF/MANIFEST.MF
META-INF/DEPENDENCIES
META-INF/LICENSE
META-INF/NOTICE
org/
org/apache/
org/apache/flink/
org/apache/flink/metrics/
org/apache/flink/metrics/datadog/
org/apache/flink/metrics/datadog/DatadogHttpClient.class

org/apache/flink/metrics/datadog/DatadogHttpReporter$DatadogHttpRequest.class
org/apache/flink/metrics/datadog/DatadogHttpReporter.class
org/apache/flink/metrics/datadog/DCounter.class
org/apache/flink/metrics/datadog/DGauge.class
org/apache/flink/metrics/datadog/DMeter.class
org/apache/flink/metrics/datadog/DMetric.class
org/apache/flink/metrics/datadog/DSeries.class
org/apache/flink/metrics/datadog/MetricType.class
META-INF/maven/
META-INF/maven/org.apache.flink/
META-INF/maven/org.apache.flink/flink-metrics-datadog/
META-INF/maven/org.apache.flink/flink-metrics-datadog/pom.xml
META-INF/maven/org.apache.flink/flink-metrics-datadog/pom.properties
META-INF/maven/org.apache.flink/force-shading/
META-INF/maven/org.apache.flink/force-shading/pom.xml
META-INF/maven/org.apache.flink/force-shading/pom.properties
org/apache/flink/shaded/
org/apache/flink/shaded/okhttp3/
org/apache/flink/shaded/okhttp3/Address.class
.
org/apache/flink/shaded/okhttp3/WebSocketListener.class
META-INF/maven/com.squareup.okhttp3/
META-INF/maven/com.squareup.okhttp3/okhttp/
META-INF/maven/com.squareup.okhttp3/okhttp/pom.xml
META-INF/maven/com.squareup.okhttp3/okhttp/pom.properties
org/apache/flink/shaded/okio/
org/apache/flink/shaded/okio/AsyncTimeout$1.class
...
org/apache/flink/shaded/okio/Util.class
META-INF/maven/com.squareup.okio/
META-INF/maven/com.squareup.okio/okio/
META-INF/maven/com.squareup.okio/okio/pom.xml
META-INF/maven/com.squareup.okio/okio/pom.properties
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-04-25 Thread StephanEwen
Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/3736
  
I think this is starting to look very good!

Given that this introduces new libraries as dependencies (okhttp, okio), 
should we pro-actively shade those away to avoid dependency conflicts?
Admittedly, it would only impact users that explicitly drop in the datadog 
reporter, but it might still be nice for those users. Given that we build a 
jr-with-dependencies anyways, the step to shading is small...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-04-24 Thread bowenli86
Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/3736
  
@zentol any further details and suggestions on the maven-jar-plugin part?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-04-20 Thread bowenli86
Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/3736
  
Green build! @zentol 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [FLINK-6013][metrics] Add Datadog HTTP metrics reporter

2017-04-20 Thread bowenli86
Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/3736
  
Aha! Updated! Apparently Chrome search is case insensitive


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [Flink-6013][metrics] Add Datadog HTTP metrics reporter

2017-04-20 Thread zentol
Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3736
  
In the current PR title "Flink" isn't upper-case.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [Flink-6013][metrics] Add Datadog HTTP metrics reporter

2017-04-20 Thread bowenli86
Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/3736
  
Where is the title that you're referring to? This PR's title at the very 
top is already "[FLINK-6013][metrics] Add Datadog HTTP metrics reporter" 
(without double quote)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [Flink-6013][metrics] Add Datadog HTTP metrics reporter

2017-04-20 Thread zentol
Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3736
  
Could you change the PR title to "[FLINK-6013][metrics] Add Datadog HTTP 
metrics reporter"? The comments aren't being mirrored to JIRA.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [Flink-6013][metrics] Add Datadog HTTP metrics reporter

2017-04-19 Thread bowenli86
Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/3736
  
@zentol ready for another round. The build fails because there are a few 
unrelated flake tests timing out in travis build


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [Flink-6013][metrics] Add Datadog HTTP metrics reporter

2017-04-19 Thread bowenli86
Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/3736
  
Addressed @zentol and @StephanEwen 's comments. Ready for another round!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [Flink-6013][metrics] Add Datadog HTTP metrics reporter

2017-04-19 Thread StephanEwen
Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/3736
  
A few more comments on this:

  - Guava is so conflict heavy that should avoid using it in the framework 
wherever possible. Adding a multiple MBs dependency to write 
`Lists.newArrayList` rather than `new ArrayList<>()` also seems a bit much. The 
use of the `Joiner` can also be replaced with a `StringBuilder`.

  - You can drop the `logback-test.xml` - I think this is an uneeded 
artifact that some other modules still have for historic reasons.

  - Would be good to also make sure this is put into the `opt` folder in 
the build target, so it can be easily dropped into the `lib` folder.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [Flink-6013][metrics] Add Datadog HTTP metrics reporter

2017-04-18 Thread bowenli86
Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/3736
  
Addressed @zentol 's comments. Ready for another round. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [Flink-6013][metrics] Add Datadog HTTP metrics reporter

2017-04-18 Thread zentol
Github user zentol commented on the issue:

https://github.com/apache/flink/pull/3736
  
If you want to enforce a specific scope config then i would suggest to 
instead ignore the configured one (which means not using 
`MetricGroup#getMetricIdentifier()`) and instead extracting the variables you 
want from `MetricGroup#getAllVariables()`. The JMXReporter does this as well 
and is one example of a reporter that completely ignores scope formats. It 
builds a series key-value pairs containing the variables for every metric, not 
unlike this reporter.

You should be able to support Meters, as the DCounter works with Numbers. 
Only exporting the rate is perfectly reasonable. You don't necessarily have to 
ignore Histograms; you can instead create a number of DCounters for every 
property that is extracted from the HistogramStatistics; it's a bit more work 
but doable imo.

Instead of storing the actual value in the DMetric why not store the Flink 
metric instead? When `DMetric#getPoints()` is called you could retrieve the 
value form the metric, you wouldn't have to update anything manually.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] flink issue #3736: [Flink-6013][metrics] Add Datadog HTTP metrics reporter

2017-04-18 Thread bowenli86
Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/3736
  
Great advice. Thanks, @zentol !  

My questions to your suggestions:
1. If I make DatadogHttpReport implement MetricReporter interface, shall I 
just ignore Histogram and Meter which are not supported by Datadog Http API and 
document that?
2. For creating DGauges/DCounters. I believe you are correct. I can create 
new objects upon metrics registration, and continue update their values, right?
3. About parsing tags. Great to know I can do it via MetricGroup! Thus, 
shall I just document the metrics scope configs should be as the following? 

```
metrics.scope.jm: jobmanager
metrics.scope.jm.job: jobmanager.job
metrics.scope.tm: taskmanager
metrics.scope.tm.job: taskmanager.job
metrics.scope.task: task
metrics.scope.operator: operator
```




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---