[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[ https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16071620#comment-16071620 ] Chesnay Schepler commented on FLINK-6221: - I believe they are built nightly. > Add Prometheus support to metrics > - > > Key: FLINK-6221 > URL: https://issues.apache.org/jira/browse/FLINK-6221 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.2.0 >Reporter: Joshua Griffith >Assignee: Maximilian Bode >Priority: Minor > Fix For: 1.4.0 > > > [Prometheus|https://prometheus.io/] is becoming popular for metrics and > alerting. It's possible to use > [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink > metrics into Prometheus but it would be far easier if Flink supported > Promethus as a metrics reporter. A [dropwizard > client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard] > exists that could be integrated into the existing metrics system. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[ https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16071312#comment-16071312 ] Maximilian Bode commented on FLINK-6221: [~Zentol] I'll update as soon as the relevant section has made it into https://ci.apache.org/projects/flink/flink-docs-release-1.4/monitoring/metrics.html#reporter to have something to link to. By the way - how often are the docs built from master, apparently not on every commit? > Add Prometheus support to metrics > - > > Key: FLINK-6221 > URL: https://issues.apache.org/jira/browse/FLINK-6221 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.2.0 >Reporter: Joshua Griffith >Assignee: Maximilian Bode >Priority: Minor > Fix For: 1.4.0 > > > [Prometheus|https://prometheus.io/] is becoming popular for metrics and > alerting. It's possible to use > [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink > metrics into Prometheus but it would be far easier if Flink supported > Promethus as a metrics reporter. A [dropwizard > client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard] > exists that could be integrated into the existing metrics system. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[ https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16071140#comment-16071140 ] ASF GitHub Bot commented on FLINK-6221: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3833 > Add Prometheus support to metrics > - > > Key: FLINK-6221 > URL: https://issues.apache.org/jira/browse/FLINK-6221 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.2.0 >Reporter: Joshua Griffith >Assignee: Maximilian Bode >Priority: Minor > > [Prometheus|https://prometheus.io/] is becoming popular for metrics and > alerting. It's possible to use > [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink > metrics into Prometheus but it would be far easier if Flink supported > Promethus as a metrics reporter. A [dropwizard > client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard] > exists that could be integrated into the existing metrics system. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[ https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063277#comment-16063277 ] ASF GitHub Bot commented on FLINK-6221: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/3833 Finally found time to try this out, and it works great, merging. > Add Prometheus support to metrics > - > > Key: FLINK-6221 > URL: https://issues.apache.org/jira/browse/FLINK-6221 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.2.0 >Reporter: Joshua Griffith >Assignee: Maximilian Bode >Priority: Minor > > [Prometheus|https://prometheus.io/] is becoming popular for metrics and > alerting. It's possible to use > [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink > metrics into Prometheus but it would be far easier if Flink supported > Promethus as a metrics reporter. A [dropwizard > client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard] > exists that could be integrated into the existing metrics system. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[ https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044326#comment-16044326 ] ASF GitHub Bot commented on FLINK-6221: --- Github user mbode commented on the issue: https://github.com/apache/flink/pull/3833 I see, makes sense. Finally got around to fixing the dependency and getting a [Green Travis](https://travis-ci.org/mbode/flink/builds/240926382). > Add Prometheus support to metrics > - > > Key: FLINK-6221 > URL: https://issues.apache.org/jira/browse/FLINK-6221 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.2.0 >Reporter: Joshua Griffith >Assignee: Maximilian Bode >Priority: Minor > > [Prometheus|https://prometheus.io/] is becoming popular for metrics and > alerting. It's possible to use > [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink > metrics into Prometheus but it would be far easier if Flink supported > Promethus as a metrics reporter. A [dropwizard > client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard] > exists that could be integrated into the existing metrics system. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[ https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034591#comment-16034591 ] ASF GitHub Bot commented on FLINK-6221: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/3833 Another thing to consider is that this rule also covers the myriad of testing utilities that so far did not have to be documented. > Add Prometheus support to metrics > - > > Key: FLINK-6221 > URL: https://issues.apache.org/jira/browse/FLINK-6221 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.2.0 >Reporter: Joshua Griffith >Assignee: Maximilian Bode >Priority: Minor > > [Prometheus|https://prometheus.io/] is becoming popular for metrics and > alerting. It's possible to use > [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink > metrics into Prometheus but it would be far easier if Flink supported > Promethus as a metrics reporter. A [dropwizard > client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard] > exists that could be integrated into the existing metrics system. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[
https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034365#comment-16034365
]
ASF GitHub Bot commented on FLINK-6221:
---
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/3833
I think it is intended. The majority of tests can be (and are) commented
with "Tests for the {@link }.", which is totally OK because it
tells us that this is the general test battery for that class and not something
else. These are also easy to add so there's not a real overhead in doing so.
But this doesn't apply to all of them; and you only really notice the
missing javadoc when you stumble on a test and you spend 10 minutes trying to
figure out what it is doing because the method names aren't helping at all.
The widely-accepted notion that the name of a class and test method are
supposed to be enough documentation for test is quite questionable IMO; and by
enforcing a javadoc on the class we give a little nudge to make the developer
hopefully think about whether something needs to be documented or not.
> Add Prometheus support to metrics
> -
>
> Key: FLINK-6221
> URL: https://issues.apache.org/jira/browse/FLINK-6221
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.2.0
>Reporter: Joshua Griffith
>Assignee: Maximilian Bode
>Priority: Minor
>
> [Prometheus|https://prometheus.io/] is becoming popular for metrics and
> alerting. It's possible to use
> [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink
> metrics into Prometheus but it would be far easier if Flink supported
> Promethus as a metrics reporter. A [dropwizard
> client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard]
> exists that could be integrated into the existing metrics system.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[ https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034152#comment-16034152 ] ASF GitHub Bot commented on FLINK-6221: --- Github user mbode commented on the issue: https://github.com/apache/flink/pull/3833 Oh, I broke the stricter checkstyle. To me, it feels a bit weird to have to put javadoc on tests, is that intended? > Add Prometheus support to metrics > - > > Key: FLINK-6221 > URL: https://issues.apache.org/jira/browse/FLINK-6221 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.2.0 >Reporter: Joshua Griffith >Assignee: Maximilian Bode >Priority: Minor > > [Prometheus|https://prometheus.io/] is becoming popular for metrics and > alerting. It's possible to use > [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink > metrics into Prometheus but it would be far easier if Flink supported > Promethus as a metrics reporter. A [dropwizard > client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard] > exists that could be integrated into the existing metrics system. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[ https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033237#comment-16033237 ] ASF GitHub Bot commented on FLINK-6221: --- Github user mbode commented on a diff in the pull request: https://github.com/apache/flink/pull/3833#discussion_r119661118 --- Diff: flink-metrics/flink-metrics-prometheus/pom.xml --- @@ -0,0 +1,129 @@ + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; +xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd;> + + 4.0.0 + + + org.apache.flink + flink-metrics + 1.3-SNAPSHOT --- End diff -- Done > Add Prometheus support to metrics > - > > Key: FLINK-6221 > URL: https://issues.apache.org/jira/browse/FLINK-6221 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.2.0 >Reporter: Joshua Griffith >Assignee: Maximilian Bode >Priority: Minor > > [Prometheus|https://prometheus.io/] is becoming popular for metrics and > alerting. It's possible to use > [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink > metrics into Prometheus but it would be far easier if Flink supported > Promethus as a metrics reporter. A [dropwizard > client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard] > exists that could be integrated into the existing metrics system. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[ https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16032868#comment-16032868 ] ASF GitHub Bot commented on FLINK-6221: --- Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/3833#discussion_r119595452 --- Diff: flink-metrics/flink-metrics-prometheus/pom.xml --- @@ -0,0 +1,129 @@ + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; +xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd;> + + 4.0.0 + + + org.apache.flink + flink-metrics + 1.3-SNAPSHOT --- End diff -- I guess that's outdated by now. Sorry :( > Add Prometheus support to metrics > - > > Key: FLINK-6221 > URL: https://issues.apache.org/jira/browse/FLINK-6221 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.2.0 >Reporter: Joshua Griffith >Assignee: Maximilian Bode >Priority: Minor > > [Prometheus|https://prometheus.io/] is becoming popular for metrics and > alerting. It's possible to use > [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink > metrics into Prometheus but it would be far easier if Flink supported > Promethus as a metrics reporter. A [dropwizard > client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard] > exists that could be integrated into the existing metrics system. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[ https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017367#comment-16017367 ] ASF GitHub Bot commented on FLINK-6221: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/3833 Shading looks correct; dependencies are included and the correct jar is included in flink-dist. > Add Prometheus support to metrics > - > > Key: FLINK-6221 > URL: https://issues.apache.org/jira/browse/FLINK-6221 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.2.0 >Reporter: Joshua Griffith >Assignee: Maximilian Bode >Priority: Minor > > [Prometheus|https://prometheus.io/] is becoming popular for metrics and > alerting. It's possible to use > [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink > metrics into Prometheus but it would be far easier if Flink supported > Promethus as a metrics reporter. A [dropwizard > client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard] > exists that could be integrated into the existing metrics system. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[ https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017308#comment-16017308 ] ASF GitHub Bot commented on FLINK-6221: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/3833 I'll take a look later today. > Add Prometheus support to metrics > - > > Key: FLINK-6221 > URL: https://issues.apache.org/jira/browse/FLINK-6221 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.2.0 >Reporter: Joshua Griffith >Assignee: Maximilian Bode >Priority: Minor > > [Prometheus|https://prometheus.io/] is becoming popular for metrics and > alerting. It's possible to use > [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink > metrics into Prometheus but it would be far easier if Flink supported > Promethus as a metrics reporter. A [dropwizard > client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard] > exists that could be integrated into the existing metrics system. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[ https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16015224#comment-16015224 ] ASF GitHub Bot commented on FLINK-6221: --- Github user mbode commented on the issue: https://github.com/apache/flink/pull/3833 @zentol Would you mind checking that I got the shading right? > Add Prometheus support to metrics > - > > Key: FLINK-6221 > URL: https://issues.apache.org/jira/browse/FLINK-6221 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.2.0 >Reporter: Joshua Griffith >Assignee: Maximilian Bode >Priority: Minor > > [Prometheus|https://prometheus.io/] is becoming popular for metrics and > alerting. It's possible to use > [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink > metrics into Prometheus but it would be far easier if Flink supported > Promethus as a metrics reporter. A [dropwizard > client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard] > exists that could be integrated into the existing metrics system. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[ https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009658#comment-16009658 ] ASF GitHub Bot commented on FLINK-6221: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/3833 Yu can find an example on how to shade here: https://github.com/apache/flink/blob/master/flink-metrics/flink-metrics-datadog/pom.xml. Shading dependencies in reporters/connectors has become a safety-precaution form our side. It is not that unlikely that user-code contains the same dependencies. For one user-code also includes other reporters, so by the very existence of this reporter there is a precedent :) Besides that something that pops up from time to time on the mailing lists is users talking to REST endpoints in their functions/source/sinks, which may also rely on http-related dependencies. > Add Prometheus support to metrics > - > > Key: FLINK-6221 > URL: https://issues.apache.org/jira/browse/FLINK-6221 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.2.0 >Reporter: Joshua Griffith >Assignee: Maximilian Bode >Priority: Minor > > [Prometheus|https://prometheus.io/] is becoming popular for metrics and > alerting. It's possible to use > [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink > metrics into Prometheus but it would be far easier if Flink supported > Promethus as a metrics reporter. A [dropwizard > client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard] > exists that could be integrated into the existing metrics system. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[
https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009654#comment-16009654
]
ASF GitHub Bot commented on FLINK-6221:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3833#discussion_r116377143
--- Diff:
flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusReporter.java
---
@@ -133,61 +136,58 @@ public void notifyOfRemovedMetric(final Metric
metric, final String metricName,
collectorsByMetricName.remove(metricName);
}
+ @SuppressWarnings("unchecked")
+ private static String getLogicalScope(MetricGroup group) {
+ return ((FrontMetricGroup)
group).getLogicalScope(CHARACTER_FILTER, SCOPE_SEPARATOR);
+ }
+
private Collector createGauge(final Gauge gauge, final String name,
final String identifier, final List labelNames, final List
labelValues) {
- return io.prometheus.client.Gauge
- .build()
- .name(name)
- .help(identifier)
- .labelNames(toArray(labelNames, String.class))
- .create()
- .setChild(new io.prometheus.client.Gauge.Child() {
- @Override
- public double get() {
- final Object value = gauge.getValue();
- if (value instanceof Double) {
- return (double) value;
- }
- if (value instanceof Number) {
- return ((Number)
value).doubleValue();
- } else if (value instanceof Boolean) {
- return ((Boolean) value) ? 1 :
0;
- } else {
- log.debug("Invalid type for
Gauge {}: {}, only number types and booleans are supported by this reporter.",
- gauge,
value.getClass().getName());
- return 0;
- }
+ return newGauge(name, identifier, labelNames, labelValues, new
io.prometheus.client.Gauge.Child() {
+ @Override
+ public double get() {
+ final Object value = gauge.getValue();
+ if (value instanceof Double) {
+ return (double) value;
+ }
+ if (value instanceof Number) {
+ return ((Number) value).doubleValue();
+ } else if (value instanceof Boolean) {
+ return ((Boolean) value) ? 1 : 0;
+ } else {
+ LOG.debug("Invalid type for Gauge {}:
{}, only number types and booleans are supported by this reporter.",
+ gauge,
value.getClass().getName());
+ return 0;
}
- }, toArray(labelValues, String.class));
+ }
+ });
}
private static Collector createGauge(final Counter counter, final
String name, final String identifier, final List labelNames, final
List labelValues) {
- return io.prometheus.client.Gauge
- .build()
- .name(name)
- .help(identifier)
- .labelNames(toArray(labelNames, String.class))
- .create()
- .setChild(new io.prometheus.client.Gauge.Child() {
- @Override
- public double get() {
- return (double) counter.getCount();
- }
- }, toArray(labelValues, String.class));
+ return newGauge(name, identifier, labelNames, labelValues, new
io.prometheus.client.Gauge.Child() {
+ @Override
+ public double get() {
+ return (double) counter.getCount();
+ }
+ });
+ }
+
+ private Collector createGauge(final Meter meter, final String name,
final String identifier, final List labelNames, final List
labelValues) {
+ return newGauge(name +
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[
https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009649#comment-16009649
]
ASF GitHub Bot commented on FLINK-6221:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3833#discussion_r116376926
--- Diff:
flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusReporter.java
---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.prometheus;
+
+import com.google.common.base.CharMatcher;
+import com.google.common.collect.ImmutableList;
+import fi.iki.elonen.NanoHTTPD;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.exporter.common.TextFormat;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.metrics.CharacterFilter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.runtime.metrics.groups.AbstractMetricGroup;
+import org.apache.flink.runtime.metrics.groups.FrontMetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+import static com.google.common.collect.Iterables.toArray;
+
+@PublicEvolving
+public class PrometheusReporter implements MetricReporter {
+ private static final Logger log =
LoggerFactory.getLogger(PrometheusReporter.class);
+
+ static final String ARG_PORT = "port";
+ private static final intDEFAULT_PORT = 9249;
+
+ private static final Pattern UNALLOWED_CHAR_PATTERN =
Pattern.compile("[^a-zA-Z0-9:_]");
+ private static final CharacterFilter CHARACTER_FILTER = new
CharacterFilter() {
+ @Override
+ public String filterCharacters(String input) {
+ return replaceInvalidChars(input);
+ }
+ };
+
+ private static final char SCOPE_SEPARATOR = '_';
+
+ private PrometheusEndpoint prometheusEndpoint;
+ private Map collectorsByMetricName = new HashMap<>();
+
+ @VisibleForTesting
+ static String replaceInvalidChars(final String input) {
+ // https://prometheus.io/docs/instrumenting/writing_exporters/
+ // Only [a-zA-Z0-9:_] are valid in metric names, any other
characters should be sanitized to an underscore.
+ return UNALLOWED_CHAR_PATTERN.matcher(input).replaceAll("_");
+ }
+
+ @Override
+ public void open(MetricConfig config) {
+ int port = config.getInteger(ARG_PORT, DEFAULT_PORT);
+ log.info("Using port {}.", port);
+ prometheusEndpoint = new PrometheusEndpoint(port);
+ try {
+ prometheusEndpoint.start(NanoHTTPD.SOCKET_READ_TIMEOUT,
true);
+ } catch (IOException e) {
+ log.error("Could not start PrometheusEndpoint on port "
+ port, e);
+ }
+ }
+
+ @Override
+ public void close() {
+ prometheusEndpoint.stop();
+ CollectorRegistry.defaultRegistry.clear();
+ }
+
+ @Override
+ public void notifyOfAddedMetric(final Metric
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[ https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009646#comment-16009646 ] ASF GitHub Bot commented on FLINK-6221: --- Github user mbode commented on the issue: https://github.com/apache/flink/pull/3833 Okay, guava is not used anymore. I am not sure about the shading part. Would you expect either prometheus clients or nanohttpd to be used in Flink user code? Or are there other advantages to shading? If so, could you point me to a module I could copy the "Flink way of shading" from? > Add Prometheus support to metrics > - > > Key: FLINK-6221 > URL: https://issues.apache.org/jira/browse/FLINK-6221 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.2.0 >Reporter: Joshua Griffith >Assignee: Maximilian Bode >Priority: Minor > > [Prometheus|https://prometheus.io/] is becoming popular for metrics and > alerting. It's possible to use > [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink > metrics into Prometheus but it would be far easier if Flink supported > Promethus as a metrics reporter. A [dropwizard > client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard] > exists that could be integrated into the existing metrics system. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[
https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009643#comment-16009643
]
ASF GitHub Bot commented on FLINK-6221:
---
Github user mbode commented on a diff in the pull request:
https://github.com/apache/flink/pull/3833#discussion_r116376242
--- Diff:
flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusReporter.java
---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.prometheus;
+
+import com.google.common.base.CharMatcher;
+import com.google.common.collect.ImmutableList;
+import fi.iki.elonen.NanoHTTPD;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.exporter.common.TextFormat;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.metrics.CharacterFilter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.runtime.metrics.groups.AbstractMetricGroup;
+import org.apache.flink.runtime.metrics.groups.FrontMetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+import static com.google.common.collect.Iterables.toArray;
+
+@PublicEvolving
+public class PrometheusReporter implements MetricReporter {
+ private static final Logger log =
LoggerFactory.getLogger(PrometheusReporter.class);
+
+ static final String ARG_PORT = "port";
+ private static final intDEFAULT_PORT = 9249;
+
+ private static final Pattern UNALLOWED_CHAR_PATTERN =
Pattern.compile("[^a-zA-Z0-9:_]");
+ private static final CharacterFilter CHARACTER_FILTER = new
CharacterFilter() {
+ @Override
+ public String filterCharacters(String input) {
+ return replaceInvalidChars(input);
+ }
+ };
+
+ private static final char SCOPE_SEPARATOR = '_';
+
+ private PrometheusEndpoint prometheusEndpoint;
+ private Map collectorsByMetricName = new HashMap<>();
+
+ @VisibleForTesting
+ static String replaceInvalidChars(final String input) {
+ // https://prometheus.io/docs/instrumenting/writing_exporters/
+ // Only [a-zA-Z0-9:_] are valid in metric names, any other
characters should be sanitized to an underscore.
+ return UNALLOWED_CHAR_PATTERN.matcher(input).replaceAll("_");
+ }
+
+ @Override
+ public void open(MetricConfig config) {
+ int port = config.getInteger(ARG_PORT, DEFAULT_PORT);
+ log.info("Using port {}.", port);
+ prometheusEndpoint = new PrometheusEndpoint(port);
+ try {
+ prometheusEndpoint.start(NanoHTTPD.SOCKET_READ_TIMEOUT,
true);
+ } catch (IOException e) {
+ log.error("Could not start PrometheusEndpoint on port "
+ port, e);
+ }
+ }
+
+ @Override
+ public void close() {
+ prometheusEndpoint.stop();
+ CollectorRegistry.defaultRegistry.clear();
+ }
+
+ @Override
+ public void notifyOfAddedMetric(final Metric
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[
https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009639#comment-16009639
]
ASF GitHub Bot commented on FLINK-6221:
---
Github user mbode commented on a diff in the pull request:
https://github.com/apache/flink/pull/3833#discussion_r116375209
--- Diff:
flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusReporter.java
---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.prometheus;
+
+import com.google.common.base.CharMatcher;
+import com.google.common.collect.ImmutableList;
+import fi.iki.elonen.NanoHTTPD;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.exporter.common.TextFormat;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.metrics.CharacterFilter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.runtime.metrics.groups.AbstractMetricGroup;
+import org.apache.flink.runtime.metrics.groups.FrontMetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+import static com.google.common.collect.Iterables.toArray;
+
+@PublicEvolving
+public class PrometheusReporter implements MetricReporter {
+ private static final Logger log =
LoggerFactory.getLogger(PrometheusReporter.class);
+
+ static final String ARG_PORT = "port";
+ private static final intDEFAULT_PORT = 9249;
+
+ private static final Pattern UNALLOWED_CHAR_PATTERN =
Pattern.compile("[^a-zA-Z0-9:_]");
+ private static final CharacterFilter CHARACTER_FILTER = new
CharacterFilter() {
+ @Override
+ public String filterCharacters(String input) {
+ return replaceInvalidChars(input);
+ }
+ };
+
+ private static final char SCOPE_SEPARATOR = '_';
+
+ private PrometheusEndpoint prometheusEndpoint;
+ private Map collectorsByMetricName = new HashMap<>();
+
+ @VisibleForTesting
+ static String replaceInvalidChars(final String input) {
+ // https://prometheus.io/docs/instrumenting/writing_exporters/
+ // Only [a-zA-Z0-9:_] are valid in metric names, any other
characters should be sanitized to an underscore.
+ return UNALLOWED_CHAR_PATTERN.matcher(input).replaceAll("_");
+ }
+
+ @Override
+ public void open(MetricConfig config) {
+ int port = config.getInteger(ARG_PORT, DEFAULT_PORT);
+ log.info("Using port {}.", port);
+ prometheusEndpoint = new PrometheusEndpoint(port);
+ try {
+ prometheusEndpoint.start(NanoHTTPD.SOCKET_READ_TIMEOUT,
true);
+ } catch (IOException e) {
+ log.error("Could not start PrometheusEndpoint on port "
+ port, e);
+ }
+ }
+
+ @Override
+ public void close() {
+ prometheusEndpoint.stop();
--- End diff --
I don't think so, `NanoHTTPD.stop()` seems to catch everything.
> Add Prometheus support to metrics
>
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[
https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009538#comment-16009538
]
ASF GitHub Bot commented on FLINK-6221:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3833#discussion_r116368530
--- Diff:
flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusReporter.java
---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.prometheus;
+
+import com.google.common.base.CharMatcher;
+import com.google.common.collect.ImmutableList;
+import fi.iki.elonen.NanoHTTPD;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.exporter.common.TextFormat;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.metrics.CharacterFilter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.runtime.metrics.groups.AbstractMetricGroup;
+import org.apache.flink.runtime.metrics.groups.FrontMetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+import static com.google.common.collect.Iterables.toArray;
+
+@PublicEvolving
+public class PrometheusReporter implements MetricReporter {
+ private static final Logger log =
LoggerFactory.getLogger(PrometheusReporter.class);
+
+ static final String ARG_PORT = "port";
--- End diff --
The indentation here in inconsistent with the rest of the code base.
> Add Prometheus support to metrics
> -
>
> Key: FLINK-6221
> URL: https://issues.apache.org/jira/browse/FLINK-6221
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.2.0
>Reporter: Joshua Griffith
>Assignee: Maximilian Bode
>Priority: Minor
>
> [Prometheus|https://prometheus.io/] is becoming popular for metrics and
> alerting. It's possible to use
> [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink
> metrics into Prometheus but it would be far easier if Flink supported
> Promethus as a metrics reporter. A [dropwizard
> client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard]
> exists that could be integrated into the existing metrics system.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[
https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009544#comment-16009544
]
ASF GitHub Bot commented on FLINK-6221:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3833#discussion_r116368532
--- Diff:
flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusReporter.java
---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.prometheus;
+
+import com.google.common.base.CharMatcher;
+import com.google.common.collect.ImmutableList;
+import fi.iki.elonen.NanoHTTPD;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.exporter.common.TextFormat;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.metrics.CharacterFilter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.runtime.metrics.groups.AbstractMetricGroup;
+import org.apache.flink.runtime.metrics.groups.FrontMetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+import static com.google.common.collect.Iterables.toArray;
+
+@PublicEvolving
+public class PrometheusReporter implements MetricReporter {
+ private static final Logger log =
LoggerFactory.getLogger(PrometheusReporter.class);
+
+ static final String ARG_PORT = "port";
+ private static final intDEFAULT_PORT = 9249;
+
+ private static final Pattern UNALLOWED_CHAR_PATTERN =
Pattern.compile("[^a-zA-Z0-9:_]");
+ private static final CharacterFilter CHARACTER_FILTER = new
CharacterFilter() {
+ @Override
+ public String filterCharacters(String input) {
+ return replaceInvalidChars(input);
+ }
+ };
+
+ private static final char SCOPE_SEPARATOR = '_';
+
+ private PrometheusEndpoint prometheusEndpoint;
+ private Map collectorsByMetricName = new HashMap<>();
--- End diff --
Could be final.
> Add Prometheus support to metrics
> -
>
> Key: FLINK-6221
> URL: https://issues.apache.org/jira/browse/FLINK-6221
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.2.0
>Reporter: Joshua Griffith
>Assignee: Maximilian Bode
>Priority: Minor
>
> [Prometheus|https://prometheus.io/] is becoming popular for metrics and
> alerting. It's possible to use
> [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink
> metrics into Prometheus but it would be far easier if Flink supported
> Promethus as a metrics reporter. A [dropwizard
> client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard]
> exists that could be integrated into the existing metrics system.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[
https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009541#comment-16009541
]
ASF GitHub Bot commented on FLINK-6221:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3833#discussion_r116368510
--- Diff:
flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusReporter.java
---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.prometheus;
+
+import com.google.common.base.CharMatcher;
+import com.google.common.collect.ImmutableList;
+import fi.iki.elonen.NanoHTTPD;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.exporter.common.TextFormat;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.metrics.CharacterFilter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.runtime.metrics.groups.AbstractMetricGroup;
+import org.apache.flink.runtime.metrics.groups.FrontMetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+import static com.google.common.collect.Iterables.toArray;
+
+@PublicEvolving
+public class PrometheusReporter implements MetricReporter {
+ private static final Logger log =
LoggerFactory.getLogger(PrometheusReporter.class);
--- End diff --
by convention this should be upper-case.
> Add Prometheus support to metrics
> -
>
> Key: FLINK-6221
> URL: https://issues.apache.org/jira/browse/FLINK-6221
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.2.0
>Reporter: Joshua Griffith
>Assignee: Maximilian Bode
>Priority: Minor
>
> [Prometheus|https://prometheus.io/] is becoming popular for metrics and
> alerting. It's possible to use
> [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink
> metrics into Prometheus but it would be far easier if Flink supported
> Promethus as a metrics reporter. A [dropwizard
> client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard]
> exists that could be integrated into the existing metrics system.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[
https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009539#comment-16009539
]
ASF GitHub Bot commented on FLINK-6221:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3833#discussion_r116368563
--- Diff:
flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusReporter.java
---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.prometheus;
+
+import com.google.common.base.CharMatcher;
+import com.google.common.collect.ImmutableList;
+import fi.iki.elonen.NanoHTTPD;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.exporter.common.TextFormat;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.metrics.CharacterFilter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.runtime.metrics.groups.AbstractMetricGroup;
+import org.apache.flink.runtime.metrics.groups.FrontMetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+import static com.google.common.collect.Iterables.toArray;
+
+@PublicEvolving
+public class PrometheusReporter implements MetricReporter {
+ private static final Logger log =
LoggerFactory.getLogger(PrometheusReporter.class);
+
+ static final String ARG_PORT = "port";
+ private static final intDEFAULT_PORT = 9249;
+
+ private static final Pattern UNALLOWED_CHAR_PATTERN =
Pattern.compile("[^a-zA-Z0-9:_]");
+ private static final CharacterFilter CHARACTER_FILTER = new
CharacterFilter() {
+ @Override
+ public String filterCharacters(String input) {
+ return replaceInvalidChars(input);
+ }
+ };
+
+ private static final char SCOPE_SEPARATOR = '_';
+
+ private PrometheusEndpoint prometheusEndpoint;
+ private Map collectorsByMetricName = new HashMap<>();
+
+ @VisibleForTesting
+ static String replaceInvalidChars(final String input) {
+ // https://prometheus.io/docs/instrumenting/writing_exporters/
+ // Only [a-zA-Z0-9:_] are valid in metric names, any other
characters should be sanitized to an underscore.
+ return UNALLOWED_CHAR_PATTERN.matcher(input).replaceAll("_");
+ }
+
+ @Override
+ public void open(MetricConfig config) {
+ int port = config.getInteger(ARG_PORT, DEFAULT_PORT);
+ log.info("Using port {}.", port);
+ prometheusEndpoint = new PrometheusEndpoint(port);
+ try {
+ prometheusEndpoint.start(NanoHTTPD.SOCKET_READ_TIMEOUT,
true);
+ } catch (IOException e) {
+ log.error("Could not start PrometheusEndpoint on port "
+ port, e);
+ }
+ }
+
+ @Override
+ public void close() {
+ prometheusEndpoint.stop();
--- End diff --
Can this throw an exception?
> Add Prometheus support to metrics
> -
>
>
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[
https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009537#comment-16009537
]
ASF GitHub Bot commented on FLINK-6221:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3833#discussion_r116368672
--- Diff:
flink-metrics/flink-metrics-prometheus/src/test/java/org/apache/flink/metrics/prometheus/PrometheusReporterTest.java
---
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.prometheus;
+
+import com.mashape.unirest.http.HttpResponse;
+import com.mashape.unirest.http.Unirest;
+import com.mashape.unirest.http.exceptions.UnirestException;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.MetricOptions;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.SimpleCounter;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.metrics.util.TestMeter;
+import org.apache.flink.runtime.metrics.MetricRegistry;
+import org.apache.flink.runtime.metrics.MetricRegistryConfiguration;
+import org.apache.flink.runtime.metrics.groups.FrontMetricGroup;
+import org.apache.flink.runtime.metrics.groups.TaskManagerMetricGroup;
+import org.apache.flink.runtime.metrics.util.TestingHistogram;
+import org.apache.flink.util.TestLogger;
+import org.junit.After;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+
+import java.util.Arrays;
+
+import static
org.apache.flink.metrics.prometheus.PrometheusReporter.ARG_PORT;
+import static
org.apache.flink.runtime.metrics.scope.ScopeFormat.SCOPE_SEPARATOR;
+import static org.hamcrest.Matchers.both;
+import static org.hamcrest.Matchers.containsString;
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+public class PrometheusReporterTest extends TestLogger {
+ private static final int NON_DEFAULT_PORT = 9429;
+
+ private static final String HOST_NAME= "hostname";
+ private static final String TASK_MANAGER = "tm";
+
+ private static final String HELP_PREFIX= "# HELP ";
+ private static final String TYPE_PREFIX= "# TYPE ";
+ private static final String DIMENSIONS = "host=\"" + HOST_NAME +
"\",tm_id=\"" + TASK_MANAGER + "\"";
+ private static final String DEFAULT_LABELS = "{" + DIMENSIONS + ",}";
+ private static final String SCOPE_PREFIX = "taskmanager_";
+
+ @Rule
+ public ExpectedException thrown = ExpectedException.none();
+
+ private MetricRegistry registry = prepareMetricRegistry();
--- End diff --
could be final.
> Add Prometheus support to metrics
> -
>
> Key: FLINK-6221
> URL: https://issues.apache.org/jira/browse/FLINK-6221
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.2.0
>Reporter: Joshua Griffith
>Assignee: Maximilian Bode
>Priority: Minor
>
> [Prometheus|https://prometheus.io/] is becoming popular for metrics and
> alerting. It's possible to use
> [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink
> metrics into Prometheus but it would be far easier if Flink supported
> Promethus as a metrics reporter. A [dropwizard
> client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard]
> exists that could be integrated into the existing metrics system.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[
https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009535#comment-16009535
]
ASF GitHub Bot commented on FLINK-6221:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3833#discussion_r116368522
--- Diff: flink-metrics/flink-metrics-prometheus/pom.xml ---
@@ -0,0 +1,128 @@
+
+
+http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;>
+
+ 4.0.0
+
+
+ org.apache.flink
+ flink-metrics
+ 1.3-SNAPSHOT
+ ..
+
+
+ flink-metrics-prometheus
+ flink-metrics-prometheus
+
+
+
+ org.apache.flink
+ flink-annotations
+ ${project.version}
+ provided
+
+
+
+ org.apache.flink
+ flink-runtime_2.10
+ ${project.version}
+ provided
+
+
+
+ org.apache.flink
+ flink-metrics-core
+ ${project.version}
+ provided
+
+
+
+ io.prometheus
+ simpleclient
+ ${prometheus.version}
+
+
+
+ io.prometheus
+ simpleclient_servlet
+ ${prometheus.version}
+
+
+
+ com.nanohttpd
+ nanohttpd
+ 2.2.0
+
+
+
+ com.google.guava
--- End diff --
We try to avoid guava unless absolutely necessary, and virtually all
modules that do depend on it shade it away.
> Add Prometheus support to metrics
> -
>
> Key: FLINK-6221
> URL: https://issues.apache.org/jira/browse/FLINK-6221
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.2.0
>Reporter: Joshua Griffith
>Assignee: Maximilian Bode
>Priority: Minor
>
> [Prometheus|https://prometheus.io/] is becoming popular for metrics and
> alerting. It's possible to use
> [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink
> metrics into Prometheus but it would be far easier if Flink supported
> Promethus as a metrics reporter. A [dropwizard
> client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard]
> exists that could be integrated into the existing metrics system.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[
https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009536#comment-16009536
]
ASF GitHub Bot commented on FLINK-6221:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3833#discussion_r116368696
--- Diff:
flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusReporter.java
---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.prometheus;
+
+import com.google.common.base.CharMatcher;
+import com.google.common.collect.ImmutableList;
+import fi.iki.elonen.NanoHTTPD;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.exporter.common.TextFormat;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.metrics.CharacterFilter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.runtime.metrics.groups.AbstractMetricGroup;
+import org.apache.flink.runtime.metrics.groups.FrontMetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+import static com.google.common.collect.Iterables.toArray;
+
+@PublicEvolving
+public class PrometheusReporter implements MetricReporter {
+ private static final Logger log =
LoggerFactory.getLogger(PrometheusReporter.class);
+
+ static final String ARG_PORT = "port";
+ private static final intDEFAULT_PORT = 9249;
+
+ private static final Pattern UNALLOWED_CHAR_PATTERN =
Pattern.compile("[^a-zA-Z0-9:_]");
+ private static final CharacterFilter CHARACTER_FILTER = new
CharacterFilter() {
+ @Override
+ public String filterCharacters(String input) {
+ return replaceInvalidChars(input);
+ }
+ };
+
+ private static final char SCOPE_SEPARATOR = '_';
+
+ private PrometheusEndpoint prometheusEndpoint;
+ private Map collectorsByMetricName = new HashMap<>();
+
+ @VisibleForTesting
+ static String replaceInvalidChars(final String input) {
+ // https://prometheus.io/docs/instrumenting/writing_exporters/
+ // Only [a-zA-Z0-9:_] are valid in metric names, any other
characters should be sanitized to an underscore.
+ return UNALLOWED_CHAR_PATTERN.matcher(input).replaceAll("_");
+ }
+
+ @Override
+ public void open(MetricConfig config) {
+ int port = config.getInteger(ARG_PORT, DEFAULT_PORT);
+ log.info("Using port {}.", port);
+ prometheusEndpoint = new PrometheusEndpoint(port);
+ try {
+ prometheusEndpoint.start(NanoHTTPD.SOCKET_READ_TIMEOUT,
true);
+ } catch (IOException e) {
+ log.error("Could not start PrometheusEndpoint on port "
+ port, e);
+ }
+ }
+
+ @Override
+ public void close() {
+ prometheusEndpoint.stop();
+ CollectorRegistry.defaultRegistry.clear();
+ }
+
+ @Override
+ public void notifyOfAddedMetric(final Metric
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[
https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009542#comment-16009542
]
ASF GitHub Bot commented on FLINK-6221:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3833#discussion_r116368588
--- Diff:
flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusReporter.java
---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.prometheus;
+
+import com.google.common.base.CharMatcher;
+import com.google.common.collect.ImmutableList;
+import fi.iki.elonen.NanoHTTPD;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.exporter.common.TextFormat;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.metrics.CharacterFilter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.runtime.metrics.groups.AbstractMetricGroup;
+import org.apache.flink.runtime.metrics.groups.FrontMetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+import static com.google.common.collect.Iterables.toArray;
+
+@PublicEvolving
+public class PrometheusReporter implements MetricReporter {
+ private static final Logger log =
LoggerFactory.getLogger(PrometheusReporter.class);
+
+ static final String ARG_PORT = "port";
+ private static final intDEFAULT_PORT = 9249;
+
+ private static final Pattern UNALLOWED_CHAR_PATTERN =
Pattern.compile("[^a-zA-Z0-9:_]");
+ private static final CharacterFilter CHARACTER_FILTER = new
CharacterFilter() {
+ @Override
+ public String filterCharacters(String input) {
+ return replaceInvalidChars(input);
+ }
+ };
+
+ private static final char SCOPE_SEPARATOR = '_';
+
+ private PrometheusEndpoint prometheusEndpoint;
+ private Map collectorsByMetricName = new HashMap<>();
+
+ @VisibleForTesting
+ static String replaceInvalidChars(final String input) {
+ // https://prometheus.io/docs/instrumenting/writing_exporters/
+ // Only [a-zA-Z0-9:_] are valid in metric names, any other
characters should be sanitized to an underscore.
+ return UNALLOWED_CHAR_PATTERN.matcher(input).replaceAll("_");
+ }
+
+ @Override
+ public void open(MetricConfig config) {
+ int port = config.getInteger(ARG_PORT, DEFAULT_PORT);
+ log.info("Using port {}.", port);
+ prometheusEndpoint = new PrometheusEndpoint(port);
+ try {
+ prometheusEndpoint.start(NanoHTTPD.SOCKET_READ_TIMEOUT,
true);
+ } catch (IOException e) {
+ log.error("Could not start PrometheusEndpoint on port "
+ port, e);
+ }
+ }
+
+ @Override
+ public void close() {
+ prometheusEndpoint.stop();
+ CollectorRegistry.defaultRegistry.clear();
+ }
+
+ @Override
+ public void notifyOfAddedMetric(final Metric
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[
https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009534#comment-16009534
]
ASF GitHub Bot commented on FLINK-6221:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3833#discussion_r116368561
--- Diff:
flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusReporter.java
---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.prometheus;
+
+import com.google.common.base.CharMatcher;
+import com.google.common.collect.ImmutableList;
+import fi.iki.elonen.NanoHTTPD;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.exporter.common.TextFormat;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.metrics.CharacterFilter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.runtime.metrics.groups.AbstractMetricGroup;
+import org.apache.flink.runtime.metrics.groups.FrontMetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+import static com.google.common.collect.Iterables.toArray;
+
+@PublicEvolving
+public class PrometheusReporter implements MetricReporter {
+ private static final Logger log =
LoggerFactory.getLogger(PrometheusReporter.class);
+
+ static final String ARG_PORT = "port";
+ private static final intDEFAULT_PORT = 9249;
+
+ private static final Pattern UNALLOWED_CHAR_PATTERN =
Pattern.compile("[^a-zA-Z0-9:_]");
+ private static final CharacterFilter CHARACTER_FILTER = new
CharacterFilter() {
+ @Override
+ public String filterCharacters(String input) {
+ return replaceInvalidChars(input);
+ }
+ };
+
+ private static final char SCOPE_SEPARATOR = '_';
+
+ private PrometheusEndpoint prometheusEndpoint;
+ private Map collectorsByMetricName = new HashMap<>();
+
+ @VisibleForTesting
+ static String replaceInvalidChars(final String input) {
+ // https://prometheus.io/docs/instrumenting/writing_exporters/
+ // Only [a-zA-Z0-9:_] are valid in metric names, any other
characters should be sanitized to an underscore.
+ return UNALLOWED_CHAR_PATTERN.matcher(input).replaceAll("_");
+ }
+
+ @Override
+ public void open(MetricConfig config) {
+ int port = config.getInteger(ARG_PORT, DEFAULT_PORT);
+ log.info("Using port {}.", port);
+ prometheusEndpoint = new PrometheusEndpoint(port);
+ try {
+ prometheusEndpoint.start(NanoHTTPD.SOCKET_READ_TIMEOUT,
true);
+ } catch (IOException e) {
+ log.error("Could not start PrometheusEndpoint on port "
+ port, e);
--- End diff --
It would be good to throw an exception here despite the current interface.
Otherwise the reporter will still be notified of added/removed metrics, even
though there's no benefit to it.
> Add Prometheus support to metrics
>
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[
https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009540#comment-16009540
]
ASF GitHub Bot commented on FLINK-6221:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3833#discussion_r116368638
--- Diff:
flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/PrometheusReporter.java
---
@@ -0,0 +1,256 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.prometheus;
+
+import com.google.common.base.CharMatcher;
+import com.google.common.collect.ImmutableList;
+import fi.iki.elonen.NanoHTTPD;
+import io.prometheus.client.Collector;
+import io.prometheus.client.CollectorRegistry;
+import io.prometheus.client.exporter.common.TextFormat;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.metrics.CharacterFilter;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.HistogramStatistics;
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.MetricReporter;
+import org.apache.flink.runtime.metrics.groups.AbstractMetricGroup;
+import org.apache.flink.runtime.metrics.groups.FrontMetricGroup;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.StringWriter;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+import static com.google.common.collect.Iterables.toArray;
+
+@PublicEvolving
+public class PrometheusReporter implements MetricReporter {
+ private static final Logger log =
LoggerFactory.getLogger(PrometheusReporter.class);
+
+ static final String ARG_PORT = "port";
+ private static final intDEFAULT_PORT = 9249;
+
+ private static final Pattern UNALLOWED_CHAR_PATTERN =
Pattern.compile("[^a-zA-Z0-9:_]");
+ private static final CharacterFilter CHARACTER_FILTER = new
CharacterFilter() {
+ @Override
+ public String filterCharacters(String input) {
+ return replaceInvalidChars(input);
+ }
+ };
+
+ private static final char SCOPE_SEPARATOR = '_';
+
+ private PrometheusEndpoint prometheusEndpoint;
+ private Map collectorsByMetricName = new HashMap<>();
+
+ @VisibleForTesting
+ static String replaceInvalidChars(final String input) {
+ // https://prometheus.io/docs/instrumenting/writing_exporters/
+ // Only [a-zA-Z0-9:_] are valid in metric names, any other
characters should be sanitized to an underscore.
+ return UNALLOWED_CHAR_PATTERN.matcher(input).replaceAll("_");
+ }
+
+ @Override
+ public void open(MetricConfig config) {
+ int port = config.getInteger(ARG_PORT, DEFAULT_PORT);
+ log.info("Using port {}.", port);
+ prometheusEndpoint = new PrometheusEndpoint(port);
+ try {
+ prometheusEndpoint.start(NanoHTTPD.SOCKET_READ_TIMEOUT,
true);
+ } catch (IOException e) {
+ log.error("Could not start PrometheusEndpoint on port "
+ port, e);
+ }
+ }
+
+ @Override
+ public void close() {
+ prometheusEndpoint.stop();
+ CollectorRegistry.defaultRegistry.clear();
+ }
+
+ @Override
+ public void notifyOfAddedMetric(final Metric
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[ https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009527#comment-16009527 ] ASF GitHub Bot commented on FLINK-6221: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/3833 As it stands a metric can have at most 10 labels, and there aren't any current plans to extend this set; according to the docs that's still ok. If this does indeed become a problem we can add a switch to rely on the scope formats instead, giving the user control as to how many labels there are. > Add Prometheus support to metrics > - > > Key: FLINK-6221 > URL: https://issues.apache.org/jira/browse/FLINK-6221 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.2.0 >Reporter: Joshua Griffith >Assignee: Maximilian Bode >Priority: Minor > > [Prometheus|https://prometheus.io/] is becoming popular for metrics and > alerting. It's possible to use > [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink > metrics into Prometheus but it would be far easier if Flink supported > Promethus as a metrics reporter. A [dropwizard > client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard] > exists that could be integrated into the existing metrics system. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[ https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009520#comment-16009520 ] ASF GitHub Bot commented on FLINK-6221: --- Github user mbode commented on the issue: https://github.com/apache/flink/pull/3833 @hadronzoo I tried to keep it as general as possible by exporting all variables available to the metric group as labels. I am not sure if this might lead to [label overuse](https://prometheus.io/docs/practices/instrumentation/#do-not-overuse-labels) at some point, did you ever run into difficulties in that context? Unfortunately I do not have a lot of experience running Prometheus in a production environment yet. > Add Prometheus support to metrics > - > > Key: FLINK-6221 > URL: https://issues.apache.org/jira/browse/FLINK-6221 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.2.0 >Reporter: Joshua Griffith >Assignee: Maximilian Bode >Priority: Minor > > [Prometheus|https://prometheus.io/] is becoming popular for metrics and > alerting. It's possible to use > [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink > metrics into Prometheus but it would be far easier if Flink supported > Promethus as a metrics reporter. A [dropwizard > client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard] > exists that could be integrated into the existing metrics system. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-6221) Add Prometheus support to metrics
[ https://issues.apache.org/jira/browse/FLINK-6221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007215#comment-16007215 ] ASF GitHub Bot commented on FLINK-6221: --- Github user hadronzoo commented on the issue: https://github.com/apache/flink/pull/3833 @mbode thanks for working on this! One thing that I've found useful when exporting Flink's statsd metrics to Prometheus is to make several of the metric fields tags: like `job_name`, `task_name`, `operator_name`, etc. This [statsd-exporter mapping](https://gist.github.com/hadronzoo/621b6a6dce7e2596d5643ce8d1e954ea) has tags that have worked well for me. I'm not tagging host names or IP addresses because Prometheus's Kubernetes support does that already, but that could be useful for people running standalone clusters. > Add Prometheus support to metrics > - > > Key: FLINK-6221 > URL: https://issues.apache.org/jira/browse/FLINK-6221 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.2.0 >Reporter: Joshua Griffith >Assignee: Maximilian Bode >Priority: Minor > > [Prometheus|https://prometheus.io/] is becoming popular for metrics and > alerting. It's possible to use > [statsd-exporter|https://github.com/prometheus/statsd_exporter] to load Flink > metrics into Prometheus but it would be far easier if Flink supported > Promethus as a metrics reporter. A [dropwizard > client|https://github.com/prometheus/client_java/tree/master/simpleclient_dropwizard] > exists that could be integrated into the existing metrics system. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
