[jira] [Commented] (FLINK-8284) Custom metrics not being exposed for Prometheus
[
https://issues.apache.org/jira/browse/FLINK-8284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16330490#comment-16330490
]
Fabian Hueske commented on FLINK-8284:
--
+1 to improve the docs.
[~JBiason], would you mind to either update this issue or create a new one and
tag it to the component "Documentation".
Thank you!
> Custom metrics not being exposed for Prometheus
> ---
>
> Key: FLINK-8284
> URL: https://issues.apache.org/jira/browse/FLINK-8284
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Metrics
>Affects Versions: 1.4.0
> Environment: Linux/CentOS 7
>Reporter: Julio Biason
>Priority: Major
>
> Following the documentation, we changed our filter that removes events with
> missing fields to a RichFilterFunction, so we can capture metrics about such
> events:
> {code:scala}
> public class MissingClientFilter extends RichFilterFunction {
> private transient Counter counter;
> @Override
> public void open(Configuration config) {
> this.counter = getRuntimeContext()
> .getMetricGroup()
> .addGroup("events")
> .counter("missingClient");
> }
> @Override
> public boolean filter(LineData line) {
> String client = line.get("client").toString();
> boolean missing = client.trim().equals("");
> if (!missing) {
> this.count();
> }
> return !missing;
> }
> private void count() {
> if (this.counter != null) {
> this.counter.inc();
> }
> }
> }
> {code}
> We also added Prometheus as our reporter:
> {noformat}
> metrics.reporters: prom
> metrics.reporter.prom.port: 9105
> metrics.reporter.prom.class:
> org.apache.flink.metrics.prometheus.PrometheusReporter
> {noformat}
> The problem is accessing port 9105 display all Flink metrics, but not ours.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8284) Custom metrics not being exposed for Prometheus
[
https://issues.apache.org/jira/browse/FLINK-8284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16330397#comment-16330397
]
Julio Biason commented on FLINK-8284:
-
Got it. Even if that seems obvious (now), I do believe it would be good to
mention this kind of stuff on the docs.
> Custom metrics not being exposed for Prometheus
> ---
>
> Key: FLINK-8284
> URL: https://issues.apache.org/jira/browse/FLINK-8284
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Metrics
>Affects Versions: 1.4.0
> Environment: Linux/CentOS 7
>Reporter: Julio Biason
>Priority: Major
>
> Following the documentation, we changed our filter that removes events with
> missing fields to a RichFilterFunction, so we can capture metrics about such
> events:
> {code:scala}
> public class MissingClientFilter extends RichFilterFunction {
> private transient Counter counter;
> @Override
> public void open(Configuration config) {
> this.counter = getRuntimeContext()
> .getMetricGroup()
> .addGroup("events")
> .counter("missingClient");
> }
> @Override
> public boolean filter(LineData line) {
> String client = line.get("client").toString();
> boolean missing = client.trim().equals("");
> if (!missing) {
> this.count();
> }
> return !missing;
> }
> private void count() {
> if (this.counter != null) {
> this.counter.inc();
> }
> }
> }
> {code}
> We also added Prometheus as our reporter:
> {noformat}
> metrics.reporters: prom
> metrics.reporter.prom.port: 9105
> metrics.reporter.prom.class:
> org.apache.flink.metrics.prometheus.PrometheusReporter
> {noformat}
> The problem is accessing port 9105 display all Flink metrics, but not ours.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8284) Custom metrics not being exposed for Prometheus
[
https://issues.apache.org/jira/browse/FLINK-8284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329054#comment-16329054
]
Chesnay Schepler commented on FLINK-8284:
-
You have to either configure multiple ports since each Job-/TaskManager has
it's own reporter, or provide each *Manager with a separate configuration.
THere's a port conflict here, and only the JobManager reporter is being started.
> Custom metrics not being exposed for Prometheus
> ---
>
> Key: FLINK-8284
> URL: https://issues.apache.org/jira/browse/FLINK-8284
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Metrics
>Affects Versions: 1.4.0
> Environment: Linux/CentOS 7
>Reporter: Julio Biason
>Priority: Major
>
> Following the documentation, we changed our filter that removes events with
> missing fields to a RichFilterFunction, so we can capture metrics about such
> events:
> {code:scala}
> public class MissingClientFilter extends RichFilterFunction {
> private transient Counter counter;
> @Override
> public void open(Configuration config) {
> this.counter = getRuntimeContext()
> .getMetricGroup()
> .addGroup("events")
> .counter("missingClient");
> }
> @Override
> public boolean filter(LineData line) {
> String client = line.get("client").toString();
> boolean missing = client.trim().equals("");
> if (!missing) {
> this.count();
> }
> return !missing;
> }
> private void count() {
> if (this.counter != null) {
> this.counter.inc();
> }
> }
> }
> {code}
> We also added Prometheus as our reporter:
> {noformat}
> metrics.reporters: prom
> metrics.reporter.prom.port: 9105
> metrics.reporter.prom.class:
> org.apache.flink.metrics.prometheus.PrometheusReporter
> {noformat}
> The problem is accessing port 9105 display all Flink metrics, but not ours.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8284) Custom metrics not being exposed for Prometheus
[
https://issues.apache.org/jira/browse/FLINK-8284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328953#comment-16328953
]
Julio Biason commented on FLINK-8284:
-
{quote}Do you see any metric for this particular operator? (like numRecordsOut)
{quote}
No, There are no `numRecordsOut` in the resulting metrics.
{quote}check for any metric related errors.
{quote}
The only thing that seems to appear is
{{2018-01-17 00:17:36,526 ERROR
org.apache.flink.runtime.metrics.MetricRegistryImpl - Could not
instantiate metrics reporter prom. Metrics might not be exposed/reported.}}
{{java.lang.RuntimeException: Could not start PrometheusReporter HTTP server on
any configured port. Ports: 9105}}
But, then again, the port IS open and I can RETRIEVE the cluster metrics.
> Custom metrics not being exposed for Prometheus
> ---
>
> Key: FLINK-8284
> URL: https://issues.apache.org/jira/browse/FLINK-8284
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Metrics
>Affects Versions: 1.4.0
> Environment: Linux/CentOS 7
>Reporter: Julio Biason
>Priority: Major
>
> Following the documentation, we changed our filter that removes events with
> missing fields to a RichFilterFunction, so we can capture metrics about such
> events:
> {code:scala}
> public class MissingClientFilter extends RichFilterFunction {
> private transient Counter counter;
> @Override
> public void open(Configuration config) {
> this.counter = getRuntimeContext()
> .getMetricGroup()
> .addGroup("events")
> .counter("missingClient");
> }
> @Override
> public boolean filter(LineData line) {
> String client = line.get("client").toString();
> boolean missing = client.trim().equals("");
> if (!missing) {
> this.count();
> }
> return !missing;
> }
> private void count() {
> if (this.counter != null) {
> this.counter.inc();
> }
> }
> }
> {code}
> We also added Prometheus as our reporter:
> {noformat}
> metrics.reporters: prom
> metrics.reporter.prom.port: 9105
> metrics.reporter.prom.class:
> org.apache.flink.metrics.prometheus.PrometheusReporter
> {noformat}
> The problem is accessing port 9105 display all Flink metrics, but not ours.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8284) Custom metrics not being exposed for Prometheus
[
https://issues.apache.org/jira/browse/FLINK-8284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318999#comment-16318999
]
Julio Biason commented on FLINK-8284:
-
I swear to ${deity} that it didn't work. Restarted the project to check logs
and suddenly the metrics are there.
Closing this as invalid. Will reopen if I find any reason for it not working in
the first place.
> Custom metrics not being exposed for Prometheus
> ---
>
> Key: FLINK-8284
> URL: https://issues.apache.org/jira/browse/FLINK-8284
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Metrics
>Affects Versions: 1.4.0
> Environment: Linux/CentOS 7
>Reporter: Julio Biason
>
> Following the documentation, we changed our filter that removes events with
> missing fields to a RichFilterFunction, so we can capture metrics about such
> events:
> {code:scala}
> public class MissingClientFilter extends RichFilterFunction {
> private transient Counter counter;
> @Override
> public void open(Configuration config) {
> this.counter = getRuntimeContext()
> .getMetricGroup()
> .addGroup("events")
> .counter("missingClient");
> }
> @Override
> public boolean filter(LineData line) {
> String client = line.get("client").toString();
> boolean missing = client.trim().equals("");
> if (!missing) {
> this.count();
> }
> return !missing;
> }
> private void count() {
> if (this.counter != null) {
> this.counter.inc();
> }
> }
> }
> {code}
> We also added Prometheus as our reporter:
> {noformat}
> metrics.reporters: prom
> metrics.reporter.prom.port: 9105
> metrics.reporter.prom.class:
> org.apache.flink.metrics.prometheus.PrometheusReporter
> {noformat}
> The problem is accessing port 9105 display all Flink metrics, but not ours.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-8284) Custom metrics not being exposed for Prometheus
[
https://issues.apache.org/jira/browse/FLINK-8284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316230#comment-16316230
]
Chesnay Schepler commented on FLINK-8284:
-
Do you see any metric for this particular operator? (like numRecordsOut)
Please set the log level to WARN/DEBUG and check for any metric related errors.
> Custom metrics not being exposed for Prometheus
> ---
>
> Key: FLINK-8284
> URL: https://issues.apache.org/jira/browse/FLINK-8284
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Metrics
>Affects Versions: 1.4.0
> Environment: Linux/CentOS 7
>Reporter: Julio Biason
>
> Following the documentation, we changed our filter that removes events with
> missing fields to a RichFilterFunction, so we can capture metrics about such
> events:
> {code:scala}
> public class MissingClientFilter extends RichFilterFunction {
> private transient Counter counter;
> @Override
> public void open(Configuration config) {
> this.counter = getRuntimeContext()
> .getMetricGroup()
> .addGroup("events")
> .counter("missingClient");
> }
> @Override
> public boolean filter(LineData line) {
> String client = line.get("client").toString();
> boolean missing = client.trim().equals("");
> if (!missing) {
> this.count();
> }
> return !missing;
> }
> private void count() {
> if (this.counter != null) {
> this.counter.inc();
> }
> }
> }
> {code}
> We also added Prometheus as our reporter:
> {noformat}
> metrics.reporters: prom
> metrics.reporter.prom.port: 9105
> metrics.reporter.prom.class:
> org.apache.flink.metrics.prometheus.PrometheusReporter
> {noformat}
> The problem is accessing port 9105 display all Flink metrics, but not ours.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-8284) Custom metrics not being exposed for Prometheus
[
https://issues.apache.org/jira/browse/FLINK-8284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308280#comment-16308280
]
Aljoscha Krettek commented on FLINK-8284:
-
[~Zentol] Could you please have a look at this?
> Custom metrics not being exposed for Prometheus
> ---
>
> Key: FLINK-8284
> URL: https://issues.apache.org/jira/browse/FLINK-8284
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Metrics
>Affects Versions: 1.4.0
> Environment: Linux/CentOS 7
>Reporter: Julio Biason
>
> Following the documentation, we changed our filter that removes events with
> missing fields to a RichFilterFunction, so we can capture metrics about such
> events:
> {code:scala}
> public class MissingClientFilter extends RichFilterFunction {
> private transient Counter counter;
> @Override
> public void open(Configuration config) {
> this.counter = getRuntimeContext()
> .getMetricGroup()
> .addGroup("events")
> .counter("missingClient");
> }
> @Override
> public boolean filter(LineData line) {
> String client = line.get("client").toString();
> boolean missing = client.trim().equals("");
> if (!missing) {
> this.count();
> }
> return !missing;
> }
> private void count() {
> if (this.counter != null) {
> this.counter.inc();
> }
> }
> }
> {code}
> We also added Prometheus as our reporter:
> {noformat}
> metrics.reporters: prom
> metrics.reporter.prom.port: 9105
> metrics.reporter.prom.class:
> org.apache.flink.metrics.prometheus.PrometheusReporter
> {noformat}
> The problem is accessing port 9105 display all Flink metrics, but not ours.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
