[
https://issues.apache.org/jira/browse/FLINK-10521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-10521.
------------------------------------
Resolution: Won't Fix
There's no good solution for this.
Reporters shouldn't have to guard every call to a metric with a try/catch block.
We could wrap every metric such that any exception is wrapped into a special
exception which has a reference to the failed metric. The reporters would have
a single try/catch block within {{report()}} (or equivalent for non-scheduled
reporters) catching this one exception, and using the contained reference to
un-register the failing metric.
But this still implies having potentially thousands of wrapper objects for the
off-chance that a metric throws an exception, which doesn't seem worth the cost
> Faulty Histogram stops Prometheus metrics from being reported
> -------------------------------------------------------------
>
> Key: FLINK-10521
> URL: https://issues.apache.org/jira/browse/FLINK-10521
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Metrics
> Affects Versions: 1.6.1
> Environment: Flink 1.6.1 cluster with one taskmanager and one
> jobmanager, prometheus and grafana, all started in a local docker environment.
> See sample project at:
> https://github.com/florianschmidt1994/flink-fault-tolerance-baseline
> Reporter: Florian Schmidt
> Priority: Major
> Attachments: Screenshot 2018-10-10 at 11.32.59.png, prometheus.log,
> taskmanager.log
>
>
> In my setup I am using the prometheus reporter and a custom implemented
> histogram metric. After a while the histogram starts throwing exceptions
> (because it is rather poorly implemented). This causes all metrics on the
> taskmanager where the histogram is running to stop being reported. By looking
> at the prometheus logs you can see that requests to
> _taskmanager:9249/metrics_ will return an empty response when a metric is
> faulty.
>
> Expected:
> A faulty metrics implementation causes this specific metric to stop being
> reported
> Actual:
> A faulty metric will cause all metrics on that taskmanager to stop being
> reported
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
