Hi Feba, I did a POC based on Beam’s doc 10.5 [1] and Flink metrics reporter [2]. You can add metrics reporter configuration in flink-conf.yaml so that metrics can be scraped from port you configured. Be sure to set disable_metrics [3] to false.
[1] https://beam.apache.org/documentation/programming-guide/#metrics <https://beam.apache.org/documentation/programming-guide/#metrics> [2] https://ci.apache.org/projects/flink/flink-docs-stable/deployment/metric_reporters.html#prometheus-orgapacheflinkmetricsprometheusprometheusreporter <https://ci.apache.org/projects/flink/flink-docs-stable/deployment/metric_reporters.html#prometheus-orgapacheflinkmetricsprometheusprometheusreporter> [3] https://beam.apache.org/documentation/runners/flink/ <https://beam.apache.org/documentation/runners/flink/> Thanks, Yu > On Apr 28, 2021, at 22:58, Feba Fathima <[email protected]> wrote: > > Hi, > > Thanks Yu. Can you please let me know how you are displaying the counter > metrics in Prometheus. Are you using any metrics sink to push the metrics > from beam and is there any kind of mechanism to pull the metrics into > Prometheus? > > Thanks in advance. > > Thanks & Regards, > Feba Fathima > > On Wed, 28 Apr, 2021, 11:28 pm Yu Zhang, <[email protected] > <mailto:[email protected]>> wrote: > Hi, > > Having similar issues. I tried with Beam 2.27.0, Flink 1.12.1 and Prometheus > 2.25.0. The distribution and gauge metrics are always 0 in both Flink UI and > Prometheus. Counter can be displayed correctly. This should be related with > issue 10928. > > Thanks, > Yu > > > On 2021/04/28 17:39:29, Alexey Romanenko <[email protected] > <mailto:[email protected]>> wrote: > > Hi, > > > > Could it be related [1][2]? > > > > [1] https://issues.apache.org/jira/browse/BEAM-7438 > > <https://issues.apache.org/jira/browse/BEAM-7438> > > [2] https://issues.apache.org/jira/browse/BEAM-10928 > > <https://issues.apache.org/jira/browse/BEAM-10928> > > > > — > > Alexey > > > > > On 28 Apr 2021, at 08:45, Feba Fathima <[email protected] > > > <mailto:[email protected]>> wrote: > > > > > > Hi, > > > > > > I have a java beam pipeline which is run using a flink runner. I tried > > > adding custom metrics to it which I want to export to Prometheus. I would > > > like to know how to export the beam custom metrics to Prometheus. Please > > > help me with this. > > > > > > Thanks in advance > > > Feba Fathima > > > >
