Re: Logging Flink metrics

Ok, got it. I would try to do it manually. Thanks a lot for your inputs and efforts. With regards On Mon, Jul 6, 2020 at 10:58 PM Chesnay Schepler wrote: > WSL is a bit buggy when it comes to allocating ports; it happily lets 2 > processes create sockets on the same port, except that the latte

Re: Logging Flink metrics

WSL is a bit buggy when it comes to allocating ports; it happily lets 2 processes create sockets on the same port, except that the latter one doesn't do anything. Super annying, and I haven't found a solution to that myself yet. You'll have to configure the ports explicitly for the JM/TM, which

Re: Logging Flink metrics

Yes. On Mon, Jul 6, 2020 at 10:43 PM Chesnay Schepler wrote: > Are you running Flink is WSL by chance? > > On 06/07/2020 19:06, Manish G wrote: > > In flink-conf.yaml: > *metrics.reporter.prom.port: 9250-9260* > > This is based on information provided here >

Re: Logging Flink metrics

Are you running Flink is WSL by chance? On 06/07/2020 19:06, Manish G wrote: In flink-conf.yaml: *metrics.reporter.prom.port: 9250-9260* This is based on information provided here

Re: Logging Flink metrics

In flink-conf.yaml: *metrics.reporter.prom.port: 9250-9260* This is based on information provided here *port - (optional) the port the Prometheus expo

Re: Logging Flink metrics

These are all JobManager metrics; have you configured prometheus to also scrape the task manager processes? On 06/07/2020 18:35, Manish G wrote: The metrics I see on prometheus is like: # HELP flink_jobmanager_job_lastCheckpointRestoreTimestamp lastCheckpointRestoreTimestamp (scope: jobmanager

Re: Logging Flink metrics

The metrics I see on prometheus is like: # HELP flink_jobmanager_job_lastCheckpointRestoreTimestamp lastCheckpointRestoreTimestamp (scope: jobmanager_job) # TYPE flink_jobmanager_job_lastCheckpointRestoreTimestamp gauge flink_jobmanager_job_lastCheckpointRestoreTimestamp{job_id="58483036154d7f72ad

Re: Logging Flink metrics

You've said elsewhere that you do see some metrics in prometheus, which are those? Why are you configuring the host for the prometheus reporter? This option is only for the PrometheusPushGatewayReporter. On 06/07/2020 18:01, Manish G wrote: Hi, So I have following in flink-conf.yml : //

Re: Logging Flink metrics

Hi, So I have following in flink-conf.yml : // metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter metrics.reporter.prom.host: 127.0.0.1 metrics.reporter.prom.port: metrics.reporter.slf4j.class: org.apache.fli

Re: Logging Flink metrics

You have explicitly configured a reporter list, resulting in the slf4j reporter being ignored: 2020-07-06 13:48:22,191 INFO org.apache.flink.configuration.GlobalConfiguration    - Loading configuration property: metrics.reporters, prom 2020-07-06 13:48:23,203 INFO org.apache.flink.run

Re: Logging Flink metrics

Please enable debug logging and search for warnings from the metric groups/registry/reporter. If you cannot find anything suspicious, you can also send the foll log to me directly. On 06/07/2020 16:29, Manish G wrote: Job is an infinite streaming one, so it keeps going. Flink configuration i

Re: Logging Flink metrics

Job is an infinite streaming one, so it keeps going. Flink configuration is as: metrics.reporter.slf4j.class: org.apache.flink.metrics.slf4j.Slf4jReporter metrics.reporter.slf4j.interval: 30 SECONDS On Mon, Jul 6, 2020 at 7:57 PM Chesnay Schepler wrote: > How long did the job run for, and wha

Re: Logging Flink metrics

How long did the job run for, and what is the configured interval? On 06/07/2020 15:51, Manish G wrote: Hi, Thanks for this. I did the configuration as mentioned at the link(changes in flink-conf.yml, copying the jar in lib directory), and registered the Meter with metrics group and invoked

Re: Logging Flink metrics

Hi, Thanks for this. I did the configuration as mentioned at the link(changes in flink-conf.yml, copying the jar in lib directory), and registered the Meter with metrics group and invoked markEvent() method in the target code. But I don't see any related logs. I am doing this all on my local comp

Re: Logging Flink metrics

Have you looked at the SLF4J reporter? https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter On 06/07/2020 13:49, Manish G wrote: Hi, Is it possible to log Flink metrics in application logs apart from publishing it t

Logging Flink metrics

Hi, Is it possible to log Flink metrics in application logs apart from publishing it to Prometheus? With regards