Re: Flink metrics missing from UI 1.7.2
Aha! This is almost certainly it. I remembered thinking something like this might be a problem. I'll need to change the deployment a bit to add this (not straightforward to edit the YAML in my case, but thanks! On Sun, Mar 24, 2019 at 10:01 AM dawid < apache-flink-user-mailing-list-arch...@davidhaglund.se> wrote: > Padarn Wilson-2 wrote > > I am running Fink 1.7.2 on Kubernetes in a setup with task manager and > job > > manager separate. > > > > I'm having trouble seeing the metrics from my Flink job in the UI > > dashboard. Actually I'm using the Datadog reporter to expose most of my > > metrics, but latency tracking does not seem to be exported. > > > > Is there anything extra that needs to be enabled to make sure metrics are > > exported and viewable to the Flink UI? > > With Flink 1.7 on Kubernetes you need to make sure the task managers are > registering to the job manager with their IP addresses and not the > hostnames, see the taskmanager-deployment.yaml manifest in [1], with the > K8S_POD_IP environment variable and setting > -Dtaskmanager.host=$(K8S_POD_IP). > > [1] > > https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/deployment/kubernetes.html#appendix > > /David > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ >
Re: Flink metrics missing from UI 1.7.2
Padarn Wilson-2 wrote > I am running Fink 1.7.2 on Kubernetes in a setup with task manager and job > manager separate. > > I'm having trouble seeing the metrics from my Flink job in the UI > dashboard. Actually I'm using the Datadog reporter to expose most of my > metrics, but latency tracking does not seem to be exported. > > Is there anything extra that needs to be enabled to make sure metrics are > exported and viewable to the Flink UI? With Flink 1.7 on Kubernetes you need to make sure the task managers are registering to the job manager with their IP addresses and not the hostnames, see the taskmanager-deployment.yaml manifest in [1], with the K8S_POD_IP environment variable and setting -Dtaskmanager.host=$(K8S_POD_IP). [1] https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/deployment/kubernetes.html#appendix /David -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Flink metrics missing from UI 1.7.2
Thanks David. I cannot see the metrics there, so let me play around a bit more and make sure they are enabled correctly. On Sat, Mar 23, 2019 at 9:19 PM David Anderson wrote: > > I have done this (actually I do it in my flink-conf.yaml), but I am not > seeing any metrics at all in the Flink UI, > > let alone the latency tracking. The latency tracking itself does not > seem to be exported to datadog (should it be?) > > The latency metrics are job metrics, and are not shown in the Flink UI. > They are available via the REST API, and I believe they should also be > exported to datadog. You will find them at > > http://localhost:8081/jobs//metrics > > with IDs like > > > latency.source_id.bc764cd8ddf7a0cff126f51c16239658.operator_id.ea632d67b7d595e5b851708ae9ad79d6.operator_subtask_index.0.latency_p90 > > On Sat, Mar 23, 2019 at 1:53 PM Padarn Wilson wrote: > >> Thanks David. >> >> I have done this (actually I do it in my flink-conf.yaml), but I am not >> seeing any metrics at all in the Flink UI, let alone the latency tracking. >> The latency tracking itself does not seem to be exported to datadog (should >> it be?) >> >> >> >> On Sat, Mar 23, 2019 at 8:43 PM David Anderson >> wrote: >> >>> Because latency tracking is expensive, it is turned off by default. You >>> turn it on by setting the interval; that looks something like this: >>> >>> env.getConfig().setLatencyTrackingInterval(1000); >>> >>> The full set of configuration options is described in the docs: >>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#metrics >>> >>> David Anderson | Training Coordinator >>> Follow us @VervericaData >>> >>> -- >>> Join Flink Forward - The Apache Flink Conference >>> Stream Processing | Event Driven | Real Time >>> >>> >>> On Sat, Mar 23, 2019 at 1:03 PM Padarn Wilson wrote: >>> > >>> > Hi User, >>> > >>> > I am running Fink 1.7.2 on Kubernetes in a setup with task manager and >>> job manager separate. >>> > >>> > I'm having trouble seeing the metrics from my Flink job in the UI >>> dashboard. Actually I'm using the Datadog reporter to expose most of my >>> metrics, but latency tracking does not seem to be exported. >>> > >>> > Is there anything extra that needs to be enabled to make sure metrics >>> are exported and viewable to the Flink UI? >>> > >>> > Thanks >>>
Re: Flink metrics missing from UI 1.7.2
> I have done this (actually I do it in my flink-conf.yaml), but I am not seeing any metrics at all in the Flink UI, > let alone the latency tracking. The latency tracking itself does not seem to be exported to datadog (should it be?) The latency metrics are job metrics, and are not shown in the Flink UI. They are available via the REST API, and I believe they should also be exported to datadog. You will find them at http://localhost:8081/jobs//metrics with IDs like latency.source_id.bc764cd8ddf7a0cff126f51c16239658.operator_id.ea632d67b7d595e5b851708ae9ad79d6.operator_subtask_index.0.latency_p90 On Sat, Mar 23, 2019 at 1:53 PM Padarn Wilson wrote: > Thanks David. > > I have done this (actually I do it in my flink-conf.yaml), but I am not > seeing any metrics at all in the Flink UI, let alone the latency tracking. > The latency tracking itself does not seem to be exported to datadog (should > it be?) > > > > On Sat, Mar 23, 2019 at 8:43 PM David Anderson > wrote: > >> Because latency tracking is expensive, it is turned off by default. You >> turn it on by setting the interval; that looks something like this: >> >> env.getConfig().setLatencyTrackingInterval(1000); >> >> The full set of configuration options is described in the docs: >> https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#metrics >> >> David Anderson | Training Coordinator >> Follow us @VervericaData >> >> -- >> Join Flink Forward - The Apache Flink Conference >> Stream Processing | Event Driven | Real Time >> >> >> On Sat, Mar 23, 2019 at 1:03 PM Padarn Wilson wrote: >> > >> > Hi User, >> > >> > I am running Fink 1.7.2 on Kubernetes in a setup with task manager and >> job manager separate. >> > >> > I'm having trouble seeing the metrics from my Flink job in the UI >> dashboard. Actually I'm using the Datadog reporter to expose most of my >> metrics, but latency tracking does not seem to be exported. >> > >> > Is there anything extra that needs to be enabled to make sure metrics >> are exported and viewable to the Flink UI? >> > >> > Thanks >> >>>
Re: Flink metrics missing from UI 1.7.2
Because latency tracking is expensive, it is turned off by default. You turn it on by setting the interval; that looks something like this: env.getConfig().setLatencyTrackingInterval(1000); The full set of configuration options is described in the docs: https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#metrics David Anderson | Training Coordinator Follow us @VervericaData -- Join Flink Forward - The Apache Flink Conference Stream Processing | Event Driven | Real Time On Sat, Mar 23, 2019 at 1:03 PM Padarn Wilson wrote: > > Hi User, > > I am running Fink 1.7.2 on Kubernetes in a setup with task manager and job manager separate. > > I'm having trouble seeing the metrics from my Flink job in the UI dashboard. Actually I'm using the Datadog reporter to expose most of my metrics, but latency tracking does not seem to be exported. > > Is there anything extra that needs to be enabled to make sure metrics are exported and viewable to the Flink UI? > > Thanks >
Flink metrics missing from UI 1.7.2
Hi User, I am running Fink 1.7.2 on Kubernetes in a setup with task manager and job manager separate. I'm having trouble seeing the metrics from my Flink job in the UI dashboard. Actually I'm using the Datadog reporter to expose most of my metrics, but latency tracking does not seem to be exported. Is there anything extra that needs to be enabled to make sure metrics are exported and viewable to the Flink UI? Thanks
