Hi Piotrek,

Thank for information, looks like isBackPressured ( and in future 
backPressuredTimeMsPerSecond) is more useful for our simple monitoring 
purposes. Looking forward for updated blog post

Thanks,
Alexey

________________________________
From: Piotr Nowojski <[email protected]>
Sent: Friday, March 19, 2021 5:01 AM
To: Alexey Trenikhun <[email protected]>
Cc: Flink User Mail List <[email protected]>
Subject: Re: inputFloatingBuffersUsage=0?

Hi,

clarification about the 2nd part. Required memory is one single exclusive 
buffer per channel, so if you are running low on memory, floating buffers are 
one of the first to go, hence you could observe some `0` for floating buffers.

For analysing backpressure I would recommend to use a different metric: 
`isBackPressured`. It was added in Flink 1.10, after this blog post had been 
written. Since 1.11 we also have `idleTimeMsPerSecond` that might be helpful.

In 1.13 all of this will become even more simple, with changes to Web UI and an 
addition of `backPressuredTimeMsPerSecond` and `busyTimeMsPerSecond`. We are 
planning to release a new updated blog post about analysing backpressure in the 
following weeks.

Best,
Piotrek

pt., 19 mar 2021 o 11:57 Piotr Nowojski 
<[email protected]<mailto:[email protected]>> napisał(a):
Hi Alexey,

Have you looked at the documentation [1]?

> inPoolUsage An estimate of the input buffers usage. (ignores 
> LocalInputChannels) Gauge
> inputFloatingBuffersUsage An estimate of the floating input buffers usage. 
> (ignores LocalInputChannels) Gauge
> inputExclusiveBuffersUsage An estimate of the exclusive input buffers usage. 
> (ignores LocalInputChannels) Gauge

Second reason might be simply that your job/task didn't manage to acquire all 
of the recommended memory. Recommended memory is `floatingBuffers + 
exclusiveBuffers * numberOfChannels`. Required memory is just `1 * 
numberOfChannels`. If there is not enough memory to satisfy recommended values, 
Flink will work with anything between recommended and required amounts.

Best,
Piotrek

[1] 
https://ci.apache.org/projects/flink/flink-docs-stable/ops/metrics.html#default-shuffle-service

czw., 18 mar 2021 o 08:30 Alexey Trenikhun 
<[email protected]<mailto:[email protected]>> napisał(a):
Hello,
While trying to investigate back pressure using hints from [1], I've noticed 
that flink_taskmanager_job_task_buffers_inPoolUsage and 
flink_taskmanager_job_task_buffers_inputFloatingBuffersUsage are always 0, 
which looks suspicious, are these metrics still populated ?

Thanks,
Alexey

[1] https://flink.apache.org/2019/07/23/flink-network-stack-2.html
Apache Flink: Flink Network Stack Vol. 2: Monitoring, Metrics, and that 
Backpressure 
Thing<https://flink.apache.org/2019/07/23/flink-network-stack-2.html>
Flink Network Stack Vol. 2: Monitoring, Metrics, and that Backpressure Thing. 
23 Jul 2019 Nico Kruber & Piotr Nowojski . In a previous blog post, we 
presented how Flink’s network stack works from the high-level abstractions to 
the low-level details.This second blog post in the series of network stack 
posts extends on this knowledge and discusses monitoring network-related 
metrics to identify ...
flink.apache.org<http://flink.apache.org>

Reply via email to