Hi Piotrek, Thank for information, looks like isBackPressured ( and in future backPressuredTimeMsPerSecond) is more useful for our simple monitoring purposes. Looking forward for updated blog post
Thanks, Alexey ________________________________ From: Piotr Nowojski <[email protected]> Sent: Friday, March 19, 2021 5:01 AM To: Alexey Trenikhun <[email protected]> Cc: Flink User Mail List <[email protected]> Subject: Re: inputFloatingBuffersUsage=0? Hi, clarification about the 2nd part. Required memory is one single exclusive buffer per channel, so if you are running low on memory, floating buffers are one of the first to go, hence you could observe some `0` for floating buffers. For analysing backpressure I would recommend to use a different metric: `isBackPressured`. It was added in Flink 1.10, after this blog post had been written. Since 1.11 we also have `idleTimeMsPerSecond` that might be helpful. In 1.13 all of this will become even more simple, with changes to Web UI and an addition of `backPressuredTimeMsPerSecond` and `busyTimeMsPerSecond`. We are planning to release a new updated blog post about analysing backpressure in the following weeks. Best, Piotrek pt., 19 mar 2021 o 11:57 Piotr Nowojski <[email protected]<mailto:[email protected]>> napisał(a): Hi Alexey, Have you looked at the documentation [1]? > inPoolUsage An estimate of the input buffers usage. (ignores > LocalInputChannels) Gauge > inputFloatingBuffersUsage An estimate of the floating input buffers usage. > (ignores LocalInputChannels) Gauge > inputExclusiveBuffersUsage An estimate of the exclusive input buffers usage. > (ignores LocalInputChannels) Gauge Second reason might be simply that your job/task didn't manage to acquire all of the recommended memory. Recommended memory is `floatingBuffers + exclusiveBuffers * numberOfChannels`. Required memory is just `1 * numberOfChannels`. If there is not enough memory to satisfy recommended values, Flink will work with anything between recommended and required amounts. Best, Piotrek [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/metrics.html#default-shuffle-service czw., 18 mar 2021 o 08:30 Alexey Trenikhun <[email protected]<mailto:[email protected]>> napisał(a): Hello, While trying to investigate back pressure using hints from [1], I've noticed that flink_taskmanager_job_task_buffers_inPoolUsage and flink_taskmanager_job_task_buffers_inputFloatingBuffersUsage are always 0, which looks suspicious, are these metrics still populated ? Thanks, Alexey [1] https://flink.apache.org/2019/07/23/flink-network-stack-2.html Apache Flink: Flink Network Stack Vol. 2: Monitoring, Metrics, and that Backpressure Thing<https://flink.apache.org/2019/07/23/flink-network-stack-2.html> Flink Network Stack Vol. 2: Monitoring, Metrics, and that Backpressure Thing. 23 Jul 2019 Nico Kruber & Piotr Nowojski . In a previous blog post, we presented how Flink’s network stack works from the high-level abstractions to the low-level details.This second blog post in the series of network stack posts extends on this knowledge and discusses monitoring network-related metrics to identify ... flink.apache.org<http://flink.apache.org>
