inputQueueSize metric
Regarding the inputQueueSize metric. What does it express after all? Observing the inputQueueSize and inPoolUsage metrics in Prometheus, I found that when the inPoolUsage is high, the inputQueueSize is low. Thisobservation does not match the description of the inputQueueSize. Therefore, does this metric express the size of the input buffers of each task, or does it express how full the input buffers of a task are? Thank you in advance.
inputQueueSize metric
Regarding the inputQueueSize metric. What does it express after all? Observing the inputQueueSize and inPoolUsage metrics in Prometheus, Ifound that when the inPoolUsage is high, the inputQueueSize is low. Thisobservation does not match the description of the inputQueueSize.Therefore, does this metric express the size of the input buffers ofeach task, or does it express how full the input buffers of a task are? Thank you in advance.
RE: Re: Re: inputQueueSize metric
Thank you for your previous response Feng. It clears things up a little. Regarding the inputQueueSize metric. What does it express after all? Observing the inputQueueSize and inPoolUsage metrics in Prometheus, I found that when the inPoolUsage is high, the inputQueueSize is low. This observation does not match the description of the inputQueueSize. Therefore, does this metric express the size of the input buffers of each task, or does it express how full the input buffers of a task are? Thank you in advance. On 2023/11/28 16:11:59 Feng Jin wrote: > Hi Dimitris > > Yes, the inputQueueLength metric can only represent the number of buffers, > and cannot obtain the actual size of the messages. > > Regarding the inputQueueSize metric, it ignores the LocalChannel. > Therefore, when tasks are connected through the Forward partition strategy, > it will consistently display a value of 0. > You may consider altering the partition strategy to the rebalance strategy > for testing purposes.st it. > > > Best, > Feng. > > > On Tue, Nov 28, 2023 at 2:32 AM Dimitris Banelas via user < > user@flink.apache.org> wrote: > > > As per the docs, the `inputQueueLength` metric refers to the number of > > queued input buffers, and cannot be used on its own in order to > > determine buffered records. > > > > For instance, if I know that there are 5 queued input buffers, I cannot > > conclude anything regarding buffered records if the size of each queue > > is not known. > > > > > > On 2023/11/27 17:28:08 Feng Jin wrote: > > > Hi Dimitris > > > > > > Maybe you can use the `inputQueueLength` metric. > > > > > > > > > Best, > > > Feng > > > > > > On Tue, Nov 28, 2023 at 12:07 AM Dimitris Mpanelas via user < > > > user@flink.apache.org> wrote: > > > > > > > Hello, > > > > > > > > I am trying to determine the buffered records in the input buffers of > > a > > > > task. I found the inputQueueSize metric. According to the docs it > > is "The > > > > real size of queued input buffers in bytes". The docs also state > > that "The > > > > size for local input channels is always 0 since the local channel take > > > > records directly from the output queue". > > > > > > > > What does this metric truly represent? In which cases should it be > > 0 and > > > > non 0? Is there a way to determine buffered records (or at least > > something > > > > similar) by using this metric? > > > > > > > > Dimitris. > > > > > > > > > > > > > >
Re: Re: inputQueueSize metric
Hi Dimitris Yes, the inputQueueLength metric can only represent the number of buffers, and cannot obtain the actual size of the messages. Regarding the inputQueueSize metric, it ignores the LocalChannel. Therefore, when tasks are connected through the Forward partition strategy, it will consistently display a value of 0. You may consider altering the partition strategy to the rebalance strategy for testing purposes.st it. Best, Feng. On Tue, Nov 28, 2023 at 2:32 AM Dimitris Banelas via user < user@flink.apache.org> wrote: > As per the docs, the `inputQueueLength` metric refers to the number of > queued input buffers, and cannot be used on its own in order to > determine buffered records. > > For instance, if I know that there are 5 queued input buffers, I cannot > conclude anything regarding buffered records if the size of each queue > is not known. > > > On 2023/11/27 17:28:08 Feng Jin wrote: > > Hi Dimitris > > > > Maybe you can use the `inputQueueLength` metric. > > > > > > Best, > > Feng > > > > On Tue, Nov 28, 2023 at 12:07 AM Dimitris Mpanelas via user < > > user@flink.apache.org> wrote: > > > > > Hello, > > > > > > I am trying to determine the buffered records in the input buffers of > a > > > task. I found the inputQueueSize metric. According to the docs it > is "The > > > real size of queued input buffers in bytes". The docs also state > that "The > > > size for local input channels is always 0 since the local channel take > > > records directly from the output queue". > > > > > > What does this metric truly represent? In which cases should it be > 0 and > > > non 0? Is there a way to determine buffered records (or at least > something > > > similar) by using this metric? > > > > > > Dimitris. > > > > > > > > >
RE: Re: inputQueueSize metric
As per the docs, the `inputQueueLength` metric refers to the number of queued input buffers, and cannot be used on its own in order to determine buffered records. For instance, if I know that there are 5 queued input buffers, I cannot conclude anything regarding buffered records if the size of each queue is not known. On 2023/11/27 17:28:08 Feng Jin wrote: > Hi Dimitris > > Maybe you can use the `inputQueueLength` metric. > > > Best, > Feng > > On Tue, Nov 28, 2023 at 12:07 AM Dimitris Mpanelas via user < > user@flink.apache.org> wrote: > > > Hello, > > > > I am trying to determine the buffered records in the input buffers of a > > task. I found the inputQueueSize metric. According to the docs it is "The > > real size of queued input buffers in bytes". The docs also state that "The > > size for local input channels is always 0 since the local channel take > > records directly from the output queue". > > > > What does this metric truly represent? In which cases should it be 0 and > > non 0? Is there a way to determine buffered records (or at least something > > similar) by using this metric? > > > > Dimitris. > > > > >
Re: inputQueueSize metric
Hi Dimitris Maybe you can use the `inputQueueLength` metric. Best, Feng On Tue, Nov 28, 2023 at 12:07 AM Dimitris Mpanelas via user < user@flink.apache.org> wrote: > Hello, > > I am trying to determine the buffered records in the input buffers of a > task. I found the inputQueueSize metric. According to the docs it is "The > real size of queued input buffers in bytes". The docs also state that "The > size for local input channels is always 0 since the local channel take > records directly from the output queue". > > What does this metric truly represent? In which cases should it be 0 and > non 0? Is there a way to determine buffered records (or at least something > similar) by using this metric? > > Dimitris. > >
inputQueueSize metric
Hello, I am trying to determine the buffered records in the input buffers of a task. I found the inputQueueSize metric. According to the docs it is "The real size of queued input buffers in bytes". The docs also state that "The size for local input channels is always 0 since the local channel take records directly from the output queue". What does this metric truly represent? In which cases should it be 0 and non 0? Is there a way to determine buffered records (or at least something similar) by using this metric? Dimitris.
