Hi Ufuk,
Thank you for your instant reply and I have got the basic idea.
I am researching flink code now, and if I have further specific
questions next time, I will send to [email protected] .
Best wishes,
Zhijiang Wang
------------------------------------------------------------------发件人：Ufuk
Celebi <[email protected]>发送时间：2015年8月10日(星期一) 17:13收件人：user
<[email protected]>，wangzhijiang999 <[email protected]>主 题：Re:
答复：how to understand the flink flow controlGood to hear. Answers to your
questions are inline.I think we should move the discussion to the [email protected]
list though if there are more questions.On Mon, Aug 10, 2015 at 9:29 AM,
wangzhijiang999 <[email protected]> wrote:Hi Ufuk, Thank you for
your detail and clear explaination! I reviewed the code based on your info and
got it clearly.For local transfer: When producer emits results to
ResultPartition, it will request Buffer from the pool. If there are no
available Buffer, it will wait .Yes, the buffer request is blocking. The
ResultPartition of producer is the InputGate of Consumer, when consumer read
buffer from InputGate and deserializer the buffer, the Buffer will be recycled
,so the producer can request Buffer again.The producer never requests a buffer
from the input gate, but from its own pool. If the input gate is slow to
recycle its buffers, the producer will become slow as well. But you are right
that the producer will not be able to request a buffer if the input gate never
recycles them. So if the consumer slows, the Buffer in the ResultPartition of
producer can not be recycled quickly, resulting in producer has no available
Buffer to emit data.If the producer waits the available Buffer to emit, it can
not process elements and read next data from its InputGate, resulting in
slowing the producer.Yes. The task thread is both consuming and producing data.
So if its blocked on the output side, the input is not consumed, which
propagates back pressure further. For remote transfer: The InputGate and
ResultPartition for each task are separate 1. For producer the Buffer in
ResultPartition will be recycled when producer write them to the channel.
2. For consumer the Buffer will be recycled when consumer read it from
InputGate and deserializer it . The consumer need to request Buffer when
reading data from channel and put them to InputGate. 3. When consumer
slows and there are no available Buffer , the consumer will not read data from
channel, so it will affect the producer writing data to channel and the
producer will have no available Buffer to emit result at last resulting in
slowing producer.Yes but there is some logic involved to stop reading from the
actual TCP channel to back pressure the remote producer. Essentially the
selector read event is unsubscribed and on the output side there is Netty's
watermark mechanism, which prevents further data from being written to the TCP
channel.
