zhijiang created FLINK-14396:
--------------------------------
Summary: Implement rudimentary non-blocking network output
Key: FLINK-14396
URL: https://issues.apache.org/jira/browse/FLINK-14396
Project: Flink
Issue Type: Task
Components: Runtime / Network
Reporter: zhijiang
Assignee: zhijiang
Fix For: 1.10.0
Considering the mailbox model and unaligned checkpoints requirements in future,
task network output should be non-blocking. In other words, as long as output
is available, it should never block for a subsequent/future single record write.
In the first version, we only implement the non-blocking output for the most
regular case, and do not solve the following cases which still keep the
previous behavior.
* Big record which might span multiple buffers
* Flatmap-like operators which might emit multiple records in every process
* Broadcast watermark which might request multiple buffers at a time
The solution is providing the RecordWriter#isAvailable method and respective
LocalBufferPool#isAvailable for judging the output beforehand. As long as there
is at-least one available buffer in LocalBufferPool, the RecordWriter is
available for network output in most cases. This doesn’t include runtime
handling of this non-blocking and availability behavior in StreamInputProcessor
Note: It requires the minimum number of buffers in output LocalBufferPool
adjusting to (numberOfSubpartitions + 1) and also adjusting the monitor of
backpressure future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
