[
https://issues.apache.org/jira/browse/FLINK-8784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16377622#comment-16377622
]
Stephan Ewen commented on FLINK-8784:
-------------------------------------
The choice to not share the network buffer pool, but partition it was made to
guard against starvation effects. Flink can never know in which order or speed
any connection will progress (that depends on the behavior of the user code),
so reserving enough memory for each connection to be able to do its maximum
work makes it safe (against deadlocks and against bad performance).
For streaming jobs, I am not sure we should change that.
For batch jobs, we can probably improve the behavior better through better
scheduling than through adjusting the network stack memory model.
> Share network buffer pool among different partition operators
> -------------------------------------------------------------
>
> Key: FLINK-8784
> URL: https://issues.apache.org/jira/browse/FLINK-8784
> Project: Flink
> Issue Type: Improvement
> Components: TaskManager
> Reporter: Garrett Li
> Priority: Major
>
> A logical network connection exists for each point-to-point exchange of data
> over the network, which typically happens at repartitioning- or broadcasting
> steps (shuffle phase). et
> Currently Flink divide the network buffer pool evenly for every partition
> operators. Sometimes it could waste too many network buffers since we might
> have a lot of repartitioning and broadcasting in the one Flink job, and NOT
> all of the partition operators in the same job have the same traffic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
