[
https://issues.apache.org/jira/browse/FLINK-10462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006711#comment-17006711
]
Yingjie Cao commented on FLINK-10462:
-------------------------------------
Recently, we encountered the same problem under different scenario and we found
connection reuse is important for short queries. I create a new Jira
FLINK-15455 which is related to this one.
To restrict the number of connections, I think we can add a new option which
indicates the maximum number of connections between two Taskmanagers and then
we map the connectionIndex generated by IntermediateResult to a new value
smaller than the maximum connection number, for example, we can do like this:
{code:java}
newConnectionIndex = oldConnectionIndex % maxNumConnections{code}
Everything else can keep unchanged and we can set the default value of
maxNumConnections to Integer.MAX_VALUE which can keep the behavior of network
connection unchanged so we don't need to worry about any potential regressions.
If the users find that there are too many connections, then he/she can set the
maxNumConnections to a relative smaller value.
What do you think?
> Remove ConnectionIndex for further sharing tcp connection in credit-based
> mode
> -------------------------------------------------------------------------------
>
> Key: FLINK-10462
> URL: https://issues.apache.org/jira/browse/FLINK-10462
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
> Affects Versions: 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.6.0, 1.6.1
> Reporter: zhijiang
> Assignee: zhijiang
> Priority: Minor
>
> Every {{IntermediateResult}} generates a random {{ConnectionIndex}} which
> will be included in {{ConnectionID}}.
> The {{RemoteInputChannel}} requests to establish tcp connection via
> {{ConnectionID}}. That means one tcp connection may be shared by multiple
> {{RemoteInputChannel}} {{s which have the same ConnectionID}}. To do so, we
> can reduce the physical connections between two \{{TaskManager}} s, and it
> brings benefits for large scale jobs.
> But this sharing is limited only for the same {{IntermediateResult}}, and I
> think it is mainly because we may temporarily switch off {{autoread}} for the
> channel during back pressure in previous network flow control. For
> credit-based mode, the channel is always open for transporting different
> intermediate data, so we can further share the tcp connection for different
> {{IntermediateResults}} to remove the limit.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
