[
https://issues.apache.org/jira/browse/DISPATCH-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17311595#comment-17311595
]
Ken Giusti commented on DISPATCH-2007:
--------------------------------------
Hi Robbie - good point regarding setting session window limits in the case of
multiple sessions. Assuming we determine a "reasonable minimum" for the
session window we'd have to set a limit on the number of simultaneously active
sessions in order to prevent unreasonable memory growth. If we consider the
hard limit on the number of active sessions allowed on a connection (approx 32K
in Proton) and pick some arbitrary smallish per session window limit (say
1MByte), then in this example we have a hypothetical session limit of 32
_Gigabytes_ per inter-router connection...
... not to mention the overhead (in both memory and CPU) to manage that number
of sessions.
So yeah, we'd have to set a max session limit per connection considerably less
than the hard limit set by Proton. And unless we're limiting the total number
of active link routes and streaming messages allowed on an inter-router
connection to the same max session limit we're right back to the problem we
have now: multiple links sharing bandwidth on a single session.
Robbie I think the key point you make is that this problem is not really one of
spreading link output across multiple sessions, but instead scheduling link
output in a fair manner so that no single link can monopolize a session's
bandwidth.
Right now, the link output processing done by the router ignores sessions
completely.
In the current implementation an inter-router connection maintains lists of
links that need processing. There is a separate list for each priority
supported by the router (10 lists). When work is available for the link it is
added to tail of the proper list based on the link's assigned priority.
When the I/O thread gets around to processing the connection it processes each
list of links in priority order. The router will write as much outgoing
message data to the link as it can, stopping only when it either runs out of
messages, credit, or is blocked either by pn_link_send() or hitting the Q3
limit. The router will then proceed to the next link in the priority list, or
the next priority list if no next link.
See [qdr_connection_process() in
src/router_core/connections.c|https://github.com/apache/qpid-dispatch/blob/master/src/router_core/connections.c#L308]
Important detail: when a link does become blocked due to Q3 or pn_link_send()
backpressure it remains on the work list *_in its current position_*! A
no-brainer fix would be to move the blocked link to the end of the work list!
That would be a good starting point..
We may want to make the output scheduler more session aware. Might be worth
trying a POC and see if it buys us anything.
> Need better allocation of sessions across inter-router links
> ------------------------------------------------------------
>
> Key: DISPATCH-2007
> URL: https://issues.apache.org/jira/browse/DISPATCH-2007
> Project: Qpid Dispatch
> Issue Type: Improvement
> Components: Router Node
> Affects Versions: 1.15.0
> Reporter: Ken Giusti
> Assignee: Ken Giusti
> Priority: Major
> Fix For: 1.16.0
>
>
> Inter-router connections establish several links for passing traffic between
> routers. Session allocation across the links is currently:
> * one session dedicated to both router control links
> * one session for EACH priority link pair (in/out) - 10 sessions total
> * one session used by all link route links
> * one session used by all streaming link
> There is an issue with sharing the session in the case of streaming & link
> routes. Due to the way session flow control works all links on the session
> share bandwidth. This means one slow link can force flow control on all of
> its peer links on the session.
> Proton allows for at most 32K sessions per connection. We should make an
> attempt to use dedicated sessions for link route and streaming links, sharing
> only when the number of link route + streaming links exceeds the proton
> limit.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
