[
https://issues.apache.org/jira/browse/DISPATCH-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17077545#comment-17077545
]
ASF GitHub Bot commented on DISPATCH-1545:
------------------------------------------
kgiusti commented on pull request #716: DISPATCH-1545: Prevent Head-of-line
blocking on shared inter-router l…
URL: https://github.com/apache/qpid-dispatch/pull/716
…inks
This patch prevents head-of-line blocking on shared links by
dynamically assigning a dedicated inter-router link for the transfer
of a streaming message.
**Patch Notes**
There are 3 primary changes made by this patch:
- Classification of inbound messages as streaming
- Modifications to the forwarding path to handle streaming messages
- The creation of a per-connection pool of links available for streaming a
message
**Classification**
There is no practical way to determine the length of an inbound message
prior to its complete arrival. This means the router has to monitor the size
of an incoming message and classify it as streaming once it gets "too big" (and
has not yet finished arriving). This requires defining a size threshold over
which the message is classified as streaming.
Note well that this classification MUST happen BEFORE the message is
forwarded by the core: the core uses the classification to determine the proper
outgoing link.
This means the I/O thread must buffer enough of an incoming message to make
the classification before the message can be handed to the core for forwarding.
In other words an incoming message will be buffered until it completely
arrives OR hits the "too big" threshold.
Rather than introducing yet another user-facing "streaming message
threshold" configuration knob (and the additional code complexity, test matrix
dimension and user documentation it would require), this patch uses the
existing Q2 threshold to classify the message. The Q2 threshold makes sense: it
already sets a topmost limit for incoming message buffering. Once Q2 is hit
the router MUST forward the message in order to unblock the link, so Q2
essentially forces us to make that classification in any case.
The patch does this buffering/threshold check in the rx handler in
router_node.c at the point in the code where the message has yet to be
forwarded. Once forwarded further changes to the Q2 state are irrelevant to
the classification process.
**Forwarding Path Modifications**
Once the classification is complete all streaming messages are flagged.
I've done this rather than constantly checking the
"qd_message_receive_complete()" throughout the message's lifecycle because the
receive status can change at any moment via the I/O thread. This opens to
possibility of race conditions.
Each forwarder has been updated to check the message's streaming flag during
the outbound link selection process. If the candidate link's connection
supports outgoing streaming links the forwarder will obtain a dedicated
outgoing anonymous link for forwarding the message. This link is either
created on demand or obtained from the connection's pool of free streaming
links.
**Connection Link Pool**
Every connection that supports outgoing streaming links maintains a free
streaming link pool. This pool has been added to the qdr_connection_t
structure. This pool is initialized empty. When a forwarder needs a dedicated
streaming link it will check the pool for an available link. If none are
available a new qdr_link_t will be created for the outgoing message. This link
is marked as "pooled" so when the message is finally delivered the qdr_link_t
will be returned to the pool, ready for the next streaming message.
The pool is designed to keep at least 128 standby links. More links can be
allocated, but links in excess of the 128 will eventually be released. This is
done by the streaming_link_scrubber core module added by this patch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Streaming deliveries can be delayed by head-of-line blocking on inter-router
> links
> ----------------------------------------------------------------------------------
>
> Key: DISPATCH-1545
> URL: https://issues.apache.org/jira/browse/DISPATCH-1545
> Project: Qpid Dispatch
> Issue Type: Bug
> Components: Router Node
> Affects Versions: 1.10.0
> Reporter: Ted Ross
> Assignee: Ken Giusti
> Priority: Major
> Fix For: 1.12.0
>
>
> Message-routed deliveries (for the same priority) are forwarded over the same
> inter-router link when moving from router to router. If a streaming delivery
> is flowing over such a link, subsequent deliveries on that link will be
> delayed until the streaming delivery is complete. A solution is needed that
> allows an arbitrary number of streaming deliveries to concurrently flow from
> router to router.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
