ted-ross commented on a change in pull request #560: NO-JIRA: add a document explaining the router's threading implementat… URL: https://github.com/apache/qpid-dispatch/pull/560#discussion_r367988503
########## File path: docs/notes/threading-info.txt ########## @@ -0,0 +1,563 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + + +===================================================== +Router Threading and Interprocess Communication Guide +===================================================== + +The Qpid Dispatch Router (qdrouterd) threading architecture is based +on two classes of threads: + + *) A Worker thread which interfaces with the Proton subsystem, and + *) The Core Routing thread which manages the routing database and +performs forwarding. + +In a running router there is a single Core Routing thread (core) and +one or more Worker threads. The number of Worker threads is +determined by the router configuration setting "workerThreads", which +defaults to four. + +IPC between these threads is done using Action queues, Work queues, +and manipulation of shared data. + + +Worker Threads +============== + +The Worker thread is responsible for interfacing with the Proton +subsystem. Only a worker thread can safely call directly into the +Proton library. The core thread must communicate with a worker thread +in order to have the worker manipulate Proton state on the core's +behalf. + +The Proton subsystem limits concurrency to a single connection. That +is, only one thread can be processing a given Proton connection (and +all of its child elements, such as links and deliveries) at a time. +The router honors this requirement by restricting access to a given +Proton connection (and its children) to a single worker thread at a +time. To say this another way, a particular Proton connection can be +processed by any worker threads but not concurrently. + +A worker thread is driven by the Proton proactor API. The worker's +main loop blocks on the proactor waiting for events, processes +incoming events, then blocks again. + + +Core Thread +=========== + +The one core thread has several responsibilities, including: + + *) Managing the forwarding state + *) Forwarding messages across the router + *) Forwarding disposition and settlement state changes across the router + *) Managing link credit flow + *) Running the management Agent + +The core thread can be thought of as sitting in between the worker +threads, moving messages and state between them. + +When a worker thread needs to forward a received message it passes the +message and associated delivery state to the core thread. The core +thread uses address information from the message to determine the +outgoing link(s) for the message. The core then queues the message to +the proper outgoing link(s). The core wakes the worker thread(s) (via +the Proton proactor) so the message(s) can be written out the Proton +link. + +When delivery disposition or settlement changes are detected by a +worker thread it notifies the core thread. The core thread then +finds the link or delivery to which the changes need to be propagated +to. This results in the core thread setting the new state in the +link/delivery and waking a worker thread to update the new state in +Proton. + +The core also manages credit flow. The core grants credit to inbound +links. The core grants an initial batch of credit to a non-anonymous +link (a link with a target address) when the target address is +available for routing. The core will continue granting credit to the +link as long as the address is routable. The core ties the +replenishment of credit with the settlement of messages arriving on +the link: when the message is settled a single credit is granted by +the core. All credit flow operations involve having the core put a +credit flow work event on the proper inbound link then waking the +worker thread to update the credit in proton. + +The core can be the destination or source of AMQP message flows +from/to clients outside the router. This is used by services like the +Management Agent to receive management requests and send responses to +management clients. The core sends and receives these messages in +exactly the same way it forwards routed traffic - via the worker +threads. + +The core thread's main loop is driven by a work queue (the +qdr_core_t's action_list). The core thread blocks on a mutex +condition waiting for work. When a worker thread needs a service from +the core thread, the worker queues a work item to the action_list then +triggers the condition. This causes the core thread to unblock and +service the action_list work items. + + +Source Code Conventions +======================= + +In general, core thread code and data structure have names that are +prefixed by "qdr_". Functions that can only be run on the core thread +- that cannot be invoked from the context of a worker thread - are +suffixed with "_CT". For example "qdr_update_delivery_CT()" is a core +function that must never be invoked by a worker thread. + +Code that is private to the Core thread is in the "src/router_core" +subdirectory of the source code repo. + +Worker thread code does not have a particular naming convention. Most +worker thread-only APIs do start with "qd_", but this prefix is not +reserved for worker thread APIs. + + +Principal Data Structures +========================= + +There are many data structures that are used throughout the code. +This section will overview a few of those data structures which are +involved in inter-thread communications. + +qdr_delivery_t +-------------- + +The qdr_delivery_t structure represents the router's view of a Proton +delivery instance (pn_delivery_t). Its definition is private to core, +however core defines an API that allows a worker to concurrently +access parts of this structure. + +There is a one-to-one relationship between a qdr_delivery_t and a +Proton pn_delivery_t. When an incoming message arrives at the router +a new incoming qdr_delivery_t is created and tied to the message's +pn_delivery_t. When a message is forwarded a new outgoing +qdr_delivery_t is created to associate with the outgoing link's +pn_delivery_t. + +It is important to understand that pn_delivery_t's - and by extension +qdr_delivery_t's - are "owned" by the parent pn_link_t +(qdr_link_t). This means that a delivery object cannot be moved from +an inbound link to an outbound link - which seems counterintiutive to +the process of routing a message. + +In reality when an inbound delivery is routed, the router create a new +outbound delivery for each outbound link. These outbound deliveries +are also represented by their own pn_delivery_t and qdr_delivery_t +instances. + +The core thread links the inbound and outbound qdr_deliver_t's Review comment: typo ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
