http://git-wip-us.apache.org/repos/asf/qpid-dispatch/blob/f77f92cb/docs/books/user-guide/reliability.adoc ---------------------------------------------------------------------- diff --git a/docs/books/user-guide/reliability.adoc b/docs/books/user-guide/reliability.adoc new file mode 100644 index 0000000..2826c55 --- /dev/null +++ b/docs/books/user-guide/reliability.adoc @@ -0,0 +1,701 @@ +//// +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License +//// + +[id='reliability'] += Reliability + +//// +There's a lot of really good information here, but most of it is best-suited at a level higher than this book. This book is really about configuring a router in an established topology, but most of the information in "Reliability" is about concepts and best practices to keep in mind as you design the topology. Alternatively, I could also see taking the examples presented here and turning them into a tutorial to teach some of the fundamental concepts inherent within router reliability. +//// + +In general, in a broker based architecture, the reliability feature is strictly related to the "store and forward" mechanism offered by each broker. Thanks to persistent journals, a broker can offer fault tolerance thus avoiding message loss; of course, it is not so true when messages are stored only in a volatile memory. + +This is completely different using {RouterName}, because each router neither takes ownership of messages nor stores them in a persistent storage. In this case, the reliability feature is offered by *path redundancy* which provides the possibility to reach the destination on different paths through the router network. In normal conditions, the best path is always chosen in terms of lowest cost but, when one or more routers go down, the topology is revisited by all remained routers and new paths are processed in order to reach always each destination. Of course, it means that the reliability is strictly related to the network topology the user chooses for his solution. + +Because a solution based on {RouterName} could be made not only by routers but by brokers too, the reliability is improved with persistent storage on them which add not only fault tolerance but temporal decoupling as well; without "store and forward" feature offered by brokers, the temporal decoupling is not possible only with routers and direct peers, both senders and receivers; the receiver must be online at same time of the sender in order to receive messages. + +== Path Redundancy + +Offering path redundancy means designing the network topology in a way that even when one or more routers go down or even connections between them, each destination is always reachable following alternate paths through the routers that are still part of the network. + +Consider the following simple scenario : + +* a network with three routers "Router.A", "Router.B" and "Router.C". +* the "Router.A" is connected to both "Router.B" and "Router.C". +* the "Router.C is connected to the "Router.B". +* all three routers listen for client connections. +* a sender client connects to the "Router.A" in order to send messages to a receiver client. +* a receiver client connects to the "Router.B" initially in order to receive messages from the sender peer. + +.Path Redundancy Enabled Topology +image::path-redundancy-01.png[Path Redundancy Enabled Topology, align="center"] + +The "Router.A" configuration is something like following. + +[options="nowrap"] +---- +router { + mode: interior + id: Router.A +} + +listener { + host: 0.0.0.0 + port: 6000 + authenticatePeer: no +} + +connector { + name: INTER_ROUTER_B + addr: 127.0.0.1 + port: 5001 + role: inter-router +} + +connector { + name: INTER_ROUTER_C + addr: 127.0.0.1 + port: 5002 + role: inter-router +} +---- + +There is only one _listener_ in order to accept client connections and two _connector_ entities for connecting to the other two routers. + +The "Router.B" configuration is the following. + +[options="nowrap"] +---- +router { + mode: interior + id: Router.B +} + +listener { + addr: 0.0.0.0 + port: 5001 + authenticatePeer: no + role: inter-router +} + +listener { + host: 0.0.0.0 + port: 6001 + authenticatePeer: no +} +---- + +It has two _listener_ entities in order to listen for connections from clients and from other routers in the network (in this case from the "Router.A" and "Router.C"). + +Finally, quite similar is the "Router.C" configuration. + +[options="nowrap"] +---- +router { + mode: interior + id: Router.C +} + +listener { + addr: 0.0.0.0 + port: 5002 + authenticatePeer: no + role: inter-router +} + +listener { + host: 0.0.0.0 + port: 6002 + authenticatePeer: no +} + +connector { + name: INTER_ROUTER_B + addr: 127.0.0.1 + port: 5001 + role: inter-router +} +---- + +It has two _listener_ entities in order to listen for connections from clients and from other routers in the network (in this case from the "Router.A") and finally it has a _connector_ (for connecting to the "Router.B") + +Consider a sender client connected to "Router.A" and attached to `my_address` address which start to send messages (that is, 10 messages) and a receiver client connected to the "Router.B" and attached to the same address. + +Starting the receiver, it waits for messages with no output on the console. + +[options="nowrap"] +---- +$ sudo python simple_recv.py -a localhost:6001/my_queue -m 10 +---- + +Starting the sender, all the messages flow through "Router.A" and "Router.B" reaching the receiver; at this point the messages are all confirmed at sender side. + +[options="nowrap"] +---- +$ sudo python simple_send.py -a localhost:6001/my_queue -m 10 +all messages confirmed +---- + +At same time, the receivers shows the messages received through the "Router.B". + +[options="nowrap"] +---- +{u'sequence': 1L} +{u'sequence': 2L} +{u'sequence': 3L} +{u'sequence': 4L} +{u'sequence': 5L} +{u'sequence': 6L} +{u'sequence': 7L} +{u'sequence': 8L} +{u'sequence': 9L} +{u'sequence': 10L} +---- + +The path redundancy is provided by the other available path through the "Router.A", "Router.C" and then "Router.B". It means that if the connection between "Router.A" and "Router.B" goes down, the alternative path is used to reach the receiver. + +Now, consider a fault on the "Router.B"; the receiver is not reachable anymore on that path but it can connect to the "Router.C" in order to continue to receive messages from the sender which does not know what's happened and it can continue to send messages to the "Router.A" in order to reach the receiver. + +.Path Redundancy after Router Failure +image::path-redundancy-02.png[Path Redundancy after Router Failure, align="center"] + +The receiver is still reachable in order to get messages from the sender as displayed in the console output. + +[options="nowrap"] +---- +$ sudo python simple_recv.py -a localhost:6002/my_queue -m 10 +{u'sequence': 1L} +{u'sequence': 2L} +{u'sequence': 3L} +{u'sequence': 4L} +{u'sequence': 5L} +{u'sequence': 6L} +{u'sequence': 7L} +{u'sequence': 8L} +{u'sequence': 9L} +{u'sequence': 10L} +---- + +== Path Redundancy and Temporal Decoupling + +In order to have temporal decoupling in a solution based on {RouterName}, adding one or more brokers is a must for its "store and forward" feature. Choosing the right topology, it is possible to have a solution which offers reliability with both path redundancy and permanent storing for messages. + +Consider the following simple scenario : + +* a network with three routers "Router.A", "Router.B" and "Router.C" and finally a broker. +* the "Router.A" is connected to both "Router.B" and "Router.C". +* initially only the "Router.B" is connected to the broker. +* all three routers listen for client connections. +* a sender client connects to the "Router.A" in order to send messages to a queue in the broker. +* a receiver client connects to the "Router.A" in order to get messages from the queue in the broker. + +.Path Redundancy and Temporal Decoupling Enabled Topology +image::path-redundancy-temp-decoupling-01.png[Path Redundancy and Temporal Decoupling Enabled Topology, align="center"] + +The receiver client can be offline when the sender starts to send messages because they'll be stored into the queue permanently; coming back online, the receiver can get messages from the queue itself without message loss. + +The "Router.A" configuration is something like following. + +[options="nowrap"] +---- +router { + mode: interior + id: Router.A +} + +listener { + host: 0.0.0.0 + port: 6000 + authenticatePeer: no +} + +connector { + name: INTER_ROUTER_B + addr: 127.0.0.1 + port: 5001 + role: inter-router +} + +connector { + name: INTER_ROUTER_C + addr: 127.0.0.1 + port: 5002 + role: inter-router +} + +address { + prefix: my_queue + waypoint: yes +} +---- + +It has a _listener_ for accepting incoming connections from clients and two _connector_ entities in order to connect to the other routers. The queue named `my_queue` on the broker is exposed by a waypoint. + +The "Router.B" configuration is the following. + +[options="nowrap"] +---- +router { + mode: interior + id: Router.B +} + +listener { + addr: 0.0.0.0 + port: 5001 + authenticatePeer: no + role: inter-router +} + +listener { + host: 0.0.0.0 + port: 6001 + authenticatePeer: no +} + +connector { + name: BROKER + addr: 127.0.0.1 + port: 5672 + role: route-container +} + +address { + prefix: my_queue + waypoint: yes +} + +autoLink { + addr: my_queue + connection: BROKER + direction: in +} + +autoLink { + addr: my_queue + connection: BROKER + direction: out +} +---- + +It can accept incoming connections from clients and from other routers (in this case the "Router.A") and connects to the broker. The queue named `my_queue` on the broker is exposed by a waypoint with the related auto-links in both directions in order to send and receive messages to/from the queue itself. + +Finally, the simple "Router.C" configuration. + +[options="nowrap"] +---- +router { + mode: interior + id: Router.C +} + +listener { + addr: 0.0.0.0 + port: 5002 + authenticatePeer: no + role: inter-router +} + +listener { + host: 0.0.0.0 + port: 6002 + authenticatePeer: no +} +---- + +It can accept incoming connections from clients and from other routers (in this case the "Router.A"). Initially there is no connection between this router and the broker. + +First of all, thanks to the broker and its "store and forward" feature, the sender can connect to the "Router.A" and start to send messages even if the receiver is not online in that moment. Using the Python sample from the Qpid Proton library, the console output is like following. + +[options="nowrap"] +---- +$ sudo python simple_send.py -a localhost:6000/my_queue -m 10 +all messages confirmed +---- + +All messages are confirmed because they reached the queue inside the broker through "Router.A" and "Router.B"; it is confirmed using the `qdstat` tool. + +[options="nowrap"] +---- +$ sudo qdstat -b localhost:6001 -a +Router Addresses + class addr phs distrib in-proc local remote cntnr in out thru to-proc from-proc + ================================================================================================================= + local $_management_internal closest 1 0 0 0 0 0 0 0 0 + local $displayname closest 1 0 0 0 0 0 0 0 0 + mobile $management 0 closest 1 0 0 0 1 0 0 1 0 + local $management closest 1 0 0 0 0 0 0 0 0 + router Router.A closest 0 0 1 0 0 0 6 0 6 + router Router.C closest 0 0 1 0 0 0 4 0 4 + mobile my_queue 1 balanced 0 0 0 0 0 0 0 0 0 + mobile my_queue 0 balanced 0 1 0 0 0 10 0 0 0 + local qdhello flood 1 1 0 0 0 0 0 97 117 + local qdrouter flood 1 0 0 0 0 0 0 7 0 + topo qdrouter flood 1 0 2 0 0 0 8 13 9 + local qdrouter.ma multicast 1 0 0 0 0 0 0 2 0 + topo qdrouter.ma multicast 1 0 2 0 0 0 0 0 1 + local temp.7f2u0zv9_U6QC5e closest 0 1 0 0 0 0 0 0 0 +---- + +For the "Router.B", there are 10 messages as output (from the router to the broker) on the `my_queue` address. + +Starting the receiver connected to the "Router.A", it gets all the available messages from the queue. + +[options="nowrap"] +---- +$ sudo python simple_recv.py -a localhost:6000/my_queue -m 10 +{u'sequence': 1L} +{u'sequence': 2L} +{u'sequence': 3L} +{u'sequence': 4L} +{u'sequence': 5L} +{u'sequence': 6L} +{u'sequence': 7L} +{u'sequence': 8L} +{u'sequence': 9L} +{u'sequence': 10L} +---- + +Using the `qdstat` tool on the "Router.B" another time, the output is like following. + +[options="nowrap"] +---- +$ sudo qdstat -b localhost:6001 -a +Router Addresses + class addr phs distrib in-proc local remote cntnr in out thru to-proc from-proc + ================================================================================================================= + local $_management_internal closest 1 0 0 0 0 0 0 0 0 + local $displayname closest 1 0 0 0 0 0 0 0 0 + mobile $management 0 closest 1 0 0 0 2 0 0 2 0 + local $management closest 1 0 0 0 0 0 0 0 0 + router Router.A closest 0 0 1 0 0 0 6 0 6 + router Router.C closest 0 0 1 0 0 0 4 0 4 + mobile my_queue 1 balanced 0 0 0 0 10 0 10 0 0 + mobile my_queue 0 balanced 0 1 0 0 0 10 0 0 0 + local qdhello flood 1 1 0 0 0 0 0 156 182 + local qdrouter flood 1 0 0 0 0 0 0 7 0 + topo qdrouter flood 1 0 2 0 0 0 10 18 11 + local qdrouter.ma multicast 1 0 0 0 0 0 0 2 0 + topo qdrouter.ma multicast 1 0 2 0 0 0 0 2 1 + local temp.Xov_ZUcyti3jjXY closest 0 1 0 0 0 0 0 0 0 +---- + +For the "Router.B", there are 10 messages as input (from the broker to the router) on the `my_queue` address. + +Now, consider a fault on the "Router.B"; in this case the broker is not reachable but it is possible to set up path redundancy through the "Router.C". + +.Path Redundancy and Temporal Decoupling after Router Failure +image::path-redundancy-temp-decoupling-02.png[Path Redundancy and Temporal Decoupling after Router Failure, align="center"] + +Using the `qdmanage` tool, it is possible to configure the waypoint on `my_queue` address, the related auto-links in both directions and finally the _connector_ instance in order to enable the connection to the broker. + +[options="nowrap"] +---- +$ sudo qdmanage -b localhost:6002 create --stdin +[ +{ "type":"connector", "name":"BROKER", "port":5672, "role":"route-container" }, +{ "type":"address", "prefix":"my_queue", "waypoint":"yes" }, +{ "type":"autoLink", "addr":"my_queue", "connection":"BROKER", "direction":"in" }, +{ "type":"autoLink", "addr":"my_queue", "connection":"BROKER", "direction":"out" } +] +[ + { + "verifyHostname": true, + "stripAnnotations": "both", + "name": "BROKER", + "allowRedirect": true, + "idleTimeoutSeconds": 16, + "maxFrameSize": 65536, + "host": "127.0.0.1", + "cost": 1, + "role": "route-container", + "maxSessions": 32768, + "type": "org.apache.qpid.dispatch.connector", + "port": "5672", + "identity": "connector/127.0.0.1:5672:BROKER", + "addr": "127.0.0.1" + }, + { + "name": null, + "prefix": "my_queue", + "ingressPhase": 0, + "waypoint": false, + "distribution": "balanced", + "type": "org.apache.qpid.dispatch.router.config.address", + "identity": "7", + "egressPhase": 0 + }, + { + "addr": "my_queue", + "name": null, + "linkRef": null, + "type": "org.apache.qpid.dispatch.router.config.autoLink", + "operStatus": "inactive", + "connection": "BROKER", + "direction": "in", + "phase": 1, + "lastError": null, + "externalAddr": null, + "identity": "8", + "containerId": null + }, + { + "addr": "my_queue", + "name": null, + "linkRef": null, + "type": "org.apache.qpid.dispatch.router.config.autoLink", + "operStatus": "inactive", + "connection": "BROKER", + "direction": "out", + "phase": 0, + "lastError": null, + "externalAddr": null, + "identity": "9", + "containerId": null + } +] +---- + +The "Router.C" configuration changes in the same way as "Router.B". It can accept incoming connections from clients and from other routers (in this case the "Router.A") and connects to the broker. The queue named `my_queue` on the broker is exposed by a waypoint with the related auto-links in both directions in order to send and receive messages to/from the queue itself. + +At this point, the sender can connect to the "Router.A" for sending messages to the queue in the broker thanks to the "Router.C". + +[options="nowrap"] +---- +$ sudo python simple_send.py -a localhost:6000/my_queue -m 10 +all messages confirmed +---- + +All messages are confirmed because they reached the queue inside the broker through "Router.A" and "Router.C"; it is confirmed using the `qdstat` tool. + +[options="nowrap"] +---- +$ sudo qdstat -b localhost:6002 -a +Router Addresses + class addr phs distrib in-proc local remote cntnr in out thru to-proc from-proc + ================================================================================================================= + local $_management_internal closest 1 0 0 0 0 0 0 1 1 + local $displayname closest 1 0 0 0 0 0 0 0 0 + mobile $management 0 closest 1 0 0 0 5 0 0 5 0 + local $management closest 1 0 0 0 0 0 0 0 0 + router Router.A closest 0 0 1 0 0 0 5 0 5 + mobile my_queue 0 balanced 0 1 0 0 0 10 0 0 0 + mobile my_queue 1 balanced 0 0 0 0 0 0 0 0 0 + local qdhello flood 1 1 0 0 0 0 0 665 647 + local qdrouter flood 1 0 0 0 0 0 0 8 0 + topo qdrouter flood 1 0 1 0 0 0 31 52 32 + local qdrouter.ma multicast 1 0 0 0 0 0 0 1 0 + topo qdrouter.ma multicast 1 0 1 0 0 0 1 2 1 + local temp.k6UMaS4P0JmtSlL closest 0 1 0 0 0 0 0 0 0 + +---- + +For the "Router.C", there are 10 messages as output (from the router to the broker) on the `my_queue` address. + +Starting the receiver connected to the "Router.A", it gets all the available messages from the queue. + +[options="nowrap"] +---- +$ sudo python simple_recv.py -a localhost:6000/my_queue -m 10 +{u'sequence': 1L} +{u'sequence': 2L} +{u'sequence': 3L} +{u'sequence': 4L} +{u'sequence': 5L} +{u'sequence': 6L} +{u'sequence': 7L} +{u'sequence': 8L} +{u'sequence': 9L} +{u'sequence': 10L} +---- + +Using the `qdstat` tool on the "Router.C" another time, the output is like following. + +[options="nowrap"] +---- +$ sudo qdstat -b localhost:6002 -a +Router Addresses + class addr phs distrib in-proc local remote cntnr in out thru to-proc from-proc + ================================================================================================================= + local $_management_internal closest 1 0 0 0 0 0 0 1 1 + local $displayname closest 1 0 0 0 0 0 0 0 0 + mobile $management 0 closest 1 0 0 0 6 0 0 6 0 + local $management closest 1 0 0 0 0 0 0 0 0 + router Router.A closest 0 0 1 0 0 0 5 0 5 + mobile my_queue 0 balanced 0 1 0 0 0 10 0 0 0 + mobile my_queue 1 balanced 0 0 0 0 10 0 10 0 0 + local qdhello flood 1 1 0 0 0 0 0 746 726 + local qdrouter flood 1 0 0 0 0 0 0 8 0 + topo qdrouter flood 1 0 1 0 0 0 34 55 35 + local qdrouter.ma multicast 1 0 0 0 0 0 0 1 0 + topo qdrouter.ma multicast 1 0 1 0 0 0 1 4 1 + local temp.Hso3moy3l+Sn+Fy closest 0 1 0 0 0 0 0 0 0 +---- + +For the "Router.C", there are 10 messages as input (from the broker to the router) on the `my_queue` address. + +== Sharded Queue + +Every broker has limits in terms of queue size but in order to overcome this problem, one possible solution is "sharding" queues : in that way a single queue is divided in more "shards" (chunks) each on a different broker. It means that such solution needs more than one broker instance in order to host a shard on each of them. Of course, a sender connected to one of these brokers can send messages to the shard hosted only on that broker. At same time, a receiver connected to a broker can get messages from the shard that is hosted on that broker and can not see available messages in the shards hosted on the other brokers, even if they are all parts of the same queue. + +[NOTE] +==== +Even if speaking about shards it is obvious that they are real queues all with same name but on different brokers. The "shard" concept is an abstract one because finally a shard is a real queue stored on a broker. +==== + +The big problem in this scenario, designed only with brokers, is that a receiver can be stucked on an empty shard without reading any messages while the shards on the other brokers have messages to deliver. it is a real problem because the receiver is interested in receiving messages from the whole queue and it does not take care if it is shared or not. Because of this problem, the receiver sees the queue as empty even if it is not so true due to the sharding and the messages available on the other shards. + +The above problem can be solved adding a {RouterName} instance in the network in front of the brokers and leverage on its waypoint feature with related auto-links. + +Consider the following simple scenario : + +* a network with one router "Router.A" and two brokers. +* the "Router.A" listens for clients connections and it is connected to both brokers. +* the brokers host shards for a queue; each broker has one shard. +* a sender client connects to the "Router.A" in order to send messages to the queue. +* a receiver client connects to the "Router.A" in order to get messages from the queue. + +.Sharded Queue Enabled Topology +image::sharded-queue-01.png[Sharded Queue Enabled Topology, align="center"] + +With such solution and connecting to the "Router.A", sender and receiver do not know anything about sharding; they want send and receive messages to/from the whole queue that is the only thing they are aware of. They are both connected to the router and see only one address (related to the queue). + +The "Router.A" configuration is something like following. + +[options="nowrap"] +---- +router { + mode: standalone + id: Router.A +} + +listener { + host: 0.0.0.0 + port: 6000 + authenticatePeer: no +} + +connector { + name: BROKER1 + addr: 127.0.0.1 + port: 5672 + role: route-container +} + +connector { + name: BROKER2 + addr: 127.0.0.1 + port: 5673 + role: route-container +} + +address { + prefix: my_queue + waypoint: yes +} + +autoLink { + addr: my_queue + connection: BROKER1 + direction: in +} + +autoLink { + addr: my_queue + connection: BROKER1 + direction: out +} + +autoLink { + addr: my_queue + connection: BROKER2 + direction: in +} + +autoLink { + addr: my_queue + connection: BROKER2 + direction: out +} +---- + +The router has a _listener_ for incoming connection from clients and two _connector_ instances in order to connect to both brokers. The whole queue is named `my_queue` hosted in terms of shards on both brokers and the router is configured with a waypoint for that address. Finally, there are two auto-links in both directions for that queue on both brokers. + +Using the Python sample from the Qpid Proton library, the sender can connect to the "Router.A" and start to send messages to the queue; the console output is like following. + +[options="nowrap"] +---- +$ sudo python simple_send.py -a localhost:6000/my_queue -m 10 +all messages confirmed +---- + +All messages are confirmed because they reached the queue and, thanks to the default `balanced` distribution on the address, the messages are delivered to both shards on the brokers (5 messages per shard). Using the `qdstat` tool on the router, the distribution is clear. + +[options="nowrap"] +---- +$ sudo qdstat -b localhost:6000 -l +Router Links + type dir conn id id peer class addr phs cap undel unsettled deliveries admin oper + ======================================================================================================================= + endpoint in 1 6 mobile my_queue 1 250 0 0 0 enabled up + endpoint out 1 7 mobile my_queue 0 250 0 0 5 enabled up + endpoint in 2 8 mobile my_queue 1 250 0 0 0 enabled up + endpoint out 2 9 mobile my_queue 0 250 0 0 5 enabled up + endpoint in 8 19 mobile $management 0 250 0 0 1 enabled up + endpoint out 8 20 local temp.qCGHruCa4UIvYrS 250 0 0 0 enabled up +---- + +There are the `out` links (from router to brokers) for the `my_queue` address (_id_ values `7` and `9`) which have each 5 deliveries. It shows messages distributed across brokers and related shards for the queue; it is confirmed by the different connections they are tied (_conn id_ values `1` and `2`). + +Starting the receiver connected to the "Router.A", it gets all the available messages from the queue. + +[options="nowrap"] +---- +$ sudo python simple_recv.py -a localhost:6000/my_queue -m 10 +{u'sequence': 1L} +{u'sequence': 2L} +{u'sequence': 3L} +{u'sequence': 4L} +{u'sequence': 5L} +{u'sequence': 6L} +{u'sequence': 7L} +{u'sequence': 8L} +{u'sequence': 9L} +{u'sequence': 10L} +---- + +As for the sender, they are received through both the brokers and related shards. it is confirmed using the `qdstat` tool. + +[options="nowrap"] +---- +$ sudo qdstat -b localhost:6000 -l +Router Links + type dir conn id id peer class addr phs cap undel unsettled deliveries admin oper + ======================================================================================================================= + endpoint in 1 6 mobile my_queue 1 250 0 0 5 enabled up + endpoint out 1 7 mobile my_queue 0 250 0 0 5 enabled up + endpoint in 2 8 mobile my_queue 1 250 0 0 5 enabled up + endpoint out 2 9 mobile my_queue 0 250 0 0 5 enabled up + endpoint in 10 22 mobile $management 0 250 0 0 1 enabled up + endpoint out 10 23 local temp.HT+f3ZilGP5o3wo 250 0 0 0 enabled up +---- + +There are the `in` links (from brokers to router) for the `my_queue` address (_id_ values `6` and `8`) which have each 5 deliveries. It shows messages distributed across brokers and related shards for the queue; it is confirmed by the different connections they are tied (_conn id_ values `1` and `2`). + +One disadvantage of sharded queues is that the receiver might receive messages "out of order" even with very good performance.
http://git-wip-us.apache.org/repos/asf/qpid-dispatch/blob/f77f92cb/docs/books/user-guide/revision-info.adoc ---------------------------------------------------------------------- diff --git a/docs/books/user-guide/revision-info.adoc b/docs/books/user-guide/revision-info.adoc new file mode 100644 index 0000000..fb19158 --- /dev/null +++ b/docs/books/user-guide/revision-info.adoc @@ -0,0 +1,20 @@ +//// +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License +//// + +_Revised on {localdate} {localtime}_ \ No newline at end of file http://git-wip-us.apache.org/repos/asf/qpid-dispatch/blob/f77f92cb/docs/books/user-guide/routing.adoc ---------------------------------------------------------------------- diff --git a/docs/books/user-guide/routing.adoc b/docs/books/user-guide/routing.adoc new file mode 100644 index 0000000..dcb5d07 --- /dev/null +++ b/docs/books/user-guide/routing.adoc @@ -0,0 +1,721 @@ +//// +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License +//// + +[id='routing'] += Routing + +Routing is the process by which messages are delivered to their destinations. To accomplish this, {RouterName} provides two routing mechanisms: _message routing_ and _link routing_. + +Message routing:: Routing is performed on messages as producers send them to a router. When a message arrives on a router, the router routes the message and its _settlement_ based on the message's _address_ and _routing pattern_. ++ +-- +.Message Routing +image::message-routing.png[Message Routing, align="center"] + +In this diagram, the message producer attaches a link to the router, and then sends a message over the link. When the router receives the message, it identifies the message's destination based on the message's address, and then uses its routing table to determine the best route to deliver the message either to its destination or to the next hop in the route. All dispositions (including settlement) are propagated along the same path that the original message transfer took. Flow control is handled between the sender and the router, and then between the router and the receiver. +-- + +Link routing:: Routing is performed on link-attach frames, which are chained together to form a virtual messaging path that directly connects a sender and receiver. Once a link route is established, the transfer of message deliveries, flow frames, and dispositions is performed across the link route. ++ +-- +.Link Routing +image::link-routing.png[Link Routing, align="center"] + +In this diagram, a router is connected to clients and to a broker, and it provides a link route to a queue on the broker (my_queue). The sender connects to the router, and the router propagates the link-attaches to the broker to form a direct link between the sender and the broker. The sender can begin sending messages to the queue, and the router passes the deliveries along the link route directly to the broker queue. +-- + +== Comparison of Message Routing and Link Routing + +While you can use either message routing or link routing to deliver messages to a destination, they differ in several important ways. Understanding these differences will enable you to choose the proper routing approach for any particular use case. + +=== When to Use Message Routing + +Message routing is the default routing mechanism. You can use it to route messages on a per-message basis between clients directly (direct-routed messaging), or to and from broker queues (brokered messaging). + +Message routing is best suited to the following requirements: + +* Default, basic message routing. ++ +{RouterName} automatically routes messages by default, so manual configuration is only required if you want routing behavior that is different than the default. + +* Message-based routing patterns. ++ +Message routing supports both anycast and multicast routing patterns. You can load-balance individual messages across multiple consumers, and multicast (or fan-out) messages to multiple subscribers. + +* Sharding messages across multiple broker instances when message delivery order is not important. ++ +Sharding messages from one producer might cause that producer's messages to be received in a different order than the order in which they were sent. + +Message routing is not suitable for any of the following requirements: + +* Dedicated path through the router network. ++ +For inter-router transfers, all message deliveries are placed on the same inter-router link. This means that the traffic for one address might affect the delivery of the traffic for another address. + +* Granular, end-to-end flow control. ++ +With message routing, end-to-end flow control is based on the settlement of deliveries and therefore might not be optimal in every case. + +* Transaction support. + +* Server-side selectors. + +=== When to Use Link Routing + +Link routing requires more detailed configuration than message routing as well as an AMQP container that can accept incoming link-attaches (typically a broker). However, link routing enables you to satisfy more advanced use cases than message routing. + +You can use link routing if you need to meet any of the following requirements: + +* Dedicated path through the router network. ++ +With link routing, each link route has dedicated inter-router links through the network. Each link has its own dedicated message buffers, which means that the address will not have "head-of-line" blocking issues with other addresses. + +* Sharding messages across multiple broker instances with guaranteed delivery order. ++ +Link routing to a sharded queue preserves the delivery order of the producer's messages by causing all messages on that link to go to the same broker instance. + +* End-to-end flow control. ++ +Flow control is "real" in that credits flow across the link route from the receiver to the sender. + +* Transaction support. ++ +Link routing supports local transactions to a broker. + +* Server-side selectors. ++ +With a link route, consumers can provide server-side selectors for broker subscriptions. + +== Configuring Message Routing + +With message routing, routing is performed on messages as producers send them to a router. When a message arrives on a router, the router routes the message and its _settlement_ based on the message's _address_ and _routing pattern_. + +With message routing, you can do the following: + +* Route messages between clients (direct-routed, or brokerless messaging) ++ +This involves configuring an address with a routing pattern. All messages sent to the address will be routed based on the routing pattern. +* Route messages through a broker queue (brokered messaging) ++ +This involves configuring a waypoint address to identify the broker queue and then connecting the router to the broker. All messages sent to the waypoint address will be routed to the broker queue. + +=== Addresses + +Addresses determine how messages flow through your router network. An address designates an endpoint in your messaging network, such as: + +* Endpoint processes that consume data or offer a service +* Topics that match multiple consumers to multiple producers +* Entities within a messaging broker: +** Queues +** Durable Topics +** Exchanges + +When a router receives a message, it uses the message's address to determine where to send the message (either its destination or one step closer to its destination). + +// Do we need to specify that these are AMQP addresses? Should they be distinguished from generic message addresses? + +// Need to add something here about the difference between discovered vs. configured mobile addresses so that it's clear that with message routing, the router can either be proactive or reactive in the way it routes messages. + +[id='routing-patterns-overview'] +=== Routing Patterns + +Each address has one of the following routing patterns, which define the path that a message with the address can take across the messaging network: + +Balanced:: An anycast method that allows multiple consumers to use the same address. Each message is delivered to a single consumer only, and {RouterName} attempts to balance the traffic load across the router network. ++ +-- +If multiple consumers are attached to the same address, each router determines which outbound path should receive a message by considering each path's current number of unsettled deliveries. This means that more messages will be delivered along paths where deliveries are settled at higher rates. + +[NOTE] +==== +{RouterName} neither measures nor uses message settlement time to determine which outbound path to use. +==== + +In this scenario, the messages are spread across both receivers regardless of path length: + +.Balanced Message Routing +image::balanced-routing.png[Balanced Message Routing, align="center"] +-- + +Closest:: An anycast method in which every message is sent along the shortest path to reach the destination, even if there are other consumers for the same address. ++ +{RouterName} determines the shortest path based on the topology cost to reach each of the consumers. If there are multiple consumers with the same lowest cost, messages will be spread evenly among those consumers. ++ +In this scenario, all messages sent by `Sender` will be delivered to `Receiver 1`: ++ +.Closest Message Routing +image::closest-routing.png[Closest Message Routing, align="center"] + +Multicast:: Messages are sent to all consumers attached to the address. Each consumer will receive one copy of the message. ++ +In this scenario, all messages are sent to all receivers: ++ +.Multicast Message Routing +image::multicast-routing.png[Multicast Message Routing, align="center"] + +=== Message Settlement + +Message settlement is negotiated between the producer and the router when the producer establishes a link to the router. Depending on the settlement pattern, messages might be delivered with any of the following degrees of reliability: + +* At most once +* At least once +* Exactly once + +{RouterName} treats all messages as either _pre-settled_ or _unsettled_, and it is responsible for propagating the settlement of each message it routes. + +Pre-settled:: Sometimes called _fire and forget_, the router settles the incoming and outgoing deliveries and propagates the settlement to the message's destination. However, it does not guarantee delivery. + +Unsettled:: The router propagates the settlement between the sender and receiver, and guarantees one of the following outcomes: ++ +* The message is delivered and settled, with the consumer's disposition indicated. +* The delivery is settled with a disposition of `RELEASED`. ++ +This means that the message did not reach its destination. +* The delivery is settled with a disposition of `MODIFIED`. ++ +This means that the message might or might not have reached its destination. The delivery is considered to be "in-doubt" and should be re-sent if "at least once" delivery is required. +* The link, session, or connection to {RouterName} was dropped, and all deliveries are "in-doubt". + +[id='routing-messages-between-clients'] +=== Routing Messages Between Clients + +You can route messages between clients without using a broker. In a brokerless scenario (sometimes called _direct-routed messaging_), {RouterName} routes messages between clients directly. + +To route messages between clients, you configure an address with a routing distribution pattern. When a router receives a message with this address, the message is routed to its destination or destinations based on the address's routing distribution pattern. + +.Procedure + +. In the router's configuration file, add an `address` section: ++ +-- +[options="nowrap",subs="+quotes"] +---- +address { + prefix: _ADDRESS_PREFIX_ + distribution: balanced|closest|multicast + ... +} +---- + +`prefix` | `pattern`:: The address or group of addresses to which the address settings should be applied. You can specify a prefix to match an exact address or beginning segment of an address. Alternatively, you can specify a pattern to match an address using wildcards. ++ +//tag::prefix-matching[] +A _prefix_ matches either an exact address or the beginning segment within an address that is delimited by either a `.` or `/` character. For example, the prefix `my_address` would match the address `my_address` as well as `my_address.1` and `my_address/1`. However, it would not match `my_address1`. +//end::prefix-matching[] ++ +//tag::pattern-matching[] +A _pattern_ matches an address that corresponds to a pattern. A pattern is a sequence of words delimited by either a `.` or `/` character. You can use wildcard characters to represent a word. The `*` character matches exactly one word, and the `#` character matches any sequence of zero or more words. ++ +The `*` and `#` characters are reserved as wildcards. Therefore, you should not use them in the message address. ++ +The following table shows some examples of address patterns: ++ +[cols="25,75"] +|=== +| This pattern... | Matches... +| `news` | `news` +| `news/*/sports` | `news/europe/sports` and `news/usa/sports`, but not `news` or `news/europe/fr/sports` +| `news/#` | `news`, `news/europe`, `news/usa`, `news/usa/sports` +|=== ++ +[NOTE] +==== +You can convert a `prefix` value to a `pattern` by appending `/\#` to it. For example, the prefix `a/b/c` is equivalent to the pattern `a/b/c/#`. +==== +//end::pattern-matching[] + +`distribution`:: The message distribution pattern. The default is `balanced`, but you can specify any of the following options: ++ +* `balanced` - Messages sent to the address will be routed to one of the receivers, and the routing network will attempt to balance the traffic load based on the rate of settlement. +* `closest` - Messages sent to the address are sent on the shortest path to reach the destination. It means that if there are multiple receivers for the same address, only the closest one will receive the message. +* `multicast` - Messages are sent to all receivers that are attached to the address in a _publish/subscribe_ model. ++ +For more information about message distribution patterns, see xref:routing-patterns-overview[Routing Patterns]. + +For information about additional attributes, see link:{qdrouterdConfManPageUrl}#_address[address] in the `qdrouterd.conf` man page. +-- + +. Add the same `address` section to any other routers that need to use the address. ++ +The `address` that you added to this router configuration file only controls how this router distributes messages sent to the address. If you have additional routers in your router network that should distribute messages for this address, then you must add the same `address` section to each of their configuration files. + +[id='routing-messages-through-broker'] +=== Routing Messages Through a Broker Queue + +You can route messages to and from a broker queue to provide clients with access to the queue through a router. In this scenario, clients connect to a router to send and receive messages, and the router routes the messages to or from the broker queue. + +You can route messages to a queue hosted on a single broker, or route messages to a _sharded queue_ distributed across multiple brokers. + +.Brokered Messaging +image::brokered-messaging.png[Brokered Messaging, align="center"] + +In this diagram, the sender connects to the router and sends messages to my_queue. The router attaches an outgoing link to the broker, and then sends the messages to my_queue. Later, the receiver connects to the router and requests messages from my_queue. The router attaches an incoming link to the broker to receive the messages from my_queue, and then delivers them to the receiver. + +You can also route messages to a _sharded queue_, which is a single, logical queue comprised of multiple, underlying physical queues. Using queue sharding, it is possible to distribute a single queue over multiple brokers. Clients can connect to any of the brokers that hold a shard to send and receive messages. + +.Brokered Messaging with Sharded Queue +image::sharded-queue-02.png[Brokered Messaging with Sharded Queue, align="center"] + +In this diagram, a sharded queue (my_queue) is distributed across two brokers. The router is connected to the clients and to both brokers. The sender connects to the router and sends messages to my_queue. The router attaches an outgoing link to each broker, and then sends messages to each shard (by default, the routing distribution is `balanced`). Later, the receiver connects to the router and requests all of the messages from my_queue. The router attaches an incoming link to one of the brokers to receive the messages from my_queue, and then delivers them to the receiver. + +.Procedure + +. xref:configure-waypoint-address[Add a waypoint address]. ++ +This address identifies the queue to which you want to route messages. +. xref:connect-router-to-broker[Add autolinks to connect the router to the broker]. ++ +Autolinks connect the router to the broker queue identified by the waypoint address. + +. xref:connect-router-to-broker[If the queue is sharded, add autolinks for each additional broker that hosts a shard]. + +[id='configure-waypoint-address'] +==== Configuring Waypoint Addresses + +A waypoint address identifies a queue on a broker to which you want to route messages. You need to configure the waypoint address on each router that needs to use the address. For example, if a client is connected to _Router A_ to send messages to the broker queue, and another client is connected to _Router B_ to receive those messages, then you would need to configure the waypoint address on both _Router A_ and _Router B_. + +.Prerequisites + +An incoming connection (`listener`) to which the clients can connect should be configured. This connection defines how the producers and consumers connect to the router to send and receive messages. For more information, see xref:adding-incoming-connections[Adding Incoming Connections]. + +// Does the broker queue have to exist before you create the waypoint address? If it doesn't exist, will you get an error? + +.Procedure + +* Create waypoint addresses on each router that needs to use the address: ++ +-- +[options="nowrap",subs="+quotes"] +---- +address { + prefix: _ADDRESS_PREFIX_ + waypoint: yes +} +---- + +`prefix` | `pattern`:: The address prefix or pattern that matches the broker queue to which you want to send messages. You can specify a prefix to match an exact address or beginning segment of an address. Alternatively, you can specify a pattern to match an address using wildcards. ++ +include::routing.adoc[tags=prefix-matching] ++ +include::routing.adoc[tags=pattern-matching] + +`waypoint`:: Set this attribute to `yes` so that the router handles messages sent to this address as a waypoint. +-- + +[id='connect-router-to-broker'] +==== Connecting a Router to the Broker + +After you add waypoint addresses to identify the broker queue, you must connect a router to the broker using autolinks. + +With autolinks, client traffic is handled on the router, not the broker. Clients attach their links to the router, and then the router uses internal autolinks to connect to the queue on the broker. Therefore, the queue will always have a single producer and a single consumer regardless of how many clients are attached to the router. + +. If this router is different than the router that is connected to the clients, then add the waypoint address. + +. Add an outgoing connection to the broker: ++ +-- +[options="nowrap",subs="+quotes"] +---- +connector { + name: _NAME_ + host: _HOST_NAME/ADDRESS_ + port: _PORT_NUMBER/NAME_ + role: route-container + ... +} +---- + +`name`:: The name of the `connector`. Specify a name that describes the broker. +`host`:: Either an IP address (IPv4 or IPv6) or hostname on which the router should connect to the broker. +`port`:: The port number or symbolic service name on which the router should connect to the broker. +`role`:: Specify `route-container` to indicate that this connection is for an external container (broker). + +For information about additional attributes, see link:{qdrouterdConfManPageUrl}#_connector[connector] in the `qdrouterd.conf` man page. +-- + +. If you want to send messages to the broker queue, create an outgoing autolink to the broker queue: ++ +-- +[options="nowrap",subs="+quotes"] +---- +autoLink { + addr: _ADDRESS_ + connection: _CONNECTOR_NAME_ + direction: out + ... +} +---- + +`addr`:: The address of the broker queue. When the autolink is created, it will be attached to this address. +`externalAddr`:: An optional alternate address for the broker queue. You use an external address if the broker queue should have a different address than that which the sender uses. In this scenario, senders send messages to the `addr` address, and then the router routes them to the broker queue represented by the `externalAddr` address. +`connection` | `containerID`:: How the router should connect to the broker. You can specify either an outgoing connection (`connection`) or the container ID of the broker (`containerID`). +`direction`:: Set this attribute to `out` to specify that this autolink can send messages from the router to the broker. + +For information about additional attributes, see link:{qdrouterdConfManPageUrl}#_autolink[autoLink] in the `qdrouterd.conf` man page. +-- + +. If you want to receive messages from the broker queue, create an incoming autolink from the broker queue: ++ +-- +[options="nowrap",subs="+quotes"] +---- +autoLink { + addr: _ADDRESS_ + connection: _CONNECTOR_NAME_ + direction: in + ... +} +---- + +`addr`:: The address of the broker queue. When the autolink is created, it will be attached to this address. +`externalAddr`:: An optional alternate address for the broker queue. You use an external address if the broker queue should have a different address than that which the receiver uses. In this scenario, receivers receive messages from the `addr` address, and the router retrieves them from the broker queue represented by the `externalAddr` address. +`connection` | `containerID`:: How the router should connect to the broker. You can specify either an outgoing connection (`connection`) or the container ID of the broker (`containerID`). +`direction`:: Set this attribute to `in` to specify that this autolink can receive messages from the broker to the router. + +For information about additional attributes, see link:{qdrouterdConfManPageUrl}#_autolink[autoLink] in the `qdrouterd.conf` man page. +-- + +=== Example: Routing Messages Through Broker Queues + +This example shows how waypoints and autolinks can route messages through a pair of queues on a broker. + +==== Router Configuration + +[options="nowrap"] +---- +connector { // <1> + name: broker + role: route-container + host: 198.51.100.1 + port: 61617 + saslMechanisms: ANONYMOUS +} + +address { // <2> + prefix: queue + waypoint: yes +} + +autoLink { // <3> + addr: queue.first + direction: in + connection: broker +} + +autoLink { // <4> + addr: queue.first + direction: out + connection: broker +} + +autoLink { // <5> + addr: queue.second + direction: in + connection: broker +} + +autoLink { // <6> + addr: queue.second + direction: out + connection: broker +} +---- +<1> The outgoing connection from the router to the broker. The `route-container` role enables the router to connect to an external AMQP container (in this case, a broker). +<2> The namespace queue on the broker to which the router should route messages. All addresses that start with `queue` will be routed to a queue on the broker. +<3> The incoming autolink from `queue.first` on the broker to the router. +<4> The outgoing autolink from the router to `queue.first` on the broker. +<5> The incoming autolink from `queue.second` on the broker to the router. +<6> The outgoing autolink from the router to `queue.second` on the broker. + +==== How the Messages are Routed + +Initially, when the broker is offline, the autolinks are inactive. + +[options="nowrap"] +---- +$ qdstat --autolinks +AutoLinks + addr dir phs extAddr link status lastErr + ======================================================== + queue.first in 1 inactive + queue.first out 0 inactive + queue.second in 1 inactive + queue.second out 0 inactive +---- + +Once the broker is online, the autolinks attempt to activate. In this case, the broker starts with the `queue.first` queue only, and the `queue.first` autolinks become active. The `queue.second` autolinks are in a failed state, because the `queue.second` queue does not exist on the broker. + +[options="nowrap"] +---- +$ qdstat --autolinks +AutoLinks + addr dir phs extAddr link status lastErr + =========================================================================== + queue.first in 1 6 active + queue.first out 0 7 active + queue.second in 1 failed Node not found: queue.second + queue.second out 0 failed Node not found: queue.second +---- + +The producer now connects to the router and sends three messages to `queue.first`. + +[options="nowrap"] +---- +$ python simple_send.py -a 127.0.0.1/queue.first -m3 +all messages confirmed +---- + +The router's address statistics show that the messages were delivered to the queue. + +[options="nowrap"] +---- +$ qdstat -a +Router Addresses + class addr phs distrib in-proc local remote cntnr in out thru to-proc from-proc + ======================================================================================================== + mobile queue.first 1 balanced 0 0 0 0 0 0 0 0 0 + mobile queue.first 0 balanced 0 1 0 0 3 3 0 0 0 +---- + +The `queue.first` address appears twice in the output: once for each phase of the address. Phase 0 is for routing messages from producers to the outgoing autolink. Phase 1 is for routing messages from the incoming autolink to the subscribed consumers. In this case, Phase 0 of the address has counted three messages in the `in` column (the messages that arrived on the router from the producer), and three messages in the `out` column (the messages that were sent from the router to the broker queue). + +The consumer now connects to the router and receives the three messages from `queue.first`. + +[options="nowrap"] +---- +$ python simple_recv.py -a 127.0.0.1:5672/queue.first -m3 +{u'sequence': int32(1)} +{u'sequence': int32(2)} +{u'sequence': int32(3)} +---- + +The router's address statistics now show that all three messages were received by the consumer from the broker queue. + +[options="nowrap"] +---- +$ qdstat -a +Router Addresses + class addr phs distrib in-proc local remote cntnr in out thru to-proc from-proc + ======================================================================================================== + mobile queue.first 1 balanced 0 0 0 0 3 3 0 0 0 + mobile queue.first 0 balanced 0 1 0 0 3 3 0 0 0 +---- + +The command output shows that Phase 1 of the address was used to deliver all three messages from the queue to the consumer. + +[NOTE] +==== +Even in a multi-router network, and with multiple producers and consumers for `queue.first`, all deliveries are routed through the queue on the connected broker. +==== + +== Configuring Link Routing + +Link routing provides an alternative strategy for brokered messaging. A link route represents a private messaging path between a sender and a receiver in which the router passes the messages between end points. You can think of a link route as a "virtual connection" or "tunnel" that travels from a sender, through the router network, to a receiver. + +With link routing, routing is performed on link-attach frames, which are chained together to form a virtual messaging path that directly connects a sender and receiver. Once a link route is established, the transfer of message deliveries, flow frames, and dispositions is performed across the link route. + +=== Link Route Addresses + +A link route address represents a broker queue, topic, or other service. When a client attaches a link route address to a router, the router propagates a link attachment to the broker resource identified by the address. + +=== Link Route Routing Patterns + +Routing patterns are not used with link routing, because there is a direct link between the sender and receiver. The router only makes a routing decision when it receives the initial link-attach request frame. Once the link is established, the router passes the messages along the link in a balanced distribution. + +=== Link Route Flow Control + +Unlike message routing, with link routing, the sender and receiver handle flow control directly: the receiver grants link credits, which is the number of messages it is able to receive. The router sends them directly to the sender, and then the sender sends the messages based on the credits that the receiver granted. + +// What additional information do we need to provide about AMQP link flow control options? Since this is handled on the client side for link routing, should we provide a simple example with a client program that implements link flow control? + +[id='creating-link-route'] +=== Creating a Link Route + +Link routes establish a link between a sender and a receiver that travels through a router. You can configure inward and outward link routes to enable the router to receive link-attaches from clients and to send them to a particular destination. + +With link routing, client traffic is handled on the broker, not the router. Clients have a direct link through the router to a broker's queue. Therefore, each client is a separate producer or consumer. + +.Procedure + +. In the router configuration file, add an outgoing connection to the broker: ++ +-- +[options="nowrap",subs="+quotes"] +---- +connector { + name: _NAME_ + host: _HOST_NAME/ADDRESS_ + port: _PORT_NUMBER/NAME_ + role: route-container + ... +} +---- + +`name`:: The name of the `connector`. You should specify a name that describes the broker. +`host`:: Either an IP address (IPv4 or IPv6) or hostname on which the router should connect to the broker. +`port`:: The port number or symbolic service name on which the router should connect to the broker. +`role`:: Specify `route-container` to indicate that this connection is for an external container (broker). + +For information about additional attributes, see link:{qdrouterdConfManPageUrl}#_connector[connector] in the `qdrouterd.conf` man page. +-- + +. If you want clients to send messages on this link route, create an incoming link route: ++ +-- +[options="nowrap",subs="+quotes"] +---- +linkRoute { + prefix: __ADDRESS_PREFIX__ + connection: __CONNECTOR_NAME__ + direction: in + ... +} +---- + +`prefix` | `pattern`:: The address prefix or pattern that matches the broker queue that should be the destination for routed link-attaches. All messages that match this prefix or pattern will be distributed along the link route. You can specify a prefix to match an exact address or beginning segment of an address. Alternatively, you can specify a pattern to match an address using wildcards. ++ +include::routing.adoc[tags=prefix-matching] ++ +include::routing.adoc[tags=pattern-matching] + +`connection` | `containerID`:: How the router should connect to the broker. You can specify either an outgoing connection (`connection`) or the container ID of the broker (`containerID`). ++ +If multiple brokers are connected to the router through this connection, requests for addresses matching the link route's prefix or pattern are balanced across the brokers. Alternatively, if you want to specify a particular broker, use `containerID` and add the broker's container ID. + +`direction`:: Set this attribute to `in` to specify that clients can send messages into the router network on this link route. + +For information about additional attributes, see link:{qdrouterdConfManPageUrl}#_linkroute[linkRoute] in the `qdrouterd.conf` man page. +-- + +. If you want clients to receive messages on this link route, create an outgoing link route: ++ +-- +[options="nowrap",subs="+quotes"] +---- +linkRoute { + prefix: __ADDRESS_PREFIX__ + connection: __CONNECTOR_NAME__ + direction: out + ... +} +---- + +`prefix` | `pattern`:: The address prefix or pattern that matches the broker queue from which you want to receive routed link-attaches. All messages that match this prefix or pattern will be distributed along the link route. You can specify a prefix to match an exact address or beginning segment of an address. Alternatively, you can specify a pattern to match an address using wildcards. ++ +include::routing.adoc[tags=prefix-matching] ++ +include::routing.adoc[tags=pattern-matching] + +`connection` | `containerID`:: How the router should connect to the broker. You can specify either an outgoing connection (`connection`) or the container ID of the broker (`containerID`). ++ +If multiple brokers are connected to the router through this connection, requests for addresses matching the link route's prefix or pattern are balanced across the brokers. Alternatively, if you want to specify a particular broker, use `containerID` and add the broker's container ID. +`direction`:: Set this attribute to `out` to specify that this link route is for receivers. + +For information about additional attributes, see link:{qdrouterdConfManPageUrl}#_linkroute[linkRoute] in the `qdrouterd.conf` man page. +-- + +=== Example: Using a Link Route to Provide Client Isolation + +This example shows how a link route can connect a client to a message broker that is on a different private network. + +.Router Network with Isolated Clients +---- + + Public Network + +-----------------+ + | +-----+ | + | B1 | Rp | | + | +/--\-+ | + | / \ | + | / \ | + +----/--------\---+ + / \ + / \ + / \ + Private Net A / \ Private Net B + +--------------/--+ +---\-------------+ + | +---/-+ | | +--\--+ | + | B2 | Ra | | | | Rb | C1 | + | +-----+ | | +-----+ | + | | | | + | | | | + +-----------------+ +-----------------+ +---- + +Client `C1` is constrained by firewall policy to connect to the router in its own network (`Rb`). However, it can use a link route to access queues, topics, and any other AMQP services that are provided on message brokers `B1` and `B2` -- even though they are on different networks. + +In this example, client `C1` needs to receive messages from `b2.event-queue`, which is hosted on broker `B2` in `Private Net A`. A link route connects the client and broker even though neither of them is aware that there is a router network between them. + +==== Router Configuration + +To enable client `C1` to receive messages from `b2.event-queue` on broker `B2`, router `Ra` must be able to do the following: + +* Connect to broker `B2` +* Route links to and from broker `B2` +* Advertise itself to the router network as a valid destination for links that have a `b2.event-queue` address. + +The relevant part of the configuration file for router `Ra` shows the following: + +-- +[options="nowrap"] +---- +connector { // <1> + name: broker + role: route-container + host: 198.51.100.1 + port: 61617 + saslMechanisms: ANONYMOUS +} + +linkRoute { // <2> + prefix: b2 + direction: in + connection: broker +} + +linkRoute { // <3> + prefix: b2 + direction: out + connection: broker +} +---- +<1> The outgoing connection from the router to broker `B2`. The `route-container` role enables the router to connect to an external AMQP container (in this case, a broker). +<2> The incoming link route for receiving links from client senders. Any sender with a target whose address begins with `b2` will be routed to broker `B2` using the `broker` connector. +<3> The outgoing link route for sending links to client receivers. Any receivers whose source address begins with `b2` will be routed to broker `B2` using the `broker` connector. +-- + +This configuration enables router `Ra` to advertise itself as a valid destination for targets and sources starting with `b2`. It also enables the router to connect to broker `B2`, and to route links to and from queues starting with the `b2` prefix. + +[NOTE] +==== +While not required, routers `Rp` and `Rb` should also have the same configuration. +==== + +==== How the Client Receives Messages + +By using the configured link route, client `C1` can receive messages from broker `B2` even though they are on different networks. + +Router `Ra` establishes a connection to broker `B2`. Once the connection is open, `Ra` tells the other routers (`Rp` and `Rb`) that it is a valid destination for link routes to the `b2` prefix. This means that sender and receiver links attached to `Rb` or `Rp` will be routed along the shortest path to `Ra`, which then routes them to broker `B2`. + +To receive messages from the `b2.event-queue` on broker `B2`, client `C1` attaches a receiver link with a source address of `b2.event-queue` to its local router, `Rb`. Because the address matches the `b2` prefix, `Rb` routes the link to `Rp`, which is the next hop in the route to its destination. `Rp` routes the link to `Ra`, which routes it to broker `B2`. Client `C1` now has a receiver established, and it can begin receiving messages. + +[NOTE] +==== +If broker `B2` is unavailable for any reason, router `Ra` will not advertise itself as a destination for `b2` addresses. In this case, routers `Rb` and `Rp` will reject link attaches that should be routed to broker `B2` with an error message indicating that there is no route available to the destination. +==== http://git-wip-us.apache.org/repos/asf/qpid-dispatch/blob/f77f92cb/docs/books/user-guide/technical-details-specifications.adoc ---------------------------------------------------------------------- diff --git a/docs/books/user-guide/technical-details-specifications.adoc b/docs/books/user-guide/technical-details-specifications.adoc new file mode 100644 index 0000000..4a57d32 --- /dev/null +++ b/docs/books/user-guide/technical-details-specifications.adoc @@ -0,0 +1,190 @@ +//// +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License +//// + +[id='technical-details-specifications'] += Technical Details and Specifications + +[id='client-compatibility'] +== Client Compatibility + +{RouterName} should, in theory, work with any client that is +compatible with AMQP 1.0. The following clients have been tested: + +[width="100%",cols="22%,78%",options="header",] +|======================================================================= +|_Client_ |_Notes_ +|qpid::messaging |The Qpid messaging clients work with {RouterName} +as long as they are configured to use the 1.0 version of the protocol. +To enable AMQP 1.0 in the C++ client, use the \{protocol:amqp1.0} +connection option. + +|Proton Reactor |The Proton Reactor API is compatible with {RouterName}. + +|Proton Messenger |Messenger works with {RouterName}. +|======================================================================= + +[id='amqp-mapping'] +== AMQP Mapping + +{RouterName} is an AMQP router and as such, it provides extensions, +code-points, and semantics for routing over AMQP. This page documents the +details of {RouterName}'s use of AMQP. + +[id='message-annotations'] +=== Message Annotations + +The following Message Annotation fields are defined by {RouterName}: + +[width="100%",cols="23%,19%,58%",options="header",] +|======================================================================= +|_Field_ |_Type_ |_Description_ +|x-opt-qd.ingress |string |The identity of the ingress router for a +message-routed message. The ingress router is the first router +encountered by a transiting message. The router will, if this field is +present, leave it unaltered. If the field is not present, the router +shall insert the field with its own identity. + +|x-opt-qd.trace |list of string |The list of routers through which this +message-routed message has transited. If this field is not present, the +router shall do nothing. If the field is present, the router shall +append its own identity to the end of the list. + +|x-opt-qd.to |string |To-Override for message-routed messages. If this +field is present, the address in this field shall be used for routing in +lieu of the _to_ field in the message properties. A router may append, +remove, or modify this annotation field depending on the policy in place +for routing the message. + +|x-opt-qd.phase |integer |The address-phase, if not zero, for messages +flowing between routers. +|======================================================================= + +[id='sourcetarget-capabilities'] +=== Source/Target Capabilities + +The following Capability values are used in Sources and Targets. + +[width="100%",cols="19%,81%",options="header",] +|======================================================================= +|_Capability_ |_Description_ +|qd.router |This capability is added to sources and targets that are +used for inter-router message exchange. This capability denotes a link +used for router-control messages flowing between routers. + +|qd.router-data |This capability is added to sources and targets that +are used for inter-router message exchange. This capability denotes a +link used for user messages being message-routed across an inter-router +connection. +|======================================================================= + +[id='dynamic-node-properties'] +=== Dynamic-Node-Properties + +The following dynamic-node-properties are used by {RouterName} in Sources. + +[width="100%",cols="23%,77%",options="header",] +|======================================================================= +|_Property_ |_Description_ +|x-opt-qd.address |The node address describing the destination desired +for a dynamic source. If this is absent, the router will terminate any +dynamic receivers. If this address is present, the router will use the +address to route the dynamic link attach to the proper destination +container. +|======================================================================= + +[id='addresses-and-address-formats'] +=== Addresses and Address Formats + +The following AMQP addresses and address patterns are used within +{RouterName}. + +[id='address-patterns'] +==== Address Patterns + +[width="100%",cols="38%,62%",options="header",] +|======================================================================= +|_Pattern_ |_Description_ +|`_local/<addr>` |An address that references a locally attached endpoint. +Messages using this address pattern shall not be routed over more than +one link. + +|`_topo/0/<router>/<addr>` | +An address that references an endpoint attached to a specific router +node in the network topology. Messages with addresses that follow this +pattern shall be routed along the shortest path to the specified router. +Note that addresses of this form are a-priori routable in that the +address itself contains enough information to route the message to its +destination. + +The `0` component immediately preceding the router-id is a placeholder +for an _area_ which may be used in the future if area routing is +implemented. + +|`<addr>` |A mobile address. An address of this format represents an +endpoint or a set of distinct endpoints that are attached to the network +in arbitrary locations. It is the responsibility of the router network +to determine which router nodes are valid destinations for mobile +addresses. +|======================================================================= + +[id='supported-addresses'] +==== Supported Addresses + +[width="100%",cols="36%,64%",options="header",] +|======================================================================= +|_Address_ |_Description_ +|`$management` |The management agent on the attached router/container. +This address would be used by an endpoint that is a management +client/console/tool wishing to access management data from the attached +container. + +|`_topo/0/Router.E/$management` |The management agent at Router.E in area +0. This address would be used by a management client wishing to access +management data from a specific container that is reachable within the +network. + +|`_local/qdhello` |The router entity in each of the connected routers. +This address is used to communicate with neighbor routers and is +exclusively for the `HELLO` discovery protocol. + +|`_local/qdrouter` |The router entity in each of the connected routers. +This address is used by a router to communicate with other routers in +the network. + +|`_topo/0/Router.E/qdrouter` |The router entity at the specifically +indicated router. This address form is used by a router to communicate +with a specific router that may or may not be a neighbor. +|======================================================================= + +[id='implementation-of-the-amqp-management-specification'] +=== Implementation of the AMQP Management Specification + +{RouterName} is manageable remotely via AMQP. It is compliant with the +emerging AMQP Management specification (draft 9). + +Differences from the specification: + +* The `name` attribute is not required when an entity is created. If + not supplied it will be set to the same value as the system-generated + "identity" attribute. Otherwise it is treated as per the standard. + +* The `REGISTER` and `DEREGISTER` operations are not implemented. The router + automatically discovers peer routers via the router network and makes + their management addresses available via the standard `GET-MGMT-NODES` + operation. --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@qpid.apache.org For additional commands, e-mail: commits-h...@qpid.apache.org
