( github gist with all the config and data from the Original Post
https://gist.github.com/danlangford/4944dcc6c0d2703ffb8555603ed27340 )
YES i was under pretty light load. a couple hundred or thousand messages at
a time were all getting funneled into the local broker. You are right once
i got 7 or 8 simultaneous connections all pushing in a couple million
message i started to see the load overflow to the other router and broker.
about 10% of messages in my setup were overflowing. thank you for your
patience. I think this config is going to work great. i do think i just
need a small adjustment to how i think about load balancing. when the cost
was the same to each broker then i would get approx 50/50 split (that was
when each router connected to each broker). with this setup where the local
cost is less its more like "balancing out once under load" and not the
traditional "balance evenly to avoid load". but now that i understand this
its fine and i know what behavior to expect.
so I do have some producers and consumers that need to use session
transactions. we have seen those work fine over amqp1.0 when connected to a
QPID Broker or Artemis broker. but with the config you see here connecting
to a qpid dispatch router (directly or through our VIP) I cannot create a
session with "transacted=true". I get a NullPointerException
javax.jms.JMSException: java.lang.NullPointerException
at
org.apache.qpid.jms.exceptions.JmsExceptionSupport.create(JmsExceptionSupport.java:86)
at
org.apache.qpid.jms.exceptions.JmsExceptionSupport.create(JmsExceptionSupport.java:108)
at org.apache.qpid.jms.JmsConnection.createResource(JmsConnection.java:609)
at
org.apache.qpid.jms.JmsLocalTransactionContext.begin(JmsLocalTransactionContext.java:125)
at org.apache.qpid.jms.JmsSession.<init>(JmsSession.java:143)
at org.apache.qpid.jms.JmsConnection.createSession(JmsConnection.java:299)
at org.myorg.mymessaging.PostOffice.buildSession(PostOffice.java:149)
...
Caused by: java.io.IOException: java.lang.NullPointerException
at
org.apache.qpid.jms.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
at
org.apache.qpid.jms.provider.amqp.AmqpTransactionCoordinator.processDeliveryUpdates(AmqpTransactionCoordinator.java:117)
at
org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:928)
at
org.apache.qpid.jms.provider.amqp.AmqpProvider.access$1800(AmqpProvider.java:93)
at
org.apache.qpid.jms.provider.amqp.AmqpProvider$18.run(AmqpProvider.java:790)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at
org.apache.qpid.jms.provider.amqp.AmqpTransactionCoordinator.processDeliveryUpdates(AmqpTransactionCoordinator.java:93)
... 10 more
are transactions expected to work with qpid dispatch router? on that note
do you think i could also get XA transactions working? would i need to use
a link route? i dont know for sure but it sounds like a link route is a
little more low level that an autolink.
thank you
On Tue, Aug 1, 2017 at 6:47 AM Ted Ross <tr...@redhat.com> wrote:
> Dan,
>
> There's one issue with your configuration which doesn't affect the load
> balancing but will cause problems with receiving messages from the
> brokers. In the address.prefix, you use "foo.#". This is a pure prefix
> and it should simply be "foo". The wildcards are coming in the next
> release but are not implemented in the code you are using.
>
> Regarding your actual question. I assume that you are testing this
> configuration under light load (i.e. sending one message at a time).
>
> The way that the balancing works is that it will route to the consumer
> (broker) with the fewest outstanding deliveries + inter-router cost. This
> means that it will favor the local broker over the remote one if there are
> no in-flight deliveries. The default (and minimum) cost for an
> inter-router connection is 1. You can set it to a higher value in the
> listener or connector.
>
> If you are sending one-at-a-time synchronous sends, they will always go to
> the local broker because the broker's zero outstanding deliveries will
> always be less than the inter-router cost of 1. If you send multiple
> deliveries asynchronously, you will see them being distributed to both
> brokers in the network. You can make the local-affinity stronger by
> increasing the inter-router cost.
>
> -Ted
>
> On Tue, Aug 1, 2017 at 3:47 AM, Dan Langford <danlangf...@gmail.com>
> wrote:
>
> > Last week I had a thread with lots of little questions around Dispatch
> > Routers. Ted Ross has been awesome to answer most of those. As a result I
> > feel like I have my QDR config shaping up a bit better. HOWEVER with some
> > more very focused questions I thought it would be best to start a new
> > thread. The problem I am seeing is that the routers are not distributing
> > the message load across other brokers on the network. Here is a little
> > diagram:
> >
> >
> > RouterA-03 ----> ArtemisBrokerA
> > |
> > |
> > |
> > \/
> > RouterB-05 ----> ArtemisBrokerB
> >
> >
> > *NOTE: i am currently using Qpid Dispatch Router from a RHEL repo. v
> 0.7.0*
> >
> >
> > Connections from clients come in through an F5 VIP which forwards those
> > connections to either host L-03-A or L-05-B. Each of those hosts have a
> > Qpid Dispatch Router installed in front of an Artemis broker. dispatch
> > router on L-05-B is listening on an additional port that dispatch router
> > L-03-A connects to for inter-router communication.
> >
> > If I go around my F5 VIP so i know I am connecting straight to L-05-B
> and I
> > send any number of messages into the router there all of those messages
> end
> > up in ArtemisBrokerB. I was hoping that some would go to BrokerB and some
> > would go via RouterA over to BrokerA. Now when BrokerB is taken down
> > CURRENTLY the messages sent to Router L-05-B ARE routed through Router
> > L-03-A and then out to ArtemisBrokerB. (Currently receiving messages pull
> > them in from both hosts so my questions only lies in message production
> at
> > the moment.)
> >
> > Do you agree that with this configuration you would expect messages to be
> > load balanced between the two routes? I would like to paste in some of
> our
> > config and results from running qdstat. If you observe something that is
> > misconfigured and are able to highlight it i would be very appreciative.
> > First the config files for QDR instances. I tried my best to scrub them
> of
> > specific IPs and hostnames.
> >
> > *L-03-A qrouterd.conf*
> >
> >
> > *router {*
> >
> > * mode: interior*
> >
> > * id: Router.A*
> >
> > *}*
> >
> > *log {*
> >
> > * module: DEFAULT*
> >
> > * enable: debug+*
> >
> > * timestamp: yes*
> >
> > *}*
> >
> > *sslProfile {*
> >
> > * name: my-ssl*
> >
> > * certFile: /opt/org/my-ssl-info.pem*
> >
> > * keyFile: /opt/org/my-ssl-info.pem*
> >
> > * password: hellokitty42*
> >
> > *}*
> >
> > *listener {*
> >
> > * role: normal*
> >
> > * host: 0.0.0.0*
> >
> > * port: 5671*
> >
> > * authenticatePeer: no*
> >
> > * saslMechanisms: ANONYMOUS*
> >
> > * sslProfile: my-ssl*
> >
> > *}*
> >
> > *connector {*
> >
> > * name: local-artemis*
> >
> > * role: route-container*
> >
> > * host: L-03-A*
> >
> > * port: 61616*
> >
> > * saslMechanisms: ANONYMOUS*
> >
> > *}*
> >
> > *connector {*
> >
> > * name: routerb*
> >
> > * role: inter-router*
> >
> > * host: L-05-B*
> >
> > * port: 6671*
> >
> > * saslMechanisms: ANONYMOUS*
> >
> > *}*
> >
> > *address {*
> >
> > * prefix: foo.#*
> >
> > * waypoint: yes*
> >
> > * distribution: balanced*
> >
> > *}*
> >
> > *autoLink {*
> >
> > * addr: foo.bar*
> >
> > * dir: in*
> >
> > * connection: local-artemis*
> >
> > *}*
> >
> > *autoLink {*
> >
> > * addr: foo.bar*
> >
> > * dir: out*
> >
> > * connection: local-artemis*
> >
> > *}*
> >
> >
> > *L-05-B qrouterd.conf*
> >
> > *router {*
> >
> > * mode: interior*
> >
> > * id: Router.B*
> >
> > *}*
> >
> > *log {*
> >
> > * module: DEFAULT*
> >
> > * enable: debug+*
> >
> > * timestamp: yes*
> >
> > *}*
> >
> > *sslProfile {*
> >
> > * name: my-ssl*
> >
> > * certFile: /opt/org/my-ssl-info.pem*
> >
> > * keyFile: /opt/org/my-ssl-info.pem*
> >
> > * password: hellokitty42*
> >
> > *}*
> >
> > *listener {*
> >
> > * role: normal*
> >
> > * host: 0.0.0.0*
> >
> > * port: 5671*
> >
> > * authenticatePeer: no*
> >
> > * saslMechanisms: ANONYMOUS*
> >
> > * sslProfile: my-ssl*
> >
> > *}*
> >
> > *listener {*
> >
> > * role: inter-router*
> >
> > * host: 0.0.0.0*
> >
> > * port: 6671*
> >
> > * authenticatePeer: no*
> >
> > * saslMechanisms: ANONYMOUS*
> >
> > * sslProfile: my-ssl*
> >
> > *}*
> >
> > *connector {*
> >
> > * name: local-artemis*
> >
> > * role: route-container*
> >
> > * host: L-05-B*
> >
> > * port: 61616*
> >
> > * saslMechanisms: ANONYMOUS*
> >
> > *}*
> >
> > *address {*
> >
> > * prefix: foo.#*
> >
> > * waypoint: yes*
> >
> > * distribution: balanced*
> >
> > *}*
> >
> > *autoLink {*
> >
> > * addr: foo.bar*
> >
> > * dir: in*
> >
> > * connection: local-artemis*
> >
> > *}*
> >
> > *autoLink {*
> >
> > * addr: foo.bar*
> >
> > * dir: out*
> >
> > * connection: local-artemis*
> >
> > *}*
> >
> >
> >
> >
> > *L-03-A ~]$ qdstat -b:5671 -g*
> >
> > *Router Statistics*
> >
> > * attr value*
> >
> > * =====================*
> >
> > * Mode interior*
> >
> > * Area 0*
> >
> > * Router Id Router.A*
> >
> >
> > *L-05-B ~]$ qdstat -b:5671 -g*
> >
> > *Router Statistics*
> >
> > * attr value*
> >
> > * =====================*
> >
> > * Mode interior*
> >
> > * Area 0*
> >
> > * Router Id Router.B*
> >
> >
> >
> >
> > *L-03-A ~]$ qdstat -b:5671 -c*
> >
> > *Connections*
> >
> > * Id host container role
> > dir security authentication*
> >
> > *
> > ============================================================
> > ======================================================*
> >
> > * 14 L-03-A:61616 L-03-A
> > route-container out no-security anonymous-user*
> >
> > * 107132 L-05-B:6671 Router.B
> > inter-router out no-security anonymous-user*
> >
> > * 107267 127.0.0.1:37916 <http://127.0.0.1:37916>
> > 180851ba-7ce4-44d0-a937-97de16e781c4 normal in no-security
> > anonymous-user*
> >
> >
> > *L-05-B ~]$ qdstat -b:5671 -c*
> >
> > *Connections*
> >
> > * Id host container role
> > dir security authentication*
> >
> > *
> > ============================================================
> > ======================================================*
> >
> > * 1 L-05-B:61616 L-05-B
> > route-container out no-security anonymous-user*
> >
> > * 2 10.<ip>.32:57182 Router.A
> > inter-router in no-security anonymous-user*
> >
> > * 143 127.0.0.1:48610 <http://127.0.0.1:48610>
> > 93e6b3e3-443e-43d9-9c2c-0d648266ec19 normal in no-security
> > anonymous-user*
> >
> >
> >
> >
> > *L-03-A ~]$ qdstat -b:5671 -n*
> >
> > *Routers in the Network*
> >
> > * router-id next-hop link*
> >
> > * ===========================*
> >
> > * Router.A (self) -*
> >
> > * Router.B - 0*
> >
> >
> > *L-05-B ~]$ qdstat -b:5671 -n*
> >
> > *Routers in the Network*
> >
> > * router-id next-hop link*
> >
> > * ===========================*
> >
> > * Router.A - 0*
> >
> > * Router.B (self) -*
> >
> >
> >
> >
> > *L-03-A ~]$ qdstat -b:5671 --linkroutes*
> >
> > *Link Routes*
> >
> >
> > *L-05-B ~]$ qdstat -b:5671 --linkroutes*
> >
> > *Link Routes*
> >
> >
> >
> >
> > *L-03-A ~]$ qdstat -b:5671 --autolinks*
> >
> > *AutoLinks*
> >
> > * addr dir phs link status lastErr*
> >
> > * ==========================================*
> >
> > * foo.bar in 1 8 active*
> >
> > * foo.bar out 0 9 active*
> >
> >
> > *L-05-B ~]$ qdstat -b:5671 --autolinks*
> >
> > *AutoLinks*
> >
> > * addr dir phs link status lastErr*
> >
> > * ==========================================*
> >
> > * foo.bar in 1 4 active*
> >
> > * foo.bar out 0 5 active*
> >
> >
> >
> >
> > *L-03-A ~]$ qdstat -b:5671 -l*
> >
> > *Router Links*
> >
> > * type dir conn id id peer class addr
> > phs cap undel unsett del presett acc rej rel mod admin oper*
> >
> > *
> > ============================================================
> > ============================================================
> > ========================*
> >
> > * endpoint in 14 8 mobile foo.bar
> 1
> > 250 0 0 80 0 80 0 0 0 enabled up*
> >
> > * endpoint out 14 9 mobile foo.bar
> 0
> > 250 0 0 40 0 40 0 0 0 enabled up*
> >
> > * router-control in 107132 15
> > 250 0 0 524 524 0 0 0 0 enabled up*
> >
> > * router-control out 107132 16 local qdhello
> > 250 0 0 527 527 0 0 0 0 enabled up*
> >
> > * inter-router in 107132 17
> > 250 0 0 0 0 0 0 0 0 enabled up*
> >
> > * inter-router out 107132 18
> > 250 0 0 0 0 0 0 0 0 enabled up*
> >
> > * endpoint in 107336 33 mobile $management
> 0
> > 250 0 0 1 0 1 0 0 0 enabled up*
> >
> > * endpoint out 107336 34 local temp.liHSN39azHuUPkg
> > 250 0 0 0 0 0 0 0 0 enabled up*
> >
> >
> > *L-05-B ~]$ qdstat -b:5671 -l*
> >
> > *Router Links*
> >
> > * type dir conn id id peer class addr
> > phs cap undel unsett del presett acc rej rel mod admin oper*
> >
> > *
> > ============================================================
> > ============================================================
> > ========================*
> >
> > * endpoint in 1 4 mobile foo.bar
> 1
> > 250 40 0 0 0 0 0 0 0 enabled up*
> >
> > * endpoint out 1 5 mobile foo.bar
> 0
> > 250 0 0 0 0 0 0 0 0 enabled up*
> >
> > * router-control out 2 6 local qdhello
> > 250 0 0 524 524 0 0 0 0 enabled up*
> >
> > * router-control in 2 7
> > 250 0 0 527 527 0 0 0 0 enabled up*
> >
> > * inter-router out 2 8
> > 250 0 0 0 0 0 0 0 0 enabled up*
> >
> > * inter-router in 2 9
> > 250 0 0 0 0 0 0 0 0 enabled up*
> >
> > * endpoint in 212 24 mobile $management
> 0
> > 250 0 0 1 0 1 0 0 0 enabled up*
> >
> > * endpoint out 212 25 local temp.BJPESwMGFo+7qiy
> > 250 0 0 0 0 0 0 0 0 enabled up*
> >
> >
> >
> >
> >
> > *L-03-A ~]$ qdstat -b:5671 -a*
> >
> > *Router Addresses*
> >
> > * class addr phs distrib in-proc local remote
> > cntnr in out thru to-proc from-proc*
> >
> > *
> > ============================================================
> > ======================================================*
> >
> > * local $_management_internal closest 1 0 0
> > 0 0 0 0 6 6*
> >
> > * local $displayname closest 1 0 0
> > 0 0 0 0 0 0*
> >
> > * mobile $management 0 closest 1 0 0
> > 0 9 0 0 9 0*
> >
> > * local $management closest 1 0 0
> > 0 0 0 0 0 0*
> >
> > * router Router.B closest 0 0 1
> > 0 0 0 19 0 19*
> >
> > * mobile foo.bar 1 balanced 0 0 0
> > 0 80 520 0 0 0*
> >
> > * mobile foo.bar 0 balanced 0 1 1
> > 0 0 40 0 0 0*
> >
> > * local qdhello flood 1 1 0
> > 0 0 0 0 269,611 269,622*
> >
> > * local qdrouter flood 1 0 0
> > 0 0 0 0 12 0*
> >
> > * topo qdrouter flood 1 0 1
> > 0 0 0 8,992 8,991 8,996*
> >
> > * local qdrouter.ma <http://qdrouter.ma> multicast 1
> > 0 0 0 0 0 0 4 0*
> >
> > * topo qdrouter.ma <http://qdrouter.ma> multicast 1
> > 0 1 0 0 0 3 0 3*
> >
> > * local temp.u6NMF1AFV0JpdBi closest 0 1 0
> > 0 0 0 0 0 0*
> >
> >
> > *L-05-B ~]$ qdstat -b:5671 -a*
> >
> > *Router Addresses*
> >
> > * class addr phs distrib in-proc local remote
> > cntnr in out thru to-proc from-proc*
> >
> > *
> > ============================================================
> > =====================================================*
> >
> > * local $_management_internal closest 1 0 0
> > 0 0 0 0 6 6*
> >
> > * local $displayname closest 1 0 0
> > 0 0 0 0 0 0*
> >
> > * mobile $management 0 closest 1 0 0
> > 0 9 0 0 9 0*
> >
> > * local $management closest 1 0 0
> > 0 0 0 0 0 0*
> >
> > * router Router.A closest 0 0 1
> > 0 0 0 6 0 6*
> >
> > * mobile foo.bar 1 balanced 0 0 0
> > 0 0 0 0 0 0*
> >
> > * mobile foo.bar 0 balanced 0 1 1
> > 0 0 0 0 0 0*
> >
> > * local qdhello flood 1 1 0
> > 0 0 0 0 570 571*
> >
> > * local qdrouter flood 1 0 0
> > 0 0 0 0 4 0*
> >
> > * topo qdrouter flood 1 0 1
> > 0 0 0 20 21 21*
> >
> > * local qdrouter.ma <http://qdrouter.ma> multicast 1
> > 0 0 0 0 0 0 4 0*
> >
> > * topo qdrouter.ma <http://qdrouter.ma> multicast 1
> > 0 1 0 0 0 0 0 1*
> >
> > * local temp.xwHdS0zzku0s3lR closest 0 1 0
> > 0 0 0 0 0 0*
> >
> >
> >
> > is there something off in my configuration? or in my understand of QDR
> > netwokin just not up to snuff? should i care a little bit less about load
> > balancing because the VIP will round robin new connections out to
> different
> > codes?
> >
> > Thank you so much for your time and insight
> >
>
