Hi Greg,
Thanks for sharing this report. Your detailed results are interesting,
especially since this is a level of benchmarking that we haven't gotten
to ourselves.
I'll take a whack at some of your questions in-line below. Hopefully
others will chime in as well.
First or all, I'm curious as to how you configured your federation
routes. The paper details the physical broker topologies but doesn't
mention which qpid-route options were used to set up your test
networks. This, of course, has performance implications.
Regards,
-Ted
gregory james marsh wrote:
Hi All,
A few other lab members and myself have just completed an Ohio State
University Technical Report dealing with Qpid broker federation
entitled "Scaling Advanced Message Queuing Protocol (AMQP)
Architecture with Broker Federation and InfiniBand". The pdf is
publicly available at http://nowlab.cse.ohio-state.edu/projects/amqp/
as well as ftp://ftp.cse.ohio-state.edu/pub/tech-report/TRList.html
Basically this report tests federation with k-nomial tree topology.
We were unable to use Qpid rdma/InfiniBand (IB) due to problem
documented on JIRA (QPID-1855). So we performed our experiments using
IPoIB via Qpid TCP connector.
Based on what we saw in our experiments, each of us has written some
questions for the list about how federation is designed & implemented
internally. I've compiled the questions below. I make reference to
figures and tables from the report where appropriate.
1. In detail how does the connection of a destination broker to a
source broker with "qpid-route route add" differ from a consumer
application declaring and binding a queue on a source broker? Are
there functional differences in how messages are routed to a federated
destination broker as opposed to a consumer? What anticipated affect
would these differences have on host system performance?
In case you haven't already seen it, there's a Wiki page on federation
at http://qpid.apache.org/using-broker-federation.html.
In Qpid federation, the destination broker behaves as a normal AMQP
client to the source broker. With "qpid-route route add" and
"qpid-route dynamic add", the destination broker declares a private,
auto-delete queue on the source broker and creates a subscription to
that queue. The "route add" command causes the queue to be bound
explicitly to an exchange on the source broker. The "dynamic add" does
not directly bind the queue but dynamically creates and deletes bindings
as consumer applications add bindings to the destination exchange.
Federation does not introduce any differences in the way messages are
forwarded. It simply uses the normal AMQP mechanisms. The "internal"
federation client exposes less capability than is offered by the Qpid
client APIs. In particular, delivery policy and flow control. These
are issues that will be addressed in future development.
2. Why does a chain of federated brokers to one consumer have a higher
bytes/sec rate on their interfaces (using SYSSTAT/sar to collect
traffic data on IPoIB interface) than a single broker to one
consumer? This experiment is explained in Sections 3.1, 4.2, Figure 8a.
There are lots of variables here. The text and diagrams in the paper
don't provide enough detail to definitively answer this question. It's
possible that a different delivery policy on your clients (vs. the
federation links) is a cause of the disparity. In M4, the federation
links are synchronous (i.e. ack per message). In 0.5 (the next
release), there's an option for batching acknowledgements on federation
links.
If your test clients use more efficient asynchronous transfers, the
federation links will have higher acknowledgement traffic than will the
normal client links (in M4).
3. Why does the activity at the root broker and producer increase as
we increase the federation tree? Our understanding was that the only
work root broker/producer does is send messages to broker(s) they are
directly connected to. (This is similar to question 2, but different
experiment. In such situations the net interfaces on the producers and
consumers (as well as brokers) of more complex trees showed elevatated
bytes/sec rates compared to less complex trees. Experiment is
explained in Sections 3.3, 4.4, Figure 14a.)
I'd be interested to see exactly how you configured your federation
routes in this scenario.
4. We observed that the tree P-B-4B-16C was performing better than
the tree P-B-4B-4C. And I thought this was because more consumers
being attached keeps the leaf brokers busy thus reducing the
contention at the root broker. Can we confirm if this understanding is
correct. (This is a comparison of the experiments performed in
Sections 3.3/4.4 and 5. Compare numbers in Tables 5, 6, 7, 8,
particularly the small message rate numbers in Tables 5 & 8 for 64B
and 256B.)
5. What is the advised measurement for performance in federated
broker-consumer design?
(We used the benchmarks we created for our SC08 workshop paper last
fall--also available at above sites. Delivered bandwidth in the
current report is our benchmark bandwidth multiplied by number of
consumers).
6. Which of the tests included with Qpid release might help us get
more insights on our topology study?
Thanks in advance for an comments and insights.
Greg Marsh
Network Based Computing Lab
Ohio State University
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
