Hi Ted,
Thanks for taking a look at the report and the comments.
I've included highlights of my M4 publisher, broker fed, and consumer codes
below. In brief my publishers are using async transfers, broker fed is using
qpid-route add (with tcp -t default to use ipoib), and consumers are using
LocalQueues. The consumer's subscription settings are below and we use these
because after trial and error we found they were giving us best peformance.
Regards,
Greg
/* IP ADDRS */
root_broker= ip address 1
level1_broker1= ip address 2
/* PUBLISHER CODE */
Connection connection;
ConnectionSettings conn_setngs;
Message message;
Session session;
// Setup the session
conn_setngs.host= root_broker;
conn_setngs.protocol= "tcp";
conn_setngs.tcpNoDelay= true;
connection.open(conn_setngs);
session = connection.newSession();
// Async msg xfer
async(session).messageTransfer(arg::content=message,
arg::destination="amq.fanout");
/* BROKER FEDERATION */
qpid-route route add $level1_broker1 $root_broker amq.fanout no_routing_key
.....and so on to form tree structures
/* CONSUMER CODE */
Connection connection;
ConnectionSettings conn_setngs;
Message message;
Session session;
LocalQueue local_my_queue;
SubscriptionSettings subscrptn_setngs;
// Setup the session
conn_setngs.host= level1_broker1;
conn_setngs.protocol= "tcp";
conn_setngs.tcpNoDelay= true;
connection.open(conn_setngs);
session = connection.newSession();
// Bind a queue
my_queue_name=session.getId().getName();
session.queueDeclare(arg::queue=my_queue_name, arg::exclusive=true,
arg::autoDelete=true);
session.exchangeBind(arg::exchange="amq.fanout",
arg::queue=my_queue_name,
arg::bindingKey="my-key");
// Set subscription settings
subscrptn_setngs.flowControl= FlowControl::unlimited();
subscrptn_setngs.autoAck= 1;
// Accept and acquire modes defined in ./src/gen/qpid/framing/enum.h
subscrptn_setngs.acceptMode= ACCEPT_MODE_EXPLICIT;
subscrptn_setngs.acquireMode= ACQUIRE_MODE_PRE_ACQUIRED;
// Subscribe to fanout via local queue
SubscriptionManager subscriptions(session);
subscriptions.subscribe(local_my_queue, my_queue_name,
subscrptn_setngs, my_queue_name);
// Get a message
producer_msg = local_my_queue.get();
// end codes
On Fri, 29 May 2009, Ted Ross wrote:
Hi Greg,
Thanks for sharing this report. Your detailed results are interesting,
especially since this is a level of benchmarking that we haven't gotten to
ourselves.
I'll take a whack at some of your questions in-line below. Hopefully others
will chime in as well.
First or all, I'm curious as to how you configured your federation routes.
The paper details the physical broker topologies but doesn't mention which
qpid-route options were used to set up your test networks. This, of course,
has performance implications.
Regards,
-Ted
gregory james marsh wrote:
Hi All,
A few other lab members and myself have just completed an Ohio State
University Technical Report dealing with Qpid broker federation entitled
"Scaling Advanced Message Queuing Protocol (AMQP) Architecture with Broker
Federation and InfiniBand". The pdf is publicly available at
http://nowlab.cse.ohio-state.edu/projects/amqp/ as well as
ftp://ftp.cse.ohio-state.edu/pub/tech-report/TRList.html
Basically this report tests federation with k-nomial tree topology. We
were unable to use Qpid rdma/InfiniBand (IB) due to problem documented on
JIRA (QPID-1855). So we performed our experiments using IPoIB via Qpid TCP
connector.
Based on what we saw in our experiments, each of us has written some
questions for the list about how federation is designed & implemented
internally. I've compiled the questions below. I make reference to
figures and tables from the report where appropriate.
1. In detail how does the connection of a destination broker to a source
broker with "qpid-route route add" differ from a consumer application
declaring and binding a queue on a source broker? Are there functional
differences in how messages are routed to a federated destination broker as
opposed to a consumer? What anticipated affect would these differences
have on host system performance?
In case you haven't already seen it, there's a Wiki page on federation at
http://qpid.apache.org/using-broker-federation.html.
In Qpid federation, the destination broker behaves as a normal AMQP client to
the source broker. With "qpid-route route add" and "qpid-route dynamic add",
the destination broker declares a private, auto-delete queue on the source
broker and creates a subscription to that queue. The "route add" command
causes the queue to be bound explicitly to an exchange on the source broker.
The "dynamic add" does not directly bind the queue but dynamically creates
and deletes bindings as consumer applications add bindings to the destination
exchange.
Federation does not introduce any differences in the way messages are
forwarded. It simply uses the normal AMQP mechanisms. The "internal"
federation client exposes less capability than is offered by the Qpid client
APIs. In particular, delivery policy and flow control. These are issues
that will be addressed in future development.
2. Why does a chain of federated brokers to one consumer have a higher
bytes/sec rate on their interfaces (using SYSSTAT/sar to collect traffic
data on IPoIB interface) than a single broker to one consumer? This
experiment is explained in Sections 3.1, 4.2, Figure 8a.
There are lots of variables here. The text and diagrams in the paper don't
provide enough detail to definitively answer this question. It's possible
that a different delivery policy on your clients (vs. the federation links)
is a cause of the disparity. In M4, the federation links are synchronous
(i.e. ack per message). In 0.5 (the next release), there's an option for
batching acknowledgements on federation links.
If your test clients use more efficient asynchronous transfers, the
federation links will have higher acknowledgement traffic than will the
normal client links (in M4).
3. Why does the activity at the root broker and producer increase as we
increase the federation tree? Our understanding was that the only work
root broker/producer does is send messages to broker(s) they are directly
connected to. (This is similar to question 2, but different experiment. In
such situations the net interfaces on the producers and consumers (as well
as brokers) of more complex trees showed elevatated bytes/sec rates
compared to less complex trees. Experiment is explained in Sections 3.3,
4.4, Figure 14a.)
I'd be interested to see exactly how you configured your federation routes in
this scenario.
4. We observed that the tree P-B-4B-16C was performing better than the
tree P-B-4B-4C. And I thought this was because more consumers being
attached keeps the leaf brokers busy thus reducing the contention at the
root broker. Can we confirm if this understanding is correct. (This is a
comparison of the experiments performed in Sections 3.3/4.4 and 5. Compare
numbers in Tables 5, 6, 7, 8, particularly the small message rate numbers
in Tables 5 & 8 for 64B and 256B.)
5. What is the advised measurement for performance in federated
broker-consumer design?
(We used the benchmarks we created for our SC08 workshop paper last
fall--also available at above sites. Delivered bandwidth in the current
report is our benchmark bandwidth multiplied by number of consumers).
6. Which of the tests included with Qpid release might help us get more
insights on our topology study?
Thanks in advance for an comments and insights.
Greg Marsh
Network Based Computing Lab
Ohio State University
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
