"V"c"'r few w s44d ! d re es2 www www s On Aug 8, 2016 3:56 AM, "Adel Boutros" <[email protected]> wrote:
> Hello guys, > > Just to wrap up, the last JMS tests performed with synchronous sending > were: > 1 Broker, 1 Dispatcher, 4 producers, 3 consumers, 4 connectors per broker > --> 6 100 msg/s. > 2 Broker, 1 Dispatcher, 4 producers, 3 consumers, 4 connectors per broker > --> 5 800 msg/s. > 2 Broker, 1 Dispatcher, 8 producers, 3 consumers, 4 connectors per broker > --> 7 600 msg/s. > 2 Broker, 1 Dispatcher, 12 producers, 3 consumers, 4 connectors per broker > --> 8 100 msg/s. > > In conclusion: > * The dispatch router itself is capable of handling high load of data. > * The Java Broker is capable of handling high load of data. > * Increasing the number of connectors increases the performance until > other components become the bottleneck (Doubling the producers increased > the throughput in the case of 2 brokers) > * Having a pool of connections as a config parameter just like > "workerThreads" might be considered as a neater option than defining > multiple connectors with their autolinks. > * JMS overhead and serialization/de-serialization might be also a > bottleneck. > > Regards, > Adel > > > From: [email protected] > > Date: Thu, 4 Aug 2016 10:58:13 +0100 > > Subject: Re: [Performance] Benchmarking Qpid dispatch router 0.6.0 with > Qpid Java Broker 6.0.0 > > To: [email protected] > > > > I havent seen the code Ulf is using, but I would guess...edit: ninja'd > > by Ulf while I was looking at something else, deleted ;) > > > > The reactor Ulf is using is a good bit lower level and has a > > significantly different threading and application usage model than the > > JMS client, so they will differ a good amount from that alone, but we > > can likely improve on the JMS clients performance still. > > > > Another big reason they will also typically differ beyond their basic > > architecture though is that they will often send very different > > messages on the wire for what may seem on the face of it like similar > > messages at the application level, as there is a good amount of > > metadata related to supporting behaviours required of a JMS client. > > Unless you were to code the reactor based sender to send more similar > > content (which obviously in some of the cases might not actually make > > sense), then the messages themselves aren't really equivalent. I'd > > guess that the messages being used in the reactor sender are body > > section only (is the body reused?), whereas the ones the JMS client is > > sending will have properties, header and annotations sections on top > > with content in each of those. Some of that content is going to be > > general purpose stuff a reactor based sender might want to send too > > (e.g message-id) whereas other bits are just JMS-client specific > > meta-data it likely wouldnt. > > > > Robbie > > > > On 4 August 2016 at 09:40, Adel Boutros <[email protected]> wrote: > > > Our producers/consumers actually logs the elapsed time. This was > slowing down the test. I deactivated the logging and with a dispatcher > only, I am at around 47 000 msg/s with asynchronous sending. > > > > > >> From: [email protected] > > >> To: [email protected] > > >> Subject: RE: [Performance] Benchmarking Qpid dispatch router 0.6.0 > with Qpid Java Broker 6.0.0 > > >> Date: Wed, 3 Aug 2016 18:39:23 +0200 > > >> > > >> And how do you measure your throughput? > > >> > > >> > From: [email protected] > > >> > To: [email protected] > > >> > Subject: RE: [Performance] Benchmarking Qpid dispatch router 0.6.0 > with Qpid Java Broker 6.0.0 > > >> > Date: Wed, 3 Aug 2016 18:38:12 +0200 > > >> > > > >> > Hello Ulf, > > >> > > > >> > I am sending messages with a byte array of 100 bytes and I am using > Berkley DB as a message store (which should be slower than having memory > only message store, no?) > > >> > > > >> > With 1 consumer, 1 producer and no broker, I am at 33k msgs/sec if > they are all on the same machine and I have set "jms.forceAsyncSend=true" > on the producer and "jms.sendAcksAsync=true" for the consumer. > > >> > > > >> > Are you using other options to get 190k? Do you think JMS might be > a bottleneck? Or something else in my config/test? > > >> > > > >> > JMS client 0.9.0 > > >> > Qpid Java Broker 6.0.1 > > >> > Dispatcher 0.6.0 > > >> > > > >> > Adel > > >> > > > >> > > Subject: Re: [Performance] Benchmarking Qpid dispatch router > 0.6.0 with Qpid Java Broker 6.0.0 > > >> > > To: [email protected] > > >> > > From: [email protected] > > >> > > Date: Wed, 3 Aug 2016 16:23:06 +0200 > > >> > > > > >> > > Hi, > > >> > > > > >> > > Excuse me if this was already mentioned somewhere, but what is > the size > > >> > > of the messages you are sending ? > > >> > > > > >> > > FWIW, I'm able to get around 30-40k msgs/sec sustained with 1 > producer, > > >> > > 1 consumer, 1 dispatch (4 worker threads) and 1 broker > (activemq-5). The > > >> > > sender sends unsettled messages as fast as it can using > qpid-proton > > >> > > reactor API which is sending async up to the window limit. > > >> > > > > >> > > With no broker involved, I'm getting ~190k msgs/sec. > > >> > > > > >> > > All of these numbers are from my 8 core laptop. Message size is > 128 bytes. > > >> > > > > >> > > I don't know the dispatcher that well, but I think it should be > able to > > >> > > handle data from each connector just fine given the numbers I > have seen. > > >> > > > > >> > > On 08/03/2016 02:41 PM, Adel Boutros wrote: > > >> > > > > > >> > > > > > >> > > > > > >> > > > Hello again, > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > As requested, I added a 2nd connector and the appropriate > autoLinks on the same host/port but with a different name. It seems to have > resolved the issue. > > >> > > > > > >> > > > 1 Broker, 1 Dispatcher, 3 producers, 0 consumers, 1 connectors > --> 5000 msg/s. > > >> > > > 1 Broker, 1 Dispatcher, 3 producers, 0 consumers, 2 connectors > --> 6600 msg/s. > > >> > > > 1 Broker, 1 Dispatcher, 4 producers, 0 consumers, 2 connectors > --> 7700 msg/s. > > >> > > > > > >> > > > I think this confirms the problem is due to the fact a single > connection is being shared by all clients (consumers/producers) and that > having a sort of pool of connections or a connection per workerThread is a > solution to consider. > > >> > > > > > >> > > > What do you think? > > >> > > > > > >> > > > I added a 3rd connector to see if it changes anything but it > > >> > > > didn't. Do you think this is maybe because the dispatcher is > not able > > >> > > > to process fast enough and saturate the 2 connectors? > > >> > > > 1 Broker, 1 Dispatcher, 4 producers, 0 consumers, 3 connectors > --> 7700 msg/s. > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > Adel > > >> > > > > > >> > > >> From: [email protected] > > >> > > >> To: [email protected] > > >> > > >> Subject: RE: [Performance] Benchmarking Qpid dispatch router > 0.6.0 with Qpid Java Broker 6.0.0 > > >> > > >> Date: Tue, 2 Aug 2016 22:21:54 +0200 > > >> > > >> > > >> > > >> Sorry for the typo. Indeed, it was with 3 producers. I used 4 > and 8 workerThread but there wasn't a difference. > > >> > > >> We want to benchmark in the worst case scenarios actually to > see what is the minimum we can guarantee. This is why we are using > synchronous sending. In the future, we will also benchmark with full > SSL/SASL to see what impact it has on the performance. > > >> > > >>> Subject: Re: [Performance] Benchmarking Qpid dispatch router > 0.6.0 with Qpid Java Broker 6.0.0 > > >> > > >>> To: [email protected] > > >> > > >>> From: [email protected] > > >> > > >>> Date: Tue, 2 Aug 2016 20:41:54 +0100 > > >> > > >>> > > >> > > >>> On 02/08/16 20:25, Adel Boutros wrote: > > >> > > >>>> How about the tests we did with consumer/producers connected > directly to the dispatcher without any broker where we had 16 000 msg/s > with 4 producers. Is it also a very low value given that there is no > persistence or storing here? It was also synchronous sending. > > >> > > >>> > > >> > > >>> The rate is low because it is synchronous. One messages is > sent to the > > >> > > >>> consumer who acknowledges it, the acknowledgement is then > conveyed back > > >> > > >>> to the sender who then can send the next message. > > >> > > >>> > > >> > > >>> The rate for a single producer through the router was 6,000 > per second. > > >> > > >>> That works out as a roundtrip time of 167 microsecs or so. In > your > > >> > > >>> table, the 16,000 rate was listed as being for 3 producers. > Based on the > > >> > > >>> rate of a single producer, the best you could hope for there > is 3 * > > >> > > >>> 6,000 i.e 18,000. (How many worker threads did you have on > the router > > >> > > >>> for that case?) > > >> > > >>> > > >> > > >>> ------------------------------------------------------------ > --------- > > >> > > >>> To unsubscribe, e-mail: [email protected] > > >> > > >>> For additional commands, e-mail: [email protected] > > >> > > >>> > > >> > > >> > > >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > -- > > >> > > Ulf > > >> > > > > >> > > ------------------------------------------------------------ > --------- > > >> > > To unsubscribe, e-mail: [email protected] > > >> > > For additional commands, e-mail: [email protected] > > >> > > > > >> > > > >> > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > >
