Re: Java Broker performance

[Oleksandr Rudyy]

Hi Tomas,

Thanks for feedback. I will merge the changes into 7.0.1.

We will be looking into performance improvements for AMQP 1.0 later
this year. Though, we do not have a date exactly when we will be doing
this work.
The changes we had implemented are tactical. They are small enough to
be ported into 7.0.x branch.

For the use cases exercised by your sample applications I cannot think
about any Broker settings which would improve the performance farther.
Hopefully the following work will increase the performance and resolve
the Broker-J lagging behind the C++ broker.

Kind Regards,
Alex


On 29 January 2018 at 09:40, Tomas Soltys  wrote:
> Hi Alex,
>
> The performance is way much better now and is acceptable for us. Thank you
> very much for your effort.
>
> Do you know why the performance is still lagging behind the C++ broker? Is
> it something that can be influenced by the broker settings?
>
> Thanks and regards,
> Tomas
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

[Tomas Soltys]

Hi Alex,

The performance is way much better now and is acceptable for us. Thank you
very much for your effort.

Do you know why the performance is still lagging behind the C++ broker? Is
it something that can be influenced by the broker settings?

Thanks and regards,
Tomas



--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

[Oleksandr Rudyy]

Hi Tomas,

We are going to port the implemented changes into 7.0.x branch for
inclusion into 7.0.1.
Please, let us know whether the sending and receiving performances are
acceptable for you.

Kind Regards,
Alex


On 23 January 2018 at 15:59, Oleksandr Rudyy  wrote:
> Hi Tomas,
>
> Today I committed changes [1] into master improving the performance of
> the broker-j for the Receiver use case. The commit is made against
> QPID-8032.
>
> I tested the changes locally on my machine and could see the
> improvements in broker-j performance. Before the change the dequeue
> operations in broker-j on my test machine were 7-8 times slower than
> the ones in cpp broker.
> After the change the performance has increased  but the broker-j is
> still 2-3 times slower than cpp broker.
>
> Please give it a try on your environment and let us know about your
> performance results.
>
> Kind Regards,
> Alex
>
>
> [1] https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=a1ed997
>
> On 22 January 2018 at 16:44, Oleksandr Rudyy  wrote:
>> Hi Tomas,
>>
>> We are currently looking into performance improvements for the Receiver.
>> We will notify you after necessary changes are made and ready for testing.
>>
>> Kind Regards,
>> Alex
>>
>>
>>
>> On 22 January 2018 at 11:20, Tomas Soltys  wrote:
>>> Hi Keith,
>>>
>>> I can confirm that sending is now much faster.
>>>
>>> *C++ broker*
>>> $ ./Broadcaster 20001
>>> real0m0.085s
>>> user0m0.027s
>>> sys 0m0.005s
>>>
>>> *Java broker*
>>> $ ./Broadcaster 20002
>>> real0m0.876s
>>> user0m0.037s
>>> sys 0m0.011s
>>>
>>> However receiving is still much slower.
>>>
>>> *C++ broker*
>>> $ time ./Receiver 20001
>>> real0m0.113s
>>> user0m0.035s
>>> sys 0m0.014s
>>>
>>> *Java broker*
>>> $ time ./Receiver 20002
>>> real0m50.168s
>>> user0m0.061s
>>> sys 0m0.032s
>>>
>>> See attached file  cpp_vs_java.gz
>>>    which
>>> contains:
>>> Broadcaster.cpp and Receiver.cpp based on qpid-proton 0.19.0
>>> Setup for C++ broker 1.37.0
>>> Setup for Java broker (master from 22-nd January 2017)
>>>
>>> Regards,
>>> Tomas
>>>
>>>
>>>
>>>
>>> --
>>> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>>>
>>> -
>>> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
>>> For additional commands, e-mail: users-h...@qpid.apache.org
>>>

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

[Oleksandr Rudyy]

Hi Tomas,

Today I committed changes [1] into master improving the performance of
the broker-j for the Receiver use case. The commit is made against
QPID-8032.

I tested the changes locally on my machine and could see the
improvements in broker-j performance. Before the change the dequeue
operations in broker-j on my test machine were 7-8 times slower than
the ones in cpp broker.
After the change the performance has increased  but the broker-j is
still 2-3 times slower than cpp broker.

Please give it a try on your environment and let us know about your
performance results.

Kind Regards,
Alex


[1] https://git-wip-us.apache.org/repos/asf?p=qpid-broker-j.git;h=a1ed997

On 22 January 2018 at 16:44, Oleksandr Rudyy  wrote:
> Hi Tomas,
>
> We are currently looking into performance improvements for the Receiver.
> We will notify you after necessary changes are made and ready for testing.
>
> Kind Regards,
> Alex
>
>
>
> On 22 January 2018 at 11:20, Tomas Soltys  wrote:
>> Hi Keith,
>>
>> I can confirm that sending is now much faster.
>>
>> *C++ broker*
>> $ ./Broadcaster 20001
>> real0m0.085s
>> user0m0.027s
>> sys 0m0.005s
>>
>> *Java broker*
>> $ ./Broadcaster 20002
>> real0m0.876s
>> user0m0.037s
>> sys 0m0.011s
>>
>> However receiving is still much slower.
>>
>> *C++ broker*
>> $ time ./Receiver 20001
>> real0m0.113s
>> user0m0.035s
>> sys 0m0.014s
>>
>> *Java broker*
>> $ time ./Receiver 20002
>> real0m50.168s
>> user0m0.061s
>> sys 0m0.032s
>>
>> See attached file  cpp_vs_java.gz
>>    which
>> contains:
>> Broadcaster.cpp and Receiver.cpp based on qpid-proton 0.19.0
>> Setup for C++ broker 1.37.0
>> Setup for Java broker (master from 22-nd January 2017)
>>
>> Regards,
>> Tomas
>>
>>
>>
>>
>> --
>> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
>> For additional commands, e-mail: users-h...@qpid.apache.org
>>

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

[Oleksandr Rudyy]

Hi Tomas,

We are currently looking into performance improvements for the Receiver.
We will notify you after necessary changes are made and ready for testing.

Kind Regards,
Alex



On 22 January 2018 at 11:20, Tomas Soltys  wrote:
> Hi Keith,
>
> I can confirm that sending is now much faster.
>
> *C++ broker*
> $ ./Broadcaster 20001
> real0m0.085s
> user0m0.027s
> sys 0m0.005s
>
> *Java broker*
> $ ./Broadcaster 20002
> real0m0.876s
> user0m0.037s
> sys 0m0.011s
>
> However receiving is still much slower.
>
> *C++ broker*
> $ time ./Receiver 20001
> real0m0.113s
> user0m0.035s
> sys 0m0.014s
>
> *Java broker*
> $ time ./Receiver 20002
> real0m50.168s
> user0m0.061s
> sys 0m0.032s
>
> See attached file  cpp_vs_java.gz
>    which
> contains:
> Broadcaster.cpp and Receiver.cpp based on qpid-proton 0.19.0
> Setup for C++ broker 1.37.0
> Setup for Java broker (master from 22-nd January 2017)
>
> Regards,
> Tomas
>
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

[Tomas Soltys]

Hi Keith,

I can confirm that sending is now much faster.

*C++ broker*
$ ./Broadcaster 20001
real0m0.085s
user0m0.027s
sys 0m0.005s

*Java broker*
$ ./Broadcaster 20002
real0m0.876s
user0m0.037s
sys 0m0.011s

However receiving is still much slower.

*C++ broker*
$ time ./Receiver 20001
real0m0.113s
user0m0.035s
sys 0m0.014s

*Java broker*
$ time ./Receiver 20002
real0m50.168s
user0m0.061s
sys 0m0.032s

See attached file  cpp_vs_java.gz
   which
contains:
Broadcaster.cpp and Receiver.cpp based on qpid-proton 0.19.0
Setup for C++ broker 1.37.0
Setup for Java broker (master from 22-nd January 2017)

Regards,
Tomas




--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

[Keith W]

Hi Tomas,

The work for QPID-8032 is done on master.   If you could repeat your
test-case with Broker-J compiled for master and let us know how
performance changes (improves, hopefully a lot).  Once I have heard
back from you I'll look to have this included in a 7.0.1 very soon.

Kind regards, Keith Wall.

On 26 November 2017 at 14:54, Keith W  wrote:
> Hi Tomas
>
> Thanks for the attachments.
>
> With your Broadcaster code, which sends persistent messages
> asynchronously, I do see inferior performance from Broker J than the
> CPP Broker.  I am using proton master (fa80534)
>
> Currently for this use-case, Broker-J commits synchronously after each
> delivery (see StandardReceivingLinkEndpoint#receiveDelivery.  The
> pertinent part is its use of an AutoCommitTransaction and the fact
> that AutoCommitTransaction#enqueue uses a synchronous #commitTran) and
> this will explain some (if not all) of the performance difference.  As
> Rob mentioned on the 10th November in this thread, the older protocols
> already have an optimisation for this use-case (involving
> AsyncAutoCommitTransaction) which should improve performance on the
> AMQP 1.0 path.   This was raised as QPID-8032. I try and include this
> in a 7.0.1 soon.
>
> This doesn't explain your observation about performance when using
> Qpid JMS Client which is doing a synchronous send of persistent
> messages, but as I commented above, I cannot reproduce the problem: I
> see very similar performance for Broker-J and CPP on my hardware.
>
> cheers Keith.
>
> On 24 November 2017 at 13:06, Tomas Soltys  wrote:
>> Hi Keith,
>>
>> Please find attached  cpp_vs_java.gz
>>   . This
>> archive contains:
>> * *java* - setup of Java broker (v7.0.0)
>> * *cpp* - setup of C++ broker (v1.36.0)
>> * *proton-client* - C++client based on Qpid proton (v0.18.1)
>> * *java_trace.log* - trace log from client sending 20 messages (10240 Bytes
>> each) to Java broker
>> * *java_trace.log* - trace log from client sending 20 messages (10240 Bytes
>> each) to C++ broker
>>
>> One thing I've noticed in logs is that C++ broker is sending dispositions in
>> chunks of 5 whereas Java broker does this for each message separately.
>>
>> Best regards,
>> Tomas
>>
>>
>>
>> --
>> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
>> For additional commands, e-mail: users-h...@qpid.apache.org
>>

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

[Keith W]

Hi Tomas

Thanks for the attachments.

With your Broadcaster code, which sends persistent messages
asynchronously, I do see inferior performance from Broker J than the
CPP Broker.  I am using proton master (fa80534)

Currently for this use-case, Broker-J commits synchronously after each
delivery (see StandardReceivingLinkEndpoint#receiveDelivery.  The
pertinent part is its use of an AutoCommitTransaction and the fact
that AutoCommitTransaction#enqueue uses a synchronous #commitTran) and
this will explain some (if not all) of the performance difference.  As
Rob mentioned on the 10th November in this thread, the older protocols
already have an optimisation for this use-case (involving
AsyncAutoCommitTransaction) which should improve performance on the
AMQP 1.0 path.   This was raised as QPID-8032. I try and include this
in a 7.0.1 soon.

This doesn't explain your observation about performance when using
Qpid JMS Client which is doing a synchronous send of persistent
messages, but as I commented above, I cannot reproduce the problem: I
see very similar performance for Broker-J and CPP on my hardware.

cheers Keith.

On 24 November 2017 at 13:06, Tomas Soltys  wrote:
> Hi Keith,
>
> Please find attached  cpp_vs_java.gz
>   . This
> archive contains:
> * *java* - setup of Java broker (v7.0.0)
> * *cpp* - setup of C++ broker (v1.36.0)
> * *proton-client* - C++client based on Qpid proton (v0.18.1)
> * *java_trace.log* - trace log from client sending 20 messages (10240 Bytes
> each) to Java broker
> * *java_trace.log* - trace log from client sending 20 messages (10240 Bytes
> each) to C++ broker
>
> One thing I've noticed in logs is that C++ broker is sending dispositions in
> chunks of 5 whereas Java broker does this for each message separately.
>
> Best regards,
> Tomas
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

[Tomas Soltys]

Hi Keith,

Please find attached  cpp_vs_java.gz
  . This
archive contains:
* *java* - setup of Java broker (v7.0.0)
* *cpp* - setup of C++ broker (v1.36.0)
* *proton-client* - C++client based on Qpid proton (v0.18.1)
* *java_trace.log* - trace log from client sending 20 messages (10240 Bytes
each) to Java broker
* *java_trace.log* - trace log from client sending 20 messages (10240 Bytes
each) to C++ broker

One thing I've noticed in logs is that C++ broker is sending dispositions in
chunks of 5 whereas Java broker does this for each message separately.

Best regards,
Tomas



--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

[Robbie Gemmell]

Your earlier mail was clear Tomas, I didn't even know you replied to
me as I don't use the Nabble interface. I just wanted to reiterate it
is also important we know which client(s) and which code any
comparisons are performed with otherwise the numbers simply cant be
reasoned about fully. While different clients might act/perform
similarly in some cases against particular servers, in others cases
they can/will be very different.

The example JMS code you gave earlier is doing a synchronous send of a
persistent message, so if you are running single producer synchronous
sends on a hard disk that can only do say ~100syncs/sec, and getting
vastly faster results in one case, then something is probably awry. In
such a situation, both brokers performance should be gated by the
synchronous sends and need to sync to disk before responding to the
client. Keiths testing suggested that is the case, seeing similar
results for both brokers in his testing.

In the case of a C[++] client doing asynchronous sends then it would
currently be expected for the C++ broker to be significantly faster
based on the details Rob gave, as even if waiting for settlement after
100 messages its still mostly async in comparison. Your example code
didnt do that however, so again seeing what you are actually doing
would be needed to fully reason about it.

You mention Qpid JMS 0.23.0 below, while earlier in the thread you
mentioned 0.27.0 when giving example code. Did you actually change
clients? Did you use the same JMS application code provided in this
thread in both cases? If you did use that example JMS code, I would
actually have to start questioning if the messages are really being
persisted (and synced to disk before response) in both cases if you
are seeing radically different numbers between them.

Following Keith's suggestion would be a good idea.

Robbie

On 24 November 2017 at 10:58, Tomas Soltys  wrote:
> Hi Robbie,
>
> I just realized that I placed my response to incorrect person. It supposed
> to be a reply to Keith's message.
>
> To your questions. The test was performed with C client based on proton
> 0.18.1. However, I got very similar results also with qpid-send tool which
> settles after each message.
>



> I've also tried sending using 0-10 and 1.0 protocol versions but no
> significant differences.
>
> I also executed test client based on Qpid JMS 0.23.0 with very similar
> results. In all cases C++ broker was able to settle and send acknowledgment
> way much faster than java broker.
>
> Is there something in the settings that can be tweaked to improve IO
> performance?
>
> Regards,
> Tomas
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

[Keith W]

Hi Tomas,

Can I suggest that you share the logs from both brokers for the two
amq_send.sh test?   Also separately, I suggest a run of amq_send.sh
with a small number of message, say message-count 20, with Proton
trace logging enabled (export PN_TRACE_FRM=true) on the client side.
Repeat this with both Brokers and share.

With regard to Broker-J performance, we know that BDB will outperform
Derby for many use-cases.  The BDB plugin has received much more
tuning over the years than Derby.  I know you have already said you
tested with both earlier in the thread, but I wanted to point it out.

cheers Keith



On 24 November 2017 at 10:58, Tomas Soltys  wrote:
> Hi Robbie,
>
> I just realized that I placed my response to incorrect person. It supposed
> to be a reply to Keith's message.
>
> To your questions. The test was performed with C client based on proton
> 0.18.1. However, I got very similar results also with qpid-send tool which
> settles after each message.
>
> I've also tried sending using 0-10 and 1.0 protocol versions but no
> significant differences.
>
> I also executed test client based on Qpid JMS 0.23.0 with very similar
> results. In all cases C++ broker was able to settle and send acknowledgment
> way much faster than java broker.
>
> Is there something in the settings that can be tweaked to improve IO
> performance?
>
> Regards,
> Tomas
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

[Tomas Soltys]

Hi Robbie,

I just realized that I placed my response to incorrect person. It supposed
to be a reply to Keith's message.

To your questions. The test was performed with C client based on proton
0.18.1. However, I got very similar results also with qpid-send tool which
settles after each message.

I've also tried sending using 0-10 and 1.0 protocol versions but no
significant differences.

I also executed test client based on Qpid JMS 0.23.0 with very similar
results. In all cases C++ broker was able to settle and send acknowledgment
way much faster than java broker.

Is there something in the settings that can be tweaked to improve IO
performance?

Regards,
Tomas



--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

[Robbie Gemmell]

Could you answer my question around confirming what client you are
using for the comparison below, the C++ one or the JMS one?

I'm guessing it is the C++ one. In which case, Rob's thoughts are the
likely explanation for why the Java broker isnt any faster with it
than you are seeing, and testing with the JMS client code given will
not be able to demonstrate that same difference between the two
brokers as it is doing a synchronous publish while the C++ code isnt.

Robbie

On 23 November 2017 at 15:33, tomas.soltys  wrote:
> Hi Keith,
>
> I'm still getting huge differences, but I still hope it is related to how I
> configured my brokers. Please find attached file  brokers.gz
>    containing
> setup of my two brokers (cpp and java). Both I tried to setup to be as
> similar as possible.
>
> With my test send tool I sent 1000 messages each 102400 bytes. Tool sends
> messages asynchronously and settle after each 100-th message.
>
> Output (2001 - cpp, 2002 - java):
>
> $ time amq_send.sh --host=localhost --port=20001 --user=BE --pass="BE"
> --node-name=broadcast --subject="broadcast.PublicRejectStream"
> --message-count=1000 --print-message=0 --message-size=102400
> --settle-rate=100
>   100 messages sent
>   200 messages sent
>   300 messages sent
>   400 messages sent
>   500 messages sent
>   600 messages sent
>   700 messages sent
>   800 messages sent
>   900 messages sent
>  1000 messages sent
> Time to send = 1 [seconds]
>
> real0m1.796s
> user0m0.125s
> sys 0m0.050s
>
> $ time amq_send.sh --host=localhost --port=20002 --user=BE --pass="BE"
> --node-name=broadcast --subject="broadcast.PublicRejectStream"
> --message-count=1000 --print-message=0 --message-size=102400
> --settle-rate=100
>   100 messages sent
>   200 messages sent
>   300 messages sent
>   400 messages sent
>   500 messages sent
>   600 messages sent
>   700 messages sent
>   800 messages sent
>   900 messages sent
>  1000 messages sent
> Time to send = 109 [seconds]
>
> real1m48.865s
> user0m1.504s
> sys 0m1.558s
>
> Best regards,
> Tomas
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

[tomas.soltys]

Hi Keith,

I'm still getting huge differences, but I still hope it is related to how I
configured my brokers. Please find attached file  brokers.gz
   containing
setup of my two brokers (cpp and java). Both I tried to setup to be as
similar as possible.

With my test send tool I sent 1000 messages each 102400 bytes. Tool sends
messages asynchronously and settle after each 100-th message.

Output (2001 - cpp, 2002 - java):

$ time amq_send.sh --host=localhost --port=20001 --user=BE --pass="BE"
--node-name=broadcast --subject="broadcast.PublicRejectStream"
--message-count=1000 --print-message=0 --message-size=102400
--settle-rate=100
  100 messages sent
  200 messages sent
  300 messages sent
  400 messages sent
  500 messages sent
  600 messages sent
  700 messages sent
  800 messages sent
  900 messages sent
 1000 messages sent
Time to send = 1 [seconds]

real0m1.796s
user0m0.125s
sys 0m0.050s

$ time amq_send.sh --host=localhost --port=20002 --user=BE --pass="BE"
--node-name=broadcast --subject="broadcast.PublicRejectStream"
--message-count=1000 --print-message=0 --message-size=102400
--settle-rate=100
  100 messages sent
  200 messages sent
  300 messages sent
  400 messages sent
  500 messages sent
  600 messages sent
  700 messages sent
  800 messages sent
  900 messages sent
 1000 messages sent
Time to send = 109 [seconds]

real1m48.865s
user0m1.504s
sys 0m1.558s

Best regards,
Tomas



--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

[Keith W]

On 15 November 2017 at 14:13, Gordon Sim  wrote:
> On 10/11/17 18:11, Keith W wrote:
>>
>> I infer that CPP Broker must be optimistically sending the
>> Disposition back to the client before the data is sync'd, so that is
>> why you see better performance (but with a lesser guarantee).
>
>
> I don't believe that is the case. The c++ broker only sends back the
> disposition once confirmed by the store (and the linear store only confirms
> once synced).
>

Agreed. I retract my statement.  I've now tested Tomas's JMS
reproduction against the C++ Broker with linearstore with durable
queue and Broker-J on the same hardware.   I got similar throughput
numbers in both cases.



> Are you sure that (a) the store was in use and (b) you were waiting for
> acknowledgement of one message before sending the next (sending
> asynchronously is obviously faster)?
>

Tomas, can you tell us more?

>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

[Gordon Sim]

On 10/11/17 18:11, Keith W wrote:

I infer that CPP Broker must be optimistically sending the
Disposition back to the client before the data is sync'd, so that is
why you see better performance (but with a lesser guarantee).


I don't believe that is the case. The c++ broker only sends back the 
disposition once confirmed by the store (and the linear store only 
confirms once synced).


Are you sure that (a) the store was in use and (b) you were waiting for 
acknowledgement of one message before sending the next (sending 
asynchronously is obviously faster)?


-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

[Robbie Gemmell]

On 10 November 2017 at 15:39, Vavricka  wrote:
> Hi,
>
> hardware:
> * Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz
> * 16 GB RAM
> * HDD ST500DM002-1BD142
>
> timings:
> Currently Java Broker 6.1.1 seems to behave as version 7.0.0 RC. 10 - 30
> messages per second. Interesting is when I increase message size to 10kB.
> Messages per second are same but throughput is increased ten times.
> When I use nonpersistent messages everything goes smooth. Thousand of 1kB
> messages are sent within 1 second.
>
> There are no extra JVM options, just the ones which are present in
> bin/qpid-server file.
>
> Heap and direct memory on broker is also default - -Xmx512m
> -XX:MaxDirectMemorySize=1536m. I tried to increase to four times larger ones
> -Xmx2048m -XX:MaxDirectMemorySize=6000m, but there was no change in messages
> per second.
>
> Unfortunately vmstat gives same values pro CPU, I am sending at least top
> output.
>
> 6.1.1:
> %Cpu(s):  6.9 us,  0.3 sy,  0.0 ni, 68.5 id, 24.1 wa,  0.0 hi,  0.2 si,  0.0
> st
>
> 7.0.0:
> %Cpu(s):  2.4 us,  0.4 sy,  0.0 ni, 71.2 id, 25.9 wa,  0.0 hi,  0.0 si,  0.0
> st
>
> When we tried on server where message store was stored on SAN disk, sending
> of messages increased to 500 msg/sec. With C++ broker on same machine we are
> able to send 5000 msg/sec.
>

Was that last comparison made using the code you gave for the C++
client against both Java and C++ brokers, or the JMS client against
both, or some other mixture?

While they might perform similarly in some cases the two clients act
very differently and so wont in others, so the precise combination is
important for comparison.

> ps. I cannot create queue in 7.0.0 version by webgui when queue contains '.'
> character, in 6.1.1 version queue with dot in name can be created by webgui
>
>
> Keith Wall wrote
>> Hi Tomas,
>>
>> Nor can I reproduce any discernible difference in performance between
>> 6.1.1 and the 7.0.0 RC with your Java code.  I have not tried the C++
>> yet.
>>
>> Can you share with us:
>>
>> * details of the hardware (including the storage) you are using for the
>> test.
>> * the timings you seeing for your tests for both the 6.1.1 case and 7.0.0
>> RC
>> * any extra JVM options you are passing either client or broker side.
>> * size of java heap (client side) and heap and direct memory (broker)
>>
>> Can I suggest that you collect vmstat type information for both runs
>> and compare?   My expectation is that CPU usage, disk I/O, and network
>> utilisation should be approximately equal between the two runs.
>>
>> cheers, Keith
>
>
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

[Vavricka]

It is ok for us to release 7.0.0 and implement this issue in 7.0.1.

Vast majority of our applications need this functionality.

Tomas


Keith Wall wrote
> The test is sending persistent messages so the broker is obliged to
> write them to disk.  After the arrival of each message transfer, the
> Broker-J awaits the sync'd to disk (after the write) before sending
> the Disposition performative back to the client.  The Qpid JMS Client
> is awaiting the Disposition, so it is only then that the
> MessageProducer#send returns and the application can send the next
> message.   I infer that CPP Broker must be optimistically sending the
> Disposition back to the client before the data is sync'd, so that is
> why you see better performance (but with a lesser guarantee).   If you
> were to switch the Qpid JMS Client to jms.forceAsyncSend[1], I would
> expect you would see greater performance.   Let me ask a higher level
> question - what messaging guarantee does this application require?
> 
> [1] https://qpid.apache.org/releases/qpid-jms-0.27.0/docs/index.html
> 
> On 10 November 2017 at 15:50, Rob Godfrey 

> rob.j.godfrey@

>  wrote:
>> On 10 November 2017 at 16:39, Vavricka 

> vavricka.tomas@

>  wrote:
>>
>>> Hi,
>>>
>>> hardware:
>>> * Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz
>>> * 16 GB RAM
>>> * HDD ST500DM002-1BD142
>>>
>>> timings:
>>> Currently Java Broker 6.1.1 seems to behave as version 7.0.0 RC. 10 - 30
>>> messages per second. Interesting is when I increase message size to
>>> 10kB.
>>> Messages per second are same but throughput is increased ten times.
>>> When I use nonpersistent messages everything goes smooth. Thousand of
>>> 1kB
>>> messages are sent within 1 second.
>>>
>>
>> So, this is more what I would expect (in the sense that 6.1 will behave
>> like 7.0 - the performance is unacceptable, but I think I understand it).
>>
>> I *think* the issue is that we have not yet implemented the optimisations
>> in the 1.0 layer for non-transactional durable messages to be processed
>> asynchronously.  Because of this the rate of message processing is
>> dependent upon how many times fsync() can be called a second.  500/s is
>> probably about right for a SAN, ~20 is what I saw from conventional hard
>> drives; for SSDs with a battery backed write cache I've gotten > 1000/s.
>> Because it is dependent upon the number of fsyncs rather than the
>> throughput of the disk, increasing the message size will not affect the
>> rate and thus you will see a linear improvement in throughput.
>>
>> Fixing this shouldn't actually be a huge change, and after it you should
>> see something more like the C++ broker behaviour.  Since this isn't (I
>> think) a regression between 7.0 and 6.1 I'd suggest that we progress with
>> the 7.0 release and then quickly follow that with a 7.0.1 that introduces
>> the necessary optimisation (we can essentially copy over the code from
>> the
>> 0-x protocol layers).
>>
>> -- Rob
>>
>>
>>>
>>> There are no extra JVM options, just the ones which are present in
>>> bin/qpid-server file.
>>>
>>> Heap and direct memory on broker is also default - -Xmx512m
>>> -XX:MaxDirectMemorySize=1536m. I tried to increase to four times larger
>>> ones
>>> -Xmx2048m -XX:MaxDirectMemorySize=6000m, but there was no change in
>>> messages
>>> per second.
>>>
>>> Unfortunately vmstat gives same values pro CPU, I am sending at least
>>> top
>>> output.
>>>
>>> 6.1.1:
>>> %Cpu(s):  6.9 us,  0.3 sy,  0.0 ni, 68.5 id, 24.1 wa,  0.0 hi,  0.2 si,
>>> 0.0
>>> st
>>>
>>> 7.0.0:
>>> %Cpu(s):  2.4 us,  0.4 sy,  0.0 ni, 71.2 id, 25.9 wa,  0.0 hi,  0.0 si,
>>> 0.0
>>> st
>>>
>>> When we tried on server where message store was stored on SAN disk,
>>> sending
>>> of messages increased to 500 msg/sec. With C++ broker on same machine we
>>> are
>>> able to send 5000 msg/sec.
>>>
>>> ps. I cannot create queue in 7.0.0 version by webgui when queue contains
>>> '.'
>>> character, in 6.1.1 version queue with dot in name can be created by
>>> webgui
>>>
>>>
>>> Keith Wall wrote
>>> > Hi Tomas,
>>> >
>>> > Nor can I reproduce any discernible difference in performance between
>>> > 6.1.1 and the 7.0.0 RC with your Java code.  I have not tried the C++
>>> > yet.
>>> >
>>> > Can you share with us:
>>> >
>>> > * details of the hardware (including the storage) you are using for
>>> the
>>> > test.
>>> > * the timings you seeing for your tests for both the 6.1.1 case and
>>> 7.0.0
>>> > RC
>>> > * any extra JVM options you are passing either client or broker side.
>>> > * size of java heap (client side) and heap and direct memory (broker)
>>> >
>>> > Can I suggest that you collect vmstat type information for both runs
>>> > and compare?   My expectation is that CPU usage, disk I/O, and network
>>> > utilisation should be approximately equal between the two runs.
>>> >
>>> > cheers, Keith
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-
>>> f2158936.html
>>>
>>> 

Re: Java Broker performance

[Keith W]

The test is sending persistent messages so the broker is obliged to
write them to disk.  After the arrival of each message transfer, the
Broker-J awaits the sync'd to disk (after the write) before sending
the Disposition performative back to the client.  The Qpid JMS Client
is awaiting the Disposition, so it is only then that the
MessageProducer#send returns and the application can send the next
message.   I infer that CPP Broker must be optimistically sending the
Disposition back to the client before the data is sync'd, so that is
why you see better performance (but with a lesser guarantee).   If you
were to switch the Qpid JMS Client to jms.forceAsyncSend[1], I would
expect you would see greater performance.   Let me ask a higher level
question - what messaging guarantee does this application require?

[1] https://qpid.apache.org/releases/qpid-jms-0.27.0/docs/index.html

On 10 November 2017 at 15:50, Rob Godfrey  wrote:
> On 10 November 2017 at 16:39, Vavricka  wrote:
>
>> Hi,
>>
>> hardware:
>> * Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz
>> * 16 GB RAM
>> * HDD ST500DM002-1BD142
>>
>> timings:
>> Currently Java Broker 6.1.1 seems to behave as version 7.0.0 RC. 10 - 30
>> messages per second. Interesting is when I increase message size to 10kB.
>> Messages per second are same but throughput is increased ten times.
>> When I use nonpersistent messages everything goes smooth. Thousand of 1kB
>> messages are sent within 1 second.
>>
>
> So, this is more what I would expect (in the sense that 6.1 will behave
> like 7.0 - the performance is unacceptable, but I think I understand it).
>
> I *think* the issue is that we have not yet implemented the optimisations
> in the 1.0 layer for non-transactional durable messages to be processed
> asynchronously.  Because of this the rate of message processing is
> dependent upon how many times fsync() can be called a second.  500/s is
> probably about right for a SAN, ~20 is what I saw from conventional hard
> drives; for SSDs with a battery backed write cache I've gotten > 1000/s.
> Because it is dependent upon the number of fsyncs rather than the
> throughput of the disk, increasing the message size will not affect the
> rate and thus you will see a linear improvement in throughput.
>
> Fixing this shouldn't actually be a huge change, and after it you should
> see something more like the C++ broker behaviour.  Since this isn't (I
> think) a regression between 7.0 and 6.1 I'd suggest that we progress with
> the 7.0 release and then quickly follow that with a 7.0.1 that introduces
> the necessary optimisation (we can essentially copy over the code from the
> 0-x protocol layers).
>
> -- Rob
>
>
>>
>> There are no extra JVM options, just the ones which are present in
>> bin/qpid-server file.
>>
>> Heap and direct memory on broker is also default - -Xmx512m
>> -XX:MaxDirectMemorySize=1536m. I tried to increase to four times larger
>> ones
>> -Xmx2048m -XX:MaxDirectMemorySize=6000m, but there was no change in
>> messages
>> per second.
>>
>> Unfortunately vmstat gives same values pro CPU, I am sending at least top
>> output.
>>
>> 6.1.1:
>> %Cpu(s):  6.9 us,  0.3 sy,  0.0 ni, 68.5 id, 24.1 wa,  0.0 hi,  0.2 si,
>> 0.0
>> st
>>
>> 7.0.0:
>> %Cpu(s):  2.4 us,  0.4 sy,  0.0 ni, 71.2 id, 25.9 wa,  0.0 hi,  0.0 si,
>> 0.0
>> st
>>
>> When we tried on server where message store was stored on SAN disk, sending
>> of messages increased to 500 msg/sec. With C++ broker on same machine we
>> are
>> able to send 5000 msg/sec.
>>
>> ps. I cannot create queue in 7.0.0 version by webgui when queue contains
>> '.'
>> character, in 6.1.1 version queue with dot in name can be created by webgui
>>
>>
>> Keith Wall wrote
>> > Hi Tomas,
>> >
>> > Nor can I reproduce any discernible difference in performance between
>> > 6.1.1 and the 7.0.0 RC with your Java code.  I have not tried the C++
>> > yet.
>> >
>> > Can you share with us:
>> >
>> > * details of the hardware (including the storage) you are using for the
>> > test.
>> > * the timings you seeing for your tests for both the 6.1.1 case and 7.0.0
>> > RC
>> > * any extra JVM options you are passing either client or broker side.
>> > * size of java heap (client side) and heap and direct memory (broker)
>> >
>> > Can I suggest that you collect vmstat type information for both runs
>> > and compare?   My expectation is that CPU usage, disk I/O, and network
>> > utilisation should be approximately equal between the two runs.
>> >
>> > cheers, Keith
>>
>>
>>
>>
>>
>> --
>> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-
>> f2158936.html
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
>> For additional commands, e-mail: users-h...@qpid.apache.org
>>
>>

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional 

Re: Java Broker performance

[Rob Godfrey]

On 10 November 2017 at 16:39, Vavricka  wrote:

> Hi,
>
> hardware:
> * Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz
> * 16 GB RAM
> * HDD ST500DM002-1BD142
>
> timings:
> Currently Java Broker 6.1.1 seems to behave as version 7.0.0 RC. 10 - 30
> messages per second. Interesting is when I increase message size to 10kB.
> Messages per second are same but throughput is increased ten times.
> When I use nonpersistent messages everything goes smooth. Thousand of 1kB
> messages are sent within 1 second.
>

So, this is more what I would expect (in the sense that 6.1 will behave
like 7.0 - the performance is unacceptable, but I think I understand it).

I *think* the issue is that we have not yet implemented the optimisations
in the 1.0 layer for non-transactional durable messages to be processed
asynchronously.  Because of this the rate of message processing is
dependent upon how many times fsync() can be called a second.  500/s is
probably about right for a SAN, ~20 is what I saw from conventional hard
drives; for SSDs with a battery backed write cache I've gotten > 1000/s.
Because it is dependent upon the number of fsyncs rather than the
throughput of the disk, increasing the message size will not affect the
rate and thus you will see a linear improvement in throughput.

Fixing this shouldn't actually be a huge change, and after it you should
see something more like the C++ broker behaviour.  Since this isn't (I
think) a regression between 7.0 and 6.1 I'd suggest that we progress with
the 7.0 release and then quickly follow that with a 7.0.1 that introduces
the necessary optimisation (we can essentially copy over the code from the
0-x protocol layers).

-- Rob


>
> There are no extra JVM options, just the ones which are present in
> bin/qpid-server file.
>
> Heap and direct memory on broker is also default - -Xmx512m
> -XX:MaxDirectMemorySize=1536m. I tried to increase to four times larger
> ones
> -Xmx2048m -XX:MaxDirectMemorySize=6000m, but there was no change in
> messages
> per second.
>
> Unfortunately vmstat gives same values pro CPU, I am sending at least top
> output.
>
> 6.1.1:
> %Cpu(s):  6.9 us,  0.3 sy,  0.0 ni, 68.5 id, 24.1 wa,  0.0 hi,  0.2 si,
> 0.0
> st
>
> 7.0.0:
> %Cpu(s):  2.4 us,  0.4 sy,  0.0 ni, 71.2 id, 25.9 wa,  0.0 hi,  0.0 si,
> 0.0
> st
>
> When we tried on server where message store was stored on SAN disk, sending
> of messages increased to 500 msg/sec. With C++ broker on same machine we
> are
> able to send 5000 msg/sec.
>
> ps. I cannot create queue in 7.0.0 version by webgui when queue contains
> '.'
> character, in 6.1.1 version queue with dot in name can be created by webgui
>
>
> Keith Wall wrote
> > Hi Tomas,
> >
> > Nor can I reproduce any discernible difference in performance between
> > 6.1.1 and the 7.0.0 RC with your Java code.  I have not tried the C++
> > yet.
> >
> > Can you share with us:
> >
> > * details of the hardware (including the storage) you are using for the
> > test.
> > * the timings you seeing for your tests for both the 6.1.1 case and 7.0.0
> > RC
> > * any extra JVM options you are passing either client or broker side.
> > * size of java heap (client side) and heap and direct memory (broker)
> >
> > Can I suggest that you collect vmstat type information for both runs
> > and compare?   My expectation is that CPU usage, disk I/O, and network
> > utilisation should be approximately equal between the two runs.
> >
> > cheers, Keith
>
>
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-
> f2158936.html
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>
>

Re: Java Broker performance

[Vavricka]

Hi,

hardware:
* Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz
* 16 GB RAM
* HDD ST500DM002-1BD142

timings:
Currently Java Broker 6.1.1 seems to behave as version 7.0.0 RC. 10 - 30
messages per second. Interesting is when I increase message size to 10kB.
Messages per second are same but throughput is increased ten times.
When I use nonpersistent messages everything goes smooth. Thousand of 1kB
messages are sent within 1 second.

There are no extra JVM options, just the ones which are present in
bin/qpid-server file.

Heap and direct memory on broker is also default - -Xmx512m
-XX:MaxDirectMemorySize=1536m. I tried to increase to four times larger ones
-Xmx2048m -XX:MaxDirectMemorySize=6000m, but there was no change in messages
per second.

Unfortunately vmstat gives same values pro CPU, I am sending at least top
output.

6.1.1:
%Cpu(s):  6.9 us,  0.3 sy,  0.0 ni, 68.5 id, 24.1 wa,  0.0 hi,  0.2 si,  0.0
st

7.0.0:
%Cpu(s):  2.4 us,  0.4 sy,  0.0 ni, 71.2 id, 25.9 wa,  0.0 hi,  0.0 si,  0.0
st

When we tried on server where message store was stored on SAN disk, sending
of messages increased to 500 msg/sec. With C++ broker on same machine we are
able to send 5000 msg/sec.

ps. I cannot create queue in 7.0.0 version by webgui when queue contains '.'
character, in 6.1.1 version queue with dot in name can be created by webgui


Keith Wall wrote
> Hi Tomas,
> 
> Nor can I reproduce any discernible difference in performance between
> 6.1.1 and the 7.0.0 RC with your Java code.  I have not tried the C++
> yet.
> 
> Can you share with us:
> 
> * details of the hardware (including the storage) you are using for the
> test.
> * the timings you seeing for your tests for both the 6.1.1 case and 7.0.0
> RC
> * any extra JVM options you are passing either client or broker side.
> * size of java heap (client side) and heap and direct memory (broker)
> 
> Can I suggest that you collect vmstat type information for both runs
> and compare?   My expectation is that CPU usage, disk I/O, and network
> utilisation should be approximately equal between the two runs.
> 
> cheers, Keith





--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

[Keith W]

Hi Tomas,

I'm testing with out of the box configuration.   I have tried your
configuration and still can't reproduce a slow down. I don't know your
ACL rules, but I added some.  The result was the same.
I'm curious to hear the answers to the questions I posed earlier.
Hopefully that will give us a clue.

Thanks Keith.


On 10 November 2017 at 13:51, Keith W  wrote:
> Hi Tomas,
>
> Nor can I reproduce any discernible difference in performance between
> 6.1.1 and the 7.0.0 RC with your Java code.  I have not tried the C++
> yet.
>
> Can you share with us:
>
> * details of the hardware (including the storage) you are using for the test.
> * the timings you seeing for your tests for both the 6.1.1 case and 7.0.0 RC
> * any extra JVM options you are passing either client or broker side.
> * size of java heap (client side) and heap and direct memory (broker)
>
> Can I suggest that you collect vmstat type information for both runs
> and compare?   My expectation is that CPU usage, disk I/O, and network
> utilisation should be approximately equal between the two runs.
>
> cheers, Keith
>
>
> On 10 November 2017 at 11:12, Rob Godfrey  wrote:
>> Hi Tomas,
>>
>> on the producing side I cannot reproduce this difference on my laptop
>> (MacBook Pro, running OS X), and I'm unaware of any changes that were made
>> to the broker that would cause such a significant slowdown (I haven't
>> looked at consuming yet).
>>
>> I presume you are running these tests on the same hardware, with the same
>> JMS client version, and such...?
>>
>> -- Rob
>>
>> On 10 November 2017 at 10:11, Rob Godfrey  wrote:
>>
>>> Thanks Tomas,
>>>
>>> we'll look into this
>>>
>>> -- Rob
>>>
>>> On 10 November 2017 at 09:59, Vavricka  wrote:
>>>
 C++ client code below

 #include 
 #include 

 #include 
 #include 
 #include 
 #include 
 #include 
 #include 
 #include 
 #include 
 #include 

 class Broadcaster : public proton::messaging_handler
 {

 private:

 std::string _account;
 std::string _password;
 std::string _host;
 unsigned int _port;
 unsigned int _count;
 unsigned int _size;
 unsigned int _sent;
 unsigned int _confirmed;
 std::string _exchange;
 std::string _routingKey;
 proton::sender _sender;

 public:

 explicit Broadcaster(const std::string ,
  const std::string ,
  const std::string ,
  unsigned int port,
  const std::string ,
  const std::string ,
  unsigned int count,
  unsigned int size)
 : _account(account)
 , _password(password)
 , _host(host)
 , _port(port)
 , _count(count)
 , _size(size)
 , _sent(0)
 , _confirmed(0)
 , _exchange(exchange)
 , _routingKey(routingKey)
 {
 }

 void on_container_start(proton::container )
 {
 proton::connection_options connectionOptions;
 connectionOptions.sasl_allow_insecure_mechs(true);
 connectionOptions.sasl_allowed_mechs("PLAIN");
 c.client_connection_options(connectionOptions);

 std::string url = "amqp://" + _account + ":" + _password + "@" +
 _host + ":" + std::to_string(_port) + "/" + _exchange;

 _sender = c.open_sender(url);
 }

 void on_sendable(proton::sender )
 {
 while (s.credit() && _sent < _count)
 {
 proton::message msg;
 msg.id(_sent + 1);
 msg.subject(_routingKey);
 msg.body(std::string(_size, '*'));
 msg.durable(true);
 s.send(msg);
 _sent++;
 std::cout << "-I sent " << _sent << " of " << _count <<
 std::endl;
 }
 }

 void on_tracker_accept(proton::tracker )
 {
 _confirmed++;
 if (_confirmed == _count)
 {
 std::cout << "-I- All messages (" << _confirmed << ")
 confirmed"
 << std::endl;
 t.connection().close();
 }
 }

 void on_transport_close(proton::transport )
 {
 _sent = _confirmed;
 }

 void run()
 {
 try
 {
 proton::default_container(*this).run();
 }
 catch (const std::exception )
 {
 std::cerr << "-E- Caught exception: " << error.what() <<
 

Re: Java Broker performance

[Keith W]

Hi Tomas,

Nor can I reproduce any discernible difference in performance between
6.1.1 and the 7.0.0 RC with your Java code.  I have not tried the C++
yet.

Can you share with us:

* details of the hardware (including the storage) you are using for the test.
* the timings you seeing for your tests for both the 6.1.1 case and 7.0.0 RC
* any extra JVM options you are passing either client or broker side.
* size of java heap (client side) and heap and direct memory (broker)

Can I suggest that you collect vmstat type information for both runs
and compare?   My expectation is that CPU usage, disk I/O, and network
utilisation should be approximately equal between the two runs.

cheers, Keith


On 10 November 2017 at 11:12, Rob Godfrey  wrote:
> Hi Tomas,
>
> on the producing side I cannot reproduce this difference on my laptop
> (MacBook Pro, running OS X), and I'm unaware of any changes that were made
> to the broker that would cause such a significant slowdown (I haven't
> looked at consuming yet).
>
> I presume you are running these tests on the same hardware, with the same
> JMS client version, and such...?
>
> -- Rob
>
> On 10 November 2017 at 10:11, Rob Godfrey  wrote:
>
>> Thanks Tomas,
>>
>> we'll look into this
>>
>> -- Rob
>>
>> On 10 November 2017 at 09:59, Vavricka  wrote:
>>
>>> C++ client code below
>>>
>>> #include 
>>> #include 
>>>
>>> #include 
>>> #include 
>>> #include 
>>> #include 
>>> #include 
>>> #include 
>>> #include 
>>> #include 
>>> #include 
>>>
>>> class Broadcaster : public proton::messaging_handler
>>> {
>>>
>>> private:
>>>
>>> std::string _account;
>>> std::string _password;
>>> std::string _host;
>>> unsigned int _port;
>>> unsigned int _count;
>>> unsigned int _size;
>>> unsigned int _sent;
>>> unsigned int _confirmed;
>>> std::string _exchange;
>>> std::string _routingKey;
>>> proton::sender _sender;
>>>
>>> public:
>>>
>>> explicit Broadcaster(const std::string ,
>>>  const std::string ,
>>>  const std::string ,
>>>  unsigned int port,
>>>  const std::string ,
>>>  const std::string ,
>>>  unsigned int count,
>>>  unsigned int size)
>>> : _account(account)
>>> , _password(password)
>>> , _host(host)
>>> , _port(port)
>>> , _count(count)
>>> , _size(size)
>>> , _sent(0)
>>> , _confirmed(0)
>>> , _exchange(exchange)
>>> , _routingKey(routingKey)
>>> {
>>> }
>>>
>>> void on_container_start(proton::container )
>>> {
>>> proton::connection_options connectionOptions;
>>> connectionOptions.sasl_allow_insecure_mechs(true);
>>> connectionOptions.sasl_allowed_mechs("PLAIN");
>>> c.client_connection_options(connectionOptions);
>>>
>>> std::string url = "amqp://" + _account + ":" + _password + "@" +
>>> _host + ":" + std::to_string(_port) + "/" + _exchange;
>>>
>>> _sender = c.open_sender(url);
>>> }
>>>
>>> void on_sendable(proton::sender )
>>> {
>>> while (s.credit() && _sent < _count)
>>> {
>>> proton::message msg;
>>> msg.id(_sent + 1);
>>> msg.subject(_routingKey);
>>> msg.body(std::string(_size, '*'));
>>> msg.durable(true);
>>> s.send(msg);
>>> _sent++;
>>> std::cout << "-I sent " << _sent << " of " << _count <<
>>> std::endl;
>>> }
>>> }
>>>
>>> void on_tracker_accept(proton::tracker )
>>> {
>>> _confirmed++;
>>> if (_confirmed == _count)
>>> {
>>> std::cout << "-I- All messages (" << _confirmed << ")
>>> confirmed"
>>> << std::endl;
>>> t.connection().close();
>>> }
>>> }
>>>
>>> void on_transport_close(proton::transport )
>>> {
>>> _sent = _confirmed;
>>> }
>>>
>>> void run()
>>> {
>>> try
>>> {
>>> proton::default_container(*this).run();
>>> }
>>> catch (const std::exception )
>>> {
>>> std::cerr << "-E- Caught exception: " << error.what() <<
>>> std::endl;
>>> throw error;
>>> }
>>> }
>>> };
>>>
>>> int main(void)
>>> {
>>> Broadcaster("C7",
>>> "C7",
>>> "pc1wj611",
>>> 20001,
>>> "broadcast",
>>> "broadcast.C7_CashTransaction",
>>> 1000,
>>> 1024).run();
>>> return 0;
>>> }
>>>
>>>
>>>
>>>
>>> --
>>> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936
>>> .html
>>>
>>> -

Re: Java Broker performance

[Vavricka]

Yes, we running it on same hardware and versions.

Couldn't be there issue with our configuration? Can you spot there some
misconfiguration in config files below?

Will be helpful if you send us your broker configuration?

config.json

{
  "id" : "cfa9a57a-6f98-4f17-b81d-22fb5a032473",
  "name" : "cps",
  "modelVersion" : "7.0",
  "accesscontrolproviders" : [ {
"id" : "65b550b1-0a22-4815-9b6f-5287f6a9b455",
"name" : "brokerAcl",
"type" : "AclFile",
"path" : "/home/vavrtom/qpid/0406/broker/work/broker.acl",
"lastUpdatedBy" : null,
"lastUpdatedTime" : null,
"createdBy" : null,
"createdTime" : null
  } ],
  "authenticationproviders" : [ {
"id" : "df3680b4-8bd7-4bb8-92b7-705d33bf1496",
"name" : "SHA_256",
"type" : "SCRAM-SHA-256",
"secureOnlyMechanisms" : [ ],
"users" : [ {
  "id" : "e98445dd-4f79-46b1-bc7f-a153f546e63e",
  "name" : "C7",
  "type" : "managed",
  "durable" : true,
  "password" :
"KAkXZolpJXtHteNgxqeNaosC3eecIN0MRZVdFcAwGm4=,,U52qJCBi8jUfjPTHMxayTwUhQ3UT+N6B7h78S+kRNUw=,fQdDP2WsuAGZ/P4PxJuSAHrpmgE/2GBMjD9/TrYw0QY=,4096",
  "lastUpdatedBy" : "admin",
  "lastUpdatedTime" : 1510235279421,
  "createdBy" : "admin",
  "createdTime" : 1510235279421
}, {
  "id" : "62ab34ac-11d8-4ed0-adc3-bd1c34a2f7af",
  "name" : "admin",
  "type" : "managed",
  "password" :
"7uKre2FrpVvO+/gABKrTHyQS5tAY92UqOWNcWPCubyw=,,j3TzWP+Bg2iZUXxMdZbVi1ftrHIvzev5PqO9ftDC4hU=,9cvIWGYPfeUD0hFLvINPKz+Ddvv/9/yz8PuMC+JR3Ro=,4096"
}, {
  "id" : "e6451384-9ef5-4823-81c6-7b1aff7251ca",
  "name" : "monit",
  "type" : "managed",
  "password" :
"b3MiUviPyxYSfB9bxRZG4FJ/uxDWjWnNvBe1B/w8JYc=,,gxUMuoFhW9oVNV/tEiLm3IdgObL2uQagi63G+3bnFgs=,cQmi/8NKL2b8ESfoF/wcffPRILoMOx5cRy+rEG2TAxw=,4096"
} ]
  }, {
"id" : "85d7cb16-4c60-45ca-94e8-47c611a7b1e4",
"name" : "external",
"type" : "External"
  } ],
  "brokerloggers" : [ {
"id" : "e637b21b-c3a8-457f-a0ff-c4327c88135b",
"name" : "logfile",
"type" : "File",
"fileName" :
"${qpid.work_dir}${file.separator}..${file.separator}..${file.separator}log${file.separator}broker.log",
"brokerloginclusionrules" : [ {
  "id" : "f8e8c9eb-6ca9-4845-8ac8-1ba614a47ce8",
  "name" : "Operational",
  "type" : "NameAndLevel",
  "level" : "INFO",
  "loggerName" : "qpid.message.*"
}, {
  "id" : "13c4e2c4-3b3a-43be-890b-cc506dae63f8",
  "name" : "Qpid",
  "type" : "NameAndLevel",
  "level" : "INFO",
  "loggerName" : "org.apache.qpid.*"
}, {
  "id" : "00ff6321-2d42-4dce-aad0-12c55c928e0d",
  "name" : "Root",
  "type" : "NameAndLevel",
  "level" : "INFO",
  "loggerName" : "ROOT"
} ]
  }, {
"id" : "b25480a7-5ed1-4035-af9d-565c9c223708",
"name" : "memory",
"type" : "Memory",
"brokerloginclusionrules" : [ {
  "id" : "ffc9a1da-e812-476c-8ccf-c8ead2f15e1e",
  "name" : "Operational",
  "type" : "NameAndLevel",
  "level" : "INFO",
  "loggerName" : "qpid.message.*"
}, {
  "id" : "8c2a2fe6-9531-469b-8f26-c5d449e97f26",
  "name" : "Qpid",
  "type" : "NameAndLevel",
  "level" : "INFO",
  "loggerName" : "org.apache.qpid.*"
}, {
  "id" : "dcb193af-fba8-49e7-9ed4-9a4a08627e8c",
  "name" : "Root",
  "type" : "NameAndLevel",
  "level" : "INFO",
  "loggerName" : "ROOT"
} ]
  } ],
  "keystores" : [ {
"id" : "f262fc1c-081b-4215-8afb-bb0ce6131857",
"name" : "keyStore",
"type" : "FileKeyStore",
"certificateAlias" : "vavrtom-amqp",
"password" : "admin",
"storeUrl" : "/home/vavrtom/qpid/0406/broker/work/keystore.jks"
  } ],
  "plugins" : [ {
"id" : "1276ff6c-61ea-4b4d-b1ad-4edc5c02f581",
"name" : "httpManagement",
"type" : "MANAGEMENT-HTTP",
"httpBasicAuthenticationEnabled" : "true"
  } ],
  "ports" : [ {
"id" : "6fdbb8b1-76ba-4048-9ca8-4e7dfb884a38",
"name" : "amqp",
"type" : "AMQP",
"port" : 20406,
"protocols" : [ "AMQP_1_0" ],
"authenticationProvider" : "SHA_256",
"threadPoolSize" : 10,
"numberOfSelectors" : 1,
"maxOpenConnections" : 1000,
"virtualhostaliases" : [ {
  "id" : "59d083e1-57f2-45c0-b8d4-c75f23ea33ef",
  "name" : "defaultAlias",
  "type" : "defaultAlias"
}, {
  "id" : "1c61ebd0-fbc6-4891-998b-59b972492fc7",
  "name" : "hostnameAlias",
  "type" : "hostnameAlias"
}, {
  "id" : "d75851d6-ea9c-4aec-a8db-7e28d5815b5e",
  "name" : "nameAlias",
  "type" : "nameAlias"
} ]
  }, {
"id" : "f52c4129-d388-4f3a-b23f-892070a131dd",
"name" : "amqps",
"type" : "AMQP",
"port" : 10406,
"protocols" : [ "AMQP_1_0" ],
"authenticationProvider" : "external",
"needClientAuth" : true,
"wantClientAuth" : true,
"keyStore" : "keyStore",
"transports" : [ "SSL" ],
"trustStores" : [ "trustStore" ],
"threadPoolSize" : 10,
"numberOfSelectors" : 1,
"maxOpenConnections" : 

Re: Java Broker performance

[Rob Godfrey]

Hi Tomas,

on the producing side I cannot reproduce this difference on my laptop
(MacBook Pro, running OS X), and I'm unaware of any changes that were made
to the broker that would cause such a significant slowdown (I haven't
looked at consuming yet).

I presume you are running these tests on the same hardware, with the same
JMS client version, and such...?

-- Rob

On 10 November 2017 at 10:11, Rob Godfrey  wrote:

> Thanks Tomas,
>
> we'll look into this
>
> -- Rob
>
> On 10 November 2017 at 09:59, Vavricka  wrote:
>
>> C++ client code below
>>
>> #include 
>> #include 
>>
>> #include 
>> #include 
>> #include 
>> #include 
>> #include 
>> #include 
>> #include 
>> #include 
>> #include 
>>
>> class Broadcaster : public proton::messaging_handler
>> {
>>
>> private:
>>
>> std::string _account;
>> std::string _password;
>> std::string _host;
>> unsigned int _port;
>> unsigned int _count;
>> unsigned int _size;
>> unsigned int _sent;
>> unsigned int _confirmed;
>> std::string _exchange;
>> std::string _routingKey;
>> proton::sender _sender;
>>
>> public:
>>
>> explicit Broadcaster(const std::string ,
>>  const std::string ,
>>  const std::string ,
>>  unsigned int port,
>>  const std::string ,
>>  const std::string ,
>>  unsigned int count,
>>  unsigned int size)
>> : _account(account)
>> , _password(password)
>> , _host(host)
>> , _port(port)
>> , _count(count)
>> , _size(size)
>> , _sent(0)
>> , _confirmed(0)
>> , _exchange(exchange)
>> , _routingKey(routingKey)
>> {
>> }
>>
>> void on_container_start(proton::container )
>> {
>> proton::connection_options connectionOptions;
>> connectionOptions.sasl_allow_insecure_mechs(true);
>> connectionOptions.sasl_allowed_mechs("PLAIN");
>> c.client_connection_options(connectionOptions);
>>
>> std::string url = "amqp://" + _account + ":" + _password + "@" +
>> _host + ":" + std::to_string(_port) + "/" + _exchange;
>>
>> _sender = c.open_sender(url);
>> }
>>
>> void on_sendable(proton::sender )
>> {
>> while (s.credit() && _sent < _count)
>> {
>> proton::message msg;
>> msg.id(_sent + 1);
>> msg.subject(_routingKey);
>> msg.body(std::string(_size, '*'));
>> msg.durable(true);
>> s.send(msg);
>> _sent++;
>> std::cout << "-I sent " << _sent << " of " << _count <<
>> std::endl;
>> }
>> }
>>
>> void on_tracker_accept(proton::tracker )
>> {
>> _confirmed++;
>> if (_confirmed == _count)
>> {
>> std::cout << "-I- All messages (" << _confirmed << ")
>> confirmed"
>> << std::endl;
>> t.connection().close();
>> }
>> }
>>
>> void on_transport_close(proton::transport )
>> {
>> _sent = _confirmed;
>> }
>>
>> void run()
>> {
>> try
>> {
>> proton::default_container(*this).run();
>> }
>> catch (const std::exception )
>> {
>> std::cerr << "-E- Caught exception: " << error.what() <<
>> std::endl;
>> throw error;
>> }
>> }
>> };
>>
>> int main(void)
>> {
>> Broadcaster("C7",
>> "C7",
>> "pc1wj611",
>> 20001,
>> "broadcast",
>> "broadcast.C7_CashTransaction",
>> 1000,
>> 1024).run();
>> return 0;
>> }
>>
>>
>>
>>
>> --
>> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936
>> .html
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
>> For additional commands, e-mail: users-h...@qpid.apache.org
>>
>>
>

Re: Java Broker performance

[Rob Godfrey]

Thanks Tomas,

we'll look into this

-- Rob

On 10 November 2017 at 09:59, Vavricka  wrote:

> C++ client code below
>
> #include 
> #include 
>
> #include 
> #include 
> #include 
> #include 
> #include 
> #include 
> #include 
> #include 
> #include 
>
> class Broadcaster : public proton::messaging_handler
> {
>
> private:
>
> std::string _account;
> std::string _password;
> std::string _host;
> unsigned int _port;
> unsigned int _count;
> unsigned int _size;
> unsigned int _sent;
> unsigned int _confirmed;
> std::string _exchange;
> std::string _routingKey;
> proton::sender _sender;
>
> public:
>
> explicit Broadcaster(const std::string ,
>  const std::string ,
>  const std::string ,
>  unsigned int port,
>  const std::string ,
>  const std::string ,
>  unsigned int count,
>  unsigned int size)
> : _account(account)
> , _password(password)
> , _host(host)
> , _port(port)
> , _count(count)
> , _size(size)
> , _sent(0)
> , _confirmed(0)
> , _exchange(exchange)
> , _routingKey(routingKey)
> {
> }
>
> void on_container_start(proton::container )
> {
> proton::connection_options connectionOptions;
> connectionOptions.sasl_allow_insecure_mechs(true);
> connectionOptions.sasl_allowed_mechs("PLAIN");
> c.client_connection_options(connectionOptions);
>
> std::string url = "amqp://" + _account + ":" + _password + "@" +
> _host + ":" + std::to_string(_port) + "/" + _exchange;
>
> _sender = c.open_sender(url);
> }
>
> void on_sendable(proton::sender )
> {
> while (s.credit() && _sent < _count)
> {
> proton::message msg;
> msg.id(_sent + 1);
> msg.subject(_routingKey);
> msg.body(std::string(_size, '*'));
> msg.durable(true);
> s.send(msg);
> _sent++;
> std::cout << "-I sent " << _sent << " of " << _count <<
> std::endl;
> }
> }
>
> void on_tracker_accept(proton::tracker )
> {
> _confirmed++;
> if (_confirmed == _count)
> {
> std::cout << "-I- All messages (" << _confirmed << ")
> confirmed"
> << std::endl;
> t.connection().close();
> }
> }
>
> void on_transport_close(proton::transport )
> {
> _sent = _confirmed;
> }
>
> void run()
> {
> try
> {
> proton::default_container(*this).run();
> }
> catch (const std::exception )
> {
> std::cerr << "-E- Caught exception: " << error.what() <<
> std::endl;
> throw error;
> }
> }
> };
>
> int main(void)
> {
> Broadcaster("C7",
> "C7",
> "pc1wj611",
> 20001,
> "broadcast",
> "broadcast.C7_CashTransaction",
> 1000,
> 1024).run();
> return 0;
> }
>
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-
> f2158936.html
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>
>

Re: Java Broker performance

[Vavricka]

C++ client code below

#include 
#include 

#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 

class Broadcaster : public proton::messaging_handler
{

private:

std::string _account;
std::string _password;
std::string _host;
unsigned int _port;
unsigned int _count;
unsigned int _size;
unsigned int _sent;
unsigned int _confirmed;
std::string _exchange;
std::string _routingKey;
proton::sender _sender;

public:

explicit Broadcaster(const std::string ,
 const std::string ,
 const std::string ,
 unsigned int port,
 const std::string ,
 const std::string ,
 unsigned int count,
 unsigned int size)
: _account(account)
, _password(password)
, _host(host)
, _port(port)
, _count(count)
, _size(size)
, _sent(0)
, _confirmed(0)
, _exchange(exchange)
, _routingKey(routingKey)
{
}

void on_container_start(proton::container )
{
proton::connection_options connectionOptions;
connectionOptions.sasl_allow_insecure_mechs(true);
connectionOptions.sasl_allowed_mechs("PLAIN");
c.client_connection_options(connectionOptions);

std::string url = "amqp://" + _account + ":" + _password + "@" +
_host + ":" + std::to_string(_port) + "/" + _exchange;

_sender = c.open_sender(url);
}

void on_sendable(proton::sender )
{
while (s.credit() && _sent < _count)
{
proton::message msg;
msg.id(_sent + 1);
msg.subject(_routingKey);
msg.body(std::string(_size, '*'));
msg.durable(true);
s.send(msg);
_sent++;
std::cout << "-I sent " << _sent << " of " << _count <<
std::endl;
}
}

void on_tracker_accept(proton::tracker )
{
_confirmed++;
if (_confirmed == _count)
{
std::cout << "-I- All messages (" << _confirmed << ") confirmed"
<< std::endl;
t.connection().close();
}
}

void on_transport_close(proton::transport )
{
_sent = _confirmed;
}

void run()
{
try
{
proton::default_container(*this).run();
}
catch (const std::exception )
{
std::cerr << "-E- Caught exception: " << error.what() <<
std::endl;
throw error;
}
}
};

int main(void)
{
Broadcaster("C7",
"C7",
"pc1wj611",
20001,
"broadcast",
"broadcast.C7_CashTransaction",
1000,
1024).run();
return 0;
}




--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

[Vavricka]

Hi, 

when consumer is not attached sending of messages is slow too. We were
running performance tests for Java Broker 6.1.1 before 6 months and there
were no issues (throughput for 1K messages was around 10MB/s).


rgodfrey wrote
> Hi Tomas, 
> 
> are you saying that there is a significant degradation in performance 
> between 7.0.0 and prior versions for this scenario, or do you see the same 
> performance in 6.1.x? 
> 
> In terms of consuming - is the performance slow if you are consuming from
> a 
> queue with 1000 messages already in it, or is the slowness just caused by 
> the slow producer? 
> 
> Thanks, 
> Rob

Code below is for JMS client 0.27.0 (without catching exceptions) which
behaves similarly as client based on proton 0.18.1.

Properties properties = new Properties();
properties.setProperty(Context.INITIAL_CONTEXT_FACTORY,
"org.apache.qpid.jms.jndi.JmsInitialContextFactory");
properties.setProperty("connectionfactory.connection",
"amqp://localhost:20406?jms.username=C7=C7");
properties.setProperty("topic.broadcastAddress", "broadcast");
InitialContext context = new InitialContext(properties);

Connection connection = ((ConnectionFactory)
context.lookup("connection")).createConnection();
Session session = connection.createSession(false,
Session.CLIENT_ACKNOWLEDGE);
MessageProducer broadcastProducer = session.createProducer((Destination)
context.lookup("broadcastAddress"));
connection.start();
char[] chars = new char[1024];
Arrays.fill(chars, 'f');
TextMessage message = session.createTextMessage(new String(chars));
message.setJMSDeliveryMode(DeliveryMode.PERSISTENT);
message.setJMSType("broadcast.C7_CashTransaction");
System.out.println(Instant.now());
int messageCount = 1000;
for (int i = 1 ; i <= messageCount ; i++)
{
broadcastProducer.send(message, DeliveryMode.PERSISTENT,
Message.DEFAULT_PRIORITY, Message.DEFAULT_TIME_TO_LIVE);
if (i % 100 == 0)
{
System.out.println(Instant.now() + " sent " + i + " messages");
}
}
System.out.println(Instant.now());

Output of client is below

[main] INFO com.deutscheboerse.amqp_1_0.examples.BroadcastSender - Creating
connection
[AmqpProvider :(1):[amqp://localhost:20406]] INFO
org.apache.qpid.jms.sasl.SaslMechanismFinder - Best match for SASL auth was:
SASL-SCRAM-SHA-256
[AmqpProvider :(1):[amqp://localhost:20406]] INFO
org.apache.qpid.jms.JmsConnection - Connection
ID:0333e24e-c00a-4dd7-91af-57fd1366512c:1 connected to remote Broker:
amqp://localhost:20406
2017-11-10T08:46:31.135Z
2017-11-10T08:46:41.770Z sent 100 messages
2017-11-10T08:46:52.449Z sent 200 messages
2017-11-10T08:47:03.958Z sent 300 messages
2017-11-10T08:47:15.174Z sent 400 messages
2017-11-10T08:47:25.514Z sent 500 messages
2017-11-10T08:47:36.045Z sent 600 messages
2017-11-10T08:47:46.432Z sent 700 messages
2017-11-10T08:47:57.797Z sent 800 messages
2017-11-10T08:48:09.434Z sent 900 messages
2017-11-10T08:48:20.419Z sent 1000 messages
2017-11-10T08:48:20.419Z


Keith Wall wrote
> Hi Tomas,
> 
> That does sound surprising.  These performance tests[1] are run, with
> BDB and using the latest Qpid JMS Client,  against both the Broker-J
> 7.0.0 and 6.1.4 code lines each day, which is consistently showing
> v7.0.0 is faster for the five use-cases exercised by the tests (see
> perftests/etc/testdefs/defaultTests.js).  I am interested to hear
> about your use-case.   Do you have code you can share?
> 
> cheers, Keith.





--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

[Keith W]

Hi Tomas,

That does sound surprising.  These performance tests[1] are run, with
BDB and using the latest Qpid JMS Client,  against both the Broker-J
7.0.0 and 6.1.4 code lines each day, which is consistently showing
v7.0.0 is faster for the five use-cases exercised by the tests (see
perftests/etc/testdefs/defaultTests.js).  I am interested to hear
about your use-case.   Do you have code you can share?

cheers, Keith.

[1] https://cwiki.apache.org/confluence/display/qpid/Running+Performance+Tests

On 9 November 2017 at 15:21, Rob Godfrey  wrote:
> Hi Tomas,
>
> are you saying that there is a significant degradation in performance
> between 7.0.0 and prior versions for this scenario, or do you see the same
> performance in 6.1.x?
>
> In terms of consuming - is the performance slow if you are consuming from a
> queue with 1000 messages already in it, or is the slowness just caused by
> the slow producer?
>
> Thanks,
> Rob
>
>
> On 9 November 2017 at 07:10, Vavricka  wrote:
>
>> Hi,
>>
>> we are testing release candidate of Java Broker 7.0.0.
>>
>> I checkout tag 7.0.0, build broker with tests without any problem.
>>
>> We have sender and receiver based on proton 0.18.1.
>> Both sender and receiver are started at same time.
>> Sender is sending 1000 persistent messages to exchange and receiver expects
>> 1000 messages in queue. Messages have size 1 kB.
>> We tried DERBY and BDB as persistent storage.
>>
>> Sending and consuming of durable messages is slow (10 messages per second).
>> When sending non-persistent messages then broker behaves as expected.
>>
>> Is there some settings that can optimize persistent storage?
>>
>> Tomas
>>
>>
>>
>> --
>> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-
>> f2158936.html
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
>> For additional commands, e-mail: users-h...@qpid.apache.org
>>
>>

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Broker performance

[Rob Godfrey]

Hi Tomas,

are you saying that there is a significant degradation in performance
between 7.0.0 and prior versions for this scenario, or do you see the same
performance in 6.1.x?

In terms of consuming - is the performance slow if you are consuming from a
queue with 1000 messages already in it, or is the slowness just caused by
the slow producer?

Thanks,
Rob


On 9 November 2017 at 07:10, Vavricka  wrote:

> Hi,
>
> we are testing release candidate of Java Broker 7.0.0.
>
> I checkout tag 7.0.0, build broker with tests without any problem.
>
> We have sender and receiver based on proton 0.18.1.
> Both sender and receiver are started at same time.
> Sender is sending 1000 persistent messages to exchange and receiver expects
> 1000 messages in queue. Messages have size 1 kB.
> We tried DERBY and BDB as persistent storage.
>
> Sending and consuming of durable messages is slow (10 messages per second).
> When sending non-persistent messages then broker behaves as expected.
>
> Is there some settings that can optimize persistent storage?
>
> Tomas
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-
> f2158936.html
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>
>

Java Broker performance

[Vavricka]

Hi, 

we are testing release candidate of Java Broker 7.0.0.

I checkout tag 7.0.0, build broker with tests without any problem.

We have sender and receiver based on proton 0.18.1.
Both sender and receiver are started at same time.
Sender is sending 1000 persistent messages to exchange and receiver expects
1000 messages in queue. Messages have size 1 kB.
We tried DERBY and BDB as persistent storage.

Sending and consuming of durable messages is slow (10 messages per second).
When sending non-persistent messages then broker behaves as expected.

Is there some settings that can optimize persistent storage?

Tomas



--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Qpid Java Broker performance tests using AMQP 1.0

[Michal Zerola]

At the end I have managed to make it run using AMQP 1.0 protocol against both
Qpid Java broker and Qpid C++ broker.

With Qpid Java broker I had to switch to REST Queue creator and also specify
management parameters. The final command then looks like:



For the C++ broker I have created queues by hand and used "NoOp" Queue
creator. The final command:





--
View this message in context: 
http://qpid.2158936.n2.nabble.com/Qpid-Java-Broker-performance-tests-using-AMQP-1-0-tp7645180p7645383.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Qpid Java Broker performance tests using AMQP 1.0

[Michal Zerola]

Hi,
I want to use performance tests which are part of the Qpid Java Broker to
measure the throughput of several brokers. I was able to configure and run
it using AMQP 0-10 protocol. However, I am struggling to make it use the
AMQP 1.0 client.

What I did:
- specified jndi-config=perftests-jndi-qpid-jms-client.properties to be used
(with 1.0 url naming)
- provided 0.9.0 qpid JMS client on class path
- specified property
java.naming.factory.initial=org.apache.qpid.jms.jndi.JmsInitialContextFactory

It doesn't seem to be enough. Is there some manual or could someone provide
a quick list of thing one needs to do in order to execute performance tests
with 1.0 client?

Thanks,

Michal



--
View this message in context: 
http://qpid.2158936.n2.nabble.com/Qpid-Java-Broker-performance-tests-using-AMQP-1-0-tp7645180.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Qpid Java Broker performance lower than expected

[vipun]

Hi Robbie,
  Thanks for the reply. We also didn't see much performance difference
between broker instances based on Derby store and berkeley db store. However
readings we obtained now are quite ok, here's the summary of readings we
have with Derby persistence:
We have also tinkered a little bit with the program Qpid Bench to allow for
multiple producers and consumers, and here's our reading.

producers   consumers   Msg_size   producers_rateconsumers_rate
threads  threads
1  1   1k921.74   877.73

1010  1k4357.87 7132.64

5050  1k5511.71 12670.5

4070  1k5753.82 24654.24

5060  1k6319.43 21525.96

Regards
Vinay

--
View this message in context: 
http://apache-qpid-users.2158936.n2.nabble.com/Qpid-Java-Broker-performance-lower-than-expected-tp6925405p6938973.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.

-
Apache Qpid - AMQP Messaging Implementation
Project:  http://qpid.apache.org
Use/Interact: mailto:users-subscr...@qpid.apache.org

Re: Qpid Java Broker performance lower than expected

[vipun]

The parameters we had used with the original QpidBench program is as follows:

-c 1 -i 1000 -s 1024 -m both --timestamp false --message-id false
--message-cache true --persistent true --jms true

--
View this message in context: 
http://apache-qpid-users.2158936.n2.nabble.com/Qpid-Java-Broker-performance-lower-than-expected-tp6925405p6939008.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.

-
Apache Qpid - AMQP Messaging Implementation
Project:  http://qpid.apache.org
Use/Interact: mailto:users-subscr...@qpid.apache.org

Re: Qpid Java Broker performance lower than expected

[Robbie Gemmell]

The reason I said that was that it takes a *lot* longer to run
some(/all) of the system tests when using the DerbyStore, but doing
some very noddy tests today with a single consumer and producer showed
there wasnt any great difference between them. Both were noticably
slower than historically, so it is something we will be looking into
improving. One particular system test I previously noticed a large
difference in uses multiple consumers and producers though, and so
that could actually be where the difference lies because the BDB store
is implemented somewhat differently to the Derby one and so possibly
has an artificial edge in that regard.

Robbie

On 26 October 2011 03:14, Danushka Menikkumbura
danushka.menikkumb...@gmail.com wrote:
 Hi Robbie,

 I did not notice that the BDB store was faster than the Derby store when I
 checked some time back.

 Thanks,
 Danushka

 On Wed, Oct 26, 2011 at 5:07 AM, Robbie Gemmell 
 robbie.gemm...@gmail.comwrote:

 Hi Vinay,

 I havent done any performance benchmarking of the Derby store to know
 what a representative number would actually be, but I will try to take
 a look at some point. I havent actually used QpidBench, so can I ask
 if there were any specific command(s) you ran so I can try the same
 scenarios?

 We havent paid much attention to performance of the Java broker for a
 while unfortunately because we have been working on various other
 issues such agetting memory usage under control and sorting out
 correctness issues etc since adding a newer protocol version and doing
 some significant refactorings and reimplementations, but as we reach
 the light at the end of the tunnel on those it is something which
 should move further up the priority list.

 It is worth nothing that there is also a BDB persistent store for the
 Java broker that you might want to look at, as I would expect it to be
 faster. It has recently been moved into the main repo, but is still an
 optional module which you need to explicitly ask for to be built
 (because BDB itself uses the Sleepycat Licence, which invokes
 restrictions upon distribution that mean it is not Apache Licence
 compatible). You can build the store module and include it (but not
 BDB itself) in the broker binary release bundle by using the following
 build command:

 ant build release-bin -Dmodules.opt=bdbstore -Ddownload-bdb=true

 You will find that downloads the bdb je jar into
 qpid/java/lib/bdbstore, and then creates a broker binary release in
 qpid/java/broker/release which includes the additional store module.
 You can make the BDB je jar available to the broker by creating a
 lib\opt subdir and copying the je jar into it, where it will get
 picked up automatically assuming you are using Java 6+. You can then
 use org.apache.qpid.server.store.berkeleydb.BDBMessageStore as the
 store class config instead of the other stores.

 Robbie

 On 24 October 2011 16:25, vipun vinay.punno...@gmail.com wrote:
  Hi,
   I'm collecting performance figures for QPID Java based broker. The
 results
  which i got after running the  QpidBench program are a little lower than
  expected. My machine which is a quad core, 8GB RAM with Windows 7 gives a
  message throughput of around 400 messages when both producer and consumer
  client instances are active.
 
  Qpid Java broker is configured to run over Derby and messaging is in
  persistent mode.  I was expecting somewhere around 1000 atleast going by
 the
  following blog which does comparisons between different messaging
 providers.
 
  http://bhavin.directi.com/rabbitmq-vs-apache-activemq-vs-apache-qpid/
 
  Do you think, the figures from my tests are correct, or what are the
  expected performance results, or are there any tweaks which need to be
 done
  for performance gains. I am running out of trunk.
 
  Thanks  Regards
  Vinay
 
  --
  View this message in context:
 http://apache-qpid-users.2158936.n2.nabble.com/Qpid-Java-Broker-performance-lower-than-expected-tp6925405p6925405.html
  Sent from the Apache Qpid users mailing list archive at Nabble.com.
 
  -
  Apache Qpid - AMQP Messaging Implementation
  Project:      http://qpid.apache.org
  Use/Interact: mailto:users-subscr...@qpid.apache.org
 
 

 -
 Apache Qpid - AMQP Messaging Implementation
 Project:      http://qpid.apache.org
 Use/Interact: mailto:users-subscr...@qpid.apache.org




-
Apache Qpid - AMQP Messaging Implementation
Project:  http://qpid.apache.org
Use/Interact: mailto:users-subscr...@qpid.apache.org

Re: Qpid Java Broker performance lower than expected

[Robbie Gemmell]

Hi Vinay,

I havent done any performance benchmarking of the Derby store to know
what a representative number would actually be, but I will try to take
a look at some point. I havent actually used QpidBench, so can I ask
if there were any specific command(s) you ran so I can try the same
scenarios?

We havent paid much attention to performance of the Java broker for a
while unfortunately because we have been working on various other
issues such agetting memory usage under control and sorting out
correctness issues etc since adding a newer protocol version and doing
some significant refactorings and reimplementations, but as we reach
the light at the end of the tunnel on those it is something which
should move further up the priority list.

It is worth nothing that there is also a BDB persistent store for the
Java broker that you might want to look at, as I would expect it to be
faster. It has recently been moved into the main repo, but is still an
optional module which you need to explicitly ask for to be built
(because BDB itself uses the Sleepycat Licence, which invokes
restrictions upon distribution that mean it is not Apache Licence
compatible). You can build the store module and include it (but not
BDB itself) in the broker binary release bundle by using the following
build command:

ant build release-bin -Dmodules.opt=bdbstore -Ddownload-bdb=true

You will find that downloads the bdb je jar into
qpid/java/lib/bdbstore, and then creates a broker binary release in
qpid/java/broker/release which includes the additional store module.
You can make the BDB je jar available to the broker by creating a
lib\opt subdir and copying the je jar into it, where it will get
picked up automatically assuming you are using Java 6+. You can then
use org.apache.qpid.server.store.berkeleydb.BDBMessageStore as the
store class config instead of the other stores.

Robbie

On 24 October 2011 16:25, vipun vinay.punno...@gmail.com wrote:
 Hi,
  I'm collecting performance figures for QPID Java based broker. The results
 which i got after running the  QpidBench program are a little lower than
 expected. My machine which is a quad core, 8GB RAM with Windows 7 gives a
 message throughput of around 400 messages when both producer and consumer
 client instances are active.

 Qpid Java broker is configured to run over Derby and messaging is in
 persistent mode.  I was expecting somewhere around 1000 atleast going by the
 following blog which does comparisons between different messaging providers.

 http://bhavin.directi.com/rabbitmq-vs-apache-activemq-vs-apache-qpid/

 Do you think, the figures from my tests are correct, or what are the
 expected performance results, or are there any tweaks which need to be done
 for performance gains. I am running out of trunk.

 Thanks  Regards
 Vinay

 --
 View this message in context: 
 http://apache-qpid-users.2158936.n2.nabble.com/Qpid-Java-Broker-performance-lower-than-expected-tp6925405p6925405.html
 Sent from the Apache Qpid users mailing list archive at Nabble.com.

 -
 Apache Qpid - AMQP Messaging Implementation
 Project:      http://qpid.apache.org
 Use/Interact: mailto:users-subscr...@qpid.apache.org



-
Apache Qpid - AMQP Messaging Implementation
Project:  http://qpid.apache.org
Use/Interact: mailto:users-subscr...@qpid.apache.org

RE: Qpid Java Broker performance lower than expected

[Vijay Devadhar]

Hi Robbie,

Thanks for detailed email responses; They really help us in understanding
the internals of qpid; In  this and earlier email to Praveen (my co-worker),
I had the sense that you were suggesting that Berkely DB (BDB) persistent
layer was better than Derby. Is that the case?

If yes, what in terms of design, architecture and interaction with messaging
layer makes BDB better than Derby store?

Thanks
Vijay

-Original Message-
From: Robbie Gemmell [mailto:robbie.gemm...@gmail.com] 
Sent: Tuesday, October 25, 2011 4:38 PM
To: users@qpid.apache.org
Subject: Re: Qpid Java Broker performance lower than expected

Hi Vinay,

I havent done any performance benchmarking of the Derby store to know
what a representative number would actually be, but I will try to take
a look at some point. I havent actually used QpidBench, so can I ask
if there were any specific command(s) you ran so I can try the same
scenarios?

We havent paid much attention to performance of the Java broker for a
while unfortunately because we have been working on various other
issues such agetting memory usage under control and sorting out
correctness issues etc since adding a newer protocol version and doing
some significant refactorings and reimplementations, but as we reach
the light at the end of the tunnel on those it is something which
should move further up the priority list.

It is worth nothing that there is also a BDB persistent store for the
Java broker that you might want to look at, as I would expect it to be
faster. It has recently been moved into the main repo, but is still an
optional module which you need to explicitly ask for to be built
(because BDB itself uses the Sleepycat Licence, which invokes
restrictions upon distribution that mean it is not Apache Licence
compatible). You can build the store module and include it (but not
BDB itself) in the broker binary release bundle by using the following
build command:

ant build release-bin -Dmodules.opt=bdbstore -Ddownload-bdb=true

You will find that downloads the bdb je jar into
qpid/java/lib/bdbstore, and then creates a broker binary release in
qpid/java/broker/release which includes the additional store module.
You can make the BDB je jar available to the broker by creating a
lib\opt subdir and copying the je jar into it, where it will get
picked up automatically assuming you are using Java 6+. You can then
use org.apache.qpid.server.store.berkeleydb.BDBMessageStore as the
store class config instead of the other stores.

Robbie

On 24 October 2011 16:25, vipun vinay.punno...@gmail.com wrote:
 Hi,
  I'm collecting performance figures for QPID Java based broker. The results
 which i got after running the  QpidBench program are a little lower than
 expected. My machine which is a quad core, 8GB RAM with Windows 7 gives a
 message throughput of around 400 messages when both producer and consumer
 client instances are active.

 Qpid Java broker is configured to run over Derby and messaging is in
 persistent mode.  I was expecting somewhere around 1000 atleast going by the
 following blog which does comparisons between different messaging providers.

 http://bhavin.directi.com/rabbitmq-vs-apache-activemq-vs-apache-qpid/

 Do you think, the figures from my tests are correct, or what are the
 expected performance results, or are there any tweaks which need to be done
 for performance gains. I am running out of trunk.

 Thanks  Regards
 Vinay

 --
 View this message in context: 
 http://apache-qpid-users.2158936.n2.nabble.com/Qpid-Java-Broker-performance-lower-than-expected-tp6925405p6925405.html
 Sent from the Apache Qpid users mailing list archive at Nabble.com.

 -
 Apache Qpid - AMQP Messaging Implementation
 Project:      http://qpid.apache.org
 Use/Interact: mailto:users-subscr...@qpid.apache.org



-
Apache Qpid - AMQP Messaging Implementation
Project:  http://qpid.apache.org
Use/Interact: mailto:users-subscr...@qpid.apache.org


-
Apache Qpid - AMQP Messaging Implementation
Project:  http://qpid.apache.org
Use/Interact: mailto:users-subscr...@qpid.apache.org

Re: Qpid Java Broker performance lower than expected

[Danushka Menikkumbura]

Hi Robbie,

I did not notice that the BDB store was faster than the Derby store when I
checked some time back.

Thanks,
Danushka

On Wed, Oct 26, 2011 at 5:07 AM, Robbie Gemmell robbie.gemm...@gmail.comwrote:

 Hi Vinay,

 I havent done any performance benchmarking of the Derby store to know
 what a representative number would actually be, but I will try to take
 a look at some point. I havent actually used QpidBench, so can I ask
 if there were any specific command(s) you ran so I can try the same
 scenarios?

 We havent paid much attention to performance of the Java broker for a
 while unfortunately because we have been working on various other
 issues such agetting memory usage under control and sorting out
 correctness issues etc since adding a newer protocol version and doing
 some significant refactorings and reimplementations, but as we reach
 the light at the end of the tunnel on those it is something which
 should move further up the priority list.

 It is worth nothing that there is also a BDB persistent store for the
 Java broker that you might want to look at, as I would expect it to be
 faster. It has recently been moved into the main repo, but is still an
 optional module which you need to explicitly ask for to be built
 (because BDB itself uses the Sleepycat Licence, which invokes
 restrictions upon distribution that mean it is not Apache Licence
 compatible). You can build the store module and include it (but not
 BDB itself) in the broker binary release bundle by using the following
 build command:

 ant build release-bin -Dmodules.opt=bdbstore -Ddownload-bdb=true

 You will find that downloads the bdb je jar into
 qpid/java/lib/bdbstore, and then creates a broker binary release in
 qpid/java/broker/release which includes the additional store module.
 You can make the BDB je jar available to the broker by creating a
 lib\opt subdir and copying the je jar into it, where it will get
 picked up automatically assuming you are using Java 6+. You can then
 use org.apache.qpid.server.store.berkeleydb.BDBMessageStore as the
 store class config instead of the other stores.

 Robbie

 On 24 October 2011 16:25, vipun vinay.punno...@gmail.com wrote:
  Hi,
   I'm collecting performance figures for QPID Java based broker. The
 results
  which i got after running the  QpidBench program are a little lower than
  expected. My machine which is a quad core, 8GB RAM with Windows 7 gives a
  message throughput of around 400 messages when both producer and consumer
  client instances are active.
 
  Qpid Java broker is configured to run over Derby and messaging is in
  persistent mode.  I was expecting somewhere around 1000 atleast going by
 the
  following blog which does comparisons between different messaging
 providers.
 
  http://bhavin.directi.com/rabbitmq-vs-apache-activemq-vs-apache-qpid/
 
  Do you think, the figures from my tests are correct, or what are the
  expected performance results, or are there any tweaks which need to be
 done
  for performance gains. I am running out of trunk.
 
  Thanks  Regards
  Vinay
 
  --
  View this message in context:
 http://apache-qpid-users.2158936.n2.nabble.com/Qpid-Java-Broker-performance-lower-than-expected-tp6925405p6925405.html
  Sent from the Apache Qpid users mailing list archive at Nabble.com.
 
  -
  Apache Qpid - AMQP Messaging Implementation
  Project:  http://qpid.apache.org
  Use/Interact: mailto:users-subscr...@qpid.apache.org
 
 

 -
 Apache Qpid - AMQP Messaging Implementation
 Project:  http://qpid.apache.org
 Use/Interact: mailto:users-subscr...@qpid.apache.org

Qpid Java Broker performance lower than expected

[vipun]

Hi,
 I'm collecting performance figures for QPID Java based broker. The results
which i got after running the  QpidBench program are a little lower than
expected. My machine which is a quad core, 8GB RAM with Windows 7 gives a
message throughput of around 400 messages when both producer and consumer
client instances are active.

Qpid Java broker is configured to run over Derby and messaging is in
persistent mode.  I was expecting somewhere around 1000 atleast going by the
following blog which does comparisons between different messaging providers.

http://bhavin.directi.com/rabbitmq-vs-apache-activemq-vs-apache-qpid/

Do you think, the figures from my tests are correct, or what are the
expected performance results, or are there any tweaks which need to be done
for performance gains. I am running out of trunk.

Thanks  Regards
Vinay 

--
View this message in context: 
http://apache-qpid-users.2158936.n2.nabble.com/Qpid-Java-Broker-performance-lower-than-expected-tp6925405p6925405.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.

-
Apache Qpid - AMQP Messaging Implementation
Project:  http://qpid.apache.org
Use/Interact: mailto:users-subscr...@qpid.apache.org