Re: Qpid Java Broker performance tests using AMQP 1.0

[Michal Zerola]

At the end I have managed to make it run using AMQP 1.0 protocol against both
Qpid Java broker and Qpid C++ broker.

With Qpid Java broker I had to switch to REST Queue creator and also specify
management parameters. The final command then looks like:



For the C++ broker I have created queues by hand and used "NoOp" Queue
creator. The final command:





--
View this message in context: 
http://qpid.2158936.n2.nabble.com/Qpid-Java-Broker-performance-tests-using-AMQP-1-0-tp7645180p7645383.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Qpid Java Broker performance tests using AMQP 1.0

[Michal Zerola]

Hi,
I want to use performance tests which are part of the Qpid Java Broker to
measure the throughput of several brokers. I was able to configure and run
it using AMQP 0-10 protocol. However, I am struggling to make it use the
AMQP 1.0 client.

What I did:
- specified jndi-config=perftests-jndi-qpid-jms-client.properties to be used
(with 1.0 url naming)
- provided 0.9.0 qpid JMS client on class path
- specified property
java.naming.factory.initial=org.apache.qpid.jms.jndi.JmsInitialContextFactory

It doesn't seem to be enough. Is there some manual or could someone provide
a quick list of thing one needs to do in order to execute performance tests
with 1.0 client?

Thanks,

Michal



--
View this message in context: 
http://qpid.2158936.n2.nabble.com/Qpid-Java-Broker-performance-tests-using-AMQP-1-0-tp7645180.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Unable to send 1MB messages in asynchronous mode

[Michal Zerola]

Hi Gordon,

I have played with the qpid.session.byte_limit and it does affect the
behavior. When I increased the limit to 1024 * 1024 * 10 I was able to send
100 messages. On the other hand, when I switched to smaller messages (1024
bytes size) and decreased the byte_limit to 1024 bytes the problem appeared
again. I have tried to debug the Java client and I have seen that:

- invoke Method in Session.java throws "timed out waiting for completion"
exception
- reason is that isFull() method returns true (comparison of internal
commandBytes counter vs byte_limit)
- looks like complete() method was not called and this is the single place
where commandBytes could be decreased (to 0); otherwise isFull() will always
return true

Thanks.




--
View this message in context: 
http://qpid.2158936.n2.nabble.com/Unable-to-send-1MB-messages-in-asynchronous-mode-tp7643793p7644331.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Unable to send 1MB messages in asynchronous mode

[Michal Zerola]

Hi Gordon,

yes, it looks exactly like the issue described in QPID-4524. It seems
strange to me that the issue was closed with a statement - expected
behavior. I have increased the sync_op_timeout to 5 minutes, but it just
delayed the same error on the client. It is also not a store performance
problem, since I can send 300 messages (all 1MB size) using C++ client in
under a minute without any problem (using async sender). I am attaching logs
again.

nosync_6.log   
nosync_cpp.log
  

To me it seems like a misunderstanding between Java JMS client and C++
broker when using 0-10 protocol. It cannot be the case that broker is simply
not able to process two consecutive 1MB messages within 5 minutes. I can
understand that with 1MB (and more) messages the async mode doesn't bring
much performance gain, however it should work.

Thanks.




--
View this message in context: 
http://qpid.2158936.n2.nabble.com/Unable-to-send-1MB-messages-in-asynchronous-mode-tp7643793p7644270.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Unable to send 1MB messages in asynchronous mode

[Michal Zerola]

Hi Gordon,

I have tested qpid client 0.32 and also 6.0.2 (neither is working against
C++ server). If I use the same code (send three 1MB messages) against Java
Qpid broker, everything works. I am attaching logs, where you can see that
after sending the first message the client waits for completion of the
command from the server. It receives multiple SessionCompleted(commands=[1,
1] ... ) in a row, however, against the Java Broker it receives only single
SessionCompleted(commands=[0, 2]) response. But after some iterations the
message is sent (right after receive of the SessionCompleted(commands=[1,
2]). The problem appears after trying to send the third message (last time I
tested it, it was already after the second ...). Right after send command
client receives SessionCompleted(commands=null) and timeouts after 1 minute.
There were no errors on the server side.

Thanks for help.

nosync_6.log   
nosync_javabroker_6.log
  




--
View this message in context: 
http://qpid.2158936.n2.nabble.com/Unable-to-send-1MB-messages-in-asynchronous-mode-tp7643793p7644185.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Unable to send 1MB messages in asynchronous mode

[Michal Zerola]

I am unable to send 1MB messages using Java JMS API to Qpid (C++ broker)
v0.34 if I use asynchronous send mode (no 'sync_publish' connection option).
First message is sent normally, but the client hangs up on second
producer.send(message) call.

If I use synchronous send mode (sync_publish='all') or smaller messages
(e.g. 10KB) then everything works. I am attaching client logs for both
asynchronous mode (where problem appears) and synchronous mode (where
sending works).

nosync.log   
sync.log   

Messages are sent as persistent to a durable queue using topic exchange.
Thanks for help.

Michal



--
View this message in context: 
http://qpid.2158936.n2.nabble.com/Unable-to-send-1MB-messages-in-asynchronous-mode-tp7643793.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java QMF library and ObjectId

[Michal Zerola]

Hello Fraser,

thank you once again. I have verified that Python qmf/console.py is using
QMF v1 (as well as qpid-tool for example) and that is why I see different
(unique) ObjectIds than comparing to Java which uses QMF v2.

We will use some different mechanism on our side which is able to deal with
that naming.

Thanks for all the explanation regarding the Java implementation.

Cheers!

Michal




--
View this message in context: 
http://qpid.2158936.n2.nabble.com/Re-Java-QMF-library-and-ObjectId-tp7613784p7613940.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java QMF library and ObjectId

[Michal Zerola]

Hello Fraser and thanks for an exhaustive answer.

There are still some foggy parts in this behavior for me, but let me clarify
firstly some parts which were not explained clearly from my side:

* we are connecting to the C++ Qpid broker
* we are really using asynchronous mechanism from Java QMF API:

From what you have written we might be really the only users using
query+subscribe mechanism :) but we are quite satisfied with it. In
particular, what we do is following:

- create qmfPredicate describing what kind of objects are we interested in
- create subscription based on the query using the predicate
- register subscription to the console
- in the onEvent() callback we receive updates and from the WorkItem we
finally mine the ListQmfConsoleData with that ObjectId(s) using which we
identify what was changed/deleted/added and store it in our internal
structures

Now why do we have a problem with the 'recycled' ObjectIds. Because of
auditing/monitoring reasons we need to keep track of also deleted objects on
the broker (for some time). Things are more complicated since objects can
reference each other - e.g. binding references queue and exchange it binds
together. However, if the references are the same for deleted and newly
created objects (with the same name) it is quite a challenge to keep track
of what is referencing what.

Of course, if it is given by the broker, there is not too much we can do
about it.

Now, what still puzzles me:

* from the Python I have been using qmf.console API and again it's
asynchronous mechanism to receive object's updates. Briefly:

...Session(self.eventListener, rcvObjects=True, manageConnections=True,
userBindings=True, rcvEvents=True)
...bindClass(org.apache.qpid.broker, queue)

where eventListener implements objectProps(self, broker, record) and
objectStats(self, broker, record) callbacks. From here I am getting Object
ID:

oid = record.getObjectId()

and here I can see that the ID has a different ('more unique') format: e.g.
the same queue recreated had 0-2-1-0-15440 and later after deletion
0-2-1-0-1592.

I can buy the argument that it uses older QMF1 protocol, but looking into
the code of console.py I can see that it supports both QMF1 + QMF2 and if
the QMF2 is available, the temporary request/response queues created should
have form qmfc-v2-... - which is exactly what I can see on the broker.

This makes me think: is my Python client communicating with QMF1 or QMF2? If
QMF2 why do I see different ObjectId's than in Java QMF if they should be
generated by the broker?

Is there something I am missing?

Thank you.

Cheers,

Michal





--
View this message in context: 
http://qpid.2158936.n2.nabble.com/Re-Java-QMF-library-and-ObjectId-tp7613784p7613803.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: Java Qpid Broker - flow to disk

[Michal Zerola]

Hi Robbie,

thank you for an extensive answer. It is clear to me now.

Michal



--
View this message in context: 
http://qpid.2158936.n2.nabble.com/Java-Qpid-Broker-flow-to-disk-tp7591254p7591439.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Java Qpid Broker - flow to disk

[Michal Zerola]

Hello,

I am curious what is the current state of the flow-to-disk implementation in
Java Qpid broker. Especially, what happens if there are no consumers and
cumulated size of persistent messages starts to reach the JVM memory limit.

Thank you,

Michal



--
View this message in context: 
http://qpid.2158936.n2.nabble.com/Java-Qpid-Broker-flow-to-disk-tp7591254.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: JDBC store in Qpid-Java broker

[Michal Zerola]

Hi Phil,

thank you for your answer!

Michal



--
View this message in context: 
http://qpid.2158936.n2.nabble.com/JDBC-store-in-Qpid-Java-broker-tp7587432p7587525.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

JDBC store in Qpid-Java broker

[Michal Zerola]

Hello,

what is the current state of the support for JDBC message store in Java Qpid
broker? I do see the mention of the SQL store in Qpid documentation, but
there is no more written (just the plain page):

http://qpid.apache.org/books/trunk/AMQP-Messaging-Broker-Java-Book/html/Java-Broker-Stores.html
http://qpid.apache.org/books/trunk/AMQP-Messaging-Broker-Java-Book/html/Java-Broker-Stores.html
  

Checking the sources also gave me the impression that it is not implemented.
I have seen some patches from 2007 regarding support for JDBC, but I do not
think they are included in current release/trunk. So to my understanding,
currently the only persistent stores which are supported are Derby and BDB
(via 3rd party library). Am I correct?

Thank you for clarification.

Michal



--
View this message in context: 
http://qpid.2158936.n2.nabble.com/JDBC-store-in-Qpid-Java-broker-tp7587432.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org

Re: SSL Client Authentication support for C++ on Windows

[Michal Zerola]

Hello, I have extended the Jakub's patch for file based certificate client
authentication.

It is attached to the same  https://issues.apache.org/jira/browse/QPID-3914
JIRA issue .

This patch extends the Jakub's one for the missing client file based
authentication under the Windows. A client may specify new Connection
options:

■ssl-cert-filename
■ssl-cert-filenamepass
■host-cert-filename
for providing the private P12 key from the file, password for accessing this
file and finally the host public certificate from the file. If specified,
the private key is loaded from the file and used for client authentication
instead of finding the certificate in the registry based store. If the host
key filename is specified, the public key from the file attempts to be added
into the Trusted Root Certification Authority store, so the host will be
trusted (I have not found other way to trust the host other than adding the
public key into the store before the handshake). Any combination of the new
connection parameters is allowed (e.g. load private key from the file and
use public host certificate from existing registry store).
This patch increases the portability of the SSL based client applications,
where certificates can be provided with the application and there is no need
to import them by hand before executing the main application.

--
View this message in context: 
http://qpid.2158936.n2.nabble.com/SSL-Client-Authentication-support-for-C-on-Windows-tp7350387p7462400.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org