Re: Are you using Apache QPID in production?
We're using it in several enterprise class projects over several product lines. Unfortunately we primarily deploy only on Windows platforms, which are treated as something of a second class citizen in the current (*nix dominated) developer community. Our biggest problems were broker fault tolerance and the ability to run the brokers as a Windows service (we submitted a patch to enable this feature.) While the C++ client does contain a rudimentary failover capability, it does not have automatic failover recovery when a failed broker comes back online, so service "splits" are possible (and common) in a long running system with less than perfect connectivity or incomplete uptime (active clients switch to the second broker and stay there, new clients connect to the first broker and don't know about each other.) Because of the way Boost was used to hide the internal implementation, we ended up having to write a complete wrapper to implement application transparent broker recovery. In the Linux broker deployments this problem does not exist due to good support for federation and clustering, but these features are built on OpenAIS, which is a *nix only solution and implemented in such a way as to be very difficult to port to Windows. The documentation is poor and incomplete, but slowly getting better. All that said, it works very well, and I would probably pick it again had I an opportunity to choose again knowing what I know now... Kerry On Mon, Oct 24, 2011 at 12:58 PM, Vijay Devadhar wrote: > Hi QPID users, > > Are you using QPID in production? What sort of lessons have you learnt > with use of > apache QPID in production? We are evaluating QPID and would love to learn > from your > experiences. Please share if you can spare a few minutes. > > Thanks, > Vijay >
Re: What is the memory footprint of an Apache qpid queue?
Hi Robbie,
I forked the broker from trunk and tried running my test against it.
Here is what I tried now. I use the DerbyMessageStore with the java broker.
I have a JAVA_MEM of "-Xms4096m -Xmx4096m" (4GB)
1) I create one queue
2) I create 25 consumers and register them with the queue
3) I create 25 producers and each enqueues 40K messages(with a session
commit every 100 message) (total messages 1 million[25*4] across all
producers).
This test worked pretty well. The memory usage held up well. It peeked and
then once it was struggling, GC kicks in and then almost all the memory is
reclaimed.
I followed this test immediately with another test, where I increased the
overall number of messages enqueued by the producers to 10 million from 1
million.
Also that worked well.
I just have a basic question to clarify that I'm understanding this right.
Are the messages persisted to disk immediate on calling producer.send() or
is it written to a buffer which is lazily written to disk? (I remember that
you mentioned the queue/consumer doesn't see the message until persisted to
disk)
Also, on the consumer side are the messages always pushed to the
consumer from disk..or is it read to a buffer and then pushed to a disk?
The reason I ask this is my use case requires strict persistent guarantee
and losing messages isn't really an option (I'm ok with taking the hit due
to synchronous reads/writes to disk). I'm hoping Derby with Qpid is the
right thing for me. I'd like to hear your thoughts.
It will be great if you can answer my question.
Thanks a lot,
Praveen
On Thu, Oct 20, 2011 at 11:54 AM, Robbie Gemmell
wrote:
> Thanks Praveen, I will try and take a look at the weekend. The sleep
> in your onMessage() handler shouldnt really affect the broker at all,
> as everything gets put to disk before going near the queue (and then
> consumer), and you said you are only sending 1 message to each queue
> anyway... so its interesting if it does.
>
> In the meantime, it might be worth seeing what behaviour you get from
> trunk (which will branched in a couple weeks to later become 0.14) to
> see if there was any improvement or not, its had a bunch of fixes and
> some major IO changes since 0.12.
>
> You can find nightly builds of the trunk broker and client at:
>
>
> https://builds.apache.org/view/M-R/view/Qpid/job/Qpid-Java-Release/lastSuccessfulBuild/artifact/trunk/qpid/java/broker/release/
>
> https://builds.apache.org/view/M-R/view/Qpid/job/Qpid-Java-Release/lastSuccessfulBuild/artifact/trunk/qpid/java/client/release/
>
> Robbie
>
> On 19 October 2011 19:57, Praveen M wrote:
> > Hi Robbie,
> >
> > Thanks for your mail.
> > I've posted a small code snippet below.(I was advised not to attach code)
> > This might not compile directly..But you'll certainly get an idea of what
> > i'm trying to do.
> >
> > The test basically tries to
> >
> > 1) Connect to qpid
> > 2) Create n number of queues as specified and register a listener for
> each
> > queue.
> > 3) enqueue a message for each queue
> > 4) wait until each of these messages are received and quits.
> >
> > An interesting behavior i saw was, when i have a big number of queues and
> > run this test,
> >
> > When the enqueues are pushed, there is a huge peek in memory usage
> > immediately and then it slowly drops down to a stable number.
> >
> > I tried a variant of the above test but I put a small sleep in my
> handler.
> > In that case, the peak after enqueues doesn't really happen.
> >
> > I suspect the reason for this is, if the consumer isn't keeping up in
> pace
> > with the producer Qpid was stopping to cache messages in memory but defer
> > everything to disk.
> > However, if the consumer is keeping pace with the producer it tries to
> push
> > as much stuff into memory..
> >
> > What I'm looking for is ~ is there a way where I can pull some controls
> to
> > ensure that Qpid doesn't flood the memory when there is a flood of
> messages
> > coming for consumer either fast moving/not. I'm not worried about my
> > throughput a lot. I'd like to restrict the amount of memory qpid uses
> when
> > there is a sudden flood of messages
> >
> > Awaiting your response.
> >
> > Praveen
> >
> >
> >
> > import java.util.*;
> >
> > import javax.jms.*;
> > import javax.jms.Queue;
> > import javax.naming.*;
> >
> >
> > public class QpidMqTransport {
> >
> >private transient Connection connection;
> >transient Session session;
> >private transient MessageProducer emptyProducer;
> >private static String connUrl = "amqp://guest:guest@test
> > /?brokerlist='tcp://localhost:5672'";
> >
> >final String INITIAL_CONTEXT_FACTORY =
> > "org.apache.qpid.jndi.PropertiesFileInitialContextFactory";
> >
> >final String CONNECTION_JNDI_NAME = "local";
> >
> >private InitialContext _ctx;
> >Map queueNameToDestination = new HashMap > Destination>();
> >Map topicNameToDestination = new HashMap > Destination>();
> >private static String options = ";{c
RE: Apache Qpid 0.10 --- 64bit Version
Hi Hamid, > I have downloaded Apache Qpid 0.10, 32bit windows installer from the > following link and tried to use the libararies with Visual Studio 2008. > > http://www.riverace.com/qpid/downloads.htm Ok. > It seems to work when I use win32bit projects. But when I try to compile the > project as 64bit, it gives errors. I have tried to search Qpid 64bit installer but > was not able to find it. Right - I didn't get enough interest in it to build one. > Please let me know, if there is a 64bit Qpid windows installer available? Or if > there is any way to use this 32bit Qpid installer for 64bit projects? You can't mix 32 and 64 bit DLLs, so you'll either need to build 32 bit projects or build 64-bit qpid. If there is sufficient interest expressed, I'll build 64 bit for Qpid 0.14 when it is released. -Steve - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:users-subscr...@qpid.apache.org
Qpid Java Broker performance lower than expected
Hi, I'm collecting performance figures for QPID Java based broker. The results which i got after running the QpidBench program are a little lower than expected. My machine which is a quad core, 8GB RAM with Windows 7 gives a message throughput of around 400 messages when both producer and consumer client instances are active. Qpid Java broker is configured to run over Derby and messaging is in persistent mode. I was expecting somewhere around 1000 atleast going by the following blog which does comparisons between different messaging providers. http://bhavin.directi.com/rabbitmq-vs-apache-activemq-vs-apache-qpid/ Do you think, the figures from my tests are correct, or what are the expected performance results, or are there any tweaks which need to be done for performance gains. I am running out of trunk. Thanks & Regards Vinay -- View this message in context: http://apache-qpid-users.2158936.n2.nabble.com/Qpid-Java-Broker-performance-lower-than-expected-tp6925405p6925405.html Sent from the Apache Qpid users mailing list archive at Nabble.com. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:users-subscr...@qpid.apache.org
Apache Qpid 0.10 --- 64bit Version
Hi, I have downloaded Apache Qpid 0.10, 32bit windows installer from the following link and tried to use the libararies with Visual Studio 2008. http://www.riverace.com/qpid/downloads.htm It seems to work when I use win32bit projects. But when I try to compile the project as 64bit, it gives errors. I have tried to search Qpid 64bit installer but was not able to find it. Please let me know, if there is a 64bit Qpid windows installer available? Or if there is any way to use this 32bit Qpid installer for 64bit projects? Thanks. -- View this message in context: http://apache-qpid-users.2158936.n2.nabble.com/Apache-Qpid-0-10-64bit-Version-tp6925206p6925206.html Sent from the Apache Qpid users mailing list archive at Nabble.com. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:users-subscr...@qpid.apache.org
Re: how to set qpid.session.max_ack_delay in cpp
Hi , check this there is no diff qpid-perftest --sync-publish=true --durable=true --count=10 --tcp-nodelay Processing 1 messages from qpid-perftest_sub_ready . done. Sending start 1 times to qpid-perftest_pub_start Processing 1 messages from qpid-perftest_pub_done . done. Processing 1 messages from qpid-perftest_sub_done . done. Total 20 transfers of 1024 bytes in 9.55857 seconds. Publish transfers/sec: 1.04628 Average: 1.04628 Subscribe transfers/sec: 1.16293 Average: 1.16293 Total transfers/sec: 2.09236 Total Mbytes/sec: 0.00204332 Best Regads, surya On Mon, Oct 24, 2011 at 3:14 PM, Gordon Sim wrote: > On 10/24/2011 08:23 AM, surya prakash wrote: > >> HI, >> >> i tried this command on my system this is giving following result can you >> check and give me the reason why it is sleeping 1 sec after sending one >> message and how to remove that sleep and where to remove. >> >> qpid-perftest --sync-publish=true --durable=true --count=10 >> Processing 1 messages from qpid-perftest_sub_ready . done. >> Sending start 1 times to qpid-perftest_pub_start >> Processing 1 messages from qpid-perftest_pub_done . >> done. >> Processing 1 messages from qpid-perftest_sub_done . done. >> >> Total 20 transfers of 1024 bytes in 9.56732 seconds. >> >> Publish transfers/sec: >> 1.04938 >> Average: 1.04938 >> >> Subscribe transfers/sec: >> 1.16709 >> Average: 1.16709 >> >> Total transfers/sec: 2.09045 >> Total Mbytes/sec: 0.00204145 >> > > There is something strange in your setup. Here is what I see: > > qpid-perftest --sync-publish=true --durable=true --count=1000 >> >> Processing 1 messages from qpid-perftest_sub_ready . done. >> Sending start 1 times to qpid-perftest_pub_start >> Processing 1 messages from qpid-perftest_pub_done . done. >> Processing 1 messages from qpid-perftest_sub_done . done. >> >> Total 2000 transfers of 1024 bytes in 0.797297 seconds. >> >> Publish transfers/sec: >> 1309.09 >> Average: 1309.09 >> >> Subscribe transfers/sec: >> 1190.33 >> Average: 1190.33 >> >> Total transfers/sec: 2508.47 >> Total Mbytes/sec: 2.44968 >> > > Try running with --durable=false for a comparison, and then perhaps with > sync-publish=false (I get about 6000 sync publishes for transient, 20,000+ > for async durable, and about 40,000+ for async transient on my laptop). > > You could also try with --tcp-nodelay both on perftest and on qpidd. > > > --**--**- > Apache Qpid - AMQP Messaging Implementation > Project: http://qpid.apache.org > Use/Interact: > mailto:users-subscribe@qpid.**apache.org > >
Re: how to set qpid.session.max_ack_delay in cpp
On 10/24/2011 08:23 AM, surya prakash wrote: HI, i tried this command on my system this is giving following result can you check and give me the reason why it is sleeping 1 sec after sending one message and how to remove that sleep and where to remove. qpid-perftest --sync-publish=true --durable=true --count=10 Processing 1 messages from qpid-perftest_sub_ready . done. Sending start 1 times to qpid-perftest_pub_start Processing 1 messages from qpid-perftest_pub_done . done. Processing 1 messages from qpid-perftest_sub_done . done. Total 20 transfers of 1024 bytes in 9.56732 seconds. Publish transfers/sec: 1.04938 Average: 1.04938 Subscribe transfers/sec: 1.16709 Average: 1.16709 Total transfers/sec: 2.09045 Total Mbytes/sec: 0.00204145 There is something strange in your setup. Here is what I see: qpid-perftest --sync-publish=true --durable=true --count=1000 Processing 1 messages from qpid-perftest_sub_ready . done. Sending start 1 times to qpid-perftest_pub_start Processing 1 messages from qpid-perftest_pub_done . done. Processing 1 messages from qpid-perftest_sub_done . done. Total 2000 transfers of 1024 bytes in 0.797297 seconds. Publish transfers/sec: 1309.09 Average: 1309.09 Subscribe transfers/sec: 1190.33 Average: 1190.33 Total transfers/sec: 2508.47 Total Mbytes/sec: 2.44968 Try running with --durable=false for a comparison, and then perhaps with sync-publish=false (I get about 6000 sync publishes for transient, 20,000+ for async durable, and about 40,000+ for async transient on my laptop). You could also try with --tcp-nodelay both on perftest and on qpidd. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:users-subscr...@qpid.apache.org
Re: how to set qpid.session.max_ack_delay in cpp
HI, i tried this command on my system this is giving following result can you check and give me the reason why it is sleeping 1 sec after sending one message and how to remove that sleep and where to remove. qpid-perftest --sync-publish=true --durable=true --count=10 Processing 1 messages from qpid-perftest_sub_ready . done. Sending start 1 times to qpid-perftest_pub_start Processing 1 messages from qpid-perftest_pub_done . done. Processing 1 messages from qpid-perftest_sub_done . done. Total 20 transfers of 1024 bytes in 9.56732 seconds. Publish transfers/sec: 1.04938 Average: 1.04938 Subscribe transfers/sec: 1.16709 Average: 1.16709 Total transfers/sec: 2.09045 Total Mbytes/sec: 0.00204145 Best Regards, surya
