That's not accurate. A spout task's nextTuple, ack, and fail methods are all called by the exact same thread. You're confusing Storm acks with acks that go back to the source message queue. Storm acks are about detecting tuple DAG completion and have nothing to do with the ack to the source message queue.
So if you weren't able to get it to work then it was a different problem. On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt < [email protected]> wrote: > I tried this once and failed, following are my findings: > > IBM MQ has a strict limitation the thread that receives the message has > to be the thread that acks it. This does not work with storm's threading > model where one thread reads the message and sends it to the bolt another > thread invokes spout's ack method when the bolt acks the message and only > then the spout can ack the message. > > Based on > http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en > we > tried looking into proprietary IBM MQ java client. The acking mechanism > relies on following 3 APIS > > > http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin( > ) > > http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit( > ) > > http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout( > ) > > We could use these with one caveat. There is no way to cherry pick what > messages, or range of messages like JMS where you can pick the message with > lowest id, will get committed. > This means in the spout we will keep an active list of pending messages, > when ack method in spout is called for a message we remove the message from > the pending list and call commit only if the pending list is empty. This > works as long as the producer is slow or the consumers are slow but if both > of them are really fast the pending list may never become empty causing lot > of memory overhead on the box as well as server side overhead of keeping > all the messages around. > we could introduce a pseudo upper limit in spout where we stop handing out > messages once pending message list reaches some upper bound but that will > just slow down the topology. > > In summary, it seems to be impossible to implement a IBM MQ spout that > does not run the risk of message loss and be performant. > > > Thanks > > Parth > > From: jeremy p <[email protected]> > Reply-To: "[email protected]" <[email protected]> > Date: Tuesday, March 31, 2015 at 11:41 AM > To: "[email protected]" <[email protected]> > Subject: Using Storm with IBM MQ Series > > Hello all, > > According to this webinar recap, Storm cannot work with IBM MQ Series > (also known as Websphere MQ) : > > http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/ > > This is a real bummer for me, because my company uses IBM MQ Series, and > we have a new project that Storm would be perfect for. Is there currently > an effort underway to make IBM MQ Series interoperate with Storm? Do you > think it's even possible to make Storm and IBM MQ Series work together? > > I'd really like to use Storm, if I can. > -- Twitter: @nathanmarz http://nathanmarz.com
