Thank you for the detailed answer! Question : what if we used something like Kafka as an intermediary between MQ Series and Storm? Like, if we just used Kafka as a way to pass messages from MQ Series to Storm? Do you think something like that would work?
--Jeremy On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt < [email protected]> wrote: > I tried this once and failed, following are my findings: > > IBM MQ has a strict limitation the thread that receives the message has > to be the thread that acks it. This does not work with storm's threading > model where one thread reads the message and sends it to the bolt another > thread invokes spout's ack method when the bolt acks the message and only > then the spout can ack the message. > > Based on > http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en > we > tried looking into proprietary IBM MQ java client. The acking mechanism > relies on following 3 APIS > > > http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin( > ) > > http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit( > ) > > http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout( > ) > > We could use these with one caveat. There is no way to cherry pick what > messages, or range of messages like JMS where you can pick the message with > lowest id, will get committed. > This means in the spout we will keep an active list of pending messages, > when ack method in spout is called for a message we remove the message from > the pending list and call commit only if the pending list is empty. This > works as long as the producer is slow or the consumers are slow but if both > of them are really fast the pending list may never become empty causing lot > of memory overhead on the box as well as server side overhead of keeping > all the messages around. > we could introduce a pseudo upper limit in spout where we stop handing out > messages once pending message list reaches some upper bound but that will > just slow down the topology. > > In summary, it seems to be impossible to implement a IBM MQ spout that > does not run the risk of message loss and be performant. > > > Thanks > > Parth > > From: jeremy p <[email protected]> > Reply-To: "[email protected]" <[email protected]> > Date: Tuesday, March 31, 2015 at 11:41 AM > To: "[email protected]" <[email protected]> > Subject: Using Storm with IBM MQ Series > > Hello all, > > According to this webinar recap, Storm cannot work with IBM MQ Series > (also known as Websphere MQ) : > > http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/ > > This is a real bummer for me, because my company uses IBM MQ Series, and > we have a new project that Storm would be perfect for. Is there currently > an effort underway to make IBM MQ Series interoperate with Storm? Do you > think it's even possible to make Storm and IBM MQ Series work together? > > I'd really like to use Storm, if I can. >
