Why couldn't a thread inside the spout be responsible for communications with IBM MQ?
That thread would read messages from the MQ, push them into a queue (most spouts do that anyways), the spout can deliver messages from the queue. ack/fail could put events in another queue. That same thread that's filling the queue can also read the ack/fail queue and communicate with IBM MQ. *Michael Rose* Senior Software Engineer, Data Science *Full*Contact | fullcontact.com <https://www.fullcontact.com/?utm_source=FullContact%20-%20Email%20Signatures&utm_medium=email&utm_content=Signature%20Link&utm_campaign=FullContact%20-%20Email%20Signatures> m: +1.720.837.1357 | t: @xorlev All Your Contacts, Updated and In One Place. Try FullContact for Free <https://www.fullcontact.com/?utm_source=FullContact%20-%20Email%20Signatures&utm_medium=email&utm_content=Signature%20Link&utm_campaign=FullContact%20-%20Email%20Signatures> On Tue, Mar 31, 2015 at 1:00 PM, Parth Brahmbhatt < [email protected]> wrote: > Sure, as long as you can figure out a performant way to push messages > from IBM MQ to kafka it should work. > > Thanks > Parth > From: jeremy p <[email protected]> > Reply-To: user <[email protected]> > Date: Tuesday, March 31, 2015 at 11:56 AM > To: user <[email protected]> > Subject: Re: Using Storm with IBM MQ Series > > Thank you for the detailed answer! Question : what if we used > something like Kafka as an intermediary between MQ Series and Storm? Like, > if we just used Kafka as a way to pass messages from MQ Series to Storm? > Do you think something like that would work? > > --Jeremy > > On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt < > [email protected]> wrote: > >> I tried this once and failed, following are my findings: >> >> IBM MQ has a strict limitation the thread that receives the message >> has to be the thread that acks it. This does not work with storm's >> threading model where one thread reads the message and sends it to the bolt >> another thread invokes spout's ack method when the bolt acks the message >> and only then the spout can ack the message. >> >> Based on >> http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en >> we >> tried looking into proprietary IBM MQ java client. The acking mechanism >> relies on following 3 APIS >> >> >> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin( >> ) >> >> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit( >> ) >> >> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout( >> ) >> >> We could use these with one caveat. There is no way to cherry pick what >> messages, or range of messages like JMS where you can pick the message with >> lowest id, will get committed. >> This means in the spout we will keep an active list of pending messages, >> when ack method in spout is called for a message we remove the message from >> the pending list and call commit only if the pending list is empty. This >> works as long as the producer is slow or the consumers are slow but if both >> of them are really fast the pending list may never become empty causing lot >> of memory overhead on the box as well as server side overhead of keeping >> all the messages around. >> we could introduce a pseudo upper limit in spout where we stop handing >> out messages once pending message list reaches some upper bound but that >> will just slow down the topology. >> >> In summary, it seems to be impossible to implement a IBM MQ spout that >> does not run the risk of message loss and be performant. >> >> >> Thanks >> >> Parth >> >> From: jeremy p <[email protected]> >> Reply-To: "[email protected]" <[email protected]> >> Date: Tuesday, March 31, 2015 at 11:41 AM >> To: "[email protected]" <[email protected]> >> Subject: Using Storm with IBM MQ Series >> >> Hello all, >> >> According to this webinar recap, Storm cannot work with IBM MQ Series >> (also known as Websphere MQ) : >> >> http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/ >> >> This is a real bummer for me, because my company uses IBM MQ Series, >> and we have a new project that Storm would be perfect for. Is there >> currently an effort underway to make IBM MQ Series interoperate with >> Storm? Do you think it's even possible to make Storm and IBM MQ Series >> work together? >> >> I'd really like to use Storm, if I can. >> > >
