All good. Thank you for your help! I really appreciate it. Please keep us updated as to your findings. Likewise, I'll let you know if I uncover anything.
--Jeremy On Wed, Apr 1, 2015 at 11:37 AM, Parth Brahmbhatt < [email protected]> wrote: > Jeremy, > > Taylor and I work for the same company and the webinar contains the same > info because it is coming from the same source. We will investigate further > what caused the IBM-MQ integration failure but we can not promise a > timeline. Once again apologies for providing inaccurate info in first place. > > Thanks > Parth > > From: jeremy p <[email protected]> > Reply-To: user <[email protected]> > Date: Wednesday, April 1, 2015 at 8:19 AM > To: user <[email protected]> > > Subject: Re: Using Storm with IBM MQ Series > > Nathan : thanks for the response! It appears that the engineer who gave > this webinar ran into the same problem as Parth : > > http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/ > > He reports that he was unable to make MQ Series work with Storm because > : > "We ran into issues with IBM MQ-Series with respect to how messages are > acknowledged by IBM MQ. IBM-MQ requires the thread that receives the > message be the same thread that acks it. Storm’s framework cannot support > this requirement, as the receiving and acking thread by design are > different threads to achieve higher throughput." > > If threading wasn't the issue, how would you explain Storm's behavior in > this situation? > > --Jeremy > > On Wed, Apr 1, 2015 at 12:11 AM, Parth Brahmbhatt < > [email protected]> wrote: > >> Thanks for correcting me Nathan. >> >> Jeremy, in that case you can give storm-jms >> <https://github.com/ptgoetz/storm-jms> a shot and see if it works for >> you. >> >> Thanks >> Parth >> >> From: Nathan Marz <[email protected]> >> Reply-To: "[email protected]" <[email protected]> >> Date: Tuesday, March 31, 2015 at 6:11 PM >> To: "[email protected]" <[email protected]> >> Subject: Re: Using Storm with IBM MQ Series >> >> That's not accurate. A spout task's nextTuple, ack, and fail methods >> are all called by the exact same thread. You're confusing Storm acks with >> acks that go back to the source message queue. Storm acks are about >> detecting tuple DAG completion and have nothing to do with the ack to the >> source message queue. >> >> So if you weren't able to get it to work then it was a different >> problem. >> >> On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt < >> [email protected]> wrote: >> >>> I tried this once and failed, following are my findings: >>> >>> IBM MQ has a strict limitation the thread that receives the message >>> has to be the thread that acks it. This does not work with storm's >>> threading model where one thread reads the message and sends it to the bolt >>> another thread invokes spout's ack method when the bolt acks the message >>> and only then the spout can ack the message. >>> >>> Based on >>> http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en >>> we >>> tried looking into proprietary IBM MQ java client. The acking mechanism >>> relies on following 3 APIS >>> >>> >>> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin( >>> ) >>> >>> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit( >>> ) >>> >>> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout( >>> ) >>> >>> We could use these with one caveat. There is no way to cherry pick what >>> messages, or range of messages like JMS where you can pick the message with >>> lowest id, will get committed. >>> This means in the spout we will keep an active list of pending messages, >>> when ack method in spout is called for a message we remove the message from >>> the pending list and call commit only if the pending list is empty. This >>> works as long as the producer is slow or the consumers are slow but if both >>> of them are really fast the pending list may never become empty causing lot >>> of memory overhead on the box as well as server side overhead of keeping >>> all the messages around. >>> we could introduce a pseudo upper limit in spout where we stop handing >>> out messages once pending message list reaches some upper bound but that >>> will just slow down the topology. >>> >>> In summary, it seems to be impossible to implement a IBM MQ spout that >>> does not run the risk of message loss and be performant. >>> >>> >>> Thanks >>> >>> Parth >>> >>> From: jeremy p <[email protected]> >>> Reply-To: "[email protected]" <[email protected]> >>> Date: Tuesday, March 31, 2015 at 11:41 AM >>> To: "[email protected]" <[email protected]> >>> Subject: Using Storm with IBM MQ Series >>> >>> Hello all, >>> >>> According to this webinar recap, Storm cannot work with IBM MQ Series >>> (also known as Websphere MQ) : >>> >>> http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/ >>> >>> This is a real bummer for me, because my company uses IBM MQ Series, >>> and we have a new project that Storm would be perfect for. Is there >>> currently an effort underway to make IBM MQ Series interoperate with >>> Storm? Do you think it's even possible to make Storm and IBM MQ Series >>> work together? >>> >>> I'd really like to use Storm, if I can. >>> >> >> >> >> -- >> Twitter: @nathanmarz >> http://nathanmarz.com >> > >
