Hey Taylor : I would love to collaborate with you on this! However, I don't have Storm up and running yet. Right now, I'm just investigating whether or not we can use Storm for this project, or if I need to pick a different technology. It sounds like we may be able to use Storm if we put some development effort into it. Would it be alright if I reached out to you once I have a Storm instance up and running?
On Wed, Apr 1, 2015 at 12:41 PM, P. Taylor Goetz <[email protected]> wrote: > Memory slightly refreshed, but I don’t have access to an IBM-MQ instance > at the moment, so I can’t verify… > > In the storm-jms spout[1], we receive a JMS message in the JMS > `onMessage()` method (the thread that invokes that method is a JMS thread) > and put it in a queue so we can perform the JMS message > acknowledgement/session commit later when the spout `ack()` method is > called (the thread that invokes that method is a Storm thread). > > I believe IBM-MQ’s JMS implementation was enforcing a policy such that the > JMS message acknowledgement/commit must be called by the same thread that > called the `onMessage()` method. Since the thread calling `ack()` (Storm > thread) != the thread calling `onMessage()`, it violated that policy. > > Again, I could be wrong. I’d be happy to help if someone with access to an > IBM-MQ instance would be interested in collaborating. > > -Taylor > > [1] https://github.com/ptgoetz/storm-jms > > On Apr 1, 2015, at 11:55 AM, P. Taylor Goetz <[email protected]> wrote: > > Hi Jeremy, > > I was the engineer who gave that webinar. And the statement you quoted is > misleading (IIRC, that’s not the wording I used when the question was > asked). > > Nathan is correct, the nextTuple(), ack() and fail() methods are all > called from the same thread. > > At the moment I don’t have access to the specific errors that were > encountered when using the storm-jms spout with IBM-MQ, but it was related > that JMS implementation enforcing a strict policy around threads and JMS > message acknowledgement (not Storm acks). I’ll try to dig up more details > to refresh my memory and get back to you. > > -Taylor > > > On Apr 1, 2015, at 11:37 AM, Parth Brahmbhatt <[email protected]> > wrote: > > Jeremy, > > Taylor and I work for the same company and the webinar contains the same > info because it is coming from the same source. We will investigate further > what caused the IBM-MQ integration failure but we can not promise a > timeline. Once again apologies for providing inaccurate info in first place. > > Thanks > Parth > > From: jeremy p <[email protected]> > Reply-To: user <[email protected]> > Date: Wednesday, April 1, 2015 at 8:19 AM > To: user <[email protected]> > Subject: Re: Using Storm with IBM MQ Series > > Nathan : thanks for the response! It appears that the engineer who gave > this webinar ran into the same problem as Parth : > > http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/ > > He reports that he was unable to make MQ Series work with Storm because > : > "We ran into issues with IBM MQ-Series with respect to how messages are > acknowledged by IBM MQ. IBM-MQ requires the thread that receives the > message be the same thread that acks it. Storm’s framework cannot support > this requirement, as the receiving and acking thread by design are > different threads to achieve higher throughput." > > If threading wasn't the issue, how would you explain Storm's behavior in > this situation? > > --Jeremy > > On Wed, Apr 1, 2015 at 12:11 AM, Parth Brahmbhatt < > [email protected]> wrote: > >> Thanks for correcting me Nathan. >> >> Jeremy, in that case you can give storm-jms >> <https://github.com/ptgoetz/storm-jms> a shot and see if it works for >> you. >> >> Thanks >> Parth >> >> From: Nathan Marz <[email protected]> >> Reply-To: "[email protected]" <[email protected]> >> Date: Tuesday, March 31, 2015 at 6:11 PM >> To: "[email protected]" <[email protected]> >> Subject: Re: Using Storm with IBM MQ Series >> >> That's not accurate. A spout task's nextTuple, ack, and fail methods >> are all called by the exact same thread. You're confusing Storm acks with >> acks that go back to the source message queue. Storm acks are about >> detecting tuple DAG completion and have nothing to do with the ack to the >> source message queue. >> >> So if you weren't able to get it to work then it was a different >> problem. >> >> On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt < >> [email protected]> wrote: >> >>> I tried this once and failed, following are my findings: >>> >>> IBM MQ has a strict limitation the thread that receives the message >>> has to be the thread that acks it. This does not work with storm's >>> threading model where one thread reads the message and sends it to the bolt >>> another thread invokes spout's ack method when the bolt acks the message >>> and only then the spout can ack the message. >>> >>> Based on >>> http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en >>> we >>> tried looking into proprietary IBM MQ java client. The acking mechanism >>> relies on following 3 APIS >>> >>> >>> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin( >>> ) >>> >>> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit( >>> ) >>> >>> http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout( >>> ) >>> >>> We could use these with one caveat. There is no way to cherry pick what >>> messages, or range of messages like JMS where you can pick the message with >>> lowest id, will get committed. >>> This means in the spout we will keep an active list of pending messages, >>> when ack method in spout is called for a message we remove the message from >>> the pending list and call commit only if the pending list is empty. This >>> works as long as the producer is slow or the consumers are slow but if both >>> of them are really fast the pending list may never become empty causing lot >>> of memory overhead on the box as well as server side overhead of keeping >>> all the messages around. >>> we could introduce a pseudo upper limit in spout where we stop handing >>> out messages once pending message list reaches some upper bound but that >>> will just slow down the topology. >>> >>> In summary, it seems to be impossible to implement a IBM MQ spout that >>> does not run the risk of message loss and be performant. >>> >>> >>> Thanks >>> >>> Parth >>> >>> From: jeremy p <[email protected]> >>> Reply-To: "[email protected]" <[email protected]> >>> Date: Tuesday, March 31, 2015 at 11:41 AM >>> To: "[email protected]" <[email protected]> >>> Subject: Using Storm with IBM MQ Series >>> >>> Hello all, >>> >>> According to this webinar recap, Storm cannot work with IBM MQ Series >>> (also known as Websphere MQ) : >>> >>> http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/ >>> >>> This is a real bummer for me, because my company uses IBM MQ Series, >>> and we have a new project that Storm would be perfect for. Is there >>> currently an effort underway to make IBM MQ Series interoperate with >>> Storm? Do you think it's even possible to make Storm and IBM MQ Series >>> work together? >>> >>> I'd really like to use Storm, if I can. >>> >> >> >> >> -- >> Twitter: @nathanmarz >> http://nathanmarz.com >> > > > >
