Hi Arun, That is a correct understanding. Our use-case is that we cannot live up to the total ordering requirement of the incoming tuples to a stateful windowed bolt due to performance reasons. Ideally we would like to construct our own stateful windowed bolt which would not try to replay the last window in a strict order, but simply drop that data (acceptable to us) or replay it in arbitrary order. Manual acking sounds good since we can probably implement this on our own then. As it is now, if I understand you corrently, there is no way of delaying acking in a stateless bolt due to the auto-acking feature. Great to hear that the patch is under way. Thanks a lot!
Best regards Alexander On Tue, Mar 8, 2016 at 9:58 AM, Arun Mahadevan <[email protected]> wrote: > Hi Alexander, > > If I understand correctly, you have a stateful bolt and a windowed bolt > that extends BaseWindowedBolt in your topology and the tuples are acked > before the execute method of the windowed bolt is invoked. I think this is > because we auto ack the non-stateful bolt (windowed bolt) tuples. This > should get fixed with the patch to remove auto acking for non stateful > bolts. > > Thanks, > Arun > > From: Alexander T > Reply-To: "[email protected]" > Date: Tuesday, March 8, 2016 at 1:55 PM > > To: "[email protected]" > Subject: Re: Stateful bolts and acking > > Hi Arun, > > Great to hear about the patch you are working on. I have another question > if you have time: Right now I'm seeing acking a bit early in the chain, > specifically for a windowed bolt we have which extends BaseWindowedBolt. > The messages seem to be acked right before they reach this bolt, and if I > replace it with a simple stateful bolt, the acking starts working as > expected again. Is there support for non-stateful windowed bolts? > > Best regards > > Alexander > > On Mon, Mar 7, 2016 at 12:16 PM, Arun Mahadevan <[email protected]> wrote: > >> Hi Alexander, >> >> You are right, the acking for the non-stateful bolts in a stateful >> topology is better handled by the bolts themselves. >> >> I plan to submit a patch soon. You can track it here - >> https://issues.apache.org/jira/browse/STORM-1608 >> >> Thanks, >> Arun >> >> From: Alexander T >> Reply-To: "[email protected]" >> Date: Monday, March 7, 2016 at 4:37 PM >> >> To: "[email protected]" >> Subject: Re: Stateful bolts and acking >> >> Hi Arun, >> >> After reading your messages a bit more thoroughly (sorry :) ) I managed >> to get the basic case working. If I understand you correctly, one should >> inherit from BaseRichBolt, which normally requires manual acking, but *not* >> perform any manual acking. And one should *not* extends BaseBasicBolt which >> normally auto-acks, since this auto-acking will not work as intended with >> stateful topologies. >> >> This would unfortunately mean that we cannot use bolts which were >> written for stateless topologies in stateful ones. Are there any plans of >> adapters or to change the interface so that they can interoperate? >> >> Best regards >> Alexander >> >> On Fri, Mar 4, 2016 at 7:26 PM, Arun Mahadevan <[email protected]> wrote: >> >>> Hi Alexander, >>> >>> The simple topology like the one you have mentioned with a single spout >>> and a stateful bolt with an empty execute works for me. I assume you are >>> anchoring the tuple in the spout’s emit. >>> >>> For a stateful topology, the acking doesn’t work if a bolt extends from >>> BaseBasicBolt or if theres an “ack" call in the bolt’s execute. >>> >>> You could open a JIRA (https://issues.apache.org/jira/browse/STORM) and >>> share the sample topology code where you see this issue. >>> >>> Thanks, >>> Arun >>> >>> >>> From: Alexander T >>> Reply-To: "[email protected]" >>> Date: Friday, March 4, 2016 at 11:15 PM >>> To: "[email protected]" >>> Subject: Re: Stateful bolts and acking >>> >>> Hi Arun, >>> >>> Thank you for your reply. I have tried both manual and automatic acking, >>> and neither works. Or rather, the acker is running but my spout is still >>> receiving failure callbacks after the message timeout. I have a simple >>> topology with a single spout reading from a file and a single empty bolt >>> doing nothing. If I change from an empty stateless bolt to an empty >>> stateful bolt the acking stops working. >>> >>> Should I proceed with sending this in as a reproducing test case? Or is >>> there anything else which I might have missed? >>> >>> Regards, >>> Alexander >>> On Mar 4, 2016 5:30 PM, "Arun Mahadevan" <[email protected]> wrote: >>> >>>> Hi Alexander, >>>> >>>> For a stateful topology the anchoring and acking is automatically taken >>>> care of. >>>> >>>> Can you check if any of your bolts inherit BaseBasicBolt or if you are >>>> manually acking. Your non-stateful bolts could inherit from BaseRichBolt >>>> instead. >>>> >>>> Thanks, >>>> Arun >>>> >>>> >>>> From: Alexander T >>>> Reply-To: "[email protected]" >>>> Date: Friday, March 4, 2016 at 8:04 PM >>>> To: "[email protected]" >>>> Subject: Stateful bolts and acking >>>> >>>> Hello! >>>> >>>> I'm on the 1.x-branch and trying to use the stateful bolts. But I >>>> cannot get it to work with acking. No matter what I try acking simply >>>> doesn't seem to have any effect. I see acking tasks suceeding in the logs >>>> (ack tuples forwarded to the acker, acker runs successfully), but still >>>> acking is not succesful and the tuples fail after the message timeout. I've >>>> not been successful in debugging the acker and the rotating map to figure >>>> out why acking is not completing. I was wondering if acking is supposed to >>>> work with stateful bolts, if so how and if there are any examples? Any >>>> ideas would be appreciated. >>>> >>>> Best regards, >>>> >>>> Alexander T >>>> >>> >> >
