It's the nature of the beast unfortunately, you're never guaranteed a clean shutdown in exceptional cases. You need to architect for that. However during a rebalance your cleanup method will be called.
Michael Rose (@Xorlev <https://twitter.com/xorlev>) Senior Platform Engineer, FullContact <http://www.fullcontact.com/> [email protected] On Thu, Dec 4, 2014 at 6:42 AM, Kobi Salant <[email protected]> wrote: > Hi Vincent, > > You are right that there is no special function for rebalancing but the > deactivate method should be called. I am not sure if you can know the > reason why the topology was deactivated but you can add you cleanups in the > deactivate and the warm ups in the activate. > > Kobi > > On Thu, Dec 4, 2014 at 2:14 PM, Vincent Russell <[email protected] > > wrote: > >> Thank you for the response Kobi, >> >> What do mean time is given to clean up? From what I have read it looks >> like the close method on the spout is not guaranteed to be called. How >> can I clean up if the spout has no notice that it is being shut down? >> On Dec 4, 2014 2:00 AM, "Kobi Salant" <[email protected]> wrote: >> >>> Hi Vincent, >>> >>> From our experience, rebalancing does not keep the inner state of the >>> bolts/spouts but prepare them from fresh. >>> It does give your components time to clean up or commit offsets so you >>> can recover cleanly afterwards. >>> >>> Kobi >>> >>> On Thu, Dec 4, 2014 at 12:21 AM, Vincent Russell < >>> [email protected]> wrote: >>> >>>> >>>> I have a question regarding serialization of member variables in >>>> spouts. We are using storm to parse files in HDFS with storm spouts. >>>> Because files usually emit many tuples we store which file we are parsing >>>> as a member variable in the spout. >>>> >>>> We have noticed that when rebalancing happens we tend to lose track of >>>> the file that is being parsed if the file isn't completed. >>>> >>>> Can member variables be deserialized during rebalancing? >>>> >>>> Is there some other good tricks to use where state must be saved >>>> between nextTuple calls to protect yourself from rebalancing. >>>> >>>> >>>> I'm using storm version 0.9.0.1. >>>> >>>> Thanks, >>>> Vincent >>>> >>> >>> >>> This message may contain confidential and/or privileged information. >>> If you are not the addressee or authorized to receive this on behalf of >>> the addressee you must not use, copy, disclose or take action based on this >>> message or any information herein. >>> If you have received this message in error, please advise the sender >>> immediately by reply email and delete this message. Thank you. >>> >> > > This message may contain confidential and/or privileged information. > If you are not the addressee or authorized to receive this on behalf of > the addressee you must not use, copy, disclose or take action based on this > message or any information herein. > If you have received this message in error, please advise the sender > immediately by reply email and delete this message. Thank you. >
