Thank you for the response Kobi, What do mean time is given to clean up? From what I have read it looks like the close method on the spout is not guaranteed to be called. How can I clean up if the spout has no notice that it is being shut down? On Dec 4, 2014 2:00 AM, "Kobi Salant" <[email protected]> wrote:
> Hi Vincent, > > From our experience, rebalancing does not keep the inner state of the > bolts/spouts but prepare them from fresh. > It does give your components time to clean up or commit offsets so you can > recover cleanly afterwards. > > Kobi > > On Thu, Dec 4, 2014 at 12:21 AM, Vincent Russell < > [email protected]> wrote: > >> >> I have a question regarding serialization of member variables in spouts. >> We are using storm to parse files in HDFS with storm spouts. Because >> files usually emit many tuples we store which file we are parsing as a >> member variable in the spout. >> >> We have noticed that when rebalancing happens we tend to lose track of >> the file that is being parsed if the file isn't completed. >> >> Can member variables be deserialized during rebalancing? >> >> Is there some other good tricks to use where state must be saved between >> nextTuple calls to protect yourself from rebalancing. >> >> >> I'm using storm version 0.9.0.1. >> >> Thanks, >> Vincent >> > > > This message may contain confidential and/or privileged information. > If you are not the addressee or authorized to receive this on behalf of > the addressee you must not use, copy, disclose or take action based on this > message or any information herein. > If you have received this message in error, please advise the sender > immediately by reply email and delete this message. Thank you. >
