To truly guarantee exact recovery your code needs to be stateless, but in most
cases with a lambda architecture, which is really what storm was initially
designed for, you can deal with keeping some or all of the data only in memory.
For example if I am doing a rolling window aggregation and I get a failure the
worst case is that the currently active windows will be missing some data. If
you have a batch job to correct the error later on you may not need to worry
about it.
- Bobby
On Friday, August 14, 2015 8:27 AM, 임정택 <[email protected]> wrote:
Hi.
First of all, you may want to read "Guaranteeing Message Processing"
<http://storm.apache.org/documentation/Guaranteeing-message-processing.html> to
see how it works.
In short, Spout and Bolt should also be stateless since worker could be
killed and restarted. (at any time)
Also data source for Spout is responsible for replaying the messages when
Spout is killed and restarted.
If data source cannot support it, your spout is responsible for. One way to
support it is storing processed offset to external system.
Hope this helps.
Regards,
Jungtaek Lim (HeartSaVioR)
2015-08-14 15:59 GMT+09:00 王立 <[email protected]>:
> Hi all,
>
> I have a question regarding to the fault tolerance mechanism in storm.
>
> I believe the key idea behind fault tolerance mechanism is the
> statelessness of Nimbus and Supervisors, i.e., storing the state on
> zookeepers, such that the failure on Nimbus and Supervisors does not result
> in state loss and they can be restarted as nothing happened.
>
> My question is: does this means the user has to ensure their spouts and
> bolts to be stateless, in order to guarantee fault tolerance?
>
> Any comment will be highly appreciated! Thanks.
>
> Regards,
> Li
>
>
--
Name : 임 정택
Blog : http://www.heartsavior.net / http://dev.heartsavior.net
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior
