Re: how to keep storm running when console ends

[Ethan Li]

That makes sense to me. I just don't know why I didn't receive Bobby's email in 
this thread. 

On Friday, August 4, 2017 10:34 AM, I PVP  wrote:
 

  #yiv9409744246 body{font-family:Helvetica, Arial;font-size:13px;}Hi J. R. 
Pauley. 
If that helps :  I use supervisord not only for Storm but also for other 
solutions within the ecosystem.
- IP VP 

On August 4, 2017 at 12:29:41 PM, Bobby Evans (ev...@yahoo-inc.com) wrote:

Java by default also tries to keep running if something unexpected happens.  If 
an exception is not caught and goes all the way up the stack, that thread will 
just print out a stack trace to stderr and then quietly stop running.  Do you 
know how many zombie daemons I have had to debug/fix because something slightly 
unexpected happened on a critical thread.  I personally added in a generic 
uncaught exception handler to all of the Hadoop daemons years ago because it 
was happening all of the time in the early days of YARN.  Storm is almost 100% 
immune to those situations, and has been since the beginning.
That is not to say that storm is not stable.  On most of our clusters the only 
time nimbus or a supervisor goes down is when we upgrade it.  We have had 
daemons running for months.  And if they do go down they typically recover in a 
few seconds, and my team gets an alert so we can debug what happened.  Running 
storm (and I would argue any critical daemon) under supervision for production 
environments is just a best practice. 
For debugging/testing something then go ahead and use nohup.

- Bobby


On Friday, August 4, 2017, 9:43:10 AM CDT, J.R. Pauley  
wrote:

thanks for all the responses. screen and nohup seem easy to adopt. 
I guess for something intended to run on large clusters I'm surprised at the 
default behavior of storm. It seems quite the opposite of what I would've 
guessed. 
I also have a Heron instance running in similar fashion and it's default 
behavior is to keep running which seems more what I expected.
On Fri, Aug 4, 2017 at 9:28 AM, M. Aaron Bossert wrote:

Try adding nohup at the beginning of each of those commands.  Nohup prevents a 
process from terminating when it's parent shell is closed.
nohup /opt/storm/apache*/bin/ storm nimbus &

Sent from my iPhone
On Aug 4, 2017, at 09:08, Ethan Li  wrote:


Hi,
Storm docs recommend to use daemontools or monit which provides error recovery 
and etc.. 
I use linux "nohup" or "screen" command for simplicity. 
Ethan 


On Friday, August 4, 2017 7:29 AM, J.R. Pauley  wrote:


this seems silly but I have not figured out how to keep nimbus, supervisor 
running after console session ends. zookeeper survives but supervisor and 
nimbus shut down when the console session ends.
I have storm 1.0.2 installed under /opt/storm and starting up 
as:/opt/storm/apache*/bin/storm nimbus&/opt/storm/apache*/bin/storm supervisor&
/opt/storm/apache*/bin/storm ui&
/opt/storm/apache*/bin/storm drpc&

I know I can add a while loop to keep the console active but there has to be a 
better way






   

Re: how to keep storm running when console ends

[I PVP]

Hi J. R. Pauley.

If that helps :  I use supervisord not only for Storm but also for other 
solutions within the ecosystem.

- IP VP


On August 4, 2017 at 12:29:41 PM, Bobby Evans 
(ev...@yahoo-inc.com) wrote:

Java by default also tries to keep running if something unexpected happens.  If 
an exception is not caught and goes all the way up the stack, that thread will 
just print out a stack trace to stderr and then quietly stop running.  Do you 
know how many zombie daemons I have had to debug/fix because something slightly 
unexpected happened on a critical thread.  I personally added in a generic 
uncaught exception handler to all of the Hadoop daemons years ago because it 
was happening all of the time in the early days of YARN.  Storm is almost 100% 
immune to those situations, and has been since the beginning.

That is not to say that storm is not stable.  On most of our clusters the only 
time nimbus or a supervisor goes down is when we upgrade it.  We have had 
daemons running for months.  And if they do go down they typically recover in a 
few seconds, and my team gets an alert so we can debug what happened.  Running 
storm (and I would argue any critical daemon) under supervision for production 
environments is just a best practice.

For debugging/testing something then go ahead and use nohup.

- Bobby



On Friday, August 4, 2017, 9:43:10 AM CDT, J.R. Pauley  
wrote:


thanks for all the responses. screen and nohup seem easy to adopt.

I guess for something intended to run on large clusters I'm surprised at the 
default behavior of storm. It seems quite the opposite of what I would've 
guessed.

I also have a Heron instance running in similar fashion and it's default 
behavior is to keep running which seems more what I expected.

On Fri, Aug 4, 2017 at 9:28 AM, M. Aaron Bossert 
> wrote:
Try adding nohup at the beginning of each of those commands.  Nohup prevents a 
process from terminating when it's parent shell is closed.

nohup /opt/storm/apache*/bin/ storm nimbus &


Sent from my iPhone

On Aug 4, 2017, at 09:08, Ethan Li 
> wrote:

Hi,

Storm 
docs
 recommend to use daemontools or monit which provides error recovery and etc..

I use linux "nohup" or "screen" command for simplicity.

Ethan




On Friday, August 4, 2017 7:29 AM, J.R. Pauley 
> wrote:


this seems silly but I have not figured out how to keep nimbus, supervisor 
running after console session ends. zookeeper survives but supervisor and 
nimbus shut down when the console session ends.

I have storm 1.0.2 installed under /opt/storm and starting up as:
/opt/storm/apache*/bin/storm nimbus&
/opt/storm/apache*/bin/storm supervisor&
/opt/storm/apache*/bin/storm ui&
/opt/storm/apache*/bin/storm drpc&

I know I can add a while loop to keep the console active but there has to be a 
better way

Re: how to keep storm running when console ends

[Bobby Evans]

Java by default also tries to keep running if something unexpected happens.  If 
an exception is not caught and goes all the way up the stack, that thread will 
just print out a stack trace to stderr and then quietly stop running.  Do you 
know how many zombie daemons I have had to debug/fix because something slightly 
unexpected happened on a critical thread.  I personally added in a generic 
uncaught exception handler to all of the Hadoop daemons years ago because it 
was happening all of the time in the early days of YARN.  Storm is almost 100% 
immune to those situations, and has been since the beginning.
That is not to say that storm is not stable.  On most of our clusters the only 
time nimbus or a supervisor goes down is when we upgrade it.  We have had 
daemons running for months.  And if they do go down they typically recover in a 
few seconds, and my team gets an alert so we can debug what happened.  Running 
storm (and I would argue any critical daemon) under supervision for production 
environments is just a best practice.  
For debugging/testing something then go ahead and use nohup.

- Bobby


On Friday, August 4, 2017, 9:43:10 AM CDT, J.R. Pauley  
wrote:

thanks for all the responses. screen and nohup seem easy to adopt. 
I guess for something intended to run on large clusters I'm surprised at the 
default behavior of storm. It seems quite the opposite of what I would've 
guessed. 
I also have a Heron instance running in similar fashion and it's default 
behavior is to keep running which seems more what I expected.
On Fri, Aug 4, 2017 at 9:28 AM, M. Aaron Bossert  wrote:

Try adding nohup at the beginning of each of those commands.  Nohup prevents a 
process from terminating when it's parent shell is closed.
nohup /opt/storm/apache*/bin/ storm nimbus &

Sent from my iPhone
On Aug 4, 2017, at 09:08, Ethan Li  wrote:


Hi,
Storm docs recommend to use daemontools or monit which provides error recovery 
and etc.. 
I use linux "nohup" or "screen" command for simplicity. 
Ethan 
 

On Friday, August 4, 2017 7:29 AM, J.R. Pauley  wrote:
 

 this seems silly but I have not figured out how to keep nimbus, supervisor 
running after console session ends. zookeeper survives but supervisor and 
nimbus shut down when the console session ends.
I have storm 1.0.2 installed under /opt/storm and starting up 
as:/opt/storm/apache*/bin/storm nimbus&/opt/storm/apache*/bin/storm supervisor&
/opt/storm/apache*/bin/storm ui&
/opt/storm/apache*/bin/storm drpc&

I know I can add a while loop to keep the console active but there has to be a 
better way

   

Re: how to keep storm running when console ends

[J.R. Pauley]

thanks for all the responses. screen and nohup seem easy to adopt.

I guess for something intended to run on large clusters I'm surprised at
the default behavior of storm. It seems quite the opposite of what I
would've guessed.

I also have a Heron instance running in similar fashion and it's default
behavior is to keep running which seems more what I expected.

On Fri, Aug 4, 2017 at 9:28 AM, M. Aaron Bossert 
wrote:

> Try adding nohup at the beginning of each of those commands.  Nohup
> prevents a process from terminating when it's parent shell is closed.
>
> nohup /opt/storm/apache*/bin/storm nimbus &
>
>
> Sent from my iPhone
>
> On Aug 4, 2017, at 09:08, Ethan Li  wrote:
>
> Hi,
>
> Storm docs
> 
>  recommend
> to use daemontools or monit which provides error recovery and etc..
>
> I use linux "nohup" or "screen" command for simplicity.
>
> Ethan
>
>
>
>
> On Friday, August 4, 2017 7:29 AM, J.R. Pauley  wrote:
>
>
> this seems silly but I have not figured out how to keep nimbus, supervisor
> running after console session ends. zookeeper survives but supervisor and
> nimbus shut down when the console session ends.
>
> I have storm 1.0.2 installed under /opt/storm and starting up as:
> /opt/storm/apache*/bin/storm nimbus&
> /opt/storm/apache*/bin/storm supervisor&
> /opt/storm/apache*/bin/storm ui&
> /opt/storm/apache*/bin/storm drpc&
>
> I know I can add a while loop to keep the console active but there has to
> be a better way
>
>
>

Re: Executor Change

[Bobby Evans]

Storm is by default stateless.  If you have state to store it is up to you to 
store that state in an external system, in a way that meets your topologies 
requirements.  At least one vs at most once.  In newer versions of storm the 
concept of stateful bolts was added.
http://storm.apache.org/releases/1.0.3/State-checkpointing.html
is the documentation for that feature.  It still saves the state in an external 
system (that you can pick) but also helps to fix some of the complexity 
associated with checkpointing and restoring state in line with replay of failed 
messages.

- Bobby


On Friday, August 4, 2017, 12:45:17 AM CDT, ? ??  
wrote:

#yiv5765339309 body {line-height:1.5;}#yiv5765339309 body 
{font-size:10.5pt;font-family:'Microsoft YaHei UI';color:rgb(0, 0, 
0);line-height:1.5;}Hi,
    It is observed that the executor of a task changes to a different worker on 
 a different supervisor. And the computation, which is similar to wordcount,  
stops and starts over. After several tests, I find that this is caused by the 
lost of the task state during executor change. So I wonder:   
   - Why does the change happen?
   - When the change happen, does the executor use a new component(bolt) 
instance? How does the instance come into being? By constructor or 
deserialization?
   - In this case, how to persist the task state?
wangchunc...@outlook.com

Re: how to keep storm running when console ends

[Bobby Evans]

You probably want to look into using something like daemon tools to monitor the 
processes.  nohup works, but it is not really a good solution.  Storm is fail 
fast.  If we hit any exception that we don't know that we can recover from the 
process will exit on the assumption that it will be relaunched and we can 
recover from the state we have written to disk/zookeeper.  Having a tool that 
will monitor and relaunch storm is the best way to keep it up and running.


- Bobby


On Friday, August 4, 2017, 8:29:12 AM CDT, M. Aaron Bossert 
 wrote:

Try adding nohup at the beginning of each of those commands.  Nohup prevents a 
process from terminating when it's parent shell is closed.
nohup /opt/storm/apache*/bin/storm nimbus &

Sent from my iPhone
On Aug 4, 2017, at 09:08, Ethan Li  wrote:


Hi,
Storm docs recommend to use daemontools or monit which provides error recovery 
and etc.. 
I use linux "nohup" or "screen" command for simplicity. 
Ethan 
 

On Friday, August 4, 2017 7:29 AM, J.R. Pauley  wrote:
 

 this seems silly but I have not figured out how to keep nimbus, supervisor 
running after console session ends. zookeeper survives but supervisor and 
nimbus shut down when the console session ends.
I have storm 1.0.2 installed under /opt/storm and starting up 
as:/opt/storm/apache*/bin/storm nimbus&/opt/storm/apache*/bin/storm supervisor&
/opt/storm/apache*/bin/storm ui&
/opt/storm/apache*/bin/storm drpc&

I know I can add a while loop to keep the console active but there has to be a 
better way

   

Re: how to keep storm running when console ends

[M. Aaron Bossert]

Try adding nohup at the beginning of each of those commands.  Nohup prevents a 
process from terminating when it's parent shell is closed.

nohup /opt/storm/apache*/bin/storm nimbus &


Sent from my iPhone

> On Aug 4, 2017, at 09:08, Ethan Li  wrote:
> 
> Hi,
> 
> Storm docs recommend to use daemontools or monit which provides error 
> recovery and etc.. 
> 
> I use linux "nohup" or "screen" command for simplicity. 
> 
> Ethan
>  
> 
> 
> 
> On Friday, August 4, 2017 7:29 AM, J.R. Pauley  wrote:
> 
> 
> this seems silly but I have not figured out how to keep nimbus, supervisor 
> running after console session ends. zookeeper survives but supervisor and 
> nimbus shut down when the console session ends.
> 
> I have storm 1.0.2 installed under /opt/storm and starting up as:
> /opt/storm/apache*/bin/storm nimbus&
> /opt/storm/apache*/bin/storm supervisor&
> /opt/storm/apache*/bin/storm ui&
> /opt/storm/apache*/bin/storm drpc&
> 
> I know I can add a while loop to keep the console active but there has to be 
> a better way
> 
> 

Re: how to keep storm running when console ends

[Ethan Li]

Hi,
Storm docs recommend to use daemontools or monit which provides error recovery 
and etc.. 
I use linux "nohup" or "screen" command for simplicity. 
Ethan 
 

On Friday, August 4, 2017 7:29 AM, J.R. Pauley  wrote:
 

 this seems silly but I have not figured out how to keep nimbus, supervisor 
running after console session ends. zookeeper survives but supervisor and 
nimbus shut down when the console session ends.
I have storm 1.0.2 installed under /opt/storm and starting up 
as:/opt/storm/apache*/bin/storm nimbus&/opt/storm/apache*/bin/storm supervisor&
/opt/storm/apache*/bin/storm ui&
/opt/storm/apache*/bin/storm drpc&

I know I can add a while loop to keep the console active but there has to be a 
better way

   

how to keep storm running when console ends

[J.R. Pauley]

this seems silly but I have not figured out how to keep nimbus, supervisor
running after console session ends. zookeeper survives but supervisor and
nimbus shut down when the console session ends.

I have storm 1.0.2 installed under /opt/storm and starting up as:
/opt/storm/apache*/bin/storm nimbus&
/opt/storm/apache*/bin/storm supervisor&
/opt/storm/apache*/bin/storm ui&
/opt/storm/apache*/bin/storm drpc&

I know I can add a while loop to keep the console active but there has to
be a better way