You should check the nimbus logs as they may reveal the actual problem.
On Thu, May 12, 2016 at 12:46 PM, Julián Bermejo Ferreiro | BEEVA <
[email protected]> wrote:
> Hi!
>
> I need your help.
>
> I'm ugrading our Storm production cluster from 0.9.4 to 1.0.1 version.
>
> All the 0.9.4 Storm procceses are running under supervision with
> Monit-5.12.2. In this case, all the start-scripts that Monit use exec
> NImbus, Supervisor and UI daemons in *background* (&).
>
> For example, the start-supervisor.sh only contains:
>
> /usr/bin/nohup ${STORM_HOME}/bin/storm supervisor &
>
> This config is strongly probed, actually is running on our production
> environment for almost 2 years.
>
> When I try the same configuration with Storm 1.0.1 and Monit 5.17.1, I
> think that the system doesn't run properly. Although all the start scripts
> are able to set up Nimbus, UI and Supervisor by themselves, when they're
> executed by Monit it crash. Monit is not able to raise none of them.
>
> I have tried by modifying start-scripts in order to set up all the Storm
> daemons in *foreground*, for example:
>
> /usr/bin/nohup ${STORM_HOME}/bin/storm supervisor
>
> In this case, Monit daemon is able to restart all of them but always in
> the second round. During the first attempt, all of them die because of
> timeout.
>
> These are Monit logs:
>
> For 1st attempt:
> error : 'nimbus' failed to start (exit status -1) --
> ....../start-nimbus.sh: Program timed out .....
> info : 'nimbus' restart action failed
>
> For 2nd attempt:
> info : 'nimbus' process is running after previous exec error (slow
> starting or manually recovered?)
>
> Although finally all procceses are running when I force failures (kill
> dash nines), the machine remains a "defunct" procces due to the first
> failed attempt.
>
> Is this problem related with the use of the conf/storm.py file?? (Of
> course I have python 2.6.6 and jdk 1.8 running on the machnies and all
> procceses run OK without Monit)
>
> Should I run NImbus, Supervisor and UI in background or in foreground?
>
> I don't know why is this happening, but I would definetly like to run a
> Storm 1.0.1 cluster as strong & stable as our Storm 0.9.4 cluster is.
>
> Could you help me guys?
>
> Thank you sooo much in advance!
>
>
> *JULIÁN BERMEJO FERREIRO*
> *Departamento de Tecnología *
> *[email protected] <[email protected]>*
> <http://www.beeva.com/>
>
>
>
>
--
Regards,
Abhishek Agarwal
