Thanks Harsha!
Just cleaned zookeeper data (stop and re-start zookeeper) and tried again, now
I can create multiple supervisors successfully!
I also tried another scenario: instead of copying the entire storm home
directory, I only use one storm home, but different storm-local dir and ports,
which both are specified in storm.yaml, I can still create multiple
supervisors. (Of course, every time before I start a new supervisor, I have to
update the storm.yaml for different storm-local dir and ports).
So my new questions are:
1) what's the best approach to create multiple supervisors on same node:
a) each supervisor has it's own storm home directory; Or
b) all supervisors have common storm home directory.
in both approaches, supervisors have its own storm-local dir and ports.
2) when start supervisor, can we specify storm to use custom configuration
(.yaml)? For example, like:
$bin/storm supervisor --config conf/myConfig.yaml
Seems like storm will always use conf/storm.yaml, and I do not see any
document mentions about specifying custom config file.
Thanks,
-Tao
-----Original Message-----
From: Harsha [mailto:[email protected]]
Sent: Friday, August 22, 2014 12:57 PM
To: [email protected]
Subject: Re: Create multiple supervisors on same node
Tao,
I tried the above steps I am able to run two supervisors on the
same node. Did you check the logs for supervisor under storm2. If
it didn't created a local_dir/storm dir than your supervisor
daemon might not be running. check for logs if there are any
errors.
-Harsha
On Fri, Aug 22, 2014, at 09:20 AM, Yu, Tao wrote:
> Thanks Harsha!
>
> I tried your way, and here is what I have (major parts) in my storm.yaml:
>
> storm.local.dir: "/opt/grid/tao/storm/storm-0.8.2/local_data/storm"
> supervisor.slots.ports:
> - 6700
> - 6701
>
> 1) I created the 1st supervisor, and I can see specified sub-folder
> "local_data/storm/supervisor" was created under "
> opt/grid/tao/storm/storm-0.8.2". That's OK!
>
> 2) then I copied the entire "storm-0.8.2" folder to a new "storm2"
> ("/opt/grid/tao/storm/storm2")
>
> 3) delete the sub-folder "local_data" under "storm2"
>
> 4) updated the storm.yaml under "storm2" with below change:
>
> storm.local.dir: "/opt/grid/tao/storm/storm2/local_data/storm"
> supervisor.slots.ports:
> - 8700
> - 8701
>
> 5) under "storm2", create a new supervisor.
>
> Then the new supervisor still has the 1st supervisor's ID. And under
> "storm2", the sub-folder "local_data/storm" was not created.
>
> Does storm still use the 1st storm home directory
> ("storm/storm-0.8.2") "local_data" folder?
>
> Thanks,
> -Tao
>
> -----Original Message-----
> From: Harsha [mailto:[email protected]]
> Sent: Friday, August 22, 2014 11:28 AM
> To: [email protected]
> Subject: Re: Create multiple supervisors on same node
>
> Tao,
> you need to delete the storm-local dir under your copied over storm
> dir ( "storm2"). Otherwise it will still pick up the same
> supervisor-id.
> -Harsha
>
> On Fri, Aug 22, 2014, at 08:16 AM, Yu, Tao wrote:
> > Thanks Derek!
> >
> > I tried your suggestion, copied the entire storm home directory
> > (which, in my case, is "storm-0.8.2") to a new directory "storm2",
> > then in "storm2" directory, I changed the conf/storm.yaml with
> > different ports, and tried to create a new supervisor. Still, got
> > the same supervisor ID as the 1st one (which I created from "storm-0.8.2"
> > directory).
> >
> > Did I do anything incorrectly?
> >
> > -Tao
> >
> > -----Original Message-----
> > From: Derek Dagit [mailto:[email protected]]
> > Sent: Friday, August 22, 2014 11:01 AM
> > To: [email protected]
> > Subject: Re: Create multiple supervisors on same node
> >
> > The two supervisors are sharing the same state, and that is how they
> > get the same randomly-generated ID.
> >
> > If I recall correctly, the default state directory is created in the
> > current working directory of the process, so that is whatever
> > directory you happen to be in when you start the supervisor.
> >
> > I think probably a good thing to do is copy the entire storm home
> > directory, change the storm.yaml in the copy to be configured with
> > different ports as you tried, and make sure to cd into the
> > appropriate directory when you launch the supervisor.
> >
> > --
> > Derek
> >
> > On 8/22/14, 9:49, Yu, Tao wrote:
> > > Hi all,
> > >
> > > Anyone knows what's the requirement to generate multiple supervisors on
> > > the same node (for same topology)? I can create the 1st supervisor, then
> > > I update the "supervisor.slots.ports" to different ports, and tried to
> > > create the 2nd supervisor on same node, it ends up creating a new
> > > supervisor but with same supervisor ID as the 1st one, so it still only
> > > has one supervisor on that node and storm UI shows 1 supervisor as well.
> > > Any suggestion on how to create the 2nd, 3rd supervisor on the same node?
> > >
> > > Any help is appreciated!
> > >
> > > thanks,
> > > -Tao
> > >
