I also tried another scenario: instead of copying the entire storm home
directory, I only use one storm home, but different storm-local dir and ports,
which both are specified in storm.yaml, I can still create multiple
supervisors. (Of course, every time before I start a new supervisor, I have to
update the storm.yaml for different storm-local dir and ports).
You will have two supervisors writing to the same log.
I recommend creating two distinct storm home directories unless you have a good
reason to have them shared. I think the code assumes it is the only supervisor
writing in storm home.
--
Derek
On 8/22/14, 14:08, Yu, Tao wrote:
Thanks Harsha!
Just cleaned zookeeper data (stop and re-start zookeeper) and tried again, now
I can create multiple supervisors successfully!
I also tried another scenario: instead of copying the entire storm home
directory, I only use one storm home, but different storm-local dir and ports,
which both are specified in storm.yaml, I can still create multiple
supervisors. (Of course, every time before I start a new supervisor, I have to
update the storm.yaml for different storm-local dir and ports).
So my new questions are:
1) what's the best approach to create multiple supervisors on same node:
a) each supervisor has it's own storm home directory; Or
b) all supervisors have common storm home directory.
in both approaches, supervisors have its own storm-local dir and ports.
2) when start supervisor, can we specify storm to use custom configuration
(.yaml)? For example, like:
$bin/storm supervisor --config conf/myConfig.yaml
Seems like storm will always use conf/storm.yaml, and I do not see any
document mentions about specifying custom config file.
Thanks,
-Tao
-----Original Message-----
From: Harsha [mailto:[email protected]]
Sent: Friday, August 22, 2014 12:57 PM
To: [email protected]
Subject: Re: Create multiple supervisors on same node
Tao,
I tried the above steps I am able to run two supervisors on the
same node. Did you check the logs for supervisor under storm2. If
it didn't created a local_dir/storm dir than your supervisor
daemon might not be running. check for logs if there are any
errors.
-Harsha
On Fri, Aug 22, 2014, at 09:20 AM, Yu, Tao wrote:
Thanks Harsha!
I tried your way, and here is what I have (major parts) in my storm.yaml:
storm.local.dir: "/opt/grid/tao/storm/storm-0.8.2/local_data/storm"
supervisor.slots.ports:
- 6700
- 6701
1) I created the 1st supervisor, and I can see specified sub-folder
"local_data/storm/supervisor" was created under"
opt/grid/tao/storm/storm-0.8.2". That's OK!
2) then I copied the entire "storm-0.8.2" folder to a new "storm2"
("/opt/grid/tao/storm/storm2")
3) delete the sub-folder "local_data" under "storm2"
4) updated the storm.yaml under "storm2" with below change:
storm.local.dir: "/opt/grid/tao/storm/storm2/local_data/storm"
supervisor.slots.ports:
- 8700
- 8701
5) under "storm2", create a new supervisor.
Then the new supervisor still has the 1st supervisor's ID. And under
"storm2", the sub-folder "local_data/storm" was not created.
Does storm still use the 1st storm home directory
("storm/storm-0.8.2") "local_data" folder?
Thanks,
-Tao
-----Original Message-----
From: Harsha [mailto:[email protected]]
Sent: Friday, August 22, 2014 11:28 AM
To: [email protected]
Subject: Re: Create multiple supervisors on same node
Tao,
you need to delete the storm-local dir under your copied over storm
dir ( "storm2"). Otherwise it will still pick up the same
supervisor-id.
-Harsha
On Fri, Aug 22, 2014, at 08:16 AM, Yu, Tao wrote:
Thanks Derek!
I tried your suggestion, copied the entire storm home directory
(which, in my case, is "storm-0.8.2") to a new directory "storm2",
then in "storm2" directory, I changed the conf/storm.yaml with
different ports, and tried to create a new supervisor. Still, got
the same supervisor ID as the 1st one (which I created from "storm-0.8.2"
directory).
Did I do anything incorrectly?
-Tao
-----Original Message-----
From: Derek Dagit [mailto:[email protected]]
Sent: Friday, August 22, 2014 11:01 AM
To: [email protected]
Subject: Re: Create multiple supervisors on same node
The two supervisors are sharing the same state, and that is how they
get the same randomly-generated ID.
If I recall correctly, the default state directory is created in the
current working directory of the process, so that is whatever
directory you happen to be in when you start the supervisor.
I think probably a good thing to do is copy the entire storm home
directory, change the storm.yaml in the copy to be configured with
different ports as you tried, and make sure to cd into the
appropriate directory when you launch the supervisor.
--
Derek
On 8/22/14, 9:49, Yu, Tao wrote:
Hi all,
Anyone knows what's the requirement to generate multiple supervisors on the same node
(for same topology)? I can create the 1st supervisor, then I update the
"supervisor.slots.ports" to different ports, and tried to create the 2nd
supervisor on same node, it ends up creating a new supervisor but with same supervisor ID
as the 1st one, so it still only has one supervisor on that node and storm UI shows 1
supervisor as well. Any suggestion on how to create the 2nd, 3rd supervisor on the same
node?
Any help is appreciated!
thanks,
-Tao
