Re: replacing one zookeeper machine with brand new machine

[Washko, Daniel]

There is a restart/stop button on the explorer page. If you turn off automatic 
restarts you can stop an instance. To remove the instance I believe there is a 
list of zookeeper nodes on the Config page. If you edit that and remove the 
zookeeper instances you are terminating and commit the change it should push it 
out. Once the restarts complete and all is green, you should be able to safely 
terminate the removed nodes.

-- 
Daniel S Washko
Solutions Architect



Phone: 757 667 1463
dwas...@gannett.comgannett.com 
On 2/5/18, 3:27 PM, "Check Peck"  wrote:

Is there an option to remove a zookeeper node from exhibitor? I am not sure
it is there I guess.

On Mon, Feb 5, 2018 at 10:21 AM, Washko, Daniel  wrote:

> The steps are the same whether Exhibitor is in the mix or not. Exhibitor
> will take care of management, though. I would recommend backing up the 
data
> in your Zookeeper ensemble just to be safe.
>
> 1) Spin up a new zookeeper and configure it to use exhibitor.
> 2) Let exhibitor bring it into the ensemble.
> 3) Use exhibitor to remove the old node.
> 4) Terminate the old node when exhibitor says it is no longer in the
> ensemble; or it is down.
>
> It has been a few years since I have worked with Exhibitor. It should
> automatically pull the new node into the ensemble. I believe there is an
> option to remove a node. You will be presented with a choice on how you
> want to initiate the changes - a rolling restart of restart all at once. I
> would recommend a rolling restart if you want to keep the ensemble live
> while you make the changes.
>
> If you have a problem with removing one of the nodes, you can edit the
> node list in exhibitor, remove that node, and save the configuration.
> Again, this will prompt for a rolling restart or parallel restart.
>
> Without exhibitor these are the steps I follow:
>
> 1) Backup the data
> 2) Spin up a new zookeeper
> 3) Identify the master
> 4) Alter the configuration on each zookeeper to add the new node and to
> add the other nodes to the new zookeeper. Be aware of the zookeeper ID, it
> has to be unique.
> 5) Perform a rolling restart of each node with the master last.
> 6) Verify the new master and that the data stored in zookeeper has
> migrated successfully to the new node.
> 7) Remove the old node from each config.
> 8) Stop zookeeper on the old node and do a rolling restart of the
> remaining zookeepr nodes with the master last.
> 9) Terminate the old node.
>
> --
> Daniel S Washko
> Solutions Architect
>
>
>
> Phone: 757 667 1463
> dwas...@gannett.comgannett.com 
>
> On 2/2/18, 3:20 PM, "Check Peck"  wrote:
>
> I have a zookeeper ensemble of 5 servers and I am using exhibitor on
> top of
> it. And I installed exhibitor and setup zookeeper by following this
> link:
>
> https://github.com/soabase/exhibitor/wiki/Building-Exhibitor
>
> Below is how all my zookeeper machines are setup in exhibitor
>
> S:1:machineA,
> S:2:machineB,
> S:3:machineC,
> S:4:machineD,
> S:5:machineE
>
> Now for some reasons, I need to replace "machineE" with brand new
> "machineF". What is the best way by which I can safely remove one
> machine
> and replace it with new machine?
>
>
>

Re: replacing one zookeeper machine with brand new machine

[Check Peck]

Is there an option to remove a zookeeper node from exhibitor? I am not sure
it is there I guess.

On Mon, Feb 5, 2018 at 10:21 AM, Washko, Daniel  wrote:

> The steps are the same whether Exhibitor is in the mix or not. Exhibitor
> will take care of management, though. I would recommend backing up the data
> in your Zookeeper ensemble just to be safe.
>
> 1) Spin up a new zookeeper and configure it to use exhibitor.
> 2) Let exhibitor bring it into the ensemble.
> 3) Use exhibitor to remove the old node.
> 4) Terminate the old node when exhibitor says it is no longer in the
> ensemble; or it is down.
>
> It has been a few years since I have worked with Exhibitor. It should
> automatically pull the new node into the ensemble. I believe there is an
> option to remove a node. You will be presented with a choice on how you
> want to initiate the changes - a rolling restart of restart all at once. I
> would recommend a rolling restart if you want to keep the ensemble live
> while you make the changes.
>
> If you have a problem with removing one of the nodes, you can edit the
> node list in exhibitor, remove that node, and save the configuration.
> Again, this will prompt for a rolling restart or parallel restart.
>
> Without exhibitor these are the steps I follow:
>
> 1) Backup the data
> 2) Spin up a new zookeeper
> 3) Identify the master
> 4) Alter the configuration on each zookeeper to add the new node and to
> add the other nodes to the new zookeeper. Be aware of the zookeeper ID, it
> has to be unique.
> 5) Perform a rolling restart of each node with the master last.
> 6) Verify the new master and that the data stored in zookeeper has
> migrated successfully to the new node.
> 7) Remove the old node from each config.
> 8) Stop zookeeper on the old node and do a rolling restart of the
> remaining zookeepr nodes with the master last.
> 9) Terminate the old node.
>
> --
> Daniel S Washko
> Solutions Architect
>
>
>
> Phone: 757 667 1463
> dwas...@gannett.comgannett.com 
>
> On 2/2/18, 3:20 PM, "Check Peck"  wrote:
>
> I have a zookeeper ensemble of 5 servers and I am using exhibitor on
> top of
> it. And I installed exhibitor and setup zookeeper by following this
> link:
>
> https://github.com/soabase/exhibitor/wiki/Building-Exhibitor
>
> Below is how all my zookeeper machines are setup in exhibitor
>
> S:1:machineA,
> S:2:machineB,
> S:3:machineC,
> S:4:machineD,
> S:5:machineE
>
> Now for some reasons, I need to replace "machineE" with brand new
> "machineF". What is the best way by which I can safely remove one
> machine
> and replace it with new machine?
>
>
>

Re: replacing one zookeeper machine with brand new machine

[Washko, Daniel]

The steps are the same whether Exhibitor is in the mix or not. Exhibitor will 
take care of management, though. I would recommend backing up the data in your 
Zookeeper ensemble just to be safe.

1) Spin up a new zookeeper and configure it to use exhibitor. 
2) Let exhibitor bring it into the ensemble. 
3) Use exhibitor to remove the old node.
4) Terminate the old node when exhibitor says it is no longer in the ensemble; 
or it is down.

It has been a few years since I have worked with Exhibitor. It should 
automatically pull the new node into the ensemble. I believe there is an option 
to remove a node. You will be presented with a choice on how you want to 
initiate the changes - a rolling restart of restart all at once. I would 
recommend a rolling restart if you want to keep the ensemble live while you 
make the changes. 

If you have a problem with removing one of the nodes, you can edit the node 
list in exhibitor, remove that node, and save the configuration. Again, this 
will prompt for a rolling restart or parallel restart.

Without exhibitor these are the steps I follow:

1) Backup the data
2) Spin up a new zookeeper
3) Identify the master
4) Alter the configuration on each zookeeper to add the new node and to add the 
other nodes to the new zookeeper. Be aware of the zookeeper ID, it has to be 
unique.
5) Perform a rolling restart of each node with the master last. 
6) Verify the new master and that the data stored in zookeeper has migrated 
successfully to the new node. 
7) Remove the old node from each config.
8) Stop zookeeper on the old node and do a rolling restart of the remaining 
zookeepr nodes with the master last.
9) Terminate the old node.

-- 
Daniel S Washko
Solutions Architect



Phone: 757 667 1463
dwas...@gannett.comgannett.com 

On 2/2/18, 3:20 PM, "Check Peck"  wrote:

I have a zookeeper ensemble of 5 servers and I am using exhibitor on top of
it. And I installed exhibitor and setup zookeeper by following this link:

https://github.com/soabase/exhibitor/wiki/Building-Exhibitor

Below is how all my zookeeper machines are setup in exhibitor

S:1:machineA,
S:2:machineB,
S:3:machineC,
S:4:machineD,
S:5:machineE

Now for some reasons, I need to replace "machineE" with brand new
"machineF". What is the best way by which I can safely remove one machine
and replace it with new machine?

Re: replacing one zookeeper machine with brand new machine

[Check Peck]

 I have a zookeeper ensemble of 5 servers and I am using exhibitor on top
of it. And I installed exhibitor and setup zookeeper by following this link:

https://github.com/soabase/exhibitor/wiki/Building-Exhibitor

Below is how all my zookeeper machines are setup in exhibitor

S:1:machineA,
S:2:machineB,
S:3:machineC,
S:4:machineD,
S:5:machineE

Now for some reasons, I need to replace "machineE" with brand new
"machineF". What is the best way by which I can safely remove one machine
and replace it with new machine?

replacing one zookeeper machine with brand new machine

[Check Peck]

I have a zookeeper ensemble of 5 servers and I am using exhibitor on top of
it. And I installed exhibitor and setup zookeeper by following this link:

https://github.com/soabase/exhibitor/wiki/Building-Exhibitor

Below is how all my zookeeper machines are setup in exhibitor

S:1:machineA,
S:2:machineB,
S:3:machineC,
S:4:machineD,
S:5:machineE

Now for some reasons, I need to replace "machineE" with brand new
"machineF". What is the best way by which I can safely remove one machine
and replace it with new machine?