Re: [ClusterLabs] OS Patching Process

[Dmitri Maziuk]

On 2016-11-24 10:41, Toni Tschampke wrote:

We recently did an upgrade for our cluster nodes from Wheezy to Jessie.


IIRC it's the MIT CS joke that they have clusters whose uptime goes way 
back past the manufacturing date of any/every piece of hardware they're 
running on. They aren't linux-ha clusters but there's no reason why that 
shouldn't be doable with linux-ha.


Dima


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] OS Patching Process

[Toni Tschampke]

We recently did an upgrade for our cluster nodes from Wheezy to Jessie. 
To reduce possible problems with drbd we first updated the kernel to the 
backport version, which is the same as the stable kernel in Jessie.
After that drbd versions matched while one node was on whezzy and one 
node already was on Jessie.


While holding the cluster in maintenance we could safely upgrade the 
second node and then going through the different configuration/syntax 
changes from older corosync/pacemaker versions to the backport versions 
from Jessie.


Since the version jump was big from corosync 1 & pacemaker 1.1.7 to 
corosync 2 & pacemaker 1.1.15 there was quite some work dealing with 
upgraded configurations but cluster works again as expected.


-Toni

--
Mit freundlichen Grüßen

Toni Tschampke | t...@halle.it
bcs kommunikationslösungen
Inh. Dipl. Ing. Carsten Burkhardt
Harz 51 | 06108 Halle (Saale) | Germany
tel +49 345 29849-0 | fax +49 345 29849-22
www.b-c-s.de | www.halle.it | www.wivewa.de


EINFACH ADRESSEN, TELEFONATE UND DOKUMENTE VERWALTEN - MIT WIVEWA -
IHREM WISSENSVERWALTER FUER IHREN BETRIEB!

Weitere Informationen erhalten Sie unter www.wivewa.de

Am 22.11.2016 um 17:35 schrieb Jason A Ramsey:

Can anyone recommend a bulletproof process for OS patching a pacemaker
cluster that manages a drbd mirror (with LVM on top of the drbd and luns
defined for an iscsi target cluster if that matters)? Any time I’ve
tried to mess with the cluster, it seems like I manage to corrupt my
drbd filesystem, and now that I have actual data on the thing, that’s
kind of a scary proposition. Thanks in advance!



--



*[ jR ]*



  /there is no path to greatness; greatness is the path/



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] OS Patching Process

[AP]

On Tue, Nov 22, 2016 at 05:22:21PM +, Jason A Ramsey wrote:
> I’ve done the opposite:
> 
> lvm on top of drbd -> iscsi lun
> 
> but I’m not trying to resize anything. I just want to patch the OS
> of the nodes and reboot them in sequence without breaking things (and,
> preferably, without taking the cluster offline).

Are you ensuring that nothing is using that node's drbd before reboot?
And if so, how?

AP

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] OS Patching Process

[Jason A Ramsey]

I’ve done the opposite:

lvm on top of drbd -> iscsi lun

but I’m not trying to resize anything. I just want to patch the OS of the nodes 
and reboot them in sequence without breaking things (and, preferably, without 
taking the cluster offline).

--
 
[ jR ]

  there is no path to greatness; greatness is the path

On 11/22/16, 11:47 AM, "emmanuel segura"  wrote:

I been using this mode: iscsi_disks -> lvm volume ->
drbd_on_top_of_lvm -> filesystem

resize: add_one_iscsi_device_to_every_cluster_node_first ->
now_add_device_the_volume_group_on_every_cluster_node ->
now_resize_the_volume_on_every_cluster_node : now you have every
cluster with the same logical volume size, now you can resize drbd and
filesystem on the active node

2016-11-22 17:35 GMT+01:00 Jason A Ramsey :
> Can anyone recommend a bulletproof process for OS patching a pacemaker
> cluster that manages a drbd mirror (with LVM on top of the drbd and luns
> defined for an iscsi target cluster if that matters)? Any time I’ve tried 
to
> mess with the cluster, it seems like I manage to corrupt my drbd 
filesystem,
> and now that I have actual data on the thing, that’s kind of a scary
> proposition. Thanks in advance!
>
>
>
> --
>
>
>
> [ jR ]
>
>
>
>   there is no path to greatness; greatness is the path
>
>
> ___
> Users mailing list: Users@clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>



-- 
  .~.
  /V\
 //  \\
/(   )\
^`~'^

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] OS Patching Process

[emmanuel segura]

I been using this mode: iscsi_disks -> lvm volume ->
drbd_on_top_of_lvm -> filesystem

resize: add_one_iscsi_device_to_every_cluster_node_first ->
now_add_device_the_volume_group_on_every_cluster_node ->
now_resize_the_volume_on_every_cluster_node : now you have every
cluster with the same logical volume size, now you can resize drbd and
filesystem on the active node

2016-11-22 17:35 GMT+01:00 Jason A Ramsey :
> Can anyone recommend a bulletproof process for OS patching a pacemaker
> cluster that manages a drbd mirror (with LVM on top of the drbd and luns
> defined for an iscsi target cluster if that matters)? Any time I’ve tried to
> mess with the cluster, it seems like I manage to corrupt my drbd filesystem,
> and now that I have actual data on the thing, that’s kind of a scary
> proposition. Thanks in advance!
>
>
>
> --
>
>
>
> [ jR ]
>
>
>
>   there is no path to greatness; greatness is the path
>
>
> ___
> Users mailing list: Users@clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>



-- 
  .~.
  /V\
 //  \\
/(   )\
^`~'^

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] OS Patching Process

[Dmitri Maziuk]

On 2016-11-22 10:35, Jason A Ramsey wrote:

Can anyone recommend a bulletproof process for OS patching a pacemaker
cluster that manages a drbd mirror (with LVM on top of the drbd and luns
defined for an iscsi target cluster if that matters)? Any time I’ve
tried to mess with the cluster, it seems like I manage to corrupt my
drbd filesystem, and now that I have actual data on the thing, that’s
kind of a scary proposition. Thanks in advance!


+1

I managed to clearly standby/unstandby mine a few times initially -- 
otherwise I wouldn't have put it in production -- but on the last 
several reboots DRBD filesystem just wouldn't unmount. Never corrupted 
anything but it's still a serious PITA. Especially with a couple of 
haresources pairs right next to it switching over perfectly every time.


Insult to injury, the RA start spewing "can't unmount, somebody's 
holding open" messages to the console at such rate that it is impossible 
to login and try lsof, fuser, or anything.


Dima


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] OS Patching Process

[Jason A Ramsey]

Can anyone recommend a bulletproof process for OS patching a pacemaker cluster 
that manages a drbd mirror (with LVM on top of the drbd and luns defined for an 
iscsi target cluster if that matters)? Any time I’ve tried to mess with the 
cluster, it seems like I manage to corrupt my drbd filesystem, and now that I 
have actual data on the thing, that’s kind of a scary proposition. Thanks in 
advance!

--

[ jR ]

  there is no path to greatness; greatness is the path
___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org