Dear All,
I need some advice regarding cluster system update. I've built a cluster
in a hope of zero downtime, but unfortunately it doesn't work this way.
Configuration
- two physical machines running Centos 7 and KVM
- on each installed virtual machine with Ubuntu 16.04 (apache, mysql,
postfix, ftp, dovecot, ispfonfig...) and drbd
The issue : Before each upgrade I really want to make a snapshot. But
there's a problem.
In order to do this, I have to
- disconnect DRBD synchronization (using LCMC cluster management tool
putting one node offline)
- at the same time make a snapshot on both nodes
This means during a snapshot a cluster is down for about 15-20 minutes.
And of course if I want to revert to this snapshot I have to do it on
both nodes at the same time (downtime again). If I revert to a snapshot
only on one node sync is broken and have to reinitialize sync of
everything. Which could take days when there's about 5TB of data.
For some time I was performing apt-get update&upgrade on inactive node,
but on one occasion mariadb was upgraded and since there was no database
(mounted on other node) the upgrade made a total mess. I was lucky I had
a snapshot. Since then I always do upgrade on active node with sync
disabled. After successful upgrade I put both nodes online, wait for
sync to finish, change role, disconnect, upgrade another one, connect.
Now the main question : what are your experiences with system upgrades
in cluster environment ? How to avoid downtime ?
Best regards,
Dušan Maček
_______________________________________________
Star us on GITHUB: https://github.com/LINBIT
drbd-user mailing list
drbd-user@lists.linbit.com
https://lists.linbit.com/mailman/listinfo/drbd-user
