On Sun, Sep 13, 2015 at 07:02:38PM +0200, Michael Scherer wrote: > Le mardi 08 septembre 2015 à 14:56 +0200, Michael Scherer a écrit : > > Le mardi 08 septembre 2015 à 11:21 +0200, Michael Scherer a écrit : > > > Hi, > > > > > > so since some nodes are stuck and likely need a reboot, I will test > > > using software watchdog on them to reboot when there is a issue as a > > > stop gap until we find what break them (but breaking is likely to happen > > > anyway, that's why we have tests) > > > > > > I will take slave21 and 22 as guinea pig, so please do not reboot them > > > if they are stuck (or if you do, tell me ). If anything weird happen on > > > them (like reboot during a test, or this kind of stuff, please tell me > > > too. > > > > > > I guess 2 to 3 weeks of tests should be enough to see if we can push > > > that to other centos 6 slaves. > > > > Seems we are missing a module on centos for the kernel support. So it > > might be less efficient. > > And slave22 didn't reboot as planned, so watchdog do not seems to work.
If a watchdog inside a VM does not work, it is most likely due to some kernel issues. You could also try to enable kdump or set some kernel options, for ideas see 'sysctl -a | grep panic'. Maybe we can setup a watchdog in Jenkins and trigger the reboot-vm job (issues a rackspace API, not from within the VM) when the watchdog does not resond? I've seen an option somewhere to configure Jenkins jobs according to a schedule... But, maybe it is easier to have SaltStack monitor and reboot VMs? The Python script for rebooting VMs and a default configuration file can be found here: https://github.com/gluster/glusterfs-patch-acceptance-tests/tree/master/rax-reboot HTH, Niels
pgpczzqIMD_6t.pgp
Description: PGP signature
_______________________________________________ Gluster-infra mailing list [email protected] http://www.gluster.org/mailman/listinfo/gluster-infra
