Re: [ClusterLabs] Ocassionally IPaddr2 resource fails to start
Hello and sorry for soo late response of mine, I somehow missed your answer. Sure let me share a bit of useful information on the count. First of all the system specific things are: - Hypervisor is a usual VMware product - VSphere - VMs OS is: Ubuntu 18.04 LTS - Pacemaker is of version: 1.1.18-0ubuntu1.1 And yes it's IProute, that has a version - 4.15.0-2ubuntu1 To be mentioned that after I moved to another way of handling this (with set failure-timeout ) I haven't seen any errors so far, on-fail action still remains "restart". But it's obvious, failure-timeout just clears all fail counters for me, so I don't see any fails now. Another thing to be mentioned, that monitor functionality for IPaddr2 resource was failing in the years past as well, I just didn't pay much attention on that. That time VM machines under my control were working over Ubuntu 14.04 and hypervisor was - Proxmox of the branch 5+ (cannot exactly remember the version, perhaps that was 5.4+). For one this could be a critical case indeed, since sometimes an absence of IP address (for a certain DB for e.g. with loading of hundreds of thousands SQL requests) can lead to a huge out age. I don't have the first idea of how to investigate this further. But, I have a staging setup where my hands are not tied, so let me know if we can research something. And have a nice day! On Mon, Oct 7, 2019 at 7:21 PM Jan Pokorný wrote: > Donat, > > On 07/10/19 09:24 -0500, Ken Gaillot wrote: > > If this always happens when the VM is being snapshotted, you can put > > the cluster in maintenance mode (or even unmanage just the IP > > resource) while the snapshotting is happening. I don't know of any > > reason why snapshotting would affect only an IP, though. > > it might be interesting if you could share the details to grow the > shared knowledge and experience in case there are some instances of > these problems reported in the future. > > In particular, it'd be interesting to hear: > > - hypervisor > > - VM OS + if plain oblivious to running virtualized, > or "the optimal arrangement" (e.g., specialized drivers, virtio, > "guest additions", etc.) > > (I think IPaddr2 is iproute2-only, hence in turn, VM OS must be Linux) > > Of course, there might be more specific things to look at if anyone > here is an expert with particular hypervisor technology and the way > the networking works with it (no, not me at all). > > -- > Poki > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ -- Best regards, Donat Zenichev ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Ocassionally IPaddr2 resource fails to start
Donat, On 07/10/19 09:24 -0500, Ken Gaillot wrote: > If this always happens when the VM is being snapshotted, you can put > the cluster in maintenance mode (or even unmanage just the IP > resource) while the snapshotting is happening. I don't know of any > reason why snapshotting would affect only an IP, though. it might be interesting if you could share the details to grow the shared knowledge and experience in case there are some instances of these problems reported in the future. In particular, it'd be interesting to hear: - hypervisor - VM OS + if plain oblivious to running virtualized, or "the optimal arrangement" (e.g., specialized drivers, virtio, "guest additions", etc.) (I think IPaddr2 is iproute2-only, hence in turn, VM OS must be Linux) Of course, there might be more specific things to look at if anyone here is an expert with particular hypervisor technology and the way the networking works with it (no, not me at all). -- Poki pgphleQBrgswu.pgp Description: PGP signature ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] Ocassionally IPaddr2 resource fails to start
Hi there! I've got a tricky case, when my IpAddr2 resource fails to start with literally no-reason: "IPSHARED_monitor_2 on my-master-1 'not running' (7): call=11, status=complete, exitreason='', last-rc-change='Wed Sep 4 06:08:07 2019', queued=0ms, exec=0ms" Resource IpAddr2 managed to fix itself and continued to work properly further after that. What I've done after, was setting 'Failure-timeout=900' seconds for my IpAddr2 resource, to prevent working of the resource on a node where it fails. I also set the 'migration-threshold=2' so IpAddr2 can fail only 2 times, and goes to a Slave side after that. Meanwhile Master gets banned for 900 seconds. After 900 seconds cluster tries to start IpAddr2 again at Master, in case it's ok, fail counter gets cleared. That's how I avoid appearing of the error I mentioned above. I tried to get so hard, why this can happen, but still no idea on the count. Any clue how to find a reason? And another question, can snap-shoting of VM machines have any impact on such? And my configurations: --- node 01: my-master-1 node 02: my-master-2 primitive IPSHARED IPaddr2 \ params ip=10.10.10.5 nic=eth0 cidr_netmask=24 \ meta migration-threshold=2 failure-timeout=900 target-role=Started \ op monitor interval=20 timeout=60 on-fail=restart location PREFER_MASTER IPSHARED 100: my-master-1 property cib-bootstrap-options: \ have-watchdog=false \ dc-version=1.1.18-2b07d5c5a9 \ cluster-infrastructure=corosync \ cluster-name=wall \ cluster-recheck-interval=5s \ start-failure-is-fatal=false \ stonith-enabled=false \ no-quorum-policy=ignore \ last-lrm-refresh=1554982967 --- Thanks in advance! -- -- BR, Donat Zenichev ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
