On 10.09.20 06:58, Alexandre DERUMIER wrote: > Thanks Thomas for the investigations. > > I'm still trying to reproduce... > I think I have some special case here, because the user of the forum with 30 > nodes had corosync cluster split. (Note that I had this bug 6 months ago,when > shuting down a node too, and the only way was stop full stop corosync on all > nodes, and start corosync again on all nodes). > > > But this time, corosync logs looks fine. (every node, correctly see node2 > down, and see remaning nodes) > > surviving node7, was the only node with HA, and LRM didn't have enable > watchog (I don't have found any log like "pve-ha-lrm: watchdog active" for > the last 6months on this nodes > > > So, the timing was: > > 10:39:05 : "halt" command is send to node2 > 10:39:16 : node2 is leaving corosync / halt -> every node is seeing it and > correctly do a new membership with 13 remaining nodes > > ...don't see any special logs (corosync,pmxcfs,pve-ha-crm,pve-ha-lrm) after > the node2 leaving. > But they are still activity on the server, pve-firewall is still logging, vms > are running fine > > > between 10:40:25 - 10:40:34 : watchdog reset nodes, but not node7. > > -> so between 70s-80s after the node2 was done, so I think that watchdog-mux > was still running fine until that. > (That's sound like lrm was stuck, and client_watchdog_timeout have expired > in watchdog-mux)
as said, if the other nodes where not using HA, the watchdog-mux had no client which could expire. > > 10:40:41 node7, loose quorum (as all others nodes have reset), > 10:40:50: node7 crm/lrm finally log. > > Sep 3 10:40:50 m6kvm7 pve-ha-crm[16196]: got unexpected error - error during > cfs-locked 'domain-ha' operation: no quorum! > Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds) > Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds) above lines also indicate very high load. Do you have some monitoring which shows the CPU/IO load before/during this event? > Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: lost lock 'ha_manager_lock - cfs > lock update failed - Permission denied > Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: lost lock 'ha_agent_m6kvm7_lock - > cfs lock update failed - Permission denied > > > > So, I really think that something have stucked lrm/crm loop, and watchdog was > not resetted because of that. > _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
