Hi, my HA-Cluster with two nodes fenced one on 14th of may. ha-idg-1 has been the DC, ha-idg-2 was fenced. It happened around 11:30 am. The log from the fenced one isn't really informative:
================================== 2019-05-14T11:22:09.948980+02:00 ha-idg-2 liblogging-stdlog: -- MARK -- 2019-05-14T11:28:21.548898+02:00 ha-idg-2 sshd[14269]: Accepted keyboard-interactive/pam for root from 10.35.34.70 port 59449 ssh2 2019-05-14T11:28:21.550602+02:00 ha-idg-2 sshd[14269]: pam_unix(sshd:session): session opened for user root by (uid=0) 2019-05-14T11:28:21.554640+02:00 ha-idg-2 systemd-logind[2798]: New session 15385 of user root. 2019-05-14T11:28:21.555067+02:00 ha-idg-2 systemd[1]: Started Session 15385 of user root. 2019-05-14T11:44:07.664785+02:00 ha-idg-2 systemd[1]: systemd 228 running in system mode. (+PAM -AUDIT +SELINUX -IMA +APPARMOR -SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GC Neustart !!! RYPT -GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD -IDN) 2019-05-14T11:44:07.664902+02:00 ha-idg-2 kernel: [ 0.000000] Linux version 4.12.14-95.13-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) ) #1 SMP Fri Mar 22 06:04:58 UTC 2019 (c01bf34) 2019-05-14T11:44:07.665492+02:00 ha-idg-2 systemd[1]: Detected architecture x86-64. 2019-05-14T11:44:07.665510+02:00 ha-idg-2 kernel: [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.12.14-95.13-default root=/dev/mapper/vg_local-lv_root resume=/ dev/disk/by-uuid/2849c504-2e45-4ec8-bbf8-724cf358ee25 splash=verbose showopts 2019-05-14T11:44:07.665510+02:00 ha-idg-2 systemd[1]: Set hostname to <ha-idg-2>. ================================= Node restarts at 11:44 am. The DC is more informative: ================================= 2019-05-14T11:24:05.105739+02:00 ha-idg-1 PackageKit: daemon quit 2019-05-14T11:24:05.106284+02:00 ha-idg-1 packagekitd[11617]: (packagekitd:11617): GLib-CRITICAL **: Source ID 15 was not found when attempting to remove it 2019-05-14T11:27:23.276813+02:00 ha-idg-1 liblogging-stdlog: -- MARK -- 2019-05-14T11:30:01.248803+02:00 ha-idg-1 cron[24140]: pam_unix(crond:session): session opened for user root by (uid=0) 2019-05-14T11:30:01.253150+02:00 ha-idg-1 systemd[1]: Started Session 17988 of user root. 2019-05-14T11:30:01.301674+02:00 ha-idg-1 CRON[24140]: pam_unix(crond:session): session closed for user root 2019-05-14T11:30:03.710784+02:00 ha-idg-1 kernel: [1015426.947016] tg3 0000:02:00.3 eth3: Link is down 2019-05-14T11:30:03.792500+02:00 ha-idg-1 kernel: [1015427.024779] bond1: link status definitely down for interface eth3, disabling it 2019-05-14T11:30:04.849892+02:00 ha-idg-1 hp-ams[2559]: CRITICAL: Network Adapter Link Down (Slot 0, Port 4) 2019-05-14T11:30:05.261968+02:00 ha-idg-1 kernel: [1015428.498127] tg3 0000:02:00.3 eth3: Link is up at 100 Mbps, full duplex 2019-05-14T11:30:05.261985+02:00 ha-idg-1 kernel: [1015428.498138] tg3 0000:02:00.3 eth3: Flow control is on for TX and on for RX 2019-05-14T11:30:05.261986+02:00 ha-idg-1 kernel: [1015428.498143] tg3 0000:02:00.3 eth3: EEE is disabled 2019-05-14T11:30:05.352500+02:00 ha-idg-1 kernel: [1015428.584725] bond1: link status definitely up for interface eth3, 100 Mbps full duplex 2019-05-14T11:30:05.983387+02:00 ha-idg-1 hp-ams[2559]: NOTICE: Network Adapter Link Down (Slot 0, Port 4) has been repaired 2019-05-14T11:30:10.520149+02:00 ha-idg-1 corosync[6957]: [TOTEM ] A processor failed, forming new configuration. 2019-05-14T11:30:16.524341+02:00 ha-idg-1 corosync[6957]: [TOTEM ] A new membership (192.168.100.10:1120) was formed. Members left: 1084777492 2019-05-14T11:30:16.524799+02:00 ha-idg-1 corosync[6957]: [TOTEM ] Failed to receive the leave message. failed: 1084777492 2019-05-14T11:30:16.525199+02:00 ha-idg-1 lvm[12430]: confchg callback. 0 joined, 1 left, 1 members 2019-05-14T11:30:16.525706+02:00 ha-idg-1 attrd[6967]: notice: Node ha-idg-2 state is now lost 2019-05-14T11:30:16.526143+02:00 ha-idg-1 cib[6964]: notice: Node ha-idg-2 state is now lost 2019-05-14T11:30:16.526480+02:00 ha-idg-1 attrd[6967]: notice: Removing all ha-idg-2 attributes for peer loss 2019-05-14T11:30:16.526742+02:00 ha-idg-1 cib[6964]: notice: Purged 1 peer with id=1084777492 and/or uname=ha-idg-2 from the membership cache 2019-05-14T11:30:16.527283+02:00 ha-idg-1 stonith-ng[6965]: notice: Node ha-idg-2 state is now lost 2019-05-14T11:30:16.527618+02:00 ha-idg-1 attrd[6967]: notice: Purged 1 peer with id=1084777492 and/or uname=ha-idg-2 from the membership cache 2019-05-14T11:30:16.527884+02:00 ha-idg-1 stonith-ng[6965]: notice: Purged 1 peer with id=1084777492 and/or uname=ha-idg-2 from the membership cache 2019-05-14T11:30:16.528156+02:00 ha-idg-1 corosync[6957]: [QUORUM] Members[1]: 1084777482 2019-05-14T11:30:16.528435+02:00 ha-idg-1 corosync[6957]: [MAIN ] Completed service synchronization, ready to provide service. 2019-05-14T11:30:16.548481+02:00 ha-idg-1 kernel: [1015439.782587] dlm: closing connection to node 1084777492 2019-05-14T11:30:16.555995+02:00 ha-idg-1 dlm_controld[12279]: 1015492 fence request 1084777492 pid 24568 nodedown time 1557826216 fence_all dlm_stonith 2019-05-14T11:30:16.626285+02:00 ha-idg-1 crmd[6969]: warning: Stonith/shutdown of node ha-idg-2 was not expected 2019-05-14T11:30:16.626534+02:00 ha-idg-1 dlm_stonith: stonith_api_time: Found 1 entries for 1084777492/(null): 0 in progress, 1 completed 2019-05-14T11:30:16.626731+02:00 ha-idg-1 dlm_stonith: stonith_api_time: Node 1084777492/(null) last kicked at: 1556884018 2019-05-14T11:30:16.626875+02:00 ha-idg-1 stonith-ng[6965]: notice: Client stonith-api.24568.6a9fa406 wants to fence (reboot) '1084777492' with device '(any)' 2019-05-14T11:30:16.627026+02:00 ha-idg-1 crmd[6969]: notice: State transition S_IDLE -> S_POLICY_ENGINE 2019-05-14T11:30:16.627165+02:00 ha-idg-1 crmd[6969]: notice: Node ha-idg-2 state is now lost 2019-05-14T11:30:16.627302+02:00 ha-idg-1 crmd[6969]: warning: Stonith/shutdown of node ha-idg-2 was not expected 2019-05-14T11:30:16.627439+02:00 ha-idg-1 stonith-ng[6965]: notice: Requesting peer fencing (reboot) of ha-idg-2 2019-05-14T11:30:16.627578+02:00 ha-idg-1 pacemakerd[6963]: notice: Node ha-idg-2 state is now lost ================================== One network interface is gone for a short period. But it's in a bonding device (round-robin), so the connection shouldn't be lost. Both nodes are connected directly, there is no switch in between. I manually (ifconfig eth3 down) stopped the interface afterwards several times ... nothing happened. The same with the second Interface (eth2). ??? Bernd -- Bernd Lentes Systemadministration Institut für Entwicklungsgenetik Gebäude 35.34 - Raum 208 HelmholtzZentrum münchen [email protected] phone: +49 89 3187 1241 phone: +49 89 3187 3827 fax: +49 89 3187 2294 http://www.helmholtz-muenchen.de/idg wer Fehler macht kann etwas lernen wer nichts macht kann auch nichts lernen Helmholtz Zentrum Muenchen Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH) Ingolstaedter Landstr. 1 85764 Neuherberg www.helmholtz-muenchen.de Stellv. Aufsichtsratsvorsitzender: MinDirig. Dr. Manfred Wolter Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler, Kerstin Guenther Registergericht: Amtsgericht Muenchen HRB 6466 USt-IdNr: DE 129521671 _______________________________________________ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
