Hi, today we've got an alert that some resources on three nodes become to Outdated state. Nothing was changed, issue occurred suddenly and is currently persisting.
drbd version: 9.0.19-1 kernel version: 4.15.18-12-pve The weird thing is that some resources working fine on same nodes, but some of them dont. I've tried to run drbdadm disconnect && drbdadm connect for all failed resources on all three nodes, but it didn't help much. ifdown/ifup for data network interface restart didn't help too. TCP-ports are open, but pve1 and pve3 resets the connection immediately. What's happening and how can we resolve this? Thank you! I attach the logs for one resource from three nodes. It was Primary on pve2 and Secondary on pve1 amd pve3 nodes. root@pve2:~# drbdadm status pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f role:Primary disk:UpToDate pve1 connection:Connecting pve3 connection:Connecting root@pve2:~# dmesg -T | grep pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f: Preparing cluster-wide state change 1999068598 (2->3 496/16) [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f: State change 1999068598: primary_nodes=4, weak_nodes=FFFFFFFFFFFFFFF9 [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f: Committing cluster-wide state change 1999068598 (0ms) [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve1: conn( Connected -> Disconnecting ) peer( Secondary -> Unknown ) [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f/0 drbd2069 pve1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off ) [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve1: ack_receiver terminated [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve1: Terminating ack_recv thread [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve1: Connection closed [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve1: conn( Disconnecting -> StandAlone ) [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve1: Terminating receiver thread [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f: Preparing cluster-wide state change 303080422 (2->1 496/16) [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f: State change 303080422: primary_nodes=4, weak_nodes=FFFFFFFFFFFFFFFB [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve3: Cluster is now split [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f: Committing cluster-wide state change 303080422 (0ms) [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve3: conn( Connected -> Disconnecting ) peer( Secondary -> Unknown ) [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f/0 drbd2069 pve3: pdsk( UpToDate -> DUnknown ) repl( Established -> Off ) [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve3: ack_receiver terminated [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve3: Terminating ack_recv thread [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve3: Connection closed [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve3: conn( Disconnecting -> StandAlone ) [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve3: Terminating receiver thread [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f/0 drbd2069: rs_discard_granularity feature disabled [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve1: conn( StandAlone -> Unconnected ) [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve1: Starting receiver thread (from drbd_w_pvc-0c26 [16956]) [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve1: conn( Unconnected -> Connecting ) [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve3: conn( StandAlone -> Unconnected ) [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve3: Starting receiver thread (from drbd_w_pvc-0c26 [16956]) [Fri Oct 25 14:47:44 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve3: conn( Unconnected -> Connecting ) [Fri Oct 25 14:47:45 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f/0 drbd2069: new current UUID: 0407098B236D1403 weak: FFFFFFFFFFFFFFFB ... root@pve1:~# drbdadm status pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f role:Secondary disk:Outdated pve2 connection:Connecting pve3 connection:Connecting root@pve1:~# dmesg -T | grep pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f [Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: Preparing remote state change 1999068598 [Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: Committing remote state change 1999068598 (primary_nodes=4) [Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: conn( Connected -> TearDown ) peer( Primary -> Unknown ) [Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f/0 drbd2069: disk( UpToDate -> Outdated ) [Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f/0 drbd2069 pve2: pdsk( UpToDate -> DUnknown ) repl( Established -> Off ) [Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: ack_receiver terminated [Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: Terminating ack_recv thread [Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: Restarting sender thread [Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: Connection closed [Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: conn( TearDown -> Unconnected ) [Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: Restarting receiver thread [Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: conn( Unconnected -> Connecting ) [Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve3: Preparing remote state change 303080422 [Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve3: Committing remote state change 303080422 (primary_nodes=4) [Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f/0 drbd2069 pve3: pdsk( UpToDate -> Outdated ) [Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve3: No reconciliation resync even though 'pve2' disappeared. (o=0) [Fri Oct 25 14:52:12 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:52:21 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:52:30 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:52:38 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:52:50 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:53:02 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:53:11 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:53:23 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:53:32 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:53:43 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:53:56 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:54:07 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 ... root@pve3:~# drbdadm status pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f role:Secondary disk:Outdated pve1 connection:Connecting pve2 connection:Connecting dmesg -T | grep pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f [Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: Preparing remote state change 1999068598 [Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: Committing remote state change 1999068598 (primary_nodes=4) [Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f/0 drbd2069 pve1: pdsk( UpToDate -> Outdated ) [Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: Preparing remote state change 303080422 [Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: Committing remote state change 303080422 (primary_nodes=4) [Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: conn( Connected -> TearDown ) peer( Primary -> Unknown ) [Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f/0 drbd2069: disk( UpToDate -> Outdated ) [Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f/0 drbd2069 pve2: pdsk( UpToDate -> DUnknown ) repl( Established -> Off ) [Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: ack_receiver terminated [Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: Terminating ack_recv thread [Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: Restarting sender thread [Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: Connection closed [Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: conn( TearDown -> Unconnected ) [Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: Restarting receiver thread [Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f pve2: conn( Unconnected -> Connecting ) [Fri Oct 25 14:52:01 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:52:13 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:52:22 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:52:33 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:52:45 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:52:54 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:53:06 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:53:15 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:53:27 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:53:35 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:53:47 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:53:56 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 [Fri Oct 25 14:54:05 2019] drbd pvc-0c26a3e2-bffd-4fee-916d-e28ea741d73f tcp:pve2: Closing unexpected connection from 10.37.20.2 ... - kvaps
_______________________________________________ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com https://lists.linbit.com/mailman/listinfo/drbd-user