Hello We are observing the following issue with resync after reboot.
After rebooting a secondary node (in a 2 or 3 node cluster), the secondary successfully connects to primary and reports UpToDate, but when a verify is launched on the secondary node that was rebooted, it reports out of sync blocks. If an "invalidate --reset-bitmap=no" is issued on the resource on the secondary node, the invalidate sync happens quickly and the next verify succeeds with no out of sync blocks. This was initially detected when we promoted a backup node and it came up with disk corruption. We traced this to the reboot occurring before the promotion. Versions The logs attached are using the 9.2.12 version of the driver on the 5.15.173 kernel, but we have also observed this issue on the 9.2.4 driver with the 5.15.166 kernel We have not seen the problem on 5.15.151 and version 9.2.4 of the driver. Attachments initsyncandverify_noreboot.txt - drbd logs from system prior to reboot , includes verify before reboot verify_after_invalidate_no_reset.txt - drbd logs after reboot show initial failed verify then, invalidate, then successful verify dynamic.res - drbd conf file - note use of separate metadata disk - we also Secondary Bring Up Secondary nodes enable drbd "persist" resource as follows """ da up all || true da secondary persist || true da disconnect persist || true da -- --discard-my-data connect persist || true """
dynamic.res
Description: dynamic.res
[ 9.039668] drbd: loading out-of-tree module taints kernel. [ 9.044657] drbd: initialized. Version: 9.2.12 (api:2/proto:118-122) [ 9.044659] drbd: GIT-hash: 2da6f528dc4ab3fd25c511f7b03531100e54ab08 build by bamboo@sta-edaptos-feature-2fapfp-7384-update-kernel-3-lw02f-xrm-h73mh, 2024-11-26 17:40:06 [ 9.044659] drbd: registered as block device major 147 [ 9.046166] drbd: registered transport class 'tcp' (version:9.2.12) [ 9.110511] drbd persist: Starting worker thread (node-id 2) [ 9.111201] drbd persist TestVerify111-1: Starting sender thread (peer-node-id 1) [ 9.113319] drbd persist/0 drbd0: meta-data IO uses: blk-bio [ 9.113485] drbd persist/0 drbd0: disk( Diskless -> Attaching ) [attach] [ 9.113490] drbd persist/0 drbd0: Maximum number of peer devices = 5 [ 9.113515] drbd persist: Method to ensure write ordering: flush [ 9.113518] drbd persist/0 drbd0: drbd_bm_resize called with capacity == 53190656 [ 9.114347] drbd persist/0 drbd0: resync bitmap: bits=6648832 words=519440 pages=1015 [ 9.114348] drbd0: detected capacity change from 0 to 53190656 [ 9.114349] drbd persist/0 drbd0: size = 25 GB (26595328 KB) [ 9.117103] drbd persist/0 drbd0: disk( Attaching -> Inconsistent ) [attach] [ 9.117105] drbd persist/0 drbd0: attached to current UUID: 0000000000000004 [ 9.118687] drbd persist TestVerify111-1: conn( StandAlone -> Unconnected ) [connect] [ 9.118951] drbd persist TestVerify111-1: Starting receiver thread (peer-node-id 1) [ 9.118971] drbd persist TestVerify111-1: conn( Unconnected -> Connecting ) [connecting] [ 9.120151] drbd persist TestVerify111-1: conn( Connecting -> Disconnecting ) [disconnect] [ 9.120176] drbd persist TestVerify111-1: Terminating sender thread [ 9.120183] drbd persist TestVerify111-1: Starting sender thread (peer-node-id 1) [ 9.128671] drbd persist TestVerify111-1: Connection closed [ 9.128675] drbd persist TestVerify111-1: helper command: /sbin/drbdadm disconnected [ 9.128721] drbd persist TestVerify111-1: helper command: /sbin/drbdadm disconnected exit code 255 (0xfffffffe) [ 9.128729] drbd persist TestVerify111-1: conn( Disconnecting -> StandAlone ) [disconnected] [ 9.128735] drbd persist TestVerify111-1: Terminating receiver thread [ 9.129449] drbd persist TestVerify111-1: conn( StandAlone -> Unconnected ) [connect] [ 9.129460] drbd persist TestVerify111-1: Starting receiver thread (peer-node-id 1) [ 9.129494] drbd persist TestVerify111-1: conn( Unconnected -> Connecting ) [connecting] [ 9.632783] drbd persist TestVerify111-1: Handshake to peer 1 successful: Agreed network protocol version 122 [ 9.632788] drbd persist TestVerify111-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG [ 9.634773] drbd persist: Preparing remote state change 2495832259: 1->2 role( Primary ) conn( Connected ) [ 9.646448] drbd persist/0 drbd0 TestVerify111-1: drbd_sync_handshake: [ 9.646450] drbd persist/0 drbd0 TestVerify111-1: self 0000000000000004:0000000000000000:B2EA3DAEC8F42136:0000000000000000 bits:0 flags:25 [ 9.646452] drbd persist/0 drbd0 TestVerify111-1: peer 0924E7BEE3B852C5:B2EA3DAEC8F42137:0000000000000000:0000000000000000 bits:850557 flags:1020 [ 9.646453] drbd persist/0 drbd0 TestVerify111-1: uuid_compare()=target-set-bitmap by rule=just-created-self [ 9.646455] drbd persist/0 drbd0 TestVerify111-1: Setting and writing the whole bitmap, fresh node [ 9.649343] drbd persist TestVerify111-1: Committing remote state change 2495832259 (primary_nodes=2) [ 9.649347] drbd persist TestVerify111-1: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [remote] [ 9.649349] drbd persist/0 drbd0 TestVerify111-1: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT ) [remote] [ 9.649352] drbd persist/0 drbd0: Disabling local AL-updates (optimization) [ 9.649354] drbd persist/0 drbd0: Setting exposed data uuid: 0924E7BEE3B852C4 [ 9.649374] drbd persist/0 drbd0: persisting effective size = 25 GB (26595328 KB) [ 9.649999] drbd persist/0 drbd0 TestVerify111-1: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0% [ 9.650160] drbd persist/0 drbd0 TestVerify111-1: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0% [ 9.650165] drbd persist/0 drbd0 TestVerify111-1: helper command: /sbin/drbdadm before-resync-target [ 9.650229] drbd persist/0 drbd0 TestVerify111-1: helper command: /sbin/drbdadm before-resync-target exit code 255 (0xfffffffe) [ 9.650233] drbd persist/0 drbd0 TestVerify111-1: setting UUIDs to B2EA3DAEC8F42136:0000000000000000:B2EA3DAEC8F42136:0000000000000000 [ 9.650236] drbd persist/0 drbd0 TestVerify111-1: repl( WFBitMapT -> SyncTarget ) [receive-bitmap] [ 9.650238] drbd persist/0 drbd0: Setting exposed data uuid: 0924E7BEE3B852C4 [ 9.650249] drbd persist/0 drbd0 TestVerify111-1: Began resync as SyncTarget (will sync 26595328 KB [6648832 bits set]). [ 54.583427] drbd persist TestVerify111-3: Starting sender thread (peer-node-id 3) [ 54.584412] drbd persist TestVerify111-3: conn( StandAlone -> Unconnected ) [connect] [ 54.584684] drbd persist TestVerify111-3: Starting receiver thread (peer-node-id 3) [ 54.584710] drbd persist TestVerify111-3: conn( Unconnected -> Connecting ) [connecting] [ 56.106044] drbd persist: Preparing remote state change 218899043: 1->3 role( Primary ) conn( Connected ) [ 56.121652] drbd persist TestVerify111-1: Committing remote state change 218899043 (primary_nodes=2) [ 56.153173] drbd persist TestVerify111-3: Handshake to peer 3 successful: Agreed network protocol version 122 [ 56.153175] drbd persist TestVerify111-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG [ 56.154426] drbd persist: Preparing cluster-wide state change 2708969553: 2->3 role( Secondary ) conn( Connected ) [ 56.163430] drbd persist/0 drbd0 TestVerify111-3: drbd_sync_handshake: [ 56.163432] drbd persist/0 drbd0 TestVerify111-3: self 0924E7BEE3B852C5:B2EA3DAEC8F42136:B2EA3DAEC8F42136:0000000000000000 bits:787296 flags:4 [ 56.163434] drbd persist/0 drbd0 TestVerify111-3: peer 0924E7BEE3B852C5:B2EA3DAEC8F42136:B2EA3DAEC8F42136:0000000000000000 bits:6646784 flags:1805 [ 56.163436] drbd persist/0 drbd0 TestVerify111-3: uuid_compare()=no-sync by rule=both-off [ 56.164168] drbd persist: State change 2708969553: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFF1 [ 56.164170] drbd persist: Committing cluster-wide state change 2708969553 (9ms) [ 56.164180] drbd persist TestVerify111-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected] [ 56.164181] drbd persist/0 drbd0 TestVerify111-3: pdsk( DUnknown -> Inconsistent ) repl( Off -> Established ) [connected] [ 56.164184] drbd persist/0 drbd0: Enabling local AL-updates [ 394.145702] drbd persist/0 drbd0 TestVerify111-1: Resync done (total 384 sec; paused 0 sec; 69256 K/sec) [ 394.145709] drbd persist/0 drbd0 TestVerify111-1: updated UUIDs 0924E7BEE3B852C4:0000000000000000:B2EA3DAEC8F42136:0000000000000000 [ 394.145715] drbd persist/0 drbd0: disk( Inconsistent -> UpToDate ) [resync-finished] [ 394.145716] drbd persist/0 drbd0 TestVerify111-1: repl( SyncTarget -> Established ) [resync-finished] [ 394.145932] drbd persist/0 drbd0 TestVerify111-1: helper command: /sbin/drbdadm after-resync-target [ 394.146077] drbd persist/0 drbd0 TestVerify111-1: helper command: /sbin/drbdadm after-resync-target exit code 255 (0xfffffffe) [ 451.168882] drbd persist/0 drbd0 TestVerify111-3: pdsk( Inconsistent -> UpToDate ) resync-susp( peer -> no ) [peer-state] [242133.852783] drbd persist: Preparing cluster-wide state change 4104304386: 2->3 repl( VerifyS ) [242133.853135] drbd persist: State change 4104304386: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFF1 [242133.853137] drbd persist: Committing cluster-wide state change 4104304386 (0ms) [242133.853145] drbd persist/0 drbd0 TestVerify111-3: repl( Established -> VerifyS ) [verify] [242133.853167] drbd persist/0 drbd0 TestVerify111-3: Starting Online Verify from sector 0 [242133.854094] drbd persist: Preparing cluster-wide state change 1620712363: 2->1 repl( VerifyS ) [242133.856726] drbd persist: State change 1620712363: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFF1 [242133.856728] drbd persist: Committing cluster-wide state change 1620712363 (3ms) [242133.856739] drbd persist/0 drbd0 TestVerify111-1: repl( Established -> VerifyS ) [verify] [242133.856758] drbd persist/0 drbd0 TestVerify111-1: Starting Online Verify from sector 0 [242524.612544] drbd persist/0 drbd0 TestVerify111-3: Online verify done (total 390 sec; paused 0 sec; 68192 K/sec) [242524.612552] drbd persist/0 drbd0 TestVerify111-3: repl( VerifyS -> Established ) [resync-finished] [242549.321856] drbd persist/0 drbd0 TestVerify111-1: Online verify done (total 415 sec; paused 0 sec; 64084 K/sec) [242549.321865] drbd persist/0 drbd0 TestVerify111-1: repl( VerifyS -> Established ) [resync-finished]
[ 9.194827] drbd: loading out-of-tree module taints kernel. [ 9.199908] drbd: initialized. Version: 9.2.12 (api:2/proto:118-122) [ 9.199909] drbd: GIT-hash: 2da6f528dc4ab3fd25c511f7b03531100e54ab08 build by bamboo@sta-edaptos-feature-2fapfp-7384-update-kernel-3-lw02f-xrm-h73mh, 2024-11-26 17:40:06 [ 9.199910] drbd: registered as block device major 147 [ 9.201286] drbd: registered transport class 'tcp' (version:9.2.12) [ 9.231989] drbd persist: Starting worker thread (node-id 2) [ 9.232963] drbd persist TestVerify111-3: Starting sender thread (peer-node-id 3) [ 9.233348] drbd persist TestVerify111-1: Starting sender thread (peer-node-id 1) [ 9.236675] drbd persist/0 drbd0: meta-data IO uses: blk-bio [ 9.236810] drbd persist/0 drbd0: disk( Diskless -> Attaching ) [attach] [ 9.236815] drbd persist/0 drbd0: Maximum number of peer devices = 5 [ 9.236840] drbd persist: Method to ensure write ordering: flush [ 9.236842] drbd persist/0 drbd0: drbd_bm_resize called with capacity == 53190656 [ 9.237691] drbd persist/0 drbd0: resync bitmap: bits=6648832 words=519440 pages=1015 [ 9.237693] drbd0: detected capacity change from 0 to 53190656 [ 9.237694] drbd persist/0 drbd0: size = 25 GB (26595328 KB) [ 9.241664] drbd persist/0 drbd0: disk( Attaching -> Outdated ) [attach] [ 9.241666] drbd persist/0 drbd0: attached to current UUID: 0924E7BEE3B852C4 [ 9.243277] drbd persist TestVerify111-3: conn( StandAlone -> Unconnected ) [connect] [ 9.243497] drbd persist TestVerify111-3: Starting receiver thread (peer-node-id 3) [ 9.243527] drbd persist TestVerify111-3: conn( Unconnected -> Connecting ) [connecting] [ 9.243583] drbd persist TestVerify111-1: conn( StandAlone -> Unconnected ) [connect] [ 9.243651] drbd persist TestVerify111-1: Starting receiver thread (peer-node-id 1) [ 9.243769] drbd persist TestVerify111-1: conn( Unconnected -> Connecting ) [connecting] [ 9.245082] drbd persist TestVerify111-3: conn( Connecting -> Disconnecting ) [disconnect] [ 9.245108] drbd persist TestVerify111-3: Terminating sender thread [ 9.245115] drbd persist TestVerify111-3: Starting sender thread (peer-node-id 3) [ 9.257791] drbd persist TestVerify111-3: Connection closed [ 9.257796] drbd persist TestVerify111-3: helper command: /sbin/drbdadm disconnected [ 9.257905] drbd persist TestVerify111-3: helper command: /sbin/drbdadm disconnected exit code 255 (0xfffffffe) [ 9.257914] drbd persist TestVerify111-3: conn( Disconnecting -> StandAlone ) [disconnected] [ 9.257920] drbd persist TestVerify111-3: Terminating receiver thread [ 9.258243] drbd persist TestVerify111-1: conn( Connecting -> Disconnecting ) [disconnect] [ 9.258271] drbd persist TestVerify111-1: Terminating sender thread [ 9.258275] drbd persist TestVerify111-1: Starting sender thread (peer-node-id 1) [ 9.263771] drbd persist TestVerify111-1: Connection closed [ 9.263776] drbd persist TestVerify111-1: helper command: /sbin/drbdadm disconnected [ 9.263815] drbd persist TestVerify111-1: helper command: /sbin/drbdadm disconnected exit code 255 (0xfffffffe) [ 9.263823] drbd persist TestVerify111-1: conn( Disconnecting -> StandAlone ) [disconnected] [ 9.263828] drbd persist TestVerify111-1: Terminating receiver thread [ 9.264566] drbd persist TestVerify111-3: conn( StandAlone -> Unconnected ) [connect] [ 9.264578] drbd persist TestVerify111-3: Starting receiver thread (peer-node-id 3) [ 9.264623] drbd persist TestVerify111-3: conn( Unconnected -> Connecting ) [connecting] [ 9.264851] drbd persist TestVerify111-1: conn( StandAlone -> Unconnected ) [connect] [ 9.264861] drbd persist TestVerify111-1: Starting receiver thread (peer-node-id 1) [ 9.264900] drbd persist TestVerify111-1: conn( Unconnected -> Connecting ) [connecting] [ 10.819888] drbd persist TestVerify111-3: Handshake to peer 3 successful: Agreed network protocol version 122 [ 10.819892] drbd persist TestVerify111-3: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG [ 10.820515] drbd persist: Preparing cluster-wide state change 1694023638: 2->3 role( Secondary ) conn( Connected ) [ 10.827508] drbd persist/0 drbd0 TestVerify111-3: drbd_sync_handshake: [ 10.827510] drbd persist/0 drbd0 TestVerify111-3: self 0924E7BEE3B852C4:0000000000000000:B2EA3DAEC8F42136:0000000000000000 bits:0 flags:21 [ 10.827512] drbd persist/0 drbd0 TestVerify111-3: peer 133E288485C8D0F2:0924E7BEE3B852C4:B2EA3DAEC8F42136:0000000000000000 bits:287 flags:1100 [ 10.827514] drbd persist/0 drbd0 TestVerify111-3: uuid_compare()=target-use-bitmap by rule=bitmap-peer [ 10.827521] drbd persist: State change 1694023638: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFF1 [ 10.827522] drbd persist: Committing cluster-wide state change 1694023638 (7ms) [ 10.827528] drbd persist TestVerify111-3: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected] [ 10.827530] drbd persist/0 drbd0 TestVerify111-3: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT ) [connected] [ 10.827534] drbd persist/0 drbd0: Setting exposed data uuid: 133E288485C8D0F2 [ 10.828493] drbd persist/0 drbd0 TestVerify111-3: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 71(1), total 71; compression: 100.0% [ 10.828677] drbd persist/0 drbd0 TestVerify111-3: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 71(1), total 71; compression: 100.0% [ 10.828682] drbd persist/0 drbd0 TestVerify111-3: helper command: /sbin/drbdadm before-resync-target [ 10.828750] drbd persist/0 drbd0 TestVerify111-3: helper command: /sbin/drbdadm before-resync-target exit code 255 (0xfffffffe) [ 10.828756] drbd persist/0 drbd0: disk( Outdated -> Inconsistent ) [receive-bitmap] [ 10.828757] drbd persist/0 drbd0 TestVerify111-1: resync-susp( no -> connection dependency ) [receive-bitmap] [ 10.828758] drbd persist/0 drbd0 TestVerify111-3: repl( WFBitMapT -> SyncTarget ) [receive-bitmap] [ 10.828779] drbd persist/0 drbd0 TestVerify111-3: Began resync as SyncTarget (will sync 1148 KB [287 bits set]). [ 10.836886] drbd persist/0 drbd0 TestVerify111-3: Resync done (total 1 sec; paused 0 sec; 1148 K/sec) [ 10.836889] drbd persist/0 drbd0 TestVerify111-3: Peer was unstable during resync [ 10.836894] drbd persist/0 drbd0 TestVerify111-1: resync-susp( connection dependency -> no ) [resync-finished] [ 10.836895] drbd persist/0 drbd0 TestVerify111-3: repl( SyncTarget -> Established ) [resync-finished] [ 10.836906] drbd persist/0 drbd0 TestVerify111-3: helper command: /sbin/drbdadm after-resync-target [ 10.837001] drbd persist/0 drbd0 TestVerify111-3: helper command: /sbin/drbdadm after-resync-target exit code 255 (0xfffffffe) [ 11.673569] drbd persist TestVerify111-1: Handshake to peer 1 successful: Agreed network protocol version 122 [ 11.673573] drbd persist TestVerify111-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG [ 11.674560] drbd persist: Preparing remote state change 3944266936: 1->2 role( Primary ) conn( Connected ) [ 11.685516] drbd persist/0 drbd0 TestVerify111-1: drbd_sync_handshake: [ 11.685519] drbd persist/0 drbd0 TestVerify111-1: self 0924E7BEE3B852C4:0924E7BEE3B852C4:B2EA3DAEC8F42136:0000000000000000 bits:0 flags:25 [ 11.685521] drbd persist/0 drbd0 TestVerify111-1: peer 133E288485C8D0F3:0924E7BEE3B852C5:0000000000000000:0000000000000000 bits:0 flags:1120 [ 11.685523] drbd persist/0 drbd0 TestVerify111-1: uuid_compare()=target-use-bitmap by rule=bitmap-peer [ 11.685780] drbd persist TestVerify111-1: Committing remote state change 3944266936 (primary_nodes=2) [ 11.685784] drbd persist TestVerify111-1: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [remote] [ 11.685785] drbd persist/0 drbd0 TestVerify111-1: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT ) [remote] [ 11.687142] drbd persist/0 drbd0 TestVerify111-1: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0% [ 11.687329] drbd persist/0 drbd0 TestVerify111-1: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0% [ 11.687334] drbd persist/0 drbd0 TestVerify111-1: helper command: /sbin/drbdadm before-resync-target [ 11.687403] drbd persist/0 drbd0 TestVerify111-1: helper command: /sbin/drbdadm before-resync-target exit code 255 (0xfffffffe) [ 11.687409] drbd persist/0 drbd0 TestVerify111-1: repl( WFBitMapT -> SyncTarget ) [receive-bitmap] [ 11.687410] drbd persist/0 drbd0 TestVerify111-3: resync-susp( no -> connection dependency ) [receive-bitmap] [ 11.687423] drbd persist/0 drbd0 TestVerify111-1: Began resync as SyncTarget (will sync 0 KB [0 bits set]). [ 11.687776] drbd persist/0 drbd0 TestVerify111-1: Resync done (total 1 sec; paused 0 sec; 0 K/sec) [ 11.687778] drbd persist/0 drbd0 TestVerify111-1: updated UUIDs 133E288485C8D0F2:0000000000000000:B2EA3DAEC8F42136:0000000000000000 [ 11.687781] drbd persist/0 drbd0: disk( Inconsistent -> UpToDate ) [resync-finished] [ 11.687781] drbd persist/0 drbd0 TestVerify111-1: repl( SyncTarget -> Established ) [resync-finished] [ 11.687782] drbd persist/0 drbd0 TestVerify111-3: resync-susp( connection dependency -> no ) [resync-finished] [ 11.687993] drbd persist/0 drbd0 TestVerify111-1: helper command: /sbin/drbdadm after-resync-target [ 11.691007] drbd persist/0 drbd0 TestVerify111-1: helper command: /sbin/drbdadm after-resync-target exit code 255 (0xfffffffe) [ 469.905587] drbd persist: Preparing cluster-wide state change 2256647405: 2->3 repl( VerifyS ) [ 469.905980] drbd persist: State change 2256647405: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFF1 [ 469.905981] drbd persist: Committing cluster-wide state change 2256647405 (0ms) [ 469.905990] drbd persist/0 drbd0 TestVerify111-3: repl( Established -> VerifyS ) [verify] [ 469.906025] drbd persist/0 drbd0 TestVerify111-3: Starting Online Verify from sector 0 [ 469.906478] drbd persist: Preparing cluster-wide state change 240032565: 2->1 repl( VerifyS ) [ 469.910168] drbd persist: State change 240032565: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFF1 [ 469.910169] drbd persist: Committing cluster-wide state change 240032565 (3ms) [ 469.910178] drbd persist/0 drbd0 TestVerify111-1: repl( Established -> VerifyS ) [verify] [ 469.910322] drbd persist/0 drbd0 TestVerify111-1: Starting Online Verify from sector 0 [ 526.910231] drbd persist/0 drbd0 TestVerify111-3: Out of sync: start=8388680, size=8 (sectors) [ 526.983995] drbd persist/0 drbd0 TestVerify111-3: Out of sync: start=8390480, size=8 (sectors) [ 530.263091] drbd persist/0 drbd0 TestVerify111-1: Out of sync: start=8388680, size=8 (sectors) [ 530.265254] drbd persist/0 drbd0 TestVerify111-1: Out of sync: start=8390480, size=8 (sectors) [ 558.196526] drbd persist/0 drbd0 TestVerify111-3: Out of sync: start=12589400, size=8 (sectors) [ 558.196621] drbd persist/0 drbd0 TestVerify111-3: Out of sync: start=12589488, size=8 (sectors) [ 558.196639] drbd persist/0 drbd0 TestVerify111-3: Out of sync: start=12589512, size=8 (sectors) [ 562.767636] drbd persist/0 drbd0 TestVerify111-1: Out of sync: start=12589400, size=8 (sectors) [ 562.767700] drbd persist/0 drbd0 TestVerify111-1: Out of sync: start=12589488, size=8 (sectors) [ 562.767718] drbd persist/0 drbd0 TestVerify111-1: Out of sync: start=12589512, size=8 (sectors) [ 591.252407] drbd persist/0 drbd0 TestVerify111-3: Out of sync: start=16777280, size=8 (sectors) [ 596.321251] drbd persist/0 drbd0 TestVerify111-1: Out of sync: start=16777280, size=8 (sectors) [ 655.525825] drbd persist/0 drbd0 TestVerify111-3: Out of sync: start=25427976, size=104 (sectors) [ 665.362851] drbd persist/0 drbd0 TestVerify111-1: Out of sync: start=25427976, size=104 (sectors) [ 862.558924] drbd persist/0 drbd0 TestVerify111-3: Online verify done (total 392 sec; paused 0 sec; 67844 K/sec) [ 862.558929] drbd persist/0 drbd0 TestVerify111-3: Online verify found 18 4k blocks out of sync! [ 862.558934] drbd persist/0 drbd0 TestVerify111-3: repl( VerifyS -> Established ) [resync-finished] [ 862.558943] drbd persist/0 drbd0 TestVerify111-3: helper command: /sbin/drbdadm out-of-sync [ 862.559058] drbd persist/0 drbd0 TestVerify111-3: helper command: /sbin/drbdadm out-of-sync exit code 255 (0xfffffffe) [ 877.514339] drbd persist/0 drbd0 TestVerify111-1: Online verify done (total 407 sec; paused 0 sec; 65344 K/sec) [ 877.514344] drbd persist/0 drbd0 TestVerify111-1: Online verify found 18 4k blocks out of sync! [ 877.514349] drbd persist/0 drbd0 TestVerify111-1: repl( VerifyS -> Established ) [resync-finished] [ 877.514357] drbd persist/0 drbd0 TestVerify111-1: helper command: /sbin/drbdadm out-of-sync [ 877.514543] drbd persist/0 drbd0 TestVerify111-1: helper command: /sbin/drbdadm out-of-sync exit code 255 (0xfffffffe) [ 1089.329950] drbd persist: Preparing cluster-wide state change 1991906621: 2->3 repl( WFBitMapT ) [ 1089.330357] drbd persist: State change 1991906621: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFF1 [ 1089.330359] drbd persist: Committing cluster-wide state change 1991906621 (0ms) [ 1089.330362] drbd persist/0 drbd0: disk( UpToDate -> Outdated ) [invalidate] [ 1089.330363] drbd persist/0 drbd0 TestVerify111-3: repl( Established -> WFBitMapT ) [invalidate] [ 1089.331701] drbd persist/0 drbd0 TestVerify111-3: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 41(1), total 41; compression: 100.0% [ 1089.331896] drbd persist/0 drbd0 TestVerify111-3: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 41(1), total 41; compression: 100.0% [ 1089.331901] drbd persist/0 drbd0 TestVerify111-3: helper command: /sbin/drbdadm before-resync-target [ 1089.331984] drbd persist/0 drbd0 TestVerify111-3: helper command: /sbin/drbdadm before-resync-target exit code 255 (0xfffffffe) [ 1089.331992] drbd persist/0 drbd0: disk( Outdated -> Inconsistent ) [receive-bitmap] [ 1089.331993] drbd persist/0 drbd0 TestVerify111-1: resync-susp( no -> connection dependency ) [receive-bitmap] [ 1089.331994] drbd persist/0 drbd0 TestVerify111-3: repl( WFBitMapT -> SyncTarget ) [receive-bitmap] [ 1089.332013] drbd persist/0 drbd0 TestVerify111-3: Began resync as SyncTarget (will sync 72 KB [18 bits set]). [ 1089.335020] drbd persist/0 drbd0 TestVerify111-3: Resync done (total 1 sec; paused 0 sec; 72 K/sec) [ 1089.335026] drbd persist/0 drbd0 TestVerify111-3: updated UUIDs 133E288485C8D0F2:0000000000000000:0924E7BEE3B852C4:B2EA3DAEC8F42136 [ 1089.335030] drbd persist/0 drbd0: disk( Inconsistent -> UpToDate ) [resync-finished] [ 1089.335031] drbd persist/0 drbd0 TestVerify111-1: resync-susp( connection dependency -> no ) [resync-finished] [ 1089.335032] drbd persist/0 drbd0 TestVerify111-3: repl( SyncTarget -> Established ) [resync-finished] [ 1089.335228] drbd persist/0 drbd0 TestVerify111-3: helper command: /sbin/drbdadm after-resync-target [ 1089.335294] drbd persist/0 drbd0 TestVerify111-3: helper command: /sbin/drbdadm after-resync-target exit code 255 (0xfffffffe) [ 1089.338428] drbd persist/0 drbd0 TestVerify111-1: received new current UUID: B9A226E34C2368A3 weak_nodes=FFFFFFFFFFFFFFF1 [ 1107.107229] drbd persist: Preparing cluster-wide state change 56453665: 2->3 repl( VerifyS ) [ 1107.107643] drbd persist: State change 56453665: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFF1 [ 1107.107645] drbd persist: Committing cluster-wide state change 56453665 (1ms) [ 1107.107653] drbd persist/0 drbd0 TestVerify111-3: repl( Established -> VerifyS ) [verify] [ 1107.107681] drbd persist/0 drbd0 TestVerify111-3: Starting Online Verify from sector 0 [ 1107.108315] drbd persist: Preparing cluster-wide state change 1793193565: 2->1 repl( VerifyS ) [ 1107.111931] drbd persist: State change 1793193565: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFF1 [ 1107.111934] drbd persist: Committing cluster-wide state change 1793193565 (4ms) [ 1107.111950] drbd persist/0 drbd0 TestVerify111-1: repl( Established -> VerifyS ) [verify] [ 1107.111966] drbd persist/0 drbd0 TestVerify111-1: Starting Online Verify from sector 0 [ 1497.669230] drbd persist/0 drbd0 TestVerify111-3: Online verify done (total 390 sec; paused 0 sec; 68192 K/sec) [ 1497.669244] drbd persist/0 drbd0 TestVerify111-3: repl( VerifyS -> Established ) [resync-finished] [ 1520.248563] drbd persist/0 drbd0 TestVerify111-1: Online verify done (total 413 sec; paused 0 sec; 64392 K/sec) [ 1520.248571] drbd persist/0 drbd0 TestVerify111-1: repl( VerifyS -> Established ) [resync-finished]
