Could you try a few things, so we can get a better picture of what's happening there: - Can you get a hash of the data on the backend device that DRBD is writing to, from before and after one of those dubious Secondary-mode writes, to verify whether or not any data is actually changed? - Can you switch the other peer into the Primary role manually (so that the node where the problem occurs should refuse to become a Primary) and see what happens when ZFS tries to write to that log?
I tried to reproduce the problem from user space (with auto-promote off and trying to read/write from/to a Secondary), where it does not seem to happen (everything is normal, cannot even read from a Secondary). However, I expect those ZFS operations to be done by some code in the kernel itself, and something may not be playing by the rules there - maybe ZFS is causing some I/O without doing a proper open/close cycle, or we are missing something in DRBD for some I/O case that's supposed to be valid. br, Robert On 11/14/19 10:28 PM, Doug Cahill wrote: > I spent some more time looking into this with another developer and I > can see while running "drbdsetup events2 r0" that there is a quick > blip when I add the drbd r0 resource to my pool as the log device: > > change resource name:r0 role:Primary > change resource name:r0 role:Secondary > > However, if I export and/or import the pool, the event never registers > again. When I write to a vdisk on this pool I can see the nr:11766480 > dw:11766452 counts increase on the peer which leads me to believe > blocks are being written, yet the state never changes. > > I also tried to run dd to the "peer" side drbd device while the > "active" side was writing data and found a message stating the peer > may not become primary while the device is opened read-only in my > syslog which doesn't make sense. The device is being written to, so > how is the block device state being tricked to thinking it is read only? > > =========in the log from the node I'm writing to the drbd resource > drbd r0 dccdx0: Preparing remote state change 892694821 > drbd r0: State change failed: Peer may not become primary while device > is opened read-only > kernel: [92771.927574] drbd r0 dccdx0: Aborting remote state change > 892694821 > > On Thu, Nov 14, 2019 at 10:39 AM Doug Cahill <[email protected] > <mailto:[email protected]>> wrote: > > On Thu, Nov 14, 2019 at 4:52 AM Roland Kammerer > <[email protected] <mailto:[email protected]>> > wrote: > > On Wed, Nov 13, 2019 at 03:08:37PM -0500, Doug Cahill wrote: > > I'm configuring a two node setup with drbd 9.0.20-1 on CentOS 7 > > (3.10.0-957.1.3.el7.x86_64) with a single resource backed by > an SSDs. I've > > explicitly enabled auto-promote in my resource configuration > to use this > > feature. > > > > The drbd device is being used in a single-primary > configuration as a zpool > > SLOG device. The zpool is only ever imported on one node at > a time and the > > import is successful during cluster failover events between > nodes. I > > confirmed through zdb that the zpool includes the configured > drbd device > > path. > > > > My concern is that the drbdadm status output shows the Role > of the drbd > > resource as "Secondary" on both sides. The documentations > reads that the > > drbd resource will be auto promoted to primary when it is > opened for > > writing. > > But also demoted when closed (don't know if this happens in your > scenario). > > > drbdadm status > > r0 role:Secondary > > disk:UpToDate > > dccdx0 role:Secondary > > peer-disk:UpToDate > > Maybe it is closed and demoted again and you look at it at the > wrong > points in time? Better look into the syslog for role changes, > or monitor > with "drbdsetup events2 r0". Do you see switches to Primary there? > > > I checked the drbdadm status while my dd write session was in > progress and I see no change from Secondary to Primary. I also > checked the stats under /sys/class and it looks the same. > > cat /sys/kernel/debug/drbd/resources/r0/connections/dccdx0/0/proc_drbd > 0: cs:Established ro:Secondary/Secondary ds:UpToDate/UpToDate C > r----- > ns:3330728 nr:0 dw:20103080 dr:26292 al:131 bm:0 lo:0 pe:[0;0] > ua:0 ap:[0;0] ep:1 wo:1 oos:0 > resync: used:0/61 hits:64 misses:4 starving:0 locked:0 changed:2 > act_log: used:0/1237 hits:28951 misses:536 starving:0 locked:0 > changed:132 > blocked on activity log: 0/0/0 > > > Best, rck > _______________________________________________ > Star us on GITHUB: https://github.com/LINBIT > drbd-user mailing list > [email protected] <mailto:[email protected]> > https://lists.linbit.com/mailman/listinfo/drbd-user > > > _______________________________________________ > Star us on GITHUB: https://github.com/LINBIT > drbd-user mailing list > [email protected] > https://lists.linbit.com/mailman/listinfo/drbd-user -- Robert ALTNOEDER - Software Developer +43-1-817-82-92 x72 <tel:+4318178292> [email protected] <mailto:[email protected]> LIN <http://www.linbit.com/en/>BIT <http://www.linbit.com/en/> | Keeping the Digital World Running DRBD HA - Disaster Recovery - Software-defined Storage t <https://twitter.com/linbit> / f <https://www.facebook.com/pg/linbitdrbd/posts/> / in <https://www.linkedin.com/company/linbit> / y <https://www.youtube.com/user/linbit> / g+ <https://plus.google.com/+Linbit/about> DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. _______________________________________________ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list [email protected] https://lists.linbit.com/mailman/listinfo/drbd-user
