Hi Joel,All drbd devices are in the same volume group on the same physical disk. The physical disk is a hardware raid-1 device consisting of 2 ssd disk. E.g:
lvdisplay /dev/vg_ssd/test-lv --- Logical volume --- LV Path /dev/vg_ssd/test-lv LV Name test-lv VG Name vg_ssd LV UUID HWkc5O-FPBp-hjCb-WHjU-1Z1T-Wb1N-nUtRmH LV Write Access read/write LV Creation host, time barentsz, 2019-06-12 09:54:27 +0200 LV Status available # open 1 LV Size 40.00 GiB Current LE 10240 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 1024 Block device 254:16 and: lvdisplay /dev/vg_ssd/vm_prace-ldap --- Logical volume --- LV Path /dev/vg_ssd/vm_prace-ldap LV Name vm_prace-ldap VG Name vg_ssd LV UUID KbAvaa-F9ER-NNfW-tPmb-C4WF-CHXP-ZheJed LV Write Access read/write LV Creation host, time barentsz, 2017-11-02 15:33:06 +0100 LV Status available # open 2 LV Size 38.00 GiB Current LE 9728 Segments 2 Allocation inherit Read ahead sectors auto - currently set to 1024 Block device 254:9
The drbd status on the failing R0 device on this host is: 0:r0/0 Connected/Connected Secondary/Primary UpToDate/UpToDate Regards, Rob On 6/12/19 10:39 AM, Joel Colledge wrote:
Hi Rob, This message indicates that mounting with DAX is failing here: https://github.com/openSUSE/kernel/blob/SLE12-SP4/drivers/dax/super.c#L131 But DRBD performs the corresponding check here: https://github.com/LINBIT/drbd-9.0/blob/drbd-9.0.18-1/drbd/drbd_dax_pmem.c#L57I don't know why the check fails for the DAX mount but succeeds for DRBD. I suggest you set up a DRBD resource using exactly this vg_ssd/test-lv that you used for the mount test. It is possible that the LV you are using for DRBD and the test LV are different in some way, e.g. they have been placed on different backing devices. If DRBD works successfully on this LV then we need to find out what the difference between the LVs is.Best regards, JoelOn Wed, Jun 12, 2019 at 10:00 AM Rob van der Wal <[email protected] <mailto:[email protected]>> wrote:Hi Joel, The additional debug setting gives one additional line: 2019-06-12T09:58:27.977088+02:00 barentsz kern warning kernel - - - [525370.955129] EXT4-fs (dm-16): DAX enabled. Warning: EXPERIMENTAL, use at your own risk 2019-06-12T09:58:27.977121+02:00 barentsz kern debug kernel - - - [525370.955135] dm-16: error: dax access failed (-95) 2019-06-12T09:58:27.977122+02:00 barentsz kern err kernel - - - [525370.955136] EXT4-fs (dm-16): DAX unsupported by block device. Turning off DAX. 2019-06-12T09:58:27.977124+02:00 barentsz kern info kernel - - - [525370.957006] EXT4-fs (dm-16): mounted filesystem with ordered data mode. Opts: dax Regards, Rob On 6/12/19 9:50 AM, Joel Colledge wrote:Hi Rob, This is strange, since the filesystem DAX access uses essentially the same checks as DRBD does. You can get more detail about the failure by doing the mount test again after enabling the relevant debug logging as follows: echo "file drivers/dax/super.c +p" > /sys/kernel/debug/dynamic_debug/control Perhaps there is a subtle difference in the checks that DRBD does which is relevant for your device on this kernel. Best regards, Joel On Wed, Jun 12, 2019 at 8:04 AM Rob van der Wal <[email protected] <mailto:[email protected]>> wrote: Hi Joel, Yes, I can mount with "-o dax": lvcreate -L 40G -n test-lv vg_ssd scan_dev_close /dev/sdb2 no DEV_IN_BCACHE set Logical volume "test-lv" created. mkfs.ext4 -b 4K /dev/vg_ssd/test-lv mount -o dax /dev/vg_ssd/test-lv /mnt df /mnt Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vg_ssd-test--lv 41022688 49176 38859976 1% /mnt But it shows a failure in syslog: [518270.833954] EXT4-fs (dm-16): DAX enabled. Warning: EXPERIMENTAL, use at your own risk [518270.833959] EXT4-fs (dm-16): DAX unsupported by block device. Turning off DAX. [518270.835599] EXT4-fs (dm-16): mounted filesystem with ordered data mode. Opts: dax Regards, Rob On 6/11/19 3:57 PM, Joel Colledge wrote:Hi Rob, This is strange. It seems that your LV is reporting that it supports PMEM/DAX. I suggest that you check that this issue also occurs without DRBD. For example, create a filesystem and try to mount it with DAX enabled: mkfs.ext4 -b 4K /dev/<your-lv> mount -o dax /dev/<your-lv> <somewhere> The check the syslog to see whether DAX was successfully enabled. You can see the expected success and failure output here: https://nvdimm.wiki.kernel.org/#filesystems Best regards, Joel PS Please "reply all" to keep the mailing list in CC On Tue, Jun 11, 2019 at 2:53 PM Rob van der Wal <[email protected]> <mailto:[email protected]> wrote:Hi Joel, I've build the module from source (downloaded fromhttps://www.linbit.com/en/drbd-community/drbd-download/). I think we don't use external metadata and we are using a ssd disk for storing. This is a part of our drbd.conf: drbd.conf: on barentsz { node-id 1; device /dev/drbd0 minor 0; disk /dev/vg_ssd/vm_prace-ldap; meta-disk internal; address ipv410.0.0.30:7788 <http://10.0.0.30:7788>; } LVM info: PV /dev/sdb2 VG vg_ssd lvm2 [446.43 GiB / 102.43 GiB free] ACTIVE '/dev/vg_ssd/vm_prace-ldap' [38.00 GiB] inherit The requested line in the syslog is: [1037978.736262] drbd r0/0 drbd0: meta-data IO uses: dax-pmem But I don't think we are using a PMEM device so this looks not correct. Regards, Rob van der Wal-- Rob van der WalSupercomputing | SURFsara | Science Park 140 | 1098 XG Amsterdam | T +31 20 800 1300 | [email protected] <mailto:[email protected]> | www.surfsara.nl <http://www.surfsara.nl> | We are ISO 27001 certified and meet the high requirements for information security.-- Joel Colledge - Software Developer+43-1-817-82-92 x53 <tel:+4318178292> [email protected] <mailto:[email protected]> LIN <http://www.linbit.com/en/>BIT <http://www.linbit.com/en/> | Keeping the Digital World Running DRBD HA - Disaster Recovery - Software-defined Storage t <https://twitter.com/linbit> / f <https://www.facebook.com/pg/linbitdrbd/posts/> / in <https://www.linkedin.com/company/linbit> / y <https://www.youtube.com/user/linbit> / g+ <https://plus.google.com/+Linbit/about> DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.-- Rob van der WalSupercomputing | SURFsara | Science Park 140 | 1098 XG Amsterdam | T +31 20 800 1300 | [email protected] <mailto:[email protected]> | www.surfsara.nl <http://www.surfsara.nl> | We are ISO 27001 certified and meet the high requirements for information security. -- Joel Colledge - Software Developer +43-1-817-82-92 x53 <tel:+4318178292> [email protected] <mailto:[email protected]>LIN <http://www.linbit.com/en/>BIT <http://www.linbit.com/en/> | Keeping the Digital World RunningDRBD HA - Disaster Recovery - Software-defined Storaget <https://twitter.com/linbit> / f <https://www.facebook.com/pg/linbitdrbd/posts/> / in <https://www.linkedin.com/company/linbit> / y <https://www.youtube.com/user/linbit> / g+ <https://plus.google.com/+Linbit/about>DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
-- Rob van der WalSupercomputing | SURFsara | Science Park 140 | 1098 XG Amsterdam | T +31 20 800 1300 | [email protected] | www.surfsara.nl |
We are ISO 27001 certified and meet the high requirements for information security.
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ drbd-user mailing list [email protected] http://lists.linbit.com/mailman/listinfo/drbd-user
