[Kernel-packages] [Bug 2084979] Re: Device Mapper accepts writes even when the underlying devices are dead (offline or transport-offline)
Hi, I have already reported this on kernel.org: https://bugzilla.kernel.org/show_bug.cgi?id=219486 BTW, if I use the O_DIRECT flag, dd returns I/O error. For example: root@ubutest:~# pvs;lvs PV VGFmt Attr PSize PFree /dev/sda3 ubuntu-vg lvm2 a-- <17.32g 4.00m /dev/sdb1 vgtestlvm2 a-- <2.00g0 LVVGAttr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert ubuntu-lv ubuntu-vg -wi-ao 17.31g lvtestvgtest-wi-a- <2.00g root@ubutest:~# echo offline > /sys/block/sdb/device/state root@ubutest:~# root@ubutest:~# dd if=f.tar of=/dev/mapper/vgtest-lvtest bs=10M count=204 oflag=direct ## oflag= dd: error writing '/dev/mapper/vgtest-lvtest': Input/output error 1+0 records in 0+0 records out 0 bytes copied, 0.00977631 s, 0.0 kB/s root@ubutest:~# -- Regards, Mirek ** Bug watch added: Linux Kernel Bug Tracker #219486 https://bugzilla.kernel.org/show_bug.cgi?id=219486 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2084979 Title: Device Mapper accepts writes even when the underlying devices are dead (offline or transport-offline) Status in linux package in Ubuntu: New Bug description: ** Steps to reproduce ** root@ubusrv:~# lsblk /dev/sda NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:00 2G 0 disk └─sda1 8:10 2G 0 part root@ubusrv:~# root@ubusrv:~# pvcreate /dev/sda1 Physical volume "/dev/sda1" successfully created. root@ubusrv:~# vgcreate vgtmp /dev/sda1 Volume group "vgtmp" successfully created root@ubusrv:~# lvcreate -n lvtmp -l 100%free vgtmp Logical volume "lvtmp" created. root@ubusrv:~# lvs LVVGAttr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert ubuntu-lv ubuntu-vg -wi-ao 18.22g lvtmp vgtmp -wi-a- <2.00g root@ubusrv:~# root@ubusrv:~# cat /sys/block/sda/device/state running root@ubusrv:~# echo offline > /sys/block/sda/device/state root@ubusrv:~# cat /sys/block/sda/device/state offline root@ubusrv:~# root@ubusrv:~# l file.tar -rw-r--r-- 1 root root 2.4G Oct 19 14:12 file.tar root@ubusrv:~# -- 1. Reading from faulty DM device: OK - expected behavior root@ubusrv:~# dd if=/dev/mapper/vgtmp-lvtmp dd: error reading '/dev/mapper/vgtmp-lvtmp': Input/output error 0+0 records in 0+0 records out 0 bytes copied, 0.00298229 s, 0.0 kB/s root@ubusrv:~# 2. Reading from faulty/offline underlying sda: OK - expected behavior root@ubusrv:~# dd if=/dev/sda1 dd: failed to open '/dev/sda1': No such device or address root@ubusrv:~# 3. Writing to faulty/offline underlying sda: OK - expected behavior root@ubusrv:~# dd of=/dev/sda1 dd: failed to open '/dev/sda1': No such device or address root@ubusrv:~# 4. And writing to faulty DM device: not OK - UNEXPECTED behavior root@ubusrv:~# dd if=file.tar of=/dev/mapper/vgtmp-lvtmp bs=10M count=204 204+0 records in 204+0 records out 2139095040 bytes (2.1 GB, 2.0 GiB) copied, 4.57038 s, 468 MB/s root@ubusrv:~# -- Where this data has been stored? Not in memory - not enough space: root@ubusrv:~# free totalusedfree shared buff/cache available Mem: 961Mi 361Mi 200Mi 1.3Mi 556Mi 599Mi Swap: 2.0Gi 0B 2.0Gi root@ubusrv:~# And the same applies to iSCSI+MPIO. -- root@ubusrv:~# multipath -ll mpatha (3600140582d6e7c50c0347908f0d094c2) dm-2 LIO-ORG,ubusrv-wrbk size=4.4G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=50 status=active |- 8:0:0:135 sdc 8:32 active ready running `- 7:0:0:135 sdb 8:16 active ready running mpathb (3600140512e2d3ad7ae048aba1ee0a33a) dm-3 LIO-ORG,ubusrv-wrth size=3.9G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=50 status=active |- 8:0:0:235 sdd 8:48 active ready running `- 7:0:0:235 sde 8:64 active ready running root@ubusrv:~# poweroff iSCSI target/server root@ubusrv:~# egrep 'mpath[ab]' /var/log/syslog | tail 2024-10-19T15:07:01.810687+02:00 ubusrv multipathd[382]: checker failed path 8:64 in map mpathb 2024-10-19T15:07:01.811658+02:00 ubusrv multipathd[382]: mpathb: remaining active paths: 1 2024-10-19T15:07:02.810464+02:00 ubusrv multipathd[382]: checker failed path 8:16 in map mpatha 2024-10-19T15:07:02.813151+02:00 ubusrv multipathd[382]: mpatha: Entering recovery mode: max_retries=12 2024-10-19T15:07:02.813182+02:00 ubusrv multipathd[382]: mpatha: remaining active paths: 0 2024-10-19T1
[Kernel-packages] [Bug 2084979] Re: Device Mapper accepts writes even when the underlying devices are dead (offline or transport-offline)
Hi Miroslaw, I finally got around to trying this out. I reproduced on: Focal with 5.4.0-200-generic Jammy with 5.15.0-122-generic Noble with 6.8.0-49-generic Plucky with 6.12.0-3-generic They all behave the same, and they all behave as you report in the bug report. I will note that every dmesg will report that the underlying disk is offline: [ 108.055848] I/O error, dev sda, sector 2048 op 0x1:(WRITE) flags 0x104000 phys_seg 32 prio class 0 [ 108.056279] Buffer I/O error on dev dm-0, logical block 0, lost async page write [ 108.056612] Buffer I/O error on dev dm-0, logical block 1, lost async page write [ 108.056996] Buffer I/O error on dev dm-0, logical block 2, lost async page write [ 108.057335] Buffer I/O error on dev dm-0, logical block 3, lost async page write [ 108.057694] Buffer I/O error on dev dm-0, logical block 4, lost async page write [ 108.058013] Buffer I/O error on dev dm-0, logical block 5, lost async page write [ 108.058342] Buffer I/O error on dev dm-0, logical block 6, lost async page write [ 108.058703] Buffer I/O error on dev dm-0, logical block 7, lost async page write [ 108.059023] Buffer I/O error on dev dm-0, logical block 8, lost async page write [ 108.059359] Buffer I/O error on dev dm-0, logical block 9, lost async page write [ 108.060034] I/O error, dev sda, sector 2304 op 0x1:(WRITE) flags 0x104000 phys_seg 32 prio class 0 [ 108.060549] I/O error, dev sda, sector 2560 op 0x1:(WRITE) flags 0x104000 phys_seg 32 prio class 0 [ 108.061012] I/O error, dev sda, sector 2816 op 0x1:(WRITE) flags 0x104000 phys_seg 32 prio class 0 [ 108.061465] I/O error, dev sda, sector 3072 op 0x1:(WRITE) flags 0x104000 phys_seg 32 prio class 0 [ 108.061908] I/O error, dev sda, sector 3328 op 0x1:(WRITE) flags 0x104000 phys_seg 32 prio class 0 [ 108.062318] I/O error, dev sda, sector 3584 op 0x1:(WRITE) flags 0x104000 phys_seg 32 prio class 0 [ 108.062744] I/O error, dev sda, sector 3840 op 0x1:(WRITE) flags 0x104000 phys_seg 32 prio class 0 [ 108.063188] I/O error, dev sda, sector 4096 op 0x1:(WRITE) flags 0x104000 phys_seg 32 prio class 0 [ 108.063616] I/O error, dev sda, sector 4352 op 0x1:(WRITE) flags 0x104000 phys_seg 32 prio class 0 Its just that dd succeeds, and exits 0. A bit frustrating I suppose. Since this affects the very latest mainline, we should probably ask upstream about it. Will you write to the upstream device mapper maintainers? Thanks, Matthew -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2084979 Title: Device Mapper accepts writes even when the underlying devices are dead (offline or transport-offline) Status in linux package in Ubuntu: New Bug description: ** Steps to reproduce ** root@ubusrv:~# lsblk /dev/sda NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:00 2G 0 disk └─sda1 8:10 2G 0 part root@ubusrv:~# root@ubusrv:~# pvcreate /dev/sda1 Physical volume "/dev/sda1" successfully created. root@ubusrv:~# vgcreate vgtmp /dev/sda1 Volume group "vgtmp" successfully created root@ubusrv:~# lvcreate -n lvtmp -l 100%free vgtmp Logical volume "lvtmp" created. root@ubusrv:~# lvs LVVGAttr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert ubuntu-lv ubuntu-vg -wi-ao 18.22g lvtmp vgtmp -wi-a- <2.00g root@ubusrv:~# root@ubusrv:~# cat /sys/block/sda/device/state running root@ubusrv:~# echo offline > /sys/block/sda/device/state root@ubusrv:~# cat /sys/block/sda/device/state offline root@ubusrv:~# root@ubusrv:~# l file.tar -rw-r--r-- 1 root root 2.4G Oct 19 14:12 file.tar root@ubusrv:~# -- 1. Reading from faulty DM device: OK - expected behavior root@ubusrv:~# dd if=/dev/mapper/vgtmp-lvtmp dd: error reading '/dev/mapper/vgtmp-lvtmp': Input/output error 0+0 records in 0+0 records out 0 bytes copied, 0.00298229 s, 0.0 kB/s root@ubusrv:~# 2. Reading from faulty/offline underlying sda: OK - expected behavior root@ubusrv:~# dd if=/dev/sda1 dd: failed to open '/dev/sda1': No such device or address root@ubusrv:~# 3. Writing to faulty/offline underlying sda: OK - expected behavior root@ubusrv:~# dd of=/dev/sda1 dd: failed to open '/dev/sda1': No such device or address root@ubusrv:~# 4. And writing to faulty DM device: not OK - UNEXPECTED behavior root@ubusrv:~# dd if=file.tar of=/dev/mapper/vgtmp-lvtmp bs=10M count=204 204+0 records in 204+0 records out 2139095040 bytes (2.1 GB, 2.0 GiB) copied, 4.57038 s, 468 MB/s root@ubusrv:~# -- Where this data has been stored? Not in memory - not enough space: root@ubusrv:~# free total
[Kernel-packages] [Bug 2084979] Re: Device Mapper accepts writes even when the underlying devices are dead (offline or transport-offline)
No ideas? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2084979 Title: Device Mapper accepts writes even when the underlying devices are dead (offline or transport-offline) Status in linux package in Ubuntu: New Bug description: ** Steps to reproduce ** root@ubusrv:~# lsblk /dev/sda NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:00 2G 0 disk └─sda1 8:10 2G 0 part root@ubusrv:~# root@ubusrv:~# pvcreate /dev/sda1 Physical volume "/dev/sda1" successfully created. root@ubusrv:~# vgcreate vgtmp /dev/sda1 Volume group "vgtmp" successfully created root@ubusrv:~# lvcreate -n lvtmp -l 100%free vgtmp Logical volume "lvtmp" created. root@ubusrv:~# lvs LVVGAttr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert ubuntu-lv ubuntu-vg -wi-ao 18.22g lvtmp vgtmp -wi-a- <2.00g root@ubusrv:~# root@ubusrv:~# cat /sys/block/sda/device/state running root@ubusrv:~# echo offline > /sys/block/sda/device/state root@ubusrv:~# cat /sys/block/sda/device/state offline root@ubusrv:~# root@ubusrv:~# l file.tar -rw-r--r-- 1 root root 2.4G Oct 19 14:12 file.tar root@ubusrv:~# -- 1. Reading from faulty DM device: OK - expected behavior root@ubusrv:~# dd if=/dev/mapper/vgtmp-lvtmp dd: error reading '/dev/mapper/vgtmp-lvtmp': Input/output error 0+0 records in 0+0 records out 0 bytes copied, 0.00298229 s, 0.0 kB/s root@ubusrv:~# 2. Reading from faulty/offline underlying sda: OK - expected behavior root@ubusrv:~# dd if=/dev/sda1 dd: failed to open '/dev/sda1': No such device or address root@ubusrv:~# 3. Writing to faulty/offline underlying sda: OK - expected behavior root@ubusrv:~# dd of=/dev/sda1 dd: failed to open '/dev/sda1': No such device or address root@ubusrv:~# 4. And writing to faulty DM device: not OK - UNEXPECTED behavior root@ubusrv:~# dd if=file.tar of=/dev/mapper/vgtmp-lvtmp bs=10M count=204 204+0 records in 204+0 records out 2139095040 bytes (2.1 GB, 2.0 GiB) copied, 4.57038 s, 468 MB/s root@ubusrv:~# -- Where this data has been stored? Not in memory - not enough space: root@ubusrv:~# free totalusedfree shared buff/cache available Mem: 961Mi 361Mi 200Mi 1.3Mi 556Mi 599Mi Swap: 2.0Gi 0B 2.0Gi root@ubusrv:~# And the same applies to iSCSI+MPIO. -- root@ubusrv:~# multipath -ll mpatha (3600140582d6e7c50c0347908f0d094c2) dm-2 LIO-ORG,ubusrv-wrbk size=4.4G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=50 status=active |- 8:0:0:135 sdc 8:32 active ready running `- 7:0:0:135 sdb 8:16 active ready running mpathb (3600140512e2d3ad7ae048aba1ee0a33a) dm-3 LIO-ORG,ubusrv-wrth size=3.9G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=50 status=active |- 8:0:0:235 sdd 8:48 active ready running `- 7:0:0:235 sde 8:64 active ready running root@ubusrv:~# poweroff iSCSI target/server root@ubusrv:~# egrep 'mpath[ab]' /var/log/syslog | tail 2024-10-19T15:07:01.810687+02:00 ubusrv multipathd[382]: checker failed path 8:64 in map mpathb 2024-10-19T15:07:01.811658+02:00 ubusrv multipathd[382]: mpathb: remaining active paths: 1 2024-10-19T15:07:02.810464+02:00 ubusrv multipathd[382]: checker failed path 8:16 in map mpatha 2024-10-19T15:07:02.813151+02:00 ubusrv multipathd[382]: mpatha: Entering recovery mode: max_retries=12 2024-10-19T15:07:02.813182+02:00 ubusrv multipathd[382]: mpatha: remaining active paths: 0 2024-10-19T15:07:04.811717+02:00 ubusrv multipathd[382]: checker failed path 8:48 in map mpathb 2024-10-19T15:07:04.813169+02:00 ubusrv multipathd[382]: mpathb: Entering recovery mode: max_retries=12 2024-10-19T15:07:04.813206+02:00 ubusrv multipathd[382]: mpathb: remaining active paths: 0 2024-10-19T15:08:02.844668+02:00 ubusrv multipathd[382]: mpatha: Disable queueing 2024-10-19T15:08:04.845680+02:00 ubusrv multipathd[382]: mpathb: Disable queueing root@ubusrv:~# root@ubusrv:~# iscsiadm -m session -P 3 | grep Att Attached SCSI devices: Attached scsi disk sdb State: transport-offline Attached scsi disk sde State: transport-offline Attached SCSI devices: Attached scsi disk sdc State: transport-offline Attached scsi disk sdd State: transport-offline ro
[Kernel-packages] [Bug 2084979] Re: Device Mapper accepts writes even when the underlying devices are dead (offline or transport-offline)
## It gets even worse - DM accepts writes even if the underlying disk has been removed Here is an example root@ubusrv:~# l /dev/sda* brw-rw 1 root disk 8, 0 Oct 26 09:06 /dev/sda brw-rw 1 root disk 8, 1 Oct 26 09:06 /dev/sda1 root@ubusrv:~# pvs PV VGFmt Attr PSize PFree /dev/sda1 vgtmp lvm2 a-- <2.00g0 /dev/vda3 ubuntu-vg lvm2 a-- 18.22g0 root@ubusrv:~# lvs LVVGAttr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert ubuntu-lv ubuntu-vg -wi-ao 18.22g lvtmp vgtmp -wi-a- <2.00g root@ubusrv:~# l /dev/mapper/vgtmp-lvtmp lrwxrwxrwx 1 root root 7 Oct 26 09:06 /dev/mapper/vgtmp-lvtmp -> ../dm-1 DM link root@ubusrv:~# Removing the underlying DM disk /dev/sda (in my case: deleting from the OS or forced removal via hypervisor) root@ubusrv:~# echo 1 > /sys/block/sda/device/delete root@ubusrv:~# l /dev/sda*## sda removed from the OS ls: cannot access '/dev/sda*': No such file or directory root@ubusrv:~# root@ubusrv:~# pvs## /dev/sda1 PV disappeared PV VGFmt Attr PSize PFree /dev/vda3 ubuntu-vg lvm2 a-- 18.22g0 root@ubusrv:~# root@ubusrv:~# lvs## lvtmp LV is also gone - OK LVVGAttr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert ubuntu-lv ubuntu-vg -wi-ao 18.22g root@ubusrv:~# root@ubusrv:~# l /dev/mapper/vgtmp-lvtmp## DM link still exists lrwxrwxrwx 1 root root 7 Oct 26 09:06 /dev/mapper/vgtmp-lvtmp -> ../dm-1 root@ubusrv:~# l /dev/dm-1 brw-rw 1 root disk 252, 1 Oct 26 09:06 /dev/dm-1 root@ubusrv:~# Writing to faulty DM device root@ubusrv:~# l file.tar -rw-r--r-- 1 root root 2.4G Oct 19 14:12 file.tar root@ubusrv:~# root@ubusrv:~# dd if=file.tar of=/dev/mapper/vgtmp-lvtmp bs=10M count=204 204+0 records in 204+0 records out 2139095040 bytes (2.1 GB, 2.0 GiB) copied, 4.33304 s, 494 MB/s completed successfully?! root@ubusrv:~# root@ubusrv:~# grep kernel /var/log/syslog | tail this means that the kernel knows that writes are failing 2024-10-26T10:36:06.405786+02:00 ubusrv kernel: Buffer I/O error on dev dm-1, logical block 0, lost async page write 2024-10-26T10:36:06.405845+02:00 ubusrv kernel: Buffer I/O error on dev dm-1, logical block 1, lost async page write 2024-10-26T10:36:06.405861+02:00 ubusrv kernel: Buffer I/O error on dev dm-1, logical block 2, lost async page write 2024-10-26T10:36:06.405876+02:00 ubusrv kernel: Buffer I/O error on dev dm-1, logical block 3, lost async page write 2024-10-26T10:36:06.405890+02:00 ubusrv kernel: Buffer I/O error on dev dm-1, logical block 4, lost async page write 2024-10-26T10:36:06.405905+02:00 ubusrv kernel: Buffer I/O error on dev dm-1, logical block 5, lost async page write 2024-10-26T10:36:06.405919+02:00 ubusrv kernel: Buffer I/O error on dev dm-1, logical block 6, lost async page write 2024-10-26T10:36:06.405972+02:00 ubusrv kernel: Buffer I/O error on dev dm-1, logical block 7, lost async page write 2024-10-26T10:36:06.406022+02:00 ubusrv kernel: Buffer I/O error on dev dm-1, logical block 8, lost async page write 2024-10-26T10:36:06.406035+02:00 ubusrv kernel: Buffer I/O error on dev dm-1, logical block 9, lost async page write root@ubusrv:~# I can only repeat the question: where has this data been stored? It's a very dangerous "feature". It may cause data loss, loss of data consistency, e.g. in DB applications. -- Regards, Mirek -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2084979 Title: Device Mapper accepts writes even when the underlying devices are dead (offline or transport-offline) Status in linux package in Ubuntu: New Bug description: ** Steps to reproduce ** root@ubusrv:~# lsblk /dev/sda NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:00 2G 0 disk └─sda1 8:10 2G 0 part root@ubusrv:~# root@ubusrv:~# pvcreate /dev/sda1 Physical volume "/dev/sda1" successfully created. root@ubusrv:~# vgcreate vgtmp /dev/sda1 Volume group "vgtmp" successfully created root@ubusrv:~# lvcreate -n lvtmp -l 100%free vgtmp Logical volume "lvtmp" created. root@ubusrv:~# lvs LVVGAttr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert ubuntu-lv ubuntu-vg -wi-ao 18.22g lvtmp vgtmp -wi-a- <2.00g root@ubusrv:~# root@ubusrv:~# cat /sys/block/sda/device/state running root@ubusrv:~# echo offline > /sys/block/sda/device/state root@ubusrv:~# cat /sys/block/sda/device/state offline root@ubusrv:~# root@ubusrv:~# l
