[Bug 1887558] Re: Multipath JBOD storage devices are not shown via /dev/mapper but each path as a single device.
Any objection to renaming this bug title to something more accurate such as "multipathd bcache disks do not get picked up by multipath-tools during boot"? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1887558 Title: Multipath JBOD storage devices are not shown via /dev/mapper but each path as a single device. To manage notifications about this bug go to: https://bugs.launchpad.net/bcache-tools/+bug/1887558/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1887558] Re: Multipath JBOD storage devices are not shown via /dev/mapper but each path as a single device.
I have successfully created a kvm based reproducer.
1) get an image
$ wget
https://cdimage.ubuntu.com/ubuntu-server/daily-live/pending/noble-live-server-amd64.iso
2) Create bcache disk
$ fallocate -l 20G image.img
$ make-bcache -C image.img
3) Boot into iso with multipathed bcache disk
$ kvm -m 2048 -boot d -cdrom ./noble-live-server-amd64.iso -device
virtio-scsi-pci,id=scsi -drive
file=image.img,if=none,id=sda,format=raw,file.locking=off -device
scsi-hd,drive=sda,serial=0001 -drive
if=none,id=sdb,file=image.img,format=raw,file.locking=off -device
scsi-hd,drive=sdb,serial=0001
Select "Try or install Ubuntu" wait for the OS to load. Press F2 to
escape to a shell to check multipathed devices.
You will see exactly what this bug describes - /dev/sd{a,b} is present
with no multipathing.
Redo these steps without make-bcache on image.img and you will see a
proper multipath /dev/mapper/mpatha present.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1887558
Title:
Multipath JBOD storage devices are not shown via /dev/mapper but each
path as a single device.
To manage notifications about this bug go to:
https://bugs.launchpad.net/bcache-tools/+bug/1887558/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1887558] Re: Multipath JBOD storage devices are not shown via /dev/mapper but each path as a single device.
Fundamentally my setup is different than the scenario set up in the bug.
I was initially confused why multipathing was even happening, so I will
explain the scenario here. In case someone else is also confused.
The hardware is a JBOD enclosure where each disk has redundant I/O,
hence the multipathing.
On boot, multipath correctly groups
* /dev/sd{a,b} -> /dev/mapper/mpatha
* /dev/sd{c,d} -> /dev/mapper/mpathb
* /dev/sd{e,f} -> /dev/mapper/mpathc
* /dev/sd{g,h} -> /dev/mapper/mpathd
At this point, the system has 4 NVME drives to use as the caching device, where
the JBOD drives are the backing device.
make-bcache -C /dev/nvme0n1p1 -B /dev/mapper/mpatha --writeback
make-bcache -C /dev/nvme1n1p1 -B /dev/mapper/mpathb --writeback
make-bcache -C /dev/nvme2n1p1 -B /dev/mapper/mpathc --writeback
make-bcache -C /dev/nvme3n1p1 -B /dev/mapper/mpathd --writeback
At this point, everything is happy. We have multipathd devices inside a
JBOD enclosure, where the physical disks are tagged as bcache devices.
The issue seems to be happening when you reboot the system, and the
bcache udev rules grab the drives and multipath fails to make
mpath{a,b,c,d} - because the drives are in use by bcache.
I don't think I see this on my system because this is an issue during
boot when multipathd is not running.
With that said, the fact that the disks are part of a JBOD enclosure I
don't think is an issue here, rather on boot disks with redundant i/o
that are marked with bcache FS do not get initialized correctly, so I
think there is still a way to reproduce this in a VM which would be nice
so we can port this into a proper dep8 tests.
The problem with reboot testing in my setup is that on boot the virtual
iscsi device is not available, I have to login manually. I'm not the
most familiar with tgtadm and iscsiadm so not sure if these tools can
make the device available at boot time which seems to be needed for this
test.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1887558
Title:
Multipath JBOD storage devices are not shown via /dev/mapper but each
path as a single device.
To manage notifications about this bug go to:
https://bugs.launchpad.net/bcache-tools/+bug/1887558/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1887558] Re: Multipath JBOD storage devices are not shown via /dev/mapper but each path as a single device.
I attempted to reproduce this issue in a VM and wasn't succesful. I
essentially copied the tgtbasedmpaths test but made the backing disk
bcache cache before setting up the targets.
1) create vm
$ lxc launch ubuntu-daily:noble --vm n-vm
$ lxc shell n-vm
# apt install -y lsscsi multipath-tools open-iscsi tgt
2) setup virtual multipathd disk
```
targetname="iqn.2016-11.foo.com:target.iscsi"
cwd=$(pwd)
testdir="/mnt/tgtmpathtest"
localhost="127.0.0.1"
portal="${localhost}:3260"
maxpaths=4
backfn="backingfile"
expectwwid="6e010001"
testdisk="/dev/disk/by-id/wwn-0x${expectwwid}"
### Setup mpath devices
# Restart tgtd to make sure modules are all loaded
service tgt restart || echo "Failed to restart tgt" >&2
# prep SINGLE test file
truncate --size 100M ${backfn}
make-bcache -C ${backfn}
# create target
tgtadm --lld iscsi --op new --mode target --tid 1 -T "${targetname}"
# allow all tomake-bcache -C ${backfn}
bind the target
tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL
# set backing file
tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b
"${cwd}/${backfn}"
# scan for targets (locally)
iscsiadm --mode discovery --type sendtargets --portal ${localhost}
# login
echo "login #1"
iscsiadm --mode node --targetname "${targetname}" --portal ${portal} --login
# duplicate this session (always 1)
for i in $(seq 2 ${maxpaths})
do
echo "extra login #${i}"
iscsiadm --mode session -r 1 --op new
done
udevadm settle
sleep 5 # sleep a bit to allow device to be created.
```
And can confirm the disks are using bcache FS by:
# udevadm info /dev/dm-0 | grep ID_FS_TYPE
E: ID_FS_TYPE=bcache
Things look fine and as they should to me
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda8:00 10G 0 disk
├─sda1 8:109G 0 part /
├─sda148:14 04M 0 part
├─sda158:15 0 106M 0 part /boot/efi
└─sda16 259:00 913M 0 part /boot
sdb8:16 0 100M 0 disk
└─mpatha 252:00 100M 0 mpath
sdc8:32 0 100M 0 disk
└─mpatha 252:00 100M 0 mpath
sdd8:48 0 100M 0 disk
└─mpatha 252:00 100M 0 mpath
sde8:64 0 100M 0 disk
└─mpatha 252:00 100M 0 mpath
# multipath -ll
mpatha (36e010001) dm-0 IET,VIRTUAL-DISK
size=100M features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 10:0:0:1 sde 8:64 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 7:0:0:1 sdb 8:16 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 8:0:0:1 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
`- 9:0:0:1 sdd 8:48 active ready running
# ls /dev/mapper
control mpatha
I'm not sure if I'm setting up this test incorrectly or not.
In fact if I run `udevadm test` I see the bcache udev rules fail.
# udevadm test /dev/mapper/mpatha
< cut here >
dm-0: /usr/lib/udev/rules.d/69-bcache.rules:21 RUN 'kmod load bcache'
dm-0: /usr/lib/udev/rules.d/69-bcache.rules:22 RUN 'bcache-register $tempnode'
dm-0: /usr/lib/udev/rules.d/69-bcache.rules:26 Importing properties from
results of 'bcache-export-cached /dev/dm-0'
dm-0: Starting 'bcache-export-cached /dev/dm-0'
Successfully forked off '(spawn)' as PID 2197.
dm-0: Process 'bcache-export-cached /dev/dm-0' failed with exit code 1.
dm-0: /usr/lib/udev/rules.d/69-bcache.rules:26 Command "bcache-export-cached
/dev/dm-0" returned 1 (error), ignoring
< cut here >
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1887558
Title:
Multipath JBOD storage devices are not shown via /dev/mapper but each
path as a single device.
To manage notifications about this bug go to:
https://bugs.launchpad.net/bcache-tools/+bug/1887558/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1887558] Re: Multipath JBOD storage devices are not shown via /dev/mapper but each path as a single device.
** Changed in: maas/3.3 Status: Triaged => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1887558 Title: Multipath JBOD storage devices are not shown via /dev/mapper but each path as a single device. To manage notifications about this bug go to: https://bugs.launchpad.net/bcache-tools/+bug/1887558/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1887558] Re: Multipath JBOD storage devices are not shown via /dev/mapper but each path as a single device.
** Merge proposal linked: https://code.launchpad.net/~alexsander-souza/maas/+git/maas/+merge/463615 ** Merge proposal linked: https://code.launchpad.net/~alexsander-souza/maas/+git/maas/+merge/459661 ** Changed in: maas/3.4 Status: Triaged => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1887558 Title: Multipath JBOD storage devices are not shown via /dev/mapper but each path as a single device. To manage notifications about this bug go to: https://bugs.launchpad.net/bcache-tools/+bug/1887558/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1887558] Re: Multipath JBOD storage devices are not shown via /dev/mapper but each path as a single device.
** Tags removed: server-triage-discuss -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1887558 Title: Multipath JBOD storage devices are not shown via /dev/mapper but each path as a single device. To manage notifications about this bug go to: https://bugs.launchpad.net/bcache-tools/+bug/1887558/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1887558] Re: Multipath JBOD storage devices are not shown via /dev/mapper but each path as a single device.
> I'd expect a modified variant of my multipath/tgt autopkgtest should work > using iscsi to create multiple > paths to the same disk We have been chasing such a setup for a while (Sergio and I), but didn't manage to get it working. Do you have a simple recipe for such a setup, Christian? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1887558 Title: Multipath JBOD storage devices are not shown via /dev/mapper but each path as a single device. To manage notifications about this bug go to: https://bugs.launchpad.net/bcache-tools/+bug/1887558/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1887558] Re: Multipath JBOD storage devices are not shown via /dev/mapper but each path as a single device.
** Tags added: server-triage-discuss -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1887558 Title: Multipath JBOD storage devices are not shown via /dev/mapper but each path as a single device. To manage notifications about this bug go to: https://bugs.launchpad.net/bcache-tools/+bug/1887558/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1887558] Re: Multipath JBOD storage devices are not shown via /dev/mapper but each path as a single device.
** Changed in: maas/3.3 Assignee: (unassigned) => Alexsander de Souza (alexsander-souza) ** Changed in: maas/3.4 Assignee: (unassigned) => Alexsander de Souza (alexsander-souza) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1887558 Title: Multipath JBOD storage devices are not shown via /dev/mapper but each path as a single device. To manage notifications about this bug go to: https://bugs.launchpad.net/bcache-tools/+bug/1887558/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1887558] Re: Multipath JBOD storage devices are not shown via /dev/mapper but each path as a single device.
** Changed in: maas Milestone: 3.5.0 => 3.5.0-beta1 ** Changed in: maas Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1887558 Title: Multipath JBOD storage devices are not shown via /dev/mapper but each path as a single device. To manage notifications about this bug go to: https://bugs.launchpad.net/bcache-tools/+bug/1887558/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
