[ +cc Heinz, Mikulas ]
> > On Tue, 5 Dec 2023, Joe Thornber wrote:
> > > Hi Eric,
> > >
> > > I just released v1.0.8 of the thinp tools:
> > >
> > > https://github.com/jthornber/thin-provisioning-tools
> >
> > Interesting, so do these tools play a role in the kernel somehow (ie,
> > kernel rust), or is this just userspace testing?
>
> The tools issue IO. LVM runs thin_check during one of the lvcreate
> calls in your reproducer script. The way the tools were issuing io was
> triggering the dm-integrity error. The fact that it then hung we
> currently believe is an md kernel bug. Mikulas and Heinz are looking
> into it.
Hi Mikulas and Heinz,
Joe mentioned that you were looking into this. Have you been able to
figure out what might be causing the strange md<=>dm-integrity interaction
in the kernel?
-Eric
> On Fri, 1 Dec 2023, David Teigland wrote:
> > On Thu, Nov 30, 2023 at 04:40:00PM -0800, Eric Wheeler wrote:
> > > integritysetup format $idev0 --integrity xxhash64 --batch-mode
> > > integritysetup format $idev1 --integrity xxhash64 --batch-mode
> > >
> > > integritysetup open --integrity xxhash64 --allow-discards $idev0
> > > dm-integrity0
> > > integritysetup open --integrity xxhash64 --allow-discards $idev1
> > > dm-integrity1
> > >
> > > mdadm --create $MD_DEV --metadata=1.2 --assume-clean --level=1
> > > --raid-devices=2 /dev/mapper/dm-integrity[01]
> > >
> > > # 1. This should be enough to trigger it:
> > > SSD_DEV=$MD_DEV
> > >
> > > # 2. If not, then wrap /dev/md9 in a linear target:
> > > #linear_add ssd $MD_DEV
> > > #SSD_DEV=/dev/mapper/ssd
> >
> > Interesting that just adding a linear layer there would have some effect.
>
> Its seemed (sometimes) to be the difference between triggering on the
> first instead of the second iteration. Without the linear wrap, it would
> sometimes not crash until the second loop, typically on `lvchange -an
> testvg`. Maybe the extra linear hop delayed the IO just enough to help
> with the race, but that is only speculation.
>
> > > # Create a writable header for the PV meta:
> > > dd if=/dev/zero bs=1M count=16 oflag=direct of=/tmp/pvheader
> > > loop=`losetup -f --show /tmp/pvheader`
> > > linear_add pv $loop /dev/nullb0
> > >
> > > # Create the VG
> > > lvmdevices --adddev $SSD_DEV
> > > lvmdevices --adddev /dev/mapper/pv
> > > vgcreate $VGNAME /dev/mapper/pv $SSD_DEV
> > >
> > > # Create the pool:
> > > lvcreate -n pool0 -L 1T $VGNAME /dev/mapper/pv
> > > lvcreate -n meta0 -L 512m $VGNAME $SSD_DEV
> > >
> > > # Make sure the meta volume is on the SSD (it should be already from
> > > above):
> > > pvmove -n meta0 /dev/mapper/pv
> >
> > I'd omit that pvmove if possible just in case it makes some unexpected
> > change. You have more than enough layers to complicate things as it is
> > without pvmove adding dm-mirror to the mix.
>
> Good point. Since we specify the PV with `lvcreate ... $SSD_DEV`, the
> pvmove isn't necessary at all. (Specifying the PV on lvcreate was added
> later, pvmove was leftover from earlier testing.)
>
> >
> > > lvconvert -y --force --force --chunksize 64k --type thin-pool
> > > --poolmetadata $VGNAME/meta0 $VGNAME/pool0
> >
> > It's not a bad idea to use mdraid over dm-integrity, but it would be
> > interesting to know if doing raid+integrity in lvm would have the same
> > problems. e.g.
> >
> >
> > lvcreate --type raid1 --raidintegrity y -m1 -L 512m -n meta0 $vg
> > /dev/ram[01]
> > lvcreate -n pool0 -L 1T $vg /dev/nullb0
> > lvconvert --type thin-pool --poolmetadata meta0 $vg/pool0
>
> Good idea! As it turns out, that crashes just as easily. Here is the
> script with LVM based RAID1, which is somewhat simpler:
>
> --------------------------------------------------------------------
> #!/bin/bash
>
> # Notice: /dev/ram0 and /dev/ram1 will be wiped unconditionally.
>
> # Configure these if you need to:
> VGNAME=testvg
> LVNAME=thin
> LVSIZE=$((10 * 1024*1024*1024/512))
>
> echo "NOTICE: THIS MAY BE UNSAFE. ONLY RUN THIS IN A TEST ENVIRONMENT!"
> echo "Press enter twice to continue or CTRL-C to abort."
>
> read
> read
>
> set -x
>
>
> # append disks into a linear target
> linear_add()
> {
> name=$1
> shift
>
> prevsize=0
> for vol in "$@"; do
> size=`blockdev --getsize $vol`
> echo "$prevsize $size linear $vol 0"
> prevsize=$size
> done \
> | dmsetup create $name
>
> echo /dev/mapper/$name
> }
>
> lvthin_add()
> {
> id=$1
> lvcreate -An -V $LVSIZE -n $LVNAME$id --thinpool pool0 $VGNAME >&2
> echo /dev/$VGNAME/$LVNAME$id
> }
>
> lvthin_snapshot()
> {
> origin=$1
> id=$2
>
> lvcreate -An -s $VGNAME/$LVNAME$origin -n $LVNAME$id >&2
> echo /dev/$VGNAME/$LVNAME$id
> }
>
> fio()
> {
> dev=$1
> /bin/fio --name=$dev --rw=randrw --direct=1 --bs=512 --numjobs=1
> --filename=$dev --time_based --runtime=$FIOTIME --ioengine=libaio --iodepth=1
> &> /dev/null
> }
>
> do_reset()
> {
> killall -9 fio
> lvchange -an $VGNAME
> rmdir /dev/$VGNAME
> dmsetup remove pv
> lvmdevices --deldev /dev/mapper/pv
> losetup -d /dev/loop?
> rmmod brd
> rmmod null_blk
> echo ==== reset done
> sleep 1
> }
>
> do_init()
> {
> modprobe null_blk gb=30000 bs=512
>
> ramsize_gb=1
> modprobe brd rd_size=$(($ramsize_gb * 1024*1024)) rd_nr=2
>
> # Create a writable header for the PV meta:
> dd if=/dev/zero bs=1M count=16 oflag=direct of=/tmp/pvheader
> loop=`losetup -f --show /tmp/pvheader`
> linear_add pv $loop /dev/nullb0
>
> # Create the VG
> lvmdevices --adddev /dev/ram0
> lvmdevices --adddev /dev/ram1
> lvmdevices --adddev /dev/mapper/pv
> vgcreate $VGNAME /dev/mapper/pv /dev/ram[01]
>
> # Create the pool:
> lvcreate -n pool0 -L 1T $VGNAME /dev/mapper/pv
> lvcreate --type raid1 --raidintegrity y -m1 -L 512m -n meta0 $VGNAME
> /dev/ram[01]
>
> lvconvert -y --force --force --chunksize 64k --type thin-pool
> --poolmetadata $VGNAME/meta0 $VGNAME/pool0
> }
>
>
> while true; do
> do_reset
> do_init
>
> thin1=`lvthin_add 1`
> fio $thin1 &
>
> thin2=`lvthin_snapshot 1 2`
> fio $thin2 &
>
> wait
>
> done
>
>
>
