Re: [PATCH 0/7] fstests: test Btrfs swapfile support

[Eryu Guan]

On Tue, Nov 06, 2018 at 02:06:30PM +0100, David Sterba wrote:
> On Mon, Nov 05, 2018 at 12:09:31AM +0800, Eryu Guan wrote:
> > On Fri, Nov 02, 2018 at 02:29:35PM -0700, Omar Sandoval wrote:
> > > From: Omar Sandoval 
> > > 
> > > This series fixes a couple of generic swapfile tests and adds some
> > > Btrfs-specific swapfile tests. Btrfs swapfile support is scheduled for
> > > 4.21 [1].
> > > 
> > > 1: https://www.spinics.net/lists/linux-btrfs/msg83454.html
> > > 
> > > Thanks!
> > 
> > Thanks for the fixes and new tests!
> > 
> > > 
> > > Omar Sandoval (7):
> > >   generic/{472,496,497}: fix $seeqres typo
> > >   generic/{472,496}: fix swap file creation on Btrfs
> > 
> > I've merged above two patches, they're two obvious bug fixes.
> > 
> > >   btrfs: test swap file activation restrictions
> > >   btrfs: test invalid operations on a swap file
> > >   btrfs: test swap files on multiple devices
> > >   btrfs: test device add/remove/replace with an active swap file
> > >   btrfs: test balance and resize with an active swap file
> > 
> > These tests look fine to me, but it'd be really great if btrfs folks
> > could help review above tests and provide Reviewed-by tags.
> 
> All look good to me,
> 
> Reviewed-by: David Sterba 

Thanks a lot for the review!

> 
> A few nits I saw:
> 
> - the command names should not be shortened, ie. 'btrfs subvolume
>   snapshot' instead of 'btrfs subvol snap'
> - the test description (eg. 3/7 and 4/7) could mention which case is
>   tested, eg. swapfile with compression or COW or snapshot
> - 4/7 has typo 'nowcow' in a comment

I've addressed these nits on commit.

Thanks,
Eryu

Re: [PATCH 0/7] fstests: test Btrfs swapfile support

[Eryu Guan]

On Fri, Nov 02, 2018 at 02:29:35PM -0700, Omar Sandoval wrote:
> From: Omar Sandoval 
> 
> This series fixes a couple of generic swapfile tests and adds some
> Btrfs-specific swapfile tests. Btrfs swapfile support is scheduled for
> 4.21 [1].
> 
> 1: https://www.spinics.net/lists/linux-btrfs/msg83454.html
> 
> Thanks!

Thanks for the fixes and new tests!

> 
> Omar Sandoval (7):
>   generic/{472,496,497}: fix $seeqres typo
>   generic/{472,496}: fix swap file creation on Btrfs

I've merged above two patches, they're two obvious bug fixes.

>   btrfs: test swap file activation restrictions
>   btrfs: test invalid operations on a swap file
>   btrfs: test swap files on multiple devices
>   btrfs: test device add/remove/replace with an active swap file
>   btrfs: test balance and resize with an active swap file

These tests look fine to me, but it'd be really great if btrfs folks
could help review above tests and provide Reviewed-by tags.

And perhaps we could add test 17[56] to 'volume' group, as they do
device operations. And similarly, add the last test to 'balance' group.

Thanks,
Eryu

> 
>  tests/btrfs/173 | 55 ++
>  tests/btrfs/173.out |  5 +++
>  tests/btrfs/174 | 66 
>  tests/btrfs/174.out | 10 ++
>  tests/btrfs/175 | 73 
>  tests/btrfs/175.out |  8 +
>  tests/btrfs/176 | 82 +
>  tests/btrfs/176.out |  5 +++
>  tests/btrfs/177 | 64 +++
>  tests/btrfs/177.out |  6 
>  tests/btrfs/group   |  5 +++
>  tests/generic/472   | 16 -
>  tests/generic/496   |  8 ++---
>  tests/generic/497   |  2 +-
>  14 files changed, 391 insertions(+), 14 deletions(-)
>  create mode 100755 tests/btrfs/173
>  create mode 100644 tests/btrfs/173.out
>  create mode 100755 tests/btrfs/174
>  create mode 100644 tests/btrfs/174.out
>  create mode 100755 tests/btrfs/175
>  create mode 100644 tests/btrfs/175.out
>  create mode 100755 tests/btrfs/176
>  create mode 100644 tests/btrfs/176.out
>  create mode 100755 tests/btrfs/177
>  create mode 100644 tests/btrfs/177.out
> 
> -- 
> 2.19.1
> 

Re: [PATCH v2 rev log added] fstests: btrfs verify hardening agaist duplicate fsid

[Eryu Guan]

On Tue, Oct 09, 2018 at 02:28:21AM +0800, Anand Jain wrote:
> We have a known bug in btrfs, that we let the device path be changed
> after the device has been mounted. So using this loop hole the new
> copied device would appears as if its mounted immediately after its
> been copied. So this test case reproduces this issue.
> 
> For example:
> 
> Initially.. /dev/mmcblk0p4 is mounted as /
> 
> lsblk
> NAMEMAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
> mmcblk0 179:00 29.2G  0 disk
> |-mmcblk0p4 179:404G  0 part /
> |-mmcblk0p2 179:20  500M  0 part /boot
> |-mmcblk0p3 179:30  256M  0 part [SWAP]
> `-mmcblk0p1 179:10  256M  0 part /boot/efi
> 
> btrfs fi show
> Label: none  uuid: 07892354-ddaa-4443-90ea-f76a06accaba
> Total devices 1 FS bytes used 1.40GiB
> devid1 size 4.00GiB used 3.00GiB path /dev/mmcblk0p4
> 
> Copy mmcblk0 to sda
> dd if=/dev/mmcblk0 of=/dev/sda
> 
> And immediately after the copy completes the change in the device
> superblock is notified which the automount scans using
> btrfs device scan and the new device sda becomes the mounted root
> device.
> 
> lsblk
> NAMEMAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
> sda   8:01 14.9G  0 disk
> |-sda48:414G  0 part /
> |-sda28:21  500M  0 part
> |-sda38:31  256M  0 part
> `-sda18:11  256M  0 part
> mmcblk0 179:00 29.2G  0 disk
> |-mmcblk0p4 179:404G  0 part
> |-mmcblk0p2 179:20  500M  0 part /boot
> |-mmcblk0p3 179:30  256M  0 part [SWAP]
> `-mmcblk0p1 179:10  256M  0 part /boot/efi
> btrfs fi show /
> Label: none  uuid: 07892354-ddaa-4443-90ea-f76a06accaba
> Total devices 1 FS bytes used 1.40GiB
> devid1 size 4.00GiB used 3.00GiB path /dev/sda4
> 
> The bug is quite nasty that you can't either unmount /dev/sda4 or
> /dev/mmcblk0p4. And the problem does not get solved until you take
> the sda out of the system on to another system to change its fsid using
> the 'btrfstune -u' command.
> 
> Signed-off-by: Anand Jain 

Hi btrfs folks,

Please help review if this patch adds a valid test for btrfs. It looks
fine to me from fstests' perspective of view, though it needs some
really minor tweaks that I can fix on commit (e.g. format of for loop,
adding space after '#' in comments line). Thanks a lot!

Eryu

> ---
> v1->v2: 
>   dont play around with dev patch use it as it is.
>   do not use SCRATCH_MNT instead create it at the TEST_DIR and its related
>changes.
>   golden out changes
>
>  tests/btrfs/173 | 88 
> +
>  tests/btrfs/173.out |  6 
>  tests/btrfs/group   |  1 +
>  3 files changed, 95 insertions(+)
>  create mode 100755 tests/btrfs/173
>  create mode 100644 tests/btrfs/173.out
> 
> diff --git a/tests/btrfs/173 b/tests/btrfs/173
> new file mode 100755
> index ..b466ae921e19
> --- /dev/null
> +++ b/tests/btrfs/173
> @@ -0,0 +1,88 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2018 Oracle. All Rights Reserved.
> +#
> +# FS QA Test 173
> +#
> +# Fuzzy test for FS image duplication.
> +#  Could be fixed by
> +#[patch] btrfs: harden agaist duplicate fsid
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +mnt=$TEST_DIR/$seq.mnt
> +_cleanup()
> +{
> + rm -rf $mnt > /dev/null 2>&1
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch_dev_pool 2
> +_scratch_dev_pool_get 2
> +
> +dev_foo=$(echo $SCRATCH_DEV_POOL | awk '{print $1}')
> +dev_bar=$(echo $SCRATCH_DEV_POOL | awk '{print $2}')
> +
> +echo dev_foo=$dev_foo >> $seqres.full
> +echo dev_bar=$dev_bar >> $seqres.full
> +echo | tee -a $seqres.full
> +
> +rm -rf $mnt > /dev/null 2>&1
> +mkdir $mnt
> +_mkfs_dev $dev_foo
> +_mount $dev_foo $mnt
> +
> +check_btrfs_mount()
> +{
> + local x=$(findmnt $mnt | grep -v TARGET | awk '{print $2}')
> + [[ $x == $dev_foo ]] && echo DEV_FOO
> + [[ $x == $dev_bar ]] && echo DEV_BAR
> +}
> +
> +echo MNT $(check_btrfs_mount)
> +
> +for sb_bytenr in 65536 67108864
> +do
> + echo -n "dd status=none if=$dev_foo of=$dev_bar bs=1 "\
> + "seek=$sb_bytenr skip=$sb_bytenr count=4096" >> $seqres.full
> + dd status=none if=$dev_foo of=$dev_bar bs=1 seek=$sb_bytenr \
> + skip=$sb_bytenr count=4096 >> $seqres.full 2>&1
> + echo ..:$? >> $seqres.full
> +done
> +
> +#Original device is mounted, scan of its clone should fail
> +$BTRFS_UTIL_PROG device scan $dev_bar >> $seqres.full 2>&1
> +echo btrfs device scan dev_bar ...:$?| 

Re: [PATCH v2 5/9] generic/102 open code dev_size _scratch_mkfs_sized()

[Eryu Guan]

On Wed, Sep 26, 2018 at 12:08:56PM +0800, Anand Jain wrote:
> 
> 
> On 09/25/2018 06:54 PM, Nikolay Borisov wrote:
> > 
> > 
> > On 25.09.2018 07:24, Anand Jain wrote:
> > > Open code helps to grep and find out parameter sent to the
> > > _scratch_mkfs_sized here.
> > > 
> > > Signed-off-by: Anand Jain 
> > 
> > IMO this is noise, you can just as simply do
> > "grep _scratch_mkfs_sized" and then open the file to inspect the actual
> > argument. But it's up to the xfstest maintainers
> 
>  I am ok. Its just a nice cleanup.
> 
> Thanks, Anand

I prefer dropping patch 5/6/7, as I don't think they're that necessary.

BTW, other patches from this series but patch 3 ("geneirc/077 fix min
size for btrfs") look fine to me, I'm taking them in this week's update.

Thanks,
Eryu

> 
> > > ---
> > >   tests/generic/102 | 3 +--
> > >   1 file changed, 1 insertion(+), 2 deletions(-)
> > > 
> > > diff --git a/tests/generic/102 b/tests/generic/102
> > > index faf940ac5070..aad496a5bc69 100755
> > > --- a/tests/generic/102
> > > +++ b/tests/generic/102
> > > @@ -31,8 +31,7 @@ _require_scratch
> > >   rm -f $seqres.full
> > > -dev_size=$((512 * 1024 * 1024)) # 512MB filesystem
> > > -_scratch_mkfs_sized $dev_size >>$seqres.full 2>&1
> > > +_scratch_mkfs_sized $((512 * 1024 * 1024)) >>$seqres.full 2>&1
> > >   _scratch_mount
> > >   for ((i = 0; i < 10; i++)); do
> > > 

Re: [PATCH v2 3/9] geneirc/077 fix min size for btrfs

[Eryu Guan]

On Tue, Sep 25, 2018 at 12:24:16PM +0800, Anand Jain wrote:
> If btrfs need to be tested at its default blockgroup which is non-mixed,
> then it needs at least 256mb.
> 
> Signed-off-by: Anand Jain 

(Sorry for the late review..)

> ---
>  tests/generic/077 | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/tests/generic/077 b/tests/generic/077
> index ef6af18c83e3..ec236992513f 100755
> --- a/tests/generic/077
> +++ b/tests/generic/077
> @@ -49,8 +49,7 @@ rm -f $seqres.full
>  _scratch_unmount >/dev/null 2>&1
>  echo "*** MKFS ***" >>$seqres.full
>  echo "" >>$seqres.full
> -SIZE=`expr 50 \* 1024 \* 1024`
> -_scratch_mkfs_sized $SIZE   >>$seqres.full 2>&1 \
> +_scratch_mkfs_sized $((256 * 1024 *1024))   >>$seqres.full 2>&1 \
>   || _fail "mkfs failed"

Hmm, this test copies "/lib/modules/" to fill the original 50M
filesystem (which seems a bad way to me)

"
# Something w/ enough data to fill 50M of fs...
filler=/lib/modules/
...
echo "*** populate filesystem, pass #1" | tee -a $seqres.full
cp -rf $filler $SCRATCH_MNT/subdir >$seqres.full 2>&1
...
"

It works most of the time as "/lib/modules" is usually larger than 50M,
but it may not fullfil the fs with 256M size.

I think we should fix the way to fill the fs too.

Thanks,
Eryu

>  _scratch_mount
>  mkdir $SCRATCH_MNT/subdir
> -- 
> 1.8.3.1
> 

Re: [PATCH] fstests: btrfs verify hardening agaist duplicate fsid

[Eryu Guan]

On Mon, Oct 01, 2018 at 04:44:35PM +0800, Anand Jain wrote:
> We have a known bug in btrfs, that we let the device path be changed
> after the device has been mounted. So using this loop hole the new
> copied device would appears as if its mounted immediately after its
> been copied. So this test case reproduces this issue.
> 
> For example:
> 
> Initially.. /dev/mmcblk0p4 is mounted as /
> 
> lsblk
> NAMEMAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
> mmcblk0 179:00 29.2G  0 disk
> |-mmcblk0p4 179:404G  0 part /
> |-mmcblk0p2 179:20  500M  0 part /boot
> |-mmcblk0p3 179:30  256M  0 part [SWAP]
> `-mmcblk0p1 179:10  256M  0 part /boot/efi
> 
> btrfs fi show
> Label: none  uuid: 07892354-ddaa-4443-90ea-f76a06accaba
> Total devices 1 FS bytes used 1.40GiB
> devid1 size 4.00GiB used 3.00GiB path /dev/mmcblk0p4
> 
> Copy mmcblk0 to sda
> dd if=/dev/mmcblk0 of=/dev/sda
> 
> And immediately after the copy completes the change in the device
> superblock is notified which the automount scans using
> btrfs device scan and the new device sda becomes the mounted root
> device.
> 
> lsblk
> NAMEMAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
> sda   8:01 14.9G  0 disk
> |-sda48:414G  0 part /
> |-sda28:21  500M  0 part
> |-sda38:31  256M  0 part
> `-sda18:11  256M  0 part
> mmcblk0 179:00 29.2G  0 disk
> |-mmcblk0p4 179:404G  0 part
> |-mmcblk0p2 179:20  500M  0 part /boot
> |-mmcblk0p3 179:30  256M  0 part [SWAP]
> `-mmcblk0p1 179:10  256M  0 part /boot/efi
> btrfs fi show /
> Label: none  uuid: 07892354-ddaa-4443-90ea-f76a06accaba
> Total devices 1 FS bytes used 1.40GiB
> devid1 size 4.00GiB used 3.00GiB path /dev/sda4
> 
> The bug is quite nasty that you can't either unmount /dev/sda4 or
> /dev/mmcblk0p4. And the problem does not get solved until you take
> the sda out of the system on to another system to change its fsid using
> the 'btrfstune -u' command.
> 
> Signed-off-by: Anand Jain 

Looks like that the test will break the whole test env as it leaves an
unmountable $SCRATCH_MNT. I'd wait for the fix to get in first before
merging the test, in case it breaks normal regression tests. (I noticed
that the test is not in 'auto' group, so it's not that dangerous.)

Also, it'd be great if test can be reviewed by btrfs folks too!

> ---
>  tests/btrfs/173 | 72 
> +
>  tests/btrfs/173.out |  5 
>  tests/btrfs/group   |  1 +
>  3 files changed, 78 insertions(+)
>  create mode 100755 tests/btrfs/173
>  create mode 100644 tests/btrfs/173.out
> 
> diff --git a/tests/btrfs/173 b/tests/btrfs/173
> new file mode 100755
> index ..f59a62e206c3
> --- /dev/null
> +++ b/tests/btrfs/173
> @@ -0,0 +1,72 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2018 Oracle. All Rights Reserved.
> +#
> +# FS QA Test 173
> +#
> +# Fuzzy test for FS image duplication.
> +#  Could be fixed by
> +#[patch] btrfs: harden agaist duplicate fsid
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch_dev_pool 2
> +_scratch_dev_pool_get 2
> +
> +dev_foo=$(echo $SCRATCH_DEV_POOL | awk '{print $1}' | rev | cut -d"/" -f1 | 
> rev)
> +dev_bar=$(echo $SCRATCH_DEV_POOL | awk '{print $2}' | rev | cut -d"/" -f1 | 
> rev)

This doesn't work if the devices in SCRATCH_DEV_POOL are symlinks, e.g.
lvm devices: /dev/mapper/testvg-testlv1, dev_foo is "testvg-testlv1" in
this case.

> +
> +_mkfs_dev /dev/$dev_foo

But /dev/testvg-testlv1 isn't existed.

_short_dev and/or _real_dev is useful in this case. e.g.

dev_foo=$(echo $SCRATCH_DEV_POOL | awk '{print $1}')
# dev_foo is like "dm-1"
dev_foo=$(_short_dev $dev_foo)
# dev_foo is like "/dev/dm-1"
dev_foo=$(_real_dev $dev_foo)

> +_mount /dev/$dev_foo $SCRATCH_MNT

It'd better to mount non-SCRATCH_DEV to other mount point, e.g.
$TEST_DIR/$seq.mnt

Thanks,
Eryu

> +
> +echo mount before btrfs image clone | tee -a $seqres.full
> +findmnt /dev/$dev_foo | grep -v TARGET | awk '{print $1" "$2}' | \
> + sed -e "s/$dev_foo/dev_foo/g" | _filter_scratch | tee -a $seqres.full
> +findmnt /dev/$dev_bar | grep -v TARGET | awk '{print $1" "$2}' | \
> + sed -e "s/$dev_bar/dev_bar/g" | _filter_scratch | tee -a $seqres.full
> +
> +for sb_bytenr in 65536 67108864
> +do
> + echo -n "dd status=none if=/dev/$dev_foo of=/dev/$dev_bar bs=1 "\
> + "seek=$sb_bytenr skip=$sb_bytenr 

Re: [PATCH] test unaligned punch hole at ENOSPC

[Eryu Guan]

On Mon, Sep 24, 2018 at 07:47:39PM +0800, Anand Jain wrote:
> Try to punch hole with unaligned size and offset when the FS
> returns ENOSPC
> 
> Signed-off-by: Anand Jain 
> ---
> This test case fails on btrfs as of now.
> 
>  tests/btrfs/172 | 66 
> +
>  tests/btrfs/172.out |  2 ++
>  tests/btrfs/group   |  1 +
>  3 files changed, 69 insertions(+)
>  create mode 100755 tests/btrfs/172
>  create mode 100644 tests/btrfs/172.out
> 
> diff --git a/tests/btrfs/172 b/tests/btrfs/172
> new file mode 100755
> index ..9c32a173f912
> --- /dev/null
> +++ b/tests/btrfs/172
> @@ -0,0 +1,66 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2018 Oracle. All Rights Reserved.
> +#
> +# FS QA Test 172
> +#
> +# Test if the unaligned (by size and offset) punch hole is successful when FS
> +# is at ENOSPC.
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs generic
> +_supported_os Linux
> +_require_scratch
> +
> +_scratch_mkfs_sized $((200 * 1024 *1024)) >> $seqres.full
> +
> +# max_inline helps to create regular extent
> +_scratch_mount "-o max_inline=0,nodatacow"
> +
> +echo "Fill fs upto ENOSPC" >> $seqres.full
> +dd status=none if=/dev/zero of=$SCRATCH_MNT/filler bs=512 >> $seqres.full 
> 2>&1
> +
> +extent_size=$(_scratch_btrfs_sectorsize)
> +unalign_by=512
> +echo extent_size=$extent_size unalign_by=$unalign_by >> $seqres.full
> +
> +hole_offset=0
> +hole_len=$unalign_by
> +run_check fallocate -p -o $hole_offset -l $hole_len $SCRATCH_MNT/filler

Please don't introduce new run_check/_run_btrfs_util_prog users, just
redirect output to /dev/null if the outputs don't matter. Please refer
to this thread

https://www.spinics.net/lists/linux-btrfs/msg80996.html

And use xfs_io fpunch command instead of bare 'fallocate -p', and check
xfs_io and kernel support on fpunch by calling

_require_xfs_io_comand "fpunch"

> +
> +hole_offset=$(($extent_size + $unalign_by))
> +hole_len=$(($extent_size - $unalign_by))
> +run_check fallocate -p -o $hole_offset -l $hole_len $SCRATCH_MNT/filler
> +
> +hole_offset=$(($extent_size * 2 + $unalign_by))
> +hole_len=$(($extent_size * 5))
> +run_check fallocate -p -o $hole_offset -l $hole_len $SCRATCH_MNT/filler
> +
> +# success, all done
> +echo "Silence is golden"
> +status=0
> +exit
> diff --git a/tests/btrfs/172.out b/tests/btrfs/172.out
> new file mode 100644
> index ..ce2de3f0d107
> --- /dev/null
> +++ b/tests/btrfs/172.out
> @@ -0,0 +1,2 @@
> +QA output created by 172
> +Silence is golden
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index feffc45b6564..7e1a638ab7e1 100644
> --- a/tests/btrfs/group
> +++ b/tests/btrfs/group
> @@ -174,3 +174,4 @@
>  169 auto quick send
>  170 auto quick snapshot
>  171 auto quick qgroup
> +172 auto quick

Add 'punch' group too.

Thanks,
Eryu

> -- 
> 1.8.3.1
> 

Re: [PATCH] fstests: btrfs/149 make it sectorsize independent

[Eryu Guan]

On Wed, Sep 05, 2018 at 02:35:12PM +0800, Anand Jain wrote:
> Originally this test case was designed to work with only 4K sectorsize.
> Now enhance it to work with any sector sizes and makes the following
> changes:
> Output file not to contain any traces of sector size.
> Use max_inline=0 mount option so that it meets the requisite of non inline
> regular extent.
> Don't log the md5sum results to the output file as the data size vary by
> the sectorsize.
> 
> Signed-off-by: Anand Jain 
> ---
>  common/btrfs|  7 +++
>  common/filter   |  5 +
>  tests/btrfs/149 | 29 -
>  tests/btrfs/149.out | 12 ++--
>  4 files changed, 38 insertions(+), 15 deletions(-)
> 
> diff --git a/common/btrfs b/common/btrfs
> index 79c687f73376..e6a218d6b63a 100644
> --- a/common/btrfs
> +++ b/common/btrfs
> @@ -367,3 +367,10 @@ _run_btrfs_balance_start()
>  
>   run_check $BTRFS_UTIL_PROG balance start $bal_opt $*
>  }
> +
> +#return the sector size of the btrfs scratch fs
> +_scratch_sectorsize()

The function name would indicate that it's a general helper function and
works for all filesystems, but it only works for btrfs. I'd be better to
name it as "_scratch_btrfs_sectorsize" or similar.

> +{
> + $BTRFS_UTIL_PROG inspect-internal dump-super $SCRATCH_DEV |\
> + grep sectorsize | awk '{print $2}'
> +}
> diff --git a/common/filter b/common/filter
> index 3965c2eb752b..e87740ddda3f 100644
> --- a/common/filter
> +++ b/common/filter
> @@ -271,6 +271,11 @@ _filter_xfs_io_pages_modified()
>   _filter_xfs_io_units_modified "Page" $PAGE_SIZE
>  }
>  
> +_filter_xfs_io_numbers()
> +{
> +_filter_xfs_io | sed -E 's/[0-9]+//g'
> +}
> +
>  _filter_test_dir()
>  {
>   # TEST_DEV may be a prefix of TEST_DIR (e.g. /mnt, /mnt/ovl-mnt)
> diff --git a/tests/btrfs/149 b/tests/btrfs/149
> index 3e955a305e0f..3958fa844c8b 100755
> --- a/tests/btrfs/149
> +++ b/tests/btrfs/149
> @@ -44,21 +44,27 @@ rm -fr $send_files_dir
>  mkdir $send_files_dir
>  
>  _scratch_mkfs >>$seqres.full 2>&1
> -_scratch_mount "-o compress"
> +# On 64K pagesize systems the compression is more efficient, so max_inline
> +# helps to create regular (non inline) extent irrespective of the final
> +# write size.
> +_scratch_mount "-o compress -o max_inline=0"
>  
>  # Write to our file using direct IO, so that this way the write ends up not
>  # getting compressed, that is, we get a regular extent which is neither
>  # inlined nor compressed.
>  # Alternatively, we could have mounted the fs without compression enabled,
>  # which would result as well in an uncompressed regular extent.
> -$XFS_IO_PROG -f -d -c "pwrite -S 0xab 0 4K" $SCRATCH_MNT/foobar | 
> _filter_xfs_io
> +sectorsize=$(_scratch_sectorsize)

This new helper takes use of "btrfs inspect-internal dump-super", so the
test needs a new _require rule:

_require_btrfs_command inspect-internal dump-super

Otherwise the patch looks fine to me. But I'd be great if other btrfs
folks could help review the change and/or help actually test the changes
with 64k sectorsize.

Thanks,
Eryu

> +$XFS_IO_PROG -f -d -c "pwrite -S 0xab 0 $sectorsize" $SCRATCH_MNT/foobar |\
> + _filter_xfs_io_numbers
>  
>  $BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT \
>   $SCRATCH_MNT/mysnap1 > /dev/null
>  
>  # Clone the regular (not inlined) extent.
> -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foobar 0 8K 4K" $SCRATCH_MNT/foobar \
> - | _filter_xfs_io
> +$XFS_IO_PROG -c \
> + "reflink $SCRATCH_MNT/foobar 0 $((2 * $sectorsize)) $sectorsize" \
> + $SCRATCH_MNT/foobar | _filter_xfs_io_numbers
>  
>  $BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT \
>   $SCRATCH_MNT/mysnap2 > /dev/null
> @@ -76,21 +82,26 @@ $BTRFS_UTIL_PROG send -p $SCRATCH_MNT/mysnap1 -f 
> $send_files_dir/2.snap \
>$SCRATCH_MNT/mysnap2 2>&1 >/dev/null | _filter_scratch
>  
>  echo "File digests in the original filesystem:"
> -md5sum $SCRATCH_MNT/mysnap1/foobar | _filter_scratch
> -md5sum $SCRATCH_MNT/mysnap2/foobar | _filter_scratch
> +sum_src_snap1=$(md5sum $SCRATCH_MNT/mysnap1/foobar | awk '{print $1}')
> +sum_src_snap2=$(md5sum $SCRATCH_MNT/mysnap2/foobar | awk '{print $1}')
> +echo "src checksum created"
>  
>  # Now recreate the filesystem by receiving both send streams and verify we 
> get
>  # the same file content that the original filesystem had.
>  _scratch_unmount
>  _scratch_mkfs >>$seqres.full 2>&1
> -_scratch_mount "-o compress"
> +_scratch_mount "-o compress,max_inline=0"
>  
>  $BTRFS_UTIL_PROG receive -f $send_files_dir/1.snap $SCRATCH_MNT > /dev/null
>  $BTRFS_UTIL_PROG receive -f $send_files_dir/2.snap $SCRATCH_MNT > /dev/null
>  
>  echo "File digests in the new filesystem:"
> -md5sum $SCRATCH_MNT/mysnap1/foobar | _filter_scratch
> -md5sum $SCRATCH_MNT/mysnap2/foobar | _filter_scratch
> +sum_dest_snap1=$(md5sum $SCRATCH_MNT/mysnap1/foobar | awk '{print $1}')
> +sum_dest_snap2=$(md5sum $SCRATCH_MNT/mysnap2/foobar | awk 

Re: [PATCH] generic: test for deduplication between different files

[Eryu Guan]

On Sun, Aug 19, 2018 at 04:41:31PM +0100, Filipe Manana wrote:
> On Sun, Aug 19, 2018 at 3:07 PM, Eryu Guan  wrote:
> > On Fri, Aug 17, 2018 at 09:39:24AM +0100, fdman...@kernel.org wrote:
> >> From: Filipe Manana 
> >>
> >> Test that deduplication of an entire file that has a size that is not
> >> aligned to the filesystem's block size into a different file does not
> >> corrupt the destination's file data.
> >>
> >> This test is motivated by a bug found in Btrfs which is fixed by the
> >> following patch for the linux kernel:
> >>
> >>   "Btrfs: fix data corruption when deduplicating between different files"
> >>
> >> XFS also fails this test, at least as of linux kernel 4.18-rc7, exactly
> >> with the same corruption as in Btrfs - some bytes of a block get replaced
> >> with zeroes after the deduplication.
> >>
> >> Signed-off-by: Filipe Manana 
> >> ---
> >>  tests/generic/505 | 84 
> >> +++
> >>  tests/generic/505.out | 33 
> >>  tests/generic/group   |  1 +
> >>  3 files changed, 118 insertions(+)
> >>  create mode 100755 tests/generic/505
> >>  create mode 100644 tests/generic/505.out
> >>
> >> diff --git a/tests/generic/505 b/tests/generic/505
> >> new file mode 100755
> >> index ..5ee232a2
> >> --- /dev/null
> >> +++ b/tests/generic/505
> >> @@ -0,0 +1,84 @@
> >> +#! /bin/bash
> >> +# SPDX-License-Identifier: GPL-2.0
> >> +# Copyright (C) 2018 SUSE Linux Products GmbH. All Rights Reserved.
> >> +#
> >> +# FS QA Test No. 505
> >> +#
> >> +# Test that deduplication of an entire file that has a size that is not 
> >> aligned
> >> +# to the filesystem's block size into a different file does not corrupt 
> >> the
> >> +# destination's file data.
> >> +#
> >> +seq=`basename $0`
> >> +seqres=$RESULT_DIR/$seq
> >> +echo "QA output created by $seq"
> >> +tmp=/tmp/$$
> >> +status=1 # failure is the default!
> >> +trap "_cleanup; exit \$status" 0 1 2 3 15
> >> +
> >> +_cleanup()
> >> +{
> >> + cd /
> >> + rm -f $tmp.*
> >> +}
> >> +
> >> +# get standard environment, filters and checks
> >> +. ./common/rc
> >> +. ./common/filter
> >> +. ./common/reflink
> >> +
> >> +# real QA test starts here
> >> +_supported_fs generic
> >> +_supported_os Linux
> >> +_require_scratch_dedupe
> >> +
> >> +rm -f $seqres.full
> >> +
> >> +_scratch_mkfs >>$seqres.full 2>&1
> >> +_scratch_mount
> >> +
> >> +# The first byte with a value of 0xae starts at an offset (2518890) which 
> >> is not
> >> +# a multiple of the block size.
> >> +$XFS_IO_PROG -f \
> >> + -c "pwrite -S 0x6b 0 2518890" \
> >> + -c "pwrite -S 0xae 2518890 102398" \
> >> + $SCRATCH_MNT/foo | _filter_xfs_io
> >> +
> >> +# Create a second file with a length not aligned to the block size, whose 
> >> bytes
> >> +# all have the value 0x6b, so that its extent(s) can be deduplicated with 
> >> the
> >> +# first file.
> >> +$XFS_IO_PROG -f -c "pwrite -S 0x6b 0 557771" $SCRATCH_MNT/bar | 
> >> _filter_xfs_io
> >> +
> >> +# The file is filled with bytes having the value 0x6b from offset 0 to 
> >> offset
> >> +# 2518889 and with the value 0xae from offset 2518890 to offset 2621287.
> >> +echo "File content before deduplication:"
> >> +od -t x1 $SCRATCH_MNT/foo
> >> +
> >> +# Now deduplicate the entire second file into a range of the first file 
> >> that
> >> +# also has all bytes with the value 0x6b. The destination range's end 
> >> offset
> >> +# must not be aligned to the block size and must be less then the offset 
> >> of
> >> +# the first byte with the value 0xae (byte at offset 2518890).
> >> +$XFS_IO_PROG -c "dedupe $SCRATCH_MNT/bar 0 1957888 557771" 
> >> $SCRATCH_MNT/foo \
> >> + | _filter_xfs_io
> >> +
> >> +# The bytes in the range starting at offset 2515659 (end of the 
> >> deduplication
> >> +# range) and ending at offset 2519040 (start offset rounded up to the 
> >> block
> >> +# size) must all have the value 0xae (and not replaced with 0x00 values).
> >
> > This doesn't seem right to me, range [2515659, 2518890) should be 0x6b
> > not 0xae, while range [2518890, 2519040) indeed should contain 0xae.
> 
> Yes, indeed. My mistake (got it right in the comment before the first
> call to "od").
> Can you fix it up (if there's nothing else to fix), or do you need me
> to send a new version?

Sure, I can fix it on commit. But I've already pushed this week's update
to upstream, so you won't see it until next week :)

Thanks,
Eryu

Re: [PATCH] generic: test for deduplication between different files

[Eryu Guan]

On Fri, Aug 17, 2018 at 09:39:24AM +0100, fdman...@kernel.org wrote:
> From: Filipe Manana 
> 
> Test that deduplication of an entire file that has a size that is not
> aligned to the filesystem's block size into a different file does not
> corrupt the destination's file data.
> 
> This test is motivated by a bug found in Btrfs which is fixed by the
> following patch for the linux kernel:
> 
>   "Btrfs: fix data corruption when deduplicating between different files"
> 
> XFS also fails this test, at least as of linux kernel 4.18-rc7, exactly
> with the same corruption as in Btrfs - some bytes of a block get replaced
> with zeroes after the deduplication.
> 
> Signed-off-by: Filipe Manana 
> ---
>  tests/generic/505 | 84 
> +++
>  tests/generic/505.out | 33 
>  tests/generic/group   |  1 +
>  3 files changed, 118 insertions(+)
>  create mode 100755 tests/generic/505
>  create mode 100644 tests/generic/505.out
> 
> diff --git a/tests/generic/505 b/tests/generic/505
> new file mode 100755
> index ..5ee232a2
> --- /dev/null
> +++ b/tests/generic/505
> @@ -0,0 +1,84 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (C) 2018 SUSE Linux Products GmbH. All Rights Reserved.
> +#
> +# FS QA Test No. 505
> +#
> +# Test that deduplication of an entire file that has a size that is not 
> aligned
> +# to the filesystem's block size into a different file does not corrupt the
> +# destination's file data.
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_supported_fs generic
> +_supported_os Linux
> +_require_scratch_dedupe
> +
> +rm -f $seqres.full
> +
> +_scratch_mkfs >>$seqres.full 2>&1
> +_scratch_mount
> +
> +# The first byte with a value of 0xae starts at an offset (2518890) which is 
> not
> +# a multiple of the block size.
> +$XFS_IO_PROG -f \
> + -c "pwrite -S 0x6b 0 2518890" \
> + -c "pwrite -S 0xae 2518890 102398" \
> + $SCRATCH_MNT/foo | _filter_xfs_io
> +
> +# Create a second file with a length not aligned to the block size, whose 
> bytes
> +# all have the value 0x6b, so that its extent(s) can be deduplicated with the
> +# first file.
> +$XFS_IO_PROG -f -c "pwrite -S 0x6b 0 557771" $SCRATCH_MNT/bar | 
> _filter_xfs_io
> +
> +# The file is filled with bytes having the value 0x6b from offset 0 to offset
> +# 2518889 and with the value 0xae from offset 2518890 to offset 2621287.
> +echo "File content before deduplication:"
> +od -t x1 $SCRATCH_MNT/foo
> +
> +# Now deduplicate the entire second file into a range of the first file that
> +# also has all bytes with the value 0x6b. The destination range's end offset
> +# must not be aligned to the block size and must be less then the offset of
> +# the first byte with the value 0xae (byte at offset 2518890).
> +$XFS_IO_PROG -c "dedupe $SCRATCH_MNT/bar 0 1957888 557771" $SCRATCH_MNT/foo \
> + | _filter_xfs_io
> +
> +# The bytes in the range starting at offset 2515659 (end of the deduplication
> +# range) and ending at offset 2519040 (start offset rounded up to the block
> +# size) must all have the value 0xae (and not replaced with 0x00 values).

This doesn't seem right to me, range [2515659, 2518890) should be 0x6b
not 0xae, while range [2518890, 2519040) indeed should contain 0xae.

Thanks,
Eryu

> +# In other words, we should have exactly the same data we had before we asked
> +# for deduplication.
> +echo "File content after deduplication and before unmounting:"
> +od -t x1 $SCRATCH_MNT/foo
> +
> +# Unmount the filesystem and mount it again. This guarantees any file data in
> +# the page cache is dropped.
> +_scratch_cycle_mount
> +
> +# The bytes in the range starting at offset 2515659 (end of the deduplication
> +# range) and ending at offset 2519040 (start offset rounded up to the block
> +# size) must all have the value 0xae (and not replaced with 0x00 values).
> +# In other words, we should have exactly the same data we had before we asked
> +# for deduplication.
> +echo "File content after unmounting:"
> +od -t x1 $SCRATCH_MNT/foo
> +
> +status=0
> +exit
> diff --git a/tests/generic/505.out b/tests/generic/505.out
> new file mode 100644
> index ..7556b9fb
> --- /dev/null
> +++ b/tests/generic/505.out
> @@ -0,0 +1,33 @@
> +QA output created by 505
> +wrote 2518890/2518890 bytes at offset 0
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +wrote 102398/102398 bytes at offset 2518890
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +wrote 557771/557771 bytes at offset 0
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +File content before 

Re: [PATCH] fstests: btrfs: Add test for corrupted orphan qgroup numbers

[Eryu Guan]

On Fri, Aug 10, 2018 at 05:10:29PM +0800, Qu Wenruo wrote:
> 
> 
> On 8/10/18 4:54 PM, Filipe Manana wrote:
> > On Fri, Aug 10, 2018 at 9:46 AM, Qu Wenruo  wrote:
> >>
> >>
> >> On 8/9/18 5:26 PM, Filipe Manana wrote:
> >>> On Thu, Aug 9, 2018 at 8:45 AM, Qu Wenruo  wrote:
>  This bug is exposed by populating a high level qgroup, and then make it
>  orphan (high level qgroup without child)
> >>>
> >>> Same comment as in the kernel patch:
> >>>
> >>> "That sentence is confusing. An orphan, by definition [1], is someone
> >>> (or something in this case) without parents.
> >>> But you mention a group without children, so that should be named
> >>> "childless" or simply say "without children".
> >>> So one part of the sentence is wrong, either what is in parenthesis or
> >>> what comes before them.
> >>>
> >>> [1] https://www.thefreedictionary.com/orphan
> >>> "
> >>>
>  with old qgroup numbers, and
>  finally do rescan.
> 
>  Normally rescan should zero out all qgroups' accounting number, but due
>  to a kernel bug which won't mark orphan qgroups dirty, their on-disk
>  data is not updated, thus old numbers remain and cause qgroup
>  corruption.
> 
>  Fixed by the following kernel patch:
>  "btrfs: qgroup: Dirty all qgroups before rescan"
> 
>  Reported-by: Misono Tomohiro 
>  Signed-off-by: Qu Wenruo 
>  ---
>   tests/btrfs/170 | 82 +
>   tests/btrfs/170.out |  3 ++
>   tests/btrfs/group   |  1 +
>   3 files changed, 86 insertions(+)
>   create mode 100755 tests/btrfs/170
>   create mode 100644 tests/btrfs/170.out
> 
>  diff --git a/tests/btrfs/170 b/tests/btrfs/170
>  new file mode 100755
>  index ..bcf8b5c0e4f3
>  --- /dev/null
>  +++ b/tests/btrfs/170
>  @@ -0,0 +1,82 @@
>  +#! /bin/bash
>  +# SPDX-License-Identifier: GPL-2.0
>  +# Copyright (c) 2018 SUSE Linux Products GmbH.  All Rights Reserved.
>  +#
>  +# FS QA Test 170
>  +#
>  +# Test if btrfs can clear orphan (high level qgroup without child) 
>  qgroup's
>  +# accounting numbers during rescan.
>  +# Fixed by the following kernel patch:
>  +# "btrfs: qgroup: Dirty all qgroups before rescan"
>  +#
>  +seq=`basename $0`
>  +seqres=$RESULT_DIR/$seq
>  +echo "QA output created by $seq"
>  +
>  +here=`pwd`
>  +tmp=/tmp/$$
>  +status=1   # failure is the default!
>  +trap "_cleanup; exit \$status" 0 1 2 3 15
>  +
>  +_cleanup()
>  +{
>  +   cd /
>  +   rm -f $tmp.*
>  +}
>  +
>  +# get standard environment, filters and checks
>  +. ./common/rc
>  +. ./common/filter
>  +
>  +# remove previous $seqres.full before test
>  +rm -f $seqres.full
>  +
>  +# real QA test starts here
>  +
>  +# Modify as appropriate.
>  +_supported_fs btrfs
>  +_supported_os Linux
>  +_require_scratch
>  +
>  +_scratch_mkfs > /dev/null 2>&1
>  +_scratch_mount
>  +
>  +
>  +# Populate the fs
>  +_run_btrfs_util_prog subvolume create "$SCRATCH_MNT/subvol"
>  +_pwrite_byte 0xcdcd 0 1M "$SCRATCH_MNT/subvol/file1" | _filter_xfs_io > 
>  /dev/null
>  +
>  +# Ensure that file reach disk, so it will also appear in snapshot
> >>>
> >>> # Ensure that buffered file data is persisted, so we won't have an
> >>> empty file in the snapshot.
>  +sync
>  +_run_btrfs_util_prog subvolume snapshot "$SCRATCH_MNT/subvol" 
>  "$SCRATCH_MNT/snapshot"
>  +
>  +
>  +_run_btrfs_util_prog quota enable "$SCRATCH_MNT"
>  +_run_btrfs_util_prog quota rescan -w "$SCRATCH_MNT"
>  +
>  +# Create high level qgroup
>  +_run_btrfs_util_prog qgroup create 1/0 "$SCRATCH_MNT"
>  +
>  +# Don't use _run_btrfs_util_prog here, as it can return 1 to info user
>  +# that qgroup is marked inconsistent, this is a bug in btrfs-progs, but
>  +# to ensure it will work, we just ignore the return value.
> >>>
> >>> Comment should go away IMHO. The preferred way is to call
> >>> $BTRFS_UTIL_PROG and have failures noticed
> >>> through differences in the golden output. There's no point in
> >>> mentioning something that currently doesn't work
> >>> if it's not used here.
> >>
> >> In this case, I think we still need to mention why we don't use
> >> _run_btrfs_util_progs, in fact if we use _run_btrfs_util_progs, the test
> >> will just fail due to the return value.
> >>
> >> In fact, it's a workaround and worthy noting IIRC.
> > 
> > Still disagree, because we are not checking the return value and rely
> > on errors printing something to stderr/stdout.
> 
> OK, either way I'll introduce a new filter here for filtering out either
> "Quota data changed, rescan scheduled" or "quotas may be inconsistent,
> rescan needed".
> 
> As there is patch floating around to change the default behavior of
> 

Re: [PATCH v2] fstests: btrfs/168 verify device ready after device delete

[Eryu Guan]

On Fri, Jul 06, 2018 at 02:01:37PM +0800, Anand Jain wrote:
> This test case verifies if the device ready return success after the
> device delete.
> 
> Signed-off-by: Anand Jain 

Need some helps from btrfs folks to see if it's a valid test. Thanks!

Eryu

> ---
> v1->v2: use _run_btrfs_util_prog instead of open coding it.
> 
>  tests/btrfs/168 | 68 
> +
>  tests/btrfs/168.out |  2 ++
>  tests/btrfs/group   |  1 +
>  3 files changed, 71 insertions(+)
>  create mode 100755 tests/btrfs/168
>  create mode 100644 tests/btrfs/168.out
> 
> diff --git a/tests/btrfs/168 b/tests/btrfs/168
> new file mode 100755
> index ..0d3e99839209
> --- /dev/null
> +++ b/tests/btrfs/168
> @@ -0,0 +1,68 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (C) 2018 Oracle. All Rights Reserved.
> +#
> +# FS QA Test 168
> +#
> +# Test if btrfs is still reported ready after the device delete
> +#
> +# This could be fixed by the following kernel commit:
> +#  btrfs: fix missing superblock update in the device delete commit 
> transaction
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_command "$BTRFS_TUNE_PROG" btrfstune
> +_require_scratch_dev_pool 2
> +
> +_scratch_dev_pool_get 2
> +dev_1=$(echo $SCRATCH_DEV_POOL | awk '{print $1}')
> +dev_2=$(echo $SCRATCH_DEV_POOL | awk '{print $2}')
> +
> +# normal delete device and then check for ready
> +run_check _scratch_pool_mkfs "-d single -m single"
> +_scratch_mount
> +_run_btrfs_util_prog device delete $dev_1 $SCRATCH_MNT
> +_run_btrfs_util_prog device ready $dev_2
> +
> +_scratch_unmount
> +# delete a seed device and then check for ready
> +run_check $BTRFS_TUNE_PROG -S 1 $dev_2
> +run_check _mount $dev_2 $SCRATCH_MNT
> +_run_btrfs_util_prog device add -f $dev_1 $SCRATCH_MNT
> +run_check mount -o rw,remount $dev_1 $SCRATCH_MNT
> +_run_btrfs_util_prog device delete $dev_2 $SCRATCH_MNT
> +_run_btrfs_util_prog device ready $dev_1
> +
> +_scratch_unmount
> +_scratch_dev_pool_put
> +
> +echo "Silence is golden"
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/168.out b/tests/btrfs/168.out
> new file mode 100644
> index ..893a41d859c8
> --- /dev/null
> +++ b/tests/btrfs/168.out
> @@ -0,0 +1,2 @@
> +QA output created by 168
> +Silence is golden
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index 5cff3bd6cc03..7bc3ea457992 100644
> --- a/tests/btrfs/group
> +++ b/tests/btrfs/group
> @@ -170,3 +170,4 @@
>  165 auto quick subvol
>  166 auto quick qgroup
>  167 auto quick replace volume
> +168 auto quick volume
> -- 
> 2.15.0
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] fstests: btrfs/168 verify device ready after device delete

[Eryu Guan]

On Tue, Jul 03, 2018 at 04:47:53PM +0800, Anand Jain wrote:
> This test case verifies if the device ready return success after the
> device delete.
> 
> Signed-off-by: Anand Jain 

Looks fine to me overall, but I may need some helps from btrfs folks :)

> ---
>  tests/btrfs/168 | 68 
> +
>  tests/btrfs/168.out |  2 ++
>  tests/btrfs/group   |  1 +
>  3 files changed, 71 insertions(+)
>  create mode 100755 tests/btrfs/168
>  create mode 100644 tests/btrfs/168.out
> 
> diff --git a/tests/btrfs/168 b/tests/btrfs/168
> new file mode 100755
> index ..cb5b8eb4b5a8
> --- /dev/null
> +++ b/tests/btrfs/168
> @@ -0,0 +1,68 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (C) 2018 Oracle. All Rights Reserved.
> +#
> +# FS QA Test 168
> +#
> +# Test if btrfs is still reported ready after the device delete
> +#
> +# This could be fixed by the following kernel commit:
> +#  btrfs: fix missing superblock update in the device delete commit 
> transaction
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_command "$BTRFS_TUNE_PROG" btrfstune
> +_require_scratch_dev_pool 2
> +
> +_scratch_dev_pool_get 2
> +dev_1=$(echo $SCRATCH_DEV_POOL | awk '{print $1}')
> +dev_2=$(echo $SCRATCH_DEV_POOL | awk '{print $2}')
> +
> +# normal delete device and then check for ready
> +run_check _scratch_pool_mkfs "-d single -m single"
> +_scratch_mount
> +run_check $BTRFS_UTIL_PROG device delete $dev_1 $SCRATCH_MNT
> +run_check $BTRFS_UTIL_PROG device ready $dev_2

Why not "_run_btrfs_util_prog device delete "

> +
> +_scratch_unmount
> +# delete a seed device and then check for ready
> +run_check $BTRFS_TUNE_PROG -S 1 $dev_2
> +run_check _mount $dev_2 $SCRATCH_MNT
> +_run_btrfs_util_prog device add -f $dev_1 $SCRATCH_MNT

Like this one?

But I still prefer dropping run_check family functions completely, just
using bare command, if the output of these commands are not
deterministic we could simply through them away.

Thanks,
Eryu

> +run_check mount -o rw,remount $dev_1 $SCRATCH_MNT
> +run_check $BTRFS_UTIL_PROG device delete $dev_2 $SCRATCH_MNT
> +run_check $BTRFS_UTIL_PROG device ready $dev_1
> +
> +_scratch_unmount
> +_scratch_dev_pool_put
> +
> +echo "Silence is golden"
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/168.out b/tests/btrfs/168.out
> new file mode 100644
> index ..893a41d859c8
> --- /dev/null
> +++ b/tests/btrfs/168.out
> @@ -0,0 +1,2 @@
> +QA output created by 168
> +Silence is golden
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index 5cff3bd6cc03..7bc3ea457992 100644
> --- a/tests/btrfs/group
> +++ b/tests/btrfs/group
> @@ -170,3 +170,4 @@
>  165 auto quick subvol
>  166 auto quick qgroup
>  167 auto quick replace volume
> +168 auto quick volume
> -- 
> 2.15.0
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] fstests: btrfs: Test if btrfs will corrupt nodatasum compressed extent when replacing device

[Eryu Guan]

On Thu, Jun 28, 2018 at 08:11:00AM +0300, Nikolay Borisov wrote:
> 
> 
> On  1.06.2018 04:34, Qu Wenruo wrote:
> > This is a long existing bug (from 2012) but exposed by a reporter
> > recently, that when compressed extent without data csum get written to
> > device-replace target device, the written data is in fact uncompressed data
> > other than the original compressed data.
> > 
> > And since btrfs still consider the data is compressed and will try to read 
> > it
> > as compressed, it can cause read error.
> > 
> > The root cause is located, and one RFC patch already sent to fix it,
> > titled "[PATCH RFC] btrfs: scrub: Don't use inode pages for device replace".
> > (The RFC is only for the extra possible way to fix the bug, the fix
> > itself should work without problem)
> > 
> > Reported-by: James Harvey 
> > Signed-off-by: Qu Wenruo 
> 
> Reviewed-by: Nikolay Borisov 

Thanks for the review! I assume the v3 patch also passes your review :)

Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] fstests: btrfs/085: replace btrfs-debug-tree with btrfs inspect-internal dump-tree

[Eryu Guan]

On Thu, Jun 21, 2018 at 03:04:22PM +0800, Lu Fengqi wrote:
> Since btrfs-dump-tree has been removed from btrfs-progs, use btrfs
> inspect-internal dump-tree instead of btrfs-dump-tree.
> 
> Signed-off-by: Lu Fengqi 

Then there's no user of $BTRFS_DEBUG_TREE_PROG, I think we could remove
the definition too in common/config

export BTRFS_DEBUG_TREE_PROG=$(type -P btrfs-debug-tree)

I'll remove it on commit.

Thanks,
Eryu

> ---
>  tests/btrfs/085 | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/btrfs/085 b/tests/btrfs/085
> index 8cc69c21c0dd..4773ed8041fd 100755
> --- a/tests/btrfs/085
> +++ b/tests/btrfs/085
> @@ -39,14 +39,14 @@ _supported_fs btrfs
>  _supported_os Linux
>  _require_scratch
>  _require_dm_target flakey
> -_require_command "$BTRFS_DEBUG_TREE_PROG" btrfs-debug-tree
> +_require_btrfs_command inspect-internal dump-tree
>  
>  rm -f $seqres.full
>  
>  has_orphan_item()
>  {
>   INO=$1
> - if $BTRFS_DEBUG_TREE_PROG $SCRATCH_DEV | \
> + if $BTRFS_UTIL_PROG inspect-internal dump-tree $SCRATCH_DEV | \
>   grep -q "key (ORPHAN ORPHAN_ITEM $INO)"; then
>   return 0
>   fi
> -- 
> 2.17.1
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] fstests: btrfs: Test if btrfs will corrupt nodatasum compressed extent when replacing device

[Eryu Guan]

On Fri, Jun 08, 2018 at 02:17:23PM +0800, Qu Wenruo wrote:
> This is a long existing bug (from 2012) but exposed by a reporter
> recently, that when compressed extent without data csum get written to
> device-replace target device, the written data is in fact uncompressed data
> other than the original compressed data.
> 
> And since btrfs still consider the data is compressed and will try to read it
> as compressed, it can cause read error.
> 
> The root cause is located, and one RFC patch already sent to fix it,
> titled "[PATCH] btrfs: scrub: Don't use inode pages for device replace".
> (The RFC is only for the extra possible way to fix the bug, the fix
> itself should work without problem)
> 
> Reported-by: James Harvey 
> Signed-off-by: Qu Wenruo 
> ---
> changelog:
> v2:
>   Now the fix patch is no longer RFC.
>   Remove _require_test as we don't really touch it.
>   Add comment on the mount cycle.
>   Add the test to group 'volume'.

Thanks for the revision! But again, I'd like to get some explicit
reviews from btrfs folks.

> ---
>  tests/btrfs/161 | 91 +
>  tests/btrfs/161.out |  2 +
>  tests/btrfs/group   |  1 +
>  3 files changed, 94 insertions(+)
>  create mode 100755 tests/btrfs/161
>  create mode 100644 tests/btrfs/161.out
> 
> diff --git a/tests/btrfs/161 b/tests/btrfs/161
> new file mode 100755
> index ..ce1b0e04
> --- /dev/null
> +++ b/tests/btrfs/161
> @@ -0,0 +1,91 @@
> +#! /bin/bash
> +# FS QA Test 161
> +#
> +# Test if btrfs will corrupt compressed data extent without data csum
> +# by replacing it with uncompressed data, when doing replacing device.
> +#
> +# This could be fixed by the following RFC patch:
   can be dropped?

Thanks,
Eryu

P.S.
*IF* you're going to send v3, could you please follow the new test
template (create new test with './new btrfs' would do the work) and
rebase against latest master? That'd be easier for me to apply the
patch, but I'm also fine with taking it as-is (after we get Reviewed-by
tag), I can convert the test and re-number it on commit as always.

> +# "[PATCH] btrfs: scrub: Don't use inode pages for device replace"
> +#
> +#---
> +# Copyright (C) 2018 SUSE Linux Products GmbH. All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch_dev_pool 2
> +_require_scratch_dev_pool_equal_size
> +
> +
> +_scratch_dev_pool_get 1
> +_spare_dev_get
> +_scratch_pool_mkfs >> $seqres.full 2>&1
> +
> +# Create nodatasum inode
> +_scratch_mount "-o nodatasum"
> +touch $SCRATCH_MNT/nodatasum_file
> +_scratch_remount "datasum,compress"
> +_pwrite_byte 0xcd 0 128K $SCRATCH_MNT/nodatasum_file > /dev/null
> +
> +# Write the compressed data back to disk
> +sync
> +
> +# Replace the device
> +_run_btrfs_util_prog replace start -Bf 1 $SPARE_DEV $SCRATCH_MNT
> +
> +# Unmount to drop all cache so next read will read from disk
> +_scratch_unmount
> +_mount $SPARE_DEV $SCRATCH_MNT
> +
> +# Now the EXTENT_DATA item still marks the extent as compressed,
> +# but the on-disk data is uncompressed, thus reading it as compressed
> +# will definitely cause EIO.
> +cat $SCRATCH_MNT/nodatasum_file > /dev/null
> +
> +_scratch_unmount
> +_spare_dev_put
> +_scratch_dev_pool_put
> +
> +echo "Silence is golden"
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/161.out b/tests/btrfs/161.out
> new file mode 100644
> index ..1752a243
> --- /dev/null
> +++ b/tests/btrfs/161.out
> @@ -0,0 +1,2 @@
> +QA output created by 161
> +Silence is golden
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index f04ee8d5..9195b368 100644
> --- 

Re: [PATCH v3] btrfs: Add test that checks rmdir(2) can delete a subvolume

[Eryu Guan]

On Wed, Jun 13, 2018 at 03:06:45PM +0900, Misono Tomohiro wrote:
> Add btrfs test that checks "rmdir" or "rm -r" command can delete a
> subvolume like an ordinary directory.
> 
> This behavior has been restricted long time but becomes allowed by
> following commit in the kernel:
>   btrfs: Allow rmdir(2) to delete an empty subvolume

As the patch is already upstream, I'll add commit id in reference too.

a79a464d5675 ("btrfs: Allow rmdir(2) to delete an empty subvolume")

> 
> The test will be skipped if kernel does not support the feature,
> which can be checked whether /sys/fs/btrfs/features/rmdir_subvol
> exists or not.
> 
> Reviewed-by: David Sterba 
> Signed-off-by: Misono Tomohiro 
> ---
> changelog:
>  v2 -> v3 - Skip test if kernel does not support the feature by
> checking sysfs
>   - Update license notation

Thanks a lot for the revision and using new template!

>  
>  This test should pass on kernel 4.18-rc1~ (or in current linus' master),
>  otherwise it will be skipped. Please change the test number appropriately
>  when applied as other pending tests exists.
> 
>  Thanks,
>  Tomohiro Misono
> 
>  tests/btrfs/200 | 128 
> 
>  tests/btrfs/200.out |   2 +
>  tests/btrfs/group   |   1 +
>  3 files changed, 131 insertions(+)
>  create mode 100755 tests/btrfs/200
>  create mode 100644 tests/btrfs/200.out
> 
> diff --git a/tests/btrfs/200 b/tests/btrfs/200
> new file mode 100755
> index ..15213eed
> --- /dev/null
> +++ b/tests/btrfs/200
> @@ -0,0 +1,128 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2018 Fujitsu. All Rights Reserved.
> +#
> +# FS QA Test btrfs/200
> +#
> +# QA test that checks rmdir(2) works for subvolumes like ordinary 
> directories.
> +#
> +# This behavior has been restricted long time but becomes allowed by 
> following
> +# patch in the kernel:
> +#   btrfs: Allow rmdir(2) to delete an empty subvolume
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +create_subvol()
> +{
> + $BTRFS_UTIL_PROG subvolume create $1 >> $seqres.full 2>&1
> +}
> +
> +create_snapshot()
> +{
> + $BTRFS_UTIL_PROG subvolume snapshot $@ >> $seqres.full 2>&1
> +}
> +
> +rmdir_subvol()
> +{
> + rmdir $1 >> $seqres.full 2>&1
> +}
> +
> +rm_r_subvol() {
> + rm -r $1 >> $seqres.full 2>&1
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch
> +
> +if [ ! -e /sys/fs/btrfs/features/rmdir_subvol ]; then
> + _notrun "The kernel does not support the deletion of subvolume by rmdir"
> +fi

Just notice that there's a _require_btrfs_fs_feature helper that could
do this exact check. I'll fix it on commit.

_require_btrfs_fs_feature "rmdir_subvol"

Thanks,
Eryu

> +
> +_scratch_mkfs > /dev/null 2>&1 || _fail "mkfs failed"
> +_scratch_mount
> +
> +# Check that an empty subvolume can be deleted by rmdir
> +create_subvol $SCRATCH_MNT/sub1
> +rmdir_subvol $SCRATCH_MNT/sub1 || \
> + echo "rmdir should delete an empty subvolume"
> +
> +# Check that non-empty subvolume cannot be deleted by rmdir
> +create_subvol $SCRATCH_MNT/sub2
> +touch $SCRATCH_MNT/sub2/file
> +rmdir_subvol $SCRATCH_MNT/sub2 && \
> + echo "rmdir should fail for non-empty subvolume"
> +rm $SCRATCH_MNT/sub2/file
> +rmdir_subvol $SCRATCH_MNT/sub2 || \
> + echo "rmdir should delete an empty subvolume"
> +
> +# Check that read-only empty subvolume can be deleted by rmdir
> +create_subvol $SCRATCH_MNT/sub3
> +create_snapshot -r $SCRATCH_MNT/sub3 $SCRATCH_MNT/snap
> +$BTRFS_UTIL_PROG property set $SCRATCH_MNT/sub3 ro true >> $seqres.full 2>&1
> +rmdir_subvol $SCRATCH_MNT/sub3 || \
> + echo "rmdir should delete an empty subvolume"
> +rmdir_subvol $SCRATCH_MNT/snap || \
> + echo "rmdir should delete a readonly empty subvolume"
> +
> +# Check that the default subvolume cannot be deleted by rmdir
> +create_subvol $SCRATCH_MNT/sub4
> +subvolid=$(_btrfs_get_subvolid $SCRATCH_MNT sub4)
> +$BTRFS_UTIL_PROG subvolume set-default $subvolid $SCRATCH_MNT \
> + >> $seqres.full 2>&1
> +rmdir_subvol $SCRATCH_MNT/sub4 && \
> + echo "rmdir should fail for the default subvolume"
> +
> +# Check that subvolume stub (created by snapshot) can be deleted by rmdir
> +# (Note: this has been always allowed)
> +create_subvol $SCRATCH_MNT/sub5
> +create_subvol $SCRATCH_MNT/sub5/sub6
> +create_snapshot $SCRATCH_MNT/sub5 $SCRATCH_MNT/snap2
> +rmdir $SCRATCH_MNT/snap2/sub6 || \
> + echo "rmdir should delete a subvolume stub (ino number == 2)"
> +
> +# Check that rm -r works for both non-snapshot 

Re: [PATCH] generic: add test for fsync of directory after creating hard link

[Eryu Guan]

On Mon, Jun 11, 2018 at 07:24:35PM +0100, fdman...@kernel.org wrote:
> From: Filipe Manana 
> 
> Test that if we create a new hard link for a file which was previously
> fsync'ed, fsync a parent directory of the new hard link and power fail,
> the parent directory exists after mounting the filesystem again. The
> parent directory must be a new directory, not yet persisted.
> 
> This test is motivated by a bug found in btrfs, where the fsync'ed parent
> directory was lost after a power failure. The bug in btrfs is fixed by a
> patch for the linux kernel titled:
> 
>  "Btrfs: sync log after logging new name"
> 
> Signed-off-by: Filipe Manana 

Looks good to me.

> ---
>  tests/generic/498 | 65 
> +++
>  tests/generic/498.out |  2 ++
>  tests/generic/group   |  1 +
>  3 files changed, 68 insertions(+)
>  create mode 100755 tests/generic/498
>  create mode 100644 tests/generic/498.out
> 
> diff --git a/tests/generic/498 b/tests/generic/498
> new file mode 100755
> index ..1cf73bda
> --- /dev/null
> +++ b/tests/generic/498
> @@ -0,0 +1,65 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (C) 2018 SUSE Linux Products GmbH. All Rights Reserved.
> +#
> +# FS QA Test No. 498
> +#
> +# Test that if we create a new hard link for a file which was previously
> +# fsync'ed, fsync a parent directory of the new hard link and power fail,
> +# the parent directory exists after mounting the filesystem again.
> +#

Thanks a lot for using new test template!

Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] fstests: btrfs: Test if btrfs will corrupt nodatasum compressed extent when replacing device

[Eryu Guan]

On Fri, Jun 01, 2018 at 09:34:48AM +0800, Qu Wenruo wrote:
> This is a long existing bug (from 2012) but exposed by a reporter
> recently, that when compressed extent without data csum get written to
> device-replace target device, the written data is in fact uncompressed data
> other than the original compressed data.
> 
> And since btrfs still consider the data is compressed and will try to read it
> as compressed, it can cause read error.
> 
> The root cause is located, and one RFC patch already sent to fix it,
> titled "[PATCH RFC] btrfs: scrub: Don't use inode pages for device replace".
> (The RFC is only for the extra possible way to fix the bug, the fix
> itself should work without problem)
> 
> Reported-by: James Harvey 
> Signed-off-by: Qu Wenruo 

Looks fine to me overall, some minor issues inline. But I'd really like
an explicit ACK from btrfs folks.

> ---
>  tests/btrfs/161 | 91 +
>  tests/btrfs/161.out |  2 +
>  tests/btrfs/group   |  1 +
>  3 files changed, 94 insertions(+)
>  create mode 100755 tests/btrfs/161
>  create mode 100644 tests/btrfs/161.out
> 
> diff --git a/tests/btrfs/161 b/tests/btrfs/161
> new file mode 100755
> index ..d4a2b474
> --- /dev/null
> +++ b/tests/btrfs/161
> @@ -0,0 +1,91 @@
> +#! /bin/bash
> +# FS QA Test 161
> +#
> +# Test if btrfs will corrupt compressed data extent without data csum
> +# by replacing it with uncompressed data, when doing replacing device.
> +#
> +# This could be fixed by the following RFC patch:
> +# "[PATCH RFC] btrfs: scrub: Don't use inode pages for device replace"
> +#
> +#---
> +# Copyright (C) 2018 SUSE Linux Products GmbH. All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_test

This is not needed, there's no point running fsck on $TEST_DEV after
test when it's not used at all.

> +_require_scratch_dev_pool 2
> +_require_scratch_dev_pool_equal_size
> +
> +
> +_scratch_dev_pool_get 1
> +_spare_dev_get
> +_scratch_pool_mkfs >> $seqres.full 2>&1
> +
> +# Create nodatasum inode
> +_scratch_mount "-o nodatasum"
> +touch $SCRATCH_MNT/nodatasum_file
> +_scratch_remount "datasum,compress"
> +_pwrite_byte 0xcd 0 128K $SCRATCH_MNT/nodatasum_file > /dev/null
> +
> +# Write the compressed data back to disk
> +sync
> +
> +# Replace the device
> +_run_btrfs_util_prog replace start -Bf 1 $SPARE_DEV $SCRATCH_MNT
> +
> +_scratch_unmount
> +
> +_mount $SPARE_DEV $SCRATCH_MNT

Better to explain the cycle mount with comments.

> +
> +# Since now the compressed extent contains *UNCOMPRESSED* data, reading it 
> will
> +# easily trigger a EIO error
> +cat $SCRATCH_MNT/nodatasum_file > /dev/null
> +
> +_scratch_unmount
> +_spare_dev_put
> +_scratch_dev_pool_put
> +
> +echo "Silence is golden"
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/161.out b/tests/btrfs/161.out
> new file mode 100644
> index ..1752a243
> --- /dev/null
> +++ b/tests/btrfs/161.out
> @@ -0,0 +1,2 @@
> +QA output created by 161
> +Silence is golden
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index f04ee8d5..f900b3d0 100644
> --- a/tests/btrfs/group
> +++ b/tests/btrfs/group
> @@ -163,3 +163,4 @@
>  158 auto quick raid scrub
>  159 auto quick
>  160 auto quick
> +161 auto quick replace

Anand introduced a new 'volume' group in commit 144c8463d38b ("btrfs:
introduce btrfs/volume group"), all 'replace' tests should be in
'volume' group now.

Thanks,
Eryu

> -- 
> 2.17.0
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To 

Re: [PATCH v2 1/4] fstests: btrfs: add seed sprout functionality test

[Eryu Guan]

On Mon, May 28, 2018 at 05:51:45PM +0800, Anand Jain wrote:
> Create a seed device and add the sprout device to it.
> 
> Signed-off-by: Anand Jain 

This series looks fine to me from fstests' point of view, there're just
some really minor common issues.

But I'd like some reviews from other btrfs folks.

> ---
> v1->v2: Use functions to do the respective operations. Add data
> verification.
> 
>  common/config   |  1 +
>  tests/btrfs/161 | 90 
> +
>  tests/btrfs/161.out |  9 ++
>  tests/btrfs/group   |  1 +
>  4 files changed, 101 insertions(+)
>  create mode 100755 tests/btrfs/161
>  create mode 100644 tests/btrfs/161.out
> 
> diff --git a/common/config b/common/config
> index af360cefc804..f0735cb0153d 100644
> --- a/common/config
> +++ b/common/config
> @@ -235,6 +235,7 @@ case "$HOSTOS" in
>  export BTRFS_UTIL_PROG="`set_prog_path btrfs`"
>  export BTRFS_SHOW_SUPER_PROG="`set_prog_path btrfs-show-super`"
>   export BTRFS_CONVERT_PROG="`set_prog_path btrfs-convert`"
> + export BTRFS_TUNE_PROG="`set_prog_path btrfstune`"
>  export XFS_FSR_PROG="`set_prog_path xfs_fsr`"
>  export MKFS_NFS_PROG="false"
>  export MKFS_CIFS_PROG="false"
> diff --git a/tests/btrfs/161 b/tests/btrfs/161
> new file mode 100755
> index ..009e95354e85
> --- /dev/null
> +++ b/tests/btrfs/161
> @@ -0,0 +1,90 @@
> +#! /bin/bash
> +# FS QA Test 161 seed sprout functionality test

It'd be good to put the all test descriptions below the test seq number,
as all other tests do.

> +#
> +#  Create a seed device, mount it and, add a new device to create a
> +#  sprout filesystem.
> +#
> +#---
> +# Copyright (c) 2018 Oracle.  All Rights Reserved.
> +# Author: Anand Jain 

The author info is not needed.

> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch_dev_pool 2

_require_command "$BTRFS_TUNE_PROG" btrfstune

> +
> +_scratch_dev_pool_get 2
> +
> +DEV_SEED=$(echo $SCRATCH_DEV_POOL | awk '{print $1}')
> +DEV_SPROUT=$(echo $SCRATCH_DEV_POOL | awk '{print $2}')

Better to use lower case for local variables.

Thanks,
Eryu

> +
> +create_seed()
> +{
> + _mkfs_dev $DEV_SEED
> + run_check _mount $DEV_SEED $SCRATCH_MNT
> + $XFS_IO_PROG -f -d -c "pwrite -S 0xab 0 256K" $SCRATCH_MNT/foobar >\
> + /dev/null
> + echo -- golden --
> + od -x $SCRATCH_MNT/foobar
> + _run_btrfs_util_prog filesystem show -m $SCRATCH_MNT
> + _scratch_unmount
> + $BTRFS_TUNE_PROG -S 1 $DEV_SEED
> + run_check _mount $DEV_SEED $SCRATCH_MNT
> +}
> +
> +create_sprout()
> +{
> + _run_btrfs_util_prog device add -f $DEV_SPROUT $SCRATCH_MNT
> + _scratch_unmount
> + run_check _mount $DEV_SPROUT $SCRATCH_MNT
> + echo -- sprout --
> + od -x $SCRATCH_MNT/foobar
> + _scratch_unmount
> +}
> +
> +create_seed
> +create_sprout
> +
> +_scratch_dev_pool_put
> +
> +status=0
> +exit
> diff --git a/tests/btrfs/161.out b/tests/btrfs/161.out
> new file mode 100644
> index ..363b8217f45c
> --- /dev/null
> +++ b/tests/btrfs/161.out
> @@ -0,0 +1,9 @@
> +QA output created by 161
> +-- golden --
> +000 abab abab abab abab abab abab abab abab
> +*
> +100
> +-- sprout --
> +000 abab abab abab abab abab abab abab abab
> +*
> +100
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index f04ee8d5297c..fe83631f0f33 100644
> --- a/tests/btrfs/group
> +++ b/tests/btrfs/group
> @@ -163,3 +163,4 @@
>  158 auto quick raid scrub
>  159 auto quick
>  160 auto quick
> +161 auto quick
> -- 
> 2.7.0
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body 

Re: [PATCH v2 5/5] generic: test invalid swap file activation

[Eryu Guan]

On Fri, May 18, 2018 at 07:37:07AM -0700, Darrick J. Wong wrote:
> On Wed, May 16, 2018 at 01:38:49PM -0700, Omar Sandoval wrote:
> > From: Omar Sandoval 
> > 
> > Swap files cannot have holes, and they must at least two pages.
> > swapon(8) and mkswap(8) have stricter restrictions, so add versions of
> > those commands without any restrictions.
> > 
> > Signed-off-by: Omar Sandoval 
> > ---
> >  .gitignore|  2 ++
> >  src/Makefile  |  2 +-
> >  src/mkswap.c  | 83 +++
> >  src/swapon.c  | 24 +
> >  tests/generic/490 | 77 +++
> >  tests/generic/490.out |  5 +++
> >  tests/generic/group   |  1 +
> >  7 files changed, 193 insertions(+), 1 deletion(-)
> >  create mode 100644 src/mkswap.c
> >  create mode 100644 src/swapon.c
> >  create mode 100755 tests/generic/490
> >  create mode 100644 tests/generic/490.out
> > 
> > diff --git a/.gitignore b/.gitignore
> > index 53029e24..efc73a7c 100644
> > --- a/.gitignore
> > +++ b/.gitignore
> > @@ -92,6 +92,7 @@
> >  /src/lstat64
> >  /src/makeextents
> >  /src/metaperf
> > +/src/mkswap
> >  /src/mmapcat
> >  /src/multi_open_unlink
> >  /src/nametest
> > @@ -111,6 +112,7 @@
> >  /src/seek_sanity_test
> >  /src/stale_handle
> >  /src/stat_test
> > +/src/swapon
> >  /src/t_access_root
> >  /src/t_dir_offset
> >  /src/t_dir_offset2
> > diff --git a/src/Makefile b/src/Makefile
> > index c42d3bb1..01fe99ef 100644
> > --- a/src/Makefile
> > +++ b/src/Makefile
> > @@ -26,7 +26,7 @@ LINUX_TARGETS = xfsctl bstat t_mtab getdevicesize 
> > preallo_rw_pattern_reader \
> > renameat2 t_getcwd e4compact test-nextquota punch-alternating \
> > attr-list-by-handle-cursor-test listxattr dio-interleaved t_dir_type \
> > dio-invalidate-cache stat_test t_encrypted_d_revalidate \
> > -   attr_replace_test
> > +   attr_replace_test swapon mkswap
> >  
> >  SUBDIRS = log-writes perf
> >  
> > diff --git a/src/mkswap.c b/src/mkswap.c
> > new file mode 100644
> > index ..d0bce2bd
> > --- /dev/null
> > +++ b/src/mkswap.c
> > @@ -0,0 +1,83 @@
> > +/* mkswap(8) without any sanity checks */
> > +
> > +#include 
> > +#include 
> > +#include 
> > +#include 
> > +#include 
> > +#include 
> > +#include 
> > +
> > +struct swap_header {
> > +   charbootbits[1024];
> > +   uint32_tversion;
> > +   uint32_tlast_page;
> > +   uint32_tnr_badpages;
> > +   unsigned char   sws_uuid[16];
> > +   unsigned char   sws_volume[16];
> > +   uint32_tpadding[117];
> > +   uint32_tbadpages[1];
> > +};
> > +
> > +int main(int argc, char **argv)
> > +{
> > +   struct swap_header *hdr;
> > +   FILE *file;
> > +   struct stat st;
> > +   long page_size;
> > +   int ret;
> > +
> > +   if (argc != 2) {
> > +   fprintf(stderr, "usage: %s PATH\n", argv[0]);
> > +   return EXIT_FAILURE;
> > +   }
> > +
> > +   page_size = sysconf(_SC_PAGESIZE);
> > +   if (page_size == -1) {
> > +   perror("sysconf");
> > +   return EXIT_FAILURE;
> > +   }
> > +
> > +   hdr = calloc(1, page_size);
> > +   if (!hdr) {
> > +   perror("calloc");
> > +   return EXIT_FAILURE;
> > +   }
> > +
> > +   file = fopen(argv[1], "r+");
> > +   if (!file) {
> > +   perror("fopen");
> > +   free(hdr);
> > +   return EXIT_FAILURE;
> > +   }
> > +
> > +   ret = fstat(fileno(file), );
> > +   if (ret) {
> > +   perror("fstat");
> > +   free(hdr);
> > +   fclose(file);
> > +   return EXIT_FAILURE;
> > +   }
> > +
> > +   hdr->version = 1;
> > +   hdr->last_page = st.st_size / page_size - 1;
> > +   memset(>sws_uuid, 0x99, sizeof(hdr->sws_uuid));
> > +   memcpy((char *)hdr + page_size - 10, "SWAPSPACE2", 10);
> > +
> > +   if (fwrite(hdr, page_size, 1, file) != 1) {
> > +   perror("fwrite");
> > +   free(hdr);
> > +   fclose(file);
> > +   return EXIT_FAILURE;
> > +   }
> > +
> > +   if (fclose(file) == EOF) {
> > +   perror("fwrite");
> > +   free(hdr);
> > +   return EXIT_FAILURE;
> > +   }
> > +
> > +   free(hdr);
> > +
> > +   return EXIT_SUCCESS;
> > +}
> > diff --git a/src/swapon.c b/src/swapon.c
> > new file mode 100644
> > index ..0cb7108a
> > --- /dev/null
> > +++ b/src/swapon.c
> > @@ -0,0 +1,24 @@
> > +/* swapon(8) without any sanity checks; simply calls swapon(2) directly. */
> > +
> > +#include 
> > +#include 
> > +#include 
> > +#include 
> > +
> > +int main(int argc, char **argv)
> > +{
> > +   int ret;
> > +
> > +   if (argc != 2) {
> > +   fprintf(stderr, "usage: %s PATH\n", argv[0]);
> > +   return EXIT_FAILURE;
> > +   }
> > +
> > +   ret = swapon(argv[1], 0);
> > +   if (ret) {
> > +   perror("swapon");
> > +   return EXIT_FAILURE;
> > +   }
> > +
> > +   return EXIT_SUCCESS;
> > +}
> > diff --git a/tests/generic/490 

Re: [PATCH v2 3/5] generic: add test for dedupe on an active swapfile

[Eryu Guan]

On Wed, May 16, 2018 at 01:38:47PM -0700, Omar Sandoval wrote:
> From: Omar Sandoval 
> 
> Similar to generic/356 that makes sure we can't reflink an active
  ^^^ dedupe
I'll fix it on commit.

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 2/5] generic: enable swapfile tests on Btrfs

[Eryu Guan]

On Wed, May 16, 2018 at 01:38:46PM -0700, Omar Sandoval wrote:
> From: Omar Sandoval 
> 
> Commit 8c96cfbfe530 ("generic/35[67]: disable swapfile tests on Btrfs")
> disabled the swapfile tests on Btrfs because it did not support
> swapfiles at the time. Now that we're adding support, we want these

So currently btrfs has no swapfile support yet, and tests still _notrun
with current v4.17-rc5 kernel? Just want to make sure that's expected.

> tests to run, but they don't. _require_scratch_swapfile always fails for
> Btrfs because swapfiles on Btrfs must be set to nocow. After fixing
> that, generic/356 and generic/357 fail for the same reason. After fixing
> _that_, both tests still fail because we don't allow reflinking a
> non-checksummed extent (which nocow implies) to a checksummed extent.
> Add a helper for formatting a swap file which does the chattr, and
> chattr the second file, which gets these tests running on kernels
> supporting Btrfs swapfiles.
> 
> Signed-off-by: Omar Sandoval 
> ---
>  common/rc | 15 ---
>  tests/generic/356 |  7 ---
>  tests/generic/357 |  6 +++---
>  3 files changed, 19 insertions(+), 9 deletions(-)
> 
> diff --git a/common/rc b/common/rc
> index ffe53236..814b8b5c 100644
> --- a/common/rc
> +++ b/common/rc
> @@ -,6 +,17 @@ _require_odirect()
>   rm -f $testfile 2>&1 > /dev/null
>  }
>  
> +_format_swapfile() {
> + local fname="$1"
> + local sz="$2"
> +
> + touch "$fname"

Better to remove $fname first, just in case it's an existing file with
some blocks allocated, as chattr(1) says

"Note: For btrfs, the 'C' flag should be set on new or empty files.  If
it is set on a file which already has data blocks, it is undefined when
the blocks assigned to the file will be fully stable"

> + chmod 0600 "$fname"
> + $CHATTR_PROG +C "$fname" > /dev/null 2>&1

It'd be good to have some comments on this chattr +C.

> + _pwrite_byte 0x61 0 "$sz" "$fname" >> $seqres.full
> + mkswap -U 27376b42-ff65-42ca-919f-6c9b62292a5c "$fname" >> $seqres.full

Is the label really needed?

Thanks,
Eryu

> +}
> +
>  # Check that the filesystem supports swapfiles
>  _require_scratch_swapfile()
>  {
> @@ -2231,10 +2242,8 @@ _require_scratch_swapfile()
>   _scratch_mount
>  
>   # Minimum size for mkswap is 10 pages
> - local size=$(($(get_page_size) * 10))
> + _format_swapfile "$SCRATCH_MNT/swap" $(($(get_page_size) * 10))
>  
> - _pwrite_byte 0x61 0 "$size" "$SCRATCH_MNT/swap" >/dev/null 2>&1
> - mkswap "$SCRATCH_MNT/swap" >/dev/null 2>&1
>   if ! swapon "$SCRATCH_MNT/swap" >/dev/null 2>&1; then
>   _scratch_unmount
>   _notrun "swapfiles are not supported"
> diff --git a/tests/generic/356 b/tests/generic/356
> index 51eeb652..b4a38f84 100755
> --- a/tests/generic/356
> +++ b/tests/generic/356
> @@ -59,11 +59,12 @@ blocks=160
>  blksz=65536
>  
>  echo "Initialize file"
> -echo >> $seqres.full
> -_pwrite_byte 0x61 0 $((blocks * blksz)) $testdir/file1 >> $seqres.full
> -mkswap -U 27376b42-ff65-42ca-919f-6c9b62292a5c $testdir/file1 >> $seqres.full
> +_format_swapfile "$testdir/file1" $((blocks * blksz))
>  swapon $testdir/file1
>  
> +touch "$testdir/file2"
> +$CHATTR_PROG +C "$testdir/file2" >/dev/null 2>&1
> +
>  echo "Try to reflink"
>  _cp_reflink $testdir/file1 $testdir/file2 2>&1 | _filter_scratch
>  
> diff --git a/tests/generic/357 b/tests/generic/357
> index 0dd0c10f..9a83a283 100755
> --- a/tests/generic/357
> +++ b/tests/generic/357
> @@ -59,9 +59,9 @@ blocks=160
>  blksz=65536
>  
>  echo "Initialize file"
> -echo >> $seqres.full
> -_pwrite_byte 0x61 0 $((blocks * blksz)) $testdir/file1 >> $seqres.full
> -mkswap -U 27376b42-ff65-42ca-919f-6c9b62292a5c $testdir/file1 >> $seqres.full
> +_format_swapfile "$testdir/file1" $((blocks * blksz))
> +touch "$testdir/file2"
> +$CHATTR_PROG +C "$testdir/file2" >/dev/null 2>&1
>  _cp_reflink $testdir/file1 $testdir/file2 2>&1 | _filter_scratch
>  
>  echo "Try to swapon"
> -- 
> 2.17.0
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs: test ENOSPC caused by many orphan items

[Eryu Guan]

On Tue, May 15, 2018 at 07:14:02PM -0700, Omar Sandoval wrote:
> On Wed, May 16, 2018 at 09:48:58AM +0800, Eryu Guan wrote:
> > On Wed, May 09, 2018 at 11:21:55PM -0700, Omar Sandoval wrote:
> > > From: Omar Sandoval <osan...@fb.com>
> > > 
> > > Btrfs has a bug where we can prematurely ENOSPC if we have lots of
> > > orphaned files, i.e., deleted files which are still open. Add a test
> > > which repeatedly creates and deletes a file while keeping all of the
> > > file descriptors open. This should succeed but doesn't on Btrfs without
> > > the fix.
> > > 
> > > Signed-off-by: Omar Sandoval <osan...@fb.com>
> > > ---
> > >  tests/generic/479 |  0
> > >  tests/generic/487 | 65 +++
> > >  tests/generic/487.out |  2 ++
> > >  tests/generic/group   |  1 +
> > >  4 files changed, 68 insertions(+)
> > >  mode change 100644 => 100755 tests/generic/479
> > >  create mode 100755 tests/generic/487
> > >  create mode 100644 tests/generic/487.out
> > > 
> > > diff --git a/tests/generic/479 b/tests/generic/479
> > > old mode 100644
> > > new mode 100755
> > > diff --git a/tests/generic/487 b/tests/generic/487
> > > new file mode 100755
> > > index ..66379cf0
> > > --- /dev/null
> > > +++ b/tests/generic/487
> > > @@ -0,0 +1,65 @@
> > > +#! /bin/bash
> > > +# FS QA Test 487
> > > +#
> > > +# Test having many file descriptors referring to deleted files open. 
> > > Regression
> > > +# test for patch "Btrfs: fix ENOSPC caused by orphan items reservations".
> > > +#
> > > +#---
> > > +# Copyright (c) 2018 Omar Sandoval.  All Rights Reserved.
> > > +#
> > > +# This program is free software; you can redistribute it and/or
> > > +# modify it under the terms of the GNU General Public License as
> > > +# published by the Free Software Foundation.
> > > +#
> > > +# This program is distributed in the hope that it would be useful,
> > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > > +# GNU General Public License for more details.
> > > +#
> > > +# You should have received a copy of the GNU General Public License
> > > +# along with this program; if not, write the Free Software Foundation,
> > > +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> > > +#---
> > > +#
> > > +
> > > +seq=`basename $0`
> > > +seqres=$RESULT_DIR/$seq
> > > +echo "QA output created by $seq"
> > > +
> > > +here=`pwd`
> > > +tmp=/tmp/$$
> > > +status=1 # failure is the default!
> > > +trap "_cleanup; exit \$status" 0 1 2 3 15
> > > +
> > > +_cleanup()
> > > +{
> > > + cd /
> > > + rm -f $tmp.*
> > > +}
> > > +
> > > +. ./common/rc
> > > +. ./common/filter
> > > +
> > > +rm -f $seqres.full
> > > +
> > > +_supported_fs generic
> > > +_supported_os Linux
> > > +_require_scratch
> > > +
> > > +_scratch_mkfs_sized $((1024 * 1024 * 1024)) >> $seqres.full 2>&1
> > > +_scratch_mount
> > > +
> > > +test_file="$SCRATCH_MNT/$seq"
> > > +
> > > +(
> > > +ulimit -n $((16 * 1024))
> > > +# ~1 files on a 1 GB filesystem should be no problem.
> > > +for ((i = 1000; i < 1; i++)); do
> > > + eval "exec $i<> \"$test_file\"" && rm "$test_file" || break
> > > +done
> > 
> > There's a helper command in src that does exactly this job, e.g.
> > 
> > $here/src/multi_open_unlink -f $SCRATCH_MNT/$seq -n 1 -s 0
> > 
> > which creates & unlinks 1 files and keeps them open for 0 second in
> > $SCRATCH_MNT using "$seq" as name prefix. This reduces the test run time
> > from 13s to 1s for me.
> > 
> > It's a straightforward change, I'll just update on commit, please let me
> > know if you have different thoughts.
> > 
> > Thanks,
> > Eryu
> 
> Great, as long as it still reproduces the bug without the fix applied,
> that's perfect. Thanks!

Yes, I've verified that it still reproduces the bug on btrfs and test
passes with the fix(es) applied.

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs: test ENOSPC caused by many orphan items

[Eryu Guan]

On Wed, May 09, 2018 at 11:21:55PM -0700, Omar Sandoval wrote:
> From: Omar Sandoval 
> 
> Btrfs has a bug where we can prematurely ENOSPC if we have lots of
> orphaned files, i.e., deleted files which are still open. Add a test
> which repeatedly creates and deletes a file while keeping all of the
> file descriptors open. This should succeed but doesn't on Btrfs without
> the fix.
> 
> Signed-off-by: Omar Sandoval 
> ---
>  tests/generic/479 |  0
>  tests/generic/487 | 65 +++
>  tests/generic/487.out |  2 ++
>  tests/generic/group   |  1 +
>  4 files changed, 68 insertions(+)
>  mode change 100644 => 100755 tests/generic/479
>  create mode 100755 tests/generic/487
>  create mode 100644 tests/generic/487.out
> 
> diff --git a/tests/generic/479 b/tests/generic/479
> old mode 100644
> new mode 100755
> diff --git a/tests/generic/487 b/tests/generic/487
> new file mode 100755
> index ..66379cf0
> --- /dev/null
> +++ b/tests/generic/487
> @@ -0,0 +1,65 @@
> +#! /bin/bash
> +# FS QA Test 487
> +#
> +# Test having many file descriptors referring to deleted files open. 
> Regression
> +# test for patch "Btrfs: fix ENOSPC caused by orphan items reservations".
> +#
> +#---
> +# Copyright (c) 2018 Omar Sandoval.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +. ./common/rc
> +. ./common/filter
> +
> +rm -f $seqres.full
> +
> +_supported_fs generic
> +_supported_os Linux
> +_require_scratch
> +
> +_scratch_mkfs_sized $((1024 * 1024 * 1024)) >> $seqres.full 2>&1
> +_scratch_mount
> +
> +test_file="$SCRATCH_MNT/$seq"
> +
> +(
> +ulimit -n $((16 * 1024))
> +# ~1 files on a 1 GB filesystem should be no problem.
> +for ((i = 1000; i < 1; i++)); do
> + eval "exec $i<> \"$test_file\"" && rm "$test_file" || break
> +done

There's a helper command in src that does exactly this job, e.g.

$here/src/multi_open_unlink -f $SCRATCH_MNT/$seq -n 1 -s 0

which creates & unlinks 1 files and keeps them open for 0 second in
$SCRATCH_MNT using "$seq" as name prefix. This reduces the test run time
from 13s to 1s for me.

It's a straightforward change, I'll just update on commit, please let me
know if you have different thoughts.

Thanks,
Eryu

> +)
> +
> +echo "Silence is golden"
> +
> +status=0
> +exit
> diff --git a/tests/generic/487.out b/tests/generic/487.out
> new file mode 100644
> index ..5f31fd97
> --- /dev/null
> +++ b/tests/generic/487.out
> @@ -0,0 +1,2 @@
> +QA output created by 487
> +Silence is golden
> diff --git a/tests/generic/group b/tests/generic/group
> index 505383f7..93581257 100644
> --- a/tests/generic/group
> +++ b/tests/generic/group
> @@ -489,3 +489,4 @@
>  484 auto quick
>  485 auto quick insert
>  486 auto quick attr
> +487 auto quick
> -- 
> 2.17.0
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] test online label ioctl

[Eryu Guan]

On Mon, Apr 30, 2018 at 04:43:18PM -0500, Eric Sandeen wrote:
> This tests the online label ioctl that btrfs has, which has been
> recently proposed for XFS.
> 
> To run, it requires an updated xfs_io with the label command and a
> filesystem that supports it
> 
> A slight change here to _require_xfs_io_command as well, so that tests
> which simply fail with "Inappropriate ioctl" can be caught in the
> common case.
> 
> Signed-off-by: Eric Sandeen 
> ---
> 
> this passes on btrfs, _notruns on xfs/ext4 of yore, and passes
> on xfs w/ my online label patchset (as long as xfs_io has the new
> capability)
> 
> diff --git a/common/rc b/common/rc
> index 9ffab7f..c53a721 100644
> --- a/common/rc
> +++ b/common/rc
> @@ -2158,6 +2158,9 @@ _require_xfs_io_command()
>   echo $testio | grep -q "Inappropriate ioctl" && \
>   _notrun "xfs_io $command support is missing"
>   ;;
> + "label")
> + testio=`$XFS_IO_PROG -c "label" $TEST_DIR 2>&1`
> + ;;
>   "open")
>   # -c "open $f" is broken in xfs_io <= 4.8. Along with the fix,
>   # a new -C flag was introduced to execute one shot commands.
> @@ -2196,7 +2199,7 @@ _require_xfs_io_command()
>   rm -f $testfile 2>&1 > /dev/null
>   echo $testio | grep -q "not found" && \
>   _notrun "xfs_io $command support is missing"
> - echo $testio | grep -q "Operation not supported" && \
> + echo $testio | grep -q "Operation not supported\|Inappropriate ioctl" 
> && \
>   _notrun "xfs_io $command failed (old kernel/wrong fs?)"
>   echo $testio | grep -q "Invalid" && \
>   _notrun "xfs_io $command failed (old kernel/wrong fs/bad args?)"
> diff --git a/tests/generic/485 b/tests/generic/485
> new file mode 100755
> index 000..79902c2
> --- /dev/null
> +++ b/tests/generic/485
> @@ -0,0 +1,99 @@
> +#! /bin/bash
> +# FS QA Test 485
> +#
> +# Test the online filesystem label set/get ioctls
> +#
> +#---
> +# Copyright (c) 2018 Red Hat, Inc.  All Rights Reserved.
> +# Author: Eric Sandeen 
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +_supported_fs generic
> +_supported_os Linux
> +_require_scratch
> +_require_xfs_io_command "label"
> +
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount
> +
> +# Make sure we can set & clear the label
> +$XFS_IO_PROG -c "label label.$seq" $SCRATCH_MNT
> +$XFS_IO_PROG -c "label" $SCRATCH_MNT
> +
> +# And that userspace can see it now, while mounted
> +blkid -s LABEL $SCRATCH_DEV | _filter_scratch
> +
> +# And that the it is still there when it's unmounted
> +_scratch_unmount
> +blkid -s LABEL $SCRATCH_DEV | _filter_scratch
> +
> +# And that it persists after a remount
> +_scratch_mount
> +$XFS_IO_PROG -c "label" $SCRATCH_MNT
> +
> +# And that a too-long label is rejected, beyond the interface max:
> +LABEL=$(perl -e "print 'l' x 257;")
> +$XFS_IO_PROG -c "label $LABEL" $SCRATCH_MNT
> +
> +# And that it succeeds right at the filesystem max:
> +case $FSTYP in
> +xfs)
> + MAXLEN=12;
> + ;;
> +btrfs)
> + MAXLEN=256

Seems this should be 255, otherwise I got failure like:

-label = "MAXLABEL"
+label: Invalid argument

and MAXLEN=255 makes the test pass with btrfs.

> + ;;
> +*)
> + MAXLEN=256
> + echo "Your filesystem supports online label, please add max length"

Perhaps we can introduce a new helper similar to _require_acl_get_max()
and _notrun the test if current $FSTYP doesn't define a maxlen on
filesystem label?

> + ;;
> +esac
> +LABEL=$(perl -e "print 'o' x $MAXLEN;")
> +$XFS_IO_PROG -c "label $LABEL" $SCRATCH_MNT | sed -e 's/o\+/MAXLABEL/'
> +
> +# And that it fails just past the filesystem max:
> +let 

Re: [PATCH v2] btrfs: Add test that checks rmdir(2) can delete a subvolume

[Eryu Guan]

On Fri, Apr 27, 2018 at 06:26:35PM +0200, David Sterba wrote:
> On Fri, Apr 27, 2018 at 05:02:45PM +0900, Misono Tomohiro wrote:
> > Add btrfs test that checks "rmdir" or "rm -r" command can delete a
> > subvolume like an ordinary drectory.
> > 
> > This behavior has been restricted long time but becomes allowed by
> > following patch in the kernel:
> >   btrfs: Allow rmdir(2) to delete an empty subvolume
> 
> AFAICS this test will fail on older kernels, eg. stable versions, but
> this is not a bug that needs to be fixed by backporting patches. The
> usual way is to use the expunges, but as this is a feature coverage, I
> think the test should detect if the kernel has the feature at all.
> Similar to the fallocate modes etc.

Agreed, if the change is treated as a behavior change not a bug on old
kernels, we probably need a way to detect the case and _notrun if kernel
doesn't support it.

> 
> For the test itself, the scenarios look sufficient, so
> Reviewed-by: David Sterba 

Thanks for the review!

Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] btrfs: Add test that checks rmdir(2) can delete a subvolume

[Eryu Guan]

On Fri, Apr 27, 2018 at 05:02:45PM +0900, Misono Tomohiro wrote:
> Add btrfs test that checks "rmdir" or "rm -r" command can delete a
> subvolume like an ordinary drectory.
> 
> This behavior has been restricted long time but becomes allowed by
> following patch in the kernel:
>   btrfs: Allow rmdir(2) to delete an empty subvolume
> 
> Signed-off-by: Tomohiro Misono 

Looks good to me, thanks! But still, I'd like to see a Reviewed-by tag
from btrfs developers.

Thanks,
Eryu

> ---
> v1 -> v2
>   - rebase to current master
>   - remove underscore from local function
>   - remove source common/btrfs
>   - remove unnecessary _filter_scratch and echo messages when fail
> 
> As I will take a whole week vacation next week, my replay will be late.
> Thanks,
> Tomohiro Misono
> 
> 
>  tests/btrfs/160 | 141 
> 
>  tests/btrfs/160.out |   2 +
>  tests/btrfs/group   |   1 +
>  3 files changed, 144 insertions(+)
>  create mode 100755 tests/btrfs/160
>  create mode 100644 tests/btrfs/160.out
> 
> diff --git a/tests/btrfs/160 b/tests/btrfs/160
> new file mode 100755
> index ..86d2dca7
> --- /dev/null
> +++ b/tests/btrfs/160
> @@ -0,0 +1,141 @@
> +#! /bin/bash
> +# FS QA Test btrfs/159
> +#
> +# QA test that checks rmdir(2) works for subvolumes like ordinary 
> directories.
> +#
> +# This behavior has been restricted long time but becomes allowed by 
> following
> +# patch in the kernel:
> +#   btrfs: Allow rmdir(2) to delete an empty subvolume
> +#
> +#---
> +# Copyright (c) 2018 Fujitsu. All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +create_subvol()
> +{
> + $BTRFS_UTIL_PROG subvolume create $1 >> $seqres.full 2>&1
> +}
> +
> +create_snapshot()
> +{
> + $BTRFS_UTIL_PROG subvolume snapshot $@ >> $seqres.full 2>&1
> +}
> +
> +rmdir_subvol()
> +{
> + rmdir $1 >> $seqres.full 2>&1
> +}
> +
> +rm_r_subvol() {
> + rm -r $1 >> $seqres.full 2>&1
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch
> +
> +_scratch_mkfs > /dev/null 2>&1 || _fail "mkfs failed"
> +_scratch_mount
> +
> +# Check that an empty subvolume can be deleted by rmdir
> +create_subvol $SCRATCH_MNT/sub1
> +rmdir_subvol $SCRATCH_MNT/sub1 || \
> + echo "rmdir should delete an empty subvolume"
> +
> +# Check that non-empty subvolume cannot be deleted by rmdir
> +create_subvol $SCRATCH_MNT/sub2
> +touch $SCRATCH_MNT/sub2/file
> +rmdir_subvol $SCRATCH_MNT/sub2 && \
> + echo "rmdir should fail for non-empty subvolume"
> +rm $SCRATCH_MNT/sub2/file
> +rmdir_subvol $SCRATCH_MNT/sub2 || \
> + echo "rmdir should delete an empty subvolume"
> +
> +# Check that read-only empty subvolume can be deleted by rmdir
> +create_subvol $SCRATCH_MNT/sub3
> +create_snapshot -r $SCRATCH_MNT/sub3 $SCRATCH_MNT/snap
> +$BTRFS_UTIL_PROG property set $SCRATCH_MNT/sub3 ro true >> $seqres.full 2>&1
> +rmdir_subvol $SCRATCH_MNT/sub3 || \
> + echo "rmdir should delete an empty subvolume"
> +rmdir_subvol $SCRATCH_MNT/snap || \
> + echo "rmdir should delete a readonly empty subvolume"
> +
> +# Check that the default subvolume cannot be deleted by rmdir
> +create_subvol $SCRATCH_MNT/sub4
> +subvolid=$(_btrfs_get_subvolid $SCRATCH_MNT sub4)
> +$BTRFS_UTIL_PROG subvolume set-default $subvolid $SCRATCH_MNT \
> + >> $seqres.full 2>&1
> +rmdir_subvol $SCRATCH_MNT/sub4 && \
> + echo "rmdir should fail for the default subvolume"
> +
> +# Check that subvolume stub (created by snapshot) can be deleted by rmdir
> +# (Note: this has been always allowed)
> +create_subvol $SCRATCH_MNT/sub5
> +create_subvol $SCRATCH_MNT/sub5/sub6
> +create_snapshot $SCRATCH_MNT/sub5 

Re: [PATCH] btrfs: Add test that checks rmdir(2) can delete a subvolume

[Eryu Guan]

On Thu, Apr 19, 2018 at 04:23:35PM +0900, Misono Tomohiro wrote:
> Add btrfs test that checks "rmdir" or "rm -r" command can delete a
> subvolume like an ordinary drectory.
> 
> This behavior has been restricted long time but becomes allowed by
> following patch in the kernel:
>   btrfs: Allow rmdir(2) to delete an empty subvolume
> 
> Signed-off-by: Tomohiro Misono 

Looks fine to me overall from fstests' perspective of view, some minor
issues inline.

But I'd like a Reviewed-by tag from btrfs developers too.

> ---
>  tests/btrfs/159 | 140 
> 
>  tests/btrfs/159.out |   2 +
>  tests/btrfs/group   |   1 +
>  3 files changed, 143 insertions(+)
>  create mode 100755 tests/btrfs/159
>  create mode 100644 tests/btrfs/159.out
> 
> diff --git a/tests/btrfs/159 b/tests/btrfs/159
> new file mode 100755
> index ..2d830502
> --- /dev/null
> +++ b/tests/btrfs/159
> @@ -0,0 +1,140 @@
> +#! /bin/bash
> +# FS QA Test btrfs/159
> +#
> +# QA test that checks rmdir(2) works for subvolumes like ordinary 
> directories.
> +#
> +# This behavior has been restricted long time but becomes allowed by 
> following
> +# patch in the kernel:
> +#   btrfs: Allow rmdir(2) to delete an empty subvolume
> +#
> +#---
> +# Copyright (c) 2018 Fujitsu. All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +_create_subvol()

There's no need to name these local helper functions with the leading
underscore.

> +{
> + $BTRFS_UTIL_PROG subvolume create $1 2>&1 | \
> + _filter_scratch >> $seqres.full

No need to do _filter_scratch if the output won't go to stdout/stderr,
it's fine or even recommended to not filter the output when dumping them
to $seqres.full for debug purpose.

There're a few other places that _filter_scratch can be dropped.

> +}
> +
> +_create_snapshot()
> +{
> + $BTRFS_UTIL_PROG subvolume snapshot $@ 2>&1 | \
> + _filter_scratch >> $seqres.full
> +}
> +
> +_rmdir_subvol()
> +{
> + rmdir $1 2>&1 | _filter_scratch

I notice that we don't expect any output from the test script ("Silence
is golden" in .out file), so I don't think it's necessary to do
redirection and filter here, just do "rmdir $1" and any error message
would break golden image and fail the test.

> + return ${PIPESTATUS[0]}

If you only care about the return value, just dump the rmdir output to
$seqres.full or /dev/null.

> +}
> +
> +_rm_r_subvol() {
> + rm -r $1 2>&1 | _filter_scratch
> + return ${PIPESTATUS[0]}

Same here.

> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/btrfs

No need to source common/btrfs, it's already sourced by common/rc based
on current $FSTYP.

> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch
> +
> +_scratch_mkfs > /dev/null 2>&1 || _fail "mkfs failed"
> +_scratch_mount
> +
> +# Check that an empty subvolume can be deleted by rmdir
> +_create_subvol $SCRATCH_MNT/sub1
> +_rmdir_subvol $SCRATCH_MNT/sub1
> +
> +# Check that non-empty subvolume cannot be deleted by rmdir
> +_create_subvol $SCRATCH_MNT/sub2
> +touch $SCRATCH_MNT/sub2/file
> +_rmdir_subvol $SCRATCH_MNT/sub2 >> $seqres.full && \
> + _fail "rmdir should fail for non-empty subvolume"

Just echo the error message, instead of exiting the test, test could
continue. Or even simpler, don't rely on the return status of
_rmdir_subvol, just put any expected output from _rmdir_subvol to .out
file, e.g.

_rmdir_subvol $SCRATCH_MNT/sub2 | _filter_scratch

So that, again, any failure message could break the golden image and
fail the test.

> +rm $SCRATCH_MNT/sub2/file
> +_rmdir_subvol $SCRATCH_MNT/sub2
> +
> +# Check that read-only empty subvolume can be deleted by rmdir
> +_create_subvol 

Re: [PATCH] btrfs: add verify chattr support for send/receive test

[Eryu Guan]

[adding linux-btrfs list to cc]

On Tue, Apr 17, 2018 at 04:44:42PM -0700, Howard McLauchlan wrote:
> This test aims to verify correct behaviour with chattr operations and
> btrfs send/receive. The intent is to check general correctness as well
> as special interactions with troublesome flags(immutable, append only).
> 
> This test is motivated by a bug in btrfs which demonstrates a lack of
> chattr support in btrfs send/receive.

>From the discussion of the kernel patch thread, it looks more like a
"new feature" or a behavior change than a bug. If it's a regression test
for a pure bug fix, it's fine to fail the test on old
kernels/btrfs-progs. But if it's a "new feature" or a behavior change,
it's better to add new _require rules to make sure the kernel and/or
userspace does have chattr support in send/receive.

I'll comment from fstests' point of view and defer above question to
btrfs developers :)

> 
> A kernel patch to fix this can be found at:
> 
> btrfs: add chattr support for send/receive
> 
> The accompanying userspace patch can be found at:
> 
> btrfs-progs: add chattr support for send/receive
> 
> Signed-off-by: Howard McLauchlan 
> ---
>  tests/btrfs/159 | 202 
>  tests/btrfs/159.out |  38 +
>  tests/btrfs/group   |   1 +
>  3 files changed, 241 insertions(+)
>  create mode 100755 tests/btrfs/159
>  create mode 100644 tests/btrfs/159.out
> 
> diff --git a/tests/btrfs/159 b/tests/btrfs/159
> new file mode 100755
> index ..b3a32594
> --- /dev/null
> +++ b/tests/btrfs/159
> @@ -0,0 +1,202 @@
> +#! /bin/bash
> +# FS QA Test 159
> +#
> +# This test verifies the correct behaviour of chattr support for btrfs
> +# send/receive; 6 cases will be tested:
> +# 1. New inode created with an inode flag set
> +# 2. Existing inode with BTRFS_INODE_APPEND is written to with sequence:
> +#chattr -a
> +#pwrite something
> +#chattr +a
> +# 3. Existing inode with BTRFS_INODE_APPEND is written to with sequence:
> +#chattr -a
> +#pwrite something
> +#chattr +d
> +# 4. Existing inode is written to with sequence:
> +#setfattr something
> +#chattr +a
> +# 5. Existing inode with BTRFS_INODE_APPEND is written to with sequence:
> +#chattr -a
> +#setfattr something
> +#setfattr something else
> +#chattr +a
> +# 6. As above, but with pwrite instead of setfattr
> +# The goal of 5 and 6 is not to test correctness, but to ensure we
> +# don't send extra chattrs that are unnecessary.
> +#
> +# We verify the md5sum of the snapshots in the receive directory to ensure 
> file
> +# contents have changed appropriately. We also observe the flags changing (or
> +# not changing) as appropriate.
> +#
> +#---
> +# Copyright (c) 2018 Facebook.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch
> +
> +send_files_dir=$TEST_DIR/btrfs-test-$seq

Take use of $TEST_DIR, so we need _require_test too.

And we may need _require_attrs and _require_test_lsattr too.

> +
> +rm -rf $send_files_dir
> +mkdir $send_files_dir
> +
> +_scratch_mkfs >>$seqres.full 2>&1
> +_scratch_mount
> +
> +# Create receive directory
> +mkdir $SCRATCH_MNT/receive
> +
> +# Create test file and set chattr flag
> +_run_btrfs_util_prog subvolume create $SCRATCH_MNT/parent
> +$XFS_IO_PROG -f -c "pwrite -S 0xaa 0K 32K" $SCRATCH_MNT/parent/foo | 
> _filter_xfs_io
> +$CHATTR_PROG +a $SCRATCH_MNT/parent/foo
> +
> +# Send/Receive initial snapshot
> +_run_btrfs_util_prog subvolume snapshot -r $SCRATCH_MNT/parent \
> + 

Re: [PATCH v2] fstests: test btrfs fsync after hole punching with no-holes mode

[Eryu Guan]

On Mon, Apr 16, 2018 at 12:28:59PM +0100, Filipe Manana wrote:
> On Mon, Apr 9, 2018 at 2:05 PM, Eryu Guan <guane...@gmail.com> wrote:
> > On Sun, Apr 08, 2018 at 09:46:24AM +0100, Filipe Manana wrote:
> >> On Sun, Apr 8, 2018 at 8:46 AM, Eryu Guan <guane...@gmail.com> wrote:
> >> > On Wed, Mar 28, 2018 at 12:55:30PM +0100, fdman...@kernel.org wrote:
> >> >> From: Filipe Manana <fdman...@suse.com>
> >> >>
> >> >> Test that when we have the no-holes mode enabled and a specific metadata
> >> >> layout, if we punch a hole and fsync the file, at replay time the whole
> >> >> hole was preserved.
> >> >>
> >> >> This issue is fixed by the following btrfs patch for the linux kernel:
> >> >>
> >> >>   "Btrfs: fix fsync after hole punching when using no-holes feature"
> >> >>
> >> >> Signed-off-by: Filipe Manana <fdman...@suse.com>
> >> >> ---
> >> >>
> >> >> V2: Made the test work when selinux is enabled, and made it use direct 
> >> >> IO
> >> >> writes to ensure 256K extents.
> >> >
> >> > Test fails with selinux enabled now on unpatched kernel. But I found
> >> > that, in my release testing, test still fails when testing with current
> >> > Linus tree (HEAD is 642e7fd23353, without selinux this time), which
> >> > should contain the mentioned fix. Does that mean the bug is not fully
> >> > fixed?
> >>
> >> The bug is fully fixed. But HEAD 642e7fd23353 does not contain the
> >> fix. The current  linus' master has it.
> >
> > I'll double check.. Thanks for the heads-up!
> 
> Hi Eryu, any reason this didn't go in the last update?
> Thanks.

Sorry, I thought I queued it for last update, but actually I queued
"generic: test for fsync after fallocate" not this one (and I double
checked that test not this one..). I'll give it another try and queue it
for next update if all look well.

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3] fstests: btrfs/159 superblock corruption test case

[Eryu Guan]

On Sat, Apr 14, 2018 at 06:43:49AM +0800, Anand Jain wrote:
> 
> > > +# Test if the superblock corruption is handled correctly:
> > > +#- Test fsid miss-match (csum ok) between primary and copy 
> > > superblock
> > > +#Fixed by the ML patch:
> > > +#btrfs: check if the fsid in the primary sb and copy sb are same
> > > +#- Test if the mount fails if the primary superblock csum is
> > > +#corrupted on any disk
> > > +#- Test if the mount does not fail if the copy1 sb csum is 
> > > corrupted
> > > +#Fixed by the ML patches:
> > > +#btrfs: verify superblock checksum during scan
> > > +#btrfs: verify checksum for all devices in mount context
> > 
> > Do you have a tree that I can pull from? I want to make sure the test
> > does pass on patched kernel, but the patchset doesn't apply on v4.16
> > kernel.
> 
> We have new discussions on whether to check for the alien-superblock and
> the superblock-checksum at the mount and scan time respectively. And
> depending on its outcome this test-case should be modified as well. So
> can you please defer this fstest patch, for now, I shall send a revised
> fstest patch when kernel patches gets integrated.

Thanks for the heads-up, I'll drop it for now.

> 
> In any case, if you want to give a try, those patches are base on kdave repo
> at [1].
>  [1]
>  https://github.com/kdave/btrfs-devel.git misc-next

Thanks!

Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3] fstests: btrfs/159 superblock corruption test case

[Eryu Guan]

On Mon, Apr 09, 2018 at 01:28:30PM +0800, Anand Jain wrote:
> Verify if the superblock corruption is handled correctly.
> 
> Signed-off-by: Anand Jain 
> ---
> v2->v3:
>  Provide the disk to be corrupted as an arg, instead of swapping the devices,
>   so drop mount_opt_minus_args().
>  159.out slightly changed.
> v1->v2:
>  $subject slightly changed
>  Added more info about the test-case
>  Keep the stuff from the ./new btrfs
>  Add mount_opt_minus_args() to get the options (if) set at the config file
>  Move DEV_GOOD & DEV_BAD to where it starts to use
>  To help debugging added run_check where possible
>  Remove {} in the out file
>  Use _filter_error_mount for mount fail cases other than -EINVAL
> 
>  tests/btrfs/159 | 149 
> 
>  tests/btrfs/159.out |  21 
>  tests/btrfs/group   |   1 +
>  3 files changed, 171 insertions(+)
>  create mode 100755 tests/btrfs/159
>  create mode 100644 tests/btrfs/159.out
> 
> diff --git a/tests/btrfs/159 b/tests/btrfs/159
> new file mode 100755
> index ..c3a50b58b0b9
> --- /dev/null
> +++ b/tests/btrfs/159
> @@ -0,0 +1,149 @@
> +#! /bin/bash
> +# FS QA Test 159
> +#
> +# Test if the superblock corruption is handled correctly:
> +#- Test fsid miss-match (csum ok) between primary and copy superblock
> +#Fixed by the ML patch:
> +#btrfs: check if the fsid in the primary sb and copy sb are same
> +#- Test if the mount fails if the primary superblock csum is
> +#corrupted on any disk
> +#- Test if the mount does not fail if the copy1 sb csum is corrupted
> +#Fixed by the ML patches:
> +#btrfs: verify superblock checksum during scan
> +#btrfs: verify checksum for all devices in mount context

Do you have a tree that I can pull from? I want to make sure the test
does pass on patched kernel, but the patchset doesn't apply on v4.16
kernel.

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] fstests: test btrfs fsync after hole punching with no-holes mode

[Eryu Guan]

On Sun, Apr 08, 2018 at 09:46:24AM +0100, Filipe Manana wrote:
> On Sun, Apr 8, 2018 at 8:46 AM, Eryu Guan <guane...@gmail.com> wrote:
> > On Wed, Mar 28, 2018 at 12:55:30PM +0100, fdman...@kernel.org wrote:
> >> From: Filipe Manana <fdman...@suse.com>
> >>
> >> Test that when we have the no-holes mode enabled and a specific metadata
> >> layout, if we punch a hole and fsync the file, at replay time the whole
> >> hole was preserved.
> >>
> >> This issue is fixed by the following btrfs patch for the linux kernel:
> >>
> >>   "Btrfs: fix fsync after hole punching when using no-holes feature"
> >>
> >> Signed-off-by: Filipe Manana <fdman...@suse.com>
> >> ---
> >>
> >> V2: Made the test work when selinux is enabled, and made it use direct IO
> >> writes to ensure 256K extents.
> >
> > Test fails with selinux enabled now on unpatched kernel. But I found
> > that, in my release testing, test still fails when testing with current
> > Linus tree (HEAD is 642e7fd23353, without selinux this time), which
> > should contain the mentioned fix. Does that mean the bug is not fully
> > fixed?
> 
> The bug is fully fixed. But HEAD 642e7fd23353 does not contain the
> fix. The current  linus' master has it.

I'll double check.. Thanks for the heads-up!

Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] fstests: test btrfs fsync after hole punching with no-holes mode

[Eryu Guan]

On Wed, Mar 28, 2018 at 12:55:30PM +0100, fdman...@kernel.org wrote:
> From: Filipe Manana 
> 
> Test that when we have the no-holes mode enabled and a specific metadata
> layout, if we punch a hole and fsync the file, at replay time the whole
> hole was preserved.
> 
> This issue is fixed by the following btrfs patch for the linux kernel:
> 
>   "Btrfs: fix fsync after hole punching when using no-holes feature"
> 
> Signed-off-by: Filipe Manana 
> ---
> 
> V2: Made the test work when selinux is enabled, and made it use direct IO
> writes to ensure 256K extents.

Test fails with selinux enabled now on unpatched kernel. But I found
that, in my release testing, test still fails when testing with current
Linus tree (HEAD is 642e7fd23353, without selinux this time), which
should contain the mentioned fix. Does that mean the bug is not fully
fixed?

--- tests/btrfs/159.out 2018-04-03 18:25:41.566105514 -0700
+++ /root/xfstests/results//btrfs/159.out.bad   2018-04-08 00:28:26.921968949 
-0700
@@ -6,4 +6,4 @@
 File digest before power failure:
 c5c0a13588a639529c979c57c336f441  SCRATCH_MNT/foobar
 File digest after power failure and log replay:
-c5c0a13588a639529c979c57c336f441  SCRATCH_MNT/foobar
+c84746bbd1b97420f076d417a6a7d81c  SCRATCH_MNT/foobar

I'll drop this patch for now.

Thanks,
Eryu

> 
>  tests/btrfs/159 | 115 
> 
>  tests/btrfs/159.out |   9 
>  tests/btrfs/group   |   1 +
>  3 files changed, 125 insertions(+)
>  create mode 100755 tests/btrfs/159
>  create mode 100644 tests/btrfs/159.out
> 
> diff --git a/tests/btrfs/159 b/tests/btrfs/159
> new file mode 100755
> index ..eb667692
> --- /dev/null
> +++ b/tests/btrfs/159
> @@ -0,0 +1,115 @@
> +#! /bin/bash
> +# FSQA Test No. 159
> +#
> +# Test that when we have the no-holes mode enabled and a specific metadata
> +# layout, if we punch a hole and fsync the file, at replay time the whole
> +# hole was preserved.
> +#
> +#---
> +#
> +# Copyright (C) 2018 SUSE Linux Products GmbH. All Rights Reserved.
> +# Author: Filipe Manana 
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + _cleanup_flakey
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/dmflakey
> +
> +# real QA test starts here
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch
> +_require_dm_target flakey
> +_require_xfs_io_command "fpunch"
> +_require_odirect
> +
> +rm -f $seqres.full
> +
> +run_test()
> +{
> + local punch_offset=$1
> +
> + # We create the filesystem with a node size of 64Kb because we need to
> + # create a specific metadata layout in order to trigger the bug we are
> + # testing. At the moment the node size can not be smaller then the
> + # system's page size, so given that the largest possible page size is
> + # 64Kb and by default the node size is set to the system's page size
> + # value, we explicitly create a filesystem with a 64Kb node size.
> + _scratch_mkfs -O no-holes -n $((64 * 1024)) >>$seqres.full 2>&1
> + _require_metadata_journaling $SCRATCH_DEV
> + _init_flakey
> + _mount_flakey
> +
> + # Create our test file with 832 extents of 256Kb each. Before each
> + # extent, there is a 256Kb hole (except for the first extent, which
> + # starts at offset 0). This creates two leafs in the filesystem tree.
> + # We use direct IO to ensure we get exactly 256K extents (with buffered
> + # IO we can get writeback triggered at any time and therefore get
> + # extents smaller than 256K).
> + for ((i = 0; i <= 831; i++)); do
> + local offset=$((i * 2 * 256 * 1024))
> + $XFS_IO_PROG -f -d -c "pwrite -S 0xab -b 256K $offset 256K" \
> + $SCRATCH_MNT/foobar >/dev/null
> + done
> +
> + # Make sure everything done so far is durably persisted.
> + sync
> +

Re: [PATCH v2] fstests: btrfs/159 superblock corruption test case

[Eryu Guan]

On Thu, Apr 05, 2018 at 02:28:49PM +0800, Anand Jain wrote:
> Verify if the superblock corruption is handled correctly.
> 
> Signed-off-by: Anand Jain 
> ---
> v1->v2:
>  $subject slightly changed
>  Added more info about the test-case
>  Keep the stuff from the ./new btrfs
>  Add mount_opt_minus_args() to get the options (if) set at the config file
>  Move DEV_GOOD & DEV_BAD to where it starts to use
>  To help debugging added run_check where possible
>  Remove {} in the out file
>  Use _filter_error_mount for mount fail cases other than -EINVAL
> 
>  tests/btrfs/159 | 177 
> 
>  tests/btrfs/159.out |  23 +++
>  tests/btrfs/group   |   1 +
>  3 files changed, 201 insertions(+)
>  create mode 100755 tests/btrfs/159
>  create mode 100644 tests/btrfs/159.out
> 
> diff --git a/tests/btrfs/159 b/tests/btrfs/159
> new file mode 100755
> index ..521cfdab0242
> --- /dev/null
> +++ b/tests/btrfs/159
> @@ -0,0 +1,177 @@
> +#! /bin/bash
> +# FS QA Test 159
> +#
> +# Test if the superblock corruption is handled correctly:
> +#- Test fsid miss-match (csum ok) between primary and copy superblock
> +#Fixed by the ML patch:
> +#btrfs: check if the fsid in the primary sb and copy sb are same
> +#- Test if the mount fails if the primary superblock csum is
> +#corrupted on any disk
> +#- Test if the mount does not fail if the copy1 sb csum is corrupted
> +#Fixed by the ML patches:
> +#btrfs: verify superblock checksum during scan
> +#btrfs: verify checksum for all devices in mount context
> +#
> +#-
> +# Copyright (c) 2018 Oracle.  All Rights Reserved.
> +# Author: Anand Jain 
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> + _scratch_dev_pool_put
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/module
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch_dev_pool 2
> +_require_loadable_fs_module "btrfs"
> +_require_command "$WIPEFS_PROG" wipefs
> +
> +_scratch_dev_pool_get 2
> +
> +mount_opt_minus_args()
> +{
> + local nr
> + local mnt_opt=""
> +
> + nr=`_scratch_mount_options | awk '{print NF}'`
> + if [ $nr -eq 4 ]; then

Seems this only works with "scratch_mount_options" returns something
like:

"-o opt dev mnt" or "-o opt1,opt2 dev mnt"

but not

"-oopt dev mnt" nor
"-o opt1 -o opt2 dev mnt" nor if $SELINUX_MOUNT_OPTIONS not empty.

Also if MOUNT_OPTIONS is "-oopt1 -oopt2", mount_opt_minus_args would
return something like "-o -oopt2,device=", which are not valid
mount options.

> + #gets only the mount option set in the config file
> + mnt_opt=`_scratch_mount_options | awk '{print $2}'`
> + fi
> + #Append the additional opts provide as func args.
> + #Make sure -o is not echo-ed if both config file mnt-option
> + #and the test case mnt-option are null.
> + if [ -z $mnt_opt ]; then
> + if [ ! -z $* ]; then
> + echo "-o $*"
> + fi
> + else
> + if [ -z $* ]; then
> + echo "-o $mnt_opt"
> + else
> + echo "-o $mnt_opt,$*"
> + fi
> + fi
> +}
> +
> +wipe()
> +{
> + $WIPEFS_PROG -a $DEV_GOOD > /dev/null 2>&1
> + $WIPEFS_PROG -a $DEV_BAD > /dev/null 2>&1
> +}
> +
> +# Test for fsid miss-match (csum ok) with primary and copy superblock.
> +check_copy1_fsid()
> +{
> + local bytenr=67108864
> + echo -e "\\ncheck_copy1_fsid\\n" | tee -a $seqres.full
> +
> + wipe
> + $MKFS_BTRFS_PROG -fq $DEV_GOOD
> + $MKFS_BTRFS_PROG -fq $DEV_BAD
> + _reload_fs_module "btrfs"
> +
> + run_check dd status=none of=$DEV_BAD if=$DEV_GOOD ibs=1 obs=1\
> + 

Re: [PATCH] fstests: generic test for fsync after fallocate

[Eryu Guan]

On Thu, Apr 05, 2018 at 10:56:14PM +0100, fdman...@kernel.org wrote:
> From: Filipe Manana 
> 
> Test that fsync operations preserve extents allocated with fallocate(2)
> that are placed beyond a file's size.
> 
> This test is motivated by a bug found in btrfs where unwritten extents
> beyond the inode's i_size were not preserved after a fsync and power
> failure. The btrfs bug is fixed by the following patch for the linux
> kernel:
> 
>  "Btrfs: fix loss of prealloc extents past i_size after fsync log replay"
> 
> Signed-off-by: Filipe Manana 

Hmm, xfs fails this test, while ext4 passes.

# diff -u tests/generic/483.out 
/root/workspace/xfstests/results//xfs_4k_crc/generic/483.out.bad
--- tests/generic/483.out   2018-04-07 23:35:00.55511 +0800
+++ /root/workspace/xfstests/results//xfs_4k_crc/generic/483.out.bad
2018-04-07 23:39:48.780659707 +0800
@@ -6,5 +6,5 @@
 0: [0..511]: data
 1: [512..2559]: unwritten
 File baz fiemap:
-0: [0..511]: data
-1: [512..6143]: unwritten
+0: [0..895]: data
+1: [896..6143]: unwritten

I'm not sure what the problem is yet, but IMHO controlling on-disk
layout of a file from userspace is hard and should be avoided if
possible.

Why not dumping md5sum to .out file like other dmflakey tests? I've
checked the md5sum of all the three test files, and they're the same on
xfs as on ext4, so the files are not corrupted on xfs.

Thanks,
Eryu

> ---
>  tests/generic/482 | 118 
> ++
>  tests/generic/482.out |  10 +
>  tests/generic/group   |   1 +
>  3 files changed, 129 insertions(+)
>  create mode 100755 tests/generic/482
>  create mode 100644 tests/generic/482.out
> 
> diff --git a/tests/generic/482 b/tests/generic/482
> new file mode 100755
> index ..43bbc913
> --- /dev/null
> +++ b/tests/generic/482
> @@ -0,0 +1,118 @@
> +#! /bin/bash
> +# FSQA Test No. 482
> +#
> +# Test that fsync operations preserve extents allocated with fallocate(2) 
> that
> +# are placed beyond a file's size.
> +#
> +#---
> +#
> +# Copyright (C) 2018 SUSE Linux Products GmbH. All Rights Reserved.
> +# Author: Filipe Manana 
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + _cleanup_flakey
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/dmflakey
> +. ./common/punch
> +
> +# real QA test starts here
> +_supported_fs generic
> +_supported_os Linux
> +_require_scratch
> +_require_dm_target flakey
> +_require_xfs_io_command "falloc" "-k"
> +_require_xfs_io_command "fiemap"
> +
> +rm -f $seqres.full
> +
> +_scratch_mkfs >>$seqres.full 2>&1
> +_require_metadata_journaling $SCRATCH_DEV
> +_init_flakey
> +_mount_flakey
> +
> +# Create our test files.
> +$XFS_IO_PROG -f -c "pwrite -S 0xea 0 256K" $SCRATCH_MNT/foo >/dev/null
> +
> +# Create a file with many extents. We later want to shrink truncate it and
> +# add a prealloc extent beyond its new size.
> +for ((i = 1; i <= 500; i++)); do
> + offset=$(((i - 1) * 4 * 1024))
> + $XFS_IO_PROG -f -s -c "pwrite -S 0xcf $offset 4K" \
> + $SCRATCH_MNT/bar >/dev/null
> +done
> +
> +# A file which already has a prealloc extent beyond its size.
> +# The fsync done on it is motivated by differences in the btrfs 
> implementation
> +# of fsync (first fsync has different logic from subsequent fsyncs).
> +$XFS_IO_PROG -f -c "pwrite -S 0xf1 0 256K" \
> +  -c "falloc -k 256K 768K" \
> +  -c "fsync" \
> +  $SCRATCH_MNT/baz >/dev/null
> +
> +# Make sure everything done so far is durably persisted.
> +sync
> +
> +# Allocate an extent beyond the size of the first test file and fsync it.
> +$XFS_IO_PROG -c "falloc -k 256K 1M"\
> +  -c "fsync" \
> +  $SCRATCH_MNT/foo
> +
> +# Do a shrinking truncate of our test file, add a prealloc extent to it after
> +# its new size and fsync it.
> +$XFS_IO_PROG -c "truncate 256K" \
> +  -c 

Re: [PATCH typo-fixed] fstests: btrfs: 159 superblock corruption test case

[Eryu Guan]

On Thu, Mar 29, 2018 at 06:28:48PM +0800, Anand Jain wrote:
> Verify if the superblock corruption is handled correctly.
> 
> Signed-off-by: Anand Jain 
> ---
>  tests/btrfs/159 | 142 
> 
>  tests/btrfs/159.out |  35 +
>  tests/btrfs/group   |   1 +
>  3 files changed, 178 insertions(+)
>  create mode 100755 tests/btrfs/159
>  create mode 100644 tests/btrfs/159.out
> 
> diff --git a/tests/btrfs/159 b/tests/btrfs/159
> new file mode 100755
> index ..f42ba4b777e7
> --- /dev/null
> +++ b/tests/btrfs/159
> @@ -0,0 +1,142 @@
> +#! /bin/bash
> +# FS QA Test 159
> +#
> +# Test if the superblock corruption is handled correctly.
> +#   - Check and validate superblock csum in mount and scan context
> +#   - Check and validate superblock for all disks in the fs
> +#   - Make sure if the copies are really a copy of the primary superblock
> +#
> +#-
> +# Copyright (c) 2018 Oracle.  All Rights Reserved.
> +# Author: Anand Jain 
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#-
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +status=1 # failure is the default!

Please use './new btrfs' to generate new test template, which defines
variables like 'tmp' and 'here', these variables may not be used in the
test explicitly, but they could be used by some helper functions,
especially the "$tmp" var. (Or please don't remove them from template :)

> +
> +_cleanup()
> +{
> + _scratch_dev_pool_put
> +}
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/module
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch_dev_pool 2
> +_require_loadable_fs_module "btrfs"
> +_require_command "$WIPEFS_PROG" wipefs
> +
> +_scratch_dev_pool_get 2
> +
> +MNT_OPT=$(echo $MOUNT_OPTIONS | cut -d" " -f2-)

I'm not sure about the purpose of this MNT_OPT, and it could be empty
and causes problems in test. Please see below.

> +DEV_GOOD=$(echo $SCRATCH_DEV_POOL | awk '{print $1}')
> +DEV_BAD=$(echo $SCRATCH_DEV_POOL | awk '{print $2}')
> +
> +wipe()
> +{
> + $WIPEFS_PROG -a $DEV_GOOD > /dev/null 2>&1
> + $WIPEFS_PROG -a $DEV_BAD > /dev/null 2>&1
> +}
> +
> +# If copy1 fsid does not match with primary fail the scan thus the mount as 
> well
> +check_copy1_fsid()
> +{
> + local bytenr=67108864
> + echo -e "\\ncheck_copy1_fsid\\n{" | tee -a $seqres.full

Hmm, I don't think the "{ }" block is necessary in .out file, IMHO it
makes the diff context harder to read when test fails. Just echo the
test name would be fine.

> +
> + wipe
> + $MKFS_BTRFS_PROG -fq $DEV_GOOD
> + $MKFS_BTRFS_PROG -fq $DEV_BAD
> + _reload_fs_module "btrfs"
> +
> + dd status=none of=$DEV_BAD if=$DEV_GOOD ibs=1 obs=1 skip=$bytenr 
> seek=$bytenr count=4K|\
> + tee -a $seqres.full
> +
> + _mount -o $MNT_OPT $DEV_BAD $SCRATCH_MNT 2>&1 | _filter_scratch_pool

When $MOUNT_OPTIONS is empty, this mount fails as

+mount: can't find /mnt/scratch in /etc/fstab

And I think we need to umount $SCRATCH_MNT here in case a buggy btrfs
module allowed the mount, as the next test assumes the device is not
mounted anywhere. Otherwise I see failures like:

+ERROR: /dev/loop1 is mounted
+mount failed

> +
> + echo -e "good\\n}" | tee -a $seqres.full
> +}
> +
> +# As in Linux kernel 4.16 if the primary is corrupted mount will fail.
> +# Which might change in the long run.
> +check_primary()
> +{
> + local bytenr=65536
> + echo -e "\\ncheck_primary\\n{" | tee -a $seqres.full
> +
> + wipe
> + _scratch_pool_mkfs "-mraid1 -draid1"
> + _reload_fs_module "btrfs"
> +
> + #To corrupt a disk block, read in hex, write in dec
> + od -j$bytenr -N1 -An -x $DEV_BAD |\
> + dd status=none of=$DEV_BAD ibs=1 obs=1 seek=$bytenr count=1|\
> + tee -a $seqres.full
> + _mount -o $MNT_OPT,device=$DEV_GOOD $DEV_BAD $SCRATCH_MNT 2>&1 | 
> _filter_scratch_pool

Same here, need to umount 

Re: [PATCH] fstests: test btrfs fsync after hole punching with no-holes mode

[Eryu Guan]

On Wed, Mar 28, 2018 at 09:48:17AM +0100, Filipe Manana wrote:
> On Wed, Mar 28, 2018 at 3:17 AM, Eryu Guan <guane...@gmail.com> wrote:
> > On Mon, Mar 26, 2018 at 11:59:21PM +0100, fdman...@kernel.org wrote:
> >> From: Filipe Manana <fdman...@suse.com>
> >>
> >> Test that when we have the no-holes mode enabled and a specific metadata
> >> layout, if we punch a hole and fsync the file, at replay time the whole
> >> hole was preserved.
> >>
> >> This issue is fixed by the following btrfs patch for the linux kernel:
> >>
> >>   "Btrfs: fix fsync after hole punching when using no-holes feature"
> >
> > I'd expect a test failure with 4.16-rc6 kernel, as the mentioned fix
> > above is not there. But test always passes for me. Did I miss anything?
> > btrfs-progs version is btrfs-progs-4.11.1-3.fc27.
> 
> It should fail on any kernel, with any btrfs-progs version (which
> should be irrelevant).
> Somehow on your system we are not getting the specific metadata layout
> needed to trigger the issue.
> 
> Can you apply the following patch on top of the test and provide the
> result 159.full file?
> 
> https://friendpaste.com/6xAuLeN4xl1AGjO9Qc5I8L
> 
> So that I can see what metadata layout you are getting.
> Thanks!

Sure, please see attachment.

Thanks,
Eryu


btrfs-159.full.bz2
Description: BZip2 compressed data

Re: [PATCH] fstests: test btrfs fsync after hole punching with no-holes mode

[Eryu Guan]

On Mon, Mar 26, 2018 at 11:59:21PM +0100, fdman...@kernel.org wrote:
> From: Filipe Manana 
> 
> Test that when we have the no-holes mode enabled and a specific metadata
> layout, if we punch a hole and fsync the file, at replay time the whole
> hole was preserved.
> 
> This issue is fixed by the following btrfs patch for the linux kernel:
> 
>   "Btrfs: fix fsync after hole punching when using no-holes feature"

I'd expect a test failure with 4.16-rc6 kernel, as the mentioned fix
above is not there. But test always passes for me. Did I miss anything?
btrfs-progs version is btrfs-progs-4.11.1-3.fc27.

> 
> Signed-off-by: Filipe Manana 
> ---
>  tests/btrfs/159 | 100 
> 
>  tests/btrfs/159.out |   5 +++
>  tests/btrfs/group   |   1 +
>  3 files changed, 106 insertions(+)
>  create mode 100755 tests/btrfs/159
>  create mode 100644 tests/btrfs/159.out
> 
> diff --git a/tests/btrfs/159 b/tests/btrfs/159
> new file mode 100755
> index ..6083975a
> --- /dev/null
> +++ b/tests/btrfs/159
> @@ -0,0 +1,100 @@
> +#! /bin/bash
> +# FSQA Test No. 159
> +#
> +# Test that when we have the no-holes mode enabled and a specific metadata
> +# layout, if we punch a hole and fsync the file, at replay time the whole
> +# hole was preserved.
> +#
> +#---
> +#
> +# Copyright (C) 2018 SUSE Linux Products GmbH. All Rights Reserved.
> +# Author: Filipe Manana 
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + _cleanup_flakey
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/dmflakey
> +
> +# real QA test starts here
> +_supported_fs generic
 ^^^ btrfs
> +_supported_os Linux
> +_require_scratch
> +_require_dm_target flakey
> +_require_xfs_io_command "fpunch"
> +
> +rm -f $seqres.full
> +
> +# We create the filesystem with a node size of 64Kb because we need to 
> create a
> +# specific metadata layout in order to trigger the bug we are testing. At the
> +# moment the node size can not be smaller then the system's page size, so 
> given
> +# that the largest possible page size is 64Kb and by default the node size is
> +# set to the system's page size value, we explictly create a filesystem with 
> a
> +# 64Kb node size.
> +_scratch_mkfs -O no-holes -n $((64 * 1024)) >>$seqres.full 2>&1
> +_require_metadata_journaling $SCRATCH_DEV
> +_init_flakey
> +_mount_flakey
> +
> +# Create our test file with 831 extents of 256Kb each. Before each extent, 
> there

I think it's 832 extents in total? As the index starts with 0.

Otherwise looks good to me.

Thanks,
Eryu

> +# is a 256Kb hole (except for the first extent, which starts at offset 0). 
> This
> +# creates two leafs in the filesystem tree, with the extent at offset 
> 216530944
> +# being the last item in the first leaf and the extent at offset 217055232 
> being
> +# the first item in the second leaf.
> +for ((i = 0; i <= 831; i++)); do
> + offset=$((i * 2 * 256 * 1024))
> + $XFS_IO_PROG -f -c "pwrite -S 0xab -b 256K $offset 256K" \
> + $SCRATCH_MNT/foobar >/dev/null
> +done
> +
> +# Now persist everything done so far.
> +sync
> +
> +# Now punch a hole that covers part of the extent at offset 216530944.
> +$XFS_IO_PROG -c "fpunch $((216530944 + 128 * 1024 - 4000)) 256K" \
> +  -c "fsync" \
> +  $SCRATCH_MNT/foobar
> +
> +echo "File digest before power failure:"
> +md5sum $SCRATCH_MNT/foobar | _filter_scratch
> +
> +# Simulate a power failure and mount the filesystem to check that replaying 
> the
> +# fsync log/journal succeeds and our test file has the expected content.
> +_flakey_drop_and_remount
> +
> +echo "File digest after power failure and log replay:"
> +md5sum $SCRATCH_MNT/foobar | _filter_scratch
> +
> +_unmount_flakey
> +_cleanup_flakey
> +
> +status=0
> +exit
> diff --git a/tests/btrfs/159.out b/tests/btrfs/159.out
> new file mode 100644

Re: [PATCH 3/3] fstests: generic: Check the fs after each FUA writes

[Eryu Guan]

On Wed, Mar 21, 2018 at 02:22:29PM +0200, Amir Goldstein wrote:
> > +
> > +_log_writes_mount
> > +$FSSTRESS_PROG $fsstress_args > /dev/null 2>&1
> 
> You should run fsstress with run_check() so output will go to $seqres.full
> this way if you are able to catch a bug, you can take the random seed
> from fsstress output and repeat the same event sequence, which
> doesn't guaranty, but can increase the chances of reproducing the bug.

I suggested dropping run_check, as I don't think we care about the
fsstress return value here (and I always try to avoid new run_check
usage), but I agree that it might be useful to save the fsstress output
to $seqres.full.

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH RFC 3/3] fstests: generic: Check the fs after each FUA writes

[Eryu Guan]

On Wed, Mar 14, 2018 at 05:02:30PM +0800, Qu Wenruo wrote:
> Basic test case which triggers fsstress with dm-log-writes, and then
> check the fs after each FUA writes.
> With needed infrastructure and special handlers for journal based fs.
> 
> Signed-off-by: Qu Wenruo 
> ---
> In my test, xfs and btrfs survives while ext4 would report error during fsck.
> 
> My current biggest concern is, we abuse $TEST_DEV and mkfs on it all by
> ourselves. Not sure if it's allowed.
> ---
>  common/dmlogwrites| 119 
>  tests/generic/481 | 124 
> ++
>  tests/generic/481.out |   2 +
>  tests/generic/group   |   1 +
>  4 files changed, 246 insertions(+)
>  create mode 100755 tests/generic/481
>  create mode 100644 tests/generic/481.out
> 
> diff --git a/common/dmlogwrites b/common/dmlogwrites
> index 467b872e..258f5887 100644
> --- a/common/dmlogwrites
> +++ b/common/dmlogwrites
> @@ -126,3 +126,122 @@ _log_writes_cleanup()
>   $UDEV_SETTLE_PROG >/dev/null 2>&1
>   _log_writes_remove
>  }
> +
> +# Convert log writes mark to entry number
> +# Result entry number is output to stdout, could be empty if not found
> +_log_writes_mark_to_entry_number()
> +{
> + local _mark=$1
> + local ret
> +
> + [ -z "$_mark" ] && _fatal \
> + "mark must be given for _log_writes_mark_to_entry_number"
> +
> + ret=$($here/src/log-writes/replay-log --find --log $LOGWRITES_DEV \
> + --end-mark $_mark 2> /dev/null)
> + [ -z "$ret" ] && return
> + ret=$(echo "$ret" | cut -f1 -d\@)
> + echo "mark $_mark has entry number $ret" >> $seqres.full
> + echo "$ret"
> +}
> +
> +# Find next fua write entry number
> +# Result entry number is output to stdout, could be empty if not found
> +_log_writes_find_next_fua()
> +{
> + local _start_entry=$1
> + local ret
> +
> + [ -z "$_start_entry" ] && _start_entry=0
> + ret=$($here/src/log-writes/replay-log --find --log $LOGWRITES_DEV \
> +   --next-fua --start-entry $_start_entry 2> /dev/null)
> + [ -z "$ret" ] && return
> +
> + ret=$(echo "$ret" | cut -f1 -d\@)
> + echo "next fua is entry number $ret" >> $seqres.full
> + echo "$ret"
> +}
> +
> +# Replay log range to specified entry
> +# $1:End entry. The entry with this number *WILL* be replayed
> +# $2:Start entry. If not specified, start from the first entry.
> +# $3:Verbose. If set to 'y' do verbose output
> +_log_writes_replay_log_entry_range()

_log_writes_replay_log_range() would be fine.

> +{
> + local _end=$1
> + local _start=$2

This arguments order ($end comes first) makes me confused when I first
read the test code. Reverse the order?

> + local _verbose=$3
> +
> + [ -z "$_end" ] && _fatal \
> + "end entry must be specified for _log_writes_replay_log_entry_range"
> +
> + [ "x$verbose" == "xy" ] && _verbose="-v"
> + [ -z "$_start" ] && _start=0
> + [ "$_start" -gt "$_end" ] && _fatal \
> + "wrong parameter order for 
> _log_writes_replay_log_entry_range:start=$_start end=$_end"
> +
> + # To ensure we replay the last entry, for _start == 0 case,
> + # we need to manually increase the end entry number to ensure
> + # it's played
> + echo "=== replay from $_start to $_end ===" >> $seqres.full
> + $here/src/log-writes/replay-log --log $LOGWRITES_DEV \
> + --replay $SCRATCH_DEV --start-entry $_start \
> + --limit $(($_end - $_start + 1)) $_verbose \
> + >> $seqres.full 2>&1
> + [ $? -ne 0 ] && _fatal "replay failed"
> +}
> +
> +_log_writes_cleanup_snapshot()
> +{
> + $UDEV_SETTLE_PROG > /dev/null 2>&1
> + $DMSETUP_PROG remove "$DMLOGWRITES_SNAPSHOT_NAME" > /dev/null 2>&1
> + $DMSETUP_PROG remove "$DMLOGWRITES_ORIGIN_NAME" > /dev/null 2>&1
> +}
> +
> +# Helper to create snapshot of a the replayed device
> +# Useful for journal based filesystem such as XFS and Ext4 to replay
> +# their journal without touching the replay device, so that we can
> +# continue replaying other than replay from the beginning.
> +# $1:Snapshot device
> +_log_writes_create_snapshot()
> +{
> + _require_dm_target snapshot

This doesn't belong here, call it in the test.

> +
> + local snapshot_dev=$1
> + local cow_base=""

Unused variable.

> +
> + [ -z "$snapshot_dev" ] && _fatal \
> + "@device must be specified for _log_writes_create_snapshot"
> + local size=$(blockdev --getsz $SCRATCH_DEV)
> + [ -z "$size" ] && _fatal \
> + "failed to get device size for _log_writes_create_snapshot"
> +
> + _log_writes_cleanup_snapshot
> +
> + DMLOGWRITES_ORIGIN_NAME="log-writes-origin"
> + DMLOGWRITES_SNAPSHOT_NAME="log-writes-snapshot"
> + DMLOGWRITES_ORIGIN_TABLE="0 $size snapshot-origin $SCRATCH_DEV"
> + DMLOGWRITES_SNAPSHOT_TABLE="0 $size snapshot 
> 

Re: [PATCH RFC 3/3] fstests: generic: Check the fs after each FUA writes

[Eryu Guan]

On Fri, Mar 16, 2018 at 01:17:07PM +0800, Qu Wenruo wrote:
> 
> 
> On 2018年03月16日 12:01, Eryu Guan wrote:
> > On Wed, Mar 14, 2018 at 05:02:30PM +0800, Qu Wenruo wrote:
> >> Basic test case which triggers fsstress with dm-log-writes, and then
> >> check the fs after each FUA writes.
> >> With needed infrastructure and special handlers for journal based fs.
> > 
> > It's not clear to me why the existing infrastructure is not sufficient
> > for the test. It'd be great if you could provide more information and/or
> > background in commit log.
> 
> The main problem of current infrastructure is we don't have the
> following things:
> 
> 1) Way to take full advantage of dm-log-writes
>The main thing is, we don't have test cases to check each FUA (this
>patch) and flush (later patch after clearing all the RFC comments).
> 
>We have some dm-flakey test cases to emulate power loss, but they are
>mostly for fsync.
>Here we are not only testing fsync, but also every superblock update.
>Which should be a super set of dm-flakey tests.
> 
> 2) Workaround for journal replay
>In fact, if we only test btrfs, we don't even need such complicated
>work, just 'replay-log --fsck "btrfs check" --check fua' will be
>enough. As btrfs check doesn't report dirty journal (log tree) as
>problem.
>But for journal based fses, their fsck all report dirty journal as
>error, which needs current snapshot works to replay them before
>running fsck.

And replay-to-fua doesn't guarantee a consistent filesystem state,
that's why we need to mount/umount the target device to replay the
filesystem journal, and to avoid replaying already-replayed-log over and
over again, we create a snapshot of the target device and mount cycle &
fsck the snapshot, right?

I'm wondering if the overhead of repeatly create & destroy snapshots is
larger than replaying log from start. Maybe snapshots take more time?

> 
> I would add them in next version if there is no extra comment on this.
> 
> > 
> >>
> >> Signed-off-by: Qu Wenruo <w...@suse.com>
> >> ---
> >> In my test, xfs and btrfs survives while ext4 would report error during 
> >> fsck.
> >>
> >> My current biggest concern is, we abuse $TEST_DEV and mkfs on it all by
> >> ourselves. Not sure if it's allowed.
> > 
> > As Amir already replied, that's not allowed, any destructive operations
> > should be done on $SCRATCH_DEV.
> 
> Yep, I'm looking for similar case who uses $SCRATCH_DEV as LVM pv do get
> extra device.
> 
> Or can we reuse the scratch_dev_pool even for ext4/xfs?

I think so, IMO pool devices are not limited to btrfs. But I think we
could use a loop device reside on $TEST_DIR? Or if snapshots take longer
time, then we don't need this extra device at all :)

I have some other comments, will reply to the RFC patch in another
email.

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH RFC 3/3] fstests: generic: Check the fs after each FUA writes

[Eryu Guan]

On Wed, Mar 14, 2018 at 05:02:30PM +0800, Qu Wenruo wrote:
> Basic test case which triggers fsstress with dm-log-writes, and then
> check the fs after each FUA writes.
> With needed infrastructure and special handlers for journal based fs.

It's not clear to me why the existing infrastructure is not sufficient
for the test. It'd be great if you could provide more information and/or
background in commit log.

> 
> Signed-off-by: Qu Wenruo 
> ---
> In my test, xfs and btrfs survives while ext4 would report error during fsck.
> 
> My current biggest concern is, we abuse $TEST_DEV and mkfs on it all by
> ourselves. Not sure if it's allowed.

As Amir already replied, that's not allowed, any destructive operations
should be done on $SCRATCH_DEV.

> ---
>  common/dmlogwrites| 119 
>  tests/generic/481 | 124 
> ++
>  tests/generic/481.out |   2 +
>  tests/generic/group   |   1 +
>  4 files changed, 246 insertions(+)
>  create mode 100755 tests/generic/481
>  create mode 100644 tests/generic/481.out
> 
> diff --git a/common/dmlogwrites b/common/dmlogwrites
> index 467b872e..258f5887 100644
> --- a/common/dmlogwrites
> +++ b/common/dmlogwrites
> @@ -126,3 +126,122 @@ _log_writes_cleanup()
>   $UDEV_SETTLE_PROG >/dev/null 2>&1
>   _log_writes_remove
>  }
> +
> +# Convert log writes mark to entry number
> +# Result entry number is output to stdout, could be empty if not found
> +_log_writes_mark_to_entry_number()
> +{
> + local _mark=$1

No need to name a local variable with leading underscore.

> + local ret
> +
> + [ -z "$_mark" ] && _fatal \
> + "mark must be given for _log_writes_mark_to_entry_number"

Please use _fail instead of _fatal (and all the other places in this
patch). I think _fatal is only used at the initializing phase of the
test harness (e.g. check required tools, env etc.) and abort the test
run in early stage. _fail is used to indicate a failure in test phase,
it also logs the error messages in $seqres.full.

> +
> + ret=$($here/src/log-writes/replay-log --find --log $LOGWRITES_DEV \
> + --end-mark $_mark 2> /dev/null)

Is this output long or short? If it's a short one, e.g. a number or a
simple string, I think it's OK to save it in a variable. But if it's
long/complex enough and/or has multiple lines, I think we'd better to
save it with a tmp file, e.g. $tmp..

> + [ -z "$ret" ] && return
> + ret=$(echo "$ret" | cut -f1 -d\@)

Because I want to avoid echoing such a variable to a pipe (even it's
quoted), it used to cause subtle problems if not quoted properly.

Also, it'd be better to give a sample output of the replay-log command
in comment if it's short enough, so people could have an idea what it
looks like and know what you're looking for in the subsequent steps
(cut -f1 -d\@, in this example), and it's easier to review.

Above comments apply to other helper functions too.

Thanks for all the work!

Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] fstests: test regression of -EEXIST on creating new file after log replay

[Eryu Guan]

On Sat, Mar 10, 2018 at 04:56:04PM -0700, Liu Bo wrote:
> The regression is introduced to btrfs in linux v4.4 and it refuses to create
> new files after log replay by returning -EEXIST.
> 
> Although the problem is on btrfs only, there is no btrfs stuff in terms of
> test, so this makes it generic.
> 
> The kernel fix is
>   Btrfs: fix unexpected -EEXIST when creating new inode
> 
> Reviewed-by: Filipe Manana 
> Signed-off-by: Liu Bo 
> ---
> v2: - Remove failed message from 481.out
> - Drop the unnecessary write in creating a file
> 
>  tests/generic/481 | 75 
> +++
>  tests/generic/481.out |  2 ++
>  tests/generic/group   |  1 +
>  3 files changed, 78 insertions(+)
>  create mode 100755 tests/generic/481
>  create mode 100644 tests/generic/481.out
> 
> diff --git a/tests/generic/481 b/tests/generic/481
> new file mode 100755
> index 000..6a7e9dd
> --- /dev/null
> +++ b/tests/generic/481
> @@ -0,0 +1,75 @@
> +#! /bin/bash
> +# FSQA Test No. 481
> +#
> +# Reproduce a regression of btrfs that leads to -EEXIST on creating new files
> +# after log replay.
> +#
> +# The kernel fix is
> +#   Btrfs: fix unexpected -EEXIST when creating new inode
> +#
> +#---
> +#
> +# Copyright (C) 2018 Oracle. All Rights Reserved.
> +# Author: Bo Liu 
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + _cleanup_flakey
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/dmflakey
> +
> +# real QA test starts here
> +_supported_fs generic
> +_supported_os Linux
> +_require_scratch
> +_require_dm_target flakey
> +
> +rm -f $seqres.full
> +
> +_scratch_mkfs >>$seqres.full 2>&1

I added a "_require_metadata_journaling $SCRATCH_DEV" here, otherwise
test fails with ext2, which doesn't have journal.

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] fstests: btrfs/146: make sure hit all stripes in the case of compression

[Eryu Guan]

On Thu, Mar 08, 2018 at 01:56:45PM +0800, Lu Fengqi wrote:
> In the case of compression, each 128K input data chunk will be compressed
> to 4K (because of the characters written are duplicate). Therefore we have
> to write (128K * 16) to make sure every stripe can be hit.
> 
> Signed-off-by: Lu Fengqi 
> ---
> 
> V2: Modify the regular expression to ensure that it matches various
> compress mount options.
> 
>  tests/btrfs/146 | 11 +--
>  1 file changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/btrfs/146 b/tests/btrfs/146
> index 7071c128ca0a..a51eda68eaf3 100755
> --- a/tests/btrfs/146
> +++ b/tests/btrfs/146
> @@ -74,9 +74,16 @@ _scratch_pool_mkfs "-d raid0 -m raid1" > $seqres.full 2>&1
>  _scratch_mount
>  
>  # How much do we need to write? We need to hit all of the stripes. btrfs uses
> -# a fixed 64k stripesize, so write enough to hit each one
> +# a fixed 64k stripesize, so write enough to hit each one. In the case of
> +# compression, each 128K input data chunk will be compressed to 4K (because 
> of
> +# the characters written are duplicate). Therefore we have to write (128K * 
> 16)
> +# to make sure every stripe can be hit.
>  number_of_devices=`echo $SCRATCH_DEV_POOL | wc -w`
> -write_kb=$(($number_of_devices * 64))
> +if ! echo $MOUNT_OPTIONS | grep -qoP 'compress(-force)?(=(?!no)|,|$)'; then

I'm wondering if we could just write ($number_of_devices * 2048)K data
unconditionally, so we could get rid of this case switch and the fancy
perl style regexp?

Thanks,
Eryu

> + write_kb=$(($number_of_devices * 64))
> +else
> + write_kb=$(($number_of_devices * 2048))
> +fi
>  _require_fs_space $SCRATCH_MNT $write_kb
>  
>  testfile=$SCRATCH_MNT/fsync-err-test
> -- 
> 2.16.2
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] fstests: btrfs/004: increase the buffer size of logical-resolve to the maximum value 64K

[Eryu Guan]

On Tue, Mar 06, 2018 at 03:02:31PM +0800, Lu Fengqi wrote:
> Because of commit e76e13ce8c0b ("fsstress: implement the
> clonerange/deduperange ioctls"), dedupe makes the number of references to
> the same extent item increase so much that the default 4K buffer of
> logical-resolve is no longer sufficient.
> 
> Signed-off-by: Lu Fengqi 

This looks fine to me. But I'd like an explicit ack from btrfs
developers.

Thanks,
Eryu

> ---
>  tests/btrfs/004 | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/btrfs/004 b/tests/btrfs/004
> index de583cc355d4..0d2efb91dba7 100755
> --- a/tests/btrfs/004
> +++ b/tests/btrfs/004
> @@ -103,7 +103,7 @@ _btrfs_inspect_addr()
>   expect_addr=$3
>   expect_inum=$4
>   file=$5
> - cmd="$BTRFS_UTIL_PROG inspect-internal logical-resolve -P $addr $mp"
> + cmd="$BTRFS_UTIL_PROG inspect-internal logical-resolve -s 65536 -P 
> $addr $mp"
>   echo "# $cmd" >> $seqres.full
>   out=`$cmd`
>   echo "$out" >> $seqres.full
> -- 
> 2.16.2
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] fstests: common/rc: fix device still mounted error with SCRATCH_DEV_POOL

[Eryu Guan]

On Mon, Jan 15, 2018 at 11:10:20PM -0800, Liu Bo wrote:
> On Mon, Jan 15, 2018 at 02:22:28PM +0800, Eryu Guan wrote:
> > On Fri, Jan 12, 2018 at 06:04:59PM -0700, Liu Bo wrote:
> > > One of btrfs tests, btrfs/011, uses SCRATCH_DEV_POOL and puts a 
> > > non-SCRATCH_DEV
> > > device as the first one when doing mkfs, and this makes
> > > _require_scratch{_nocheck} fail to umount $SCRATCH_MNT since it checks 
> > > mount
> > > point with SCRATCH_DEV only, and for sure it finds nothing to umount and 
> > > the
> > > following tests complain about 'device still mounted' alike errors.
> > > 
> > > Introduce a helper to address this special case where both btrfs and 
> > > scratch
> > > dev pool are in use.
> > > 
> > > Signed-off-by: Liu Bo <bo.li@oracle.com>
> > 
> > Hmm, I didn't see this problem, I ran btrfs/011 then another tests that
> > uses $SCRATCH_DEV, and the second test ran fine too. Can you please
> > provide more details?
> 
> Sure, so I was using 4 devices of size being 2500M, btrfs/011 bailed
> out when doing a cp due to enospc then _fail is called to abort the
> test, and the mount point now is associated with a different device
> other than SCRATCH_DEV, so that _require_scratch_nocheck in btrfs/012
> was not able to umount SCRATCH_MNT.

Yeah, that's the exact case I described as below. I think adding
_scratch_umount >/dev/null 2>&1 in _cleanup() would resolve your issue.

> 
> > 
> > Anyway, I think we should fix btrfs/011 to either not use $SCRATCH_DEV
> > in replace operations (AFAIK, other btrfs replace tests do this) or
> > umount all devices before exit. And I noticed btrfs/011 does umount
> > $SCRATCH_MNT at the end of workout(), so usually all should be fine
> > (perhaps it would leave a device mounted if interrupted in the middle of
> > test run, because _cleanup() doesn't do umount).
> 
> That's true, if you want, I could fix all btrfs replace tests to
> umount SCRATCH_MNT right before exit.

I think only the tests that replace $SCRATCH_DEV (as what btrfs/011
does) need fixes, _require_scratch would umount $SCRATCH_MNT for other
tests.

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] fstests: common/rc: fix device still mounted error with SCRATCH_DEV_POOL

[Eryu Guan]

On Fri, Jan 12, 2018 at 06:04:59PM -0700, Liu Bo wrote:
> One of btrfs tests, btrfs/011, uses SCRATCH_DEV_POOL and puts a 
> non-SCRATCH_DEV
> device as the first one when doing mkfs, and this makes
> _require_scratch{_nocheck} fail to umount $SCRATCH_MNT since it checks mount
> point with SCRATCH_DEV only, and for sure it finds nothing to umount and the
> following tests complain about 'device still mounted' alike errors.
> 
> Introduce a helper to address this special case where both btrfs and scratch
> dev pool are in use.
> 
> Signed-off-by: Liu Bo 

Hmm, I didn't see this problem, I ran btrfs/011 then another tests that
uses $SCRATCH_DEV, and the second test ran fine too. Can you please
provide more details?

Anyway, I think we should fix btrfs/011 to either not use $SCRATCH_DEV
in replace operations (AFAIK, other btrfs replace tests do this) or
umount all devices before exit. And I noticed btrfs/011 does umount
$SCRATCH_MNT at the end of workout(), so usually all should be fine
(perhaps it would leave a device mounted if interrupted in the middle of
test run, because _cleanup() doesn't do umount).

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/5] fstests: filter: Introduce filter to filter out offset for xfs_io

[Eryu Guan]

On Thu, Jan 11, 2018 at 02:55:57PM +0800, Qu Wenruo wrote:
> Some test cases (AFAIK, btrfs RAID recovery test cases) read out certain
> location to verify its data.
> 
> Such read is mostly OK, but the golden output contains the on-disk
> offset, which can differ due to underlying chunk change.
> (This time is mkfs chunk layout change for btrfs)
> 
> So introduce macro _filter_xfs_io_offset to filter out the offset part
> wrote 65536/65536 bytes at offset 136708096
>
> And offset from "pread -v"
> 0826:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  
> ^
> 
> Reported-by: Nikolay Borisov 
> Signed-off-by: Qu Wenruo 
> ---
>  common/filter | 10 +-
>  1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/common/filter b/common/filter
> index 9c33efac..77afcbc4 100644
> --- a/common/filter
> +++ b/common/filter
> @@ -226,6 +226,15 @@ _filter_xfs_io()
>  sed -e "s/[0-9/.]* [GMKiBbytes]*, [0-9]* ops\; [0-9/:. sec]* 
> ([infa0-9/.]* [EPGMKiBbytes]*\/sec and [infa0-9/.]* ops\/sec)/XXX Bytes, X 
> ops\; XX:XX:XX.X (XXX YYY\/sec and XXX ops\/sec)/"
>  }
>  
> +# Also filter out the offset part of xfs_io output
> +# Some test cases may be affected by underlaying extent/chunk layout change,
> +# so wipe out this part to avoid golden output difference
> +_filter_xfs_io_offset()
> +{
> +# filter out " at offset XXX" and offset of "pread -v"
> +sed -e "s/ at offset [0-9]*$//" -e "s/^[0-9a-f]\+:/:/"
> +}
> +

I'd put _filter_xfs_io part of _filter_xfs_io_offset, so the callers
don't have to chain the filter themselves, e.g.

- $XFS_IO_PROG ... | _filter_xfs_io | _filter_xfs_io_offset
+ $XFS_IO_PROG ... | _filter_xfs_io_offset

And I perfer folding this 5-patch patchset into a single patch, so the
introduction and the usage of the new helper function together provide a
complete context in git log.

Thanks,
Eryu

>  # stderr filter for xfs_io to handle change of error output format (e.g.
>  # pwrite64 -> pwrite).
>  _filter_xfs_io_error()
> @@ -277,7 +286,6 @@ _filter_xfs_io_pages_modified()
>   _filter_xfs_io_units_modified "Page" $PAGE_SIZE
>  }
>  
> -
>  _filter_test_dir()
>  {
>   # TEST_DEV may be a prefix of TEST_DIR (e.g. /mnt, /mnt/ovl-mnt)
> -- 
> 2.15.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] generic/015: Change the test filesystem size to 101mb

[Eryu Guan]

On Mon, Jan 08, 2018 at 10:43:30AM +0200, Nikolay Borisov wrote:
> This test has been failing for btrfs for quite some time,
> at least since 4.7. There are 2 implementation details of btrfs that
> it exposes:
> 
> 1. Currently btrfs filesystem under 100mb are created in Mixed block
> group mode. Freespace accounting for it is not 100% accurate - I've

mkfs.btrfs does this too? Because I noticed _scratch_mkfs_sized adds
'--mixed' mkfs option explicitly.

> observed around 100-200kb discrepancy between a newly created filesystem,
> then writing a file and deleting it and checking the free space. This
> falls within %3 and not %1 as hardcoded in the test. 
> 
> 2. BTRFS won't flush it's delayed allocation on file deletion if less
> than 32mb are deleted. On such files we need to perform sync (missing
> in the test) or wait until time elapses for transaction commit.
> 
> Since mixed mode is somewhat deprecated and btrfs is not really intended
> to be used on really small devices let's just adjust the test to
> create a 101mb fs, which doesn't use mixed mode and really test
> freespace accounting.
> 
> Signed-off-by: Nikolay Borisov 
> ---
>  tests/generic/015 | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/generic/015 b/tests/generic/015
> index 78f2b13..416c4ae 100755
> --- a/tests/generic/015
> +++ b/tests/generic/015
> @@ -53,7 +53,7 @@ _supported_os Linux
>  _require_scratch
>  _require_no_large_scratch_dev
>  
> -_scratch_mkfs_sized `expr 50 \* 1024 \* 1024` >/dev/null 2>&1 \
> +_scratch_mkfs_sized `expr 101 \* 1024 \* 1024` >/dev/null 2>&1 \

Better to have some comments in the code too, to explain why we choose
101m filesystem size.

Thanks,
Eryu

>  || _fail "mkfs failed"
>  _scratch_mount || _fail "mount failed"
>  out=$SCRATCH_MNT/fillup.$$
> -- 
> 2.7.4
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] fstests: btrfs/158: reproduce a scrub bug on raid6 corruption

[Eryu Guan]

On Tue, Jan 02, 2018 at 01:35:00PM -0700, Liu Bo wrote:
> This is to reproduce a bug of scrub, with which scrub is unable to
> repair raid6 corruption as expected.
> 
> The kernel side fixes are
>   Btrfs: make raid6 rebuild retry more
>   Btrfs: fix scrub to repair raid6 corruption
> 
> Signed-off-by: Liu Bo 

Looks fine overall, I tested it with 4.15-rc6 kernel and test failed as
expected, re-tested successfully after applying the patches mentioned in
commit log.

Just some really minor issues below, and I can fix them on commit if
what I suggest looks sane to you.

> ---
>  tests/btrfs/158 | 114 
> 
>  tests/btrfs/158.out |  10 +
>  tests/btrfs/group   |   1 +
>  3 files changed, 125 insertions(+)
>  create mode 100755 tests/btrfs/158
>  create mode 100644 tests/btrfs/158.out
> 
> diff --git a/tests/btrfs/158 b/tests/btrfs/158
> new file mode 100755
> index 000..43afc2d
> --- /dev/null
> +++ b/tests/btrfs/158
> @@ -0,0 +1,114 @@
> +#! /bin/bash
> +# FS QA Test 158
> +#
> +# The test case is check if scrub is able fix raid6 data corruption,
> +# ie. if there is data corruption on two disks in the same horizontal
> +# stripe, e.g.  due to bitrot.
> +#
> +# The kernel fixes are
> +#Btrfs: make raid6 rebuild retry more
> +#Btrfs: fix scrub to repair raid6 corruption
> +#
> +#---
> +# Copyright (c) 2017 Oracle.  All Rights Reserved.
    2018?
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch_dev_pool 4
> +_require_btrfs_command inspect-internal dump-tree
> +
> +get_physical_stripe0()
> +{
> + $BTRFS_UTIL_PROG inspect-internal dump-tree -t 3 $SCRATCH_DEV | \
> + grep " DATA\|RAID6" -A 10 | \
> + $AWK_PROG '($1 ~ /stripe/ && $3 ~ /devid/ && $2 ~ /0/) { print $6 }'
> +}
> +
> +get_physical_stripe1()
> +{
> + $BTRFS_UTIL_PROG inspect-internal dump-tree -t 3 $SCRATCH_DEV | \
> + grep " DATA\|RAID6" -A 10 | \
> + $AWK_PROG '($1 ~ /stripe/ && $3 ~ /devid/ && $2 ~ /1/) { print $6 }'
> +}
> +
> +_scratch_dev_pool_get 4
> +# step 1: create a raid6 btrfs and create a 4K file
> +echo "step 1..mkfs.btrfs" >>$seqres.full
> +
> +mkfs_opts="-d raid6 -b 1G"
> +_scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1
> +
> +# -o nospace_cache makes sure data is written to the start position of the 
> data
> +# chunk
> +_scratch_mount -o nospace_cache
> +
> +# [0,64K) is written to stripe 0 and [64K, 128K) is written to stripe 1
> +$XFS_IO_PROG -f -d -c "pwrite -S 0xaa 0 128K" -c "fsync" \
> + "$SCRATCH_MNT/foobar" | _filter_xfs_io
> +
> +_scratch_unmount
> +
> +stripe_0=`get_physical_stripe0`
> +stripe_1=`get_physical_stripe1`
> +dev4=`echo $SCRATCH_DEV_POOL | awk '{print $4}'`
> +dev3=`echo $SCRATCH_DEV_POOL | awk '{print $3}'`
> +
> +# step 2: corrupt the 1st and 2nd stripe (stripe 0 and 1)
> +echo "step 2..simulate bitrot at offset $stripe_0 of device_4($dev4) and 
> offset $stripe_1 of device_3($dev3)" >>$seqres.full
> +
> +$XFS_IO_PROG -f -d -c "pwrite -S 0xbb $stripe_0 64K" $dev4 | _filter_xfs_io
> +$XFS_IO_PROG -f -d -c "pwrite -S 0xbb $stripe_1 64K" $dev3 | _filter_xfs_io
> +
> +# step 3: read foobar to repair the bitrot

Comment meant to be "scrub to repair the bitrot"?

> +echo "step 3..repair the bitrot" >> $seqres.full
> +_scratch_mount -o nospace_cache
> +
> +btrfs scrub start -B $SCRATCH_MNT >> $seqres.full 2>&1

$BTRFS_UTIL_PROG ...

> +
> +od -x $SCRATCH_MNT/foobar
> +
> +_scratch_dev_pool_put
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/158.out b/tests/btrfs/158.out
> new file mode 100644
> index 000..1f5ad3f
> --- /dev/null

Re: [PATCH] fstests: btrfs: Add test case to check if btrfs can handle full fs trim correctly

[Eryu Guan]

On Thu, Dec 07, 2017 at 08:43:43AM +0800, Qu Wenruo wrote:
> Ping.
> 
> Any comment on this?

It's been pushed out to upstream, see commit 88231c0c0b9d

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] fstests: btrfs: reproduce a read failure on raid6 setup

[Eryu Guan]

On Mon, Dec 04, 2017 at 03:33:23PM -0700, Liu Bo wrote:
> This test case is to reproduce a bug of raid6 reconstruction process.
> 
> The kernel fix are
> Btrfs: do not merge rbios if their fail stripe index are not identical
> Btrfs: make raid6 rebuild retry more
> 
> Signed-off-by: Liu Bo 

Test failed as expected with v4.15-rc2 kernel and passed with above
patches applied.

I fixed two trailing whitespace issues and some of the indentions to
avoid too-long line, also replaced 'repair' group with a new 'raid'
group. I think 'repair' group is meant for testing filesystem repair
tools, not the internal raid repair mechanism, so a new 'raid' group
looks better to me, as we're having more raid-specific tests recently.

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs/057: Fix test case to work on 64K page size

[Eryu Guan]

On Wed, Dec 06, 2017 at 12:15:45PM +0530, Harish wrote:
> On platforms with a page size greater than 4Kb, at the moment btrfs
> doesn't support a node/leaf size smaller than the page size, but it
> supports a larger one. So use the max supported node size (64Kb) so
> that the test runs on any platform currently supported by Linux.
> 
> Signed-off-by: Harish 

Looks fine to me. Also cc btrfs list for review.

Thanks,
Eryu

> ---
>  tests/btrfs/057 | 7 +--
>  tests/btrfs/057.out | 4 ++--
>  2 files changed, 7 insertions(+), 4 deletions(-)
> 
> diff --git a/tests/btrfs/057 b/tests/btrfs/057
> index 8d35579..06b2be9 100755
> --- a/tests/btrfs/057
> +++ b/tests/btrfs/057
> @@ -48,8 +48,11 @@ _require_scratch
>  
>  rm -f $seqres.full
>  
> -# use small leaf size to get higher btree height.
> -run_check _scratch_mkfs "-b 1g --nodesize 4096"
> +# Currently in btrfs the node/leaf size can not be smaller than the page
> +# size (but it can be greater than the page size). So use the largest
> +# supported node/leaf size (64Kb) so that the test can run on any platform
> +# that Linux supports.
> +run_check _scratch_mkfs "-b 1g --nodesize 65536"
>  
>  # inode cache is saved in the FS tree itself for every
>  # individual FS tree,that affects the sizes reported by qgroup show
> diff --git a/tests/btrfs/057.out b/tests/btrfs/057.out
> index 60cb92d..f2d9e9c 100644
> --- a/tests/btrfs/057.out
> +++ b/tests/btrfs/057.out
> @@ -1,3 +1,3 @@
>  QA output created by 057
> -4096 4096
> -4096 4096
> +65536 65536
> +65536 65536
> -- 
> 2.7.4
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs/124: add balance --full-balance option

[Eryu Guan]

On Tue, Dec 05, 2017 at 04:30:33PM +0800, Qu Wenruo wrote:
> 
> 
> On 2017年12月05日 16:26, Anand Jain wrote:
> > btrfs balance needs --full-balance option since 4.6, so check the
> > version and then use it.
> > 
> > As this may be useful for other btrfs tests as well, so this patch
> > adds _btrfs_full_balance_option() to the common/btrfs file.
> > 
> > Signed-off-by: Anand Jain 
> > ---
> >  common/btrfs| 13 +
> >  tests/btrfs/124 |  3 ++-
> >  2 files changed, 15 insertions(+), 1 deletion(-)
> > 
> > diff --git a/common/btrfs b/common/btrfs
> > index c09206c6f292..8ca6486b26dc 100644
> > --- a/common/btrfs
> > +++ b/common/btrfs
> > @@ -356,3 +356,16 @@ _btrfs_compression_algos()
> > echo "${feature#/sys/fs/btrfs/features/compress_}"
> > done
> >  }
> > +
> > +#btrfs-progs adds required --full-balance option since v4.6 so check for
> > +#that and then return the required option.
> > +_btrfs_full_balance_option()
> > +{
> > +   ver=$($BTRFS_UTIL_PROG --version|awk '{print $2}'|cut -d"-" -f1 | cut 
> > -d"v" -f2)
> 
> I'm never a fan of version check.
> Backporting or modified package version can easily screw this up.

Agreed, avoid version checking if possible.

> 
> Why not directly checking the help message of "btrfs balance start"?
> 
> Thanks,
> Qu
> 
> > +   tup1=$(echo $ver| cut -d"." -f1)
> > +   tup2=$(echo $ver| cut -d"." -f2)

And please declare local variables as 'local'.

> > +
> > +   RET=""
> > +   (("$tup1" > "4")) && RET="--full-balance"
> > +   (("$tup1" == "4")) && (("tup2" >= "6")) && RET="--full-balance"
> > +}
> > diff --git a/tests/btrfs/124 b/tests/btrfs/124
> > index a6486270a972..ec1d24b5ef42 100755
> > --- a/tests/btrfs/124
> > +++ b/tests/btrfs/124
> > @@ -129,7 +129,8 @@ _run_btrfs_util_prog device scan
> >  _scratch_mount >> $seqres.full 2>&1
> >  _run_btrfs_util_prog filesystem show
> >  echo >> $seqres.full
> > -_run_btrfs_util_prog balance start ${SCRATCH_MNT}
> > +_btrfs_full_balance_option
> > +_run_btrfs_util_prog balance start $RET ${SCRATCH_MNT}

This usage depends on the implicit global variable RET, which seems a
bit hard to follow. How about let the new helper return the expected
option string, and use that string in test, either assign it to a local
var or use it in command line directly.

Think about it more, perhaps we can introduce a new helper, e.g.
_run_btrfs_balance, and hide all these details, just call balance with
--full-balance when possible?

Thanks,
Eryu

> >  
> >  checkpoint2=`md5sum $SCRATCH_MNT/tf2`
> >  echo $checkpoint2 >> $seqres.full 2>&1
> > 
> 



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] btrfs/154: test for device dynamic rescan

[Eryu Guan]

On Wed, Nov 15, 2017 at 11:05:15AM +0800, Anand Jain wrote:
> Make sure missing device is included in the alloc list when it is
> scanned on a mounted FS.
> 
> This test case needs btrfs kernel patch which is in the ML
>   [PATCH] btrfs: handle dynamically reappearing missing device
> Without the kernel patch, the test will run, but reports as
> failed, as the device scanned won't appear in the alloc_list.
> 
> Signed-off-by: Anand Jain 
> ---
> v2: Fixed review comments.
>  tests/btrfs/154 | 186 
> 
>  tests/btrfs/154.out |  10 +++
>  tests/btrfs/group   |   1 +
>  3 files changed, 197 insertions(+)
>  create mode 100755 tests/btrfs/154
>  create mode 100644 tests/btrfs/154.out
> 
> diff --git a/tests/btrfs/154 b/tests/btrfs/154
> new file mode 100755
> index ..73a185157389
> --- /dev/null
> +++ b/tests/btrfs/154
> @@ -0,0 +1,186 @@
> +#! /bin/bash
> +# FS QA Test 154
> +#
> +# Test for reappearing missing device functionality.
> +#   This test will fail without the btrfs kernel patch
> +#   [PATCH] btrfs: handle dynamically reappearing missing device
> +#
> +#-
> +# Copyright (c) 2017 Oracle.  All Rights Reserved.
> +# Author: Anand Jain 
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#-
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/module
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch_dev_pool 2
> +_require_loadable_fs_module "btrfs"
> +
> +_scratch_dev_pool_get 2
> +
> +DEV1=`echo $SCRATCH_DEV_POOL | awk '{print $1}'`
> +DEV2=`echo $SCRATCH_DEV_POOL | awk '{print $2}'`
> +
> +echo DEV1=$DEV1 >> $seqres.full
> +echo DEV2=$DEV2 >> $seqres.full
> +
> +# Balance won't be successful if filled too much
> +DEV1_SZ=`blockdev --getsize64 $DEV1`
> +DEV2_SZ=`blockdev --getsize64 $DEV2`
> +
> +# get min
> +MAX_FS_SZ=`echo -e "$DEV1_SZ\n$DEV2_SZ" | sort | head -1`
> +# Need disks with more than 2G
> +if [ $MAX_FS_SZ -lt 20 ]; then
> + _scratch_dev_pool_put
> + _notrun "Smallest dev size $MAX_FS_SZ, Need at least 2G"
> +fi
> +
> +MAX_FS_SZ=1
> +bs="1M"
> +COUNT=$(($MAX_FS_SZ / 100))
> +CHECKPOINT1=0
> +CHECKPOINT2=0
> +
> +setup()
> +{
> + echo >> $seqres.full
> + echo "MAX_FS_SZ=$MAX_FS_SZ COUNT=$COUNT" >> $seqres.full
> + echo "setup"
> + echo "-setup-" >> $seqres.full
> + _scratch_pool_mkfs "-mraid1 -draid1" >> $seqres.full 2>&1
> + _scratch_mount >> $seqres.full 2>&1
> + dd if=/dev/urandom of="$SCRATCH_MNT"/tf bs=$bs count=1 \
> + >>$seqres.full 2>&1
> + _run_btrfs_util_prog filesystem show -m ${SCRATCH_MNT}
> + _run_btrfs_util_prog filesystem df $SCRATCH_MNT
> + COUNT=$(( $COUNT - 1 ))
> + echo "unmount" >> $seqres.full
> + _scratch_unmount
> +}
> +
> +degrade_mount_write()
> +{
> + echo >> $seqres.full
> + echo "--degraded mount: max_fs_sz $max_fs_sz bytes--" >> $seqres.full
> + echo
> + echo "degraded mount"
> +
> + echo "clean btrfs ko" >> $seqres.full
> + # un-scan the btrfs devices
> + _reload_fs_module "btrfs"
> + _mount -o degraded $DEV1 $SCRATCH_MNT >>$seqres.full 2>&1
> + cnt=$(( $COUNT/10 ))
> + dd if=/dev/urandom of="$SCRATCH_MNT"/tf1 bs=$bs count=$cnt \
> + >>$seqres.full 2>&1
> + COUNT=$(( $COUNT - $cnt ))
> + _run_btrfs_util_prog filesystem show -m $SCRATCH_MNT
> + _run_btrfs_util_prog filesystem df $SCRATCH_MNT
> + CHECKPOINT1=`md5sum $SCRATCH_MNT/tf1`
> + echo $SCRATCH_MNT/tf1:$CHECKPOINT1 >> $seqres.full
> +}
> +
> +scan_missing_dev_and_write()
> +{
> + echo >> $seqres.full
> + echo "--scan missing $DEV2--" >> $seqres.full
> + echo
> + echo "scan missing 

Re: [PATCH] btrfs/154: test for device dynamic rescan

[Eryu Guan]

On Mon, Nov 13, 2017 at 10:25:41AM +0800, Anand Jain wrote:
> Make sure missing device is included in the alloc list when it is
> scanned on a mounted FS.
> 
> This test case needs btrfs kernel patch which is in the ML
>   [PATCH] btrfs: handle dynamically reappearing missing device
> Without the kernel patch, the test will run, but reports as
> failed, as the device scanned won't appear in the alloc_list.
> 
> Signed-off-by: Anand Jain 

Tested without the fix and test failed as expected, test passed after
applying the fix.

Some minor nits below.

> ---
>  tests/btrfs/154 | 188 
> 
>  tests/btrfs/154.out |  10 +++
>  tests/btrfs/group   |   1 +
>  3 files changed, 199 insertions(+)
>  create mode 100755 tests/btrfs/154
>  create mode 100644 tests/btrfs/154.out
> 
> diff --git a/tests/btrfs/154 b/tests/btrfs/154
> new file mode 100755
> index ..8b06fc4d9347
> --- /dev/null
> +++ b/tests/btrfs/154
> @@ -0,0 +1,188 @@
> +#! /bin/bash
> +# FS QA Test 154
> +#
> +# Test for reappearing missing device functionality.
> +#   This test will fail without the btrfs kernel patch
> +#   [PATCH] btrfs: handle dynamically reappearing missing device
> +#
> +#-
> +# Copyright (c) 2017 Oracle.  All Rights Reserved.
> +# Author: Anand Jain 
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#-
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/module
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch_dev_pool 2
> +_test_unmount

This is not needed now, _require_loadable_fs_module will umount & mount
test dev as necessary.

> +_require_loadable_fs_module "btrfs"
> +
> +_scratch_dev_pool_get 2
> +
> +DEV1=`echo $SCRATCH_DEV_POOL | awk '{print $1}'`
> +DEV2=`echo $SCRATCH_DEV_POOL | awk '{print $2}'`
> +
> +echo DEV1=$DEV1 >> $seqres.full
> +echo DEV2=$DEV2 >> $seqres.full
> +
> +# Balance won't be successful if filled too much
> +DEV1_SZ=`blockdev --getsize64 $DEV1`
> +DEV2_SZ=`blockdev --getsize64 $DEV2`
> +
> +# get min
> +MAX_FS_SZ=`echo -e "$DEV1_SZ\n$DEV2_SZ" | sort | head -1`
> +# Need disks with more than 2G
> +if [ $MAX_FS_SZ -lt 20 ]; then
> + _scratch_dev_pool_put
> + _test_mount

Then no need to _test_mount.

> + _notrun "Smallest dev size $MAX_FS_SZ, Need at least 2G"
> +fi
> +
> +MAX_FS_SZ=1
> +bs="1M"
> +COUNT=$(($MAX_FS_SZ / 100))
> +CHECKPOINT1=0
> +CHECKPOINT2=0
> +
> +setup()
> +{
> + echo >> $seqres.full
> + echo "MAX_FS_SZ=$MAX_FS_SZ COUNT=$COUNT" >> $seqres.full
> + echo "setup"
> + echo "-setup-" >> $seqres.full
> + _scratch_pool_mkfs "-mraid1 -draid1" >> $seqres.full 2>&1
> + _scratch_mount >> $seqres.full 2>&1
> + dd if=/dev/urandom of="$SCRATCH_MNT"/tf bs=$bs count=1 \
> + >>$seqres.full 2>&1
> + _run_btrfs_util_prog filesystem show -m ${SCRATCH_MNT}
> + _run_btrfs_util_prog filesystem df $SCRATCH_MNT
> + COUNT=$(( $COUNT - 1 ))
> + echo "unmount" >> $seqres.full
> + _scratch_unmount
> +}
> +
> +degrade_mount_write()
> +{
> + echo >> $seqres.full
> + echo "--degraded mount: max_fs_sz $max_fs_sz bytes--" >> $seqres.full
> + echo
> + echo "degraded mount"
> +
> + echo "clean btrfs ko" >> $seqres.full
> + # un-scan the btrfs devices
> + _reload_fs_module "btrfs"
> + _mount -o degraded $DEV1 $SCRATCH_MNT >>$seqres.full 2>&1
> + cnt=$(( $COUNT/10 ))
> + dd if=/dev/urandom of="$SCRATCH_MNT"/tf1 bs=$bs count=$cnt \
> + >>$seqres.full 2>&1
> + COUNT=$(( $COUNT - $cnt ))
> + _run_btrfs_util_prog filesystem show -m $SCRATCH_MNT
> + _run_btrfs_util_prog filesystem df $SCRATCH_MNT
> + 

Re: [PATCH] fstests: btrfs: add test case for raid6 reconstruction bug

[Eryu Guan]

On Wed, Nov 01, 2017 at 06:01:23PM -0600, Liu Bo wrote:
> This is to reproduce a raid6 reconstruction bug after two drives
> getting offline and online via hotplug.
> 
> Signed-off-by: James Alandt 
> Signed-off-by: Liu Bo 

I don't have 5 deletable pool devices to actually test this patch, but
just comment issues I can see from the code.

> ---
>  tests/btrfs/152   | 121 
> ++
>  tests/btrfs/group |   1 +
>  2 files changed, 122 insertions(+)
>  create mode 100755 tests/btrfs/152
> 
> diff --git a/tests/btrfs/152 b/tests/btrfs/152
> new file mode 100755
> index 000..2b3d98c
> --- /dev/null
> +++ b/tests/btrfs/152
> @@ -0,0 +1,121 @@
> +#! /bin/bash
> +# FS QA Test 152
> +#
> +# This test is to reproduce a bug of btrfs raid6 reconstruction.

Better to provide more information about the bug and how would you like
to test it, it helps others to understand the bug and test, also it will
be helpful to debug test failure if it fails, say 5 years later.

> +#
> +#---
> +# Copyright (c) 2017 Oracle.  All Rights Reserved.
> +# Author: Liu Bo 
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> + if [ $removed == 1 ]; then
 $removed -eq 1 looks better

> + _scratch_umount
> + _devmgt_add ${disk1_remove}
> + _devmgt_add ${disk2_remove}
> + fi

Better to initialize all the variables used here before _cleanup.

> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch
> +_require_scratch_nocheck

Above two _requires are not needed.

> +_require_scratch_dev_pool 5

Because you have this one.

> +_require_deletable_scratch_dev_pool
> +
> +# umount test so that we can unload btrfs module
> +_test_unmount
> +_require_btrfs_loadable

Perhaps you can take use of the new helpers introduced by Darrick's
pending patch "fstests: add module reloading helpers" (you can find the
patch in list), and with new helpers you don't have to umount test dev
before "_require_loadable_fs_module btrfs", which will be the
replacement of _require_btrfs_loadable.

> +
> +_scratch_dev_pool_get 5
> +
> +devs=( $SCRATCH_DEV_POOL )
> +n=${#devs[@]}
> +
> +mkfs_opts="-draid6 -mraid6 -n64K -b 1G"
> +_scratch_pool_mkfs $mkfs_opts >> $seqres.full 2>&1
> +
> +_scratch_mount -o nospace_cache

Please comment any non-obvious & non-default mkfs/mount options like
-n64k, -o nospace_cache :)

> +
> +# 1st 4k write
> +$XFS_IO_PROG -f -c "pwrite -W 0 4K" $SCRATCH_MNT/f1 | _filter_xfs_io
> +sync

No need to sync with "-W" pwrite.

> +
> +# hotplug disk1
> +disk1=${devs[$((n-1))]}
> +disk1_short=`_short_dev ${disk1}`
> +disk1_remove=`readlink -f /sys/block/${disk1_short} | rev | cut -d "/" -f 3 
> | rev`

Factor this into a common function? I see similar code in btrfs/003 to
do the same task.

> +_devmgt_remove ${disk1_remove} $disk1_short
> +removed=1
> +
> +# 2nd 4K write
> +$XFS_IO_PROG -f -c "pwrite -W 0 4K" $SCRATCH_MNT/f2 | _filter_xfs_io
> +sync

Same here (and other places below), sync not needed.

> +
> +# hotplug disk2
> +disk2=${devs[$((n-2))]}
> +disk2_short=`_short_dev ${disk2}`
> +disk2_remove=`readlink -f /sys/block/${disk2_short} | rev | cut -d "/" -f 3 
> | rev`
> +_devmgt_remove ${disk2_remove} $disk2_short
> +removed=1
> +
> +# 3rd 4K write
> +$XFS_IO_PROG -f -c "pwrite -W 0 4K" $SCRATCH_MNT/f3 | _filter_xfs_io
> +sync
> +
> +_scratch_unmount
> +
> +# bring disk1 & disk2 online
> +_devmgt_add ${disk1_remove}
> +_devmgt_add ${disk2_remove}
> +removed=0
> +
> +# reload the module to simulate 'reboot'
> +_reload_btrfs_ko

With the new helper, this should be "_reload_fs_module btrfs"

> +
> +# have every disk available for 

Re: [PATCH] xfstests: Add message indicating btrfs-progs support FST in read-only mode

[Eryu Guan]

On Thu, Oct 26, 2017 at 07:16:02AM +, Gu, Jinxiang wrote:
> Hi,
> 
> > -Original Message-
> > From: Eryu Guan [mailto:eg...@redhat.com]
> > Sent: Thursday, October 26, 2017 2:52 PM
> > To: Gu, Jinxiang/顾 金香 <g...@cn.fujitsu.com>
> > Cc: fste...@vger.kernel.org; linux-btrfs@vger.kernel.org
> > Subject: Re: [PATCH] xfstests: Add message indicating btrfs-progs support 
> > FST in read-only mode
> > 
> > On Thu, Oct 26, 2017 at 01:57:46PM +0800, Gu Jinxiang wrote:
> > > From: Gu JinXiang <g...@cn.fujitsu.com>
> > >
> > > btrfs-progs now support FST in read-only mode, so when space_cache=v2
> > > enabled, this test case will fail.
> > > Add message to help user to understand this status.
> > 
> > Sorry, I don't quite understand the new 'FST' feature. But is it a bug that 
> > we want to fix when mounting with space_cache=v2
> > option, or we just couldn't do btrfs-convert in this case? If it's a real 
> > bug, I'd say let the test fail as it is, and
> > track bug in tools like bugzilla not comments/messages in the test; if it's 
> > the latter case, then just _notrun the test
> > if space_cache=v2 option is specified, e.g.
> > 
> > _exclude_scratch_mount_option "space_cache=v2"
> Thank you for your suggestion.
> As an known behavior when use btrfs-convert when space_cache=v2 enabled, I 
> prefer to modify this case not to be run.
> I all send a new patch for this.

But from what Qu said in his reply, it looks like this is a real bug in
btrfs-convert, so I don't think _notrun is a good idea. I'd leave the
test as it is, because that's a bug and this test finds it.

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] xfstests: Add message indicating btrfs-progs support FST in read-only mode

[Eryu Guan]

On Thu, Oct 26, 2017 at 01:57:46PM +0800, Gu Jinxiang wrote:
> From: Gu JinXiang 
> 
> btrfs-progs now support FST in read-only mode, so when space_cache=v2
> enabled, this test case will fail.
> Add message to help user to understand this status.

Sorry, I don't quite understand the new 'FST' feature. But is it a bug
that we want to fix when mounting with space_cache=v2 option, or we just
couldn't do btrfs-convert in this case? If it's a real bug, I'd say let
the test fail as it is, and track bug in tools like bugzilla not
comments/messages in the test; if it's the latter case, then just
_notrun the test if space_cache=v2 option is specified, e.g.

_exclude_scratch_mount_option "space_cache=v2"

Thanks,
Eryu

> 
> Signed-off-by: Gu JinXiang 
> ---
>  tests/btrfs/012 | 6 ++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/tests/btrfs/012 b/tests/btrfs/012
> index 85c82f07..529e6eca 100755
> --- a/tests/btrfs/012
> +++ b/tests/btrfs/012
> @@ -96,6 +96,12 @@ cp -aR /lib/modules/`uname -r`/ $SCRATCH_MNT/new
>  
>  _scratch_unmount
>  
> +space_cache_version=$(echo "$MOUNT_OPTIONS" | grep "space_cache=v2")
> +if [ -n "$space_cache_version" ]; then
> + _fail "since used space_cache=v2 when mount," \
> +"and for FST btrfs-progs support is read-only."\
> +   "so btrfs-convert rollback will fail"
> +fi
>  # Now restore the ext4 device
>  $BTRFS_CONVERT_PROG -r $SCRATCH_DEV >> $seqres.full 2>&1 || \
>   _fail "btrfs-convert rollback failed"
> -- 
> 2.13.5
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] Fstest: btrfs/151: test if device delete ends up with losing raid profile

[Eryu Guan]

On Fri, Oct 13, 2017 at 01:40:05PM -0600, Liu Bo wrote:
> Currently running 'btrfs device delete' can end up with losing data
> raid profile (if any), this test is to reproduce the problem.
> 
> The fix is
>  "Btrfs: avoid losing data raid profile when deleting a device"
> 
> Signed-off-by: Liu Bo 
> ---
>  tests/btrfs/151 | 73 
> +
>  tests/btrfs/151.out |  2 ++
>  tests/btrfs/group   |  1 +
>  3 files changed, 76 insertions(+)
>  create mode 100755 tests/btrfs/151
>  create mode 100644 tests/btrfs/151.out
> 
> diff --git a/tests/btrfs/151 b/tests/btrfs/151
> new file mode 100755
> index 000..1866cb6
> --- /dev/null
> +++ b/tests/btrfs/151
> @@ -0,0 +1,73 @@
> +#! /bin/bash
> +# FS QA Test 151
> +#
> +# Test if it's losing data chunk's raid profile after 'btrfs device
> +# remove'.
> +#
> +# The fix is
> +#Btrfs: avoid losing data raid profile when deleting a device
> +#
> +#---
> +# Copyright (c) 2017 Oracle.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch
> +_require_scratch_dev_pool 3
> +
> +# create raid1 for data
> +_scratch_pool_mkfs "-d raid1 -b 1G" >> $seqres.full 2>&1
> +
> +# we need an empty data chunk, so nospace_cache is required.
> +_scratch_mount -onospace_cache
> +
> +# if data chunk is empty, 'btrfs device remove' can change raid1 to
> +# single.
> +$BTRFS_UTIL_PROG device delete 2 $SCRATCH_MNT >> $seqres.full
> +
> +df_ret=`$BTRFS_UTIL_PROG filesystem df $SCRATCH_MNT`
> +echo $df_ret | $AWK_PROG -F ':' '/Data,/ {print $1}'

Forgot to commit the changes before sending? It looks like the same as v1.

Thanks,
Eryu

> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/151.out b/tests/btrfs/151.out
> new file mode 100644
> index 000..0a1de06
> --- /dev/null
> +++ b/tests/btrfs/151.out
> @@ -0,0 +1,2 @@
> +QA output created by 151
> +Data, RAID1
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index e73bb1b..a7ff7b0 100644
> --- a/tests/btrfs/group
> +++ b/tests/btrfs/group
> @@ -153,3 +153,4 @@
>  148 auto quick rw
>  149 auto quick send compress
>  150 auto quick dangerous
> +151 auto quick
> -- 
> 2.5.0
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Fstest: btrfs/151: test if device delete ends up with losing raid profile

[Eryu Guan]

On Mon, Oct 09, 2017 at 11:39:21AM -0600, Liu Bo wrote:
> Currently running 'btrfs device delete' can end up with losing data raid
> profile (if any), this test is to reproduce the problem.
> 
> The fix is
>  "Btrfs: avoid losing data raid profile when deleting a device"
> 
> Signed-off-by: Liu Bo 
> ---
>  tests/btrfs/151 | 73 
> +
>  tests/btrfs/151.out |  2 ++
>  tests/btrfs/group   |  1 +
>  3 files changed, 76 insertions(+)
>  create mode 100755 tests/btrfs/151
>  create mode 100644 tests/btrfs/151.out
> 
> diff --git a/tests/btrfs/151 b/tests/btrfs/151
> new file mode 100755
> index 000..1866cb6
> --- /dev/null
> +++ b/tests/btrfs/151
> @@ -0,0 +1,73 @@
> +#! /bin/bash
> +# FS QA Test 151
> +#
> +# Test if it's losing data chunk's raid profile after 'btrfs device
> +# remove'.
> +#
> +# The fix is
> +#Btrfs: avoid losing data raid profile when deleting a device
> +#
> +#---
> +# Copyright (c) 2017 Oracle.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch

Better to have _require_btrfs_dev_del_by_devid too.

> +_require_scratch_dev_pool 3

Hmm, this only reproduces for me if I have exact three devices in
SCRATCH_DEV_POOL, large dev pool passed the test. If this can only be
reproduced with a three-device setup, I think we need to limit the dev
pool size to at most 3, e.g.

# 
_scratch_dev_pool_get 3

_scratch_dev_pool_put
status=0
exit

> +
> +# create raid1 for data
> +_scratch_pool_mkfs "-d raid1 -b 1G" >> $seqres.full 2>&1
> +
> +# we need an empty data chunk, so nospace_cache is required.
> +_scratch_mount -onospace_cache
> +
> +# if data chunk is empty, 'btrfs device remove' can change raid1 to
> +# single.
> +$BTRFS_UTIL_PROG device delete 2 $SCRATCH_MNT >> $seqres.full
> +
> +df_ret=`$BTRFS_UTIL_PROG filesystem df $SCRATCH_MNT`
> +echo $df_ret | $AWK_PROG -F ':' '/Data,/ {print $1}'

I'd like to avoid saving outputs in a variable, it's easy to cause
subtle problems without proper quoting, as Dave found out in this thread

[whacky issue] xfs/277 endlessly looping in _require_xfs_io_command
http://www.spinics.net/lists/fstests/msg07501.html

How about:

# save btrfs filesystem df output for debug purpose
$BTRFS_UTIL_PROG filesystem df $SCRATCH_MNT >>$seqres.full 2>&1
$BTRFS_UTIL_PROG filesystem df $SCRATCH_MNT | \
$AWK_PROG -F ':' '/Data,/ {print $1}'

Thanks,
Eryu

> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/151.out b/tests/btrfs/151.out
> new file mode 100644
> index 000..0a1de06
> --- /dev/null
> +++ b/tests/btrfs/151.out
> @@ -0,0 +1,2 @@
> +QA output created by 151
> +Data, RAID1
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index e73bb1b..a7ff7b0 100644
> --- a/tests/btrfs/group
> +++ b/tests/btrfs/group
> @@ -153,3 +153,4 @@
>  148 auto quick rw
>  149 auto quick send compress
>  150 auto quick dangerous
> +151 auto quick
> -- 
> 2.5.0
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ANNOUNCE] fsperf: a simple fs/block performance testing framework

[Eryu Guan]

On Mon, Oct 09, 2017 at 04:17:31PM +1100, Dave Chinner wrote:
> On Sun, Oct 08, 2017 at 10:25:10PM -0400, Josef Bacik wrote:
> > On Mon, Oct 09, 2017 at 11:51:37AM +1100, Dave Chinner wrote:
> > > On Fri, Oct 06, 2017 at 05:09:57PM -0400, Josef Bacik wrote:
> > > > Hello,
> > > > 
> > > > One thing that comes up a lot every LSF is the fact that we have no 
> > > > general way
> > > > that we do performance testing.  Every fs developer has a set of 
> > > > scripts or
> > > > things that they run with varying degrees of consistency, but nothing 
> > > > central
> > > > that we all use.  I for one am getting tired of finding regressions 
> > > > when we are
> > > > deploying new kernels internally, so I wired this thing up to try and 
> > > > address
> > > > this need.
> > > > 
> > > > We all hate convoluted setups, the more brain power we have to put in 
> > > > to setting
> > > > something up the less likely we are to use it, so I took the xfstests 
> > > > approach
> > > > of making it relatively simple to get running and relatively easy to 
> > > > add new
> > > > tests.  For right now the only thing this framework does is run fio 
> > > > scripts.  I
> > > > chose fio because it already gathers loads of performance data about 
> > > > it's runs.
> > > > We have everything we need there, latency, bandwidth, cpu time, and all 
> > > > broken
> > > > down by reads, writes, and trims.  I figure most of us are familiar 
> > > > enough with
> > > > fio and how it works to make it relatively easy to add new tests to the
> > > > framework.
> > > > 
> > > > I've posted my code up on github, you can get it here
> > > > 
> > > > https://github.com/josefbacik/fsperf
> > > > 
> > > > All (well most) of the results from fio are stored in a local sqlite 
> > > > database.
> > > > Right now the comparison stuff is very crude, it simply checks against 
> > > > the
> > > > previous run and it only checks a few of the keys by default.  You can 
> > > > check
> > > > latency if you want, but while writing this stuff up it seemed that 
> > > > latency was
> > > > too variable from run to run to be useful in a "did my thing regress or 
> > > > improve"
> > > > sort of way.
> > > > 
> > > > The configuration is brain dead simple, the README has examples.  All 
> > > > you need
> > > > to do is make your local.cfg, run ./setup and then run ./fsperf and you 
> > > > are good
> > > > to go.
> > > 
> > > Why re-invent the test infrastructure? Why not just make it a
> > > tests/perf subdir in fstests?
> > > 
> > 
> > Probably should have led with that shouldn't I have?  There's nothing 
> > keeping me
> > from doing it, but I didn't want to try and shoehorn in a python thing into
> > fstests.  I need python to do the sqlite and the json parsing to dump into 
> > the
> > sqlite database.
> > 
> > Now if you (and others) are not opposed to this being dropped into 
> > tests/perf
> > then I'll work that up.  But it's definitely going to need to be done in 
> > python.
> > I know you yourself have said you aren't opposed to using python in the 
> > past, so
> > if that's still the case then I can definitely wire it all up.
> 
> I have no problems with people using python for stuff like this but,
> OTOH, I'm not the fstests maintainer anymore :P

I have no problem either if python is really needed, after all this is a
very useful infrastructure improvement. But the python version problem
brought up by Ted made me a bit nervous, we need to work that round
carefully.

OTOH, I'm just curious, what is the specific reason that python is a
hard requirement? If we can use perl, that'll be much easier for
fstests.

BTW, opinions from key fs developers/fstests users, like you, are also
very important and welcomed :)

Thanks,
Eryu

> 
> > > > The plan is to add lots of workloads as we discover regressions and 
> > > > such.  We
> > > > don't want anything that takes too long to run otherwise people won't 
> > > > run this,
> > > > so the existing tests don't take much longer than a few minutes each.  
> > > > I will be
> > > > adding some more comparison options so you can compare against averages 
> > > > of all
> > > > previous runs and such.
> > > 
> > > Yup, that fits exactly into what fstests is for... :P
> > > 
> > > Integrating into fstests means it will be immediately available to
> > > all fs developers, it'll run on everything that everyone already has
> > > setup for filesystem testing, and it will have familiar mkfs/mount
> > > option setup behaviour so there's no new hoops for everyone to jump
> > > through to run it...
> > > 
> > 
> > TBF I specifically made it as easy as possible because I know we all hate 
> > trying
> > to learn new shit.
> 
> Yeah, it's also hard to get people to change their workflows to add
> a whole new test harness into them. It's easy if it's just a new
> command to an existing workflow :P
> 
> > I figured this was different enough to warrant a separate
> > project, especially since I'm going to add block 

Re: [PATCH v2] fstests: btrfs/150 regression test for reading compressed data

[Eryu Guan]

On Tue, Sep 26, 2017 at 05:18:51PM -0700, Liu Bo wrote:
> On Tue, Sep 26, 2017 at 04:37:52PM -0700, Liu Bo wrote:
> > On Tue, Sep 26, 2017 at 05:02:36PM +0800, Eryu Guan wrote:
> > > On Fri, Sep 22, 2017 at 05:21:27PM -0600, Liu Bo wrote:
> > > > We had a bug in btrfs compression code which could end up with a
> > > > kernel panic.
> > > > 
> > > > This is adding a regression test for the bug and I've also sent a
> > > > kernel patch to fix the bug.
> > > > 
> > > > The patch is "Btrfs: fix kernel oops while reading compressed data".
> > > > 
> > > > Signed-off-by: Liu Bo <bo.li@oracle.com>
> > > 
> > > Hmm, I can't reproduce the panic with 4.13 kernel, which doesn't have
> > > the fix applied. Can you please help confirm if it panics on your test
> > > environment?
> > >
> > 
> > Yes, it is reproducible on my box, hrm...I'll be running it more times
> > to double check.
> > 
> 
> It worked for me...both v4.13 and v4.14.0-rc2 have the following
> messages[1].
> 
> This requires two config:
> CONFIG_FAULT_INJECTION=y
> CONFIG_FAULT_INJECTION_DEBUG_FS=y
> 
> Could you please check again?

I re-compiled 4.14-rc2 kernel on my test vm with FAIL_MAKE_REQUEST
enabled (which requires FAULT_INJECTION), and I can reproduce the crash
now. It was so weired that previously I did have FAIL_MAKE_REQUEST
enabled and test ran normally without hitting the bug, but now I can hit
the bug quite reliably. Not sure what was happning in my previous test..

Thanks for confirming!

Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] fstests: btrfs/150 regression test for reading compressed data

[Eryu Guan]

On Fri, Sep 22, 2017 at 05:21:27PM -0600, Liu Bo wrote:
> We had a bug in btrfs compression code which could end up with a
> kernel panic.
> 
> This is adding a regression test for the bug and I've also sent a
> kernel patch to fix the bug.
> 
> The patch is "Btrfs: fix kernel oops while reading compressed data".
> 
> Signed-off-by: Liu Bo 

Hmm, I can't reproduce the panic with 4.13 kernel, which doesn't have
the fix applied. Can you please help confirm if it panics on your test
environment?

> ---
> v2: - Fix ambiguous copyright.
> - Use /proc/$pid/make-it-fail to specify IO failure
> - Use bash -c to run test only when pid is odd.
> - Add test to dangerous group.
> 
>  tests/btrfs/150 | 103 
> 
>  tests/btrfs/150.out |   3 ++
>  tests/btrfs/group   |   1 +
>  3 files changed, 107 insertions(+)
>  create mode 100755 tests/btrfs/150
>  create mode 100644 tests/btrfs/150.out
> 
> diff --git a/tests/btrfs/150 b/tests/btrfs/150
> new file mode 100755
> index 000..8891c38
> --- /dev/null
> +++ b/tests/btrfs/150
> @@ -0,0 +1,103 @@
> +#! /bin/bash
> +# FS QA Test btrfs/150
> +#
> +# This is a regression test which ends up with a kernel oops in btrfs.
> +# It occurs when btrfs's read repair happens while reading a compressed
> +# extent.
> +# The patch to fix it is
> +#Btrfs: fix kernel oops while reading compressed data
> +#
> +#---
> +# Copyright (c) 2017 Oracle.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch
> +_require_fail_make_request
> +_require_scratch_dev_pool 2 

Trailing whitespace in above line.

> +
> +SYSFS_BDEV=`_sysfs_dev $SCRATCH_DEV`
> +enable_io_failure()
> +{
> +echo 100 > $DEBUGFS_MNT/fail_make_request/probability
> +echo 1000 > $DEBUGFS_MNT/fail_make_request/times
> +echo 0 > $DEBUGFS_MNT/fail_make_request/verbose
> +echo 1 > $SYSFS_BDEV/make-it-fail
> +}
> +
> +disable_io_failure()
> +{
> +echo 0 > $SYSFS_BDEV/make-it-fail
> +echo 0 > $DEBUGFS_MNT/fail_make_request/probability
> +echo 0 > $DEBUGFS_MNT/fail_make_request/times
> +}
> +
> +_scratch_pool_mkfs "-d raid1 -b 1G" >> $seqres.full 2>&1
> +
> +# It doesn't matter which compression algorithm we use.
> +_scratch_mount -ocompress
> +
> +# Create a file with all data being compressed
> +$XFS_IO_PROG -f -c "pwrite -W 0 8K" $SCRATCH_MNT/foobar | _filter_xfs_io
> +
> +# Raid1 consists of two copies and btrfs decides which copy to read by 
> reader's
> +# %pid.  Now we inject errors to copy #1 and copy #0 is good.  We want to 
> read
> +# the bad copy to trigger read-repair.
> +while [[ -z $result ]]; do
> + # invalidate the page cache
> + $XFS_IO_PROG -f -c "fadvise -d 0 8K" $SCRATCH_MNT/foobar

Does 'echo 3 > /proc/sys/vm/drop_caches' work?

Thanks,
Eryu

> +
> + enable_io_failure
> +
> + result=$(bash -c "
> + if [ \$((\$\$ % 2)) == 1 ]; then
> + echo 1 > /proc/\$\$/make-it-fail
> + exec $XFS_IO_PROG -c \"pread 0 8K\" \$SCRATCH_MNT/foobar
> + fi")
> +
> + disable_io_failure
> +done
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/150.out b/tests/btrfs/150.out
> new file mode 100644
> index 000..c492c24
> --- /dev/null
> +++ b/tests/btrfs/150.out
> @@ -0,0 +1,3 @@
> +QA output created by 150
> +wrote 8192/8192 bytes at offset 0
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index 70c3f05..e73bb1b 100644
> --- a/tests/btrfs/group
> +++ b/tests/btrfs/group
> @@ -152,3 +152,4 @@
>  147 auto quick send
>  148 

Re: [PATCH 1/2] xfstests: Split MOUNT_OPTIONS to TEST_FS_MOUNT_OPTS and MOUNT_OPTIONS

[Eryu Guan]

On Mon, Sep 18, 2017 at 04:11:09PM +0800, Gu Jinxiang wrote:
> Resovle the inconsistent of mount option.
> Btrfs use MOUNT_OPTIONS for both scrath_dev and test_dev. Change to
> MOUNT_OPTIONS for scratch mount, and TEST_FS_MOUNT_OPTS for test dev
> mount.
> 
> Signed-off-by: Gu Jinxiang 
> ---
> As mentioned by https://patchwork.kernel.org/patch/9742039/, the usage
> of MOUNT_OPTIONS is inconsistent.

Sorry for the late reply, because I don't think this is the right fix,
and I was trying to sort out & refactor the mount option settings
through fstests, but found that it required more time than I thought..

The new _check_test_btrfs_filesystem() you introduced here is a (almost)
complete copy of _check_btrfs_filesystem, the only difference is that
it's mounting $device with $TEST_FS_MOUNT_OPTS instead of
$MOUNT_OPTIONS. At least you could factor out a common helper so you
don't have to repeat 99% of the code.

But that only fixes _check_test_btrfs_filesystems, there're similar
problems in _check_generic_filesystems and other places that do mount
operation. I'd like to see a well-defined interface to return the correct
mount options to callers that want to do mount operations.

I'll look into this and see if I can work something out. I really
appreciate if anyone has other thoughts/suggestions!

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/2] Add new common filter function

[Eryu Guan]

On Fri, Sep 01, 2017 at 10:14:41AM +0300, Amir Goldstein wrote:
> On Fri, Sep 1, 2017 at 10:04 AM, Eryu Guan <eg...@redhat.com> wrote:
> > On Fri, Sep 01, 2017 at 02:39:44PM +0900, Misono, Tomohiro wrote:
> >> Several tests uses both _filter_test_dir and _filter_scratch
> >> concatenated by pipe to filter $TEST_DIR and $SCRATCH_MNT. However, this
> >> would fail if the shorter string is a substring of the other (like
> >> "/mnt" and "/mnt2").
> >>
> >> This patch introduces new common filter function to safely call both
> >> _filter_test_dir and _filter_scratch.
> >>
> >> I chedked this with btrfs/029, generic/409,410,411, and generic/381,383,
> >> xfs/106,108 (which calls _filter_quota). Thanks Eryu for advice.
> >>
> >> Signed-off-by: Tomohiro Misono <misono.tomoh...@jp.fujitsu.com>
> >
> > Thanks! Though I don't think we need two separate patches, so I merged
> > the two patches together at commit time.
> >
> 
> FYI, there is still a way for a creative user to mess this up:
> 
> TEST_DEV=/test
> TEST_DIR=/test/ovl-mnt
> SCRATCH_DEV=/ovl
> SCRATCH_MNT=/ovl/ovl-mnt
> 
> $SCRATCH_DEV is a substring of $TEST_DIR
> and _filter_scratch is run first.
> 
> It's not that crazy to get to this config with the 'new'
> -overlay command, e.g.:
> TEST_DEV=/dev/sda
> TEST_DIR=/test
> SCRATCH_DEV=/dev/sdb
> SCRATCH_MNT=/ovl
> 
> Will be auto converted to the values above.

Yeah, overlay makes it more complicated.

> 
> This patch didn't break this use case.

Good to know! I just tested this config with a quick overlay tests run,
it did pass.

Seems like we need some tests for fstests itself, like
./check selftest ? :)

Thanks,
Eryu

> 
> >> ---
> >>  common/filter | 11 +++
> >>  1 file changed, 11 insertions(+)
> >>
> >> diff --git a/common/filter b/common/filter
> >> index 1ef342b..75570f9 100644
> >> --- a/common/filter
> >> +++ b/common/filter
> >> @@ -295,6 +295,17 @@ _filter_scratch()
> >>   -e "/.use_space/d"
> >>  }
> >>
> >> +_filter_testdir_and_scratch()
> >> +{
> >> + # filter both $TEST_DIR and $SCRATCH_MNT, but always filter the 
> >> longer
> >> + # string first if the other string is a substring of the first one
> >> + if echo "$TEST_DIR" | grep -q "$SCRATCH_MNT"; then
> >> +   _filter_test_dir | _filter_scratch
> >> + else
> >> +   _filter_scratch | _filter_test_dir
> >
> > And fixed the indention here, used tab :)
> >
> > Thanks,
> > Eryu
> >
> >> + fi
> >> +}
> >> +
> >>  # Turn any device in the scratch pool into SCRATCH_DEV
> >>  _filter_scratch_pool()
> >>  {
> >> --
> >> 2.9.5
> >>
> > --
> > To unsubscribe from this list: send the line "unsubscribe fstests" in
> > the body of a message to majord...@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/2] Add new common filter function

[Eryu Guan]

On Fri, Sep 01, 2017 at 02:39:44PM +0900, Misono, Tomohiro wrote:
> Several tests uses both _filter_test_dir and _filter_scratch
> concatenated by pipe to filter $TEST_DIR and $SCRATCH_MNT. However, this
> would fail if the shorter string is a substring of the other (like
> "/mnt" and "/mnt2").
> 
> This patch introduces new common filter function to safely call both
> _filter_test_dir and _filter_scratch.
> 
> I chedked this with btrfs/029, generic/409,410,411, and generic/381,383,
> xfs/106,108 (which calls _filter_quota). Thanks Eryu for advice.
> 
> Signed-off-by: Tomohiro Misono 

Thanks! Though I don't think we need two separate patches, so I merged
the two patches together at commit time.

> ---
>  common/filter | 11 +++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/common/filter b/common/filter
> index 1ef342b..75570f9 100644
> --- a/common/filter
> +++ b/common/filter
> @@ -295,6 +295,17 @@ _filter_scratch()
>   -e "/.use_space/d"
>  }
>  
> +_filter_testdir_and_scratch()
> +{
> + # filter both $TEST_DIR and $SCRATCH_MNT, but always filter the longer
> + # string first if the other string is a substring of the first one
> + if echo "$TEST_DIR" | grep -q "$SCRATCH_MNT"; then
> +   _filter_test_dir | _filter_scratch
> + else
> +   _filter_scratch | _filter_test_dir

And fixed the indention here, used tab :)

Thanks,
Eryu

> + fi
> +}
> +
>  # Turn any device in the scratch pool into SCRATCH_DEV
>  _filter_scratch_pool()
>  {
> -- 
> 2.9.5
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs/029: fix wrong usage of name filter

[Eryu Guan]

On Fri, Sep 01, 2017 at 09:44:59AM +0900, Misono, Tomohiro wrote:
> Ok. I will do that if you won't, though I'm not sure other combination of 
> filters would pose the similar problem.

Thanks! Then I'll test :)

Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs/029: fix wrong usage of name filter

[Eryu Guan]

On Thu, Aug 31, 2017 at 08:53:09AM +0900, Misono, Tomohiro wrote:
> On 2017/08/30 20:09, Eryu Guan wrote:
> > On Wed, Aug 30, 2017 at 04:38:16PM +0900, Misono, Tomohiro wrote:
> >> btrfs/029 uses _filter_testdirs() to filter the name of $TEST_DIR and
> >> $SCRATCH_MNT directory.
> >>
> >> In this function, it calls both _filter_test_dir and _filter_scratch
> >> concatenapted by pipe. Therefore if $TEST_DIR is a prefix of
> >> $SCRATCH_MNT, this filter function gives wrong filtered name for
> >> $SCRATCH_MNT and the test fails.
> > 
> > Sorry, I'm a bit confused, how could $TEST_DIR be a prefix of
> > $SCRATCH_MNT? Won't that fail the test at setup time?
> 
> I used "/mnt" for $TEST_DIR and "/mnt_scratch" for $SCRATCH_MNT and hit
> this problem because "/mnt_scratch" is filtered to "$TEST_DIR_scrach"
> instead of "$SCRATCH_MNT".
> 
> I think these are valid directory names and other btrfs tests run correctly
> with these names.

Ah, yes, that's possible and a valid (though not usual) test setup. The
filter becomes complex when one string is prefix of another string, we
have similar problems when filtering TEST_DIR vs TEST_DEV and
SCRATCH_MNT vs SCRATCH_DEV in _filter_test_dir and _filter_scratch.

But the fix only works around btrfs/029, but there're other places we
use the two filters together, like in _filter_quota, TEST_DIR filter
results would be wrong if SCRATCH_MNT is a prefix of TEST_DIR, because
in _filter_quota it calls _filter_scratch first.

I think one solution is that we filter the longer string first, e.g.
move _filter_testdirs to common/filter and update it to something like:

# filter both test and scratch mount points and devices, but always
# filter the longer string if the other string is a substring of the
# first one.
_filter_testdirs()
{
if echo "$TEST_DIR" | grep -q "$SCRATCH_MNT"; then
_filter_test_dir | _filter_scratch
else
_filter_scratch | _filter_test_dir
fi
}

And use this new helper when needed. I found 4 places that need update,
they're btrfs/029, generic/409, generic/410, generic/411 and
_filter_quota().

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs/029: fix wrong usage of name filter

[Eryu Guan]

On Wed, Aug 30, 2017 at 04:38:16PM +0900, Misono, Tomohiro wrote:
> btrfs/029 uses _filter_testdirs() to filter the name of $TEST_DIR and
> $SCRATCH_MNT directory.
> 
> In this function, it calls both _filter_test_dir and _filter_scratch
> concatenated by pipe. Therefore if $TEST_DIR is a prefix of
> $SCRATCH_MNT, this filter function gives wrong filtered name for
> $SCRATCH_MNT and the test fails.

Sorry, I'm a bit confused, how could $TEST_DIR be a prefix of
$SCRATCH_MNT? Won't that fail the test at setup time?

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] fstests: btrfs: enhance regression test for nocsum dio read's repair

[Eryu Guan]

On Mon, Aug 14, 2017 at 03:03:13PM +0800, Lu Fengqi wrote:
> I catch this following error from dmesg when this testcase fails.
> 
> [17446.661127] Buffer I/O error on dev sdb1, logical block 64, async page read
> 
> We expect to inject disk IO errors on the device when xfs_io reads the
> specific file, but other processes may trigger IO error earlier. So, we
> can use task-filter to solve this problem.
> 
> Signed-off-by: Lu Fengqi 

This looks OK to me. Does btrfs/143 need a similar fix?

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs: test if receive with qgroups corrupts metadata

[Eryu Guan]

On Thu, Jul 20, 2017 at 04:03:33PM -0700, Justin Maggard wrote:
> This test case does some concurrent send/receives with qgroups enabled.
> Currently (4.13-rc1) this usually results in btrfs check errors, and
> often also results in a WARN_ON in record_root_in_trans().
> 
> Bisecting points to 6426c7ad697d (btrfs: qgroup: Fix qgroup accounting
> when creating snapshot) as the culprit.

Thanks for the new test! But I'd need some help from btrfs-list to
review if this is a sane test for btrfs.

BTW, you're missing a S-O-B line here :)

> ---
>  tests/btrfs/149 | 101 
> 
>  tests/btrfs/149.out |  17 +
>  tests/btrfs/group   |   1 +
>  3 files changed, 119 insertions(+)
>  create mode 100755 tests/btrfs/149
>  create mode 100644 tests/btrfs/149.out
> 
> diff --git a/tests/btrfs/149 b/tests/btrfs/149
> new file mode 100755
> index 000..5c1912d
> --- /dev/null
> +++ b/tests/btrfs/149
> @@ -0,0 +1,101 @@
> +#! /bin/bash
> +# FS QA Test No. btrfs/149
> +#
> +# Test that incremental send/receive operations don't corrupt metadata when
> +# qgroups are enabled.
> +#
> +#---
> +#
> +# Copyright (c) 2017 NETGEAR, Inc.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# real QA test starts here
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch
> +
> +rm -f $seqres.full
> +
> +_scratch_mkfs >>$seqres.full 2>&1
> +_scratch_mount
> +
> +# Enable quotas
> +$BTRFS_UTIL_PROG quota enable $SCRATCH_MNT
> +
> +# Create 2 source and 4 destination subvolumes
> +for subvol in subvol1 subvol2 recv1_1 recv1_2 recv2_1 recv2_2; do
> + $BTRFS_UTIL_PROG subvolume create $SCRATCH_MNT/$subvol

The only thing I've noticed is that you need to _filter_scratch here and
some other places, so in .out file you don't assume and hardcode
/mnt/scratch as SCRATCH_MNT.

Thanks,
Eryu

> +done
> +mkdir $SCRATCH_MNT/subvol{1,2}/.snapshots
> +
> +# Create base snapshots and send them
> +$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT/subvol1 \
> + $SCRATCH_MNT/subvol1/.snapshots/1
> +$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT/subvol2 \
> + $SCRATCH_MNT/subvol2/.snapshots/1
> +for recv in recv1_1 recv1_2 recv2_1 recv2_2; do
> + $BTRFS_UTIL_PROG send $SCRATCH_MNT/subvol1/.snapshots/1 | \
> + $BTRFS_UTIL_PROG receive $SCRATCH_MNT/${recv}
> +done
> +
> +# Now do 10 loops of concurrent incremental send/receives
> +for i in `seq 1 10`; do
> + prev=$i
> + curr=$((i+1))
> +
> + $BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT/subvol1 \
> + $SCRATCH_MNT/subvol1/.snapshots/${curr} > /dev/null
> + ($BTRFS_UTIL_PROG send -p $SCRATCH_MNT/subvol1/.snapshots/${prev} \
> + $SCRATCH_MNT/subvol1/.snapshots/${curr} 2> /dev/null | \
> + $BTRFS_UTIL_PROG receive $SCRATCH_MNT/recv1_1) > /dev/null &
> + ($BTRFS_UTIL_PROG send -p $SCRATCH_MNT/subvol1/.snapshots/${prev} \
> + $SCRATCH_MNT/subvol1/.snapshots/${curr} 2> /dev/null | \
> + $BTRFS_UTIL_PROG receive $SCRATCH_MNT/recv1_2) > /dev/null &
> +
> + $BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT/subvol2 \
> + $SCRATCH_MNT/subvol2/.snapshots/${curr} > /dev/null
> + ($BTRFS_UTIL_PROG send -p $SCRATCH_MNT/subvol2/.snapshots/${prev} \
> + $SCRATCH_MNT/subvol2/.snapshots/${curr} 2> /dev/null | \
> + $BTRFS_UTIL_PROG receive $SCRATCH_MNT/recv2_1) > /dev/null &
> + ($BTRFS_UTIL_PROG send -p $SCRATCH_MNT/subvol2/.snapshots/${prev} \
> + $SCRATCH_MNT/subvol2/.snapshots/${curr} 2> /dev/null | \
> + $BTRFS_UTIL_PROG receive $SCRATCH_MNT/recv2_2) > /dev/null &
> + wait
> +done
> +
> +_scratch_unmount
> +
> +status=0
> +exit
> diff --git a/tests/btrfs/149.out b/tests/btrfs/149.out
> new file mode 100644
> index 000..3ea9101
> --- 

Re: [PATCH] btrfs: test direct IO write against raid5/6 filesystems

[Eryu Guan]

On Thu, Jul 13, 2017 at 03:10:40PM +0100, fdman...@kernel.org wrote:
> From: Filipe Manana 
> 
> Test that a direct IO write works against raid5/6 filesystems and that
> after the write operation we are able to read back the correct data
> and scrub operations don't find any errors.
> 
> This test is motivated by a regression introduced in the merge window
> for the 4.13 linux kernel, which was undetected by the current set of
> test cases. The issue is fixed by the following patch:
> 
>   "Btrfs: fix write corruption due to bio cloning on raid5/6"
> 
> Signed-off-by: Filipe Manana 

Thanks for the test! I did minor updates and committed:

- add _require_odirect in the test
- add rw group

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] btrfs/146: Test various btrfs operations rounding behavior

[Eryu Guan]

On Fri, Jun 23, 2017 at 04:25:43PM +0800, Eryu Guan wrote:
> On Wed, Jun 21, 2017 at 10:50:35AM +0300, Nikolay Borisov wrote:
> > When changing the size of disks/filesystem we should always be
> > rounding down to a multiple of sectorsize
> > 
> > Signed-off-by: Nikolay Borisov <nbori...@suse.com>
> 
> Thanks for the update! But I still need some reviews from btrfs list to
> see if this is a valid test.

Still looking for reviews on this test. Thanks a lot!

Eryu

> 
> (There're still two places in _cleanup() that have trailing whitespace
> issue, but I can fix them at commit time if test passes review of other
> btrfs developers.)
> 
> Thanks,
> Eryu
> 
> > ---
> > 
> > Changes since v1: 
> >  - Worked on incorporated feedback by Eryu 
> >  - Changed test number to 146 to avoid clashes
> > 
> >  tests/btrfs/146 | 147 
> > 
> >  tests/btrfs/146.out |  20 +++
> >  tests/btrfs/group   |   1 +
> >  3 files changed, 168 insertions(+)
> >  create mode 100755 tests/btrfs/146
> >  create mode 100644 tests/btrfs/146.out
> > 
> > diff --git a/tests/btrfs/146 b/tests/btrfs/146
> > new file mode 100755
> > index 000..7e6d40f
> > --- /dev/null
> > +++ b/tests/btrfs/146
> > @@ -0,0 +1,147 @@
> > +#! /bin/bash
> > +# FS QA Test No. btrfs/146
> > +#
> > +# Test that various code paths which deal with modifying the total size
> > +# of devices/superblock correctly round the value to a multiple of
> > +# sector size
> > +#
> > +#---
> > +#
> > +# Copyright (C) 2017 SUSE Linux Products GmbH. All Rights Reserved.
> > +# Author: Nikolay Borisov <nbori...@suse.com>
> > +#
> > +# This program is free software; you can redistribute it and/or
> > +# modify it under the terms of the GNU General Public License as
> > +# published by the Free Software Foundation.
> > +#
> > +# This program is distributed in the hope that it would be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program; if not, write the Free Software Foundation,
> > +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> > +#---
> > +#
> > +
> > +seq=`basename $0`
> > +seqres=$RESULT_DIR/$seq
> > +echo "QA output created by $seq"
> > +
> > +tmp=/tmp/$$
> > +status=1   # failure is the default!
> > +trap "_cleanup; exit \$status" 0 1 2 3 15
> > +
> > +_cleanup()
> > +{
> > +   $UMOUNT_PROG $loop_mnt 
> > +   _destroy_loop_device $loop_dev1
> > +   _destroy_loop_device $loop_dev2
> > +   cd /
> > +   rm -f $tmp.*
> > +   
> > +}
> > +
> > +# get standard environment, filters and checks
> > +. ./common/rc
> > +. ./common/filter
> > +
> > +# real QA test starts here
> > +_supported_fs btrfs
> > +_supported_os Linux
> > +_require_scratch
> > +_require_loop
> > +_require_block_device $SCRATCH_DEV
> > +_require_btrfs_command inspect-internal dump-super
> > +# ~2.1 gigabytes just to be on the safe side
> > +_require_fs_space $SCRATCH_MNT $(( 2202009 ))
> > +
> > +rm -f $seqres.full
> > +
> > +_scratch_mkfs >>$seqres.full 2>&1
> > +_scratch_mount
> > +
> > +
> > +# Size of devices which are going to be half a page larger than
> > +# default sectorsize (4kb)
> > +expectedsize=$(( 1 * 1024 * 1024 * 1024 ))
> > +filesize=$(( $expectedsize + 2048 ))
> > +loop_mnt=$SCRATCH_MNT/mount
> > +fs_img1=$SCRATCH_MNT/disk1.img
> > +fs_img2=$SCRATCH_MNT/disk2.img
> > +
> > +mkdir $loop_mnt
> > +
> > +#create files to hold dummy devices
> > +$XFS_IO_PROG -f -c "falloc 0 $filesize" $fs_img1 &> /dev/null
> > +$XFS_IO_PROG -f -c "falloc 0 $filesize" $fs_img2 &> /dev/null
> > +
> > +loop_dev1=$(_create_loop_device $fs_img1)
> > +loop_dev2=$(_create_loop_device $fs_img2)
> > +
> > +#create fs only on the first device
> > +_mkfs_dev $loop_dev1
> > +_mount $loop_dev1 $loop_mnt
> > +
> > +echo "Size from mkfs"
> > +$BTRFS_UTIL_PROG inspect-int

Re: [PATCH v2] btrfs/146: Test various btrfs operations rounding behavior

[Eryu Guan]

On Wed, Jun 21, 2017 at 10:50:35AM +0300, Nikolay Borisov wrote:
> When changing the size of disks/filesystem we should always be
> rounding down to a multiple of sectorsize
> 
> Signed-off-by: Nikolay Borisov 

Thanks for the update! But I still need some reviews from btrfs list to
see if this is a valid test.

(There're still two places in _cleanup() that have trailing whitespace
issue, but I can fix them at commit time if test passes review of other
btrfs developers.)

Thanks,
Eryu

> ---
> 
> Changes since v1: 
>  - Worked on incorporated feedback by Eryu 
>  - Changed test number to 146 to avoid clashes
> 
>  tests/btrfs/146 | 147 
> 
>  tests/btrfs/146.out |  20 +++
>  tests/btrfs/group   |   1 +
>  3 files changed, 168 insertions(+)
>  create mode 100755 tests/btrfs/146
>  create mode 100644 tests/btrfs/146.out
> 
> diff --git a/tests/btrfs/146 b/tests/btrfs/146
> new file mode 100755
> index 000..7e6d40f
> --- /dev/null
> +++ b/tests/btrfs/146
> @@ -0,0 +1,147 @@
> +#! /bin/bash
> +# FS QA Test No. btrfs/146
> +#
> +# Test that various code paths which deal with modifying the total size
> +# of devices/superblock correctly round the value to a multiple of
> +# sector size
> +#
> +#---
> +#
> +# Copyright (C) 2017 SUSE Linux Products GmbH. All Rights Reserved.
> +# Author: Nikolay Borisov 
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + $UMOUNT_PROG $loop_mnt 
> + _destroy_loop_device $loop_dev1
> + _destroy_loop_device $loop_dev2
> + cd /
> + rm -f $tmp.*
> + 
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# real QA test starts here
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch
> +_require_loop
> +_require_block_device $SCRATCH_DEV
> +_require_btrfs_command inspect-internal dump-super
> +# ~2.1 gigabytes just to be on the safe side
> +_require_fs_space $SCRATCH_MNT $(( 2202009 ))
> +
> +rm -f $seqres.full
> +
> +_scratch_mkfs >>$seqres.full 2>&1
> +_scratch_mount
> +
> +
> +# Size of devices which are going to be half a page larger than
> +# default sectorsize (4kb)
> +expectedsize=$(( 1 * 1024 * 1024 * 1024 ))
> +filesize=$(( $expectedsize + 2048 ))
> +loop_mnt=$SCRATCH_MNT/mount
> +fs_img1=$SCRATCH_MNT/disk1.img
> +fs_img2=$SCRATCH_MNT/disk2.img
> +
> +mkdir $loop_mnt
> +
> +#create files to hold dummy devices
> +$XFS_IO_PROG -f -c "falloc 0 $filesize" $fs_img1 &> /dev/null
> +$XFS_IO_PROG -f -c "falloc 0 $filesize" $fs_img2 &> /dev/null
> +
> +loop_dev1=$(_create_loop_device $fs_img1)
> +loop_dev2=$(_create_loop_device $fs_img2)
> +
> +#create fs only on the first device
> +_mkfs_dev $loop_dev1
> +_mount $loop_dev1 $loop_mnt
> +
> +echo "Size from mkfs"
> +$BTRFS_UTIL_PROG inspect-internal dump-super /dev/loop0 | grep total
> +
> +#resize down to 768mb + 2k
> +$BTRFS_UTIL_PROG filesystem resize 824182784 $loop_mnt >>$seqres.full 2>&1
> +sync
> +
> +echo ""
> +
> +echo "Size after resizing down"
> +$BTRFS_UTIL_PROG inspect-internal dump-super $loop_dev1 | grep total
> +
> +echo ""
> +
> +#now resize back up to 1 gb
> +$BTRFS_UTIL_PROG filesystem resize max $loop_mnt >>$seqres.full 2>&1
> +sync
> +
> +echo "Size after resizing up"
> +$BTRFS_UTIL_PROG inspect-internal dump-super /dev/loop0 | grep total
> +
> +# Add fs load for later balance operations, we need to do this
> +# before adding a second device
> +$FSSTRESS_PROG -w -n 15000 -p 2 -d $loop_mnt >> $seqres.full 2>&1
> +
> +# add second unaligned device, this checks the btrfs_init_new_device codepath
> +# device should be aligned
> +$BTRFS_UTIL_PROG device add $loop_dev2 $loop_mnt >>$seqres.full 2>&1
> +sync
> +
> +echo ""
> +
> +# Ensure that adding a device doesn't cause the superbock to be
> +# unaligned
> +echo "Size of superblock after device addition"
> +$BTRFS_UTIL_PROG inspect-internal dump-super $loop_dev1 | grep total
> +
> +echo ""
> +
> +# The newly added 

Re: [PATCH] btrfs/145: Test various btrfs operations rounding behavior

[Eryu Guan]

On Thu, Jun 15, 2017 at 04:45:38PM +0300, Nikolay Borisov wrote:
> When changing the size of disks/filesystem we should always be
> rounding down to a multiple of sectorsize. This is a test for the following
> btrfs patche: 
> 
> btrfs: Round up values which are written for total_bytes_size
> 
> Signed-off-by: Nikolay Borisov 

[ cc btrfs list ]

I'd like to use some help from other btrfs developers on this test :)

I'll comment from fstests perspective of view.

> ---
>  tests/btrfs/145 | 147 
> 
>  tests/btrfs/145.out |  20 +++
>  tests/btrfs/group   |   1 +
>  3 files changed, 168 insertions(+)
>  create mode 100755 tests/btrfs/145
>  create mode 100644 tests/btrfs/145.out
> 
> diff --git a/tests/btrfs/145 b/tests/btrfs/145
> new file mode 100755
> index 000..4461b37
> --- /dev/null
> +++ b/tests/btrfs/145
> @@ -0,0 +1,147 @@
> +#! /bin/bash
> +# FS QA Test No. btrfs/145
> +# 

Trailing whitespace issue in above line, and many other lines in this
test, please remove them all.

> +# Test that various code paths which deal with modifying the total size
> +# of devices/superblock correctly round the value to a multiple of 
> +# sector size
> +#
> +#---
> +#
> +# Copyright (C) 2017 SUSE Linux Products GmbH. All Rights Reserved.
> +# Author: Nikolay Borisov 
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + $UMOUNT_PROG $loop_mnt 
> + _destroy_loop_device $loop_dev1
> + _destroy_loop_device $loop_dev2
> + cd /
> + rm -f $tmp.*
> + 
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# real QA test starts here
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch
> +_require_loop
> +_require_block_device $SCRATCH_DEV

I see 'btrfs inspect-internal dump-super' is used, I think we need

_require_btrfs_command inspect-internal dump-super

> +
> +rm -f $seqres.full
> +
> +_scratch_mkfs >>$seqres.full 2>&1
> +_scratch_mount
> +
> +
> +#Size of devices which are going to be half a page larger than 
> +#default sectorsize (4kb)

Nitpick, please add a space after '#' for comments.

> +expectedsize=$(( 1 * 1024 * 1024 * 1024 )) 
> +filesize=$(( $expectedsize + 2048 ))
> +loop_mnt=$SCRATCH_MNT/mount
> +fs_img1=$SCRATCH_MNT/disk1.img
> +fs_img2=$SCRATCH_MNT/disk2.img
> +
> +mkdir $loop_mnt
> +
> +#create files to hold dummy devices
> +$XFS_IO_PROG -f -c "falloc 0 $filesize" $fs_img1 &> /dev/null
> +$XFS_IO_PROG -f -c "falloc 0 $filesize" $fs_img2 &> /dev/null

Perhaps we need to make sure $SCRATCH_MNT has enough free space to hold
the two fs imgs.

_require_fs_space $SCRATCH_MNT 

> +
> +loop_dev1=$(_create_loop_device $fs_img1)
> +loop_dev2=$(_create_loop_device $fs_img2) 
> +
> +#create fs only on the first device
> +_mkfs_dev $loop_dev1 
> +_mount $loop_dev1 $loop_mnt
> +
> +echo "Size from mkfs"
> +$BTRFS_UTIL_PROG inspect-internal dump-super /dev/loop0 | grep total
^^ $loop_dev1
> +
> +#resize down to 768mb + 2k
> +$BTRFS_UTIL_PROG fi resize 824182784 $loop_mnt >>$seqres.full 2>&1 

Better to use full subcommand name, i.e. s/fi/filesystem/, I remembered
there was a patch to do this

2fd80ca btrfs: use full subcommand names

> +sync
> +
> +echo ""
> +
> +echo "Size after resizing down"
> +$BTRFS_UTIL_PROG inspect-internal dump-super $loop_dev1 | grep total
> +
> +echo ""
> +
> +#now resize back up to 1 gb
> +$BTRFS_UTIL_PROG fi resize max $loop_mnt >>$seqres.full 2>&1 
> +sync
> +
> +echo "Size after resizing up"
> +$BTRFS_UTIL_PROG inspect-internal dump-super /dev/loop0 | grep total
> +
> +#Add fs load for later balance operations, we need to do this 
> +#before adding a second device
> +$FSSTRESS_PROG -w -n 15000 -p 2 -d $loop_mnt >> $seqres.full 2>&1
> +
> +#add second unaligned device, this checks the btrfs_init_new_device codepath 
> +#device should be aligned
> +$BTRFS_UTIL_PROG device add $loop_dev2 

Re: [xfstests PATCH v4 5/5] btrfs: make a btrfs version of writeback error reporting test

[Eryu Guan]

On Wed, Jun 14, 2017 at 07:55:17AM -0400, Jeff Layton wrote:
> On Tue, 2017-06-13 at 16:40 +0800, Eryu Guan wrote:
> > On Mon, Jun 12, 2017 at 08:42:13AM -0400, Jeff Layton wrote:
> > > Make a new btrfs/999 test that works the way Chris Mason suggested:
> > > 
> > > Build a filesystem with 2 devices that stripes the data across
> > > both devices, but mirrors metadata across both. Then, make one
> > > of the devices fail and see how fsync is handled.
> > > 
> > > Signed-off-by: Jeff Layton <jlay...@redhat.com>
> > > ---
> > >  tests/btrfs/999   | 93 
> > > +++
> > 
> > Missing btrfs/999.out file
> > 
> > >  tests/btrfs/group |  1 +
> > >  2 files changed, 94 insertions(+)
> > >  create mode 100755 tests/btrfs/999
> > > 
> > > diff --git a/tests/btrfs/999 b/tests/btrfs/999
> > > new file mode 100755
> > > index ..84031cc0d913
> > > --- /dev/null
> > > +++ b/tests/btrfs/999
> > > @@ -0,0 +1,93 @@
> > > +#! /bin/bash
> > > +# FS QA Test No. 999
> > > +#
> > > +# Open a file several times, write to it, fsync on all fds and make sure 
> > > that
> > > +# they all return 0. Change the device to start throwing errors. Write 
> > > again
> > > +# on all fds and fsync on all fds. Ensure that we get errors on all of 
> > > them.
> > > +# Then fsync on all one last time and verify that all return 0.
> > > +#
> > > +#---
> > > +# Copyright (c) 2017, Jeff Layton <jlay...@redhat.com>
> > > +#
> > > +# This program is free software; you can redistribute it and/or
> > > +# modify it under the terms of the GNU General Public License as
> > > +# published by the Free Software Foundation.
> > > +#
> > > +# This program is distributed in the hope that it would be useful,
> > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > > +# GNU General Public License for more details.
> > > +#
> > > +# You should have received a copy of the GNU General Public License
> > > +# along with this program; if not, write the Free Software Foundation,
> > > +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> > > +#---
> > > +
> > > +seq=`basename $0`
> > > +seqres=$RESULT_DIR/$seq
> > > +echo "QA output created by $seq"
> > > +
> > > +here=`pwd`
> > > +tmp=/tmp/$$
> > > +status=1# failure is the default!
> > > +trap "_cleanup; exit \$status" 0 1 2 3 15
> > > +
> > > +_cleanup()
> > > +{
> > > +cd /
> > > +rm -rf $tmp.* $testdir
> > > +_dmerror_cleanup
> > > +}
> > > +
> > > +# get standard environment, filters and checks
> > > +. ./common/rc
> > > +. ./common/filter
> > > +. ./common/dmerror
> > > +
> > > +# real QA test starts here
> > > +_supported_os Linux
> > > +_require_dm_target error
> > > +_require_test_program fsync-err
> > > +_require_test_program dmerror
> > > +
> > > +# bring up dmerror device
> > > +_scratch_unmount
> > > +_dmerror_init
> > > +
> > > +# Replace first device with error-test device
> > > +old_SCRATCH_DEV=$SCRATCH_DEV
> > > +SCRATCH_DEV_POOL=`echo $SCRATCH_DEV_POOL | perl -pe 
> > > "s#$SCRATCH_DEV#$DMERROR_DEV#"`
> > > +SCRATCH_DEV=$DMERROR_DEV
> > > +
> > > +_require_scratch
> > > +_require_scratch_dev_pool
> > 
> > Need "_require_scratch_dev_pool_equal_size" too, since test creates
> > raid1 profile for metadata.

Sorry, it's not needed here. I got confused with device replace
operation, only "replace" needs this require rule. Thanks for
confirming!

Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [xfstests PATCH v4 5/5] btrfs: make a btrfs version of writeback error reporting test

[Eryu Guan]

On Mon, Jun 12, 2017 at 08:42:13AM -0400, Jeff Layton wrote:
> Make a new btrfs/999 test that works the way Chris Mason suggested:
> 
> Build a filesystem with 2 devices that stripes the data across
> both devices, but mirrors metadata across both. Then, make one
> of the devices fail and see how fsync is handled.
> 
> Signed-off-by: Jeff Layton 
> ---
>  tests/btrfs/999   | 93 
> +++

Missing btrfs/999.out file

>  tests/btrfs/group |  1 +
>  2 files changed, 94 insertions(+)
>  create mode 100755 tests/btrfs/999
> 
> diff --git a/tests/btrfs/999 b/tests/btrfs/999
> new file mode 100755
> index ..84031cc0d913
> --- /dev/null
> +++ b/tests/btrfs/999
> @@ -0,0 +1,93 @@
> +#! /bin/bash
> +# FS QA Test No. 999
> +#
> +# Open a file several times, write to it, fsync on all fds and make sure that
> +# they all return 0. Change the device to start throwing errors. Write again
> +# on all fds and fsync on all fds. Ensure that we get errors on all of them.
> +# Then fsync on all one last time and verify that all return 0.
> +#
> +#---
> +# Copyright (c) 2017, Jeff Layton 
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> +cd /
> +rm -rf $tmp.* $testdir
> +_dmerror_cleanup
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/dmerror
> +
> +# real QA test starts here
> +_supported_os Linux
> +_require_dm_target error
> +_require_test_program fsync-err
> +_require_test_program dmerror
> +
> +# bring up dmerror device
> +_scratch_unmount
> +_dmerror_init
> +
> +# Replace first device with error-test device
> +old_SCRATCH_DEV=$SCRATCH_DEV
> +SCRATCH_DEV_POOL=`echo $SCRATCH_DEV_POOL | perl -pe 
> "s#$SCRATCH_DEV#$DMERROR_DEV#"`
> +SCRATCH_DEV=$DMERROR_DEV
> +
> +_require_scratch
> +_require_scratch_dev_pool

Need "_require_scratch_dev_pool_equal_size" too, since test creates
raid1 profile for metadata.

Thanks,
Eryu

> +
> +rm -f $seqres.full
> +
> +echo "Format and mount"
> +
> +_scratch_pool_mkfs "-d raid0 -m raid1" > $seqres.full 2>&1
> +_scratch_mount
> +
> +# How much do we need to write? We need to hit all of the stripes. btrfs uses
> +# a fixed 64k stripesize, so write enough to hit each one
> +number_of_devices=`echo $SCRATCH_DEV_POOL | wc -w`
> +write_kb=$(($number_of_devices * 64))
> +_require_fs_space $SCRATCH_MNT $write_kb
> +
> +testfile=$SCRATCH_MNT/fsync-err-test
> +
> +SCRATCH_DEV=$old_SCRATCH_DEV
> +$here/src/fsync-err -b $(($write_kb * 1024)) -d $here/src/dmerror $testfile
> +
> +# success, all done
> +_dmerror_load_working_table
> +
> +# fs may be corrupt after this -- attempt to repair it
> +_repair_scratch_fs >> $seqres.full
> +
> +# remove dmerror device
> +_dmerror_cleanup
> +
> +status=0
> +exit
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index 6f19619e877c..8dbdfbfe29fd 100644
> --- a/tests/btrfs/group
> +++ b/tests/btrfs/group
> @@ -145,3 +145,4 @@
>  141 auto quick
>  142 auto quick
>  143 auto quick
> +999 auto quick
> -- 
> 2.13.0
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [xfstests PATCH v4 2/5] ext4: allow ext4 to use $SCRATCH_LOGDEV

[Eryu Guan]

On Mon, Jun 12, 2017 at 08:42:10AM -0400, Jeff Layton wrote:
> The writeback error handling test requires that you put the journal on a
> separate device. This allows us to use dmerror to simulate data
> writeback failure, without affecting the journal.
> 
> xfs already has infrastructure for this (a'la $SCRATCH_LOGDEV), so wire
> up the ext4 code so that it can do the same thing when _scratch_mkfs is
> called.
> 
> Signed-off-by: Jeff Layton 
> Reviewed-by: Darrick J. Wong 
> ---
>  common/rc | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/common/rc b/common/rc
> index 87e6ff08b18d..08807ac7c22a 100644
> --- a/common/rc
> +++ b/common/rc
> @@ -676,6 +676,9 @@ _scratch_mkfs_ext4()
>   local tmp=`mktemp`
>   local mkfs_status
>  
> + [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_LOGDEV" ] && \
> + $mkfs_cmd -O journal_dev $SCRATCH_LOGDEV && \
> + mkfs_cmd="$mkfs_cmd $MKFS_OPTIONS -J device=$SCRATCH_LOGDEV"

This $MKFS_OPTIONS should be added to the first command when creating
the journal device so that journal dev has the same block size as data
dev, there's no need to update mkfs_cmd string.

The external log dev support for ext3 patch has similar issue.

Thanks,
Eryu

>  
>   _scratch_do_mkfs "$mkfs_cmd" "$mkfs_filter" $* 2>$tmp.mkfserr 
> 1>$tmp.mkfsstd
>   mkfs_status=$?
> -- 
> 2.13.0
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [xfstests PATCH v4 1/5] generic: add a writeback error handling test

[Eryu Guan]

On Mon, Jun 12, 2017 at 08:42:09AM -0400, Jeff Layton wrote:
> I'm working on a set of kernel patches to change how writeback errors
> are handled and reported in the kernel. Instead of reporting a
> writeback error to only the first fsync caller on the file, I aim
> to make the kernel report them once on every file description.
> 
> This patch adds a test for the new behavior. Basically, open many fds
> to the same file, turn on dm_error, write to each of the fds, and then
> fsync them all to ensure that they all get an error back.
> 
> To do that, I'm adding a new tools/dmerror script that the C program
> can use to load the error table. For now, that's all it can do, but
> we can fill it out with other commands as necessary.
> 
> Signed-off-by: Jeff Layton 
> ---
>  .gitignore |   1 +
>  common/dmerror |  13 ++-
>  doc/auxiliary-programs.txt |  16 
>  src/Makefile   |   2 +-
>  src/dmerror|  44 +
>  src/fsync-err.c| 223 
> +
>  tests/generic/999  |  77 
>  tests/generic/999.out  |   3 +
>  tests/generic/group|   1 +
>  9 files changed, 374 insertions(+), 6 deletions(-)
>  create mode 100755 src/dmerror
>  create mode 100644 src/fsync-err.c
>  create mode 100755 tests/generic/999
>  create mode 100644 tests/generic/999.out
> 
> diff --git a/.gitignore b/.gitignore
> index 39664b0a7f53..56e863b2c8dc 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -72,6 +72,7 @@
>  /src/fs_perms
>  /src/fssum
>  /src/fstest
> +/src/fsync-err
>  /src/fsync-tester
>  /src/ftrunc
>  /src/genhashnames
> diff --git a/common/dmerror b/common/dmerror
> index d46c5d0b7266..238baa213b1f 100644
> --- a/common/dmerror
> +++ b/common/dmerror
> @@ -23,22 +23,25 @@ if [ $? -eq 0 ]; then
>   _notrun "Cannot run tests with DAX on dmerror devices"
>  fi
>  
> -_dmerror_init()
> +_dmerror_setup()
>  {
>   local dm_backing_dev=$SCRATCH_DEV
>  
> - $DMSETUP_PROG remove error-test > /dev/null 2>&1
> -
>   local blk_dev_size=`blockdev --getsz $dm_backing_dev`
>  
>   DMERROR_DEV='/dev/mapper/error-test'
>  
>   DMLINEAR_TABLE="0 $blk_dev_size linear $dm_backing_dev 0"
>  
> + DMERROR_TABLE="0 $blk_dev_size error $dm_backing_dev 0"
> +}
> +
> +_dmerror_init()
> +{
> + _dmerror_setup
> + $DMSETUP_PROG remove error-test > /dev/null 2>&1
>   $DMSETUP_PROG create error-test --table "$DMLINEAR_TABLE" || \
>   _fatal "failed to create dm linear device"
> -
> - DMERROR_TABLE="0 $blk_dev_size error $dm_backing_dev 0"
>  }
>  
>  _dmerror_mount()
> diff --git a/doc/auxiliary-programs.txt b/doc/auxiliary-programs.txt
> index 21ef118596b6..bcab453c4335 100644
> --- a/doc/auxiliary-programs.txt
> +++ b/doc/auxiliary-programs.txt
> @@ -16,6 +16,8 @@ note the dependency with:
>  Contents:
>  
>   - af_unix   -- Create an AF_UNIX socket
> + - dmerror   -- fault injection block device control
> + - fsync-err -- tests fsync error reporting after failed writeback
>   - open_by_handle-- open_by_handle_at syscall exercise
>   - stat_test -- statx syscall exercise
>   - t_dir_type-- print directory entries and their file type
> @@ -30,6 +32,20 @@ af_unix
>  
>   The af_unix program creates an AF_UNIX socket at the given location.
>  
> +dmerror
> +
> + dmerror is a program for creating, destroying and controlling a
> + fault injection device. The device can be set up as initially
> + working and then flip to throwing errors for testing purposes.
> +
> +fsync-err
> +
> + Specialized program for testing how the kernel reports errors that
> + occur during writeback. Works in conjunction with the dmerror script
> + in tools/ to write data to a device, and then force it to fail
> + writeback and test that errors are reported during fsync and cleared
> + afterward.
> +
>  open_by_handle
>  
>   The open_by_handle program exercises the open_by_handle_at() system
> diff --git a/src/Makefile b/src/Makefile
> index 6b0e4b022485..abd0fff34a64 100644
> --- a/src/Makefile
> +++ b/src/Makefile
> @@ -13,7 +13,7 @@ TARGETS = dirstress fill fill2 getpagesize holes lstat64 \
>   multi_open_unlink dmiperf unwritten_sync genhashnames t_holes \
>   t_mmap_writev t_truncate_cmtime dirhash_collide t_rename_overwrite \
>   holetest t_truncate_self t_mmap_dio af_unix t_mmap_stale_pmd \
> - t_mmap_cow_race t_mmap_fallocate
> + t_mmap_cow_race t_mmap_fallocate fsync-err

src/dmerror should be installed on 'make install' too.

-   $(LTINSTALL) -m 755 fill2attr fill2fs fill2fs_check scaleread.sh 
$(PKG_LIB_DIR)/src
+   $(LTINSTALL) -m 755 fill2attr fill2fs fill2fs_check scaleread.sh 
dmerror $(PKG_LIB_DIR)/src

>  
>  LINUX_TARGETS = xfsctl bstat t_mtab getdevicesize preallo_rw_pattern_reader \
>   preallo_rw_pattern_writer ftrunc trunc 

Re: [xfstests PATCH v4 0/5] new tests for writeback error reporting behavior

[Eryu Guan]

On Mon, Jun 12, 2017 at 08:42:08AM -0400, Jeff Layton wrote:
> v4: respin set based on Eryu's comments
> 
> These tests are companion tests to the patchset I recently posted with
> the cover letter:
> 
> [PATCH v6 00/20] fs: enhanced writeback error reporting with errseq_t 
> (pile #1)
> 
> This set just adds 3 new xfstests to test writeback behavior. One generic
> filesystem test, one test for raw block devices, and one test for btrfs.
> The tests work with dmerror to ensure that writeback fails, and then
> tests how the kernel reports errors afterward.
> 
> xfs, ext2/3/4 and btrfs all pass on a kernel with the patchset above.

xfs, ext3/4 passed generic/999, btrfs passed btrfs/999 (with some local
modifications, see reply to btrfs/999), but ext2 (using ext4 driver) and
btrfs failed generic/999 on my host. (See test log at the end of mail.)

In the ext2 case, this test requires an external log device to run, but
ext2 has no journal at all, I wonder if we should _notrun on ext2.

btrfs doesn't support external log device either, it should not run this
generic test either.

I think _require_logdev() should be updated too, to do a check on
$FSTYP, only allows filesystems that have external log device support to
continue to run.

> 
> The one comment I couldn't really address from earlier review is that
> we don't have a great way for xfstests to tell what sort of error
> reporting behavior it should expect from the running kernel. That
> makes it difficult to tell whether failure is expected during a given
> run.
> 
> Maybe that's OK though and we should just let unconverted filesystems
> fail this test?

If there's really no good way to tell if current fs supports this new
behavior, I think this is fine, strictly speaking, it's not a new
feature anyway.

Thanks,
Eryu

P.S. ext2 and btrfs failure in generic/999 run (I renumbered it to
generic/441)

[root@ibm-x3550m3-05 xfstests]# ./check generic/441 generic/442
FSTYP -- ext2
PLATFORM  -- Linux/x86_64 ibm-x3550m3-05 4.12.0-rc4.jlayton+
MKFS_OPTIONS  -- /dev/sdc2
MOUNT_OPTIONS -- -o acl,user_xattr -o context=system_u:object_r:root_t:s0 
/dev/sdc2 /mnt/testarea/scratch

generic/441 4s ... - output mismatch (see 
/root/xfstests/results//generic/441.out.bad)
--- tests/generic/441.out   2017-06-13 15:52:09.928413126 +0800
+++ /root/xfstests/results//generic/441.out.bad 2017-06-13 
16:21:41.414112226 +0800
@@ -1,3 +1,3 @@
 QA output created by 441
 Format and mount
-Test passed!
+Third fsync on fd[0] failed: Input/output error
...
(Run 'diff -u tests/generic/441.out 
/root/xfstests/results//generic/441.out.bad'  to see the entire diff)

[root@ibm-x3550m3-05 xfstests]# ./check generic/441 generic/442
FSTYP -- btrfs
PLATFORM  -- Linux/x86_64 ibm-x3550m3-05 4.12.0-rc4.jlayton+
MKFS_OPTIONS  -- /dev/sdc2
MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/sdc2 
/mnt/testarea/scratch

generic/441 4s ... - output mismatch (see 
/root/xfstests/results//generic/441.out.bad)
--- tests/generic/441.out   2017-06-13 15:52:09.928413126 +0800
+++ /root/xfstests/results//generic/441.out.bad 2017-06-13 
16:25:17.483273992 +0800
@@ -1,3 +1,3 @@
 QA output created by 441
 Format and mount
-Test passed!
+Third fsync on fd[0] failed: Read-only file system
...
(Run 'diff -u tests/generic/441.out 
/root/xfstests/results//generic/441.out.bad'  to see the entire diff)
> 
> Jeff Layton (5):
>   generic: add a writeback error handling test
>   ext4: allow ext4 to use $SCRATCH_LOGDEV
>   generic: test writeback error handling on dmerror devices
>   ext3: allow it to put journal on a separate device when doing
> scratch_mkfs
>   btrfs: make a btrfs version of writeback error reporting test
> 
>  .gitignore |   1 +
>  common/dmerror |  13 ++-
>  common/rc  |  14 ++-
>  doc/auxiliary-programs.txt |  16 
>  src/Makefile   |   2 +-
>  src/dmerror|  44 +
>  src/fsync-err.c| 223 
> +
>  tests/btrfs/999|  93 +++
>  tests/btrfs/group  |   1 +
>  tests/generic/998  |  64 +
>  tests/generic/998.out  |   2 +
>  tests/generic/999  |  77 
>  tests/generic/999.out  |   3 +
>  tests/generic/group|   2 +
>  14 files changed, 548 insertions(+), 7 deletions(-)
>  create mode 100755 src/dmerror
>  create mode 100644 src/fsync-err.c
>  create mode 100755 tests/btrfs/999
>  create mode 100755 tests/generic/998
>  create mode 100644 tests/generic/998.out
>  create mode 100755 tests/generic/999
>  create mode 100644 tests/generic/999.out
> 
> -- 
> 2.13.0
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs: test Btrfs delalloc accounting overflow

[Eryu Guan]

On Wed, Jun 07, 2017 at 10:34:13AM -0700, Omar Sandoval wrote:
> On Wed, Jun 07, 2017 at 05:36:45PM +0800, Eryu Guan wrote:
> > On Tue, Jun 06, 2017 at 11:57:10PM -0700, Omar Sandoval wrote:
> > > From: Omar Sandoval <osan...@fb.com>
> > > 
> > > This is a regression test for "[PATCH] Btrfs: fix delalloc accounting
> > > leak caused by u32 overflow". It creates a bunch of delalloc extents and
> > > merges them together to make sure the accounting is done right.
> > > 
> > > Signed-off-by: Omar Sandoval <osan...@fb.com>
> > 
> > I refer to the patch by its commit id 8b8b08cbfb90 as well, and replaced
> > xfs_io with $XFS_IO_PROG, and queued for next update.
> 
> Hi, Eryu,
> 
> 8b8b08cbfb90 is actually a different fix, my patch isn't upstream yet
> (but Dave has it queued up for 4.13).

Ahh, I see, I only searched "fix delalloc accounting" in commit log and
didn't notice the difference between the rest of the log. Sorry about
that, and thanks for the heads up!

Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs: test Btrfs delalloc accounting overflow

[Eryu Guan]

On Tue, Jun 06, 2017 at 11:57:10PM -0700, Omar Sandoval wrote:
> From: Omar Sandoval 
> 
> This is a regression test for "[PATCH] Btrfs: fix delalloc accounting
> leak caused by u32 overflow". It creates a bunch of delalloc extents and
> merges them together to make sure the accounting is done right.
> 
> Signed-off-by: Omar Sandoval 

I refer to the patch by its commit id 8b8b08cbfb90 as well, and replaced
xfs_io with $XFS_IO_PROG, and queued for next update.

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] generic: test Btrfs delalloc accounting overflow

[Eryu Guan]

On Tue, Jun 06, 2017 at 05:03:05PM -0700, Omar Sandoval wrote:
> On Sat, Jun 03, 2017 at 12:37:00AM -0700, Christoph Hellwig wrote:
> > This looks like a btrfs-specific test, and not like a generic one
> > to me.
> 
> Nothing about the workload itself is btrfs-specific, we just have the
> extra check at the end. But I don't really care, I can make it a btrfs
> test unless Eryu has already applied it.

What's not very clear to me is that how do we check the accountings for
non-btrfs filesystems, i.e. what would cause a test failure on non-btrfs
filesystems besides an xfs_io write failure? I'd prefer a btrfs-specific
test if there's no good way to do the check. Or if we want to keep it a
generic test, some comments on the non-btrfs case would be good. I'm
fine with either way.

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] generic: test Btrfs delalloc accounting overflow

[Eryu Guan]

On Fri, Jun 02, 2017 at 02:46:52PM +0200, David Sterba wrote:
> On Fri, Jun 02, 2017 at 12:07:37PM +0300, Nikolay Borisov wrote:
> > > +# Make sure that we didn't leak any metadata space.
> > > +if [[ $FSTYP = btrfs ]]; then
> > > + uuid="$(findmnt -n -o UUID "$TEST_DIR")"
> > 
> > if we are on btrfs and we don't have findmnt this test will likely fail.
> > Perhaps include a _require_command findmnt
> 
> I think utilities like findmnt should be checked at the beginning of the
> whole testuiste, not in each test that uses them. As findmnt is part of

Agreed. I think we can define a FINDMNT_PROG in common/config and refuse
to run any test if it's mising, as what we did to $MOUNT_PROG and other
must-have commands. (There's already a bare call to findmnt in
common/rc, change it to call $FINDMNT_PROG too).

We can do this in a separate patch.

Thanks,
Eryu

> util-linux, missing it would also mean that eg 'mount' is missing.
> Highly unlikely.
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] fstests: common: Make _test_mount to include MOUNT_OPTIONS to allow consistent _test_cycle_mount

[Eryu Guan]

On Wed, May 24, 2017 at 05:27:24PM +0800, Qu Wenruo wrote:
> 
> 
> At 05/24/2017 05:22 PM, Eryu Guan wrote:
> > On Wed, May 24, 2017 at 03:58:11PM +0800, Qu Wenruo wrote:
> > > 
> > > 
> > > At 05/24/2017 01:16 PM, Qu Wenruo wrote:
> > > > 
> > > > 
> > > > At 05/24/2017 01:08 PM, Eryu Guan wrote:
> > > > > On Wed, May 24, 2017 at 12:28:34PM +0800, Qu Wenruo wrote:
> > > > > > 
> > > > > > 
> > > > > > At 05/24/2017 12:24 PM, Eryu Guan wrote:
> > > > > > > On Wed, May 24, 2017 at 08:22:25AM +0800, Qu Wenruo wrote:
> > > > > > > > 
> > > > > > > > 
> > > > > > > > At 05/23/2017 07:13 PM, Eryu Guan wrote:
> > > > > > > > > On Tue, May 23, 2017 at 04:02:05PM +0800, Qu Wenruo wrote:
> > > > > > > > > > [BUG]
> > > > > > > > > > If using MOUNT_OPTIONS="-o nodatasum" and btrfs to run 
> > > > > > > > > > genierc/142
> > > > > > > > > > generic/143 and generic/154, it will cause false alert like:
> > > > > > > > > > cp: failed to clone '/mnt/test/test-154/file2'
> > > > > > > > > > from '/mnt/test/test-154/file1': Invalid
> > > > > > > > > > argument
> > > > > > > > > 
> > > > > > > > > MOUNT_OPTIONS is for scratch mount, and
> > > > > > > > > TEST_FS_MOUNT_OPTS is for test
> > > > > > > > > dev mount, so I think setting TEST_FS_MOUNT_OPTS to "-o 
> > > > > > > > > nodatasum"
> > > > > > > > > should fix your problem.
> > > > > > > > 
> > > > > > > > Nope, the problem is the inconsistent of TEST_MNT setup.
> > > > > > > 
> > > > > > > It does fix the failure for me, did I miss anything?
> > > > > > > 
> > > > > > > # MOUNT_OPTIONS="-o nodatasum" TEST_FS_MOUNT_OPTS="-o
> > > > > > > nodatasum" ./check generic/142 generic/143 generic/154
> > > > > > > FSTYP -- btrfs
> > > > > > > PLATFORM  -- Linux/x86_64 dhcp-66-86-11 4.12.0-rc1
> > > > > > > MKFS_OPTIONS  -- /dev/sda6
> > > > > > > MOUNT_OPTIONS -- -o nodatasum -o
> > > > > > > context=system_u:object_r:root_t:s0 /dev/sda6
> > > > > > > /mnt/testarea/scratch
> > > > > > > 
> > > > > > > generic/142 2s ... 1s
> > > > > > > generic/143  18s
> > > > > > > generic/154  1s
> > > > > > > Ran: generic/142 generic/143 generic/154
> > > > > > > Passed all 3 tests
> > > > > > > 
> > > > > > 
> > > > > > But if you only export MOUNT_OPTIONS, it will fail, due to the 
> > > > > > different
> > > > > > mount options between test_cycle_mount().
> > > > > 
> > > > > That's correct. Sorry, I didn't make it clear in my first reply. I 
> > > > > meant
> > > > > that you should set both TEST_FS_MOUNT_OPTS and MOUNT_OPTIONS to
> > > > > "-onodatasum", for both test dev and scratch dev.
> > > > 
> > > > That's just a workaround, not a root fix.
> > > > 
> > > > Not to mention quite a lot test cases lose its coverage.
> > > > As after _test_cycle_mount(), they are just testing default mount 
> > > > option.
> > > > 
> > > > > 
> > > > > > 
> > > > > > To make it clear:
> > > > > > If test mount follows TEST_FS_MOUNT_OPTS, then both the first mount 
> > > > > > and
> > > > > > test_cycle_mount should follow TEST_FS_MOUNT_OPTS.
> > > > > 
> > > > > _test_mount does follow TEST_FS_MOUNT_OPTS, not MOUNT_OPTIONS, no 
> > > > > matter
> > > > > which mount it is.
> > > > 
> > > > While test mount setup by check script follows MOUNT_OPTIONS.
> > > 
> > > Well, $MOUNT_OPTIONS is appended by check_generic_filsystem() or
> > > check_btrfs_filesystem() for btrfs.
> > 
> > Yeah, you're right, I found it too. The _check__filesystem

Re: [PATCH] fstests: common: Make _test_mount to include MOUNT_OPTIONS to allow consistent _test_cycle_mount

[Eryu Guan]

On Wed, May 24, 2017 at 03:58:11PM +0800, Qu Wenruo wrote:
> 
> 
> At 05/24/2017 01:16 PM, Qu Wenruo wrote:
> > 
> > 
> > At 05/24/2017 01:08 PM, Eryu Guan wrote:
> > > On Wed, May 24, 2017 at 12:28:34PM +0800, Qu Wenruo wrote:
> > > > 
> > > > 
> > > > At 05/24/2017 12:24 PM, Eryu Guan wrote:
> > > > > On Wed, May 24, 2017 at 08:22:25AM +0800, Qu Wenruo wrote:
> > > > > > 
> > > > > > 
> > > > > > At 05/23/2017 07:13 PM, Eryu Guan wrote:
> > > > > > > On Tue, May 23, 2017 at 04:02:05PM +0800, Qu Wenruo wrote:
> > > > > > > > [BUG]
> > > > > > > > If using MOUNT_OPTIONS="-o nodatasum" and btrfs to run 
> > > > > > > > genierc/142
> > > > > > > > generic/143 and generic/154, it will cause false alert like:
> > > > > > > > cp: failed to clone '/mnt/test/test-154/file2'
> > > > > > > > from '/mnt/test/test-154/file1': Invalid
> > > > > > > > argument
> > > > > > > 
> > > > > > > MOUNT_OPTIONS is for scratch mount, and
> > > > > > > TEST_FS_MOUNT_OPTS is for test
> > > > > > > dev mount, so I think setting TEST_FS_MOUNT_OPTS to "-o nodatasum"
> > > > > > > should fix your problem.
> > > > > > 
> > > > > > Nope, the problem is the inconsistent of TEST_MNT setup.
> > > > > 
> > > > > It does fix the failure for me, did I miss anything?
> > > > > 
> > > > > # MOUNT_OPTIONS="-o nodatasum" TEST_FS_MOUNT_OPTS="-o
> > > > > nodatasum" ./check generic/142 generic/143 generic/154
> > > > > FSTYP -- btrfs
> > > > > PLATFORM  -- Linux/x86_64 dhcp-66-86-11 4.12.0-rc1
> > > > > MKFS_OPTIONS  -- /dev/sda6
> > > > > MOUNT_OPTIONS -- -o nodatasum -o
> > > > > context=system_u:object_r:root_t:s0 /dev/sda6
> > > > > /mnt/testarea/scratch
> > > > > 
> > > > > generic/142 2s ... 1s
> > > > > generic/143  18s
> > > > > generic/154  1s
> > > > > Ran: generic/142 generic/143 generic/154
> > > > > Passed all 3 tests
> > > > > 
> > > > 
> > > > But if you only export MOUNT_OPTIONS, it will fail, due to the different
> > > > mount options between test_cycle_mount().
> > > 
> > > That's correct. Sorry, I didn't make it clear in my first reply. I meant
> > > that you should set both TEST_FS_MOUNT_OPTS and MOUNT_OPTIONS to
> > > "-onodatasum", for both test dev and scratch dev.
> > 
> > That's just a workaround, not a root fix.
> > 
> > Not to mention quite a lot test cases lose its coverage.
> > As after _test_cycle_mount(), they are just testing default mount option.
> > 
> > > 
> > > > 
> > > > To make it clear:
> > > > If test mount follows TEST_FS_MOUNT_OPTS, then both the first mount and
> > > > test_cycle_mount should follow TEST_FS_MOUNT_OPTS.
> > > 
> > > _test_mount does follow TEST_FS_MOUNT_OPTS, not MOUNT_OPTIONS, no matter
> > > which mount it is.
> > 
> > While test mount setup by check script follows MOUNT_OPTIONS.
> 
> Well, $MOUNT_OPTIONS is appended by check_generic_filsystem() or
> check_btrfs_filesystem() for btrfs.

Yeah, you're right, I found it too. The _check__filesystem() mounts
with MOUNT_OPTIONS unconditionally.

> 
> Which will remount the device with $MOUNT_OPTIONS.
> Further more, this function doesn't care if it's scratch device or test
> device, just use MOUNT_OPTIONS.
> 
> So either we split it for scratch and test device, and follows different
> mount options,
> or just like my submitted patch, to make _test_mount() to follow
> MOUNT_OPTIONS.

Fixing _check__filesystem() is the correct way. And I guess we can
refactor out a common function and call it in
_check_[xfs|btrfs|generic]_filesystem.

Thanks for looking into this!

Eryu

> 
> Thanks,
> Qu
> 
> > 
> > Just check the following very basic test script:
> > --
> > #! /bin/bash
> > # FS QA Test 010
> > #
> > # what am I here for?
> > #
> > seq=`basename $0`
> > seqres=$RESULT_DIR/$seq
> > echo "QA output created by $seq"
> > 
> > here=`pwd`
> > tmp=/tmp/$$
> > status=1# failure is the default!
> > trap "_cleanup; exit \$status" 0 1 2 3 15
> > 
> > _cleanup()
> > {
> >  cd /
> >  rm -f $tmp.*
> > }
> > 
> > # get standard environment, filters and checks
> > . ./common/rc
> > . ./common/filter
> > 
> > # remove previous $seqres.full before test
> > rm -f $seqres.full
> > 
> > # real QA test starts here
> > 
> > # Modify as appropriate.
> > _supported_fs generic
> > _supported_os Linux
> > _require_test
> > 
> > mount > $tmp.mount1
> > _test_cycle_mount
> > mount > $tmp.mount2
> > 
> > diff  $tmp.mount1 $tmp.mount2
> > 
> > echo "Silence is golden"
> > # success, all done
> > status=0
> > exit
> > 
> > --
> > 
> > Thanks,
> > Qu
> > > 
> > > Thanks,
> > > Eryu
> > > -- 
> > > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > > the body of a message to majord...@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > 
> > > 
> 
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] fstests: common: Make _test_mount to include MOUNT_OPTIONS to allow consistent _test_cycle_mount

[Eryu Guan]

On Wed, May 24, 2017 at 12:28:34PM +0800, Qu Wenruo wrote:
> 
> 
> At 05/24/2017 12:24 PM, Eryu Guan wrote:
> > On Wed, May 24, 2017 at 08:22:25AM +0800, Qu Wenruo wrote:
> > > 
> > > 
> > > At 05/23/2017 07:13 PM, Eryu Guan wrote:
> > > > On Tue, May 23, 2017 at 04:02:05PM +0800, Qu Wenruo wrote:
> > > > > [BUG]
> > > > > If using MOUNT_OPTIONS="-o nodatasum" and btrfs to run genierc/142
> > > > > generic/143 and generic/154, it will cause false alert like:
> > > > > cp: failed to clone '/mnt/test/test-154/file2' from 
> > > > > '/mnt/test/test-154/file1': Invalid argument
> > > > 
> > > > MOUNT_OPTIONS is for scratch mount, and TEST_FS_MOUNT_OPTS is for test
> > > > dev mount, so I think setting TEST_FS_MOUNT_OPTS to "-o nodatasum"
> > > > should fix your problem.
> > > 
> > > Nope, the problem is the inconsistent of TEST_MNT setup.
> > 
> > It does fix the failure for me, did I miss anything?
> > 
> > # MOUNT_OPTIONS="-o nodatasum" TEST_FS_MOUNT_OPTS="-o nodatasum" ./check 
> > generic/142 generic/143 generic/154
> > FSTYP -- btrfs
> > PLATFORM  -- Linux/x86_64 dhcp-66-86-11 4.12.0-rc1
> > MKFS_OPTIONS  -- /dev/sda6
> > MOUNT_OPTIONS -- -o nodatasum -o context=system_u:object_r:root_t:s0 
> > /dev/sda6 /mnt/testarea/scratch
> > 
> > generic/142 2s ... 1s
> > generic/143  18s
> > generic/154  1s
> > Ran: generic/142 generic/143 generic/154
> > Passed all 3 tests
> > 
> 
> But if you only export MOUNT_OPTIONS, it will fail, due to the different
> mount options between test_cycle_mount().

That's correct. Sorry, I didn't make it clear in my first reply. I meant
that you should set both TEST_FS_MOUNT_OPTS and MOUNT_OPTIONS to
"-onodatasum", for both test dev and scratch dev.

> 
> To make it clear:
> If test mount follows TEST_FS_MOUNT_OPTS, then both the first mount and
> test_cycle_mount should follow TEST_FS_MOUNT_OPTS.

_test_mount does follow TEST_FS_MOUNT_OPTS, not MOUNT_OPTIONS, no matter
which mount it is.

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] fstests: common: Make _test_mount to include MOUNT_OPTIONS to allow consistent _test_cycle_mount

[Eryu Guan]

On Wed, May 24, 2017 at 08:22:25AM +0800, Qu Wenruo wrote:
> 
> 
> At 05/23/2017 07:13 PM, Eryu Guan wrote:
> > On Tue, May 23, 2017 at 04:02:05PM +0800, Qu Wenruo wrote:
> > > [BUG]
> > > If using MOUNT_OPTIONS="-o nodatasum" and btrfs to run genierc/142
> > > generic/143 and generic/154, it will cause false alert like:
> > > cp: failed to clone '/mnt/test/test-154/file2' from 
> > > '/mnt/test/test-154/file1': Invalid argument
> > 
> > MOUNT_OPTIONS is for scratch mount, and TEST_FS_MOUNT_OPTS is for test
> > dev mount, so I think setting TEST_FS_MOUNT_OPTS to "-o nodatasum"
> > should fix your problem.
> 
> Nope, the problem is the inconsistent of TEST_MNT setup.

It does fix the failure for me, did I miss anything?

# MOUNT_OPTIONS="-o nodatasum" TEST_FS_MOUNT_OPTS="-o nodatasum" ./check 
generic/142 generic/143 generic/154
FSTYP -- btrfs
PLATFORM  -- Linux/x86_64 dhcp-66-86-11 4.12.0-rc1
MKFS_OPTIONS  -- /dev/sda6
MOUNT_OPTIONS -- -o nodatasum -o context=system_u:object_r:root_t:s0 /dev/sda6 
/mnt/testarea/scratch

generic/142 2s ... 1s
generic/143  18s
generic/154  1s
Ran: generic/142 generic/143 generic/154
Passed all 3 tests

Thanks,
Eryu

> 
> As I described, the TEST_MNT is setup *with* MOUNT_OPTIONS for the 1st time,
> maybe by "check" script.
> 
> But _test_cycle_mount() uses TEST_FS_MOUNT_OPTS otherthan MOUNT_OPTIONS, and
> leads to the problem.
> 
> If using TEST_FS_MOUNT_OPTS, then 1st setup should also use
> TEST_FS_MOUNT_OPTS.
> 
> Thanks,
> Qu
> 
> > 
> > Yeah, MOUNT_OPTIONS and TEST_FS_MOUNT_OPTS (and almost all other global
> > variables) are not documented properly..
> > 
> > Thanks,
> > Eryu
> > 
> > > 
> > > [REASON]
> > > It is caused by _test_cycle_mount function, which unmount test device,
> > > but when trying to re-mount it again using _test_mount(), we don't pass
> > > $MOUNT_OPTIONS.
> > > 
> > > So this makes mount options differs between _test_cycle_mount().
> > > 
> > > And btrfs doesn't allow different csum flags between reflink source and
> > > destination inodes, so it returns -EINVAL for reflink operation.
> > > 
> > > [FIX]
> > > Fix it by passing $MOUNT_OPTIONS to _test_mount(), so that
> > > _test_cycle_mount() won't cause different mount options.
> > > So btrfs with "-o nodatasum" mount option can pass generic/14[23]
> > > and generic/154 without false alert.
> > > 
> > > Signed-off-by: Qu Wenruo <quwen...@cn.fujitsu.com>
> > > ---
> > >   common/rc | 3 ++-
> > >   1 file changed, 2 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/common/rc b/common/rc
> > > index ba215961..a591907c 100644
> > > --- a/common/rc
> > > +++ b/common/rc
> > > @@ -522,7 +522,8 @@ _test_mount()
> > >   return $?
> > >   fi
> > >   _test_options mount
> > > -_mount -t $FSTYP $TEST_OPTIONS $TEST_FS_MOUNT_OPTS 
> > > $SELINUX_MOUNT_OPTIONS $* $TEST_DEV $TEST_DIR
> > > +_mount -t $FSTYP $TEST_OPTIONS $TEST_FS_MOUNT_OPTS 
> > > $SELINUX_MOUNT_OPTIONS \
> > > + $MOUNT_OPTIONS $* $TEST_DEV $TEST_DIR
> > >   }
> > >   _test_unmount()
> > > -- 
> > > 2.13.0
> > > 
> > > 
> > > 
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe fstests" in
> > > the body of a message to majord...@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> > 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] fstests: common: Make _test_mount to include MOUNT_OPTIONS to allow consistent _test_cycle_mount

[Eryu Guan]

On Tue, May 23, 2017 at 04:02:05PM +0800, Qu Wenruo wrote:
> [BUG]
> If using MOUNT_OPTIONS="-o nodatasum" and btrfs to run genierc/142
> generic/143 and generic/154, it will cause false alert like:
> cp: failed to clone '/mnt/test/test-154/file2' from 
> '/mnt/test/test-154/file1': Invalid argument

MOUNT_OPTIONS is for scratch mount, and TEST_FS_MOUNT_OPTS is for test
dev mount, so I think setting TEST_FS_MOUNT_OPTS to "-o nodatasum"
should fix your problem.

Yeah, MOUNT_OPTIONS and TEST_FS_MOUNT_OPTS (and almost all other global
variables) are not documented properly..

Thanks,
Eryu

> 
> [REASON]
> It is caused by _test_cycle_mount function, which unmount test device,
> but when trying to re-mount it again using _test_mount(), we don't pass
> $MOUNT_OPTIONS.
> 
> So this makes mount options differs between _test_cycle_mount().
> 
> And btrfs doesn't allow different csum flags between reflink source and
> destination inodes, so it returns -EINVAL for reflink operation.
> 
> [FIX]
> Fix it by passing $MOUNT_OPTIONS to _test_mount(), so that
> _test_cycle_mount() won't cause different mount options.
> So btrfs with "-o nodatasum" mount option can pass generic/14[23]
> and generic/154 without false alert.
> 
> Signed-off-by: Qu Wenruo 
> ---
>  common/rc | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/common/rc b/common/rc
> index ba215961..a591907c 100644
> --- a/common/rc
> +++ b/common/rc
> @@ -522,7 +522,8 @@ _test_mount()
>  return $?
>  fi
>  _test_options mount
> -_mount -t $FSTYP $TEST_OPTIONS $TEST_FS_MOUNT_OPTS 
> $SELINUX_MOUNT_OPTIONS $* $TEST_DEV $TEST_DIR
> +_mount -t $FSTYP $TEST_OPTIONS $TEST_FS_MOUNT_OPTS 
> $SELINUX_MOUNT_OPTIONS \
> + $MOUNT_OPTIONS $* $TEST_DEV $TEST_DIR
>  }
>  
>  _test_unmount()
> -- 
> 2.13.0
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 6/6] fstests: regression test for nocsum buffered read's repair

[Eryu Guan]

On Tue, May 09, 2017 at 11:56:11AM -0600, Liu Bo wrote:
> This is to test whether buffered read retry-repair code is able to work in
> raid1 case as expected.
> 
> Please note that without checksum, btrfs doesn't know if the data used to
> repair is correct, so repair is more of resync which makes sure that both
> of the copy has the same content.
> 
> Commit 20a7db8ab3f2 ("btrfs: add dummy callback for readpage_io_failed and 
> drop
> checks") introduced the regression.
> 
> The upstream fix is
>   Btrfs: bring back repair during read

btrfs/14[1-3] all could refer to the upstream patch along with its
commit id.

> 
> Signed-off-by: Liu Bo 
> ---
>  tests/btrfs/143 | 155 
> 
>  tests/btrfs/143.out |  39 +
>  tests/btrfs/group   |   1 +
>  3 files changed, 195 insertions(+)
>  create mode 100755 tests/btrfs/143
>  create mode 100644 tests/btrfs/143.out
> 
> diff --git a/tests/btrfs/143 b/tests/btrfs/143
> new file mode 100755
> index 000..5263e78
> --- /dev/null
> +++ b/tests/btrfs/143
> @@ -0,0 +1,155 @@
> +#! /bin/bash
> +# FS QA Test 143
> +#
> +# Regression test for btrfs buffered read's repair during read without 
> checksum.
> +#
> +# This is to test whether buffered read retry-repair code is able to work in
> +# raid1 case as expected.
> +#
> +# Please note that without checksum, btrfs doesn't know if the data used to
> +# repair is correct, so repair is more of resync which makes sure that both
> +# of the copy has the same content.
> +#
> +# Commit 20a7db8ab3f2 ("btrfs: add dummy callback for readpage_io_failed and 
> drop
> +# checks") introduced the regression.
> +#
> +# The upstream fix is
> +#Btrfs: bring back repair during read
> +#
> +#---
> +# Copyright (c) 2017 Liu Bo.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch_dev_pool 2

btrfs/142 and btrfs/143 need a "_require_fail_make_request"

> +
> +_require_btrfs_command inspect-internal dump-tree
> +_require_command "$FILEFRAG_PROG" filefrag
> +
> +_check_repair()
> +{
> + filter=${1:-cat}
> + dmesg | tac | sed -ne "0,\#run fstests $seqnum at $date_time#p" | tac | 
> $filter | grep -q -e "read error corrected"
> + if [ $? -eq 0 ]; then
> + echo 1
> + else
> + echo 0
> + fi
> +}

This function can be removed.

Thanks,
Eryu

> +
> +get_physical()
> +{
> +# $1 is logical address
> +# print chunk tree and find devid 2 which is $SCRATCH_DEV
> +$BTRFS_UTIL_PROG inspect-internal dump-tree -t 3 $SCRATCH_DEV | \
> + grep $1 -A 6 | awk '($1 ~ /stripe/ && $3 ~ /devid/ && $4 ~ /1/) { print 
> $6 }'
> +}
> +
> +SYSFS_BDEV=`_sysfs_dev $SCRATCH_DEV`
> +
> +start_fail()
> +{
> + echo 100 > $DEBUGFS_MNT/fail_make_request/probability
> + echo 4 > $DEBUGFS_MNT/fail_make_request/times
> + echo 0 > $DEBUGFS_MNT/fail_make_request/verbose
> + echo 1 > $SYSFS_BDEV/make-it-fail
> +}
> +
> +stop_fail()
> +{
> + echo 0 > $DEBUGFS_MNT/fail_make_request/probability
> + echo 0 > $DEBUGFS_MNT/fail_make_request/times
> + echo 0 > $SYSFS_BDEV/make-it-fail
> +}
> +
> +_scratch_dev_pool_get 2
> +# step 1, create a raid1 btrfs which contains one 128k file.
> +echo "step 1..mkfs.btrfs" >>$seqres.full
> +
> +mkfs_opts="-d raid1 -b 1G"
> +_scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1
> +
> +# -o nospace_cache makes sure data is written to the start position of the 
> data
> +# chunk
> +_scratch_mount -o nospace_cache,nodatasum
> +
> +$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" "$SCRATCH_MNT/foobar" 
> | _filter_xfs_io
> +
> +# 

Re: [PATCH 3/6] fstests: regression test for btrfs dio read repair

[Eryu Guan]

On Tue, May 09, 2017 at 11:56:08AM -0600, Liu Bo wrote:
> This case tests whether dio read can repair the bad copy if we have
> a good copy.
> 
> Commit 2dabb3248453 ("Btrfs: Direct I/O read: Work on sectorsized blocks")
> introduced the regression.
> 
> The upstream fix is
>   Btrfs: fix invalid dereference in btrfs_retry_endio

I noticed this is in upstream now, you can refer to it along with hash
tag too.

> 
> Signed-off-by: Liu Bo 
> ---
>  tests/btrfs/140 | 115 
> 
>  tests/btrfs/140.out |  39 ++
>  tests/btrfs/group   |   1 +
>  3 files changed, 155 insertions(+)
>  create mode 100755 tests/btrfs/140
>  create mode 100644 tests/btrfs/140.out
> 
> diff --git a/tests/btrfs/140 b/tests/btrfs/140
> new file mode 100755
> index 000..09a9939
> --- /dev/null
> +++ b/tests/btrfs/140
> @@ -0,0 +1,115 @@
> +#! /bin/bash
> +# FS QA Test 140
> +#
> +# Regression test for btrfs DIO read's repair during read.
> +#
> +# Commit 2dabb3248453 ("Btrfs: Direct I/O read: Work on sectorsized blocks")
> +# introduced the regression.
> +# The upstream fix is
> +#Btrfs: fix invalid dereference in btrfs_retry_endio
> +#
> +#---
> +# Copyright (c) 2017 Liu Bo.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch_dev_pool 2
> +
> +_require_btrfs_command inspect-internal dump-tree
> +_require_command "$FILEFRAG_PROG" filefrag
> +_require_odirect
> +
> +get_physical()
> +{
> + # $1 is logical address
> + # print chunk tree and find devid 2 which is $SCRATCH_DEV
> + $BTRFS_UTIL_PROG inspect-internal dump-tree -t 3 $SCRATCH_DEV | \
> + grep $1 -A 6 | awk '($1 ~ /stripe/ && $3 ~ /devid/ && $4 ~ /1/) { print 
> $6 }'
> +}
> +
> +_scratch_dev_pool_get 2
> +# step 1, create a raid1 btrfs which contains one 128k file.
> +echo "step 1..mkfs.btrfs" >>$seqres.full
> +
> +mkfs_opts="-d raid1 -b 1G"
> +_scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1
> +
> +# -o nospace_cache makes sure data is written to the start position of the 
> data
> +# chunk
> +_scratch_mount -o nospace_cache
> +
> +$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" "$SCRATCH_MNT/foobar" 
> | _filter_xfs_io
> +
> +# step 2, corrupt the first 64k of one copy (on SCRATCH_DEV which is the 
> first
> +# one in $SCRATCH_DEV_POOL
> +echo "step 2..corrupt file extent" >>$seqres.full
> +
> +${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar >> $seqres.full
> +logical_in_btrfs=`${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar | _filter_filefrag 
> | cut -d '#' -f 1`
> +physical_on_scratch=`get_physical ${logical_in_btrfs}`
> +
> +_scratch_unmount
> +$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 64K $physical_on_scratch 64K" 
> $SCRATCH_DEV | _filter_xfs_io
> +
> +_scratch_mount
> +
> +# step 3, 128k dio read (this read can repair bad copy)
> +echo "step 3..repair the bad copy" >>$seqres.full
> +
> +# since raid1 consists of two copies, and the following read may read the 
> good
> +# copy directly, so lets loop 10 times here and discard output that dio reads
> +# give
> +for i in `seq 1 10`; do
> + $XFS_IO_PROG -d -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" > 
> /dev/null
> + _get_current_dmesg | grep -q -e "csum failed" && break
> +done

Half of the time I got test failure because pread from SCRATCH_DEV read
0xbb instead of 0xaa on v4.11 kernel (bug should be fixed there), tested
on two different hosts and could hit failure on both hosts.

Similar failure happened to all the 4 tests randomly. I thought it was
because "csum failed" was never hit, so I tried a "while true; do" loop,
and that did fix the btrfs/140 failure for me, 

Re: [PATCH 2/6] fstests: add _get_current_dmesg

[Eryu Guan]

On Tue, May 09, 2017 at 11:56:07AM -0600, Liu Bo wrote:
> _get_current_dmesg can be used to grep customized pattern.
> 
> Signed-off-by: Liu Bo 

I can't apply this patch on top of current master, perhaps it needs a
rebase :)

> ---
>  common/rc | 9 +++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
> 
> diff --git a/common/rc b/common/rc
> index 78a2101..111ed69 100644
> --- a/common/rc
> +++ b/common/rc
> @@ -3215,6 +3215,12 @@ _get_device_size()
>   grep `_short_dev $1` /proc/partitions | awk '{print $3}'
>  }
>  
> +# get current dmesg log
> +_get_current_dmesg()
> +{
> + dmesg | tac | sed -ne "0,\#run fstests $seqnum at $date_time#p" | tac
> +}
> +
>  # check dmesg log for WARNING/Oops/etc.
>  _check_dmesg()
>  {
> @@ -3230,8 +3236,7 @@ _check_dmesg()
>  
>   # search the dmesg log of last run of $seqnum for possible failures
>   # use sed \cregexpc address type, since $seqnum contains "/"

This second line of comment should go to _get_current_dmesg.

Thanks,
Eryu

> - dmesg | tac | sed -ne "0,\#run fstests $seqnum at $date_time#p" | \
> - tac | $filter >$seqres.dmesg
> + _get_current_dmesg | $filter >$seqres.dmesg
>   grep -q -e "kernel BUG at" \
>-e "WARNING:" \
>-e "BUG:" \
> -- 
> 2.5.0
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3] fstests: regression test for btrfs dio read repair

[Eryu Guan]

On Fri, Apr 28, 2017 at 11:25:52AM -0600, Liu Bo wrote:
> This case tests whether dio read can repair the bad copy if we have
> a good copy.
> 
> Commit 2dabb3248453 ("Btrfs: Direct I/O read: Work on sectorsized blocks")
> introduced the regression.
> 
> The upstream fix is
>   Btrfs: fix invalid dereference in btrfs_retry_endio
> 
> Signed-off-by: Liu Bo 

Sorry for the late review, and many thanks to Filipe's reviews! I agreed
with Filipe that the common helpers can be placed in common/btrfs and/or
common/rc files.

Some thoughts inline.

> ---
> v2: - Add regression commit and the fix to the description
> - Use btrfs inspect-internal dump-tree to get rid of the dependence 
> btrfs-map-logical
> - Add comments in several places
> 
> v3: - Add 'mkfs -b 1G' to limit filesystem size to 2G in raid1 profile so that
>   we get a consistent output.
> 
>  tests/btrfs/140 | 167 
> 
>  tests/btrfs/140.out |  39 
>  tests/btrfs/group   |   1 +
>  3 files changed, 207 insertions(+)
>  create mode 100755 tests/btrfs/140
>  create mode 100644 tests/btrfs/140.out
> 
> diff --git a/tests/btrfs/140 b/tests/btrfs/140
> new file mode 100755
> index 000..dcd8807
> --- /dev/null
> +++ b/tests/btrfs/140
> @@ -0,0 +1,167 @@
> +#! /bin/bash
> +# FS QA Test 140
> +#
> +# Regression test for btrfs DIO read's repair during read.
> +#
> +# Commit 2dabb3248453 ("Btrfs: Direct I/O read: Work on sectorsized blocks")
> +# introduced the regression.
> +# The upstream fix is
> +#Btrfs: fix invalid dereference in btrfs_retry_endio
> +#
> +#---
> +# Copyright (c) 2017 Liu Bo.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch_dev_pool 2
> +
> +_require_btrfs_command inspect-internal dump-tree
> +_require_command "$FILEFRAG_PROG" filefrag
> +_require_odirect
> +
> +# helpe to convert 'file offset' to btrfs logical offset
> +FILEFRAG_FILTER='
> + if (/blocks? of (\d+) bytes/) {
> + $blocksize = $1;
> + next
> + }
> + ($ext, $logical, $physical, $length) =
> + (/^\s*(\d+):\s+(\d+)..\s+\d+:\s+(\d+)..\s+\d+:\s+(\d+):/)
> + or next;
> + ($flags) = /.*:\s*(\S*)$/;
> + print $physical * $blocksize, "#",
> +   $length * $blocksize, "#",
> +   $logical * $blocksize, "#",
> +   $flags, " "'
  ^^^ "\n" so one extent per line?

This can be embedded in the filter function, like what _filter_mkfs
does.

> +
> +# this makes filefrag output script readable by using a perl helper.
> +# output is one extent per line, with three numbers separated by '#'
> +# the numbers are: physical, length, logical (all in bytes)
> +# sample output: "1234#10#5678" -> physical 1234, length 10, logical 5678
> +_filter_extents()

Global functions start with underscore, local functions don't.

> +{
> + tee -a $seqres.full | $PERL_PROG -ne "$FILEFRAG_FILTER"

I don't think this tee belongs here, see below.

> +}
> +
> +_check_file_extents()
> +{
> + cmd="filefrag -v $1"

$FILEFRAG_PROG

> + echo "# $cmd" >> $seqres.full

Just call filefrag -v again to dump the filefrag output to $seqres.full

> + out=`$cmd | _filter_extents`
> + if [ -z "$out" ]; then
> + return 1
> + fi
> + echo "after filter: $out" >> $seqres.full
> + echo $out
> + return 0
> +}

Hmm, seems that all you want from all these filters is the logical byte
of the first extent in the file. How about converting
_check_file_extents to _filter_filefrag and put it in common/filter?
And _get_physical can also be converted to 

Re: [PATCH] fstests: introduce btrfs-map-logical

[Eryu Guan]

On Wed, Apr 12, 2017 at 02:52:23PM +0200, David Sterba wrote:
> > > I understand that we need to do corruption so that we can test if the 
> > > repair works, but I'm not sure if the output format will change, or if 
> > > the program will get replace by "btrfs inspect-internal" group.
> > 
> > In the long-term it will be repleaced, but there's no ETA.
> 
> Possibly, if fstests maintainer agrees, we can add btrfs-map-logical to
> fstests. It's small and uses headers from libbtrfs, so this would become
> a new dependency but I believe is still bearable.

IMHO, I think the ability to poke btrfs internal really should be
provided by btrfs-progs package and maintained by btrfs community.
fstests provides some fs-independent c helpers to assist testing, but
not necessarily needs to "understand" filesystem internals.

For historical reason, building fstests requires xfsprogs development
headers, we'd better not introduce new fs-specific dependencies.

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Does btrfs get nlink on directories wrong? -- was Re: [PATCH 2/4] xfstests: Add first statx test [ver #5]

[Eryu Guan]

On Wed, Apr 05, 2017 at 03:32:30PM +0300, Amir Goldstein wrote:
> On Wed, Apr 5, 2017 at 3:30 PM, David Sterba  wrote:
> > On Wed, Apr 05, 2017 at 11:53:41AM +0100, David Howells wrote:
> >> I've added a test to xfstests that exercises the new statx syscall.  
> >> However,
> >> it fails on btrfs:
> >>
> >>  Test statx on a directory
> >> +[!] stx_nlink differs, 1 != 2
> >> +Failed
> >> +stat_test failed
> >>
> >> because a new directory it creates has an nlink of 1, not 2.  Is this a 
> >> case
> >> of my making an incorrect assumption or is it an fs bug?
> >
> > Afaik nlink == 1 means that there's no accounting of subdirectories, and
> > it's a valid value. The 'find' utility can use nlink to optimize
> > directory traversal but otherwise I'm not aware of other usage.
> >
> > All directories in btrfs have nlink == 1.
> 
> FYI,
> 
> Overlayfs uses nlink = 1 for merge dirs to silence 'find' et al.
> Ext4 uses nlink = 1 for directories with more than 32K subdirs
> (EXT4_FEATURE_RO_COMPAT_DIR_NLINK).
> 
> But in both those fs newly created directories will have nlink = 2.

Is there a conclusion on this? Seems the test should be updated
accordingly?

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/3] common/rc: test that xfs_io's falloc command supports specific flags

[Eryu Guan]

On Tue, Apr 04, 2017 at 07:34:29AM +0100, fdman...@kernel.org wrote:
> From: Filipe Manana <fdman...@suse.com>
> 
> For example NFS 4.2 supports fallocate but it does not support its
> KEEP_SIZE flag, so we want to skip tests that use fallocate with that
> flag on filesystems that don't support it.
> 
> Suggested-by: Eryu Guan <eg...@redhat.com>
> Signed-off-by: Filipe Manana <fdman...@suse.com>
> ---
>  common/rc | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/common/rc b/common/rc
> index e1ab2c6..3d0f089 100644
> --- a/common/rc
> +++ b/common/rc
> @@ -2021,8 +2021,8 @@ _require_xfs_io_command()
>   "chproj")
>   testio=`$XFS_IO_PROG -F -f -c "chproj 0" $testfile 2>&1`
>   ;;
> - "falloc" )
> - testio=`$XFS_IO_PROG -F -f -c "falloc 0 1m" $testfile 2>&1`
> + "falloc*" )
> + testio=`$XFS_IO_PROG -F -f -c "$command 0 1m" $testfile 2>&1`

Sorry, I was wrong about this. It would break the subsequent
$XFS_IO_PROG -c "help $command" | grep ... command if another $param is
specified. Seems adding $param to falloc command is the right way, as
what Darrick did to fiemap in his new test.

-   testio=`$XFS_IO_PROG -F -f -c "falloc 0 1m" $testfile 2>&1`
+   testio=`$XFS_IO_PROG -F -f -c "falloc $param 0 1m" $testfile 
2>&1`

Do you mind me updating these three patches accordingly? Or can you send
out new version if you like?

Thanks! And sorry again!

Eryu

P.S. I'm thinking of converting all the case switches (except the
default one) in _require_xfs_io_command() to actually run the $command
with $param, and doing other cleanups, but that won't block this patch
and I can do it in another patch.

>   ;;
>   "fpunch" | "fcollapse" | "zero" | "fzero" | "finsert" | "funshare")
>   testio=`$XFS_IO_PROG -F -f -c "pwrite 0 20k" -c "fsync" \
> -- 
> 2.7.0.rc3
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] fstests: generic: Check if cycle mount and sleep can affect fiemap result

[Eryu Guan]

On Thu, Apr 06, 2017 at 11:28:01AM -0500, Eric Sandeen wrote:
> On 4/6/17 11:26 AM, Theodore Ts'o wrote:
> > On Wed, Apr 05, 2017 at 10:35:26AM +0800, Eryu Guan wrote:
> >>
> >> Test fails with ext3/2 when driving with ext4 driver, fiemap changed
> >> after umount/mount cycle, then changed back to original result after
> >> sleeping some time. An ext4 bug? (cc'ed linux-ext4 list.)
> > 
> > I haven't had time to look at this, but I'm not sure this test is a
> > reasonable one on the face of it.
> > 
> > A file system may choose to optimize a file's extent tree for whatever
> > reason it wants, whenever it wants, including on an unmount --- and
> > that would not be an invalid thing to do.  So to have an xfstests that
> > causes a test failure if a file system were to, say, do some cleanup
> > at mount or unmount time, or when the file is next opened, to merge
> > adjacent extents together (and hence change what is returned by
> > FIEMAP) might be strange, or even weird --- but is this any of user
> > space's business?  Or anything we want to enforce as wrong wrong wrong
> > by xfstests?

So I was asking for a review from ext4 side instead of queuing it for
next xfstests update :)

> 
> I had the same question.  If the exact behavior isn't defined anywhere,
> I don't know what we can be testing, TBH.

Agreed, I was about to ask for the expected behavior today if there was
no new review comments on this patch.

Thanks for the comments and review!

Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html