Re: [PATCH v2] fstests: btrfs/149 make it sectorsize independent
On 09/10/2018 03:59 PM, Nikolay Borisov wrote: On 10.09.2018 06:25, Anand Jain wrote: Originally this test case was designed to work with only 4K sectorsize. Now enhance it to work with any sector sizes and makes the following changes: Output file not to contain any traces of sector size. Use max_inline=0 mount option so that it meets the requisite of non inline regular extent. Don't log the md5sum results to the output file as the data size vary by the sectorsize. Signed-off-by: Anand Jain You also need to amend the description of the test, currently it says: # Test that an incremental send/receive operation will not fail when the # destination filesystem has compression enabled and the source filesystem # has a 4K extent at a file offset 0 that is not compressed and that is # shared. The '4k' need to be replaced by 'sector sized extent' Right. Will fix. --- v1->v2: rename _scratch_sectorsize() to _scratch_btrfs_sectorsize() add _require_btrfs_command inspect-internal dump-super common/btrfs| 7 +++ common/filter | 5 + tests/btrfs/149 | 30 +- tests/btrfs/149.out | 12 ++-- 4 files changed, 39 insertions(+), 15 deletions(-) diff --git a/common/btrfs b/common/btrfs index 79c687f73376..26dc0bb9600f 100644 --- a/common/btrfs +++ b/common/btrfs @@ -367,3 +367,10 @@ _run_btrfs_balance_start() run_check $BTRFS_UTIL_PROG balance start $bal_opt $* } + +#return the sector size of the btrfs scratch fs +_scratch_btrfs_sectorsize() +{ + $BTRFS_UTIL_PROG inspect-internal dump-super $SCRATCH_DEV |\ + grep sectorsize | awk '{print $2}' +} diff --git a/common/filter b/common/filter index 3965c2eb752b..e87740ddda3f 100644 --- a/common/filter +++ b/common/filter @@ -271,6 +271,11 @@ _filter_xfs_io_pages_modified() _filter_xfs_io_units_modified "Page" $PAGE_SIZE } +_filter_xfs_io_numbers() +{ +_filter_xfs_io | sed -E 's/[0-9]+//g' +} + _filter_test_dir() { # TEST_DEV may be a prefix of TEST_DIR (e.g. /mnt, /mnt/ovl-mnt) diff --git a/tests/btrfs/149 b/tests/btrfs/149 index 3e955a305e0f..9b0f00b0ed81 100755 --- a/tests/btrfs/149 +++ b/tests/btrfs/149 @@ -36,6 +36,7 @@ _require_test _require_scratch _require_scratch_reflink _require_odirect +_require_btrfs_command inspect-internal dump-super send_files_dir=$TEST_DIR/btrfs-test-$seq @@ -44,21 +45,27 @@ rm -fr $send_files_dir mkdir $send_files_dir _scratch_mkfs >>$seqres.full 2>&1 -_scratch_mount "-o compress" +# On 64K pagesize systems the compression is more efficient, so max_inline +# helps to create regular (non inline) extent irrespective of the final +# write size. +_scratch_mount "-o compress -o max_inline=0" # Write to our file using direct IO, so that this way the write ends up not # getting compressed, that is, we get a regular extent which is neither # inlined nor compressed. # Alternatively, we could have mounted the fs without compression enabled, # which would result as well in an uncompressed regular extent. -$XFS_IO_PROG -f -d -c "pwrite -S 0xab 0 4K" $SCRATCH_MNT/foobar | _filter_xfs_io +sectorsize=$(_scratch_btrfs_sectorsize) +$XFS_IO_PROG -f -d -c "pwrite -S 0xab 0 $sectorsize" $SCRATCH_MNT/foobar |\ + _filter_xfs_io_numbers $BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT \ $SCRATCH_MNT/mysnap1 > /dev/null # Clone the regular (not inlined) extent. -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foobar 0 8K 4K" $SCRATCH_MNT/foobar \ - | _filter_xfs_io +$XFS_IO_PROG -c \ + "reflink $SCRATCH_MNT/foobar 0 $((2 * $sectorsize)) $sectorsize" \ + $SCRATCH_MNT/foobar | _filter_xfs_io_numbers $BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT \ $SCRATCH_MNT/mysnap2 > /dev/null @@ -76,21 +83,26 @@ $BTRFS_UTIL_PROG send -p $SCRATCH_MNT/mysnap1 -f $send_files_dir/2.snap \ $SCRATCH_MNT/mysnap2 2>&1 >/dev/null | _filter_scratch It's not visible in this hunk but before we do the send -p there are also references to an 8k offsset, this needs to be reworded with the more generic 'sector sized' language Thanks will fix that too. echo "File digests in the original filesystem:" No point in echo'ing the above string given you no longer echo the checksums. OK. -md5sum $SCRATCH_MNT/mysnap1/foobar | _filter_scratch -md5sum $SCRATCH_MNT/mysnap2/foobar | _filter_scratch +sum_src_snap1=$(md5sum $SCRATCH_MNT/mysnap1/foobar | awk '{print $1}') +sum_src_snap2=$(md5sum $SCRATCH_MNT/mysnap2/foobar | awk '{print $1}') +echo "src checksum created" I think it will be worth it outputting the checksums in the .full log since if the test is going to fail for someone likely it will be reproducible for their particular config and so it pays to have the checksums in the full log. Yep will have md5sum logged to the full log. Thanks, Anand # Now recreate the filesystem by receiving both send
Re: [PATCH v2] fstests: btrfs/149 make it sectorsize independent
On 10.09.2018 06:25, Anand Jain wrote: > Originally this test case was designed to work with only 4K sectorsize. > Now enhance it to work with any sector sizes and makes the following > changes: > Output file not to contain any traces of sector size. > Use max_inline=0 mount option so that it meets the requisite of non inline > regular extent. > Don't log the md5sum results to the output file as the data size vary by > the sectorsize. > > Signed-off-by: Anand Jain You also need to amend the description of the test, currently it says: # Test that an incremental send/receive operation will not fail when the # destination filesystem has compression enabled and the source filesystem # has a 4K extent at a file offset 0 that is not compressed and that is # shared. The '4k' need to be replaced by 'sector sized extent' > --- > v1->v2: rename _scratch_sectorsize() to _scratch_btrfs_sectorsize() > add _require_btrfs_command inspect-internal dump-super > > common/btrfs| 7 +++ > common/filter | 5 + > tests/btrfs/149 | 30 +- > tests/btrfs/149.out | 12 ++-- > 4 files changed, 39 insertions(+), 15 deletions(-) > > diff --git a/common/btrfs b/common/btrfs > index 79c687f73376..26dc0bb9600f 100644 > --- a/common/btrfs > +++ b/common/btrfs > @@ -367,3 +367,10 @@ _run_btrfs_balance_start() > > run_check $BTRFS_UTIL_PROG balance start $bal_opt $* > } > + > +#return the sector size of the btrfs scratch fs > +_scratch_btrfs_sectorsize() > +{ > + $BTRFS_UTIL_PROG inspect-internal dump-super $SCRATCH_DEV |\ > + grep sectorsize | awk '{print $2}' > +} > diff --git a/common/filter b/common/filter > index 3965c2eb752b..e87740ddda3f 100644 > --- a/common/filter > +++ b/common/filter > @@ -271,6 +271,11 @@ _filter_xfs_io_pages_modified() > _filter_xfs_io_units_modified "Page" $PAGE_SIZE > } > > +_filter_xfs_io_numbers() > +{ > +_filter_xfs_io | sed -E 's/[0-9]+//g' > +} > + > _filter_test_dir() > { > # TEST_DEV may be a prefix of TEST_DIR (e.g. /mnt, /mnt/ovl-mnt) > diff --git a/tests/btrfs/149 b/tests/btrfs/149 > index 3e955a305e0f..9b0f00b0ed81 100755 > --- a/tests/btrfs/149 > +++ b/tests/btrfs/149 > @@ -36,6 +36,7 @@ _require_test > _require_scratch > _require_scratch_reflink > _require_odirect > +_require_btrfs_command inspect-internal dump-super > > send_files_dir=$TEST_DIR/btrfs-test-$seq > > @@ -44,21 +45,27 @@ rm -fr $send_files_dir > mkdir $send_files_dir > > _scratch_mkfs >>$seqres.full 2>&1 > -_scratch_mount "-o compress" > +# On 64K pagesize systems the compression is more efficient, so max_inline > +# helps to create regular (non inline) extent irrespective of the final > +# write size. > +_scratch_mount "-o compress -o max_inline=0" > > # Write to our file using direct IO, so that this way the write ends up not > # getting compressed, that is, we get a regular extent which is neither > # inlined nor compressed. > # Alternatively, we could have mounted the fs without compression enabled, > # which would result as well in an uncompressed regular extent. > -$XFS_IO_PROG -f -d -c "pwrite -S 0xab 0 4K" $SCRATCH_MNT/foobar | > _filter_xfs_io > +sectorsize=$(_scratch_btrfs_sectorsize) > +$XFS_IO_PROG -f -d -c "pwrite -S 0xab 0 $sectorsize" $SCRATCH_MNT/foobar |\ > + _filter_xfs_io_numbers > > $BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT \ > $SCRATCH_MNT/mysnap1 > /dev/null > > # Clone the regular (not inlined) extent. > -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foobar 0 8K 4K" $SCRATCH_MNT/foobar \ > - | _filter_xfs_io > +$XFS_IO_PROG -c \ > + "reflink $SCRATCH_MNT/foobar 0 $((2 * $sectorsize)) $sectorsize" \ > + $SCRATCH_MNT/foobar | _filter_xfs_io_numbers > > $BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT \ > $SCRATCH_MNT/mysnap2 > /dev/null > @@ -76,21 +83,26 @@ $BTRFS_UTIL_PROG send -p $SCRATCH_MNT/mysnap1 -f > $send_files_dir/2.snap \ >$SCRATCH_MNT/mysnap2 2>&1 >/dev/null | _filter_scratch > It's not visible in this hunk but before we do the send -p there are also references to an 8k offsset, this needs to be reworded with the more generic 'sector sized' language > echo "File digests in the original filesystem:" No point in echo'ing the above string given you no longer echo the checksums. > -md5sum $SCRATCH_MNT/mysnap1/foobar | _filter_scratch > -md5sum $SCRATCH_MNT/mysnap2/foobar | _filter_scratch > +sum_src_snap1=$(md5sum $SCRATCH_MNT/mysnap1/foobar | awk '{print $1}') > +sum_src_snap2=$(md5sum $SCRATCH_MNT/mysnap2/foobar | awk '{print $1}') > +echo "src checksum created" I think it will be worth it outputting the checksums in the .full log since if the test is going to fail for someone likely it will be reproducible for their particular config and so it pays to have the checksums in the full log. > > # Now recreate the filesystem by receiving both send streams and verify we > get >
[PATCH v2] fstests: btrfs/149 make it sectorsize independent
Originally this test case was designed to work with only 4K sectorsize. Now enhance it to work with any sector sizes and makes the following changes: Output file not to contain any traces of sector size. Use max_inline=0 mount option so that it meets the requisite of non inline regular extent. Don't log the md5sum results to the output file as the data size vary by the sectorsize. Signed-off-by: Anand Jain --- v1->v2: rename _scratch_sectorsize() to _scratch_btrfs_sectorsize() add _require_btrfs_command inspect-internal dump-super common/btrfs| 7 +++ common/filter | 5 + tests/btrfs/149 | 30 +- tests/btrfs/149.out | 12 ++-- 4 files changed, 39 insertions(+), 15 deletions(-) diff --git a/common/btrfs b/common/btrfs index 79c687f73376..26dc0bb9600f 100644 --- a/common/btrfs +++ b/common/btrfs @@ -367,3 +367,10 @@ _run_btrfs_balance_start() run_check $BTRFS_UTIL_PROG balance start $bal_opt $* } + +#return the sector size of the btrfs scratch fs +_scratch_btrfs_sectorsize() +{ + $BTRFS_UTIL_PROG inspect-internal dump-super $SCRATCH_DEV |\ + grep sectorsize | awk '{print $2}' +} diff --git a/common/filter b/common/filter index 3965c2eb752b..e87740ddda3f 100644 --- a/common/filter +++ b/common/filter @@ -271,6 +271,11 @@ _filter_xfs_io_pages_modified() _filter_xfs_io_units_modified "Page" $PAGE_SIZE } +_filter_xfs_io_numbers() +{ +_filter_xfs_io | sed -E 's/[0-9]+//g' +} + _filter_test_dir() { # TEST_DEV may be a prefix of TEST_DIR (e.g. /mnt, /mnt/ovl-mnt) diff --git a/tests/btrfs/149 b/tests/btrfs/149 index 3e955a305e0f..9b0f00b0ed81 100755 --- a/tests/btrfs/149 +++ b/tests/btrfs/149 @@ -36,6 +36,7 @@ _require_test _require_scratch _require_scratch_reflink _require_odirect +_require_btrfs_command inspect-internal dump-super send_files_dir=$TEST_DIR/btrfs-test-$seq @@ -44,21 +45,27 @@ rm -fr $send_files_dir mkdir $send_files_dir _scratch_mkfs >>$seqres.full 2>&1 -_scratch_mount "-o compress" +# On 64K pagesize systems the compression is more efficient, so max_inline +# helps to create regular (non inline) extent irrespective of the final +# write size. +_scratch_mount "-o compress -o max_inline=0" # Write to our file using direct IO, so that this way the write ends up not # getting compressed, that is, we get a regular extent which is neither # inlined nor compressed. # Alternatively, we could have mounted the fs without compression enabled, # which would result as well in an uncompressed regular extent. -$XFS_IO_PROG -f -d -c "pwrite -S 0xab 0 4K" $SCRATCH_MNT/foobar | _filter_xfs_io +sectorsize=$(_scratch_btrfs_sectorsize) +$XFS_IO_PROG -f -d -c "pwrite -S 0xab 0 $sectorsize" $SCRATCH_MNT/foobar |\ + _filter_xfs_io_numbers $BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT \ $SCRATCH_MNT/mysnap1 > /dev/null # Clone the regular (not inlined) extent. -$XFS_IO_PROG -c "reflink $SCRATCH_MNT/foobar 0 8K 4K" $SCRATCH_MNT/foobar \ - | _filter_xfs_io +$XFS_IO_PROG -c \ + "reflink $SCRATCH_MNT/foobar 0 $((2 * $sectorsize)) $sectorsize" \ + $SCRATCH_MNT/foobar | _filter_xfs_io_numbers $BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT \ $SCRATCH_MNT/mysnap2 > /dev/null @@ -76,21 +83,26 @@ $BTRFS_UTIL_PROG send -p $SCRATCH_MNT/mysnap1 -f $send_files_dir/2.snap \ $SCRATCH_MNT/mysnap2 2>&1 >/dev/null | _filter_scratch echo "File digests in the original filesystem:" -md5sum $SCRATCH_MNT/mysnap1/foobar | _filter_scratch -md5sum $SCRATCH_MNT/mysnap2/foobar | _filter_scratch +sum_src_snap1=$(md5sum $SCRATCH_MNT/mysnap1/foobar | awk '{print $1}') +sum_src_snap2=$(md5sum $SCRATCH_MNT/mysnap2/foobar | awk '{print $1}') +echo "src checksum created" # Now recreate the filesystem by receiving both send streams and verify we get # the same file content that the original filesystem had. _scratch_unmount _scratch_mkfs >>$seqres.full 2>&1 -_scratch_mount "-o compress" +_scratch_mount "-o compress,max_inline=0" $BTRFS_UTIL_PROG receive -f $send_files_dir/1.snap $SCRATCH_MNT > /dev/null $BTRFS_UTIL_PROG receive -f $send_files_dir/2.snap $SCRATCH_MNT > /dev/null echo "File digests in the new filesystem:" -md5sum $SCRATCH_MNT/mysnap1/foobar | _filter_scratch -md5sum $SCRATCH_MNT/mysnap2/foobar | _filter_scratch +sum_dest_snap1=$(md5sum $SCRATCH_MNT/mysnap1/foobar | awk '{print $1}') +sum_dest_snap2=$(md5sum $SCRATCH_MNT/mysnap2/foobar | awk '{print $1}') +echo "dest checksum created" + +[[ $sum_src_snap1 == $sum_dest_snap1 ]] && echo "src and dest checksum matched" +[[ $sum_src_snap2 == $sum_dest_snap2 ]] && echo "src and dest checksum matched" status=0 exit diff --git a/tests/btrfs/149.out b/tests/btrfs/149.out index 303de928d35a..6ba251799ff2 100644 --- a/tests/btrfs/149.out +++ b/tests/btrfs/149.out @@ -1,14 +1,14 @@ QA output created by 149 -wrote 4096/4096 bytes at offset 0 +wrote