[PATCH v5 2/2] generic: make 17[1-4] work well when btrfs compression is enabled

[Wang Xiaoguang]

When enabling btrfs compression, original codes can not fill fs
correctly, here we introduce _fill_fs() in common/rc, which'll keep
creating and writing files until enospc error occurs. Note _fill_fs
is copied from tests/generic/256, but with some minor modifications.

Signed-off-by: Wang Xiaoguang 
---
V2: In common/, I did't find an existing function suitable for
these 4 test cases to fill fs, so I still use _pwrite_byte() with
a big enough file length fo fill fs. Note, for btrfs, metadata space
still is not full, only data space is full, but it's OK for these
4 test cases.

All these 4 cases pass in xfs and btrfs(without compression), if
btrfs has compression enabled, these 4 cases will fail for false
enospc error, I have sent kernel patches to fix this bug.

V3: Introduce  _fill_fs in common/rc to fill fs.
V4: Fix some issues suggested by Eryu and Darrick.
V5: Put _fill_fs() in common/populate.
---
 common/populate   | 67 +++
 tests/generic/171 |  4 ++--
 tests/generic/172 |  5 +++--
 tests/generic/173 |  4 ++--
 tests/generic/174 |  4 ++--
 tests/generic/256 | 66 +-
 6 files changed, 82 insertions(+), 68 deletions(-)

diff --git a/common/populate b/common/populate
index 3b9b531..78e9809 100644
--- a/common/populate
+++ b/common/populate
@@ -539,3 +539,70 @@ _scratch_fuzz_test() {
(find "${SCRATCH_MNT}/test.1/" -type f -size -1048576k -print0 | xargs 
-0 cat) >/dev/null 2>&1
 }
 
+# Fill a file system by repeatedly creating files in the given folder
+# starting with the given file size.  Files are reduced in size when
+# they can no longer fit until no more files can be created.
+_fill_fs()
+{
+   local file_size=$1
+   local dir=$2
+   local block_size=$3
+   local switch_user=$4
+   local file_count=1
+   local bytes_written=0
+   local use_falloc=1;
+
+   if [ $# -ne 4 ]; then
+   echo "Usage: _fill_fs filesize dir blocksize switch_user"
+   exit 1
+   fi
+
+   if [ $switch_user -eq 0 ]; then
+   mkdir -p $dir
+   else
+   _user_do "mkdir -p $dir"
+   fi
+   if [ ! -d $dir ]; then
+   return 0;
+   fi
+
+   testio=`$XFS_IO_PROG -F -fc "falloc 0 $block_size" $dir/$$.xfs_io 2>&1`
+   echo $testio | grep -q "not found" && use_falloc=0
+   echo $testio | grep -q "Operation not supported" && use_falloc=0
+
+   if [ $file_size -lt $block_size ]; then
+   $file_size = $block_size
+   fi
+
+   while [ $file_size -ge $block_size ]; do
+   bytes_written=0
+   if [ $switch_user -eq 0 ]; then
+   if [ $use_falloc -eq 0 ]; then
+   $XFS_IO_PROG -fc "pwrite -b 8388608 0 
$file_size" \
+   $dir/$file_count
+   else
+   $XFS_IO_PROG -fc "falloc 0 $file_size" \
+   $dir/$file_count
+   fi
+   else
+   if [ $use_falloc -eq 0 ]; then
+   _user_do "$XFS_IO_PROG -f -c \"pwrite -b 
8388608 0 \
+   $file_size\" $dir/$file_count"
+   else
+   _user_do "$XFS_IO_PROG -f -c \"falloc 0 \
+   $file_size\" $dir/$file_count"
+   fi
+   fi
+
+   if [ -f $dir/$file_count ]; then
+   bytes_written=$(stat -c '%s' $dir/$file_count)
+   fi
+
+   # If there was no room to make the file, then divide it in
+   # half, and keep going
+   if [ $bytes_written -lt $file_size ]; then
+   file_size=$((file_size / 2))
+   fi
+   file_count=$((file_count + 1))
+   done
+}
diff --git a/tests/generic/171 b/tests/generic/171
index a69f798..d0cd192 100755
--- a/tests/generic/171
+++ b/tests/generic/171
@@ -38,6 +38,7 @@ _cleanup()
 
 # get standard environment, filters and checks
 . ./common/rc
+. ./common/populate
 . ./common/filter
 . ./common/attr
 . ./common/reflink
@@ -76,8 +77,7 @@ sync
 
 echo "Allocate the rest of the space"
 nr_free=$(stat -f -c '%f' $testdir)
-touch $testdir/file0 $testdir/file1
-_pwrite_byte 0x61 0 $((blksz * nr_free)) $testdir/eat_my_space >> $seqres.full 
2>&1
+_fill_fs $((nr_free * blksz)) $testdir/space $blksz 0 >> $seqres.full 2>&1
 sync
 
 echo "CoW the big file"
diff --git a/tests/generic/172 b/tests/generic/172
index 8192290..d943e64 100755
--- a/tests/generic/172
+++ b/tests/generic/172
@@ -38,6 +38,7 @@ _cleanup()
 
 # get standard environment, filters and checks
 . ./common/rc
+. ./common/populate
 . ./common/filter
 . ./common/attr
 . ./common/reflink
@@ 

[PATCH v5 1/2] common/populate: use _require_xfs_io_command() in right place

[Wang Xiaoguang]

In original common/populate codes, we put _require_xfs_io_command "falloc" and
_require_xfs_io_command "fpunch" in the begin of common/populate, but it's
not appropriate, for fs, which does not support falloc and punch, will not
be able to use other helper functions in common/populate, so here I choose
to put _require_xfs_io_command "falloc" or "punch" in helper function which
really use falloc and fpunch.

And xfs/120 uses fpunch, add _require_xfs_io_command "fpunch".

Signed-off-by: Wang Xiaoguang 
---
 common/populate | 7 ---
 tests/xfs/120   | 1 +
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/common/populate b/common/populate
index d0003c5..3b9b531 100644
--- a/common/populate
+++ b/common/populate
@@ -22,9 +22,6 @@
 #  Mountain View, CA 94043, USA, or: http://www.sgi.com
 #---
 
-_require_xfs_io_command "falloc"
-_require_xfs_io_command "fpunch"
-
 _require_xfs_db_blocktrash_z_command() {
test "${FSTYP}" = "xfs" || _notrun "cannot run xfs_db on ${FSTYP}"
$XFS_DB_PROG -x -f -c 'blocktrash -z' "${TEST_DEV}" | grep -q 'nothing 
on stack' || _notrun "blocktrash -z not supported"
@@ -90,6 +87,8 @@ __populate_fill_fs() {
 # types of metadata block
 _scratch_xfs_populate() {
_scratch_mount
+   _require_xfs_io_command "fpunch"
+
blksz="$(stat -f -c '%s' "${SCRATCH_MNT}")"
dblksz="$(xfs_info "${SCRATCH_MNT}" | grep naming.*bsize | sed -e 
's/^.*bsize=//g' -e 's/\([0-9]*\).*$/\1/g')"
leaf_lblk="$((32 * 1073741824 / blksz))"
@@ -192,6 +191,8 @@ _scratch_xfs_populate() {
 # types of metadata block
 _scratch_ext4_populate() {
_scratch_mount
+   _require_xfs_io_command "fpunch"
+
blksz="$(stat -f -c '%s' "${SCRATCH_MNT}")"
dblksz="${blksz}"
leaf_lblk="$((32 * 1073741824 / blksz))"
diff --git a/tests/xfs/120 b/tests/xfs/120
index 3deece6..631e2f2 100755
--- a/tests/xfs/120
+++ b/tests/xfs/120
@@ -47,6 +47,7 @@ _cleanup()
 _supported_fs xfs
 _supported_os Linux
 
+_require_xfs_io_command "fpunch"
 _require_scratch
 test -n "${FORCE_FUZZ}" || _require_scratch_xfs_crc
 _require_attrs
-- 
2.9.0



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 4/4] xfstests: btrfs/134: add test for incremental send which renames a directory already being deleted

[robbieko]

Hi Eryu Guan,

Yes, it need apply
[PATCH] "Btrfs: incremental send, do not skip generation inconsistency 
check for inode 256."

and test again, it will failed.

because current code there is a problem, but just will not happen.

Thansk
Robbie Ko

Eryu Guan 於 2016-11-01 15:20 寫到:

On Fri, Oct 28, 2016 at 09:44:06AM +0800, robbieko wrote:

From: Robbie Ko 

Test that an incremental send operation dosen't work because
it tries to rename a directory which is already deleted.

This test exercises scenarios used to fail in btrfs and are fixed by
the following patch for the linux kernel:

"Btrfs: incremental send, add generation check for inode is waiting 
for move."


I was testing with v4.9-rc1+ kernel and btrfs-progs v4.6. Seems above
patch is not merged in 4.9-rc1 kernel, but test passed for me, is that
expected?

Thanks,
Eryu



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 1/4] xfstests: btrfs/131: add test for an incremental send with name collision

[robbieko]

Hi Eryu Guan,

OK, I will modify it.

Thanks.
Robbie Ko

Eryu Guan 於 2016-11-01 15:17 寫到:

On Fri, Oct 28, 2016 at 09:44:03AM +0800, robbieko wrote:

From: Robbie Ko 

Test that an incremental send operation doesn't work because
there's a name collision in the destination and it's not checked
corretly before the rename operation applies.

This test exercises scenarios used to fail in btrfs and are fixed by
the following patch for the linux kernel:

"Btrfs: incremental send, do not skip generation inconsistency check 
for inode 256."


Signed-off-by: Robbie Ko 


Sorry for the late review!


---
 tests/btrfs/131 | 111 


 tests/btrfs/131.out |   2 +
 tests/btrfs/group   |   1 +
 3 files changed, 114 insertions(+)
 create mode 100755 tests/btrfs/131
 create mode 100644 tests/btrfs/131.out

diff --git a/tests/btrfs/131 b/tests/btrfs/131
new file mode 100755
index 000..2e2e0bc
--- /dev/null
+++ b/tests/btrfs/131
@@ -0,0 +1,111 @@
+#! /bin/bash
+# FS QA Test No. btrfs/131
+#
+# Test that an incremental send operation doesn't work because
+# there's a name collision in the destination and it's not checked
+# corretly before the rename operation applies.
+#
+#---
+# Copyright (C) 2016 Synology Inc. All Rights Reserved.
+# Author: Robbie Ko 
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software 
Foundation,

+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#---
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+tmp=/tmp/$$
+status=1   # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+   cd /
+   rm -fr $send_files_dir
+   rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_fs btrfs
+_supported_os Linux
+_require_test
+_require_scratch
+_require_fssum
+
+send_files_dir=$TEST_DIR/btrfs-test-$seq
+
+rm -f $seqres.full
+rm -fr $send_files_dir
+mkdir $send_files_dir
+
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+mkdir $SCRATCH_MNT/a1
+mkdir $SCRATCH_MNT/a2
+
+# Filesystem looks like:
+#
+# . (ino 
256)
+# |--- a1/  (ino 
257)

+# |
+# |--- a2/  (ino 
258)

+#
+_run_btrfs_util_prog subvolume snapshot -r $SCRATCH_MNT 
$SCRATCH_MNT/mysnap1

+
+_run_btrfs_util_prog send $SCRATCH_MNT/mysnap1 -f 
$send_files_dir/1.snap


I found these "run_check" based helpers make the result harder to read
when test failed, I can't see immediate error messages from the diff
output but have to open $seqres.full file to see the details.

Can you please use $BTRFS_UTIL_PROG directly, in all these four tests?
One concern about using bare $BTRFS_UTIL_PROG is that output messages 
of

btrfs command are always changed. But I think, at least, we can ignore
the stdout and capture the stderr only. e.g.

$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT \
$SCRATCH_MNT/mysnap1 >/dev/null

So if everything works fine no output can be seen, but if snapshot
creation is failed, we can see error messages from stderr, which could
fail the test.


+
+_scratch_unmount
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+touch $SCRATCH_MNT/a2
+
+# Filesystem now looks like:
+#
+# . (ino 
256)
+# |--- a2   (ino 
257)

+#
+_run_btrfs_util_prog subvolume snapshot -r $SCRATCH_MNT 
$SCRATCH_MNT/mysnap2

+
+_run_btrfs_util_prog receive $SCRATCH_MNT -f $send_files_dir/1.snap
+rm $send_files_dir/1.snap
+
+run_check $FSSUM_PROG -A -f -w $send_files_dir/1.fssum 
$SCRATCH_MNT/mysnap1
+run_check $FSSUM_PROG -A -f -w $send_files_dir/2.fssum 
$SCRATCH_MNT/mysnap2


And this "run_check" is not necessary either. Just run $FSSUM_PROG
directly and let it print "OK", and match it in .out file.

Thanks,
Eryu


+
+_run_btrfs_util_prog send $SCRATCH_MNT/mysnap1 -f 
$send_files_dir/1.snap
+_run_btrfs_util_prog send -p $SCRATCH_MNT/mysnap1 
$SCRATCH_MNT/mysnap2 \

+   -f $send_files_dir/2.snap
+
+# Now recreate the filesystem by receiving 

[PATCH] btrfs: Call kunmap if zlib_inflateInit2 fails

[Nick Terrell]

If zlib_inflateInit2 fails, the input page is never unmapped.
Add a call to kunmap when it fails.

Signed-off-by: Nick Terrell 
---
 fs/btrfs/zlib.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/fs/btrfs/zlib.c b/fs/btrfs/zlib.c
index 88d274e..30e19c9 100644
--- a/fs/btrfs/zlib.c
+++ b/fs/btrfs/zlib.c
@@ -250,6 +250,7 @@ static int zlib_decompress_biovec(struct list_head *ws, 
struct page **pages_in,
 
if (Z_OK != zlib_inflateInit2(>strm, wbits)) {
printk(KERN_WARNING "BTRFS: inflateInit failed\n");
+   kunmap(pages_in[page_in_index]);
return -EIO;
}
while (workspace->strm.total_in < srclen) {
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] generic: create and delete files repeatedly to exercise ENOSPC behaviour

[Dave Chinner]

On Tue, Nov 01, 2016 at 07:19:30PM +0800, Wang Xiaoguang wrote:
> In btrfs, sometimes though the number of created files' consumed disk space
> are not larger than fs's free space, we can still get some ENOSPC error, it
> may be that btrfs does not try hard to reclaim disk space(I have sent kernel
> patch to resolve this kind of enospc error. Note, this false enospc error
> will not always happen even in kernel without my fixing patch).
> 
> Currently only in btrfs, I get this ENOSPC error, xfs and ext4 work well.
.
> +RUN_TIME=$((600 * $TIME_FACTOR))
> +fs_size=$((15 * 1024 * 1024 * 1024))
> +_scratch_mkfs_sized $fs_size > $seqres.full 2>&1
> +_scratch_mount > $seqres.full 2>&1
> +
> +testfile1=$SCRATCH_MNT/testfile1
> +testfile2=$SCRATCH_MNT/testfile2
> +filesize1=$(((fs_size * 80) / 100))
> +filesize2=$(((fs_size * 5) / 100))
> +
> +do_test()
> +{
> + while [ -f $SCRATCH_MNT/run ]; do
> + $XFS_IO_PROG -fc "pwrite 0 $filesize1" $testfile1 > /dev/null
> + $XFS_IO_PROG -fc "pwrite 0 $filesize2" $testfile2 > /dev/null
> + rm -f $testfile1 $testfile2 
> + done
> +}

What are you trying to test here that other ENOSPC tests
don't exercise? Why cant you just use preallocation to trigger
ENOSPC repeatedly instead of writing data? That would allow multiple
iterations per second, not one every few minutes...

Cheers,

Dave.
-- 
Dave Chinner
da...@fromorbit.com
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 1/4] btrfs-progs: utils: Introduce function to escape characters

[Qu Wenruo]

At 11/01/2016 06:08 PM, David Sterba wrote:

On Tue, Nov 01, 2016 at 04:01:43PM +0800, Qu Wenruo wrote:

Introduce new function, escape_string_inplace(), to escape specified
characters in place.


Sorry, the pointer to seq_path was misleading. The actual escape
function is mangle_path and it copies one string to another. As we just
print the path, we can simply switch and call putchar.



Putchar() method is indeed much easier to implement.

But it makes us hard to do further formatting, like aligning the path to 
given width. (At least we are still using 32 chars alignment for path)


So I still prefer the current full function string escaping and still 
use %-32s for formatting.



And the idea of implementing escape_string_inplace() as a pure string 
manipulation function can make it more agile for later use.

For example, we can reuse it for print-tree.

Although the current esacpe_string_inplace() can be further fine-tuned 
to avoid calling memmove() in a loop, which I can update it in next version.


Thanks,
Qu


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v4] generic: make 17[1-4] work well when btrfs compression is enabled

[Dave Chinner]

On Tue, Nov 01, 2016 at 04:49:34PM +0800, Wang Xiaoguang wrote:
> hi Darrick,
> 
> Common/populate needs xfs_io supports falloc and fpunch,
> so I didn't put _fill_fs() in common/populate.

Tests will include common/rc first, and so pick up the functionality
_fill_fs requires before it's included from common/populate. So
there is no reason to put it in common/rc.

Cheers,

Dave.
-- 
Dave Chinner
da...@fromorbit.com
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/3] btrfs-progs: send: fix handling of multiple snapshots (-p option)

[Tsutomu Itoh]

Sorry for the late reply.

On 2016/10/29 0:10, David Sterba wrote:
> On Wed, Oct 19, 2016 at 11:35:03AM +0900, Tsutomu Itoh wrote:
>> We cannot send multiple snapshots at once by -p option.
> 
> We cannot like that it's broken, or we cannot because it's not supposed
> to work that way. I guess it's the former, but the changelog text is a
> bit confusing.

Sorry, I'm not good at English.

>>
>> [before]
>> # btrfs send -f /tmp/data0 -p Snap0 Snap[12]
>> At subvol Snap1
>> At subvol Snap2
>> ERROR: parent determination failed for 0
>> # 
>>
>> [after]
>> # btrfs send -f /tmp/data0 -p Snap0 Snap[12]
>> At subvol Snap1
>> At subvol Snap2
>> # 
> 
> I'm not sure it's fixed, I wrote a simple test, attached, that triggers
> the bug even with the patch applied.
> 

In the attached script,

   run_check $SUDO_HELPER btrfs send -f "$here"/send.stream -p subv-snap1 
subv-snap2 subv-snap3

I think that 'btrfs' is a mistake of '$TOP/btrfs'.

Thanks,
Tsutomu

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/3] btrfs-progs: Add a command to show bg info

[divya . indi]

On 10/28/2016 09:00 AM, David Sterba wrote:

On Mon, Oct 17, 2016 at 05:35:14PM -0700, Divya Indi wrote:

Add a new subcommand to btrfs inspect-internal

btrfs inspect-internal bg_analysis 
Gives information about all the block groups.

The sample output from the cover letter should also go here (or just
here).

Below are some comments, but overall, I think the block group dumping
should be more advanced than just what this patch does. It's a good
start, but given the rich structure of the blockgroups and chunks, I'd
rather see the basics done right from the beginning.

The blockgroups and chunks can be viewed in a logical or physical way.
I've sent a patch some time agou, that dumps the physical structure, but
got reminded that the balance analysis is more interesting on the
logical level.

Ideally I'd like to keep both ways to show the information. The physical
way is easier, just iterate the device extents and print start, lenght
etc. The logical is more tricky as it's a tree structure, when one
logical chunk could comprised of several physical chunks. And this must
reflect all current raid profiles, where we're mixing mirorring and
striping, and chunks of special kind (parity).
Since most of this information is available through the chunk, we can 
try to implement these as part of v2.


Now, I'm not asking you to implement all of that, but I want to make
sure that code that touches the area of interest does not block further
extensions. I understand that you want to add the ability to optimize
the balance.


Signed-off-by: Divya Indi 
Reviewed-by: Ashish Samant 
Reviewed-by: Liu Bo 
---
  cmds-inspect.c |  114 
  1 files changed, 114 insertions(+), 0 deletions(-)

diff --git a/cmds-inspect.c b/cmds-inspect.c
index f435ea9..0e2f15a 100644
--- a/cmds-inspect.c
+++ b/cmds-inspect.c
@@ -147,6 +147,118 @@ static int get_bg_info(int fd, struct 
btrfs_ioctl_search_args *bg_args,
}
return ret;
  }
+
+static const char * const cmd_inspect_bg_analysis_usage[] = {
+   "btrfs inspect-internal bg_analysis  ",
+   "Get information about all block groups",
+   "",
+   "",
+   NULL
+};
+
+static int cmd_inspect_bg_analysis(int argc, char **argv)
+{
+   struct btrfs_ioctl_search_args args;
+   struct btrfs_ioctl_search_args bg_args;
+   struct btrfs_ioctl_search_header *header;
+   struct btrfs_ioctl_search_header *bg_header;
+   struct btrfs_ioctl_search_key *sk;
+   struct btrfs_ioctl_search_key *bg_sk;
+   struct btrfs_block_group_item *bg;
+   struct btrfs_chunk *chunk;
+   unsigned long off = 0;
+   unsigned long bg_off = 0;
+   DIR *dirstream = NULL;
+   int fd;
+   int i;
+   int ret = 0;
+   u64 used;
+   u64 flags;
+   char bg_type[32] = {0};
+
+   if (check_argc_exact(argc, 2))
+   usage(cmd_inspect_bg_analysis_usage);
+
+   fd = btrfs_open_dir(argv[optind], , 1);
+   if (fd < 0)
+   return 1;
+
+   memset(, 0, sizeof(args));
+   sk = 
+   sk->min_offset = sk->min_transid = 0;
+   sk->max_offset = sk->max_transid = (u64)-1;
+   printf("%20s%20s%20s%20s\n", "Type", "Start", "Len", "Used");
+   while (1) {
+
+   /* Walk through the chunk tree and retrieve all the chunks */
+   ret = get_chunks(fd, );
+   if (ret < 0)
+   goto out;
+
+   /*
+* it should not happen.
+*/
+   if (sk->nr_items == 0)
+   break;

So is this an error condition? If yes, then it should be handled.


+
+   off = 0;
+   memset(_args, 0, sizeof(bg_args));
+   bg_sk = _args.key;
+
+   bg_sk->tree_id = BTRFS_EXTENT_TREE_OBJECTID;
+   bg_sk->min_type = BTRFS_BLOCK_GROUP_ITEM_KEY;
+   bg_sk->max_type = BTRFS_BLOCK_GROUP_ITEM_KEY;
+   bg_sk->min_transid =  0;
+   bg_sk->max_transid = (u64)-1;
+
+   for (i = 0; i < sk->nr_items; i++) {
+   header = (struct btrfs_ioctl_search_header *)(args.buf
+ + off);
+   off += sizeof(*header);
+   if (header->type == BTRFS_CHUNK_ITEM_KEY) {
+   chunk = (struct btrfs_chunk *)
+   (args.buf + off);
+
+   /* For every chunk lookup an exact match(bg) in
+* the extent tree and read its used values */

Comment formatting


+   ret = get_bg_info(fd, _args, header->offset,
+ chunk->length);
+   if (ret < 0)
+

Re: [PATCH 1/3] btrfs-progs: Generic functions to retrieve chunks and their bg info

[divya . indi]

On 10/28/2016 08:44 AM, David Sterba wrote:

On Mon, Oct 17, 2016 at 05:35:13PM -0700, Divya Indi wrote:

An efficient alternative to retrieving block groups:
get_chunks(): Walk the chunk tree to retrieve the chunks.
get_bg_info(): For each retrieved chunk, lookup an exact match of block
group in the extent tree.

Signed-off-by: Divya Indi 
Reviewed-by: Ashish Samant 
Reviewed-by: Liu Bo 
---
  cmds-inspect.c |   66 
  1 files changed, 66 insertions(+), 0 deletions(-)

diff --git a/cmds-inspect.c b/cmds-inspect.c
index 4b7cea0..f435ea9 100644
--- a/cmds-inspect.c
+++ b/cmds-inspect.c
@@ -81,6 +81,72 @@ out:
return !!ret;
  }
  
+static void bg_flags_to_str(u64 flags, char *ret)

+{
+   int empty = 1;
+
+   if (flags & BTRFS_BLOCK_GROUP_DATA) {
+   empty = 0;
+   strcpy(ret, "DATA");
+   }
+   if (flags & BTRFS_BLOCK_GROUP_METADATA) {
+   if (!empty)
+   strcat(ret, "|");
+   strcat(ret, "METADATA");
+   }
+   if (flags & BTRFS_BLOCK_GROUP_SYSTEM) {
+   if (!empty)
+   strcat(ret, "|");
+   strcat(ret, "SYSTEM");
+   }
+}
+
+/* Walking through the chunk tree to retrieve chunks. */

No empty newline.


+
+static int get_chunks(int fd, struct btrfs_ioctl_search_args *chunk_args)
+{
+   struct btrfs_ioctl_search_key *sk;
+   int ret;
+   int e;
+
+   sk = _args->key;
+
+   sk->tree_id = BTRFS_CHUNK_TREE_OBJECTID;
+   sk->min_objectid = sk->max_objectid = BTRFS_FIRST_CHUNK_TREE_OBJECTID;
+   sk->max_type = sk->min_type = BTRFS_CHUNK_ITEM_KEY;

Please don't do multiple asignments in one statement.


+   sk->nr_items = 4096;
+
+   ret = ioctl(fd, BTRFS_IOC_TREE_SEARCH, chunk_args);
+   e = errno;

This is useless asignment, I've removed it from the code, please don't
reintrduce it.


+   if (ret < 0) {
+   fprintf(stderr, "ret %d error '%s'\n", ret,
+   strerror(e));
+   }
+   return ret;
+}



+
+/* Given the objectid, find the block group item in the extent tree */
+static int get_bg_info(int fd, struct btrfs_ioctl_search_args *bg_args,
+  u64 objectid, unsigned long length)
+{
+   struct btrfs_ioctl_search_key *bg_sk;
+   int ret;
+   int e;
+
+   bg_sk = _args->key;
+
+   bg_sk->min_objectid = bg_sk->max_objectid = objectid;
+   bg_sk->nr_items = 1;
+   bg_sk->min_offset = bg_sk->max_offset = length;

Same here.


+
+   ret = ioctl(fd, BTRFS_IOC_TREE_SEARCH, bg_args);
+   e = errno;
+   if (ret < 0) {
+   fprintf(stderr, "ret %d error '%s'\n", ret,
+   strerror(e));

Please take a look how the error messages are constructed when the tree
search ioctl fails, there are enough examples in the code.


+   }
+   return ret;
+}
  static const char * const cmd_inspect_inode_resolve_usage[] = {
"btrfs inspect-internal inode-resolve [-v]  ",
"Get file system paths for the given inode",

Actually, I'm not sure if such functions should exist at all, as they
only hide the search ioctl but don't do any validation of the returned
keys and data.
The intent was to avoid the same assignments and calls in both the sub 
commands, but I see your point.

 Noted- for v2.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/3] btrfs-progs: Add command to check if balance op is req

[divya . indi]

On 10/31/2016 09:33 AM, David Sterba wrote:

On Fri, Oct 28, 2016 at 05:29:45PM +0100, Graham Cobb wrote:

On 28/10/16 16:20, David Sterba wrote:

I tend to agree with this approach. The usecase, with some random sample
balance options:

  $ btrfs balance start --analyze -dusage=10 -musage=5 /path

Wouldn't a "balance analyze" command be better than "balance start
--analyze"? I would have guessed the latter started the balance but
printed some analysis as well (before or, probably more usefully,
afterwards).

Right, separate command seems better.
What about btrfs inspect-internal bg_analysis (new name: 
show-block-groups)? It can still be a subcommand for inspect-internal?

So, wel have 2 new sub commands:
1) btrfs balance analyze
2) btrfs inspect-internal show-block-groups

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 2/4] btrfs-progs: introduce new send-dump object

[Qu Wenruo]

At 11/01/2016 06:22 PM, David Sterba wrote:

On Tue, Nov 01, 2016 at 04:01:44PM +0800, Qu Wenruo wrote:

+/*
+ * Underlying PRINT_DUMP, the only difference is how we handle
+ * the full path.
+ */
+static int __print_dump(int subvol, void *user, const char *path,
+   const char *title, const char *fmt, ...)


printf-like the __attribute__ ((format (printf, ...)))


+{
+   struct btrfs_dump_send_args *r = user;
+   char real_title[TITLE_WIDTH + 1] = { 0 };
+   char full_path[PATH_MAX] = {0};
+   char *out_path;
+   va_list args;
+   int ret;
+
+   if (subvol) {
+   PATH_CAT_OR_RET(title, r->full_subvol_path, r->root_path, path, 
ret);
+   out_path = r->full_subvol_path;
+   } else {
+   PATH_CAT_OR_RET(title, full_path, r->full_subvol_path, path, 
ret);
+   out_path = full_path;
+   }
+   string_escape_inplace(out_path, " \n\t\\");
+
+   /* Append ':' to title */
+   strncpy(real_title, title, TITLE_WIDTH - 1);
+   strncat(real_title, ":", TITLE_WIDTH);


I'd rather avoid such string operations, ':', just print everything.


I'm completely OK to remove the ':'.




+
+   /* Unified header, */
+   printf("%-*s%-*s", TITLE_WIDTH, real_title, PATH_WIDTH, out_path);


PATH_WIDTH is used only here, please hardcode it into the format string.

The rest of the patch looks good. I think I've seen some artifacts in
the output, but we can tune this later.


Would you like to share the artifacts so I can handle them in next update?




+   ret = strftime(dest, max_size, "%Y-%m-%d %H:%M:%S", tm);


We should use the RFC 3339 format, as it's standardized and also is a
string without whitespace.


Will use RFC 3339 in next version.

Thanks,
Qu


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Snapper & apt-btrfs-snapshot on Ubuntu

[Hans van Kranenburg]

On 11/02/2016 01:00 AM, Ahmed Badr wrote:
> 
> Btrfs newbie here

Congrats!

> trying to figure out how to use it properly on Ubuntu
> 16.04 server.
> 
> I have root formated using Btrfs, and installed Snapper and everything
> seems to works fine. I installed apt-btrfs-snapshot which takes snapshot
> before and after apt-get just like OpenSUSE/Zypper and that seems to work
> fine too.
> 
> The problem is, unlike OpenSUSE/Zypper, on Ubuntu neither Snapper nor
> apt-btrfs-snapshot see the snapshots made by the other program. So I cannot
> list, compare or manage the apt-btrfs snapshots using snapper. Any way to
> fix this?

To get more info, try 'btrfs sub list -q /mountpoint'. This will show
you in where both tools create their snapshots, and it will show
(parent_uuid field) which subvolume they're taken from.

While all of the snapshots with a matching parent_uuid are snapshots of
the same thing in linear time for btrfs, the tools may have their own
added administration on top (like snapshot names, or directories they're
placed in), which prevents them from being able to do something with the
other ones. This is not the fault of btrfs, it's because those tools
probably are designed to ignore anything they didn't cause to happen
themselves.

-- 
Hans van Kranenburg
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Snapper & apt-btrfs-snapshot on Ubuntu

[Ahmed Badr]

Hi all,

Btrfs newbie here trying to figure out how to use it properly on Ubuntu
16.04 server.

I have root formated using Btrfs, and installed Snapper and everything
seems to works fine. I installed apt-btrfs-snapshot which takes snapshot
before and after apt-get just like OpenSUSE/Zypper and that seems to work
fine too.

The problem is, unlike OpenSUSE/Zypper, on Ubuntu neither Snapper nor
apt-btrfs-snapshot see the snapshots made by the other program. So I cannot
list, compare or manage the apt-btrfs snapshots using snapper. Any way to
fix this?

Thanks.
--
A.B.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PULL] Btrfs fixes for 4.9-rc4

[Chris Mason]

On Tue, Nov 01, 2016 at 02:02:25PM +0100, David Sterba wrote:

Hi,

please pull the following assorted small fixes to 4.9, thanks. The branch is
"rc3" as it was meant for rc3 but I did not get to sending the pull request,
and I don't want to recreate the branch just because of a slightly wrong
name.


Thanks Dave, I'll get this in.

-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: cannot mount read-write because of unsupported optional features (2)

[Omar Sandoval]

On Tue, Nov 01, 2016 at 04:03:38PM +0100, Tobias Holst wrote:
> Ah thanks! Looks like I missed this bug on the list...
> 
> But as I understand it, it should be mountable with btrfs-progs
> >=4.7.3 or when mounting it with "-o clear_cache,nospace_cache". But
> both doesn't work. Is it only mountable with kernel >=4.9 anymore?
> That would be interesting since I never ran a kernel newer than 4.8...
> 
> Regards,
> Tobias

Hi, Tobias,

The fix was backported to v4.7 and v4.8 stable, so it's not just v4.9.
The btrfs-progs version doesn't matter for mounting. If you use a stable
kernel newer than v4.7.10 or v4.8.4, you will be able to mount it. If
you really need to use it with an older kernel, you can clear the free
space tree with -o clear_cache,nospace_cache from the newer kernel.

Sorry for the trouble, we really had to add this incompatability bit
even for x86 because of a btrfs-progs bug that could potentially corrupt
the free space tree.
-- 
Omar
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: cannot mount read-write because of unsupported optional features (2)

[Tobias Holst]

Ah thanks! Looks like I missed this bug on the list...

But as I understand it, it should be mountable with btrfs-progs
>=4.7.3 or when mounting it with "-o clear_cache,nospace_cache". But
both doesn't work. Is it only mountable with kernel >=4.9 anymore?
That would be interesting since I never ran a kernel newer than 4.8...

Regards,
Tobias


2016-11-01 6:24 GMT+01:00 Qu Wenruo :
>
>
> At 11/01/2016 12:46 PM, Tobias Holst wrote:
>>
>> Hi
>>
>> I can't mount my boot partition anymore. When I try it by entering
>> "mount /dev/sdi1 /mnt/boot/" I get:
>>>
>>> mount: wrong fs type, bad option, bad superblock on /dev/sdi1,
>>>   missing codepage or helper program, or other error
>>>
>>>   In some cases useful info is found in syslog - try
>>>   dmesg | tail or so.
>>
>>
>> "dmesg | tail" gives me:
>>>
>>> BTRFS info (device sdi1): using free space tree
>>> BTRFS info (device sdi1): has skinny extents
>>> BTRFS error (device sdi1): cannot mount read-write because of unsupported
>>> optional features (2)
>>> BTRFS: open_ctree failed
>>
>>
>> I am using an Ubuntu 16.10 Live CD with Kernel 4.8 and btrfs-progs v4.8.2.
>>
>> "btrfs inspect-internal dump-super /dev/sdi1" gives me the following:
>>>
>>> superblock: bytenr=65536, device=/dev/sdi1
>>> -
>>> csum_type   0 (crc32c)
>>> csum_size   4
>>> csum0x0f346b08 [match]
>>> bytenr  65536
>>> flags   0x1
>>>( WRITTEN )
>>> magic   _BHRfS_M [match]
>>> fsid67ac5740-1ced-4d59-8999-03bb3195ec49
>>> label   t-hyper-boot
>>> generation  64
>>> root20971520
>>> sys_array_size  129
>>> chunk_root_generation   43
>>> root_level  1
>>> chunk_root  12587008
>>> chunk_root_level0
>>> log_root0
>>> log_root_transid0
>>> log_root_level  0
>>> total_bytes 2147483648
>>> bytes_used  102813696
>>> sectorsize  4096
>>> nodesize4096
>>> leafsize4096
>>> stripesize  4096
>>> root_dir6
>>> num_devices 1
>>> compat_flags0x0
>>> compat_ro_flags 0x3
>
> compat_ro_flags 0x3 is FREE_SPACE_TREE and FREE_SPACE_TREE_VALID.
>
> FREE_SPACE_TREE_VALID is introduced later to fix a endian bug in free space
> tree.
>
> And that seems to be cause.
>
> Thanks,
> Qu
>
>>> incompat_flags  0x34d
>>>( MIXED_BACKREF |
>>>  MIXED_GROUPS |
>>>  COMPRESS_LZO |
>>>  EXTENDED_IREF |
>>>  SKINNY_METADATA |
>>>  NO_HOLES )
>>> cache_generation53
>>> uuid_tree_generation64
>>> dev_item.uuid   273d5800-add1-45bb-8a11-ecd6d8c1503e
>>> dev_item.fsid   67ac5740-1ced-4d59-8999-03bb3195ec49 [match]
>>> dev_item.type   0
>>> dev_item.total_bytes2147483648
>>> dev_item.bytes_used 667680768
>>> dev_item.io_align   4096
>>> dev_item.io_width   4096
>>> dev_item.sector_size4096
>>> dev_item.devid  1
>>> dev_item.dev_group  0
>>> dev_item.seek_speed 0
>>> dev_item.bandwidth  0
>>> dev_item.generation 0
>>
>>
>> Does anyone have an idea why I can't mount it anymore?
>>
>> Regards,
>> Tobias
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

dmesg traces from enospc_debug & kernel 4.9-r3

[E V]

Testing 4.9-rc3 on my big backup system running rsync. Saw 10 traces
over the last day, I broke them down into groups for were they differ
with a full trace at the bottom for reference. Hopefully useful to
somebody. Everything is running fine so far. Can send all the full
traces if someone is interested.

2 of these:
trace6:[14843.507199]  [] ?
btrfs_alloc_tree_block+0x3bc/0x4d0 [btrfs]
trace6-[14843.507254]  [] ?
__btrfs_cow_block+0x148/0x5d0 [btrfs]
trace6-[14843.507294]  [] ?
btrfs_cow_block+0x114/0x1d0 [btrfs]
trace6-[14843.507334]  [] ?
btrfs_search_slot+0x206/0xa40 [btrfs]
trace6-[14843.507376]  [] ?
__btrfs_qgroup_release_data+0x88/0x120 [btrfs]

4 of these:
trace8:[15611.350323]  [] ?
btrfs_alloc_tree_block+0x3bc/0x4d0 [btrfs]
trace8-[15611.350378]  [] ? leaf_space_used+0xd2/0x110 [btrfs]
trace8-[15611.350417]  [] ?
__btrfs_cow_block+0x148/0x5d0 [btrfs]
trace8-[15611.350458]  [] ?
btrfs_cow_block+0x114/0x1d0 [btrfs]
trace8-[15611.350498]  [] ?
btrfs_search_slot+0x206/0xa40 [btrfs]

1 each of the following 4
trace1:[14494.245397]  [] ?
btrfs_alloc_tree_block+0x3bc/0x4d0 [btrfs]
trace1-[14494.245457]  [] ?
read_extent_buffer_pages+0x1c6/0x2a0 [btrfs]
trace1-[14494.245513]  [] ?
__btrfs_cow_block+0x148/0x5d0 [btrfs]
trace1-[14494.245553]  [] ?
btrfs_cow_block+0x114/0x1d0 [btrfs]
trace1-[14494.245593]  [] ?
btrfs_search_slot+0x206/0xa40 [btrfs]
--
trace10:[83879.470211]  [] ?
btrfs_alloc_tree_block+0x3bc/0x4d0 [btrfs]
trace10-[83879.470259]  [] ?
autoremove_wake_function+0x30/0x30
trace10-[83879.470299]  [] ?
__btrfs_cow_block+0x148/0x5d0 [btrfs]
trace10-[83879.470339]  [] ?
btrfs_cow_block+0x114/0x1d0 [btrfs]
trace10-[83879.470378]  [] ? push_leaf_left+0xf0/0x170 [btrfs]
--
trace7:[15069.248346]  [] ?
btrfs_alloc_tree_block+0x3bc/0x4d0 [btrfs]
trace7-[15069.248394]  [] ? __activate_page+0x1b5/0x250
trace7-[15069.248423]  [] ? put_pages_list+0x70/0x70
trace7-[15069.248452]  [] ? pagevec_lru_move_fn+0xb8/0xd0
trace7-[15069.248490]  [] ?
__btrfs_cow_block+0x148/0x5d0 [btrfs]
--
trace9:[21545.901275]  [] ?
btrfs_alloc_tree_block+0x3bc/0x4d0 [btrfs]
trace9-[21545.901323]  [] ? update_curr+0x7e/0x100
trace9-[21545.901361]  [] ?
__btrfs_cow_block+0x148/0x5d0 [btrfs]
trace9-[21545.901401]  [] ?
btrfs_cow_block+0x114/0x1d0 [btrfs]
trace9-[21545.901443]  [] ?
btrfs_search_slot+0x206/0xa40 [btrfs]

A full one for reference
$ cat trace10
[83879.469227] use_block_rsv: 214 callbacks suppressed
[83879.469257] [ cut here ]
[83879.469309] WARNING: CPU: 1 PID: 14025 at
fs/btrfs/extent-tree.c:8325 btrfs_alloc_tree_block+0x3bc/0x4d0 [btrfs]
[83879.469360] BTRFS: block rsv returned -28
[83879.469787] CPU: 1 PID: 14025 Comm: kworker/u34:4 Tainted: G
W I 4.9.0-rc3 #1
[83879.469893] Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
[83879.469926]   812946c9 c9000e5ab968

[83879.469979]  8105a27f 8808198d8158 c9000e5ab9b8
4000
[83879.470033]  8808198d8000 88039e154f00 
8105a2ea
[83879.470087] Call Trace:
[83879.470113]  [] ? dump_stack+0x46/0x5d
[83879.470143]  [] ? __warn+0xbf/0xe0
[83879.470170]  [] ? warn_slowpath_fmt+0x4a/0x50
[83879.470211]  [] ?
btrfs_alloc_tree_block+0x3bc/0x4d0 [btrfs]
[83879.470259]  [] ? autoremove_wake_function+0x30/0x30
[83879.470299]  [] ? __btrfs_cow_block+0x148/0x5d0 [btrfs]
[83879.470339]  [] ? btrfs_cow_block+0x114/0x1d0 [btrfs]
[83879.470378]  [] ? push_leaf_left+0xf0/0x170 [btrfs]
[83879.470418]  [] ? split_leaf+0x57a/0x710 [btrfs]
[83879.470456]  [] ? leaf_space_used+0xd2/0x110 [btrfs]
[83879.470495]  [] ? btrfs_search_slot+0x926/0xa40 [btrfs]
[83879.470540]  [] ? btrfs_get_token_32+0xee/0x110 [btrfs]
[83879.470584]  [] ?
btrfs_csum_file_blocks+0x3f8/0x610 [btrfs]
[83879.470643]  [] ?
add_pending_csums.isra.48+0x41/0x60 [btrfs]
[83879.470702]  [] ?
btrfs_finish_ordered_io+0x31b/0x640 [btrfs]
[83879.470761]  [] ? normal_work_helper+0xc3/0x2f0 [btrfs]
[83879.470794]  [] ? process_one_work+0x14b/0x400
[83879.470824]  [] ? worker_thread+0x5d/0x470
[83879.470853]  [] ? rescuer_thread+0x310/0x310
[83879.470882]  [] ? kthread+0xcb/0xf0
[83879.470909]  [] ? kthread_park+0x50/0x50
[83879.470939]  [] ? ret_from_fork+0x22/0x30
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] btrfs: store and load values of stripes_min/stripes_max in balance status item

[David Sterba]

The balance status item contains currently known filter values, but the
stripes filter was unintentionally not among them. This would mean, that
interrupted and automatically restarted balance does not apply the
stripe filters.

Fixes: dee32d0ac3719ef8d640efaf0884111df444730f
CC: sta...@vger.kernel.org # 4.4+
Signed-off-by: David Sterba 
---
 fs/btrfs/ctree.h | 4 
 1 file changed, 4 insertions(+)

diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 9d8edcb0813c..223c706aea1f 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -2210,6 +2210,8 @@ btrfs_disk_balance_args_to_cpu(struct btrfs_balance_args 
*cpu,
cpu->target = le64_to_cpu(disk->target);
cpu->flags = le64_to_cpu(disk->flags);
cpu->limit = le64_to_cpu(disk->limit);
+   cpu->stripes_min = le32_to_cpu(disk->stripes_min);
+   cpu->stripes_max = le32_to_cpu(disk->stripes_max);
 }
 
 static inline void
@@ -2228,6 +2230,8 @@ btrfs_cpu_balance_args_to_disk(struct 
btrfs_disk_balance_args *disk,
disk->target = cpu_to_le64(cpu->target);
disk->flags = cpu_to_le64(cpu->flags);
disk->limit = cpu_to_le64(cpu->limit);
+   disk->stripes_min = cpu_to_le32(cpu->stripes_min);
+   disk->stripes_max = cpu_to_le32(cpu->stripes_max);
 }
 
 /* struct btrfs_super_block */
-- 
2.10.1

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PULL] Btrfs fixes for 4.9-rc4

[David Sterba]

Hi,

please pull the following assorted small fixes to 4.9, thanks. The branch is
"rc3" as it was meant for rc3 but I did not get to sending the pull request,
and I don't want to recreate the branch just because of a slightly wrong
name.


The following changes since commit d9ed71e5457c8c5bf1dc706e06468eab9e2aa87e:

  Merge branch 'fst-fixes' of 
git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux into for-linus-4.9 
(2016-10-12 13:16:00 -0700)

are available in the git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-4.9-rc3

for you to fetch changes up to 9d1032cc49a8a1065e79ee323de66bcb4fdbd535:

  btrfs: fix WARNING in btrfs_select_ref_head() (2016-10-24 18:20:29 +0200)


Dan Carpenter (1):
  Btrfs: remove some no-op casts

Goldwyn Rodrigues (1):
  btrfs: qgroup: Prevent qgroup->reserved from going subzero

Liu Bo (1):
  Btrfs: kill BUG_ON in do_relocation

Wang Xiaoguang (3):
  btrfs: make file clone aware of fatal signals
  btrfs: pass correct args to btrfs_async_run_delayed_refs()
  btrfs: fix WARNING in btrfs_select_ref_head()

 fs/btrfs/extent-tree.c |  3 +++
 fs/btrfs/extent_io.c   |  8 
 fs/btrfs/inode.c   | 13 +
 fs/btrfs/ioctl.c   |  5 +
 fs/btrfs/relocation.c  |  9 -
 5 files changed, 29 insertions(+), 9 deletions(-)
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] generic: create and delete files repeatedly to exercise ENOSPC behaviour

[Eryu Guan]

On Tue, Nov 01, 2016 at 07:19:30PM +0800, Wang Xiaoguang wrote:
> In btrfs, sometimes though the number of created files' consumed disk space
> are not larger than fs's free space, we can still get some ENOSPC error, it
> may be that btrfs does not try hard to reclaim disk space(I have sent kernel
> patch to resolve this kind of enospc error. Note, this false enospc error
> will not always happen even in kernel without my fixing patch).
> 
> Currently only in btrfs, I get this ENOSPC error, xfs and ext4 work well.
> 
> Signed-off-by: Wang Xiaoguang 
> ---
>  tests/generic/389 | 78 
> +++
>  tests/generic/389.out |  2 ++
>  tests/generic/group   |  1 +
>  3 files changed, 81 insertions(+)
>  create mode 100755 tests/generic/389
>  create mode 100644 tests/generic/389.out
> 
> diff --git a/tests/generic/389 b/tests/generic/389
> new file mode 100755
> index 000..96bc12e
> --- /dev/null
> +++ b/tests/generic/389
> @@ -0,0 +1,78 @@
> +#! /bin/bash
> +# FS QA Test 389
> +#
> +# Create and delete files repeatedly to exercise ENOSPC behaviour. 

Trailing whitespace in this line.

> +#
> +#---
> +# Copyright (c) 2016 Fujitsu.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# Modify as appropriate.
> +_supported_fs generic
> +_supported_os Linux
> +_require_scratch
> +
> +RUN_TIME=$((600 * $TIME_FACTOR))

Hmm, does it really need 600s to run? I think it's better to limit the
runtime within 300s and make it an 'auto' test. I, personally, prefer a
"loop count" based test, I'd find out a minimum loop count that could
reproduce the ENOSPC problem more reliably on btrfs (for example, say
75%) and make the count scale with LOAD_FACTOR.

> +fs_size=$((15 * 1024 * 1024 * 1024))

And does it really need 15G on SCRATCH_DEV? A smaller fs size makes test
run faster, and gives the test more chance to be run, because not
everyone has a 15G SCRATCH_DEV.

> +_scratch_mkfs_sized $fs_size > $seqres.full 2>&1
> +_scratch_mount > $seqres.full 2>&1

Append to $seqres.full not overwrite.

> +
> +testfile1=$SCRATCH_MNT/testfile1
> +testfile2=$SCRATCH_MNT/testfile2
> +filesize1=$(((fs_size * 80) / 100))
> +filesize2=$(((fs_size * 5) / 100))

Better to have some comments on the filesizes chosen here. e.g. someone
may wonder that why it's testing ENOSPC condition with 85% full, not 99%
or 100%.

> +
> +do_test()
> +{
> + while [ -f $SCRATCH_MNT/run ]; do
> + $XFS_IO_PROG -fc "pwrite 0 $filesize1" $testfile1 > /dev/null
> + $XFS_IO_PROG -fc "pwrite 0 $filesize2" $testfile2 > /dev/null
> + rm -f $testfile1 $testfile2 

Trailing whitespace here.

> + done
> +}
> +
> +echo "Silence is golden"
> +touch $SCRATCH_MNT/run
> +do_test &
> +sleep $RUN_TIME
> +rm -f $SCRATCH_MNT/run
> +wait
> +
> +status=0
> +exit
> diff --git a/tests/generic/389.out b/tests/generic/389.out
> new file mode 100644
> index 000..e8c24bb
> --- /dev/null
> +++ b/tests/generic/389.out
> @@ -0,0 +1,2 @@
> +QA output created by 389
> +Silence is golden

Can you please rebase on top of current master? generic/389 is already
taken, and it makes applying & testing the patch a litter harder :)

> diff --git a/tests/generic/group b/tests/generic/group
> index fc32cfd..b6d4013 100644
> --- a/tests/generic/group
> +++ b/tests/generic/group
> @@ -391,3 +391,4 @@
>  386 auto quick quota
>  387 auto clone
>  388 auto log metadata
> +389 enospc

Perhaps we can add it to 'rw' group too.

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 2/3] btrfs-progs: receive: Introduce option to exam and dump send stream

[David Sterba]

On Tue, Nov 01, 2016 at 09:10:09AM +0800, Qu Wenruo wrote:
> I'm just a little curious about the stream samples.
> Should I create a send-dump-test to include all these samples? Or put 
> them somewhere else?

Misc tests would be fine for now, as you did in the new patchset.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] generic: create and delete files repeatedly to exercise ENOSPC behaviour

[Wang Xiaoguang]

In btrfs, sometimes though the number of created files' consumed disk space
are not larger than fs's free space, we can still get some ENOSPC error, it
may be that btrfs does not try hard to reclaim disk space(I have sent kernel
patch to resolve this kind of enospc error. Note, this false enospc error
will not always happen even in kernel without my fixing patch).

Currently only in btrfs, I get this ENOSPC error, xfs and ext4 work well.

Signed-off-by: Wang Xiaoguang 
---
 tests/generic/389 | 78 +++
 tests/generic/389.out |  2 ++
 tests/generic/group   |  1 +
 3 files changed, 81 insertions(+)
 create mode 100755 tests/generic/389
 create mode 100644 tests/generic/389.out

diff --git a/tests/generic/389 b/tests/generic/389
new file mode 100755
index 000..96bc12e
--- /dev/null
+++ b/tests/generic/389
@@ -0,0 +1,78 @@
+#! /bin/bash
+# FS QA Test 389
+#
+# Create and delete files repeatedly to exercise ENOSPC behaviour. 
+#
+#---
+# Copyright (c) 2016 Fujitsu.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#---
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1   # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+   cd /
+   rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# Modify as appropriate.
+_supported_fs generic
+_supported_os Linux
+_require_scratch
+
+RUN_TIME=$((600 * $TIME_FACTOR))
+fs_size=$((15 * 1024 * 1024 * 1024))
+_scratch_mkfs_sized $fs_size > $seqres.full 2>&1
+_scratch_mount > $seqres.full 2>&1
+
+testfile1=$SCRATCH_MNT/testfile1
+testfile2=$SCRATCH_MNT/testfile2
+filesize1=$(((fs_size * 80) / 100))
+filesize2=$(((fs_size * 5) / 100))
+
+do_test()
+{
+   while [ -f $SCRATCH_MNT/run ]; do
+   $XFS_IO_PROG -fc "pwrite 0 $filesize1" $testfile1 > /dev/null
+   $XFS_IO_PROG -fc "pwrite 0 $filesize2" $testfile2 > /dev/null
+   rm -f $testfile1 $testfile2 
+   done
+}
+
+echo "Silence is golden"
+touch $SCRATCH_MNT/run
+do_test &
+sleep $RUN_TIME
+rm -f $SCRATCH_MNT/run
+wait
+
+status=0
+exit
diff --git a/tests/generic/389.out b/tests/generic/389.out
new file mode 100644
index 000..e8c24bb
--- /dev/null
+++ b/tests/generic/389.out
@@ -0,0 +1,2 @@
+QA output created by 389
+Silence is golden
diff --git a/tests/generic/group b/tests/generic/group
index fc32cfd..b6d4013 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -391,3 +391,4 @@
 386 auto quick quota
 387 auto clone
 388 auto log metadata
+389 enospc
-- 
2.7.4



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 2/2] btrfs: fix false enospc for compression

[Wang Xiaoguang]

hi,

I have rebased these 2 patches against Linux 4.9-rc2, sorry for being late.
After applying these patches,  Stefan does not see any ENOSPC error :)
If you have free time, please have a check, thanks.

Regards,
Xiaoguang Wang

On 11/01/2016 06:18 PM, Wang Xiaoguang wrote:

When testing btrfs compression, sometimes we got ENOSPC error, though fs
still has much free space, xfstests generic/171, generic/172, generic/173,
generic/174, generic/175 can reveal this bug in my test environment when
compression is enabled.

After some debuging work, we found that it's btrfs_delalloc_reserve_metadata()
which sometimes tries to reserve plenty of metadata space, even for very small
data range. In btrfs_delalloc_reserve_metadata(), the number of metadata bytes
we try to reserve is calculated by the difference between outstanding_extents
and reserved_extents. Please see below case for how ENOSPC occurs:

   1, Buffered write 128MB data in unit of 128KB, so finially we'll have inode
outstanding extents be 1, and reserved_extents be 1024. Note it's
btrfs_merge_extent_hook() that merges these 128KB units into one big
outstanding extent, but do not change reserved_extents.

   2, When writing dirty pages, for compression, cow_file_range_async() will
split above big extent in unit of 128KB(compression extent size is 128KB).
When first split opeartion finishes, we'll have 2 outstanding extents and 1024
reserved extents, and just right now the currently generated ordered extent is
dispatched to run and complete, then btrfs_delalloc_release_metadata()(see
btrfs_finish_ordered_io()) will be called to release metadata, after that we
will have 1 outstanding extents and 1 reserved extents(also see logic in
drop_outstanding_extent()). Later cow_file_range_async() continues to handles
left data range[128KB, 128MB), and if no other ordered extent was dispatched
to run, there will be 1023 outstanding extents and 1 reserved extent.

   3, Now if another bufferd write for this file enters, then
btrfs_delalloc_reserve_metadata() will at least try to reserve metadata
for 1023 outstanding extents' metadata, for 16KB node size, it'll be 
1023*16384*2*8,
about 255MB, for 64K node size, it'll be 1023*65536*8*2, about 1GB metadata, so
obviously it's not sane and can easily result in enospc error.

The root cause is that for compression, its max extent size will no longer be
BTRFS_MAX_EXTENT_SIZE(128MB), it'll be 128KB, so current metadata reservation
method in btrfs is not appropriate or correct, here we introduce:
enum btrfs_metadata_reserve_type {
BTRFS_RESERVE_NORMAL,
BTRFS_RESERVE_COMPRESS,
};
and expand btrfs_delalloc_reserve_metadata() and btrfs_delalloc_reserve_space()
by adding a new enum btrfs_metadata_reserve_type argument. When a data range 
will
go through compression, we use BTRFS_RESERVE_COMPRESS to reserve metatata.
Meanwhile we introduce EXTENT_COMPRESS flag to mark a data range that will go
through compression path.

With this patch, we can fix these false enospc error for compression.

Signed-off-by: Wang Xiaoguang 
Tested-by: Holger Hoffstätte 
Tested-by: Stefan Priebe 
---
  fs/btrfs/ctree.h |  31 ++--
  fs/btrfs/extent-tree.c   |  55 +
  fs/btrfs/extent_io.c |  59 +-
  fs/btrfs/extent_io.h |   2 +
  fs/btrfs/file.c  |  28 +--
  fs/btrfs/free-space-cache.c  |   6 +-
  fs/btrfs/inode-map.c |   5 +-
  fs/btrfs/inode.c | 185 ---
  fs/btrfs/ioctl.c |  12 ++-
  fs/btrfs/relocation.c|  14 +++-
  fs/btrfs/tests/inode-tests.c |  15 ++--
  11 files changed, 315 insertions(+), 97 deletions(-)

diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 52a8535..457b106 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -98,6 +98,19 @@ static const int btrfs_csum_sizes[] = { 4 };
  
  #define BTRFS_DIRTY_METADATA_THRESH	SZ_32M
  
+/*

+ * for compression, max file extent size would be limited to 128K, so when
+ * reserving metadata for such delalloc writes, pass BTRFS_RESERVE_COMPRESS to
+ * btrfs_delalloc_reserve_metadata() or btrfs_delalloc_reserve_space() to
+ * calculate metadata, for none-compression, use BTRFS_RESERVE_NORMAL.
+ */
+enum btrfs_metadata_reserve_type {
+   BTRFS_RESERVE_NORMAL,
+   BTRFS_RESERVE_COMPRESS,
+};
+int inode_need_compress(struct inode *inode);
+u64 btrfs_max_extent_size(enum btrfs_metadata_reserve_type reserve_type);
+
  #define BTRFS_MAX_EXTENT_SIZE SZ_128M
  
  struct btrfs_mapping_tree {

@@ -2693,10 +2706,14 @@ int btrfs_subvolume_reserve_metadata(struct btrfs_root 
*root,
  void btrfs_subvolume_release_metadata(struct btrfs_root *root,
  struct btrfs_block_rsv *rsv,
  u64 qgroup_reserved);
-int btrfs_delalloc_reserve_metadata(struct 

[PATCH] btrfs: Remove some dead code

[Christophe JAILLET]

'btrfs_iget()' can not return an error pointer, so this test can be
removed.

Signed-off-by: Christophe JAILLET 
---
 fs/btrfs/free-space-cache.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index e4b48f377d3a..afd8b0c10acd 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -75,8 +75,6 @@ static struct inode *__lookup_free_space_inode(struct 
btrfs_root *root,
btrfs_release_path(path);
 
inode = btrfs_iget(root->fs_info->sb, , root, NULL);
-   if (!inode)
-   return ERR_PTR(-ENOENT);
if (IS_ERR(inode))
return inode;
if (is_bad_inode(inode)) {
-- 
2.9.3

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 2/2] btrfs: fix false enospc for compression

[Wang Xiaoguang]

When testing btrfs compression, sometimes we got ENOSPC error, though fs
still has much free space, xfstests generic/171, generic/172, generic/173,
generic/174, generic/175 can reveal this bug in my test environment when
compression is enabled.

After some debuging work, we found that it's btrfs_delalloc_reserve_metadata()
which sometimes tries to reserve plenty of metadata space, even for very small
data range. In btrfs_delalloc_reserve_metadata(), the number of metadata bytes
we try to reserve is calculated by the difference between outstanding_extents
and reserved_extents. Please see below case for how ENOSPC occurs:

  1, Buffered write 128MB data in unit of 128KB, so finially we'll have inode
outstanding extents be 1, and reserved_extents be 1024. Note it's
btrfs_merge_extent_hook() that merges these 128KB units into one big
outstanding extent, but do not change reserved_extents.

  2, When writing dirty pages, for compression, cow_file_range_async() will
split above big extent in unit of 128KB(compression extent size is 128KB).
When first split opeartion finishes, we'll have 2 outstanding extents and 1024
reserved extents, and just right now the currently generated ordered extent is
dispatched to run and complete, then btrfs_delalloc_release_metadata()(see
btrfs_finish_ordered_io()) will be called to release metadata, after that we
will have 1 outstanding extents and 1 reserved extents(also see logic in
drop_outstanding_extent()). Later cow_file_range_async() continues to handles
left data range[128KB, 128MB), and if no other ordered extent was dispatched
to run, there will be 1023 outstanding extents and 1 reserved extent.

  3, Now if another bufferd write for this file enters, then
btrfs_delalloc_reserve_metadata() will at least try to reserve metadata
for 1023 outstanding extents' metadata, for 16KB node size, it'll be 
1023*16384*2*8,
about 255MB, for 64K node size, it'll be 1023*65536*8*2, about 1GB metadata, so
obviously it's not sane and can easily result in enospc error.

The root cause is that for compression, its max extent size will no longer be
BTRFS_MAX_EXTENT_SIZE(128MB), it'll be 128KB, so current metadata reservation
method in btrfs is not appropriate or correct, here we introduce:
enum btrfs_metadata_reserve_type {
BTRFS_RESERVE_NORMAL,
BTRFS_RESERVE_COMPRESS,
};
and expand btrfs_delalloc_reserve_metadata() and btrfs_delalloc_reserve_space()
by adding a new enum btrfs_metadata_reserve_type argument. When a data range 
will
go through compression, we use BTRFS_RESERVE_COMPRESS to reserve metatata.
Meanwhile we introduce EXTENT_COMPRESS flag to mark a data range that will go
through compression path.

With this patch, we can fix these false enospc error for compression.

Signed-off-by: Wang Xiaoguang 
Tested-by: Holger Hoffstätte 
Tested-by: Stefan Priebe 
---
 fs/btrfs/ctree.h |  31 ++--
 fs/btrfs/extent-tree.c   |  55 +
 fs/btrfs/extent_io.c |  59 +-
 fs/btrfs/extent_io.h |   2 +
 fs/btrfs/file.c  |  28 +--
 fs/btrfs/free-space-cache.c  |   6 +-
 fs/btrfs/inode-map.c |   5 +-
 fs/btrfs/inode.c | 185 ---
 fs/btrfs/ioctl.c |  12 ++-
 fs/btrfs/relocation.c|  14 +++-
 fs/btrfs/tests/inode-tests.c |  15 ++--
 11 files changed, 315 insertions(+), 97 deletions(-)

diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 52a8535..457b106 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -98,6 +98,19 @@ static const int btrfs_csum_sizes[] = { 4 };
 
 #define BTRFS_DIRTY_METADATA_THRESHSZ_32M
 
+/*
+ * for compression, max file extent size would be limited to 128K, so when
+ * reserving metadata for such delalloc writes, pass BTRFS_RESERVE_COMPRESS to
+ * btrfs_delalloc_reserve_metadata() or btrfs_delalloc_reserve_space() to
+ * calculate metadata, for none-compression, use BTRFS_RESERVE_NORMAL.
+ */
+enum btrfs_metadata_reserve_type {
+   BTRFS_RESERVE_NORMAL,
+   BTRFS_RESERVE_COMPRESS,
+};
+int inode_need_compress(struct inode *inode);
+u64 btrfs_max_extent_size(enum btrfs_metadata_reserve_type reserve_type);
+
 #define BTRFS_MAX_EXTENT_SIZE SZ_128M
 
 struct btrfs_mapping_tree {
@@ -2693,10 +2706,14 @@ int btrfs_subvolume_reserve_metadata(struct btrfs_root 
*root,
 void btrfs_subvolume_release_metadata(struct btrfs_root *root,
  struct btrfs_block_rsv *rsv,
  u64 qgroup_reserved);
-int btrfs_delalloc_reserve_metadata(struct inode *inode, u64 num_bytes);
-void btrfs_delalloc_release_metadata(struct inode *inode, u64 num_bytes);
-int btrfs_delalloc_reserve_space(struct inode *inode, u64 start, u64 len);
-void btrfs_delalloc_release_space(struct inode *inode, u64 start, u64 len);
+int btrfs_delalloc_reserve_metadata(struct 

[PATCH 1/2] btrfs: improve inode's outstanding_extents computation

[Wang Xiaoguang]

This issue was revealed by modifying BTRFS_MAX_EXTENT_SIZE(128MB) to 64KB,
When modifying BTRFS_MAX_EXTENT_SIZE(128MB) to 64KB, fsstress test often
gets these warnings from btrfs_destroy_inode():
WARN_ON(BTRFS_I(inode)->outstanding_extents);
WARN_ON(BTRFS_I(inode)->reserved_extents);

Simple test program below can reproduce this issue steadily.
Note: you need to modify BTRFS_MAX_EXTENT_SIZE to 64KB to have test,
otherwise there won't be such WARNING.
#include 
#include 
#include 
#include 
#include 

int main(void)
{
int fd;
char buf[68 *1024];

memset(buf, 0, 68 * 1024);
fd = open("testfile", O_CREAT | O_EXCL | O_RDWR);
pwrite(fd, buf, 68 * 1024, 64 * 1024);
return;
}

When BTRFS_MAX_EXTENT_SIZE is 64KB, and buffered data range is:
64KB128K132KB
|---|---|
 64 + 4KB

1) for above data range, btrfs_delalloc_reserve_metadata() will reserve
metadata and set BTRFS_I(inode)->outstanding_extents to 2.
(68KB + 64KB - 1) / 64KB == 2

Outstanding_extents: 2

2) then btrfs_dirty_page() will be called to dirty pages and set
EXTENT_DELALLOC flag. In this case, btrfs_set_bit_hook() will be called
twice.
The 1st set_bit_hook() call will set DEALLOC flag for the first 64K.
64KB128KB
|---|
64KB DELALLOC
Outstanding_extents: 2

Set_bit_hooks() uses FIRST_DELALLOC flag to avoid re-increase
outstanding_extents counter.
So for 1st set_bit_hooks() call, it won't modify outstanding_extents,
it's still 2.

Then FIRST_DELALLOC flag is *CLEARED*.

3) 2nd btrfs_set_bit_hook() call.
Because FIRST_DELALLOC have been cleared by previous set_bit_hook(),
btrfs_set_bit_hook() will increase BTRFS_I(inode)->outstanding_extents by
one, so now BTRFS_I(inode)->outstanding_extents is 3.
64KB128KB132KB
|---||
64K DELALLOC   4K DELALLOC
Outstanding_extents: 3

But the correct outstanding_extents number should be 2, not 3.
The 2nd btrfs_set_bit_hook() call just screwed up this, and leads to the
WARN_ON().

Normally, we can solve it by only increasing outstanding_extents in
set_bit_hook().
But the problem is for delalloc_reserve/release_metadata(), we only have
a 'length' parameter, and calculate in-accurate outstanding_extents.
If we only rely on set_bit_hook() release_metadata() will crew things up
as it will decrease inaccurate number.

So the fix we use is:
1) Increase *INACCURATE* outstanding_extents at delalloc_reserve_meta
   Just as a place holder.
2) Increase *accurate* outstanding_extents at set_bit_hooks()
   This is the real increaser.
3) Decrease *INACCURATE* outstanding_extents before returning
   This makes outstanding_extents to correct value.

For 128M BTRFS_MAX_EXTENT_SIZE, due to limitation of
__btrfs_buffered_write(), each iteration will only handle about 2MB
data.
So btrfs_dirty_pages() won't need to handle cases cross 2 extents.

Signed-off-by: Wang Xiaoguang 
Tested-by: Holger Hoffstätte 
Tested-by: Stefan Priebe 
---
 fs/btrfs/ctree.h |  2 ++
 fs/btrfs/inode.c | 65 ++--
 fs/btrfs/ioctl.c |  6 ++
 3 files changed, 62 insertions(+), 11 deletions(-)

diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 0b8ce2b..52a8535 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -3139,6 +3139,8 @@ int btrfs_start_delalloc_roots(struct btrfs_fs_info 
*fs_info, int delay_iput,
   int nr);
 int btrfs_set_extent_delalloc(struct inode *inode, u64 start, u64 end,
  struct extent_state **cached_state, int dedupe);
+int btrfs_set_extent_defrag(struct inode *inode, u64 start, u64 end,
+   struct extent_state **cached_state);
 int btrfs_create_subvol_root(struct btrfs_trans_handle *trans,
 struct btrfs_root *new_root,
 struct btrfs_root *parent_root,
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 2b790bd..e427b81 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -1601,6 +1601,9 @@ static void btrfs_split_extent_hook(struct inode *inode,
if (!(orig->state & EXTENT_DELALLOC))
return;
 
+   if (btrfs_is_free_space_inode(inode))
+   return;
+
size = orig->end - orig->start + 1;
if (size > BTRFS_MAX_EXTENT_SIZE) {
u64 num_extents;
@@ -1643,6 +1646,9 @@ static void btrfs_merge_extent_hook(struct inode *inode,
if (!(other->state & 

Re: [PATCH v3 2/4] btrfs-progs: introduce new send-dump object

[David Sterba]

On Tue, Nov 01, 2016 at 04:01:44PM +0800, Qu Wenruo wrote:
> +/*
> + * Underlying PRINT_DUMP, the only difference is how we handle
> + * the full path.
> + */
> +static int __print_dump(int subvol, void *user, const char *path,
> + const char *title, const char *fmt, ...)

printf-like the __attribute__ ((format (printf, ...)))

> +{
> + struct btrfs_dump_send_args *r = user;
> + char real_title[TITLE_WIDTH + 1] = { 0 };
> + char full_path[PATH_MAX] = {0};
> + char *out_path;
> + va_list args;
> + int ret;
> +
> + if (subvol) {
> + PATH_CAT_OR_RET(title, r->full_subvol_path, r->root_path, path, 
> ret);
> + out_path = r->full_subvol_path;
> + } else {
> + PATH_CAT_OR_RET(title, full_path, r->full_subvol_path, path, 
> ret);
> + out_path = full_path;
> + }
> + string_escape_inplace(out_path, " \n\t\\");
> +
> + /* Append ':' to title */
> + strncpy(real_title, title, TITLE_WIDTH - 1);
> + strncat(real_title, ":", TITLE_WIDTH);

I'd rather avoid such string operations, ':', just print everything.

> +
> + /* Unified header, */
> + printf("%-*s%-*s", TITLE_WIDTH, real_title, PATH_WIDTH, out_path);

PATH_WIDTH is used only here, please hardcode it into the format string.

The rest of the patch looks good. I think I've seen some artifacts in
the output, but we can tune this later.

> + ret = strftime(dest, max_size, "%Y-%m-%d %H:%M:%S", tm);

We should use the RFC 3339 format, as it's standardized and also is a
string without whitespace.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 1/4] btrfs-progs: utils: Introduce function to escape characters

[David Sterba]

On Tue, Nov 01, 2016 at 04:01:43PM +0800, Qu Wenruo wrote:
> Introduce new function, escape_string_inplace(), to escape specified
> characters in place.

Sorry, the pointer to seq_path was misleading. The actual escape
function is mangle_path and it copies one string to another. As we just
print the path, we can simply switch and call putchar.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v4] generic: make 17[1-4] work well when btrfs compression is enabled

[Wang Xiaoguang]

hi Darrick,

Common/populate needs xfs_io supports falloc and fpunch,
so I didn't put _fill_fs() in common/populate.

Regards,
Xiaoguang Wang

On 11/01/2016 04:45 PM, Wang Xiaoguang wrote:

When enabling btrfs compression, original codes can not fill fs
correctly, here we introduce _fill_fs() in common/rc, which'll keep
creating and writing files until enospc error occurs. Note _fill_fs
is copied from tests/generic/256, but with some minor modifications.

Signed-off-by: Wang Xiaoguang 
---
V2: In common/, I did't find an existing function suitable for
 these 4 test cases to fill fs, so I still use _pwrite_byte() with
 a big enough file length fo fill fs. Note, for btrfs, metadata space
 still is not full, only data space is full, but it's OK for these
 4 test cases.

 All these 4 cases pass in xfs and btrfs(without compression), if
 btrfs has compression enabled, these 4 cases will fail for false
 enospc error, I have sent kernel patches to fix this bug.

V3: Introduce  _fill_fs in common/rc to fill fs.
V4: Fix some issues suggested by Eryu and Darrick.
---
  common/rc | 68 +++
  tests/generic/171 |  3 +--
  tests/generic/172 |  4 ++--
  tests/generic/173 |  3 +--
  tests/generic/174 |  3 +--
  tests/generic/256 | 65 
  6 files changed, 78 insertions(+), 68 deletions(-)

diff --git a/common/rc b/common/rc
index 7a9fc90..7628a0e 100644
--- a/common/rc
+++ b/common/rc
@@ -4003,6 +4003,74 @@ _require_xfs_mkfs_without_validation()
fi
  }
  
+# Fill a file system by repeatedly creating files in the given folder

+# starting with the given file size.  Files are reduced in size when
+# they can no longer fit until no more files can be created.
+_fill_fs()
+{
+   local file_size=$1
+   local dir=$2
+   local block_size=$3
+   local switch_user=$4
+   local file_count=1
+   local bytes_written=0
+   local use_falloc=1;
+
+   if [ $# -ne 4 ]; then
+   echo "Usage: _fill_fs filesize dir blocksize switch_user"
+   exit 1
+   fi
+
+   if [ $switch_user -eq 0 ]; then
+   mkdir -p $dir
+   else
+   _user_do "mkdir -p $dir"
+   fi
+   if [ ! -d $dir ]; then
+   return 0;
+   fi
+
+   testio=`$XFS_IO_PROG -F -fc "falloc 0 $block_size" $dir/$$.xfs_io 2>&1`
+   echo $testio | grep -q "not found" && use_falloc=0
+   echo $testio | grep -q "Operation not supported" && use_falloc=0
+
+   if [ $file_size -lt $block_size ]; then
+   $file_size = $block_size
+   fi
+
+   while [ $file_size -ge $block_size ]; do
+   bytes_written=0
+   if [ $switch_user -eq 0 ]; then
+   if [ $use_falloc -eq 0 ]; then
+   $XFS_IO_PROG -fc "pwrite -b 8388608 0 
$file_size" \
+   $dir/$file_count
+   else
+   $XFS_IO_PROG -fc "falloc 0 $file_size" \
+   $dir/$file_count
+   fi
+   else
+   if [ $use_falloc -eq 0 ]; then
+   _user_do "$XFS_IO_PROG -f -c \"pwrite -b 
8388608 0 \
+   $file_size\" $dir/$file_count"
+   else
+   _user_do "$XFS_IO_PROG -f -c \"falloc 0 \
+   $file_size\" $dir/$file_count"
+   fi
+   fi
+
+   if [ -f $dir/$file_count ]; then
+   bytes_written=$(stat -c '%s' $dir/$file_count)
+   fi
+
+   # If there was no room to make the file, then divide it in
+   # half, and keep going
+   if [ $bytes_written -lt $file_size ]; then
+   file_size=$((file_size / 2))
+   fi
+   file_count=$((file_count + 1))
+   done
+}
+
  init_rc
  
  

diff --git a/tests/generic/171 b/tests/generic/171
index a69f798..906e4db 100755
--- a/tests/generic/171
+++ b/tests/generic/171
@@ -76,8 +76,7 @@ sync
  
  echo "Allocate the rest of the space"

  nr_free=$(stat -f -c '%f' $testdir)
-touch $testdir/file0 $testdir/file1
-_pwrite_byte 0x61 0 $((blksz * nr_free)) $testdir/eat_my_space >> $seqres.full 
2>&1
+_fill_fs $((nr_free * blksz)) $testdir/space $blksz 0 >> $seqres.full 2>&1
  sync
  
  echo "CoW the big file"

diff --git a/tests/generic/172 b/tests/generic/172
index 8192290..f6fcdc9 100755
--- a/tests/generic/172
+++ b/tests/generic/172
@@ -57,6 +57,7 @@ testdir=$SCRATCH_MNT/test-$seq
  mkdir $testdir
  
  echo "Reformat with appropriate size"

+blksz="$(get_block_size $testdir)"
  umount $SCRATCH_MNT
  
  

[PATCH v4] generic: make 17[1-4] work well when btrfs compression is enabled

[Wang Xiaoguang]

When enabling btrfs compression, original codes can not fill fs
correctly, here we introduce _fill_fs() in common/rc, which'll keep
creating and writing files until enospc error occurs. Note _fill_fs
is copied from tests/generic/256, but with some minor modifications.

Signed-off-by: Wang Xiaoguang 
---
V2: In common/, I did't find an existing function suitable for
these 4 test cases to fill fs, so I still use _pwrite_byte() with
a big enough file length fo fill fs. Note, for btrfs, metadata space
still is not full, only data space is full, but it's OK for these
4 test cases.

All these 4 cases pass in xfs and btrfs(without compression), if
btrfs has compression enabled, these 4 cases will fail for false
enospc error, I have sent kernel patches to fix this bug.

V3: Introduce  _fill_fs in common/rc to fill fs.
V4: Fix some issues suggested by Eryu and Darrick.
---
 common/rc | 68 +++
 tests/generic/171 |  3 +--
 tests/generic/172 |  4 ++--
 tests/generic/173 |  3 +--
 tests/generic/174 |  3 +--
 tests/generic/256 | 65 
 6 files changed, 78 insertions(+), 68 deletions(-)

diff --git a/common/rc b/common/rc
index 7a9fc90..7628a0e 100644
--- a/common/rc
+++ b/common/rc
@@ -4003,6 +4003,74 @@ _require_xfs_mkfs_without_validation()
fi
 }
 
+# Fill a file system by repeatedly creating files in the given folder
+# starting with the given file size.  Files are reduced in size when
+# they can no longer fit until no more files can be created.
+_fill_fs()
+{
+   local file_size=$1
+   local dir=$2
+   local block_size=$3
+   local switch_user=$4
+   local file_count=1
+   local bytes_written=0
+   local use_falloc=1;
+
+   if [ $# -ne 4 ]; then
+   echo "Usage: _fill_fs filesize dir blocksize switch_user"
+   exit 1
+   fi
+
+   if [ $switch_user -eq 0 ]; then
+   mkdir -p $dir
+   else
+   _user_do "mkdir -p $dir"
+   fi
+   if [ ! -d $dir ]; then
+   return 0;
+   fi
+
+   testio=`$XFS_IO_PROG -F -fc "falloc 0 $block_size" $dir/$$.xfs_io 2>&1`
+   echo $testio | grep -q "not found" && use_falloc=0
+   echo $testio | grep -q "Operation not supported" && use_falloc=0
+
+   if [ $file_size -lt $block_size ]; then
+   $file_size = $block_size
+   fi
+
+   while [ $file_size -ge $block_size ]; do
+   bytes_written=0
+   if [ $switch_user -eq 0 ]; then
+   if [ $use_falloc -eq 0 ]; then
+   $XFS_IO_PROG -fc "pwrite -b 8388608 0 
$file_size" \
+   $dir/$file_count
+   else
+   $XFS_IO_PROG -fc "falloc 0 $file_size" \
+   $dir/$file_count
+   fi
+   else
+   if [ $use_falloc -eq 0 ]; then
+   _user_do "$XFS_IO_PROG -f -c \"pwrite -b 
8388608 0 \
+   $file_size\" $dir/$file_count"
+   else
+   _user_do "$XFS_IO_PROG -f -c \"falloc 0 \
+   $file_size\" $dir/$file_count"
+   fi
+   fi
+
+   if [ -f $dir/$file_count ]; then
+   bytes_written=$(stat -c '%s' $dir/$file_count)
+   fi
+
+   # If there was no room to make the file, then divide it in
+   # half, and keep going
+   if [ $bytes_written -lt $file_size ]; then
+   file_size=$((file_size / 2))
+   fi
+   file_count=$((file_count + 1))
+   done
+}
+
 init_rc
 
 

diff --git a/tests/generic/171 b/tests/generic/171
index a69f798..906e4db 100755
--- a/tests/generic/171
+++ b/tests/generic/171
@@ -76,8 +76,7 @@ sync
 
 echo "Allocate the rest of the space"
 nr_free=$(stat -f -c '%f' $testdir)
-touch $testdir/file0 $testdir/file1
-_pwrite_byte 0x61 0 $((blksz * nr_free)) $testdir/eat_my_space >> $seqres.full 
2>&1
+_fill_fs $((nr_free * blksz)) $testdir/space $blksz 0 >> $seqres.full 2>&1
 sync
 
 echo "CoW the big file"
diff --git a/tests/generic/172 b/tests/generic/172
index 8192290..f6fcdc9 100755
--- a/tests/generic/172
+++ b/tests/generic/172
@@ -57,6 +57,7 @@ testdir=$SCRATCH_MNT/test-$seq
 mkdir $testdir
 
 echo "Reformat with appropriate size"
+blksz="$(get_block_size $testdir)"
 umount $SCRATCH_MNT
 
 file_size=$((168 * 1024 * 1024))
@@ -72,8 +73,7 @@ _cp_reflink $testdir/bigfile $testdir/clonefile
 sync
 
 echo "Allocate the rest of the space"
-touch $testdir/file0 $testdir/file1
-_pwrite_byte 0x61 0 $fs_size 

[PATCH v3 2/4] btrfs-progs: introduce new send-dump object

[Qu Wenruo]

Introduce send-dump.[ch] which implements a new btrfs_send_ops to
exam and output all operations inside a send stream.

It has a better output format than the old and no longer compilable
send-test tool, but still tries to be script friendly.

Provides the basis for later "inspect-internal dump-send" command.

Signed-off-by: Qu Wenruo 
Signed-off-by: David Sterba 
---
 Makefile.in |   2 +-
 send-dump.c | 299 
 send-dump.h |  29 ++
 3 files changed, 329 insertions(+), 1 deletion(-)
 create mode 100644 send-dump.c
 create mode 100644 send-dump.h

diff --git a/Makefile.in b/Makefile.in
index b53cf2c..c535c19 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -93,7 +93,7 @@ objects = ctree.o disk-io.o kernel-lib/radix-tree.o 
extent-tree.o print-tree.o \
  extent-cache.o extent_io.o volumes.o utils.o repair.o \
  qgroup.o raid56.o free-space-cache.o kernel-lib/list_sort.o props.o \
  ulist.o qgroup-verify.o backref.o string-table.o task-utils.o \
- inode.o file.o find-root.o free-space-tree.o help.o
+ inode.o file.o find-root.o free-space-tree.o help.o send-dump.o
 cmds_objects = cmds-subvolume.o cmds-filesystem.o cmds-device.o cmds-scrub.o \
   cmds-inspect.o cmds-balance.o cmds-send.o cmds-receive.o \
   cmds-quota.o cmds-qgroup.o cmds-replace.o cmds-check.o \
diff --git a/send-dump.c b/send-dump.c
new file mode 100644
index 000..b4d8a19
--- /dev/null
+++ b/send-dump.c
@@ -0,0 +1,299 @@
+/*
+ * Copyright (C) 2016 Fujitsu.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program.
+ */
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include "utils.h"
+#include "commands.h"
+#include "send-utils.h"
+#include "send-stream.h"
+#include "send-dump.h"
+
+#define PATH_CAT_OR_RET(function_name, outpath, path1, path2, ret) \
+({ \
+   ret = path_cat_out(outpath, path1, path2);  \
+   if (ret < 0) {  \
+   error("%s: path invalid: %s\n", function_name, path2);  \
+   return ret; \
+   }   \
+})
+
+#define TITLE_WIDTH16
+#define PATH_WIDTH 32
+
+/*
+ * Underlying PRINT_DUMP, the only difference is how we handle
+ * the full path.
+ */
+static int __print_dump(int subvol, void *user, const char *path,
+   const char *title, const char *fmt, ...)
+{
+   struct btrfs_dump_send_args *r = user;
+   char real_title[TITLE_WIDTH + 1] = { 0 };
+   char full_path[PATH_MAX] = {0};
+   char *out_path;
+   va_list args;
+   int ret;
+
+   if (subvol) {
+   PATH_CAT_OR_RET(title, r->full_subvol_path, r->root_path, path, 
ret);
+   out_path = r->full_subvol_path;
+   } else {
+   PATH_CAT_OR_RET(title, full_path, r->full_subvol_path, path, 
ret);
+   out_path = full_path;
+   }
+   string_escape_inplace(out_path, " \n\t\\");
+
+   /* Append ':' to title */
+   strncpy(real_title, title, TITLE_WIDTH - 1);
+   strncat(real_title, ":", TITLE_WIDTH);
+
+   /* Unified header, */
+   printf("%-*s%-*s", TITLE_WIDTH, real_title, PATH_WIDTH, out_path);
+   va_start(args, fmt);
+   /* Operation specified ones */
+   vprintf(fmt, args);
+   va_end(args);
+   printf("\n");
+   return 0;
+}
+
+/* For subvolume/snapshot operation only */
+#define PRINT_DUMP_SUBVOL(user, path, title, fmt, ...) \
+   __print_dump(1, user, path, title, fmt, ##__VA_ARGS__)
+
+/* For other operations */
+#define PRINT_DUMP(user, path, title, fmt, ...) \
+   __print_dump(0, user, path, title, fmt, ##__VA_ARGS__)
+
+static int print_subvol(const char *path, const u8 *uuid, u64 ctransid,
+   void *user)
+{
+   char uuid_str[BTRFS_UUID_UNPARSED_SIZE];
+
+   uuid_unparse(uuid, uuid_str);
+
+   return PRINT_DUMP_SUBVOL(user, path, "subvol", "uuid=%s transid=%llu",
+uuid_str, ctransid);
+}
+
+static int print_snapshot(const char *path, const u8 *uuid, u64 ctransid,
+  

[PATCH v3 1/4] btrfs-progs: utils: Introduce function to escape characters

[Qu Wenruo]

Introduce new function, escape_string_inplace(), to escape specified
characters in place.

Signed-off-by: Qu Wenruo 
---
 utils.c | 17 +
 utils.h |  2 ++
 2 files changed, 19 insertions(+)

diff --git a/utils.c b/utils.c
index 3f54245..69faae1 100644
--- a/utils.c
+++ b/utils.c
@@ -4251,3 +4251,20 @@ unsigned int rand_range(unsigned int upper)
 */
return (unsigned int)(jrand48(rand_seed) % upper);
 }
+
+void string_escape_inplace(char *restrict string, char *restrict escape_chars)
+{
+   int i, j;
+
+   for (i = 0; i < strlen(escape_chars); i++) {
+   j = 0;
+   while (j < strlen(string)) {
+   if (string[j] == escape_chars[i]) {
+   memmove(string + j, string + j + 1,
+   strlen(string) -j);
+   continue;
+   }
+   j++;
+   }
+   }
+}
diff --git a/utils.h b/utils.h
index 1a2dbcd..b36d411 100644
--- a/utils.h
+++ b/utils.h
@@ -457,4 +457,6 @@ unsigned int rand_range(unsigned int upper);
 /* Also allow setting the seed manually */
 void init_rand_seed(u64 seed);
 
+void string_escape_inplace(char *restrict string, char *restrict escape_chars);
+
 #endif
-- 
2.10.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v3 4/4] btrfs-progs: remove send-test tool

[Qu Wenruo]

Since new "receive --dump" has better output and structure, it's time
to remove old and function-weak send-test tool.

Signed-off-by: Qu Wenruo 
Signed-off-by: David Sterba 
---
 Makefile.in |   6 +-
 send-test.c | 447 
 2 files changed, 1 insertion(+), 452 deletions(-)
 delete mode 100644 send-test.c

diff --git a/Makefile.in b/Makefile.in
index c535c19..9295d32 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -428,10 +428,6 @@ test-ioctl: ioctl-test ioctl-test-32 ioctl-test-64
$(Q)./ioctl-test-32 > ioctl-test-32.log
$(Q)./ioctl-test-64 > ioctl-test-64.log
 
-send-test: $(objects) $(libs) send-test.o
-   @echo "[LD] $@"
-   $(Q)$(CC) $(CFLAGS) -o send-test $(objects) $(libs) send-test.o 
$(LDFLAGS) $(LIBS)
-
 library-test: $(libs_shared) library-test.o
@echo "[LD] $@"
$(Q)$(CC) $(CFLAGS) -o library-test library-test.o $(LDFLAGS) -lbtrfs
@@ -464,7 +460,7 @@ clean: $(CLEANDIRS)
@echo "Cleaning"
$(Q)$(RM) -f -- $(progs) cscope.out *.o *.o.d \
kernel-lib/*.o kernel-lib/*.o.d \
- dir-test ioctl-test quick-test send-test library-test 
library-test-static \
+ dir-test ioctl-test quick-test library-test library-test-static \
  btrfs.static mkfs.btrfs.static \
  $(check_defs) \
  $(libs) $(lib_links) \
diff --git a/send-test.c b/send-test.c
deleted file mode 100644
index 4645b89..000
--- a/send-test.c
+++ /dev/null
@@ -1,447 +0,0 @@
-/*
- * Copyright (C) 2013 SUSE.  All rights reserved.
- *
- * This code is adapted from cmds-send.c and cmds-receive.c,
- * Both of which are:
- *
- * Copyright (C) 2012 Alexander Block.  All rights reserved.
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public
- * License v2 as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
- * General Public License for more details.
- *
- * You should have received a copy of the GNU General Public
- * License along with this program; if not, write to the
- * Free Software Foundation, Inc., 59 Temple Place - Suite 330,
- * Boston, MA 021110-1307, USA.
- */
-
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-
-/*
- * This should be compilable without the rest of the btrfs-progs
- * source distribution.
- */
-#if BTRFS_FLAT_INCLUDES
-#include "send-utils.h"
-#include "send-stream.h"
-#else
-#include 
-#include 
-#endif /* BTRFS_FLAT_INCLUDES */
-
-static int pipefd[2];
-struct btrfs_ioctl_send_args io_send = {0, };
-static char *subvol_path;
-static char *root_path;
-
-struct recv_args {
-   char *full_subvol_path;
-   char *root_path;
-};
-
-void usage(int error)
-{
-   printf("send-test  \n");
-   if (error)
-   exit(error);
-}
-
-static int print_subvol(const char *path, const u8 *uuid, u64 ctransid,
-   void *user)
-{
-   struct recv_args *r = user;
-   char uuid_str[BTRFS_UUID_UNPARSED_SIZE];
-
-   r->full_subvol_path = path_cat(r->root_path, path);
-   uuid_unparse(uuid, uuid_str);
-
-   printf("subvol\t%s\t%llu\t%s\n", uuid_str,
-  (unsigned long long)ctransid, r->full_subvol_path);
-
-   return 0;
-}
-
-static int print_snapshot(const char *path, const u8 *uuid, u64 ctransid,
- const u8 *parent_uuid, u64 parent_ctransid,
- void *user)
-{
-   struct recv_args *r = user;
-   char uuid_str[BTRFS_UUID_UNPARSED_SIZE];
-   char parent_uuid_str[BTRFS_UUID_UNPARSED_SIZE];
-
-   r->full_subvol_path = path_cat(r->root_path, path);
-   uuid_unparse(uuid, uuid_str);
-   uuid_unparse(parent_uuid, parent_uuid_str);
-
-   printf("snapshot\t%s\t%llu\t%s\t%llu\t%s\n", uuid_str,
-  (unsigned long long)ctransid, parent_uuid_str,
-  (unsigned long long)parent_ctransid, r->full_subvol_path);
-
-   return 0;
-}
-
-static int print_mkfile(const char *path, void *user)
-{
-   struct recv_args *r = user;
-   char *full_path = path_cat(r->full_subvol_path, path);
-
-   printf("mkfile\t%s\n", full_path);
-
-   free(full_path);
-   return 0;
-}
-
-static int print_mkdir(const char *path, void *user)
-{
-   struct recv_args *r = user;
-   char *full_path = path_cat(r->full_subvol_path, path);
-
-   printf("mkdir\t%s\n", full_path);
-
-   free(full_path);
-   return 0;
-}
-
-static int print_mknod(const char *path, u64 mode, u64 dev, void *user)
-{
-   struct recv_args *r = user;
-   char *full_path = 

[PATCH 5/5] btrfs-progs: misc-test: Add send stream dump test

[Qu Wenruo]

* PLEASE DON'T MERGE THIS PATCH*

With 2 send stream which contains all operations for user to check the
output.

This is just used for checking output format and find any possible
deflects or ugly layout.

It doesn't really checks any thing and doesn't follow normal test output
redirection.

Signed-off-by: Qu Wenruo 
---
 .../016-send-dump-output/creation.stream.xz| Bin 0 -> 984 bytes
 .../016-send-dump-output/deletion.stream.xz| Bin 0 -> 408 bytes
 tests/misc-tests/016-send-dump-output/test.sh  |  23 +
 3 files changed, 23 insertions(+)
 create mode 100644 tests/misc-tests/016-send-dump-output/creation.stream.xz
 create mode 100644 tests/misc-tests/016-send-dump-output/deletion.stream.xz
 create mode 100755 tests/misc-tests/016-send-dump-output/test.sh

diff --git a/tests/misc-tests/016-send-dump-output/creation.stream.xz 
b/tests/misc-tests/016-send-dump-output/creation.stream.xz
new file mode 100644
index 
..521c5d97170e7ad41ac8b3305461df09e2cb4c86
GIT binary patch
literal 984
zcmV;}11J3bH+ooF000E$*0e?f03iVu0001VFXf})8bkw?T>vp13XGw}P}F=IbdLR(
z>U`@9XCIGr1@6OEPWlaQgT5hCS#G{B*Zx$SeQxx(I;AlUUp>USnJ^DpJDxOMlV57u
zelh2}f67+r{m^CB^YzYr3tjQ|k)GN6FCPlJzZl~%dVK~u^utMa*9rwMg9+N$g{SND
zcZ-NRsrbN=(OpirB^JtZ_@@|0ko=lrZK$K~N%4wfsccFuJN7=oxM?(C{$t
zrJ3+}UkQ}_E*UQZWqzP7z377g)WK7w_xN6@JI=Ju(YvI$Mt_bx^7=ASE}(b8_B}4i
zasV`w=9w+*M6KB1PI0epqZRs16o(yJ_TJRHtLJs)H=r2l{lUm#HgtEO+R509mJ9Dp
zH53E2E>5wavINe=AaNq|VGT6MOCuv!R7O%jU@r%Yrdpfo?l6$2s-G=tmJ

[PATCH v3 0/4] Introduce dump option for btrfs-receive

[Qu Wenruo]

The branch can be fetched from github:
https://github.com/adam900710/btrfs-progs/tree/send_dump

The branch doesn't contain the fake test case.

Introduce new "--dump" option for btrfs-receive, which will exam and dump
metadata info of a send stream.
This is quite handy to debug send stream.

Since such function is provided by old send-test tool, which doesn't even
compile now, remove the old send-test tool.

changelog:
v2:
  Move from inspect subcommand to receive subcommand.
v3:
  Add output for ctime/mtime/atime
  (Human readable local time like "-11-01 11:11:11")
  Rearrange the output from "key1: value1, key2: value2" to
  "key1=value1 key2=value2". Suggested by David
  Rename macro path_cat_or_error() to PATH_CAT_OR_RET(). Suggested by David
  Change pointer to array to avoid memory allocation error. Suggested by David
  Add the 5th patch to add a fake test case to show how the new output looks
  like.

Qu Wenruo (4):
  btrfs-progs: utils: Introduce function to escape characters
  btrfs-progs: introduce new send-dump object
  btrfs-progs: receive: introduce option to dump send stream
  btrfs-progs: remove send-test tool

 Documentation/btrfs-receive.asciidoc |  15 +-
 Makefile.in  |   8 +-
 cmds-receive.c   |  35 ++-
 send-dump.c  | 299 +++
 send-dump.h  |  29 +++
 send-test.c  | 447 ---
 utils.c  |  17 ++
 utils.h  |   2 +
 8 files changed, 394 insertions(+), 458 deletions(-)
 create mode 100644 send-dump.c
 create mode 100644 send-dump.h
 delete mode 100644 send-test.c

-- 
2.10.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v3 3/4] btrfs-progs: receive: introduce option to dump send stream

[Qu Wenruo]

Introduce new option, '--dump' for receive subcommand.

With this command, user can dump the metadata of a send stream.
Which is quite useful for education purpose or bug reporting.

Signed-off-by: Qu Wenruo 
Signed-off-by: David Sterba 
---
 Documentation/btrfs-receive.asciidoc | 15 ++-
 cmds-receive.c   | 35 +++
 2 files changed, 45 insertions(+), 5 deletions(-)

diff --git a/Documentation/btrfs-receive.asciidoc 
b/Documentation/btrfs-receive.asciidoc
index e246603..f12e949 100644
--- a/Documentation/btrfs-receive.asciidoc
+++ b/Documentation/btrfs-receive.asciidoc
@@ -9,12 +9,19 @@ SYNOPSIS
 
 *btrfs receive* [options] 
 
+or
+
+*btrfs receive* --dump [options]
+
 DESCRIPTION
 ---
 
 Receive a stream of changes and replicate one or more subvolumes that were
 previously used with *btrfs send* The received subvolumes are stored to
-'path'.
+'path', if '--dump' option is not given.
+
+If '--dump' option is given, *btrfs receive* will only do the validation of
+the stream, and print the stream metadata.
 
 *btrfs receive* will fail int the following cases:
 
@@ -56,6 +63,12 @@ By default the mountpoint is searched in '/proc/self/mounts'.
 If you do not have '/proc', eg. in a chroot environment, use this option to 
tell
 us where this filesystem is mounted.
 
+--dump::
+print the stream metadata
++
+Does not accept the 'path' parameter. So with this option, *btrfs receive* 
won't
+modify your filesystem, and can be run by non-privileged users.
+
 EXIT STATUS
 ---
 *btrfs receive* returns a zero exit status if it succeeds. Non zero is
diff --git a/cmds-receive.c b/cmds-receive.c
index d0525bf..1dcdb1a 100644
--- a/cmds-receive.c
+++ b/cmds-receive.c
@@ -49,6 +49,7 @@
 #include "send.h"
 #include "send-stream.h"
 #include "send-utils.h"
+#include "send-dump.h"
 
 static int g_verbose = 0;
 
@@ -1214,6 +1215,7 @@ int cmd_receive(int argc, char **argv)
struct btrfs_receive r;
int receive_fd = fileno(stdin);
u64 max_errors = 1;
+   int dump = 0;
int ret = 0;
 
memset(, 0, sizeof(r));
@@ -1226,9 +1228,11 @@ int cmd_receive(int argc, char **argv)
 
while (1) {
int c;
+   enum { GETOPT_VAL_DUMP = 257 };
static const struct option long_opts[] = {
{ "max-errors", required_argument, NULL, 'E' },
{ "chroot", no_argument, NULL, 'C' },
+   { "dump", no_argument, NULL, GETOPT_VAL_DUMP },
{ NULL, 0, NULL, 0 }
};
 
@@ -1265,6 +1269,9 @@ int cmd_receive(int argc, char **argv)
goto out;
}
break;
+   case GETOPT_VAL_DUMP:
+   dump = 1;
+   break;
case '?':
default:
error("receive args invalid");
@@ -1272,7 +1279,9 @@ int cmd_receive(int argc, char **argv)
}
}
 
-   if (check_argc_exact(argc - optind, 1))
+   if (dump && check_argc_exact(argc - optind, 0))
+   usage(cmd_receive_usage);
+   if (!dump && check_argc_exact(argc - optind, 1))
usage(cmd_receive_usage);
 
tomnt = argv[optind];
@@ -1285,17 +1294,33 @@ int cmd_receive(int argc, char **argv)
}
}
 
-   ret = do_receive(, tomnt, realmnt, receive_fd, max_errors);
+   if (dump) {
+   struct btrfs_dump_send_args dump_args;
+
+   dump_args.root_path[0] = '.';
+   dump_args.root_path[1] = '\0';
+   dump_args.full_subvol_path[0] = '.';
+   dump_args.full_subvol_path[1] = '\0';
+   ret = btrfs_read_and_process_send_stream(receive_fd,
+   _print_send_ops, _args, 0, 0);
+   if (ret < 0)
+   error("failed to dump the send stream: %s",
+ strerror(-ret));
+   } else {
+   ret = do_receive(, tomnt, realmnt, receive_fd, max_errors);
+   }
+
if (receive_fd != fileno(stdin))
close(receive_fd);
-
 out:
 
return !!ret;
 }
 
 const char * const cmd_receive_usage[] = {
-   "btrfs receive [-ve] [-f ] [--max-errors ] ",
+   "btrfs receive [options] ",
+   "or",
+   "btrfs receive --dump [options]",
"Receive subvolumes from stdin.",
"Receives one or more subvolumes that were previously",
"sent with btrfs send. The received subvolumes are stored",
@@ -1322,5 +1347,7 @@ const char * const cmd_receive_usage[] = {
"-m   The root mount point of the destination fs.",
" If you do not have /proc use this to tell us where ",
" this file system is mounted.",
+   "--dump   Exam 

Re: [PATCH v2 4/4] xfstests: btrfs/134: add test for incremental send which renames a directory already being deleted

[Eryu Guan]

On Fri, Oct 28, 2016 at 09:44:06AM +0800, robbieko wrote:
> From: Robbie Ko 
> 
> Test that an incremental send operation dosen't work because
> it tries to rename a directory which is already deleted.
> 
> This test exercises scenarios used to fail in btrfs and are fixed by
> the following patch for the linux kernel:
> 
> "Btrfs: incremental send, add generation check for inode is waiting for move."

I was testing with v4.9-rc1+ kernel and btrfs-progs v4.6. Seems above
patch is not merged in 4.9-rc1 kernel, but test passed for me, is that
expected?

Thanks,
Eryu
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 1/4] xfstests: btrfs/131: add test for an incremental send with name collision

[Eryu Guan]

On Fri, Oct 28, 2016 at 09:44:03AM +0800, robbieko wrote:
> From: Robbie Ko 
> 
> Test that an incremental send operation doesn't work because
> there's a name collision in the destination and it's not checked
> corretly before the rename operation applies.
> 
> This test exercises scenarios used to fail in btrfs and are fixed by
> the following patch for the linux kernel:
> 
> "Btrfs: incremental send, do not skip generation inconsistency check for 
> inode 256."
> 
> Signed-off-by: Robbie Ko 

Sorry for the late review!

> ---
>  tests/btrfs/131 | 111 
> 
>  tests/btrfs/131.out |   2 +
>  tests/btrfs/group   |   1 +
>  3 files changed, 114 insertions(+)
>  create mode 100755 tests/btrfs/131
>  create mode 100644 tests/btrfs/131.out
> 
> diff --git a/tests/btrfs/131 b/tests/btrfs/131
> new file mode 100755
> index 000..2e2e0bc
> --- /dev/null
> +++ b/tests/btrfs/131
> @@ -0,0 +1,111 @@
> +#! /bin/bash
> +# FS QA Test No. btrfs/131
> +#
> +# Test that an incremental send operation doesn't work because
> +# there's a name collision in the destination and it's not checked
> +# corretly before the rename operation applies.
> +#
> +#---
> +# Copyright (C) 2016 Synology Inc. All Rights Reserved.
> +# Author: Robbie Ko 
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#---
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -fr $send_files_dir
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# real QA test starts here
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_test
> +_require_scratch
> +_require_fssum
> +
> +send_files_dir=$TEST_DIR/btrfs-test-$seq
> +
> +rm -f $seqres.full
> +rm -fr $send_files_dir
> +mkdir $send_files_dir
> +
> +_scratch_mkfs >>$seqres.full 2>&1
> +_scratch_mount
> +
> +mkdir $SCRATCH_MNT/a1
> +mkdir $SCRATCH_MNT/a2
> +
> +# Filesystem looks like:
> +#
> +# . (ino 256)
> +# |--- a1/  (ino 257)
> +# |
> +# |--- a2/  (ino 258)
> +#
> +_run_btrfs_util_prog subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/mysnap1
> +
> +_run_btrfs_util_prog send $SCRATCH_MNT/mysnap1 -f $send_files_dir/1.snap

I found these "run_check" based helpers make the result harder to read
when test failed, I can't see immediate error messages from the diff
output but have to open $seqres.full file to see the details.

Can you please use $BTRFS_UTIL_PROG directly, in all these four tests?
One concern about using bare $BTRFS_UTIL_PROG is that output messages of
btrfs command are always changed. But I think, at least, we can ignore
the stdout and capture the stderr only. e.g.

$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT \
$SCRATCH_MNT/mysnap1 >/dev/null

So if everything works fine no output can be seen, but if snapshot
creation is failed, we can see error messages from stderr, which could
fail the test.

> +
> +_scratch_unmount
> +_scratch_mkfs >>$seqres.full 2>&1
> +_scratch_mount
> +touch $SCRATCH_MNT/a2
> +
> +# Filesystem now looks like:
> +#
> +# . (ino 256)
> +# |--- a2   (ino 257)
> +#
> +_run_btrfs_util_prog subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/mysnap2
> +
> +_run_btrfs_util_prog receive $SCRATCH_MNT -f $send_files_dir/1.snap
> +rm $send_files_dir/1.snap
> +
> +run_check $FSSUM_PROG -A -f -w $send_files_dir/1.fssum $SCRATCH_MNT/mysnap1
> +run_check $FSSUM_PROG -A -f -w $send_files_dir/2.fssum $SCRATCH_MNT/mysnap2

And this "run_check" is not necessary either. Just run $FSSUM_PROG
directly and let it print "OK", and match it in .out file.

Thanks,
Eryu

> +
> +_run_btrfs_util_prog send $SCRATCH_MNT/mysnap1 -f $send_files_dir/1.snap
> +_run_btrfs_util_prog send -p