Re: [PATCH v3] xfstests: btrfs/316: cross-subvolume sparse copy

2013-07-03 Thread Dave Chinner
On Tue, Jul 02, 2013 at 11:51:21AM -0400, Eric Sandeen wrote: On Jul 2, 2013, at 10:28 AM, Koen De Wit koen.de@oracle.com wrote: Dave, Thanks for the review. I will clean up the commit message and do a full mail-to-myself-and-test-patch round trip to avoid errors like the wrong

[PATCH] Btrfs: fix extent buffer leak after backref walking

2013-07-03 Thread Liu Bo
commit 47fb091fb787420cd195e66f162737401cce023f(Btrfs: fix unlock after free on rewinded tree blocks) takes an extra increment on the reference of allocated dummy extent buffer, so now we cannot free this dummy one, and end up with extent buffer leak. Signed-off-by: Liu Bo bo.li@oracle.com

Re: WARNING: at fs/btrfs/backref.c:903 find_parent_nodes+0x616/0x815 [btrfs]()

2013-07-03 Thread Liu Bo
On Wed, Jul 03, 2013 at 11:37:09AM +0900, Tomasz Chmielewski wrote: I've upgraded to linux 3.10 and enabled extended inode refs and skinny metadata extent refs with these commands: btrfstune -r /dev/sdc1 btrfstune -x /dev/sdc1 Since then, I have WARNING: at fs/btrfs/backref.c:903

Re: [PATCH] Btrfs: fix extent buffer leak after backref walking

2013-07-03 Thread Jan Schmidt
On Wed, July 03, 2013 at 08:40 (+0200), Liu Bo wrote: commit 47fb091fb787420cd195e66f162737401cce023f(Btrfs: fix unlock after free on rewinded tree blocks) takes an extra increment on the reference of allocated dummy extent buffer, so now we cannot free this dummy one, and end up with

[PATCH] Xfstests/btrfs/308: add snapshot-aware defrag for partial extents testcases

2013-07-03 Thread Liu Bo
This is to test whether snapshot-aware defrag can work well on partial extents. Signed-off-by: Liu Bo bo.li@oracle.com --- tests/btrfs/308 | 93 +++ tests/btrfs/308.out |2 + tests/btrfs/group |1 + 3 files changed, 96

Re: [PATCH] Btrfs: fix a bug of snapshot-aware defrag to make it work on partial extents

2013-07-03 Thread Liu Bo
On Mon, Jul 01, 2013 at 04:22:06PM -0400, Josef Bacik wrote: On Mon, Jul 01, 2013 at 10:13:26PM +0800, Liu Bo wrote: For partial extents, snapshot-aware defrag does not work as expected, since a) we use the wrong logical offset to search for parents, which should be disk_bytenr +

Re: [PATCH v3] xfstests: btrfs/316: cross-subvolume sparse copy

2013-07-03 Thread Koen De Wit
On 07/03/2013 08:37 AM, Dave Chinner wrote: On Tue, Jul 02, 2013 at 11:51:21AM -0400, Eric Sandeen wrote: On Jul 2, 2013, at 10:28 AM, Koen De Wit koen.de@oracle.com wrote: Dave, Thanks for the review. I will clean up the commit message and do a full mail-to-myself-and-test-patch

Re: btrfs send /receive : having problems sending a snapshot back to the original partition

2013-07-03 Thread Miguel Negrão
Em 02-07-2013 13:05, Stefan Behrens escreveu: The reason why you are not able to send an incremental snapshot back (as you tried it in your previous mail) is that the implementation cannot handle it. The receive side gets a parent_uuid (in case of incremental transfers using the -p option), this

Re: [PATCH v3] xfstests: btrfs/316: cross-subvolume sparse copy

2013-07-03 Thread Dave Chinner
On Wed, Jul 03, 2013 at 12:02:52PM +0200, Koen De Wit wrote: On 07/03/2013 08:37 AM, Dave Chinner wrote: On Tue, Jul 02, 2013 at 11:51:21AM -0400, Eric Sandeen wrote: On Jul 2, 2013, at 10:28 AM, Koen De Wit koen.de@oracle.com wrote: Dave, Thanks for the review. I will clean

Re: [PATCH] btrfs-progs: Fix segmentation fault

2013-07-03 Thread David Sterba
On Sat, Jun 22, 2013 at 02:39:21PM +0900, Kusanagi Kouichi wrote: 'btrfs subvolume delete' crashes with segv if it runs in a detached mount. One does not even need to do the lazy unmount. If the there are 2 processes attempting to delete the same snapshot and one deletes the snapshot directory

[PATCH v4] xfstests: btrfs/316: cross-subvolume sparse copy

2013-07-03 Thread Koen De Wit
This testscript creates reflinks to files on different subvolumes, overwrites the original files and reflinks, and moves reflinked files between subvolumes. Signed-off-by: Koen De Wit koen.de@oracle.com --- Two new common/rc functions used in this script (_require_cp_reflink and

[PATCH] Btrfs: optimize function btrfs_read_chunk_tree

2013-07-03 Thread Filipe David Borba Manana
After reading all device items from the chunk tree, don't exit the loop and then navigate down the tree again to find the chunk items. Instead, if while reading the device items we find a chunk item in the leaf keep iterating over the leaf and process the chunk items - at the moment chunk items

[PATCH v4] xfstests: btrfs/311: sparse copy between different filesystems/mountpoints

2013-07-03 Thread Koen De Wit
Check if creating a sparse copy (reflink) of a file on btrfs expectedly fails when it's done between different filesystems or different mount points of the same filesystem. For both situations, these actions are executed: - Copy a file with the reflink=auto option. A normal copy should be

[RFC PATCH 00/12] Btrfs-progs: introduce chunk recover function

2013-07-03 Thread Miao Xie
This patchset introduced chunk recover function, which was implemented by scanning the whoel disks in the filesystem. Now, we can recover Single, Dup, RAID1 chunks, and RAID0, RAID10, RAID5, RAID6 metadata chunks. Miao Xie (11): Btrfs-progs: don't close the file descriptor 0 when closing a

[PATCH 10/12] Btrfs-progs: introduce list_{first, next}_entry/list_splice_tail{_init}

2013-07-03 Thread Miao Xie
Signed-off-by: Miao Xie mi...@cn.fujitsu.com --- list.h | 68 +- 1 file changed, 59 insertions(+), 9 deletions(-) diff --git a/list.h b/list.h index 50f4619..db7a58c 100644 --- a/list.h +++ b/list.h @@ -223,18 +223,18 @@ static

[PATCH 03/12] Btrfs-progs: Don't free the devices when close the ctree

2013-07-03 Thread Miao Xie
Some commands(such as btrfs-convert) access the devices again after we close the ctree, so it is better that we don't free the devices objects when the ctree is closed, or we need re-allocate the memory for the devices. We needn't worry the memory leak problem, because all the memory will be freed

[PATCH 06/12] Btrfs-progs: use rb-tree instead of extent cache tree for fs/file roots

2013-07-03 Thread Miao Xie
Because the fs/file roots are not extents, so it is better to use rb-tree to manage them. Fix it. Signed-off-by: Miao Xie mi...@cn.fujitsu.com --- ctree.h | 4 ++-- disk-io.c | 50 -- 2 files changed, 38 insertions(+), 16 deletions(-) diff

[PATCH 02/12] Btrfs-progs: don't close the file descriptor 0 when closing a device

2013-07-03 Thread Miao Xie
As we know, the file descriptor 0 is a special number, so we shouldn't use it to initialize the file descriptor of the devices, or we might close this special file descriptor by mistake when we close the devices. -1 is a better choice. Signed-off-by: Miao Xie mi...@cn.fujitsu.com ---

[PATCH 11/12] Btrfs-progs: Add chunk rebuild function for RAID1/SINGLE/DUP

2013-07-03 Thread Miao Xie
Add chunk rebuild for RAID1/SINGLE/DUP to chunk-recover command. Before this patch chunk-recover can only scan and reuse the old chunk data to recover. With this patch, chunk-recover can use the reference between chunk/block group/dev extent to rebuild the whole chunk tree even when old chunks

[PATCH 12/12] Btrfs-progs: recover raid0/raid10/raid5/raid6 metadata chunk

2013-07-03 Thread Miao Xie
According to the bytenr of the extent buffer record, we can calculate the index of the stripes, and we also know which device and where we read out the extent buffer record, that means we can know the relationship between the device extent and the stripes in the chunk, by this relationship, we can

[PATCH 01/12] Btrfs-progs: fix missing recow roots when making btrfs filesystem

2013-07-03 Thread Miao Xie
From: Wang Shilong wangsl-f...@cn.fujitsu.com When making btrfs filesystem. we firstly write root leaf to specified filed, and then we recow the root. If we don't recow, some trees are not in the correct block group. Steps to reproduce: dd if=/dev/zero of=test.img bs=1M count=100

[PATCH 05/12] Btrfs-progs: introduce common insert/search/delete functions for rb-tree

2013-07-03 Thread Miao Xie
In fact, the code of many rb-tree insert/search/delete functions is similar, so we can abstract them, and implement common functions for rb-tree, and then simplify them. Signed-off-by: Miao Xie mi...@cn.fujitsu.com --- btrfs-list.c | 19 +++- cmds-check.c | 111

[PATCH 07/12] Btrfs-progs: extend the extent cache for the device extent

2013-07-03 Thread Miao Xie
As we know, btrfs can manage several devices in the same fs, so [offset, size] is not sufficient for unique identification of an device extent, we need the device id to identify the device extents which have the same offset and size, but are not in the same device. So, we added a member variant

[PATCH 08/12] Btrfs-progs: Add block group check funtion

2013-07-03 Thread Miao Xie
This patch adds the function to check correspondence between block group, chunk and device extent. Original-signed-off-by: Cheng Yang chenyang.f...@cn.fujitsu.com Signed-off-by: Miao Xie mi...@cn.fujitsu.com --- Makefile | 2 +- btrfsck.h| 118 + cmds-check.c | 555

[PATCH 04/12] Btrfs-progs: cleanup similar code in open_ctree_* and close_ctree

2013-07-03 Thread Miao Xie
Signed-off-by: Miao Xie mi...@cn.fujitsu.com --- btrfs-find-root.c | 137 +++- disk-io.c | 473 +++--- disk-io.h | 12 ++ 3 files changed, 307 insertions(+), 315 deletions(-) diff --git a/btrfs-find-root.c

Re: [PATCH 02/12] Btrfs-progs: don't close the file descriptor 0 when closing a device

2013-07-03 Thread Filipe David Manana
On Wed, Jul 3, 2013 at 2:25 PM, Miao Xie mi...@cn.fujitsu.com wrote: +++ b/disk-io.c @@ -1270,12 +1270,13 @@ static int close_all_devices(struct btrfs_fs_info *fs_info) while (!list_empty(list)) { device = list_entry(list-next, struct btrfs_device, dev_list);

[PATCH 1/2] Btrfs-progs: make pretty_sizes() works less error prone

2013-07-03 Thread Wang Shilong
From: Wang Shilong wangsl-f...@cn.fujitsu.com In the original code, pretty_sizes() may return NULL in two cases: 1 Allocating memory dynamically fails 2 Overflow happens(size exceeds YB) The original codes don't handle error gracefully and some places forget to free memory. However, just

[PATCH RESEND] Btrfs-progs: fix compile warning in btrfs_free_block_groups()

2013-07-03 Thread Wang Shilong
From: Wang Shilong wangsl-f...@cn.fujitsu.com extent-tree.c: In function 'btrfs_free_block_groups': extent-tree.c:3189:12: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast] Signed-off-by: Wang Shilong wangsl-f...@cn.fujitsu.com --- extent-tree.c | 3 ++- 1 file

Re: [PATCH RESEND] Btrfs-progs: fix compile warning in btrfs_free_block_groups()

2013-07-03 Thread Stefan Behrens
On Thu, 4 Jul 2013 00:24:45 +0800, Wang Shilong wrote: From: Wang Shilong wangsl-f...@cn.fujitsu.com extent-tree.c: In function 'btrfs_free_block_groups': extent-tree.c:3189:12: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast] Signed-off-by: Wang Shilong

[PATCH] Btrfs-progs: fix check in btrfs_lookup_extent_info()

2013-07-03 Thread Filipe David Borba Manana
We want to test if path-slots[0] is greater than zero. Testing for path-slots was a logical error, as it corresponds to a memory address (start of fixed array) and therefore is always non-zero. Signed-off-by: Filipe David Borba Manana fdman...@gmail.com --- extent-tree.c |2 +- 1 file

[PATCH v7 3/8] Btrfs: create UUID tree if required

2013-07-03 Thread Stefan Behrens
This tree is not created by mkfs.btrfs. Therefore when a filesystem is mounted writable and the UUID tree does not exist, this tree is created if required. The tree is also added to the fs_info structure and initialized, but this commit does not yet read or write UUID tree elements.

[PATCH v7 6/8] Btrfs: introduce uuid-tree-gen field

2013-07-03 Thread Stefan Behrens
In order to be able to detect the case that a filesystem is mounted with an old kernel, add a uuid-tree-gen field like the free space cache is doing it. It is part of the super block and written with each commit. Old kernels do not know this field and don't update it. Signed-off-by: Stefan

[PATCH v7 5/8] Btrfs: fill UUID tree initially

2013-07-03 Thread Stefan Behrens
When the UUID tree is initially created, a task is spawned that walks through the root tree. For each found subvolume root_item, the uuid and received_uuid entries in the UUID tree are added. This is such a quick operation so that in case somebody wants to unmount the filesystem while the task is

[PATCH v7 4/8] Btrfs: maintain subvolume items in the UUID tree

2013-07-03 Thread Stefan Behrens
When a new subvolume or snapshot is created, a new UUID item is added to the UUID tree. Such items are removed when the subvolume is deleted. The ioctl to set the received subvolume UUID is also touched and will now also add this received UUID into the UUID tree together with the ID of the

[PATCH v7 8/8] Btrfs: add mount option to force UUID tree checking

2013-07-03 Thread Stefan Behrens
This should never be needed, but since all functions are there to check and rebuild the UUID tree, a mount option is added that allows to force this check and rebuild procedure. Signed-off-by: Stefan Behrens sbehr...@giantdisaster.de --- fs/btrfs/ctree.h | 1 + fs/btrfs/disk-io.c | 3 ++-

[PATCH v7 0/8] Btrfs: introduce a tree for UUID to subvol ID mapping

2013-07-03 Thread Stefan Behrens
Mapping UUIDs to subvolume IDs is an operation with a high effort today. Today, the algorithm even has quadratic effort (based on the number of existing subvolumes), which means, that it takes minutes to send/receive a single subvolume if 10,000 subvolumes exist. But even linear effort would be

[PATCH v7 1/8] Btrfs: introduce a tree for items that map UUIDs to something

2013-07-03 Thread Stefan Behrens
Mapping UUIDs to subvolume IDs is an operation with a high effort today. Today, the algorithm even has quadratic effort (based on the number of existing subvolumes), which means, that it takes minutes to send/receive a single subvolume if 10,000 subvolumes exist. But even linear effort would be

[PATCH v7 2/8] Btrfs: support printing UUID tree elements

2013-07-03 Thread Stefan Behrens
This commit adds support to print UUID tree elements to print-tree.c. Signed-off-by: Stefan Behrens sbehr...@giantdisaster.de --- fs/btrfs/print-tree.c | 24 1 file changed, 24 insertions(+) diff --git a/fs/btrfs/print-tree.c b/fs/btrfs/print-tree.c index

[PATCH v7 7/8] Btrfs: check UUID tree during mount if required

2013-07-03 Thread Stefan Behrens
If the filesystem was mounted with an old kernel that was not aware of the UUID tree, this is detected by looking at the uuid_tree_generation field of the superblock (similar to how the free space cache is doing it). If a mismatch is detected at mount time, a thread is started that does two

Re: [PATCH 1/2] Btrfs-progs: make pretty_sizes() works less error prone

2013-07-03 Thread David Sterba
On Thu, Jul 04, 2013 at 12:07:37AM +0800, Wang Shilong wrote: From: Wang Shilong wangsl-f...@cn.fujitsu.com In the original code, pretty_sizes() may return NULL in two cases: 1 Allocating memory dynamically fails Turning it into static allocation looks much better than calling malloc for ~30

Re: [PATCH 1/3] Btrfs-progs: add missing write check for mkfs

2013-07-03 Thread Filipe David Manana
On Wed, Jul 3, 2013 at 6:09 PM, David Sterba dste...@suse.cz wrote: On Sun, Jun 30, 2013 at 12:51:44PM +0100, Filipe David Borba Manana wrote: Assert that the write of the device tree root succeeds. This verification is currently done for all other tree roots, however it was missing for the

[PATCH v2 3/3] Btrfs-progs: remove unused code

2013-07-03 Thread Filipe David Borba Manana
V2: removed unused statement in extent-cache.c:tree_insert() too. Signed-off-by: Filipe David Borba Manana fdman...@gmail.com --- extent-cache.c |1 - volumes.c |2 -- 2 files changed, 3 deletions(-) diff --git a/extent-cache.c b/extent-cache.c index 3dd6434..a8bab59 100644 ---

[PATCH v2 1/3] Btrfs-progs: add missing write check for mkfs

2013-07-03 Thread Filipe David Borba Manana
Assert that the writes of the device and chunk tree roots succeed. This verification is currently done for all other tree roots, however it was missing for those 2 trees. Would these tree root writes fail, but all others succeed, it would lead to a corrupted/incomplete btrfs filesystem, or, more

Scrub causes oom after removal of failed disk (linux 3.10)

2013-07-03 Thread Torbjørn
Hi btrfs devs, I have a btrfs raid10 array consisting of 2TB drives. I added a new drive to the array, then balanced. The balance failed after ~50GB was moved to the new drive. The balance fixed lots of errors according to dmesg. Server rebooted The newly added drive were no longer detected

Re: [RFC PATCH 00/12] Btrfs-progs: introduce chunk recover function

2013-07-03 Thread Chris Mason
Quoting Miao Xie (2013-07-03 09:25:08) This patchset introduced chunk recover function, which was implemented by scanning the whoel disks in the filesystem. Now, we can recover Single, Dup, RAID1 chunks, and RAID0, RAID10, RAID5, RAID6 metadata chunks. Really nice. I've integrated this with

Re: [PATCH 1/3] Btrfs-progs: add missing write check for mkfs

2013-07-03 Thread David Sterba
On Wed, Jul 03, 2013 at 06:25:59PM +0100, Filipe David Manana wrote: Also, replacing the BUG_ON with something else would be nice of course. If you don't mind, I would prefer to do that as a different patch, since the BUG_ON() use is what is currently done everywhere in this function at

Re: [PATCH 02/12] Btrfs-progs: don't close the file descriptor 0 when closing a device

2013-07-03 Thread Miao Xie
On wed, 3 Jul 2013 15:17:02 +0100, Filipe David Manana wrote: On Wed, Jul 3, 2013 at 2:25 PM, Miao Xie mi...@cn.fujitsu.com wrote: +++ b/disk-io.c @@ -1270,12 +1270,13 @@ static int close_all_devices(struct btrfs_fs_info *fs_info) while (!list_empty(list)) { device

Re: [PATCH 2/3] Btrfs: fix the deadlock between the transaction start/attach and commit

2013-07-03 Thread Miao Xie
On Wed, 26 Jun 2013 20:53:00 +0300, Alex Lyakas wrote: Hi Miao, On Mon, Jun 17, 2013 at 4:51 AM, Miao Xie mi...@cn.fujitsu.com wrote: On sun, 16 Jun 2013 13:38:42 +0300, Alex Lyakas wrote: Hi Miao, On Thu, Jun 13, 2013 at 6:08 AM, Miao Xie mi...@cn.fujitsu.com wrote: On wed, 12 Jun

Re: [RFC PATCH 00/12] Btrfs-progs: introduce chunk recover function

2013-07-03 Thread Liu Bo
On Wed, Jul 03, 2013 at 04:36:44PM -0400, Chris Mason wrote: Quoting Miao Xie (2013-07-03 09:25:08) This patchset introduced chunk recover function, which was implemented by scanning the whoel disks in the filesystem. Now, we can recover Single, Dup, RAID1 chunks, and RAID0, RAID10, RAID5,