[GIT PULL] [RFC PATCH 0/4] btrfs: Implement delayed directory name index insertion and deletion

2010-12-01 Thread Miao Xie
Compare with Ext3/4, the performance of file creation and deletion on btrfs is very poor. the reason is that btrfs must do a lot of b+ tree insertions, such as inode item, directory name item, directory name index and so on. If we can do some delayed b+ tree insertion or deletion, we can improve

[RFC PATCH 1/4] btrfs: introduce a function btrfs_insert_dir_index_item()

2010-12-01 Thread Miao Xie
restructure btrfs_insert_dir_item() and introduce a function btrfs_insert_dir_index_item() to insert dir index item. Signed-off-by: Miao Xie mi...@cn.fujitsu.com --- fs/btrfs/ctree.h|6 fs/btrfs/dir-item.c | 69 +++--- 2 files changed,

[RFC PATCH 3/4] btrfs: reduce the times of mmap() in fill_inode_item()

2010-12-01 Thread Miao Xie
With the old code, we must map the page every time we want to set a member variable of the inode item, it is inefficient. We just do it at first. By this way, we can improve the performance of file creation and deletion by ~2% Signed-off-by: Miao Xie mi...@cn.fujitsu.com --- fs/btrfs/inode.c |

[RFC PATCH 4/4] btrfs: implement delayed dir index insertion and deletion

2010-12-01 Thread Miao Xie
Compare with Ext3/4, the performance of file creation and deletion on btrfs is very poor. the reason is that btrfs must do a lot of b+ tree insertions, such as inode item, directory name item, directory name index and so on. If we can do some delayed b+ tree insertion or deletion, we can improve

Re: [PATCH] Btrfs: dynamically remove unused block groups

2010-12-01 Thread Josef Bacik
On Tue, Nov 30, 2010 at 09:53:41PM -0700, Anthony Roberts wrote: Hello, What happens in the event the filesystem has mostly been cleared out, but there's a few things left? For example, several of the chunks might be at very low usage, but not zero. Would the user be able to defragment

[RFC PATCH 0/4 v2] Btrfs: Add readonly support to replace BUG_ON phrase

2010-12-01 Thread liubo
Btrfs has a number of BUG_ON()s, which may lead btrfs to unpleasant panic. Meanwhile, they are very ugly and should be handled more propriately. There are mainly two ways to deal with these BUG_ON()s. 1. For those errors which can be handled well by callers, we just return their error number to

[RFC PATCH 1/4 v2] Btrfs: add filesystem state for error handle

2010-12-01 Thread liubo
Add filesystem state and a flags to tell if the filesystem is valid or insane now. Signed-off-by: Liu Bo liubo2...@cn.fujitsu.com --- fs/btrfs/ctree.h | 11 +++ 1 files changed, 11 insertions(+), 0 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index 8db9234..92b5ca2

[RFC PATCH 2/4 v2] Btrfs: avoid transaction stuff when readonly

2010-12-01 Thread liubo
When the filesystem is readonly, avoid transaction stuff by checking MS_RDONLY at start transaction time. Signed-off-by: Liu Bo liubo2...@cn.fujitsu.com --- fs/btrfs/transaction.c |3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/fs/btrfs/transaction.c

[RFC PATCH 3/4 v2] Btrfs: add readonly support for error handle

2010-12-01 Thread liubo
This patch provide a new error handle interface for those errors that handled by current BUG_ONs. In order to protect btrfs from panic, when it comes to those BUG_ON errors, the interface forces btrfs readonly and saves the FS state to disk. And the filesystem can be umounted, although mabye

[RFC PATCH 4/4 v2] Btrfs: deal with filesystem state at mount, umount

2010-12-01 Thread liubo
Since there is a filesystem state, we should deal with it carefully at mount, umount and remount. - At mount, the FS state should be checked if there is error on these FS. If it does have, btrfsck is recommended. - At umount, the FS state should be saved into disk for consistency.

Fsck, parent transid verify failed

2010-12-01 Thread Tommy Jonsson
Hi folks! Been using btrfs for quite a while now, worked great until now. Got power-loss on my machine and now i have the parent transid verify failed on X wanted X found X problem. So I can't get it to mount. My btrfs is spread over sda (2tb), sdc(2tb), sdd(1tb). Is this something that an

Re: What to do about subvolumes?

2010-12-01 Thread Mike Hommey
On Wed, Dec 01, 2010 at 09:21:36AM -0500, Josef Bacik wrote: 1) Users need to be able to create their own subvolumes. The permission semantics will be absolutely the same as creating directories, so I don't think this is too tricky. We want this because you can only take snapshots of

Re: What to do about subvolumes?

2010-12-01 Thread C Anthony Risinger
On Wed, Dec 1, 2010 at 8:21 AM, Josef Bacik jo...@redhat.com wrote: === How do we want subvolumes to work from a user perspective? === 1) Users need to be able to create their own subvolumes.  The permission semantics will be absolutely the same as creating directories, so I don't think

Re: What to do about subvolumes?

2010-12-01 Thread Chris Mason
Excerpts from Josef Bacik's message of 2010-12-01 09:21:36 -0500: Hello, Various people have complained about how BTRFS deals with subvolumes recently, specifically the fact that they all have the same inode number, and there's no discrete seperation from one subvolume to another. Christoph

Re: What to do about subvolumes?

2010-12-01 Thread Chris Mason
Excerpts from C Anthony Risinger's message of 2010-12-01 09:51:55 -0500: On Wed, Dec 1, 2010 at 8:21 AM, Josef Bacik jo...@redhat.com wrote: === How do we want subvolumes to work from a user perspective? === 1) Users need to be able to create their own subvolumes.  The permission

Re: What to do about subvolumes?

2010-12-01 Thread C Anthony Risinger
On Wed, Dec 1, 2010 at 10:01 AM, Chris Mason chris.ma...@oracle.com wrote: Excerpts from C Anthony Risinger's message of 2010-12-01 09:51:55 -0500: On Wed, Dec 1, 2010 at 8:21 AM, Josef Bacik jo...@redhat.com wrote: === How do we want subvolumes to work from a user perspective? === 1)

Re: dm-crypt barrier support is effective

2010-12-01 Thread Matt
On Mon, Nov 15, 2010 at 12:24 AM, Matt jackdac...@gmail.com wrote: On Sun, Nov 14, 2010 at 10:54 PM, Milan Broz mb...@redhat.com wrote: On 11/14/2010 10:49 PM, Matt wrote: only with the dm-crypt scaling patch I could observe the data-corruption even with v5 I sent on Friday? Are you sure

Re: What to do about subvolumes?

2010-12-01 Thread Mike Hommey
On Wed, Dec 01, 2010 at 11:01:37AM -0500, Chris Mason wrote: Excerpts from C Anthony Risinger's message of 2010-12-01 09:51:55 -0500: On Wed, Dec 1, 2010 at 8:21 AM, Josef Bacik jo...@redhat.com wrote: === How do we want subvolumes to work from a user perspective? === 1) Users need

Re: What to do about subvolumes?

2010-12-01 Thread Hugo Mills
On Wed, Dec 01, 2010 at 09:21:36AM -0500, Josef Bacik wrote: === Quotas === This is a huge topic in and of itself, but Christoph mentioned wanting to have an idea of what we wanted to do with it, so I'm putting it here. There are really 2 things here 1) Limiting the size of subvolumes.

disk space caching generation missmatch

2010-12-01 Thread Johannes Hirte
After enabling disk space caching I've observed several log entries like this: btrfs: free space inode generation (0) did not match free space cache generation (169594) for block group 15464398848 I'm not sure, but it seems this happens on every reboot. Is this something to worry about?

Re: What to do about subvolumes?

2010-12-01 Thread C Anthony Risinger
On Wed, Dec 1, 2010 at 10:38 AM, Hugo Mills hugo-l...@carfax.org.uk wrote: On Wed, Dec 01, 2010 at 09:21:36AM -0500, Josef Bacik wrote: === Quotas === This is a huge topic in and of itself, but Christoph mentioned wanting to have an idea of what we wanted to do with it, so I'm putting it

Re: What to do about subvolumes?

2010-12-01 Thread Mike Hommey
On Wed, Dec 01, 2010 at 04:38:00PM +, Hugo Mills wrote: On Wed, Dec 01, 2010 at 09:21:36AM -0500, Josef Bacik wrote: === Quotas === This is a huge topic in and of itself, but Christoph mentioned wanting to have an idea of what we wanted to do with it, so I'm putting it here.

Re: dm-crypt barrier support is effective

2010-12-01 Thread Mike Snitzer
On Wed, Dec 01 2010 at 11:05am -0500, Matt jackdac...@gmail.com wrote: On Mon, Nov 15, 2010 at 12:24 AM, Matt jackdac...@gmail.com wrote: On Sun, Nov 14, 2010 at 10:54 PM, Milan Broz mb...@redhat.com wrote: On 11/14/2010 10:49 PM, Matt wrote: only with the dm-crypt scaling patch I could

Re: dm-crypt barrier support is effective

2010-12-01 Thread Matt
On Wed, Dec 1, 2010 at 5:52 PM, Mike Snitzer snit...@redhat.com wrote: On Wed, Dec 01 2010 at 11:05am -0500, Matt jackdac...@gmail.com wrote: On Mon, Nov 15, 2010 at 12:24 AM, Matt jackdac...@gmail.com wrote: On Sun, Nov 14, 2010 at 10:54 PM, Milan Broz mb...@redhat.com wrote: On

Re: What to do about subvolumes?

2010-12-01 Thread Josef Bacik
On Wed, Dec 01, 2010 at 04:38:00PM +, Hugo Mills wrote: On Wed, Dec 01, 2010 at 09:21:36AM -0500, Josef Bacik wrote: === Quotas === This is a huge topic in and of itself, but Christoph mentioned wanting to have an idea of what we wanted to do with it, so I'm putting it here.

Re: dm-crypt barrier support is effective

2010-12-01 Thread Milan Broz
On 12/01/2010 06:35 PM, Matt wrote: Thanks for pointing to v6 ! I hadn't noticed that there was a new one :) Well, so I'll restore my box to a working/productive state and will try out v6 (I'm pretty confident that it'll work without problems). It's the same as previous, just with fixed

Re: On Removing BUG_ON macros

2010-12-01 Thread Josef Bacik
On Thu, Nov 11, 2010 at 12:32:06PM +0800, Ian Kent wrote: On Mon, 2010-11-08 at 23:02 +0800, Ian Kent wrote: On Mon, 2010-11-08 at 09:15 -0500, Josef Bacik wrote: On Mon, Nov 08, 2010 at 10:06:13PM +0800, Ian Kent wrote: On Mon, 2010-11-08 at 07:42 -0500, Josef Bacik wrote: On Mon,

Re: What to do about subvolumes?

2010-12-01 Thread Josef Bacik
On Wed, Dec 01, 2010 at 07:33:39PM +0100, Goffredo Baroncelli wrote: On Wednesday, 01 December, 2010, Josef Bacik wrote: Hello, Hi Josef === What are subvolumes? === They are just another tree. In BTRFS we have various b-trees to describe the filesystem. A few of them

Re: What to do about subvolumes?

2010-12-01 Thread C Anthony Risinger
On Wed, Dec 1, 2010 at 12:36 PM, Josef Bacik jo...@redhat.com wrote: On Wed, Dec 01, 2010 at 07:33:39PM +0100, Goffredo Baroncelli wrote: Another point that I want like to discuss is how manage the pivoting between the subvolumes. One of the most beautiful feature of btrfs is the snapshot

Re: disk space caching generation missmatch

2010-12-01 Thread Josef Bacik
On Wed, Dec 01, 2010 at 05:46:14PM +0100, Johannes Hirte wrote: After enabling disk space caching I've observed several log entries like this: btrfs: free space inode generation (0) did not match free space cache generation (169594) for block group 15464398848 I'm not sure, but it seems

Re: What to do about subvolumes?

2010-12-01 Thread C Anthony Risinger
On Wed, Dec 1, 2010 at 12:48 PM, C Anthony Risinger anth...@extof.me wrote: On Wed, Dec 1, 2010 at 12:36 PM, Josef Bacik jo...@redhat.com wrote: On Wed, Dec 01, 2010 at 07:33:39PM +0100, Goffredo Baroncelli wrote: Another point that I want like to discuss is how manage the pivoting between

Re: What to do about subvolumes?

2010-12-01 Thread Goffredo Baroncelli
On Wednesday, 01 December, 2010, you (C Anthony Risinger) wrote: [...] i forgot to mention, but a quick 'n dirty solution would be to simply not enable users to do this by accident. mkfs.btrfs could create a new subvol, then mark it as default... this way the user has to manually mount with

Re: dm-crypt barrier support is effective

2010-12-01 Thread Jon Nelson
On Wed, Dec 1, 2010 at 12:24 PM, Milan Broz mb...@redhat.com wrote: On 12/01/2010 06:35 PM, Matt wrote: Thanks for pointing to v6 ! I hadn't noticed that there was a new one :) Well, so I'll restore my box to a working/productive state and will try out v6 (I'm pretty confident that it'll work

Re: What to do about subvolumes?

2010-12-01 Thread Hugo Mills
On Wed, Dec 01, 2010 at 12:38:30PM -0500, Josef Bacik wrote: On Wed, Dec 01, 2010 at 04:38:00PM +, Hugo Mills wrote: On Wed, Dec 01, 2010 at 09:21:36AM -0500, Josef Bacik wrote: === Quotas === This is a huge topic in and of itself, but Christoph mentioned wanting to have an

Re: What to do about subvolumes?

2010-12-01 Thread J. Bruce Fields
On Wed, Dec 01, 2010 at 09:21:36AM -0500, Josef Bacik wrote: Hello, Various people have complained about how BTRFS deals with subvolumes recently, specifically the fact that they all have the same inode number, and there's no discrete seperation from one subvolume to another. Christoph

Re: disk space caching generation missmatch

2010-12-01 Thread Johannes Hirte
On Wednesday 01 December 2010 18:40:18 Josef Bacik wrote: On Wed, Dec 01, 2010 at 05:46:14PM +0100, Johannes Hirte wrote: After enabling disk space caching I've observed several log entries like this: btrfs: free space inode generation (0) did not match free space cache generation

Re: What to do about subvolumes?

2010-12-01 Thread Josef Bacik
On Wed, Dec 01, 2010 at 02:44:04PM -0500, J. Bruce Fields wrote: On Wed, Dec 01, 2010 at 09:21:36AM -0500, Josef Bacik wrote: Hello, Various people have complained about how BTRFS deals with subvolumes recently, specifically the fact that they all have the same inode number, and

Re: dm-crypt barrier support is effective

2010-12-01 Thread Heinz Diehl
On 01.12.2010, Milan Broz wrote: Anyway, I run several tests on 2.6.37-rc3+ and see no integrity problems (using xfs,ext3 and ext4 over dmcrypt). Not that this might help, but just for testing purposes, I have run all the -rcX from 2.6.36 on with Milan's patch (XFS filesystem) under heavy

Re: What to do about subvolumes?

2010-12-01 Thread J. Bruce Fields
On Wed, Dec 01, 2010 at 02:54:33PM -0500, Josef Bacik wrote: Oh well crud, I was hoping that I could leave the inode numbers as 256 for everything, but I forgot about readdir. So the inode item in the parent would have to have a unique inode number that would get spit out in readdir, but

Re: disk space caching generation missmatch

2010-12-01 Thread Josef Bacik
On Wed, Dec 01, 2010 at 08:56:14PM +0100, Johannes Hirte wrote: On Wednesday 01 December 2010 18:40:18 Josef Bacik wrote: On Wed, Dec 01, 2010 at 05:46:14PM +0100, Johannes Hirte wrote: After enabling disk space caching I've observed several log entries like this: btrfs: free

Re: What to do about subvolumes?

2010-12-01 Thread Jeff Layton
On Wed, 1 Dec 2010 09:21:36 -0500 Josef Bacik jo...@redhat.com wrote: There is one tricky thing. When you create a subvolume, the directory inode that is created in the parent subvolume has the inode number of 256. So if you have a bunch of subvolumes in the same parent subvolume, you are

Re: What to do about subvolumes?

2010-12-01 Thread Josef Bacik
On Wed, Dec 01, 2010 at 03:00:08PM -0500, J. Bruce Fields wrote: On Wed, Dec 01, 2010 at 02:54:33PM -0500, Josef Bacik wrote: Oh well crud, I was hoping that I could leave the inode numbers as 256 for everything, but I forgot about readdir. So the inode item in the parent would have to

Re: What to do about subvolumes?

2010-12-01 Thread J. Bruce Fields
On Wed, Dec 01, 2010 at 03:09:52PM -0500, Josef Bacik wrote: On Wed, Dec 01, 2010 at 03:00:08PM -0500, J. Bruce Fields wrote: On Wed, Dec 01, 2010 at 02:54:33PM -0500, Josef Bacik wrote: Oh well crud, I was hoping that I could leave the inode numbers as 256 for everything, but I forgot

Re: What to do about subvolumes?

2010-12-01 Thread Freddie Cash
On Wed, Dec 1, 2010 at 11:35 AM, Hugo Mills hugo-l...@carfax.org.uk wrote: On Wed, Dec 01, 2010 at 12:38:30PM -0500, Josef Bacik wrote: If you delete your subvolume A, like use the btrfs tool to delete it, you will only be stuck with what you changed in snapshot B.  So if you only changed

Re: dm-crypt barrier support is effective

2010-12-01 Thread Milan Broz
On 12/01/2010 08:34 PM, Jon Nelson wrote: Perhaps this is useful: for myself, I found that when I started using 2.6.37rc3 that postgresql starting having a *lot* of problems with corruption. Specifically, I noted zeroed pages, corruption in headers, all sorts of stuff on /newly created/

Re: What to do about subvolumes?

2010-12-01 Thread Jeff Layton
On Wed, 1 Dec 2010 21:46:03 +0100 Goffredo Baroncelli kreij...@libero.it wrote: On Wednesday, 01 December, 2010, Jeff Layton wrote: A more common use case than CIFS or samba is going to be things like backup programs. They commonly look at inode numbers in order to identify hardlinks and

Re: disk space caching generation missmatch

2010-12-01 Thread Johannes Hirte
On Wednesday 01 December 2010 21:03:13 Josef Bacik wrote: On Wed, Dec 01, 2010 at 08:56:14PM +0100, Johannes Hirte wrote: On Wednesday 01 December 2010 18:40:18 Josef Bacik wrote: On Wed, Dec 01, 2010 at 05:46:14PM +0100, Johannes Hirte wrote: After enabling disk space caching I've

hunt for 2.6.37 dm-crypt+ext4 corruption? (was: Re: dm-crypt barrier support is effective)

2010-12-01 Thread Mike Snitzer
On Wed, Dec 01 2010 at 3:45pm -0500, Milan Broz mb...@redhat.com wrote: On 12/01/2010 08:34 PM, Jon Nelson wrote: Perhaps this is useful: for myself, I found that when I started using 2.6.37rc3 that postgresql starting having a *lot* of problems with corruption. Specifically, I noted

Re: What to do about subvolumes?

2010-12-01 Thread Hugo Mills
On Wed, Dec 01, 2010 at 12:24:28PM -0800, Freddie Cash wrote: On Wed, Dec 1, 2010 at 11:35 AM, Hugo Mills hugo-l...@carfax.org.uk wrote:  The idea is you are only charged for what blocks you have on the disk.  Thanks,   My point was that it's perfectly possible to have blocks on the

Re: disk space caching generation missmatch

2010-12-01 Thread Johannes Hirte
On Wednesday 01 December 2010 22:22:45 Johannes Hirte wrote: On Wednesday 01 December 2010 21:03:13 Josef Bacik wrote: On Wed, Dec 01, 2010 at 08:56:14PM +0100, Johannes Hirte wrote: On Wednesday 01 December 2010 18:40:18 Josef Bacik wrote: On Wed, Dec 01, 2010 at 05:46:14PM +0100,

Re: What to do about subvolumes?

2010-12-01 Thread Freddie Cash
On Wed, Dec 1, 2010 at 1:28 PM, Hugo Mills hugo-l...@carfax.org.uk wrote: On Wed, Dec 01, 2010 at 12:24:28PM -0800, Freddie Cash wrote: On Wed, Dec 1, 2010 at 11:35 AM, Hugo Mills hugo-l...@carfax.org.uk wrote:  The idea is you are only charged for what blocks you have on the disk.  Thanks,

Re: What to do about subvolumes?

2010-12-01 Thread Michael Vrable
On Wed, Dec 01, 2010 at 03:09:52PM -0500, Josef Bacik wrote: On Wed, Dec 01, 2010 at 03:00:08PM -0500, J. Bruce Fields wrote: I think you're already fine: # mkdir TMP # dd if=/dev/zero of=TMP-image bs=1M count=512 # mkfs.btrfs TMP-image # mount -oloop TMP-image

Re: [RFC PATCH 4/4 v2] Btrfs: deal with filesystem state at mount, umount

2010-12-01 Thread Tsutomu Itoh
Hi, I found 1 typo. (2010/12/01 19:21), liubo wrote: Since there is a filesystem state, we should deal with it carefully at mount, umount and remount. - At mount, the FS state should be checked if there is error on these FS. If it does have, btrfsck is recommended. - At umount, the FS

Re: [RFC PATCH 4/4 v2] Btrfs: deal with filesystem state at mount, umount

2010-12-01 Thread liubo
On 12/02/2010 10:29 AM, Tsutomu Itoh wrote: Hi, I found 1 typo. (2010/12/01 19:21), liubo wrote: Since there is a filesystem state, we should deal with it carefully at mount, umount and remount. - At mount, the FS state should be checked if there is error on these FS. If it does

Re: [RFC PATCH 2/4 v2] Btrfs: avoid transaction stuff when readonly

2010-12-01 Thread liubo
On 12/01/2010 06:20 PM, liubo wrote: When the filesystem is readonly, avoid transaction stuff by checking MS_RDONLY at start transaction time. This patch may lead btrfs panic. Since btrfs allows transaction under readonly fs state, which is a bit weird, btrfs does not even check the

Re: [RFC PATCH 2/4 v2] Btrfs: avoid transaction stuff when readonly

2010-12-01 Thread Yan, Zheng
On Thu, Dec 2, 2010 at 11:42 AM, liubo liubo2...@cn.fujitsu.com wrote: On 12/01/2010 06:20 PM, liubo wrote: When the filesystem is readonly, avoid transaction stuff by checking MS_RDONLY at start transaction time. This patch may lead btrfs panic. Since btrfs allows transaction under

Re: What to do about subvolumes?

2010-12-01 Thread Mike Fedyk
On Wed, Dec 1, 2010 at 3:32 PM, Freddie Cash fjwc...@gmail.com wrote: On Wed, Dec 1, 2010 at 1:28 PM, Hugo Mills hugo-l...@carfax.org.uk wrote: On Wed, Dec 01, 2010 at 12:24:28PM -0800, Freddie Cash wrote: On Wed, Dec 1, 2010 at 11:35 AM, Hugo Mills hugo-l...@carfax.org.uk wrote:  The idea is

Re: [RFC PATCH 2/4 v2] Btrfs: avoid transaction stuff when readonly

2010-12-01 Thread liubo
On 12/02/2010 12:28 PM, Yan, Zheng wrote: On Thu, Dec 2, 2010 at 11:42 AM, liubo liubo2...@cn.fujitsu.com wrote: On 12/01/2010 06:20 PM, liubo wrote: When the filesystem is readonly, avoid transaction stuff by checking MS_RDONLY at start transaction time. This patch may lead btrfs panic.

Re: [RFC PATCH 2/4 v2] Btrfs: avoid transaction stuff when readonly

2010-12-01 Thread Mike Fedyk
On Wed, Dec 1, 2010 at 8:28 PM, Yan, Zheng yanzh...@21cn.com wrote: On Thu, Dec 2, 2010 at 11:42 AM, liubo liubo2...@cn.fujitsu.com wrote: On 12/01/2010 06:20 PM, liubo wrote: When the filesystem is readonly, avoid transaction stuff by checking MS_RDONLY at start transaction time. This

Re: [RFC PATCH 2/4 v2] Btrfs: avoid transaction stuff when readonly

2010-12-01 Thread liubo
On 12/02/2010 01:41 PM, Mike Fedyk wrote: On Wed, Dec 1, 2010 at 8:28 PM, Yan, Zheng yanzh...@21cn.com wrote: On Thu, Dec 2, 2010 at 11:42 AM, liubo liubo2...@cn.fujitsu.com wrote: On 12/01/2010 06:20 PM, liubo wrote: When the filesystem is readonly, avoid transaction stuff by checking