Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
2011/2/24 Maria Wikström ma...@ponstudios.se: mån 2011-02-21 klockan 09:51 +0800 skrev Zhong, Xin: The backtrace in your attachment looks like a known bug of 2.6.37 which have already been fixed in 2.6.38. I have no idea why latest btrfs still hang in your environment if there's no debug info... Haha, yes that's very hard :) 2.6.38-rc6 and btrfs-unstable behaves the same way. I can close the process with ctrl+c and it disappear a few seconds later. There is no CPU usage. Reading works because I can start htop and watch svn info disappear, but everything writing to btrfs slows down to a crawl. It takes about 1 minute to log in. So I had to put the logs on an other partition using ext3 to get the output from sysrq+t. I believe I've been experiencing this issue also. However, my problem usually results in a No space left on device error rather than a lock-up or crash. But I've bisected my issue to this patch, and my btrfs fi show and btrfs fi df looks similar to others who've posted to this tread with all my space being allocated, but not used. I've been playing around with ftrace to try to get some information on the issue. Since I'm getting a soft error, I can used a command like echo 1 tracing_on; emerge -1av openmotif; echo 0 tracing_on to end the trace as soon as the build fails. The traces are probably too large for the M/L (~275kb compressed), so I've put them up on my local server in case anybody can find them useful: http://dontpanic.dyndns.org/emerge-openmotif-ftrace.gz I'm still only capturing the tail end of the problem, but maybe someone will find these insightful. Let me know if you want me to increase the ftrace buffer size or insert some trace_printk debugging statements. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
Excerpts from Johannes Hirte's message of 2011-02-23 18:02:44 -0500: On Wednesday 23 February 2011 22:56:27 Chris Mason wrote: Excerpts from Zhong, Xin's message of 2011-02-23 02:27:05 -0500: In the dmesg of rc4, I can see svn hang in shrink_dellalloc and there's two flush-btrfs threads hang there too. Josef, it seems you are the expert in this area. Could you take a quick look? Thanks! Ok, it does look like the fluhs-btrfs threads are busy trying to flush things. Could you please do a btrfs-show and a btrfs fi df /xxx (where xxx is your mount point) and send the results here? -chris failed to read /dev/sr0 Label: none uuid: 00eab15f-c4cf-4403-a529-9bc11fa50167 Total devices 1 FS bytes used 47.72GB devid1 size 65.69GB used 65.69GB path /dev/sda2 Label: none uuid: c6f4e6e6-c4ba-4394-9e9c-bbc3d0b32793 Total devices 1 FS bytes used 9.48GB devid1 size 20.01GB used 20.01GB path /dev/sda1 Btrfs v0.19-35-g1b444cd-dirty and btrfs fi df on / Data: total=15.49GB, used=8.35GB System, DUP: total=8.00MB, used=12.00KB System: total=4.00MB, used=0.00 Metadata, DUP: total=2.25GB, used=1.13GB /home Data: total=63.42GB, used=47.47GB System: total=4.00MB, used=16.00KB Metadata: total=2.27GB, used=251.34MB The bug is reproducable on both filesystems. Ok, you've got a good amount of metadata space free, but it is frantically trying to make room for the delayed allocation. Let me see if I can recreate this setup here. -chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
On Thu, Feb 24, 2011 at 10:00 AM, Chris Mason chris.ma...@oracle.com wrote: Excerpts from Mitch Harder's message of 2011-02-24 10:55:15 -0500: 2011/2/24 Maria Wikström ma...@ponstudios.se: mån 2011-02-21 klockan 09:51 +0800 skrev Zhong, Xin: The backtrace in your attachment looks like a known bug of 2.6.37 which have already been fixed in 2.6.38. I have no idea why latest btrfs still hang in your environment if there's no debug info... Haha, yes that's very hard :) 2.6.38-rc6 and btrfs-unstable behaves the same way. I can close the process with ctrl+c and it disappear a few seconds later. There is no CPU usage. Reading works because I can start htop and watch svn info disappear, but everything writing to btrfs slows down to a crawl. It takes about 1 minute to log in. So I had to put the logs on an other partition using ext3 to get the output from sysrq+t. I believe I've been experiencing this issue also. However, my problem usually results in a No space left on device error rather than a lock-up or crash. But I've bisected my issue to this patch, and my btrfs fi show and btrfs fi df looks similar to others who've posted to this tread with all my space being allocated, but not used. Sorry, which patch did you bisect the problem down to? The patch at the head of this thread: Btrfs: pwrite blocked when writing from the mmaped buffer of the same page -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
Excerpts from Mitch Harder's message of 2011-02-24 11:03:07 -0500: On Thu, Feb 24, 2011 at 10:00 AM, Chris Mason chris.ma...@oracle.com wrote: Excerpts from Mitch Harder's message of 2011-02-24 10:55:15 -0500: 2011/2/24 Maria Wikström ma...@ponstudios.se: mån 2011-02-21 klockan 09:51 +0800 skrev Zhong, Xin: The backtrace in your attachment looks like a known bug of 2.6.37 which have already been fixed in 2.6.38. I have no idea why latest btrfs still hang in your environment if there's no debug info... Haha, yes that's very hard :) 2.6.38-rc6 and btrfs-unstable behaves the same way. I can close the process with ctrl+c and it disappear a few seconds later. There is no CPU usage. Reading works because I can start htop and watch svn info disappear, but everything writing to btrfs slows down to a crawl. It takes about 1 minute to log in. So I had to put the logs on an other partition using ext3 to get the output from sysrq+t. I believe I've been experiencing this issue also. However, my problem usually results in a No space left on device error rather than a lock-up or crash. But I've bisected my issue to this patch, and my btrfs fi show and btrfs fi df looks similar to others who've posted to this tread with all my space being allocated, but not used. Sorry, which patch did you bisect the problem down to? The patch at the head of this thread: Btrfs: pwrite blocked when writing from the mmaped buffer of the same page Hmmm, that patch shouldn't be changing our performance under delalloc pressure, and it really shouldn't impact early enospc. -chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
On Thu, Feb 24, 2011 at 10:19 AM, Chris Mason chris.ma...@oracle.com wrote: Excerpts from Mitch Harder's message of 2011-02-24 11:03:07 -0500: On Thu, Feb 24, 2011 at 10:00 AM, Chris Mason chris.ma...@oracle.com wrote: Excerpts from Mitch Harder's message of 2011-02-24 10:55:15 -0500: 2011/2/24 Maria Wikström ma...@ponstudios.se: mån 2011-02-21 klockan 09:51 +0800 skrev Zhong, Xin: The backtrace in your attachment looks like a known bug of 2.6.37 which have already been fixed in 2.6.38. I have no idea why latest btrfs still hang in your environment if there's no debug info... Haha, yes that's very hard :) 2.6.38-rc6 and btrfs-unstable behaves the same way. I can close the process with ctrl+c and it disappear a few seconds later. There is no CPU usage. Reading works because I can start htop and watch svn info disappear, but everything writing to btrfs slows down to a crawl. It takes about 1 minute to log in. So I had to put the logs on an other partition using ext3 to get the output from sysrq+t. I believe I've been experiencing this issue also. However, my problem usually results in a No space left on device error rather than a lock-up or crash. But I've bisected my issue to this patch, and my btrfs fi show and btrfs fi df looks similar to others who've posted to this tread with all my space being allocated, but not used. Sorry, which patch did you bisect the problem down to? The patch at the head of this thread: Btrfs: pwrite blocked when writing from the mmaped buffer of the same page Hmmm, that patch shouldn't be changing our performance under delalloc pressure, and it really shouldn't impact early enospc. I've bisected this issue around where this patch went into git, and I've also constructed a testing patch that reverts this patch, and placed it on top of the current Btrfs git sources (I understand this patch addresses a real issue, this was just for testing). It could be that this patch just uncovers another problem, but all my tests seem to point to this patch triggering this issue. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH] Btrfs: add ioctl to set compress or cow per file/dir
On 2011-02-24, at 2:40 AM, liubo wrote: #define FS_DIRECTIO_FL 0x0010 /* Use direct i/o */ +#define FS_NOCOW_FL 0x0020 /* Do not cow file */ +#define FS_COW_FL0x0010 /* Cow file */ #define FS_RESERVED_FL 0x8000 /* reserved for ext2 lib */ I'm assuming that FS_COW_FL should not be the same as FS_DIRECTIO_FL? Cheers, Andreas -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH] Btrfs: add ioctl to set compress or cow per file/dir
Excerpts from Andreas Dilger's message of 2011-02-24 13:37:52 -0500: On 2011-02-24, at 2:40 AM, liubo wrote: #define FS_DIRECTIO_FL0x0010 /* Use direct i/o */ +#define FS_NOCOW_FL0x0020 /* Do not cow file */ +#define FS_COW_FL0x0010 /* Cow file */ #define FS_RESERVED_FL0x8000 /* reserved for ext2 lib */ I'm assuming that FS_COW_FL should not be the same as FS_DIRECTIO_FL? No, we can do DIRECTIO with COW. -chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped buffer of the same page
On Mon, Feb 21, 2011 at 09:51:11AM +0800, Zhong, Xin wrote:
The backtrace in your attachment looks like a known bug of 2.6.37 which have
already been fixed in 2.6.38. I have no idea why latest btrfs still hang in
your environment if there's no debug info...
Hi list.
I'm watching this list for a while because it seems, that I'm also
affected by this bug. In the archives I found someone with Gentoo system
with freezing `svn info' (thats why I joined). Well, seems that I have
same issue here.
Attached latest _rc kernel sysrq+t output (first part when the `svn
info' freezed on libgcrypt, and second after ctrl+c that emerge).
Seems that my backtrace is small compared to Marias. I'm missing some
features in kernel to get larger backtraces?
Piotr Szymaniak.
-Original Message-
From: Maria Wikström [mailto:ma...@ponstudios.se]
Sent: Friday, February 18, 2011 7:32 PM
To: Zhong, Xin
Cc: Johannes Hirte; linux-btrfs@vger.kernel.org
Subject: RE: [PATCH v2]Btrfs: pwrite blocked when writing from the mmaped
buffer of the same page
Seems like my reply got eaten by the lists spam filter, so I resend with
attachment compressed. Should have thought of that :p
fre 2011-02-11 klockan 12:39 +0800 skrev Zhong, Xin:
Hi,
Could you paste the output of sysrq+t here? Thanks!
Yes, it's in the attachment. I tried latest btrfs from git (last commit Mon
Feb 14 00:45:29 2011 +) but it hang so bad that I couldn't get the output
from sysrq+t to hit the disk. So the output is from vanilla
2.6.37
// Maria
-Original Message-
From: Johannes Hirte [mailto:johannes.hi...@fem.tu-ilmenau.de]
Sent: Wednesday, February 02, 2011 7:35 AM
To: Zhong, Xin
Cc: Maria Wikström; linux-btrfs@vger.kernel.org
Subject: Re: [PATCH v2]Btrfs: pwrite blocked when writing from the
mmaped buffer of the same page
On Friday 28 January 2011 04:53:24 Zhong, Xin wrote:
Could you describe the steps to recreate it?
It will be a great help for me to look further. Thanks!
It's a little strange. I have to systems with btrfs, both
Gentoo-based. One is affected by this bug the other is not. On the
affected system it is enough to do a 'emerge dev-libs/libgcrypt' that
should normaly compile and install libgcrypt. The emerge command is
part of portage, the package management of Gentoo.
The strace output looks similar to the one from Maria:
N?r??yb?X??ǧv?^?){.n?+{?n?߲)w* jg??? ?ݢj/???z?ޖ??2?ޙ?)ߡ?a???? ?G???h??j:+v???w??٥
--
Druzyna futbolowa tutaj jest do niczego, ale ucze sie troche pilki
noznej. Trener mowi, ze pilka nozna to futbol dla inteligentnych, a
futbol to futbol dla kretynow.
-- Stephen King, The Dead Zone
vanilla-2.6.38-rc6_sysrq-t.bz2
Description: BZip2 compressed data
pgpwOlW7jCAZg.pgp
Description: PGP signature
Re: [2.6.38-rc6] create-rebalance-mount crash...
Hi, Chris and Liu
On thu, 24 Feb 2011 09:35:32 -0500, Chris Mason wrote:
[SNIP]
[PATCH] btrfs: fix OOPS of empty filesystem after balance
btrfs will exclude unused block groups via a thread.
When a empty filesystem is balanced, the block group with tag DATA may be
dropped,
and after umount, this will lead to OOPS when we mount it again.
Thanks for tracking this down! Comment below:
[SNIP]
-static void init_global_block_rsv(struct btrfs_fs_info *fs_info)
+static int init_global_block_rsv(struct btrfs_fs_info *fs_info)
{
struct btrfs_space_info *space_info;
+space_info = __find_space_info(fs_info, BTRFS_BLOCK_GROUP_DATA);
+if (!space_info)
+return -EAGAIN;
+
space_info = __find_space_info(fs_info, BTRFS_BLOCK_GROUP_SYSTEM);
fs_info-chunk_block_rsv.space_info = space_info;
fs_info-chunk_block_rsv.priority = 10;
@@ -3884,6 +3888,8 @@ static void init_global_block_rsv(struct btrfs_fs_info
*fs_info)
btrfs_add_durable_block_rsv(fs_info,fs_info-delalloc_block_rsv);
update_global_block_rsv(fs_info);
+
+return 0;
}
static void release_global_block_rsv(struct btrfs_fs_info *fs_info)
@@ -8514,7 +8520,13 @@ int btrfs_read_block_groups(struct btrfs_root *root)
set_block_group_ro(cache);
}
-init_global_block_rsv(info);
+again:
+ret = init_global_block_rsv(info);
+if (ret == -EAGAIN) {
+update_space_info(info, BTRFS_BLOCK_GROUP_DATA, 0, 0,
+space_info);
+goto again;
+}
ret = 0;
Are we looping here because we expect the init_global_block_rsv to fail
more than once? If so we need a cond_resched or something in there.
But if the EAGAIN is only returned once we should avoid the loop and
open code the call again.
I don't think we should create a space information object in
init_global_block_rsv(),
which just does initialize the global block reservation object.
I think it is better to split btrfs_read_block_group() to three steps.
Step 1: create and initialize the space information object.
Step 2: read the block groups and update the space information.
Step 3: initialize the global block reservation object.
In this way, the logic of the source is clear, and avoid sometrivial mistake.
BTW: I found the btrfs filesystem just has three types of data(file data, meta
data,
system meta data), why not add a space information array with three elements
into fs_info?
In this way, we can simplify the source code of the space information, and
needn't use
RCU lock to protect the space information object list. (I didn't find a lock to
protect
the space information object list in the write-side. Is it right?)
Regards
Miao
-chris
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH] Btrfs: add ioctl to set compress or cow per file/dir
On 02/24/2011 10:54 PM, Chris Mason wrote: Excerpts from liubo's message of 2011-02-24 04:40:55 -0500: Data compression and data cow are controlled across the entire FS by mount options right now. ioctls are needed to set this on a per file or per directory basis. This has been proposed previously, but VFS developers wanted us to use generic ioctls rather than btrfs-specific ones. We need to fit these into the existing per-inode flags, and to use the generic FS_IOCTL_SETFLAGS ioctl. For data compression, there are the existing compression flags of vfs inode, while for datacow, there is no flag to indicate it, which we need to add. So, what we will do is to add datacow flag in vfs inode flags and then to set or to unset btrfs compress/cow flag on the corresponding btrfs inode's flag per file or per directory. Moreover, we also add a compression type ioctl to make this feature more flexible. I really expect some advices and comments on the followings: - In this patch, I made a special ioctl to set compress type, and to record the compress_type per inode on disk, I've consumed some reserved space of btrfs_inode_item, so is this acceptable? I don't expect people to mix compression types on the disk. There really should just be one true compression method (probably LZO once it has been established for a while). So, I'd prefer that we store this in the super, and just have flags in the inode for enabling or disabling compression. It sounds nice and will make code neatly. :) So, all files directories will share the same compress type stored in the super. Meanwhile, I got another idea from my collegue, could we just owe the whole compress type thing to new proper mount options, ie, mount xxx xxx -o compress=a,inode_compress=b? Seems that this makes mount more flexible. It does make it more flexible, but I think sometimes extra flexibility leads to more QA time and isn't often used by the actual users ;) ok. - When we are inclined to set inode's compression type, should it be a force mode? This is much like the difference between mount as compress and mount as compress-force. I'd store this as flags in the super too. ok. - For directory basis, after compress/cow ioctl on it, any files that are created or renamed in it, or moved into it, will inherit the directory's compress and datacow attribute. Here comes to some disputes, is it right that renamed and moved files also inherit the father directory's compress datacow attribute? And if what we are dealing with is directory, should this behaviour be recursive or not? I'm inclined to leave these recursive things to btrfs-progs if this is necessary. I'd say that if we rename a file into a directory it does inherit, but not make it recursive. ok, got it. I will send a new version based on this thread. Thanks a lot for reviewing! thanks, liubo -chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH] Btrfs: add ioctl to set compress or cow per file/dir
On 02/25/2011 02:39 AM, Chris Mason wrote: Excerpts from Andreas Dilger's message of 2011-02-24 13:37:52 -0500: On 2011-02-24, at 2:40 AM, liubo wrote: #define FS_DIRECTIO_FL0x0010 /* Use direct i/o */ +#define FS_NOCOW_FL0x0020 /* Do not cow file */ +#define FS_COW_FL0x0010 /* Cow file */ #define FS_RESERVED_FL0x8000 /* reserved for ext2 lib */ I'm assuming that FS_COW_FL should not be the same as FS_DIRECTIO_FL? No, we can do DIRECTIO with COW. Sorry for my fault, thanks for pointing it out. thanks, liubo -chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
