Re: Rename BTRfs to MuchSlowerFS ?
On 09/05/2011 06:29 PM, Hugo Mills wrote: On Mon, Sep 05, 2011 at 12:25:21PM -0400, Christoph Hellwig wrote: On Mon, Sep 05, 2011 at 06:23:23PM +0200, Tomasz Chmielewski wrote: That's because dpkg is known for using (f)sync very heavily. btrfs honours the sync request in all cases, so it's much much slower than ext3, which doesn't. Hmm, is it really the case with ext3/ext4 (ignoring fsync in some cases)? Sounds like a bug in ext3/ext4 then. Is it documented anywhere where ext3/ext4 would just silently ignore fsync? They don't. Unteil recently ext3 and reiserfs would not flush the disk caches unless enabled by a mount option, but even that has recently been fixed. Apologies for disseminating my misunderstanding, then. Hugo. I'm wondering if this may be the very same unacceptable behavior as discussed earlier also in the following message threads: btrfs always writes something after sync news://news.gmane.org:119/cabvos09hntxyuscjbg+byegc5y53j4det6b49rx1aarpqfv...@mail.gmail.com Re: Writes in idle/How to debug filesystem activity news://news.gmane.org:119/4e5c8a20.7030...@wpkg.org Re: Applications using fsync cause hangs for several seconds every few minutes news://news.gmane.org:119/4e41a6a4.9070...@uvm.edu Btrfs slowdown news://news.gmane.org:119/cao47_-9blkwugdeuzalqhsq9tzkauao8fmqey1ppk9a2hb+...@mail.gmail.com PLEASE TEST: Everybody who is seeing weird and long hangs news://news.gmane.org:119/4e36c47e.70...@redhat.com and this: curious writes on mounted, not used btrfs filesystem http://thread.gmane.org/gmane.comp.file-systems.btrfs/10840 and probably this: 2.6.37: Multi-second I/O latency while untarring http://www.spinics.net/lists/linux-btrfs/msg08516.html Hasn't the slowdown been fixed yet? For those experiencing the slowdown issue btrfs is entirely useless... (I had to stop using btrfs because of these frequent slowdowns, while btrfs was doing most likely needless disk writes for multiple seconds, I couldn't work at all) Can someone look into this if this is not fixed? There are numerous analysis reports in the above threads, without any constructive result. Thanks -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Applications using fsync cause hangs for several seconds every few minutes
This is most probably related to the same regression seen after 2.6.38, my blocked comment on 3 August included an indication to that the behavior was present in my distro 2.6.38 kernel too, it just was appearing after a considerably longer uptime (on my desktop system using btrfs as rootfs on an Intel ICH10 driven SATA HDD). I have reverted my / to ext4 since, and I'm okay with it, although I would be very happy to see some improvement on this serious-for-me issue. Btrfs slowdown news://news.gmane.org:119/cao47_-9blkwugdeuzalqhsq9tzkauao8fmqey1ppk9a2hb+...@mail.gmail.com Also, a patch by Josef Bacik was an attempt for fixing this, but no one reported about testing it on an affected system, it did not eliminate the slowdowns for me: PLEASE TEST: Everybody who is seeing weird and long hangs news://news.gmane.org:119/4e36c47e.70...@redhat.com My comment was going as an aswer to Mck's post in Btrfs slowdown thread, where I reported about this in a little more detail - but it never appeared on the list. I try including it now: I'm confirming this too. Following advices given on #btrfs irc, I have applied Josef's second patch for fs/btrfs/extent_io.c and I'm reporting that it did NOT make the slowdowns disappear on 3.0 kernels (even with some rather different configs). The HDD thrashing appeared on all other kernel versions I tried, higher than 2.6.37. Initially, I had been into looking for a latest known good kernel (to prepare a proper git bisect as cwillu advised) and at first I also felt like 2.6.38 does not show this miserable behaviour. But later it turned out this was only for approximately 2 days of uptime. Given enough time, the lock-ups appeared on 2.6.38 too. Although they were not that apparent than on later kernel versions, and the individual lockups took much less time with 2.6.38 running for 2 days (binary Sabayon Linux repository kernel). My HDD, with btrfs as / on it emits very distinct (and loud enough) noises with a slightly different character for reads and writes - and I can actually hear the disk's repetitive seek pattern during a such thrashing period. Based on that, I guess it must be the exact same thing happening with 2.6.38 as with later kernels because they sound very similar. They last much shorter but they have a similarly repetitive seeking nature with other I/O severely throttled and I believe it is write what is mostly what's happening during a lockup. So I concluded that I failed to identify a known good version so far. I didn't have time to get into earlier kernels than .38. (Tried .37, but for too brief of uptime to claim they did not appear when I was on .37) Similar with my current kernel. It started happening after about 12 hours of running the machine using # uname -a Linux insula 3.0.0-git15genseed #2 SMP PREEMPT Tue Aug 2 20:10:05 CEST 2011 x86_64 Intel(R) Core(TM)2 Duo CPU E4500 @ 2.20GHz GenuineIntel GNU/Linux As appended string reflects, it is a custom kernel, it has Josef's patch applied with the config attached.Tried to patch my distro's 3.0 kernel, no change was experienced with regards to the issue (iirc it was even a lot worse). Let me know if I can contribute with anything that would be valuable for the developers towards elimination of this very nasty bug. Now, after 23 hours of uptime, my PC has become almost unusable. Currently there's about 8 seconds thrashing, 10 seconds not thrashing, and during thrashing, all other (disk) I/O is practically blocked. SysRq+W under thrashing (dunno how informative it is, but here's one): [62279.779382] SysRq : Show Blocked State [62279.779389] taskPC stack pid father [62279.779404] btrfs-submit-0 D 5616 4678 2 0x [62279.779413] 88012b1370d0 0046 8801 8182c020 [62279.779422] 880128d39fd8 00010480 4000 880128d38000 [62279.779429] 880128d39fd8 00010480 88012b1370d0 00010480 [62279.779437] Call Trace: [62279.779449] [812779c6] ? cfq_set_request+0x33e/0x37e [62279.779456] [81277063] ? cfq_cic_lookup+0x35/0x139 [62279.779462] [812773a2] ? cfq_may_queue+0x51/0x6e [62279.779470] [8143ed81] ? io_schedule+0x4e/0x63 [62279.779477] [8126b276] ? get_request_wait+0xaa/0x10e [62279.779484] [8104f2ad] ? wake_up_bit+0x23/0x23 [62279.779490] [8126c2a6] ? __make_request+0x175/0x26b [62279.779496] [8126a267] ? generic_make_request+0x224/0x289 [62279.779502] [8126a37f] ? submit_bio+0xb3/0xbc [62279.779509] [81372238] ? dm_any_congested+0x4f/0x57 [62279.779516] [81206de6] ? run_scheduled_bios+0x246/0x3b1 [62279.779523] [8120c791] ? worker_loop+0x180/0x4bb [62279.779529] [8120c611] ? btrfs_queue_worker+0x24e/0x24e [62279.779535] [8104eee7] ? kthread+0x7a/0x82 [62279.779542]
Re: Applications using fsync cause hangs for several seconds every few minutes
Are these processes principally btrfs-submit and btrfs-transacti in particular? Then it may be related to my very similar issue reported earlier. On 08/18/2011 08:47 AM, Chris Samuel wrote: On 18/08/11 00:29, Michael Cronenworth wrote: I'm running kernel 3.0 (Fedora 15's 2.6.40) on two boxes and I have not seen slow downs or hangs. I use Firefox. I've got btrfs on an external USB drive with the 3.0.1 kernel and I see that sync seems to take an age, according to iotop it seems that the btrfs processes are hitting it quite hard, IIRC. cheers, Chris -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Applications using fsync cause hangs for several seconds every few minutes
On 08/18/2011 09:29 AM, Andrew Guertin wrote: * Many processes occasionally hang for a short time * When this happens, my cpu monitor shows a short burst of cpu activity (100% of 1 core) followed by a longer period of IO * When this happens, iotop shows [btrfs-submit-0] and [btrfs-transacti] at the top of the list * Behavior slowly increases in duration (and frequency?) over time, and goes away with a reboot * Heavy IO makes behavior appear faster ... and the following behaviors before commit 4e69b59: * Occasional spikes of IO on cpu monitor concurrent with [btrfs-submit-0] and [btrfs-transacti] at top of iotop * No hangs, even when that occurs Yes, exactly that happened in my case too. Yours is a much more precise description! I did not diagnose 2.6.38 further because I just wanted to establish a known-good version and at first sight (2 days uptime) my HDD behavior showed that it cannot be good if _any_ HDD thrashing appears at all in the first place... I was able to work with the computer during those IO spikes on 2.6.38 too, although it was observable that the HDD is being thrased (meanwhile, LED was almost constant lit). But it didn't cause other programs to be unresponsive, I confirm... I wasn't taking notes or anything though, so I'm not 100% certain I was observing or interpreting or remembering everything correctly. --Andrew -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
