Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On 2018年02月14日 00:24, Holger Hoffstätte wrote: > On 02/13/18 13:54, Qu Wenruo wrote: >> On 2018年02月13日 20:26, Holger Hoffstätte wrote: >>> On 02/13/18 12:40, Qu Wenruo wrote: >> The problem is not about how much space it takes, but how many extents >> are here in the filesystem. >>> >>> I

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On 02/13/18 13:54, Qu Wenruo wrote: > On 2018年02月13日 20:26, Holger Hoffstätte wrote: >> On 02/13/18 12:40, Qu Wenruo wrote: > The problem is not about how much space it takes, but how many extents > are here in the filesystem. >> >> I have no idea why btrfs' mount even needs to touch all bl

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On 2018年02月13日 20:26, Holger Hoffstätte wrote: > On 02/13/18 12:40, Qu Wenruo wrote: The problem is not about how much space it takes, but how many extents are here in the filesystem. > > I have no idea why btrfs' mount even needs to touch all block groups to > get going (which seems t

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On Tue, Feb 13, 2018 at 4:46 AM, Qu Wenruo wrote: > > > On 2018年02月13日 20:06, John Ettedgui wrote: > That's fairly straightforward to do, though it should be quite slow so I'd hope not to have to do that too often. >>> >>> Then it could be tried on the most frequently updated files t

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On 2018年02月13日 20:06, John Ettedgui wrote: >>> That's fairly straightforward to do, though it should be quite slow so >>> I'd hope not to have to do that too often. >> >> Then it could be tried on the most frequently updated files then. > > That's an interesting idea. > More than 3/4 of the

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On 02/13/18 12:40, Qu Wenruo wrote: >>> The problem is not about how much space it takes, but how many extents >>> are here in the filesystem. I have no idea why btrfs' mount even needs to touch all block groups to get going (which seems to be the root of the problem), but here's a not so crazy id

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On Tue, Feb 13, 2018 at 3:40 AM, Qu Wenruo wrote: > > > On 2018年02月13日 19:25, John Ettedgui wrote: >> On Tue, Feb 13, 2018 at 3:04 AM, Qu Wenruo wrote: >>> >>> >>> >>> The problem is not about how much space it takes, but how many extents >>> are here in the filesystem. >>> >>> For new fs filled

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On 2018年02月13日 19:25, John Ettedgui wrote: > On Tue, Feb 13, 2018 at 3:04 AM, Qu Wenruo wrote: >> >> >> On 2018年02月13日 18:21, John Ettedgui wrote: >>> Hello there, >>> >>> have you found anything good since then? >> >> Unfortunately, not really much to speed it up. > Oh :/ >> >> This reminds me

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On Tue, Feb 13, 2018 at 3:04 AM, Qu Wenruo wrote: > > > On 2018年02月13日 18:21, John Ettedgui wrote: >> Hello there, >> >> have you found anything good since then? > > Unfortunately, not really much to speed it up. Oh :/ > > This reminds me of the old (and crazy) idea to skip block group build > for

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On 2018年02月13日 18:21, John Ettedgui wrote: > On Thu, Jul 21, 2016 at 1:19 AM, Qu Wenruo wrote: >> >> >> No more. >> >> The dump is already good enough for me to dig for some time. >> >> We don't usually get such large extent tree dump from a real world use case. >> >> It would help us in several

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On Thu, Jul 21, 2016 at 1:19 AM, Qu Wenruo wrote: > > > No more. > > The dump is already good enough for me to dig for some time. > > We don't usually get such large extent tree dump from a real world use case. > > It would help us in several ways, from determine how fragmented a block > group is

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

At 07/21/2016 11:47 PM, Graham Cobb wrote: On 21/07/16 09:19, Qu Wenruo wrote: We don't usually get such large extent tree dump from a real world use case. Let us know if you want some more :-) I have a heavily used single disk BTRFS filesystem with about 3.7TB in use and about 9 million ex

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On 07/25/2016 09:01 AM, David Sterba wrote: On Mon, Jul 18, 2016 at 09:42:50AM -0400, Josef Bacik wrote: This makes search for BLOCK_GROUP_ITEM very very very slow if extent tree is really big. On the handle, we search CHUNK_ITEM very very fast, because CHUNK_ITEM are in their own tree. (CHUNK

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On Mon, Jul 18, 2016 at 09:42:50AM -0400, Josef Bacik wrote: > > > > This makes search for BLOCK_GROUP_ITEM very very very slow if extent tree is > > really big. > > > > On the handle, we search CHUNK_ITEM very very fast, because CHUNK_ITEM are > > in > > their own tree. > > (CHUNK_ITEM and BLOCK_

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On 21/07/16 09:19, Qu Wenruo wrote: > We don't usually get such large extent tree dump from a real world use > case. Let us know if you want some more :-) I have a heavily used single disk BTRFS filesystem with about 3.7TB in use and about 9 million extents. I am happy to provide an extent dump

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

At 07/21/2016 04:13 PM, John Ettedgui wrote: On Thu, Jul 21, 2016 at 1:10 AM Qu Wenruo mailto:quwen...@cn.fujitsu.com>> wrote: Thanks for the info, pretty helpful. After a simple analysis, the defrag did do a pretty good job. --

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

Thanks for the info, pretty helpful. After a simple analysis, the defrag did do a pretty good job. --- | Avg Extent size | Median Extent size | Data Extents | -

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

At 07/18/2016 09:42 PM, Josef Bacik wrote: On 07/16/2016 07:53 PM, Qu Wenruo wrote: On 07/15/2016 07:29 PM, Christian Rohmann wrote: Hey Qu, all On 07/15/2016 05:56 AM, Qu Wenruo wrote: The good news is, we have patch to slightly speedup the mount, by avoiding reading out unrelated tree

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

Qu Wenruo posted on Mon, 18 Jul 2016 17:07:47 +0800 as excerpted: >> Since I'm really surprised on the mount time reduce, especially >> considering the fact that for compression case, max extent size is >> limited to 128K, IMHO defrag won't help much. >> >> Is the 128K limit for the wh

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On 07/16/2016 07:53 PM, Qu Wenruo wrote: On 07/15/2016 07:29 PM, Christian Rohmann wrote: Hey Qu, all On 07/15/2016 05:56 AM, Qu Wenruo wrote: The good news is, we have patch to slightly speedup the mount, by avoiding reading out unrelated tree blocks. In our test environment, it takes 15%

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

At 07/18/2016 04:53 PM, John Ettedgui wrote: On Mon, Jul 18, 2016 at 1:42 AM Qu Wenruo mailto:quwen...@cn.fujitsu.com>> wrote: > So following that, another partition got its mounting time reduced by > about 70% by running a manual defrag (I kept compression on and used > -clzo for

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

At 07/18/2016 04:20 PM, John Ettedgui wrote: On Sun, Jul 17, 2016 at 6:14 PM Qu Wenruo mailto:quwen...@cn.fujitsu.com>> wrote: Well, compression=no only affects any write after the mount option. And balance won't help to convert compressed extent to non-compressed one. But ma

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

At 07/16/2016 07:17 PM, John Ettedgui wrote: On Thu, Jul 14, 2016 at 10:54 PM John Ettedgui mailto:john.etted...@gmail.com>> wrote: On Thu, Jul 14, 2016 at 10:26 PM Qu Wenruo mailto:quwen...@cn.fujitsu.com>> wrote: > Would increasing the leaf size help as well? > nodatac

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On 07/15/2016 07:29 PM, Christian Rohmann wrote: Hey Qu, all On 07/15/2016 05:56 AM, Qu Wenruo wrote: The good news is, we have patch to slightly speedup the mount, by avoiding reading out unrelated tree blocks. In our test environment, it takes 15% less time to mount a fs filled with 16K f

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

Hey Qu, all On 07/15/2016 05:56 AM, Qu Wenruo wrote: > > The good news is, we have patch to slightly speedup the mount, by > avoiding reading out unrelated tree blocks. > > In our test environment, it takes 15% less time to mount a fs filled > with 16K files(2T used space). > > https://patchwor

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

Am Fri, 15 Jul 2016 13:24:45 +0800 schrieb Qu Wenruo : > > as for defrag, all my partitions are already on > > autodefrag, so I assume that should be good. Or is manual once in a > > while a good idea as well? > AFAIK autodefrag will only help if you're doing appending write. > > Manual one wil

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

At 07/15/2016 12:39 PM, John Ettedgui wrote: On Thu, Jul 14, 2016 at 8:56 PM Qu Wenruo mailto:quwen...@cn.fujitsu.com>> wrote: Sorry for the late reply. Oh it's all good, it's only a been a few days. [Slow mount] In fact we also reproduce the same problem, and found the problem.

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

Sorry for the late reply. [Slow mount] In fact we also reproduce the same problem, and found the problem. It's related to the size of extent tree. If the extent tree is large enough, mount needs to do quite a lot of IO to read out all block group items. And such read is random small read (defa

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

2015 22:42 To: "Duncan" <1i5t5.dun...@cox.net> Cc: linux-btrfs@vger.kernel.org Subject: Re: mount btrfs takes 30 minutes, btrfs check runs out of memory I'll try without autodefrag anyways tomorrow just to make sure. And then file a bug report too with however it decides to be

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

rg Subject: Re: mount btrfs takes 30 minutes, btrfs check runs out of memory Chris Murphy posted on Thu, 13 Aug 2015 17:19:41 -0600 as excerpted: > Well I think others have suggested 3000 snapshots and quite a few things > will get very slow. But then also you have autodefrag and I forget the

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

I have 2 snapshots a few days apart for incrementally backing up the volume but that's it. I'll try without autodefrag tomorrow. Vincent -Original Message- From: "Chris Murphy" Sent: Thursday, August 13, 2015 19:19 To: "Btrfs BTRFS" Subject: Re: mount

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

Chris Murphy posted on Thu, 13 Aug 2015 17:19:41 -0600 as excerpted: > Well I think others have suggested 3000 snapshots and quite a few things > will get very slow. But then also you have autodefrag and I forget the > interaction of this with many snapshots since the snapshot aware defrag > code

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On Thu, Aug 13, 2015 at 4:38 PM, Vincent Olivier wrote: > Hi, > > I think I might be having this problem too. 12 x 4TB RAID10 (original makefs, > not converted from ext or whatnot). Says it has ~6TiB left. Centos 7. Dual > Xeon CPU. 32GB RAM. ELRepo Kernel 4.1.5. Fstab options: > noatime,autode

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

he problem when I mount manually without the "noauto,x-systemd.automount" options. Can anyone help ? Thanks. Vincent -Original Message- From: "Austin S Hemmelgarn" Sent: Wednesday, August 5, 2015 07:30 To: "John Ettedgui" Cc: "Qu Wenruo" , &q

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On 2015-08-04 13:36, John Ettedgui wrote: On Tue, Aug 4, 2015 at 4:28 AM, Austin S Hemmelgarn wrote: On 2015-08-04 00:58, John Ettedgui wrote: On Mon, Aug 3, 2015 at 8:01 PM, Qu Wenruo wrote: Although the best practice is staying away from such converted fs, either using pure, newly create

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On Tue, Aug 4, 2015 at 4:28 AM, Austin S Hemmelgarn wrote: > On 2015-08-04 00:58, John Ettedgui wrote: >> >> On Mon, Aug 3, 2015 at 8:01 PM, Qu Wenruo wrote: >>> >>> Although the best practice is staying away from such converted fs, either >>> using pure, newly created btrfs, or convert back to e

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On Mon, Aug 3, 2015 at 9:01 PM, Qu Wenruo wrote: > Oh, converted... > That's too bad. :( > > [[What's wrong with convert]] > Although btrfs is flex enough in theory to fit itself into the free space of > ext* and works fine, > But in practice, ext* is too fragmental in the standard of btrfs, not

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On 2015-08-04 00:58, John Ettedgui wrote: On Mon, Aug 3, 2015 at 8:01 PM, Qu Wenruo wrote: Although the best practice is staying away from such converted fs, either using pure, newly created btrfs, or convert back to ext* before any balance. Unfortunately I don't have enough hard drive space

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

John Ettedgui posted on Mon, 03 Aug 2015 21:58:09 -0700 as excerpted: > Current mount options in fstab: > defaults,noatime,compress=lzo,space_cache,autodefrag 0 0 Just a few hints for a tidier fstab. Feel free to ignore if you don't care, as the practical difference in mount options is nil.

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On Mon, Aug 3, 2015 at 8:01 PM, Qu Wenruo wrote: > Oh, converted... > That's too bad. :( > > [[What's wrong with convert]] > Although btrfs is flex enough in theory to fit itself into the free space of > ext* and works fine, > But in practice, ext* is too fragmental in the standard of btrfs, not t

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

John Ettedgui wrote on 2015/08/03 18:55 -0700: On Mon, Aug 3, 2015 at 6:39 PM, Qu Wenruo wrote: Oh, you were using trace-cmd, that's why the data is so huge. Oh, I thought it was just automating the work for me, but without any sort of impact. I was originally hoping you just copy the tra

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On Mon, Aug 3, 2015 at 6:55 PM, John Ettedgui wrote: > On Mon, Aug 3, 2015 at 6:39 PM, Qu Wenruo wrote: >> >> Oh, you were using trace-cmd, that's why the data is so huge. > Oh, I thought it was just automating the work for me, but without any > sort of impact. >> >> I was originally hoping you j

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On Mon, Aug 3, 2015 at 6:39 PM, Qu Wenruo wrote: > > Oh, you were using trace-cmd, that's why the data is so huge. Oh, I thought it was just automating the work for me, but without any sort of impact. > > I was originally hoping you just copy the trace file, which is human > readable and not so hu

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

John Ettedgui wrote on 2015/07/31 21:35 -0700: On Thu, Jul 30, 2015 at 10:45 PM, John Ettedgui wrote: On Thu, Jul 30, 2015 at 10:40 PM, Qu Wenruo wrote: It seems that you're using Chromium while doing the dump. :) If no CD drive, I'll recommend to use Archlinux installation iso to make a b

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

Quoting John Ettedgui at 2015-07-30-21:10:27(-0700): > On Thu, Jul 30, 2015 at 7:34 PM, Qu Wenruo wrote: > > > > Hi John, > > Thanks for the trace output. > You are welcome, thank you for looking at it! > > > > But it seems that, your root partition is also btrfs, causing a lot of btrfs > > trace

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On Sat, 1 Aug 2015 02:35:39 PM John Ettedgui wrote: > >> It seems that you're using Chromium while doing the dump. :) > >> If no CD drive, I'll recommend to use Archlinux installation iso to make > >> a bootable USB stick and do the dump. > >> (just download and dd would do the trick) > >> As its k

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On Thu, Jul 30, 2015 at 10:45 PM, John Ettedgui wrote: > On Thu, Jul 30, 2015 at 10:40 PM, Qu Wenruo wrote: >> >> It seems that you're using Chromium while doing the dump. :) >> If no CD drive, I'll recommend to use Archlinux installation iso to make a >> bootable USB stick and do the dump. >> (j

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On Thu, Jul 30, 2015 at 10:40 PM, Qu Wenruo wrote: > > > > It seems that you're using Chromium while doing the dump. :) > Ooops I did not think that would be an issue :/ > If no CD drive, I'll recommend to use Archlinux installation iso to make a > bootable USB stick and do the dump. > (just down

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

John Ettedgui wrote on 2015/07/30 22:15 -0700: On Thu, Jul 30, 2015 at 9:52 PM, Qu Wenruo wrote: I'm not familiar with ftrace either, but your trace is good enough already, the only thing needed is to avoid using btrfs as root partition(at least /var/). I've stopped all journaling services f

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

John Ettedgui wrote on 2015/07/30 21:09 -0700: On Thu, Jul 30, 2015 at 7:34 PM, Qu Wenruo wrote: Hi John, Thanks for the trace output. You are welcome, thank you for looking at it! But it seems that, your root partition is also btrfs, causing a lot of btrfs trace from your systemd journa

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On Thu, Jul 30, 2015 at 7:34 PM, Qu Wenruo wrote: > > > Hi John, > Thanks for the trace output. You are welcome, thank you for looking at it! > > But it seems that, your root partition is also btrfs, causing a lot of btrfs > trace from your systemd journal. > Oh yes sorry about that. I actually ha

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

John Ettedgui wrote on 2015/07/29 18:55 +: Hello, I have the same issue and would like to add myself to this thread. My btrfs partition is about 10tb on top of lvm2 and has been taking about a minute to mount in the past few months. Qu Wenruo cn.fujitsu.com > write

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

Hi, Georgi Georgiev wrote on 2015/07/29 14:46 +0900: Using BTRFS on a very large filesystem, and as we put and more data to it, the time it takes to mount it grew to, presently, about 30 minutes. Is there something wrong with the filesystem? Is there a way to bring this time down? ... Here is