Re: Status of FST and mount times

On 2018-02-21 10:56, Hans van Kranenburg wrote: On 02/21/2018 04:19 PM, Ellis H. Wilson III wrote: $ sudo btrfs fi df /mnt/btrfs Data, single: total=3.32TiB, used=3.32TiB System, DUP: total=8.00MiB, used=384.00KiB Metadata, DUP: total=16.50GiB, used=15.82GiB GlobalReserve, single:

Re: Status of FST and mount times

On 2018年02月21日 22:49, Ellis H. Wilson III wrote: > On 02/20/2018 08:49 PM, Qu Wenruo wrote: > On 2018年02月16日 22:12, Ellis H. Wilson III wrote: >> $ sudo btrfs-debug-tree -t chunk /dev/sdb | grep CHUNK_ITEM | wc -l >> 3454 > >> Increasing node size may reduce extent tree size.

Re: Status of FST and mount times

On Wed, Feb 21, 2018 at 9:49 AM, Ellis H. Wilson III wrote: > On 02/20/2018 08:49 PM, Qu Wenruo wrote: > > On 2018年02月16日 22:12, Ellis H. Wilson III wrote: >> >> $ sudo btrfs-debug-tree -t chunk /dev/sdb | grep CHUNK_ITEM | wc -l >> 3454 > > >>

Re: Status of FST and mount times

On 02/21/2018 04:19 PM, Ellis H. Wilson III wrote: > On 02/21/2018 10:03 AM, Hans van Kranenburg wrote: >> On 02/21/2018 03:49 PM, Ellis H. Wilson III wrote: >>> On 02/20/2018 08:49 PM, Qu Wenruo wrote: My suggestion is to use balance to reduce number of block groups, so we could do less

Re: Status of FST and mount times

On 02/21/2018 03:49 PM, Ellis H. Wilson III wrote: > On 02/20/2018 08:49 PM, Qu Wenruo wrote: > On 2018年02月16日 22:12, Ellis H. Wilson III wrote: >> $ sudo btrfs-debug-tree -t chunk /dev/sdb | grep CHUNK_ITEM | wc -l >> 3454 > >> Increasing node size may reduce extent tree size.

Re: Status of FST and mount times

On 02/20/2018 08:49 PM, Qu Wenruo wrote: On 2018年02月16日 22:12, Ellis H. Wilson III wrote: $ sudo btrfs-debug-tree -t chunk /dev/sdb | grep CHUNK_ITEM | wc -l 3454 Increasing node size may reduce extent tree size. Although at most reduce one level AFAIK. But considering that the higher the

Re: Status of FST and mount times

On 2018年02月20日 23:41, Austin S. Hemmelgarn wrote: > On 2018-02-20 09:59, Ellis H. Wilson III wrote: >> On 02/16/2018 07:59 PM, Qu Wenruo wrote: >>> On 2018年02月16日 22:12, Ellis H. Wilson III wrote: $ sudo btrfs-debug-tree -t chunk /dev/sdb | grep CHUNK_ITEM | wc -l 3454 >>> >>> OK, this

Re: Status of FST and mount times

On 2018-02-20 09:59, Ellis H. Wilson III wrote: On 02/16/2018 07:59 PM, Qu Wenruo wrote: On 2018年02月16日 22:12, Ellis H. Wilson III wrote: $ sudo btrfs-debug-tree -t chunk /dev/sdb | grep CHUNK_ITEM | wc -l 3454 OK, this explains everything. There are too many chunks. This means at mount you

Re: Status of FST and mount times

On 02/16/2018 07:59 PM, Qu Wenruo wrote: On 2018年02月16日 22:12, Ellis H. Wilson III wrote: $ sudo btrfs-debug-tree -t chunk /dev/sdb | grep CHUNK_ITEM | wc -l 3454 OK, this explains everything. There are too many chunks. This means at mount you need to search for block group item 3454 times.

Re: Status of FST and mount times

On 2018年02月16日 22:12, Ellis H. Wilson III wrote: > On 02/15/2018 08:55 PM, Qu Wenruo wrote: >> On 2018年02月16日 00:30, Ellis H. Wilson III wrote: >>> Very helpful information.  Thank you Qu and Hans! >>> >>> I have about 1.7TB of homedir data newly rsync'd data on a single >>> enterprise 7200rpm

Re: Status of FST and mount times

On 02/16/2018 09:42 AM, Ellis H. Wilson III wrote: On 02/16/2018 09:20 AM, Hans van Kranenburg wrote: Well, imagine you have a big tree (an actual real life tree outside) and you need to pick things (e.g. apples) which are hanging everywhere. So, what you need to to is climb the tree, climb on

Re: Status of FST and mount times

On 02/16/2018 09:20 AM, Hans van Kranenburg wrote: Well, imagine you have a big tree (an actual real life tree outside) and you need to pick things (e.g. apples) which are hanging everywhere. So, what you need to to is climb the tree, climb on a branch all the way to the end where the first

Re: Status of FST and mount times

On 02/16/2018 03:12 PM, Ellis H. Wilson III wrote: > On 02/15/2018 08:55 PM, Qu Wenruo wrote: >> On 2018年02月16日 00:30, Ellis H. Wilson III wrote: >>> Very helpful information.  Thank you Qu and Hans! >>> >>> I have about 1.7TB of homedir data newly rsync'd data on a single >>> enterprise 7200rpm

Re: Status of FST and mount times

On 02/15/2018 08:55 PM, Qu Wenruo wrote: On 2018年02月16日 00:30, Ellis H. Wilson III wrote: Very helpful information.  Thank you Qu and Hans! I have about 1.7TB of homedir data newly rsync'd data on a single enterprise 7200rpm HDD and the following output for btrfs-debug: extent tree key

Re: Status of FST and mount times

On 2018年02月16日 00:30, Ellis H. Wilson III wrote: > On 02/15/2018 06:12 AM, Hans van Kranenburg wrote: >> On 02/15/2018 02:42 AM, Qu Wenruo wrote: >>> Just as said by Nikolay, the biggest problem of slow mount is the size >>> of extent tree (and HDD seek time) >>> >>> The easiest way to get a

Re: Status of FST and mount times

On 2018-02-15 11:58, Ellis H. Wilson III wrote: On 02/15/2018 11:51 AM, Austin S. Hemmelgarn wrote: There are scaling performance issues with directory listings on BTRFS for directories with more than a few thousand files, but they're not well documented (most people don't hit them because

Re: Status of FST and mount times

On 02/15/2018 11:51 AM, Austin S. Hemmelgarn wrote: There are scaling performance issues with directory listings on BTRFS for directories with more than a few thousand files, but they're not well documented (most people don't hit them because most applications are designed around the

Re: Status of FST and mount times

On 2018-02-15 10:42, Ellis H. Wilson III wrote: On 02/14/2018 06:24 PM, Duncan wrote: Frame-of-reference here: RAID0.  Around 70TB raw capacity.  No compression.  No quotas enabled.  Many (potentially tens to hundreds) of subvolumes, each with tens of snapshots.  No control over size or number

Re: Status of FST and mount times

On 02/15/2018 01:14 AM, Chris Murphy wrote: On Wed, Feb 14, 2018 at 9:00 AM, Ellis H. Wilson III wrote: Frame-of-reference here: RAID0. Around 70TB raw capacity. No compression. No quotas enabled. Many (potentially tens to hundreds) of subvolumes, each with tens of

Re: Status of FST and mount times

On 02/14/2018 06:24 PM, Duncan wrote: Frame-of-reference here: RAID0. Around 70TB raw capacity. No compression. No quotas enabled. Many (potentially tens to hundreds) of subvolumes, each with tens of snapshots. No control over size or number of files, but directory tree (entries per dir and

Re: Status of FST and mount times

On 02/15/2018 06:12 AM, Hans van Kranenburg wrote: On 02/15/2018 02:42 AM, Qu Wenruo wrote: Just as said by Nikolay, the biggest problem of slow mount is the size of extent tree (and HDD seek time) The easiest way to get a basic idea of how large your extent tree is using debug tree: #

Re: Status of FST and mount times

On 02/15/2018 02:42 AM, Qu Wenruo wrote: > > > On 2018年02月15日 01:08, Nikolay Borisov wrote: >> >> >> On 14.02.2018 18:00, Ellis H. Wilson III wrote: >>> Hi again -- back with a few more questions: >>> >>> Frame-of-reference here: RAID0.  Around 70TB raw capacity.  No >>> compression.  No quotas

Re: Status of FST and mount times

On Wed, Feb 14, 2018 at 9:00 AM, Ellis H. Wilson III wrote: > Frame-of-reference here: RAID0. Around 70TB raw capacity. No compression. > No quotas enabled. Many (potentially tens to hundreds) of subvolumes, each > with tens of snapshots. Even if non-catastrophic to lose

Re: Status of FST and mount times

On Wed, Feb 14, 2018 at 10:08 AM, Nikolay Borisov wrote: > V1 for large filesystems is jut awful. Facebook have been experiencing > the pain hence they implemented v2. You can view the spacecache tree as > the complement version of the extent tree. v1 cache is implemented as a

Re: Status of FST and mount times

On 2018年02月15日 10:15, Duncan wrote: > Qu Wenruo posted on Thu, 15 Feb 2018 09:42:27 +0800 as excerpted: > >> The easiest way to get a basic idea of how large your extent tree is >> using debug tree: >> >> # btrfs-debug-tree -r -t extent >> >> You would get something like: >> btrfs-progs v4.15

Re: Status of FST and mount times

Qu Wenruo posted on Thu, 15 Feb 2018 09:42:27 +0800 as excerpted: > The easiest way to get a basic idea of how large your extent tree is > using debug tree: > > # btrfs-debug-tree -r -t extent > > You would get something like: > btrfs-progs v4.15 extent tree key (EXTENT_TREE ROOT_ITEM 0)

Re: Status of FST and mount times

On 2018年02月15日 01:08, Nikolay Borisov wrote: > > > On 14.02.2018 18:00, Ellis H. Wilson III wrote: >> Hi again -- back with a few more questions: >> >> Frame-of-reference here: RAID0.  Around 70TB raw capacity.  No >> compression.  No quotas enabled.  Many (potentially tens to hundreds) of >>

Re: Status of FST and mount times

Ellis H. Wilson III posted on Wed, 14 Feb 2018 11:00:29 -0500 as excerpted: > Hi again -- back with a few more questions: > > Frame-of-reference here: RAID0. Around 70TB raw capacity. No > compression. No quotas enabled. Many (potentially tens to hundreds) of > subvolumes, each with tens of

Re: Status of FST and mount times

On 02/14/2018 12:08 PM, Nikolay Borisov wrote: V1 for large filesystems is jut awful. Facebook have been experiencing the pain hence they implemented v2. You can view the spacecache tree as the complement version of the extent tree. v1 cache is implemented as a hidden inode and even though

Re: Status of FST and mount times

On 14.02.2018 18:00, Ellis H. Wilson III wrote: > Hi again -- back with a few more questions: > > Frame-of-reference here: RAID0.  Around 70TB raw capacity.  No > compression.  No quotas enabled.  Many (potentially tens to hundreds) of > subvolumes, each with tens of snapshots.  No control over