Re: [PATCH 08/35] btrfs: release metadata before running delayed refs

2018-09-03 Thread Nikolay Borisov
On 30.08.2018 20:41, Josef Bacik wrote: > We want to release the unused reservation we have since it refills the > delayed refs reserve, which will make everything go smoother when > running the delayed refs if we're short on our reservation. > > Signed-off-by: Josef Bacik Reviewed-by:

Re: [PATCH 06/35] btrfs: check if free bgs for commit

2018-09-03 Thread Nikolay Borisov
On 30.08.2018 20:41, Josef Bacik wrote: > may_commit_transaction will skip committing the transaction if we don't > have enough pinned space or if we're trying to find space for a SYSTEM > chunk. However if we have pending free block groups in this transaction > we still want to commit as we

[PATCH] btrfs-progs: tests: Add test for missing device delete error value

2018-09-03 Thread Nikolay Borisov
Add a test which ensures the kernel returns the correct error value when missing device removal is requested. This test verifies that kernel refactoring didn't broken the return value. Signed-off-by: Nikolay Borisov --- tests/misc-tests/011-delete-missing-device/test.sh | 17 +

[PATCH 0/3] cleanup couple of device-related functions' retval

2018-09-03 Thread Nikolay Borisov
Currently btrfs_find_device_by_path, btrfs_find_device_missing_or_by_path and btrfs_find_device_by_devspec are called in a chain and they all return an integer value to signal error and at the same time use one of their parameters as an output. This patch set refactors those functions starting

[PATCH 3/3] btrfs: Make btrfs_find_device_by_devspec return btrfs_device directly

2018-09-03 Thread Nikolay Borisov
Instead of returning an error value and using one of the parameters for returning the actual object we are interested in just refactor the function to directly return btrfs_device *. Also bubble up the error handling for the special BTRFS_ERROR_DEV_MISSING_NOT_FOUND value into btrfs_rm_device. No

[PATCH 2/3] btrfs: Make btrfs_find_device_missing_or_by_path return directly a device

2018-09-03 Thread Nikolay Borisov
This function returns a numeric error value and additionally the device found in one of its input parameters. Simplify this by making the function directly return a pointer to btrfs_device. Additionally adjust the caller to handle the case when we want to remove the 'missing' device and ENOENT is

[PATCH 1/3] btrfs: Make btrfs_find_device_by_path return struct btrfs_device

2018-09-03 Thread Nikolay Borisov
Currently this function returns an error code as well as uses one of its arguments as a return value for struct btrfs_device. Change the function so that it returns btrfs_device directly and use the usual "encode error in pointer" mechanics if something goes wrong. No functional changes.

[PATCH 1/2] btrfs-progs: fix kernel version parsing on some versions past 3.0

2018-09-03 Thread Adam Borowski
The code fails if the third section is missing (like "4.18") or is followed by anything but "." or "-". This happens for example if we're not exactly at a tag and CONFIG_LOCALVERSION_AUTO=n (which results in "4.18.5+"). Signed-off-by: Adam Borowski --- fsfeatures.c | 5 + 1 file changed, 1

[PATCH 2/2] btrfs-progs: defrag: open files RO on new enough kernels or if root

2018-09-03 Thread Adam Borowski
Fixes EXTXBSY races. Signed-off-by: Adam Borowski --- cmds-filesystem.c | 10 -- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/cmds-filesystem.c b/cmds-filesystem.c index 06c8311b..4c9df69f 100644 --- a/cmds-filesystem.c +++ b/cmds-filesystem.c @@ -26,6 +26,7 @@

Re: IO errors when building RAID1.... ?

2018-09-03 Thread Pierre Couderc
On 09/03/2018 05:15 AM, Chris Murphy wrote: On Sat, Sep 1, 2018 at 1:03 AM, Pierre Couderc wrote: On 08/31/2018 08:52 PM, Chris Murphy wrote: Bad sector which is failing write. This is fatal, there isn't anything the block layer or Btrfs (or ext4 or XFS) can do about it. Well, ext234 do

Re: IO errors when building RAID1.... ?

2018-09-03 Thread Adam Borowski
On Sun, Sep 02, 2018 at 09:15:25PM -0600, Chris Murphy wrote: > For > 10 years drive firmware handles bad sector remapping internally. > It remaps the sector logical address to a reserve physical sector. > > NTFS and ext[234] have a means of accepting a list of bad sectors, and > will avoid using

RAID1 & BTRFS critical (device sda2): corrupt leaf, bad key order

2018-09-03 Thread Etienne Champetier
Hello btfrs hackers, I have a computer acting as backup server with BTRFS RAID1, and I would like to know the different options to rebuild this RAID (I saw this thread https://www.spinics.net/lists/linux-btrfs/msg68679.html but there was no raid 1) # uname -a Linux servmaison 4.4.0-134-generic

Re: [PATCH 12/35] btrfs: add ALLOC_CHUNK_FORCE to the flushing code

2018-09-03 Thread Nikolay Borisov
On 30.08.2018 20:42, Josef Bacik wrote: > + if (flush_state == ALLOC_CHUNK_FORCE && !commit_cycles) > + flush_state++; This is a bit obscure. So if we allocated a chunk and !commit_cycles just break from the loop? What's the reasoning behind this ?

Re: [PATCH v2] btrfs-progs: defrag: open files RO on new enough kernels

2018-09-03 Thread Nikolay Borisov
On 3.09.2018 14:31, Adam Borowski wrote: > Defragging an executable conflicts both way with it being run, resulting in > ETXTBSY. This either makes defrag fail or prevents the program from being > executed. > > Kernels 4.19-rc1 and later allow defragging files you could have possibly >

RAID1 & BTRFS critical (device sda2): corrupt leaf, bad key order

2018-09-03 Thread Etienne Champetier
Hello linux-btfrs, I have a computer acting as backup server with BTRFS RAID1, and I would like to know the different options to rebuild this RAID (I saw this thread https://www.spinics.net/lists/linux-btrfs/msg68679.html but there was no raid 1) # uname -a Linux servmaison 4.4.0-134-generic

[PATCH v2] btrfs-progs: defrag: open files RO on new enough kernels

2018-09-03 Thread Adam Borowski
Defragging an executable conflicts both way with it being run, resulting in ETXTBSY. This either makes defrag fail or prevents the program from being executed. Kernels 4.19-rc1 and later allow defragging files you could have possibly opened rw, even if the passed descriptor is ro.

Re: [PATCH 2/2] btrfs-progs: defrag: open files RO on new enough kernels or if root

2018-09-03 Thread Nikolay Borisov
On 3.09.2018 13:14, Adam Borowski wrote: > Fixes EXTXBSY races. > > Signed-off-by: Adam Borowski > --- > cmds-filesystem.c | 10 -- > 1 file changed, 8 insertions(+), 2 deletions(-) > > diff --git a/cmds-filesystem.c b/cmds-filesystem.c > index 06c8311b..4c9df69f 100644 > ---

Re: [PATCH 2/2] btrfs-progs: defrag: open files RO on new enough kernels or if root

2018-09-03 Thread Adam Borowski
On Mon, Sep 03, 2018 at 02:01:21PM +0300, Nikolay Borisov wrote: > On 3.09.2018 13:14, Adam Borowski wrote: > > Fixes EXTXBSY races. > > You have to be more eloquent than that and explain at least one race > condition. If you try to defrag an executable that's currently running: ERROR: cannot

Re: [PATCH 2/2] btrfs-progs: defrag: open files RO on new enough kernels or if root

2018-09-03 Thread Adam Borowski
On Mon, Sep 03, 2018 at 02:04:23PM +0300, Nikolay Borisov wrote: > On 3.09.2018 13:14, Adam Borowski wrote: > > - fd = open(fpath, O_RDWR); > > + fd = open(fpath, defrag_ro); > > Looking at the kernel code I think this is in fact incorrect, because in > ioctl.c we have: > >

Re: [PATCH 3/3] btrfs: Make btrfs_find_device_by_devspec return btrfs_device directly

2018-09-03 Thread Qu Wenruo
On 2018/9/3 下午5:46, Nikolay Borisov wrote: > Instead of returning an error value and using one of the parameters for > returning the actual object we are interested in just refactor the > function to directly return btrfs_device *. Also bubble up the error > handling for the special

[PATCH v3] btrfs-progs: defrag: open files RO on new enough kernels

2018-09-03 Thread Adam Borowski
Defragging an executable conflicts both way with it being run, resulting in ETXTBSY. This either makes defrag fail or prevents the program from being executed. Kernels 4.19-rc1 and later allow defragging files you could have possibly opened rw, even if the passed descriptor is ro (commit

Re: [PATCH 1/3] btrfs: Make btrfs_find_device_by_path return struct btrfs_device

2018-09-03 Thread Qu Wenruo
On 2018/9/3 下午5:46, Nikolay Borisov wrote: > Currently this function returns an error code as well as uses one of > its arguments as a return value for struct btrfs_device. Change the > function so that it returns btrfs_device directly and use the usual > "encode error in pointer" mechanics if

Re: [PATCH 2/3] btrfs: Make btrfs_find_device_missing_or_by_path return directly a device

2018-09-03 Thread Qu Wenruo
On 2018/9/3 下午5:46, Nikolay Borisov wrote: > This function returns a numeric error value and additionally the > device found in one of its input parameters. Simplify this by making > the function directly return a pointer to btrfs_device. Additionally > adjust the caller to handle the case when

Re: [PATCH 2/2] btrfs-progs: defrag: open files RO on new enough kernels or if root

2018-09-03 Thread Nikolay Borisov
On 3.09.2018 13:14, Adam Borowski wrote: > Fixes EXTXBSY races. You have to be more eloquent than that and explain at least one race condition. > > Signed-off-by: Adam Borowski > --- > cmds-filesystem.c | 10 -- > 1 file changed, 8 insertions(+), 2 deletions(-) > > diff --git

Re: [PATCH 06/35] btrfs: check if free bgs for commit

2018-09-03 Thread Nikolay Borisov
On 3.09.2018 12:06, Nikolay Borisov wrote: > > > On 30.08.2018 20:41, Josef Bacik wrote: >> may_commit_transaction will skip committing the transaction if we don't >> have enough pinned space or if we're trying to find space for a SYSTEM >> chunk. However if we have pending free block

Re: RAID1 & BTRFS critical (device sda2): corrupt leaf, bad key order

2018-09-03 Thread Qu Wenruo
On 2018/9/3 下午10:18, Etienne Champetier wrote: > Hello btfrs hackers, > > I have a computer acting as backup server with BTRFS RAID1, and I > would like to know the different options to rebuild this RAID > (I saw this thread > https://www.spinics.net/lists/linux-btrfs/msg68679.html but there

Does ssd auto detected work for microSD cards?

2018-09-03 Thread GWB
Curious instance here, but perhaps this is the expected behaviour: mount | grep btrfs /dev/sdb3 on / type btrfs (rw,ssd,subvol=@) /dev/sdb3 on /home type btrfs (rw,ssd,subvol=@home) /dev/sde1 on /media/gwb09/btrfs-32G-MicroSDc type btrfs (rw,nosuid,nodev,uhelper=udisks2) This is on an Ubuntu 14

Re: fsck lowmem mode only: ERROR: errors found in fs roots

2018-09-03 Thread Christoph Anton Mitterer
Hey. On Fri, 2018-08-31 at 10:33 +0800, Su Yue wrote: > Can you please fetch btrfs-progs from my repo and run lowmem check > in readonly? > Repo: https://github.com/Damenly/btrfs-progs/tree/lowmem_debug > It's based on v4.17.1 plus additonal output for debug only. I've adapted your patch to

Re: Does ssd auto detected work for microSD cards?

2018-09-03 Thread Chris Murphy
On Mon, Sep 3, 2018 at 7:53 PM, GWB wrote: > Curious instance here, but perhaps this is the expected behaviour: > > mount | grep btrfs > /dev/sdb3 on / type btrfs (rw,ssd,subvol=@) > /dev/sdb3 on /home type btrfs (rw,ssd,subvol=@home) > /dev/sde1 on /media/gwb09/btrfs-32G-MicroSDc type btrfs >

[PATCH 0/4] btrfs: quota: Skip unmodified tree blocks for balance

2018-09-03 Thread Qu Wenruo
This patchset can be fetched from github: https://github.com/adam900710/linux/tree/qgroup_balance_skip_trees The base commit is v4.19-rc1 tag. There are a lot of reports of system hang for balance on quota enabled fs. It's most obvious for large fs. The hang is caused by tons of unmodified

[PATCH 1/4] btrfs: qgroup: Introduce trace event to analyse the number of dirty extents accounted

2018-09-03 Thread Qu Wenruo
Number of qgroup dirty extents is directly linked to the performance overhead, so add a new trace event, trace_qgroup_num_dirty_extents(), to record how many dirty extents is processed in btrfs_qgroup_account_extents(). This will be pretty handy to analyse later balance performance improvement.

[PATCH 3/4] btrfs: qgroup: Introduce function to find all new tree blocks of tree reloc tree

2018-09-03 Thread Qu Wenruo
Introduce new function, qgroup_trace_new_subtree_blocks(), to iterate all new tree blocks in a tree reloc tree. So that qgroup could skip unrelated tree blocks during balance, which should hugely speedup balance speed when quota is enabled. The function qgroup_trace_new_subtree_blocks() itself

[PATCH 2/4] btrfs: qgroup: Introduce function to trace two swaped extents

2018-09-03 Thread Qu Wenruo
Introduce a new function, qgroup_trace_extent_swap(), which will be used later for balance qgroup speedup. The basis idea of balance is swapping tree blocks between tree reloc tree and the real file tree. The swap will happen in highest tree block, but there may be a lot of tree blocks involved.

[PATCH 4/4] btrfs: qgroup: Use generation aware subtree swap to mark dirty extents

2018-09-03 Thread Qu Wenruo
Before this patch, for quota enabled balance, btrfs needs to mark the whole subtree dirty for quota. E.g. OO = Old tree blocks (from file tree) NN = New tree blocks (from tree reloc tree) File tree (src) Tree reloc tree (dst) OO (a)

Re: RAID1 & BTRFS critical (device sda2): corrupt leaf, bad key order

2018-09-03 Thread Chris Murphy
On Mon, Sep 3, 2018 at 7:52 AM, Etienne Champetier wrote: > Hello linux-btfrs, > > I have a computer acting as backup server with BTRFS RAID1, and I > would like to know the different options to rebuild this RAID > (I saw this thread > https://www.spinics.net/lists/linux-btrfs/msg68679.html but

Re: IO errors when building RAID1.... ?

2018-09-03 Thread Chris Murphy
On Mon, Sep 3, 2018 at 4:23 AM, Adam Borowski wrote: > On Sun, Sep 02, 2018 at 09:15:25PM -0600, Chris Murphy wrote: >> For > 10 years drive firmware handles bad sector remapping internally. >> It remaps the sector logical address to a reserve physical sector. >> >> NTFS and ext[234] have a means