Re: [RFC PATCH 1/6] btrfs-progs: splitcmd-gen.sh: create btrfs- binaries for selected subcommands

2018-08-29 Thread Misono Tomohiro
On 2018/08/30 2:24, Axel Burri wrote: > Create separate binaries for each subcommand ("btrfs foo bar"). > Least invasive approach, generate c-files for each command: > > # ./splitcmd-gen.sh > # make V=1 btrfs-subvolume-show > # make V=1 btrfs-send > # [...] > > Alternative

Re: [PATCH v2] btrfs: Always check nocow for quota enabled case to make sure we won't reserve unnecessary data space

2018-08-29 Thread Qu Wenruo
On 2018/8/30 上午9:57, Misono Tomohiro wrote: > On 2018/08/28 14:21, Qu Wenruo wrote: >> On 2018/8/24 下午4:09, Misono Tomohiro wrote: >> [snip] BTW, what's the possibility of such problem in your test environment? >>> >>> It's like one in several times. >>> It may depend on hardware

[PATCH] btrfs: extent-tree.c: Remove redundant variable from btrfs_cross_ref_exist()

2018-08-29 Thread Misono Tomohiro
Since commit d7df2c796d7e ("Btrfs attach delayed ref updates to delayed ref heads"), check_delaed_ref() won't return -ENOENT. In btrfs_cross_ref_exist(), two variable 'ret' and 'ret2' is originally used to handle -ENOENT error case. Since the code is not needed anymore, let's just remove 'ret2'.

Re: [PATCH v2] btrfs: Always check nocow for quota enabled case to make sure we won't reserve unnecessary data space

2018-08-29 Thread Misono Tomohiro
On 2018/08/28 14:21, Qu Wenruo wrote: > On 2018/8/24 下午4:09, Misono Tomohiro wrote: > [snip] >>> >>> BTW, what's the possibility of such problem in your test environment? >> >> It's like one in several times. >> It may depend on hardware performance? (the machine is not so fast), >> >> I also

Re: How to erase a RAID1 ?

2018-08-29 Thread Qu Wenruo
[Forgot to Cc the list] On 2018/8/29 下午10:04, Pierre Couderc wrote: > > On 08/29/2018 02:52 PM, Qu Wenruo wrote: >> >> On 2018/8/29 下午8:49, Pierre Couderc wrote: >>> I want to reinstall a RAID1 btrfs system (wchis is now under debian >>> stretch, and will be reinstalled in stretch). >> If you

Re: [PATCH 6/6] btrfs-progs: rescue-super: Don't double free fs_devices

2018-08-29 Thread Qu Wenruo
On 2018/8/29 下午11:38, Nikolay Borisov wrote: > > > On 3.08.2018 08:50, Qu Wenruo wrote: >> [BUG] >> During fuzz/007 we hit the following error: >> -- >> == RUN MAYFAIL /home/adam/btrfs/btrfs-progs/btrfs rescue super-recover >> -y -v >>

Re: [PATCH 3/6] btrfs-progs: Don't report dirty leaked eb using BUG_ON

2018-08-29 Thread Qu Wenruo
On 2018/8/29 下午10:52, Nikolay Borisov wrote: > > > On 3.08.2018 08:50, Qu Wenruo wrote: >> Another BUG_ON() during fuzz/003: >> -- >> == RUN MAYFAIL /home/adam/btrfs/btrfs-progs/btrfs check --init-csum-tree >>

Re: [RFC PATCH 0/6] btrfs-progs: build distinct binaries for specific btrfs subcommands

2018-08-29 Thread Austin S. Hemmelgarn
On 2018-08-29 13:24, Axel Burri wrote: This patch allows to build distinct binaries for specific btrfs subcommands, e.g. "btrfs-subvolume-show" which would be identical to "btrfs subvolume show". Motivation: While btrfs-progs offer the all-inclusive "btrfs" command, it gets pretty cumbersome

[RFC PATCH 5/6] btrfs-progs: Makefile: move progs_splitcmd variable to Makefile.install_setcap

2018-08-29 Thread Axel Burri
Signed-off-by: Axel Burri --- Makefile| 17 - Makefile.install_setcap | 2 ++ splitcmd-gen.sh | 17 ++--- 3 files changed, 20 insertions(+), 16 deletions(-) diff --git a/Makefile b/Makefile index acf5a677..3f16434c 100644 --- a/Makefile +++

[RFC PATCH 1/6] btrfs-progs: splitcmd-gen.sh: create btrfs- binaries for selected subcommands

2018-08-29 Thread Axel Burri
Create separate binaries for each subcommand ("btrfs foo bar"). Least invasive approach, generate c-files for each command: # ./splitcmd-gen.sh # make V=1 btrfs-subvolume-show # make V=1 btrfs-send # [...] Alternative approach: instead of including the c-file, link with obj in

[RFC PATCH 3/6] btrfs-progs: Makefile: add "install-splitcmd-setcap" target, installs splitcmd binaries with appropriate capabilities

2018-08-29 Thread Axel Burri
Install all $progs_install_splitcmd, and set appropriate linux file capabilities(7) using setcap(8). NOTE: while installing, group is hardcoded to "btrfs"! This needs further discussion. Signed-off-by: Axel Burri --- Makefile| 36 Makefile.inc.in |

[RFC PATCH 6/6] btrfs-progs: add splitcmd binaries to gitignore

2018-08-29 Thread Axel Burri
Signed-off-by: Axel Burri --- .gitignore | 9 + 1 file changed, 9 insertions(+) diff --git a/.gitignore b/.gitignore index 144ebb3b..299019e0 100644 --- a/.gitignore +++ b/.gitignore @@ -42,6 +42,15 @@ library-test-static /fssum testsuite-id +btrfs-filesystem-usage

[RFC PATCH 0/6] btrfs-progs: build distinct binaries for specific btrfs subcommands

2018-08-29 Thread Axel Burri
This patch allows to build distinct binaries for specific btrfs subcommands, e.g. "btrfs-subvolume-show" which would be identical to "btrfs subvolume show". Motivation: While btrfs-progs offer the all-inclusive "btrfs" command, it gets pretty cumbersome to restrict privileges to the subcommands

[RFC PATCH 2/6] btrfs-progs: add btrfs- source files generated by splitcmd-gen.sh

2018-08-29 Thread Axel Burri
Another approach would be to generate the splitted commands in the Makefile on-demand, which is probably not desired. Signed-off-by: Axel Burri --- btrfs-filesystem-usage.c | 23 +++ btrfs-qgroup-destroy.c | 23 +++ btrfs-receive.c| 23

[RFC PATCH 4/6] btrfs-progs: Makefile: include Makefile.install_setcap generated by splitcmd-gen.sh

2018-08-29 Thread Axel Burri
Signed-off-by: Axel Burri --- Makefile| 11 ++- Makefile.install_setcap | 10 ++ splitcmd-gen.sh | 5 + 3 files changed, 17 insertions(+), 9 deletions(-) create mode 100644 Makefile.install_setcap diff --git a/Makefile b/Makefile index

Re: [PATCH 6/6] btrfs-progs: rescue-super: Don't double free fs_devices

2018-08-29 Thread Nikolay Borisov
On 3.08.2018 08:50, Qu Wenruo wrote: > [BUG] > During fuzz/007 we hit the following error: > -- > == RUN MAYFAIL /home/adam/btrfs/btrfs-progs/btrfs rescue super-recover -y > -v > /home/adam/btrfs/btrfs-progs/tests//fuzz-tests/images/bko-200409.raw.restored.scratch > ERROR: tree_root

Re: [PATCH 5/6] btrfs-progs: Exit gracefull when we failed to alloc dev extent

2018-08-29 Thread Nikolay Borisov
On 3.08.2018 08:50, Qu Wenruo wrote: > Another BUG_ON() during fuzz/003: > -- > == RUN MAYFAIL /home/adam/btrfs/btrfs-progs/btrfs check --repair > /home/adam/btrfs/btrfs-progs/tests//fuzz-tests/images/bko-199833-reloc-recovery-crash.raw.restored > [1/7] checking root items > Fixed 0

Re: [PATCH 3/6] btrfs-progs: Don't report dirty leaked eb using BUG_ON

2018-08-29 Thread Nikolay Borisov
On 3.08.2018 08:50, Qu Wenruo wrote: > Another BUG_ON() during fuzz/003: > -- > == RUN MAYFAIL /home/adam/btrfs/btrfs-progs/btrfs check --init-csum-tree > /home/adam/btrfs/btrfs-progs/tests//fuzz-tests/images/bko-161821.raw.restored > [1/7] checking root items > Fixed 0 roots. > [2/7]

Re: btrfs fi du unreliable?

2018-08-29 Thread Graham Cobb
On 29/08/18 14:31, Jorge Bastos wrote: > Thanks, that makes sense, so it's only possible to see how much space > a snapshot is using with quotas enable, I remember reading that > somewhere before, though there was a new way after reading this latest > post . My extents lists scripts

Re: [PATCH v3 1/2] btrfs: Enhance btrfs_trim_fs function to handle error better

2018-08-29 Thread Nikolay Borisov
On 29.08.2018 16:53, Qu Wenruo wrote: > > > On 2018/8/29 下午9:43, Nikolay Borisov wrote: >> >> >> On 29.08.2018 08:15, Qu Wenruo wrote: >>> Function btrfs_trim_fs() doesn't handle errors in a consistent way, if >>> error happens when trimming existing block groups, it will skip the >>>

Re: [PATCH 2/6] btrfs-progs: Exit gracefully when failed to repair root dir item

2018-08-29 Thread Nikolay Borisov
On 3.08.2018 08:50, Qu Wenruo wrote: > Another BUG_ON() during fuzz/003: > -- > == RUN MAYFAIL /home/adam/btrfs/btrfs-progs/btrfs check --init-csum-tree > /home/adam/btrfs/btrfs-progs/tests//fuzz-tests/images/bko-161821.raw.restored > [1/7] checking root items > Fixed 0 roots. > [2/7]

Re: [PATCH 1/6] btrfs-progs: Exit gracefully if we hit ENOSPC when allocating tree block

2018-08-29 Thread Nikolay Borisov
On 3.08.2018 08:50, Qu Wenruo wrote: > When running test fuzz/003, we could hit the following BUG_ON: > -- > == RUN MAYFAIL /home/adam/btrfs/btrfs-progs/btrfs check --init-csum-tree > /home/adam/btrfs/btrfs-progs/tests//fuzz-tests/images/bko-155621-bad-block-group-offset.raw.restored

Re: [PATCH v3 2/2] btrfs: Ensure btrfs_trim_fs can trim the whole fs

2018-08-29 Thread Nikolay Borisov
On 29.08.2018 08:15, Qu Wenruo wrote: > [BUG] > fstrim on some btrfs only trims the unallocated space, not trimming any > space in existing block groups. > > [CAUSE] > Before fstrim_range passed to btrfs_trim_fs(), it get truncated to > range [0, super->total_bytes). > So later btrfs_trim_fs()

Re: How to erase a RAID1 ?

2018-08-29 Thread Pierre Couderc
On 08/29/2018 02:52 PM, Qu Wenruo wrote: On 2018/8/29 下午8:49, Pierre Couderc wrote: I want to reinstall a RAID1 btrfs system (wchis is now under debian stretch, and will be reinstalled in stretch). If you still want to use btrfs, just umount the original fs, and # mkfs.btrfs -f The

Re: [PATCH v3 1/2] btrfs: Enhance btrfs_trim_fs function to handle error better

2018-08-29 Thread Qu Wenruo
On 2018/8/29 下午9:43, Nikolay Borisov wrote: > > > On 29.08.2018 08:15, Qu Wenruo wrote: >> Function btrfs_trim_fs() doesn't handle errors in a consistent way, if >> error happens when trimming existing block groups, it will skip the >> remaining blocks and continue to trim unallocated space

Re: [PATCH v3 1/2] btrfs: Enhance btrfs_trim_fs function to handle error better

2018-08-29 Thread Nikolay Borisov
On 29.08.2018 08:15, Qu Wenruo wrote: > Function btrfs_trim_fs() doesn't handle errors in a consistent way, if > error happens when trimming existing block groups, it will skip the > remaining blocks and continue to trim unallocated space for each device. > > And the return value will only

Re: btrfs fi du unreliable?

2018-08-29 Thread Jorge Bastos
On Wed, Aug 29, 2018 at 1:42 PM Remi Gauvin wrote: > > > Exclusive means... exclusive... to that 1 snapshop/subvolume. If the > data also exists on the 23'rd snapshot, it's not exclusive. > > If you wanted to report how much data is exclusive to a group of > snapshots, (say, July 22 *and* 23rd).

Re: How to erase a RAID1 ?

2018-08-29 Thread Qu Wenruo
On 2018/8/29 下午8:49, Pierre Couderc wrote: > I want to reinstall a RAID1 btrfs system (wchis is now under debian > stretch, and will be reinstalled in stretch). If you still want to use btrfs, just umount the original fs, and # mkfs.btrfs -f Then a completely new btrfs. > > How to

How to erase a RAID1 ?

2018-08-29 Thread Pierre Couderc
I want to reinstall a RAID1 btrfs system (wchis is now under debian stretch, and will be reinstalled in stretch). How to correctly "erase" it ? Not truly hard erase it, but so that old data does not appear... It is not clear in the wiki. Thanks PC

Re: [PATCH 0/4] Userspace support for FSID change

2018-08-29 Thread Qu Wenruo
On 2018/8/29 下午8:33, Nikolay Borisov wrote: > > > On 29.08.2018 15:09, Qu Wenruo wrote: >> >> >> On 2018/8/29 下午4:35, Nikolay Borisov wrote: >>> Here is the userspace tooling support for utilising the new metadata_uuid >>> field, >>> enabling the change of fsid without having to rewrite

Re: [PATCH 0/4] Userspace support for FSID change

2018-08-29 Thread Austin S. Hemmelgarn
On 2018-08-29 08:33, Nikolay Borisov wrote: On 29.08.2018 15:09, Qu Wenruo wrote: On 2018/8/29 下午4:35, Nikolay Borisov wrote: Here is the userspace tooling support for utilising the new metadata_uuid field, enabling the change of fsid without having to rewrite every metadata block. This

Re: btrfs fi du unreliable?

2018-08-29 Thread Remi Gauvin
On 2018-08-29 08:00 AM, Jorge Bastos wrote: > > Look for example at snapshots from July 21st and 22nd, total used > space went from 199 to 277GiB, this is mostly from new added files, as > I confirmed from browsing those snapshots, there were no changes on > the 23rd, and a lot of files were

Re: [PATCH 0/4] Userspace support for FSID change

2018-08-29 Thread Nikolay Borisov
On 29.08.2018 15:09, Qu Wenruo wrote: > > > On 2018/8/29 下午4:35, Nikolay Borisov wrote: >> Here is the userspace tooling support for utilising the new metadata_uuid >> field, >> enabling the change of fsid without having to rewrite every metadata block. >> This >> patchset consists of

Re: [PATCH 0/4] Userspace support for FSID change

2018-08-29 Thread Qu Wenruo
On 2018/8/29 下午4:35, Nikolay Borisov wrote: > Here is the userspace tooling support for utilising the new metadata_uuid > field, > enabling the change of fsid without having to rewrite every metadata block. > This > patchset consists of adding support for the new field to various tools and

btrfs fi du unreliable?

2018-08-29 Thread Jorge Bastos
Hi, Read on another post that you can use sudo btrfs fi du -s ' to get a report on how much exclusive space is being used by the snapshots, I wasn't aware of this command and believed that there wasn't an easy way to get this, but after using it on some of my btrfs file systems used for backups

Re: DRDY errors are not consistent with scrub results

2018-08-29 Thread Hugo Mills
On Wed, Aug 29, 2018 at 09:58:58AM +, Duncan wrote: > Cerem Cem ASLAN posted on Wed, 29 Aug 2018 09:58:21 +0300 as excerpted: > > > Thinking again, this is totally acceptable. If the requirement was a > > good health disk, then I think I must check the disk health by myself. > > I may believe

Re: DRDY errors are not consistent with scrub results

2018-08-29 Thread Duncan
Cerem Cem ASLAN posted on Wed, 29 Aug 2018 09:58:21 +0300 as excerpted: > Thinking again, this is totally acceptable. If the requirement was a > good health disk, then I think I must check the disk health by myself. > I may believe that the disk is in a good state, or make a quick test or > make

Re: DRDY errors are not consistent with scrub results

2018-08-29 Thread ein
On 08/28/2018 12:51 AM, Cerem Cem ASLAN wrote: > Hi, > Good morning. > > I'm not sure how to proceed at the moment. Taking succesfull backups > made me think that everything might be okay but I'm not sure if I > should continue trusting the drive or not. What additional checks > should I

Re: Strange behavior (possible bugs) in btrfs

2018-08-29 Thread Filipe Manana
On Tue, Aug 28, 2018 at 9:35 PM Jayashree Mohan wrote: > > Hi Filipe, > > This is to follow up the status of crash consistency bugs we reported > on btrfs. We see that there has been a patch(not in the kernel yet) > (https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg77875.html) > that

[PATCH 1/2] btrfs: Introduce support for FSID change without metadata rewrite

2018-08-29 Thread Nikolay Borisov
This field is going to be used when the user wants to change the UUID of the filesystem without having to rewrite all metadata blocks. This field adds another level of indirection such that when the FSID is changed what really happens is the current uuid (the one with which the fs was created) is

[PATCH 2/2] btrfs: Remove fsid/metadata_fsid fields from btrfs_info

2018-08-29 Thread Nikolay Borisov
Currently btrfs_fs_info structure contains a copy of the fsid/metadata_uuid fields. Same values are also contained in the btrfs_fs_devices structure which fs_info has a reference to. Let's reduce duplication by removing the fields from fs_info and always refer to the ones in fs_devices. No

[PATCH 0/2] FSID change kernel support

2018-08-29 Thread Nikolay Borisov
Here is the kernel support for another level of indirection when dealing with FSID. I've taken inspiration from XFS so if people are familiar how FSID is changed there it's identical in BTRFS. In short, another field is introduced which will record the original fsid (which is stamped into every

[PATCH 1/4] btrfs-progs: Add support for metadata_uuid field.

2018-08-29 Thread Nikolay Borisov
Add support for a new metadata_uuid field. This is just a preparatory commit which switches all users of the fsid field for metdata comparison purposes to utilize the new field. This more or less mirrors the kernel patch, additionally: * Update 'btrfs inspect-internal dump-super' to account for

[PATCH 2/4] btrfstune: Add support for changing the user uuid

2018-08-29 Thread Nikolay Borisov
This allows us to change the use-visible UUID on filesytems from userspace if desired, by copying the existing UUID to the new location for metadata comparisons. If this is done, an incompat flag must be set to prevent older filesystems from mounting the filesystem, but the original UUID can be

[PATCH 4/4] btrfs-progs: Remove fsid/metdata_uuid fields from fs_info

2018-08-29 Thread Nikolay Borisov
Signed-off-by: Nikolay Borisov --- btrfstune.c | 5 +++-- check/main.c| 2 +- chunk-recover.c | 11 +-- ctree.c | 11 ++- ctree.h | 2 -- disk-io.c | 18 +- volumes.c | 5 +++-- 7 files changed, 27 insertions(+), 27

[PATCH 3/4] btrfs-progs: tests: Add tests for changing fsid feature

2018-08-29 Thread Nikolay Borisov
Add a bunch of tests exercising the new btrfstune functionality. In particular check that various restrictions are implemented correctly, test that btrfs-image works as expected and also test the output of btrfs inspect-internal dump-super is correct. Signed-off-by: Nikolay Borisov ---

[PATCH 0/4] Userspace support for FSID change

2018-08-29 Thread Nikolay Borisov
Here is the userspace tooling support for utilising the new metadata_uuid field, enabling the change of fsid without having to rewrite every metadata block. This patchset consists of adding support for the new field to various tools and files (Patch 1). The actual implementation of the new

Re: DRDY errors are not consistent with scrub results

2018-08-29 Thread Cerem Cem ASLAN
Chris Murphy , 29 Ağu 2018 Çar, 02:58 tarihinde şunu yazdı: > > On Tue, Aug 28, 2018 at 5:04 PM, Cerem Cem ASLAN > wrote: > > What I want to achive is that I want to add the problematic disk as > > raid1 and see how/when it fails and how BTRFS recovers these fails. > > While the party goes on,

Re: [LKP] [lkp-robot] [mm] 9092c71bb7: blogbench.write_score -12.3% regression

2018-08-29 Thread Huang, Ying
"Huang, Ying" writes: > Josef Bacik writes: > >> On Thu, Aug 02, 2018 at 01:55:23PM +0800, Huang, Ying wrote: >>> "Huang, Ying" writes: >>> >>> > Hi, Chris, >>> > >>> > Chris Mason writes: >>> > >>> >> On 19 Jun 2018, at 23:51, Huang, Ying wrote: >>> > "Huang, Ying" writes: >>> >