Re: Scrub aborts due to corrupt leaf

2018-08-28 Thread Qu Wenruo
On 2018/8/28 下午9:56, Chris Murphy wrote: > On Tue, Aug 28, 2018 at 7:42 AM, Qu Wenruo wrote: >> >> >> On 2018/8/28 下午9:29, Larkin Lowrey wrote: >>> On 8/27/2018 10:12 PM, Larkin Lowrey wrote: On 8/27/2018 12:46 AM, Qu Wenruo wrote: > >> The system uses ECC memory and edac-util has

Re: [PATCH 0/6] btrfs-progs: Variant fixes for fuzz-tests

2018-08-28 Thread Qu Wenruo
Gentle ping. These fixes are pretty small, I'd like to see them merged before I need to rebase them again and again. Thanks, Qu On 2018/8/3 下午1:50, Qu Wenruo wrote: > This can be fetched from github: > https://github.com/adam900710/btrfs-progs/tree/fixes_for_fuzz_test > > The base HEAD is: >

[PATCH v3 1/2] btrfs: Enhance btrfs_trim_fs function to handle error better

2018-08-28 Thread Qu Wenruo
Function btrfs_trim_fs() doesn't handle errors in a consistent way, if error happens when trimming existing block groups, it will skip the remaining blocks and continue to trim unallocated space for each device. And the return value will only reflect the final error from device trimming. This

[PATCH v3 2/2] btrfs: Ensure btrfs_trim_fs can trim the whole fs

2018-08-28 Thread Qu Wenruo
[BUG] fstrim on some btrfs only trims the unallocated space, not trimming any space in existing block groups. [CAUSE] Before fstrim_range passed to btrfs_trim_fs(), it get truncated to range [0, super->total_bytes). So later btrfs_trim_fs() will only be able to trim block groups in range [0,

[PATCH v3 0/2] btrfs: trim enhancement to allow btrfs really trim block groups

2018-08-28 Thread Qu Wenruo
This patchset can be fetched from github: https://github.com/adam900710/linux/tree/trim_fix Which is based on v4.19-rc1 tag. This patchset introduces 2 enhancement, one to output better error messages during trim, the other one is to ensure we could really trim block groups if logical bytenr of

Re: Scrub aborts due to corrupt leaf

2018-08-28 Thread Qu Wenruo
On 2018/8/28 下午9:56, Chris Murphy wrote: > On Tue, Aug 28, 2018 at 7:42 AM, Qu Wenruo wrote: >> >> >> On 2018/8/28 下午9:29, Larkin Lowrey wrote: >>> On 8/27/2018 10:12 PM, Larkin Lowrey wrote: On 8/27/2018 12:46 AM, Qu Wenruo wrote: > >> The system uses ECC memory and edac-util has

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

2018-08-28 Thread Chris Murphy
On Tue, Aug 28, 2018 at 1:14 PM, Menion wrote: > You are correct, indeed in order to cleanup you need > > 1) someone realize that snapshots have been created > 2) apt-brtfs-snapshot is manually installed on the system > > Assuming also that the snapshots created during do-release-upgrade are >

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

2018-08-28 Thread Chris Murphy
On Tue, Aug 28, 2018 at 8:56 AM, Menion wrote: > [sudo] password for menion: > ID gen top level path > -- --- - > 257 600627 5 /@ > 258 600626 5 /@home > 296 599489 5 >

Re: DRDY errors are not consistent with scrub results

2018-08-28 Thread Chris Murphy
On Tue, Aug 28, 2018 at 5:04 PM, Cerem Cem ASLAN wrote: > What I want to achive is that I want to add the problematic disk as > raid1 and see how/when it fails and how BTRFS recovers these fails. > While the party goes on, the main system shouldn't be interrupted > since this is a production

Re: DRDY errors are not consistent with scrub results

2018-08-28 Thread Cerem Cem ASLAN
What I want to achive is that I want to add the problematic disk as raid1 and see how/when it fails and how BTRFS recovers these fails. While the party goes on, the main system shouldn't be interrupted since this is a production system. For example, I would never expect to be ended up with such a

WARNING in clone_finish_inode_update while de-duplicating with bedup

2018-08-28 Thread Sam Tygier
Hello, I get the following WARNING while de-duplicating with bedup 0.10.1. I am running Debian with backports kernel: Linux lithium 4.17.0-0.bpo.1-amd64 #1 SMP Debian 4.17.8-1~bpo9+1 (2018-07-23) x86_64 GNU/Linux I ran: sudo bedup scan /media/btrfs/ sudo bedup dedupe /media/btrfs/

Re: DRDY errors are not consistent with scrub results

2018-08-28 Thread Chris Murphy
On Tue, Aug 28, 2018 at 12:50 PM, Cerem Cem ASLAN wrote: > I've successfully moved everything to another disk. (The only hard > part was configuring the kernel parameters, as my root partition was > on LVM which is on LUKS partition. Here are the notes, if anyone > needs: >

Re: Strange behavior (possible bugs) in btrfs

2018-08-28 Thread Jayashree Mohan
Hi Filipe, This is to follow up the status of crash consistency bugs we reported on btrfs. We see that there has been a patch(not in the kernel yet) (https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg77875.html) that resolves one of the reported bugs. However, the other bugs we reported

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

2018-08-28 Thread Austin S. Hemmelgarn
On 2018-08-28 15:14, Menion wrote: You are correct, indeed in order to cleanup you need 1) someone realize that snapshots have been created 2) apt-brtfs-snapshot is manually installed on the system Your second requirement is only needed if you want the nice automated cleanup. There's

Re: DRDY errors are not consistent with scrub results

2018-08-28 Thread Cerem Cem ASLAN
I've successfully moved everything to another disk. (The only hard part was configuring the kernel parameters, as my root partition was on LVM which is on LUKS partition. Here are the notes, if anyone needs: https://github.com/ceremcem/smith-sync/blob/master/create-bootable-backup.md) Now I'm

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

2018-08-28 Thread Noah Massey
On Tue, Aug 28, 2018 at 1:25 PM Menion wrote: > > Ok, I have removed the snapshot and the free expected space is here, thank > you! > As a side note: apt-btrfs-snapshot was not installed, but it is > present in Ubuntu repository and I have used it (and I like the idea > of automatic snapshot

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

2018-08-28 Thread Menion
Ok, I have removed the snapshot and the free expected space is here, thank you! As a side note: apt-btrfs-snapshot was not installed, but it is present in Ubuntu repository and I have used it (and I like the idea of automatic snapshot during upgrade) This means that the do-release-upgrade does

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

2018-08-28 Thread Austin S. Hemmelgarn
On 2018-08-28 12:05, Noah Massey wrote: On Tue, Aug 28, 2018 at 11:47 AM Austin S. Hemmelgarn wrote: On 2018-08-28 11:27, Noah Massey wrote: On Tue, Aug 28, 2018 at 10:59 AM Menion wrote: [sudo] password for menion: ID gen top level path -- --- -

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

2018-08-28 Thread Noah Massey
On Tue, Aug 28, 2018 at 11:47 AM Austin S. Hemmelgarn wrote: > > On 2018-08-28 11:27, Noah Massey wrote: > > On Tue, Aug 28, 2018 at 10:59 AM Menion wrote: > >> > >> [sudo] password for menion: > >> ID gen top level path > >> -- --- - > >> 257

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

2018-08-28 Thread Austin S. Hemmelgarn
On 2018-08-28 11:27, Noah Massey wrote: On Tue, Aug 28, 2018 at 10:59 AM Menion wrote: [sudo] password for menion: ID gen top level path -- --- - 257 600627 5 /@ 258 600626 5 /@home 296 599489 5

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

2018-08-28 Thread Noah Massey
On Tue, Aug 28, 2018 at 10:59 AM Menion wrote: > > [sudo] password for menion: > ID gen top level path > -- --- - > 257 600627 5 /@ > 258 600626 5 /@home > 296 599489 5 >

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

2018-08-28 Thread Menion
[sudo] password for menion: ID gen top level path -- --- - 257 600627 5 /@ 258 600626 5 /@home 296 599489 5 /@apt-snapshot-release-upgrade-bionic-2018-08-27_15:29:55 297 599489 5

Re: Scrub aborts due to corrupt leaf

2018-08-28 Thread Chris Murphy
On Tue, Aug 28, 2018 at 7:42 AM, Qu Wenruo wrote: > > > On 2018/8/28 下午9:29, Larkin Lowrey wrote: >> On 8/27/2018 10:12 PM, Larkin Lowrey wrote: >>> On 8/27/2018 12:46 AM, Qu Wenruo wrote: > The system uses ECC memory and edac-util has not reported any errors. > However, I will run a

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

2018-08-28 Thread Chris Murphy
On Tue, Aug 28, 2018 at 3:34 AM, Menion wrote: > Hi all > I have run a distro upgrade on my Ubuntu 16.04 that runs ppa kernel > 4.17.2 with btrfsprogs 4.17.0 > The root filesystem is BTRFS single created by the Ubuntu Xenial > installer (so on kernel 4.4.0) on an internal mmc, located in >

Re: Scrub aborts due to corrupt leaf

2018-08-28 Thread Qu Wenruo
On 2018/8/28 下午9:29, Larkin Lowrey wrote: > On 8/27/2018 10:12 PM, Larkin Lowrey wrote: >> On 8/27/2018 12:46 AM, Qu Wenruo wrote: >>> The system uses ECC memory and edac-util has not reported any errors. However, I will run a memtest anyway. >>> So it should not be the memory problem.

Re: Scrub aborts due to corrupt leaf

2018-08-28 Thread Larkin Lowrey
On 8/27/2018 10:12 PM, Larkin Lowrey wrote: On 8/27/2018 12:46 AM, Qu Wenruo wrote: The system uses ECC memory and edac-util has not reported any errors. However, I will run a memtest anyway. So it should not be the memory problem. BTW, what's the current generation of the fs? # btrfs

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

2018-08-28 Thread Qu Wenruo
On 2018/8/28 下午9:07, Menion wrote: > Ok, thanks for your replay > This is a root FS, how can I defragment it? If it's a rootfs, then it's a little strange. Normally package manager should overwrite the whole file during package update transaction, thus the extent booking doesn't happen as

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

2018-08-28 Thread Menion
Ok, thanks for your replay This is a root FS, how can I defragment it? If I try to launch it I get this output: menion@Menionubuntu:~$ sudo btrfs filesystem defragment -r / ERROR: defrag failed on /bin/bash: Text file busy ERROR: defrag failed on /bin/dash: Text file busy ERROR: defrag failed on

Re: 14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

2018-08-28 Thread Qu Wenruo
On 2018/8/28 下午5:34, Menion wrote: > Hi all > I have run a distro upgrade on my Ubuntu 16.04 that runs ppa kernel > 4.17.2 with btrfsprogs 4.17.0 > The root filesystem is BTRFS single created by the Ubuntu Xenial > installer (so on kernel 4.4.0) on an internal mmc, located in > /dev/mmcblk0p3 >

Re: [PATCH] btrfs: extent-tree: Detect bytes_may_use underflow earlier

2018-08-28 Thread Qu Wenruo
On 2018/8/28 下午7:48, Nikolay Borisov wrote: > > > On 28.08.2018 14:46, Qu Wenruo wrote: >> >> >> On 2018/8/28 下午4:48, Nikolay Borisov wrote: >>> >>> >>> On 28.08.2018 09:46, Qu Wenruo wrote: Although we have space_info::bytes_may_use underflow detection in

Re: [PATCH] btrfs: extent-tree: Detect bytes_may_use underflow earlier

2018-08-28 Thread Nikolay Borisov
On 28.08.2018 14:46, Qu Wenruo wrote: > > > On 2018/8/28 下午4:48, Nikolay Borisov wrote: >> >> >> On 28.08.2018 09:46, Qu Wenruo wrote: >>> Although we have space_info::bytes_may_use underflow detection in >>> btrfs_free_reserved_data_space_noquota(), we have more callers who are >>>

Re: [PATCH] btrfs: extent-tree: Detect bytes_may_use underflow earlier

2018-08-28 Thread Qu Wenruo
On 2018/8/28 下午4:48, Nikolay Borisov wrote: > > > On 28.08.2018 09:46, Qu Wenruo wrote: >> Although we have space_info::bytes_may_use underflow detection in >> btrfs_free_reserved_data_space_noquota(), we have more callers who are >> subtracting number from space_info::bytes_may_use. >> >> So

Re: corruption_errs

2018-08-28 Thread Austin S. Hemmelgarn
On 2018-08-27 18:53, John Petrini wrote: Hi List, I'm seeing corruption errors when running btrfs device stats but I'm not sure what that means exactly. I've just completed a full scrub and it reported no errors. I'm hoping someone here can enlighten me. Thanks! The first thing to understand

Re: BTRFS support per-subvolume compression, isn't it?

2018-08-28 Thread Austin S. Hemmelgarn
On 2018-08-27 17:05, Eugene Bright wrote: Greetings! BTRFS wiki says there is no per-subvolume compression option [1]. At the same time next command allow me to set properties per-subvolume: btrfs property set /volume compression zstd Corresponding get command shows distinct

14Gb of space lost after distro upgrade on BTFS root partition (long thread with logs)

2018-08-28 Thread Menion
Hi all I have run a distro upgrade on my Ubuntu 16.04 that runs ppa kernel 4.17.2 with btrfsprogs 4.17.0 The root filesystem is BTRFS single created by the Ubuntu Xenial installer (so on kernel 4.4.0) on an internal mmc, located in /dev/mmcblk0p3 After the upgrade I have cleaned apt cache and

Re: [PATCH] btrfs: extent-tree: Detect bytes_may_use underflow earlier

2018-08-28 Thread Nikolay Borisov
On 28.08.2018 09:46, Qu Wenruo wrote: > Although we have space_info::bytes_may_use underflow detection in > btrfs_free_reserved_data_space_noquota(), we have more callers who are > subtracting number from space_info::bytes_may_use. > > So instead of doing underflow detection for every caller,

Re: [PATCH RFC] btrfs: clone: Flush data before doing clone

2018-08-28 Thread Qu Wenruo
On 2018/8/28 下午1:54, Qu Wenruo wrote: > Due to the limitation of btrfs_cross_ref_exist(), run_delalloc_nocow() > can still fall back to CoW even only (unrelated) part of the > preallocated extent is shared. > > This makes the follow case to do unnecessary CoW: > > # xfs_io -f -c "falloc 0 2M"

[PATCH] btrfs: extent-tree: Detect bytes_may_use underflow earlier

2018-08-28 Thread Qu Wenruo
Although we have space_info::bytes_may_use underflow detection in btrfs_free_reserved_data_space_noquota(), we have more callers who are subtracting number from space_info::bytes_may_use. So instead of doing underflow detection for every caller, introduce a new wrapper update_bytes_may_use() to