Re: Ideas to reuse filesystem's checksum to enhance dm-raid1/10/5/6?

On 16.11.2017 09:38, Qu Wenruo wrote: > > > On 2017年11月16日 14:54, Nikolay Borisov wrote: >> >> >> On 16.11.2017 04:18, Qu Wenruo wrote: >>> Hi all, >>> >>> [Background] >>> Recently I'm considering the possibility to use checksum from filesystem >>> to enhance device-mapper raid. >>> >>> The

Re: Ideas to reuse filesystem's checksum to enhance dm-raid1/10/5/6?

On 2017年11月16日 14:54, Nikolay Borisov wrote: > > > On 16.11.2017 04:18, Qu Wenruo wrote: >> Hi all, >> >> [Background] >> Recently I'm considering the possibility to use checksum from filesystem >> to enhance device-mapper raid. >> >> The idea behind it is quite simple, since most modern

Re: Ideas to reuse filesystem's checksum to enhance dm-raid1/10/5/6?

On 16.11.2017 04:18, Qu Wenruo wrote: > Hi all, > > [Background] > Recently I'm considering the possibility to use checksum from filesystem > to enhance device-mapper raid. > > The idea behind it is quite simple, since most modern filesystems have > checksum for their metadata, and even some

Re: [PATCH 0/4] factor __btrfs_open_devices()

On 11/16/2017 01:03 AM, David Sterba wrote: On Thu, Nov 09, 2017 at 11:45:22PM +0800, Anand Jain wrote: This is in preparation to help bring the missing device back to the alloc list dynamically. As of now if we run 'btrfs dev scan ' on a mounted FS, nothing happens, there is no code which

Ideas to reuse filesystem's checksum to enhance dm-raid1/10/5/6?

Hi all, [Background] Recently I'm considering the possibility to use checksum from filesystem to enhance device-mapper raid. The idea behind it is quite simple, since most modern filesystems have checksum for their metadata, and even some (btrfs) have checksum for data. And for btrfs RAID1/10

Re: [PATCH v2 2/3] btrfs: tree-checker: Add checker for dir item

On Wed, Nov 08, 2017 at 09:59:39AM +0200, Nikolay Borisov wrote: > > > On 8.11.2017 02:54, Qu Wenruo wrote: > > Add checker for dir item, for key types DIR_ITEM, DIR_INDEX and > > XATTR_ITEM. > > > > This checker does comprehensive check for: > > 1) dir_item header and its data size > >

Re: [PATCH] btrfs: fix deadlock when writing out space cache

On 11/15/2017 06:46 PM, Liu Bo wrote: On Wed, Nov 15, 2017 at 04:20:52PM -0500, Josef Bacik wrote: From: Josef Bacik If we fail to prepare our pages for whatever reason (out of memory in our case) we need to make sure to drop the block_group->data_rwsem, otherwise hilarity

Re: [PATCH] btrfs: move some zstd work data from stack to workspace

On 11/15/17, 9:30 AM, "David Sterba" wrote: > * ZSTD_inBuffer in_buf > * ZSTD_outBuffer out_buf > > are used in all functions to pass the compression parameters and the > local variables consume some space. We can move them to the workspace > and reduce the stack consumption: >

Re: [PATCH 1/2] btrfs: Fix wild memory access in compression level parser

On 2017年11月15日 23:11, David Sterba wrote: > On Mon, Nov 06, 2017 at 10:43:18AM +0800, Qu Wenruo wrote: >> [BUG] >> Kernel panic when mounting with "-o compress" mount option. >> KASAN will report like: >> -- >> == >> BUG: KASAN:

[PATCH v2] fstests: btrfs/143: make test case more reliable

This changes to use '_scratch_cycle_mount' to drop all caches btrfs could have in order to avoid an issue that drop_caches somehow doesn't work on Nikolay's box. Also use bash -c to run 'read' only when %pid is odd so that we can read the faulty disk. Reported-by: Nikolay Borisov

[PATCH v3] Btrfs: avoid losing data raid profile when deleting a device

We've avoided data losing raid profile when doing balance, but it turns out that deleting a device could also result in the same problem. Say we have 3 disks, and they're created with '-d raid1' profile. - We have chunk P (the only data chunk on the empty btrfs). - Suppose that chunk P's two

[PATCH v2] Btrfs: set plug for fsync

Setting plug can merge adjacent IOs before dispatching IOs to the disk driver. Without plug, it'd not be a problem for single disk usecases, but for multiple disks using raid profile, a large IO can be split to several IOs of stripe length, and plug can be helpful to bring them together for each

Re: [PATCH] btrfs: fix deadlock when writing out space cache

On Wed, Nov 15, 2017 at 04:20:52PM -0500, Josef Bacik wrote: > From: Josef Bacik > > If we fail to prepare our pages for whatever reason (out of memory in > our case) we need to make sure to drop the block_group->data_rwsem, > otherwise hilarity ensues. > Reviewed-by: Liu Bo

Re: Tiered storage?

Roy Sigurd Karlsbakk posted on Wed, 15 Nov 2017 15:10:08 +0100 as excerpted: >>> As for dedupe there is (to my knowledge) nothing fully automatic yet. >>> You have to run a program to scan your filesystem but all the >>> deduplication is done in the kernel. >>> duperemove works apparently quite

Re: zstd compression

Austin S. Hemmelgarn posted on Wed, 15 Nov 2017 07:57:06 -0500 as excerpted: > The 'compress' and 'compress-force' mount options only impact newly > written data. The compression used is stored with the metadata for the > extents themselves, so any existing data on the volume will be read just >

Re: [PATCH] btrfs: fix deadlock when writing out space cache

On Wed, Nov 15, 2017 at 04:20:52PM -0500, Josef Bacik wrote: > From: Josef Bacik > > If we fail to prepare our pages for whatever reason (out of memory in > our case) we need to make sure to drop the block_group->data_rwsem, > otherwise hilarity ensues. > > Signed-off-by: Josef

[PATCH] btrfs: fix deadlock when writing out space cache

From: Josef Bacik If we fail to prepare our pages for whatever reason (out of memory in our case) we need to make sure to drop the block_group->data_rwsem, otherwise hilarity ensues. Signed-off-by: Josef Bacik --- fs/btrfs/free-space-cache.c | 6 +- 1 file

Re: Read before you deploy btrfs + zstd

Austin S. Hemmelgarn posted on Tue, 14 Nov 2017 07:38:03 -0500 as excerpted: > On 2017-11-14 02:34, Martin Steigerwald wrote: >> Hello David. >> >> David Sterba - 13.11.17, 23:50: >>> while 4.14 is still fresh, let me address some concerns I've seen on >>> linux forums already. >>> >>> The newly

Re: Read before you deploy btrfs + zstd

On 11/15/17, 6:41 AM, "David Sterba" wrote: > The branch is now in a state that can be tested. Turns out the memory > requirements are too much for grub, so the boot fails with "not enough > memory". The calculated value > > ZSTD_BTRFS_MAX_INPUT: 131072 >

Re: Read before you deploy btrfs + zstd

Imran Geriskovan posted on Wed, 15 Nov 2017 20:16:56 +0200 as excerpted: > On 11/15/17, Martin Steigerwald wrote: >> Somehow I am happy that I still have a plain Ext4 for /boot. :) > > You may use uncompressed btrfs for /boot. > Both Syslinux (my choice) and Grub supports

[josef-btrfs:slab-priority 1/1] mm/vmscan.c:336: undefined reference to `__udivdi3'

tree: https://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git slab-priority head: bd319083ec02fd19b9f3522935d3c6c0528e1864 commit: bd319083ec02fd19b9f3522935d3c6c0528e1864 [1/1] mm: use sc->priority for slab shrink targets config: i386-randconfig-n0-201746 (attached as .config)

Re: Read before you deploy btrfs + zstd

On 11/15/17, Martin Steigerwald wrote: > Somehow I am happy that I still have a plain Ext4 for /boot. :) You may use uncompressed btrfs for /boot. Both Syslinux (my choice) and Grub supports it. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the

[PATCH 1/6] btrfs: switch btrfs_trans_handle::adding_csums to bool

The semantics of adding_csums matches bool, 'short' was most likely used to save space in a698d0755adb6f2 ("Btrfs: add a type field for the transaction handle"). Signed-off-by: David Sterba --- fs/btrfs/inode.c | 4 ++-- fs/btrfs/transaction.h | 2 +- 2 files changed, 3

[PATCH 0/6] Shrink some data structures

Unused members removed, other reordered. struct btrfs_transaction: - /* size: 432, cachelines: 7, members: 27 */ - /* sum members: 416, holes: 4, sum holes: 16 */ - /* last cacheline: 48 bytes */ + /* size: 416, cachelines: 7, members: 27 */ + /* sum members: 412,

[PATCH 6/6] btrfs: reoder btrfs_transaction members for better packing

There are now 20 bytes of holes, we can reduce that to 4 by minor changes. Moving 'aborted' to the status and flags is also more logical, similar for num_dirty_bgs. The size goes from 432 to 416. Signed-off-by: David Sterba --- fs/btrfs/transaction.h | 4 ++-- 1 file changed,

[PATCH 4/6] btrfs: reoder btrfs_trans_handle members for better packing

Recent updates to the structure left some holes, reorder the types so the packing is tight. The size goes from 112 to 104 on 64bit. Signed-off-by: David Sterba --- fs/btrfs/transaction.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git

[PATCH 2/6] btrfs: remove unused member of btrfs_trans_handle

Last user was removed in a monster commit a22285a6a32390195235171 ("Btrfs: Integrate metadata reservation with start_transaction") in 2010. Signed-off-by: David Sterba --- fs/btrfs/transaction.h | 1 - 1 file changed, 1 deletion(-) diff --git a/fs/btrfs/transaction.h

[PATCH 3/6] btrfs: switch to refcount_t type for btrfs_trans_handle::use_count

The use_count is a reference counter, we can use the refcount_t type, though we don't use the atomicity. This is not a performance critical code and we could catch the underflows. The type is changed from long, but the number of references will fit an int. Signed-off-by: David Sterba

[PATCH 5/6] btrfs: use narrower type for btrfs_transaction::num_dirty_bgs

The u64 is an overkill here, we could not possibly create that many blockgroups in one transaction. Signed-off-by: David Sterba --- fs/btrfs/extent-tree.c | 2 +- fs/btrfs/transaction.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git

[PATCH] btrfs: move some zstd work data from stack to workspace

* ZSTD_inBuffer in_buf * ZSTD_outBuffer out_buf are used in all functions to pass the compression parameters and the local variables consume some space. We can move them to the workspace and reduce the stack consumption: zstd.c:zstd_decompress-24 (136 -> 112)

Re: [PATCH 0/4] factor __btrfs_open_devices()

On Thu, Nov 09, 2017 at 11:45:22PM +0800, Anand Jain wrote: > This is in preparation to help bring the missing device back to the > alloc list dynamically. As of now if we run 'btrfs dev scan ' > on a mounted FS, nothing happens, there is no code which handles this > condition. So the idea is to

Re: [PATCH 2/4] btrfs: let variable required be declared inside the loop

On Thu, Nov 09, 2017 at 05:53:58PM +0200, Nikolay Borisov wrote: > On 9.11.2017 17:45, Anand Jain wrote: > > A preparation patch to create the actual device open in a new function. > Just say you are reducing the visibility of some variables to the loop. This patch is IMHO too fine-grained and

Re: [PATCH 3/7] btrfs: use i_size_read() instead of open code

On Tue, Nov 07, 2017 at 07:44:59AM +0800, Anand Jain wrote: > > > On 11/07/2017 12:52 AM, David Sterba wrote: > > On Mon, Nov 06, 2017 at 04:36:14PM +0800, Anand Jain wrote: > >> As i_size_read() takes care of 32bit smp or preempt cases as well. > > > > Can bdev->bd_inode->i_size change so that

Re: [PATCH] btrfs: ref-verify: Remove unused parameter from walk_up_tree() to kill warning

On Wed, Nov 15, 2017 at 04:04:40PM +0100, Geert Uytterhoeven wrote: > With gcc-4.1.2: > > fs/btrfs/ref-verify.c: In function ‘btrfs_build_ref_tree’: > fs/btrfs/ref-verify.c:1017: warning: ‘root’ is used uninitialized in this > function > > The variable is indeed passed uninitialized,

Re: Read before you deploy btrfs + zstd

David Sterba - 15.11.17, 15:39: > On Tue, Nov 14, 2017 at 07:53:31PM +0100, David Sterba wrote: > > On Mon, Nov 13, 2017 at 11:50:46PM +0100, David Sterba wrote: > > > Up to now, there are no bootloaders supporting ZSTD. > > > > I've tried to implement the support to GRUB, still incomplete and

Re: [PATCH v2 2/3] btrfs: tree-checker: Add checker for dir item

On Wed, Nov 08, 2017 at 08:54:25AM +0800, Qu Wenruo wrote: > Add checker for dir item, for key types DIR_ITEM, DIR_INDEX and > XATTR_ITEM. > > This checker does comprehensive check for: > 1) dir_item header and its data size >Against item boundary and maximum name/xattr length. >This part

Re: [PATCH v2 1/3] btrfs: tree-checker: Fix false panic for sanity test

On Wed, Nov 08, 2017 at 08:54:24AM +0800, Qu Wenruo wrote: > [BUG] > If we run btrfs with CONFIG_BTRFS_FS_RUN_SANITY_TESTS=y, it will > instantly cause kernel panic like: > > -- > ... > assertion failed: 0, file: fs/btrfs/disk-io.c, line: 3853 > ... > Call Trace: >

Re: [PATCH v2 3/3] btrfs: Cleanup existing name_len checks

On Wed, Nov 08, 2017 at 08:54:26AM +0800, Qu Wenruo wrote: > Since tree-checker has verified leaf when reading from disk, we don't > need the existing verify_dir_item() or btrfs_is_name_len_valid(). > > Signed-off-by: Qu Wenruo Reviewed-by: David Sterba -- To

Re: [PATCH v2 1/3] btrfs: tree-checker: Fix false panic for sanity test

On Wed, Nov 08, 2017 at 04:03:35PM +0800, Qu Wenruo wrote: > >> +int btrfs_check_leaf_full(struct btrfs_root *root, struct extent_buffer > >> *leaf) > >> +{ > >> + return check_leaf(root, leaf, true); > >> +} > >> + > >> +int btrfs_check_leaf_relaxed(struct btrfs_root *root, > >> +

Re: Btrfs: add a extent ref verify tool

On Tue, Nov 14, 2017 at 11:46 PM, Linux Kernel Mailing List wrote: > Web: > https://git.kernel.org/torvalds/c/fd708b81d972a0714b02a60eb4792fdbf15868c4 > Commit: fd708b81d972a0714b02a60eb4792fdbf15868c4 > Parent:

Re: [PATCH v2 2/2] Btrfs: fix reported number of inode blocks after buffered append writes

On Sat, Nov 04, 2017 at 04:20:07AM +, fdman...@kernel.org wrote: > From: Filipe Manana > > The patch from commit a7e3b975a0f9 ("Btrfs: fix reported number of inode > blocks") introduced a regression where if we do a buffered write starting > at position equal to or greater

Re: [PATCH 1/2] btrfs: Fix wild memory access in compression level parser

On Mon, Nov 06, 2017 at 10:43:18AM +0800, Qu Wenruo wrote: > [BUG] > Kernel panic when mounting with "-o compress" mount option. > KASAN will report like: > -- > == > BUG: KASAN: wild-memory-access in strncmp+0x31/0xc0 > Read of

[PATCH] btrfs: ref-verify: Remove unused parameter from walk_up_tree() to kill warning

With gcc-4.1.2: fs/btrfs/ref-verify.c: In function ‘btrfs_build_ref_tree’: fs/btrfs/ref-verify.c:1017: warning: ‘root’ is used uninitialized in this function The variable is indeed passed uninitialized, but it is never used by the callee. However, not all versions of gcc are smart

Re: Read before you deploy btrfs + zstd

On Tue, Nov 14, 2017 at 07:53:31PM +0100, David Sterba wrote: > On Mon, Nov 13, 2017 at 11:50:46PM +0100, David Sterba wrote: > > Up to now, there are no bootloaders supporting ZSTD. > > I've tried to implement the support to GRUB, still incomplete and hacky > but most of the code is there. The

Re: Tiered storage?

>> As for dedupe there is (to my knowledge) nothing fully automatic yet. >> You have to run a program to scan your filesystem but all the >> deduplication is done in the kernel. >> duperemove works apparently quite well when I tested it, but there may >> be some performance implications. >

Re: [RFC PATCH 0/3] btrfs-progs: Add support for running the tools from

On Wed, Nov 01, 2017 at 01:42:42AM +, Gu, Jinxiang wrote: > > On 2017年10月31日 16:43, Gu Jinxiang wrote: > > > Add support for running the tools from a given path (for > > > example,/usr/bin) by setting $EXEC while running tests. > > > > > > Achieved: > > > Specify the location of binary and run

Re: zstd compression

On 2017-11-15 05:35, Imran Geriskovan wrote: On 11/15/17, Lukas Pirl wrote: you might be interested in the thread "Read before you deploy btrfs + zstd"¹. Thanks. I've read it. Bootloader is not an issue since /boot is on another uncompressed fs. Let me make my question

Re: Tiered storage?

On 2017-11-15 02:11, waxhead wrote: As a regular BTRFS user I can tell you that there is no such thing as hot data tracking yet. Some people seem to use bcache together with btrfs and come asking for help on the mailing list. Bcache works fine recently. It was only with older versions that

Re: Tiered storage?

On 2017-11-15 04:26, Marat Khalili wrote: On 15/11/17 10:11, waxhead wrote: hint: you need more than two for raid1 if you want to stay safe Huh? Two is not enough? Having three or more makes a difference? (Or, you mean hot spare?) They're probably referring to an issue where a two device

Hello Dear...

Hello Dear... I know that this message will come to you as a surprise. I hoped that you will not expose or betray this trust and confident that I am about to repose on you, my name is M, Shakour Rosarita. I am 19 years old Girl, female, from Tartu Syria, (never married) 61 kg, white in

Re: zstd compression

On 11/15/17, Lukas Pirl wrote: > you might be interested in the thread "Read before you deploy > btrfs + zstd"¹. Thanks. I've read it. Bootloader is not an issue since /boot is on another uncompressed fs. Let me make my question more generic: Can there be any issues for

Re: [PATCH] btrfs: handle dynamically reappearing missing device

the system] url: https://github.com/0day-ci/linux/commits/Anand-Jain/btrfs-handle-dynamically-reappearing-missing-device/20171115-143047 base: https://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git next config: sparc64-allyesconfig (attached as .config) compiler: sparc64-linux-gnu-gcc

Re: [PATCH v2] btrfs/154: test for device dynamic rescan

On 11/15/2017 02:47 PM, Eryu Guan wrote: On Wed, Nov 15, 2017 at 11:05:15AM +0800, Anand Jain wrote: Make sure missing device is included in the alloc list when it is scanned on a mounted FS. This test case needs btrfs kernel patch which is in the ML [PATCH] btrfs: handle dynamically

Re: zstd compression

Hi Imran, On 11/15/2017 09:51 AM, Imran Geriskovan wrote as excerpted: > Any further advices? you might be interested in the thread "Read before you deploy btrfs + zstd"¹. Cheers, Lukas ¹ https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg69871.html -- To unsubscribe from this list:

Re: Tiered storage?

On 15/11/17 10:11, waxhead wrote: hint: you need more than two for raid1 if you want to stay safe Huh? Two is not enough? Having three or more makes a difference? (Or, you mean hot spare?) -- With Best Regards, Marat Khalili -- To unsubscribe from this list: send the line "unsubscribe

zstd compression

Kernel 4.14 now includes btrfs zstd compression support. My question: I currently have a fs mounted and used with "compress=lzo" option. What happens if I change it to "compress=zstd"? My guess is that existing files will be read and uncompressed via lzo. And new files will be written with zstd