Re: [f2fs-dev] [PATCH v6] f2fs: compress: support compress level

2020-12-03 Thread Gao Xiang
Hi Chao, On Fri, Dec 04, 2020 at 03:09:20PM +0800, Chao Yu wrote: > On 2020/12/4 8:31, Gao Xiang wrote: > > could make more sense), could you leave some CR numbers about these > > algorithms on typical datasets (enwik9, silisia.tar or else.) with 16k > > cluster size? > > Just from a quick test w

[f2fs-dev] [PATCH v2] f2fs: fix race of pending_pages in decompression

2020-12-03 Thread Daeho Jeong
From: Daeho Jeong I found out f2fs_free_dic() is invoked in a wrong timing, but f2fs_verify_bio() still needed the dic info and it triggered the below kernel panic. It has been caused by the race condition of pending_pages value between decompression and verity logic, when the same compression cl

Re: [f2fs-dev] [PATCH v6] f2fs: compress: support compress level

2020-12-03 Thread Chao Yu
On 2020/12/4 8:31, Gao Xiang wrote: could make more sense), could you leave some CR numbers about these algorithms on typical datasets (enwik9, silisia.tar or else.) with 16k cluster size? Just from a quick test with enwik9 on vm: Original blocks:244382 lz4

Re: [f2fs-dev] [PATCH] f2fs: fix race of pending_pages in decompression

2020-12-03 Thread Daeho Jeong
Thanks for the explanation about verity. I got your point. Thanks~ 2020년 12월 4일 (금) 오후 2:18, Eric Biggers 님이 작성: > > On Fri, Dec 04, 2020 at 02:00:34PM +0900, Daeho Jeong wrote: > > I think I don't understand how verity works. > > Right after verity is enabled on a file, is the verity logic workin

Re: [f2fs-dev] [PATCH] f2fs: fix race of pending_pages in decompression

2020-12-03 Thread Eric Biggers
On Fri, Dec 04, 2020 at 02:00:34PM +0900, Daeho Jeong wrote: > I think I don't understand how verity works. > Right after verity is enabled on a file, is the verity logic working > for the whole file data area? > Or it's just working for the data area which is updated after verity is > enabled? >

Re: [f2fs-dev] [PATCH] f2fs: fix race of pending_pages in decompression

2020-12-03 Thread Daeho Jeong
I think I don't understand how verity works. Right after verity is enabled on a file, is the verity logic working for the whole file data area? Or it's just working for the data area which is updated after verity is enabled? 2020년 12월 4일 (금) 오후 1:51, Eric Biggers 님이 작성: > > On Fri, Dec 04, 2020 at

Re: [f2fs-dev] [PATCH] f2fs: fix race of pending_pages in decompression

2020-12-03 Thread Eric Biggers
On Fri, Dec 04, 2020 at 01:48:24PM +0900, Daeho Jeong wrote: > Eric, > > I have another question. > I understand enabling the verity can be possible in the middle of I/O. > Is the opposite way also possible? Actually, I couldn't find any > disabling function of it, though. > No, disabling fs-ver

Re: [f2fs-dev] [PATCH] f2fs: fix race of pending_pages in decompression

2020-12-03 Thread Daeho Jeong
Eric, I have another question. I understand enabling the verity can be possible in the middle of I/O. Is the opposite way also possible? Actually, I couldn't find any disabling function of it, though. 2020년 12월 4일 (금) 오후 1:31, Daeho Jeong 님이 작성: > > > Are you sure? I thought that compression (an

Re: [f2fs-dev] [PATCH] f2fs: fix race of pending_pages in decompression

2020-12-03 Thread Daeho Jeong
> Are you sure? I thought that compression (and encryption) apply to the whole > file, including any Merkle tree blocks past i_size. This "dic" structure is only for "de"compression, so we don't need to worry about going beyond i_size case. > Also, even if you include the i_size check, it's stil

Re: [f2fs-dev] [PATCH] f2fs: fix race of pending_pages in decompression

2020-12-03 Thread Eric Biggers
On Fri, Dec 04, 2020 at 12:43:23PM +0900, Daeho Jeong wrote: > STEP_VERITY is enabled by f2fs_need_verity() and the function is like below. > We already know the second condition (idx < DIV_ROUND_UP...) is > satisfied when invoking f2fs_alloc_dic(). > > static inline bool f2fs_need_verity(const st

Re: [f2fs-dev] [PATCH] f2fs: fix race of pending_pages in decompression

2020-12-03 Thread Daeho Jeong
STEP_VERITY is enabled by f2fs_need_verity() and the function is like below. We already know the second condition (idx < DIV_ROUND_UP...) is satisfied when invoking f2fs_alloc_dic(). static inline bool f2fs_need_verity(const struct inode *inode, pgoff_t idx) { return fsverity_active(inode)

Re: [f2fs-dev] [PATCH] f2fs: fix race of pending_pages in decompression

2020-12-03 Thread Eric Biggers
On Fri, Dec 04, 2020 at 09:58:47AM +0900, Daeho Jeong wrote: > diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c > index 87090da8693d..cdf72e153da0 100644 > --- a/fs/f2fs/compress.c > +++ b/fs/f2fs/compress.c > @@ -803,8 +803,6 @@ void f2fs_decompress_pages(struct bio *bio, struct page > *page,

Re: [f2fs-dev] [PATCH v6] f2fs: compress: support compress level

2020-12-03 Thread Gao Xiang
On Fri, Dec 04, 2020 at 11:11:03AM +0800, Chao Yu wrote: > On 2020/12/4 10:47, Gao Xiang wrote: ... > > > future (and add more dependency to algorithms, you might see BWT-based bzip2 > > removal patch > > Oops, is that really allowed? I don't this is a good idea...and I don't see > there > are

Re: [f2fs-dev] [PATCH v6] f2fs: compress: support compress level

2020-12-03 Thread Chao Yu
On 2020/12/4 10:47, Gao Xiang wrote: On Fri, Dec 04, 2020 at 10:38:08AM +0800, Chao Yu wrote: On 2020/12/4 10:06, Gao Xiang wrote: On Fri, Dec 04, 2020 at 09:56:27AM +0800, Chao Yu wrote: ... Keep lz4hc dirty data under writeback could block writeback, make kswapd busy, and direct memory

Re: [f2fs-dev] [PATCH v6] f2fs: compress: support compress level

2020-12-03 Thread Gao Xiang
On Fri, Dec 04, 2020 at 10:38:08AM +0800, Chao Yu wrote: > On 2020/12/4 10:06, Gao Xiang wrote: > > On Fri, Dec 04, 2020 at 09:56:27AM +0800, Chao Yu wrote: ... > > > > > Keep lz4hc dirty data under writeback could block writeback, make kswapd > > busy, and direct memory reclaim path, I guess t

Re: [f2fs-dev] [PATCH v6] f2fs: compress: support compress level

2020-12-03 Thread Chao Yu
On 2020/12/4 10:06, Gao Xiang wrote: On Fri, Dec 04, 2020 at 09:56:27AM +0800, Chao Yu wrote: Hi Xiang, On 2020/12/4 8:31, Gao Xiang wrote: Hi Chao, On Thu, Dec 03, 2020 at 11:32:34AM -0800, Eric Biggers wrote: ... What is the use case for storing the compression level on-disk? Keep in m

Re: [f2fs-dev] [PATCH v6] f2fs: compress: support compress level

2020-12-03 Thread Gao Xiang
On Fri, Dec 04, 2020 at 09:56:27AM +0800, Chao Yu wrote: > Hi Xiang, > > On 2020/12/4 8:31, Gao Xiang wrote: > > Hi Chao, > > > > On Thu, Dec 03, 2020 at 11:32:34AM -0800, Eric Biggers wrote: > > > > ... > > > > > > > > What is the use case for storing the compression level on-disk? > > > > >

Re: [f2fs-dev] [PATCH v6] f2fs: compress: support compress level

2020-12-03 Thread Chao Yu
Hi Xiang, On 2020/12/4 8:31, Gao Xiang wrote: Hi Chao, On Thu, Dec 03, 2020 at 11:32:34AM -0800, Eric Biggers wrote: ... What is the use case for storing the compression level on-disk? Keep in mind that compression levels are an implementation detail; the exact compressed data that is prod

Re: [f2fs-dev] [PATCH v6] f2fs: compress: support compress level

2020-12-03 Thread Chao Yu
On 2020/12/4 3:32, Eric Biggers wrote: On Thu, Dec 03, 2020 at 02:17:15PM +0800, Chao Yu wrote: +config F2FS_FS_LZ4HC + bool "LZ4HC compression support" + depends on F2FS_FS_COMPRESSION + depends on F2FS_FS_LZ4 + select LZ4HC_COMPRESS + default y + help +

[f2fs-dev] [PATCH] f2fs: fix race of pending_pages in decompression

2020-12-03 Thread Daeho Jeong
From: Daeho Jeong I found out f2fs_free_dic() is invoked in a wrong timing, but f2fs_verify_bio() still needed the dic info and it triggered the below kernel panic. It has been caused by the race condition of pending_pages value between decompression and verity logic, when the same compression cl

Re: [f2fs-dev] [PATCH v6] f2fs: compress: support compress level

2020-12-03 Thread Gao Xiang
Hi Chao, On Thu, Dec 03, 2020 at 11:32:34AM -0800, Eric Biggers wrote: ... > > What is the use case for storing the compression level on-disk? > > Keep in mind that compression levels are an implementation detail; the exact > compressed data that is produced by a particular algorithm at a part

Re: [f2fs-dev] [PATCH v7 0/8] add support for direct I/O with fscrypt using blk-crypto

2020-12-03 Thread Satya Tangirala via Linux-f2fs-devel
On Tue, Nov 17, 2020 at 12:15:26PM -0500, Theodore Y. Ts'o wrote: > What is the expected use case for Direct I/O using fscrypt? This > isn't a problem which is unique to fscrypt, but one of the really > unfortunate aspects of the DIO interface is the silent fallback to > buffered I/O. We've lived

[f2fs-dev] [PATCH 1/2] f2fs: don't allow any writes on readonly mount

2020-12-03 Thread Jaegeuk Kim
generic_make_request: Trying to write to read-only block-device dm-5 (partno 0) WARNING: CPU: 7 PID: 546 at block/blk-core.c:2190 generic_make_request_checks+0x664/0x690 pc : generic_make_request_checks+0x664/0x690 lr : generic_make_request_checks+0x664/0x690 Call trace: generic_make_request_chec

[f2fs-dev] [PATCH 2/2] f2fs: introduce max_io_bytes, a sysfs entry, to limit bio size

2020-12-03 Thread Jaegeuk Kim
This patch adds max_io_bytes to limit bio size when f2fs tries to merge consecutive IOs. This can give a testing point to split out bios and check end_io handles those bios correctly. This is used to capture a recent bug on the decompression and fsverity flow. Signed-off-by: Jaegeuk Kim --- Docu

Re: [f2fs-dev] [PATCH v6 1/3] lib: zstd: Add kernel-specific API

2020-12-03 Thread Nick Terrell via Linux-f2fs-devel
> On Dec 2, 2020, at 9:03 PM, Michał Mirosław wrote: > > On Thu, Dec 03, 2020 at 03:59:21AM +, Nick Terrell wrote: >> On Dec 2, 2020, at 7:14 PM, Michał Mirosław wrote: >>> On Thu, Dec 03, 2020 at 01:42:03AM +, Nick Terrell wrote: On Dec 2, 2020, at 5:16 PM, Michał Mirosław

[f2fs-dev] [PATCH v7 0/3] Update to zstd-1.4.6

2020-12-03 Thread Nick Terrell
From: Nick Terrell Please pull from [email protected]:terrelln/linux.git tags/v7-zstd-1.4.6 to get these changes. Alternatively the patchset is included. This patchset upgrades the zstd library to the latest upstream release. The current zstd version in the kernel is a modified version of upst

[f2fs-dev] [PATCH v7 1/3] lib: zstd: Add kernel-specific API

2020-12-03 Thread Nick Terrell
From: Nick Terrell This patch: - Moves `include/linux/zstd.h` -> `include/linux/zstd_lib.h` - Adds a new API in `include/linux/zstd.h` that is functionally equivalent to the in-use subset of the current API. Functions are renamed to avoid symbol collisions with zstd, to make it clear it is

[f2fs-dev] [PATCH v7 2/3] lib: zstd: Add decompress_sources.h for decompress_unzstd

2020-12-03 Thread Nick Terrell
From: Nick Terrell Adds decompress_sources.h which includes every .c file necessary for zstd decompression. This is used in decompress_unzstd.c so the internal structure of the library isn't exposed. This allows us to upgrade the zstd library version without modifying any callers. Instead we jus

Re: [f2fs-dev] [PATCH v6] f2fs: compress: support compress level

2020-12-03 Thread Eric Biggers
On Thu, Dec 03, 2020 at 02:17:15PM +0800, Chao Yu wrote: > +config F2FS_FS_LZ4HC > + bool "LZ4HC compression support" > + depends on F2FS_FS_COMPRESSION > + depends on F2FS_FS_LZ4 > + select LZ4HC_COMPRESS > + default y > + help > + Support LZ4HC compress algorithm, if

Re: [f2fs-dev] [PATCH v7 6/8] ext4: support direct I/O with fscrypt using blk-crypto

2020-12-03 Thread Theodore Y. Ts'o
On Tue, Nov 17, 2020 at 02:07:06PM +, Satya Tangirala wrote: > From: Eric Biggers > > Wire up ext4 with fscrypt direct I/O support. Direct I/O with fscrypt is > only supported through blk-crypto (i.e. CONFIG_BLK_INLINE_ENCRYPTION must > have been enabled, the 'inlinecrypt' mount option must h

Re: [f2fs-dev] [PATCH] f2fs: introduce a new per-sb directory in sysfs

2020-12-03 Thread Chao Yu
Jaegeuk, Can you comment on this patch? On 2020/11/27 17:01, Chao Yu wrote: Add a new directory 'stat' in path of /sys/fs/f2fs//, later we can add new readonly stat sysfs file into this directory, it will make directory less mess. Signed-off-by: Chao Yu --- fs/f2fs/f2fs.h | 5 +++- fs/f

[f2fs-dev] Disinfection

2020-12-03 Thread Diego Sánchez via Linux-f2fs-devel
Good morning, looking for companies interested in raising additional capital by diversifying their offer in soaps, liquids and gels for hand disinfection and cosmetics for body and hair care. The distribution of innovative products corresponding to the current preferences of customers in the f

Re: [f2fs-dev] Using secure erase in f2fs

2020-12-03 Thread Mikhail Novosyolov via Linux-f2fs-devel
3 декабря 2020 г. 2:41:14 GMT+03:00, Daeho Jeong пишет: >Hi Mikhail, > >As you noticed, f2fs already sends discard commands on released blocks. >Do you want to fill the blocks with zero data, instead of sending >discard commands? Yes, I want to fill the blocks with zero data for multiple times

Re: [f2fs-dev] [PATCH v3] f2fs: avoid race condition for shinker count

2020-12-03 Thread Chao Yu
On 2020/12/3 15:55, Jaegeuk Kim wrote: How about adding another variable to monitor it? That makes more sense. :) I've checked this, LGTM. Reviewed-by: Chao Yu Thanks, From bdc5a805487f0210df7ef4e85ce5a4f0471bca72 Mon Sep 17 00:00:00 2001 From: Jaegeuk Kim Date: Fri, 6 Nov 2020 13:22:0

Re: [f2fs-dev] [PATCH v3] f2fs: avoid race condition for shinker count

2020-12-03 Thread Jaegeuk Kim
On 12/03, Chao Yu wrote: > On 2020/12/3 14:07, Jaegeuk Kim wrote: > > On 11/11, Jaegeuk Kim wrote: > > > Light reported sometimes shinker gets nat_cnt < dirty_nat_cnt resulting in > > > wrong do_shinker work. Let's avoid to get stale data by using > > > nat_tree_lock. > > > > > > Reported-by: Lig