Re: [f2fs-dev] [PATCH 1/3] f2fs: avoid using __GFP_NOFAIL

2017-11-06 Thread Michal Hocko
ck(&im->ino_lock); > - radix_tree_preload_end(); > + > + if (preloaded) > + radix_tree_preload_end(); > > if (e != tmp) > kmem_cache_free(ino_entry_slab, tmp); > -- > 2.14.1.145.gb3622a4ee -- Michal Hocko

Re: [f2fs-dev] [PATCH 1/3] f2fs: avoid using __GFP_NOFAIL

2017-11-06 Thread Michal Hocko
onditions. -- Michal Hocko SUSE Labs -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Linux-f2fs

Re: [f2fs-dev] [PATCH] mm: workingset: fix NULL ptr dereference

2018-04-09 Thread Michal Hocko
question. Why do you make all allocations for the mapping NOFS automatically? What kind of reclaim recursion problems are you trying to prevent? -- Michal Hocko SUSE Labs -- Check out the vibrant tech community on one of t

Re: [f2fs-dev] [PATCH] mm: workingset: fix NULL ptr dereference

2018-04-09 Thread Michal Hocko
On Mon 09-04-18 06:41:14, Matthew Wilcox wrote: > On Mon, Apr 09, 2018 at 02:48:52PM +0200, Michal Hocko wrote: > > On Mon 09-04-18 20:25:06, Chao Yu wrote: > > [...] > > > diff --git a/fs/f2fs/inode.c b/fs/f2fs/inode.c > > > index c852e800..cc63f8c448f0 1

Re: [f2fs-dev] [PATCH] mm: workingset: fix NULL ptr dereference

2018-04-10 Thread Michal Hocko
644 > --- a/mm/filemap.c > +++ b/mm/filemap.c > @@ -785,7 +785,7 @@ int replace_page_cache_page(struct page *old, struct page > *new, gfp_t gfp_mask) > VM_BUG_ON_PAGE(!PageLocked(new), new); > VM_BUG_ON_PAGE(new->mapping, new); > >

Re: [f2fs-dev] [PATCH] mm: workingset: fix NULL ptr dereference

2018-04-10 Thread Michal Hocko
On Tue 10-04-18 05:05:28, Matthew Wilcox wrote: > On Tue, Apr 10, 2018 at 10:26:43AM +0200, Michal Hocko wrote: > > On Mon 09-04-18 12:40:44, Matthew Wilcox wrote: > > > The problem is that the mapping gfp flags are used not only for allocating > > > pages, but also fo

Re: [f2fs-dev] [PATCH] mm: workingset: fix NULL ptr dereference

2018-04-10 Thread Michal Hocko
CLAIM_MASK filtering and a warning in slab for __GFP_ZERO looks like a reasonable step forward. -- Michal Hocko SUSE Labs -- Check out the vibrant tech community on one of the world's most engaging tech sites,

Re: [f2fs-dev] deadlock during writeback when using f2fs filesystem

2018-06-01 Thread Michal Hocko
(GFP_NOFS | __GFP_ZERO) > #define GFP_F2FS_HIGH_ZERO (GFP_NOFS | __GFP_ZERO | __GFP_HIGHMEM) > +#define GFP_F2FS_NODE_MAPPING (GFP_NOWAIT | __GFP_IO | __GFP_ZERO) > > Thanks, > Sahitya. > -- > -- > Sent by a consultant of the Qualcomm Innovation Center, Inc. &g

Re: [f2fs-dev] deadlock during writeback when using f2fs filesystem

2018-06-01 Thread Michal Hocko
On Fri 01-06-18 16:50:50, Sahitya Tummala wrote: > On Fri, Jun 01, 2018 at 12:26:09PM +0200, Michal Hocko wrote: > > On Fri 01-06-18 15:02:35, Sahitya Tummala wrote: > > > Hi, > > > > > > We are observing a deadlock scenario during FS writeback under

Re: [f2fs-dev] mm: mkfs.ext4 invoked oom-killer on i386 - pagecache_get_page

2020-05-19 Thread Michal Hocko
of the low mem is in the Normal zone which is completely missing here. How have you got to that configuration? I have to say I haven't seen anything like that on i386. The failing request is GFP_USER so highmem is not really allowed but free pages are way above watermarks so the allocation should hav

Re: [f2fs-dev] mm: mkfs.ext4 invoked oom-killer on i386 - pagecache_get_page

2020-05-19 Thread Michal Hocko
On Tue 19-05-20 10:11:25, Arnd Bergmann wrote: > On Tue, May 19, 2020 at 9:52 AM Michal Hocko wrote: > > > > On Mon 18-05-20 19:40:55, Naresh Kamboju wrote: > > > Thanks for looking into this problem. > > > > > > On Sat, 2 May 2020 at 02:28, Andrew Morto

Re: [f2fs-dev] mm: mkfs.ext4 invoked oom-killer on i386 - pagecache_get_page

2020-05-21 Thread Michal Hocko
ed. If memory.{low,min} is not used then the patch should be effectively a nop. Btw. do you see the problem when booting with cgroup_disable=memory kernel command line parameter? I suspect that something might be initialized for memcg incorrectly and the patch just makes it more visible for some reason

Re: [f2fs-dev] mm: mkfs.ext4 invoked oom-killer on i386 - pagecache_get_page

2020-05-21 Thread Michal Hocko
On Thu 21-05-20 16:11:11, Naresh Kamboju wrote: > On Thu, 21 May 2020 at 15:25, Michal Hocko wrote: > > > > On Wed 20-05-20 20:09:06, Chris Down wrote: > > > Hi Naresh, > > > > > > Naresh Kamboju writes: > > > > As a part of investigation

Re: [f2fs-dev] mm: mkfs.ext4 invoked oom-killer on i386 - pagecache_get_page

2020-05-21 Thread Michal Hocko
On Thu 21-05-20 05:24:27, Hugh Dickins wrote: > On Thu, 21 May 2020, Michal Hocko wrote: > > On Thu 21-05-20 16:11:11, Naresh Kamboju wrote: > > > On Thu, 21 May 2020 at 15:25, Michal Hocko wrote: > > > > > > > > On Wed 20-05-20 20:09

Re: [f2fs-dev] mm: mkfs.ext4 invoked oom-killer on i386 - pagecache_get_page

2020-05-21 Thread Michal Hocko
On Thu 21-05-20 11:55:16, Michal Hocko wrote: > On Wed 20-05-20 20:09:06, Chris Down wrote: > > Hi Naresh, > > > > Naresh Kamboju writes: > > > As a part of investigation on this issue LKFT teammate Anders Roxell > > > git bisected the problem an

Re: [f2fs-dev] mm: mkfs.ext4 invoked oom-killer on i386 - pagecache_get_page

2020-05-28 Thread Michal Hocko
0 Not tainted > > > > > 5.7.0-rc6-next-20200519+ #1 > > > > > [ 35.532121] Hardware name: Supermicro SYS-5019S-ML/X11SSH-F, BIOS > > > > > 2.2 05/23/2018 > > > > > [ 35.539507] EIP: mem_cgroup_get_nr_swap_pages+0x28/0x60 > > Swap

Re: [f2fs-dev] mm: mkfs.ext4 invoked oom-killer on i386 - pagecache_get_page

2020-05-28 Thread Michal Hocko
s this still hold? Because I still have a hard time to understand how those three patches could have the observed effects. -- Michal Hocko SUSE Labs ___ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] mm: mkfs.ext4 invoked oom-killer on i386 - pagecache_get_page

2020-05-29 Thread Michal Hocko
ld confirm. Maybe my debugging patch is incomplete or used incorrectly (maybe it would be esier to use printk rather than trace_printk?). -- Michal Hocko SUSE Labs ___ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] mm: mkfs.ext4 invoked oom-killer on i386 - pagecache_get_page

2020-06-11 Thread Michal Hocko
On Fri 29-05-20 11:49:20, Michal Hocko wrote: > On Fri 29-05-20 02:56:44, Chris Down wrote: > > Yafang Shao writes: > > > Look at this patch[1] carefully you will find that it introduces the > > > same issue that I tried to fix in another patch [2]. Even more sad is >

Re: [f2fs-dev] mm: mkfs.ext4 invoked oom-killer on i386 - pagecache_get_page

2020-06-12 Thread Michal Hocko
On Fri 12-06-20 15:13:22, Naresh Kamboju wrote: > On Thu, 11 Jun 2020 at 15:25, Michal Hocko wrote: > > > > On Fri 29-05-20 11:49:20, Michal Hocko wrote: > > > On Fri 29-05-20 02:56:44, Chris Down wrote: > > > > Yafang Shao writes: > > > Agreed. E

Re: [f2fs-dev] mm: mkfs.ext4 invoked oom-killer on i386 - pagecache_get_page

2020-06-17 Thread Michal Hocko
On Wed 17-06-20 19:07:20, Naresh Kamboju wrote: > On Thu, 21 May 2020 at 22:04, Michal Hocko wrote: > > > > On Thu 21-05-20 11:55:16, Michal Hocko wrote: > > > On Wed 20-05-20 20:09:06, Chris Down wrote: > > > > Hi Naresh, > > > > >

Re: [f2fs-dev] mm: mkfs.ext4 invoked oom-killer on i386 - pagecache_get_page

2020-06-17 Thread Michal Hocko
in > and mem_cgroup_below_low changed to `>`? If that fixes it, then that gives a > strong hint about what's going on here. This would work but I believe an explicit check for the root memcg would be easier to spot the reasoning. -- Michal Hocko SUSE Labs ___ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] mm: mkfs.ext4 invoked oom-killer on i386 - pagecache_get_page

2020-06-17 Thread Michal Hocko
On Wed 17-06-20 21:23:05, Naresh Kamboju wrote: > On Wed, 17 Jun 2020 at 19:41, Michal Hocko wrote: > > > > [Our emails have crossed] > > > > On Wed 17-06-20 14:57:58, Chris Down wrote: > > > Naresh Kamboju writes: > > > > mkfs -t ext4 /dev

Re: [f2fs-dev] mm: mkfs.ext4 invoked oom-killer on i386 - pagecache_get_page

2020-06-18 Thread Michal Hocko
> > should not be exposed to other MM parts. > > I agree that the current semantics are mentally taxing and we should > generally avoid exposing the implementation details outside of memcg where > possible. Do you have a suggested rework? :-) I would really prefer to do that wo

[f2fs-dev] [PATCH 1/2] mm: add PF_MEMALLOC_NOFS

2016-04-26 Thread Michal Hocko
From: Michal Hocko GFP_NOFS context is used for the following 4 reasons currently - to prevent from deadlocks when the lock held by the allocation context would be needed during the memory reclaim - to prevent from stack overflows during the reclaim because

[f2fs-dev] [PATCH 0/2] scop GFP_NOFS api

2016-04-26 Thread Michal Hocko
Hi, we have discussed this topic at LSF/MM this year. There was a general interest in the scope GFP_NOFS allocation context among some FS developers. For those who are not aware of the discussion or the issue I am trying to sort out (or at least start in that direction) please have a look at patch

[f2fs-dev] [PATCH 2/2] mm, debug: report when GFP_NO{FS, IO} is used explicitly from memalloc_no{fs, io}_{save, restore} context

2016-04-26 Thread Michal Hocko
From: Michal Hocko THIS PATCH IS FOR TESTING ONLY AND NOT MEANT TO HIT LINUS TREE It is desirable to reduce the direct GFP_NO{FS,IO} usage at minimum and prefer scope usage defined by memalloc_no{fs,io}_{save,restore} API. Let's help this process and add a debugging tool to catch wh

[f2fs-dev] [PATCH 1.1/2] xfs: abstract PF_FSTRANS to PF_MEMALLOC_NOFS

2016-04-27 Thread Michal Hocko
From: Michal Hocko xfs has defined PF_FSTRANS to declare a scope GFP_NOFS semantic quite some time ago. We would like to make this concept more generic and use it for other filesystems as well. Let's start by giving the flag a more genric name PF_MEMALLOC_NOFS which is in line with an ex

Re: [f2fs-dev] [PATCH 1/2] mm: add PF_MEMALLOC_NOFS

2016-04-27 Thread Michal Hocko
On Wed 27-04-16 09:07:02, Dave Chinner wrote: > On Tue, Apr 26, 2016 at 01:56:11PM +0200, Michal Hocko wrote: > > From: Michal Hocko > > > > GFP_NOFS context is used for the following 4 reasons currently > > - to prevent from deadlocks when the lock held by the a

Re: [f2fs-dev] [PATCH 1.2/2] mm: introduce memalloc_nofs_{save, restore} API

2016-04-27 Thread Michal Hocko
On Wed 27-04-16 13:54:35, Michal Hocko wrote: > From: Michal Hocko > Ups missed Dave's note about: > GFP_NOFS context is used for the following 4 reasons currently > - to prevent from deadlocks when the lock held by the allocation > context would be need

[f2fs-dev] [PATCH 1.2/2] mm: introduce memalloc_nofs_{save, restore} API

2016-04-27 Thread Michal Hocko
From: Michal Hocko GFP_NOFS context is used for the following 4 reasons currently - to prevent from deadlocks when the lock held by the allocation context would be needed during the memory reclaim - to prevent from stack overflows during the reclaim because

Re: [f2fs-dev] [PATCH 2/2] mm, debug: report when GFP_NO{FS, IO} is used explicitly from memalloc_no{fs, io}_{save, restore} context

2016-04-27 Thread Michal Hocko
On Wed 27-04-16 08:58:45, Dave Chinner wrote: > On Tue, Apr 26, 2016 at 01:56:12PM +0200, Michal Hocko wrote: > > From: Michal Hocko > > > > THIS PATCH IS FOR TESTING ONLY AND NOT MEANT TO HIT LINUS TREE > > > > It is desirable to reduce the direct GFP_NO{FS,IO

Re: [f2fs-dev] [PATCH 1.2/2] mm: introduce memalloc_nofs_{save, restore} API

2016-05-01 Thread Michal Hocko
rom 1968c0a8ace4090a9deca8f4c1a206ee546e595a Mon Sep 17 00:00:00 2001 From: Michal Hocko Date: Wed, 27 Apr 2016 22:32:57 +0200 Subject: [PATCH] fold me "mm: introduce memalloc_nofs_{save,restore} API" Lockdep infrastructure is hooked into early hot paths of the allocator so __lockdep_trace_alloc has to check for PF_

Re: [f2fs-dev] [PATCH 1.2/2] mm: introduce memalloc_nofs_{save, restore} API

2016-05-01 Thread Michal Hocko
Hi Dave, On Wed 27-04-16 13:54:35, Michal Hocko wrote: [...] > diff --git a/fs/xfs/kmem.h b/fs/xfs/kmem.h > index 0d83f332e5c2..b35688a54c9a 100644 > --- a/fs/xfs/kmem.h > +++ b/fs/xfs/kmem.h > @@ -50,7 +50,7 @@ kmem_flags_convert(xfs_km_flags_t flags) > l

Re: [f2fs-dev] [PATCH 1.2/2] mm: introduce memalloc_nofs_{save, restore} API

2016-05-01 Thread Michal Hocko
On Wed 27-04-16 22:09:27, Michal Hocko wrote: [...] > [ 53.993480] [] mark_held_locks+0x5e/0x74 > [ 53.993480] [] lockdep_trace_alloc+0xb2/0xb5 > [ 53.993480] [] kmem_cache_alloc+0x36/0x2b0 Scratch that. I got it. It is the lockdep annotation which I got wrong with my patch.

Re: [f2fs-dev] [PATCH 0/2] scop GFP_NOFS api

2016-05-01 Thread Michal Hocko
On Fri 29-04-16 15:35:42, NeilBrown wrote: > On Tue, Apr 26 2016, Michal Hocko wrote: > > > Hi, > > we have discussed this topic at LSF/MM this year. There was a general > > interest in the scope GFP_NOFS allocation context among some FS > > developers. For

Re: [f2fs-dev] [PATCH 1.1/2] xfs: abstract PF_FSTRANS to PF_MEMALLOC_NOFS

2016-05-01 Thread Michal Hocko
On Wed 27-04-16 11:41:51, Andreas Dilger wrote: > On Apr 27, 2016, at 5:54 AM, Michal Hocko wrote: [...] > > --- a/fs/xfs/kmem.c > > +++ b/fs/xfs/kmem.c > > @@ -80,13 +80,13 @@ kmem_zalloc_large(size_t size, xfs_km_flags_t flags) > > * context via PF_MEMALLOC_NOI

Re: [f2fs-dev] [PATCH 0/2] scop GFP_NOFS api

2016-05-03 Thread Michal Hocko
for you, or who you really are waiting on to free that > memory. > > Whenever trying to free memory I think you need to do best-effort > without blocking. I agree with that. Or at least you have to wait on something that is _guaranteed_ to make a forward progress.

Re: [f2fs-dev] [RFC PATCH] mm: introduce kv[mz]alloc helpers

2016-12-08 Thread Michal Hocko
On Thu 08-12-16 14:00:20, David Hildenbrand wrote: > Am 08.12.2016 um 11:33 schrieb Michal Hocko: > > From: Michal Hocko > > > > Using kmalloc with the vmalloc fallback for larger allocations is a > > common pattern in the kernel code. Yet we do not have any common

[f2fs-dev] [RFC PATCH] mm: introduce kv[mz]alloc helpers

2016-12-08 Thread Michal Hocko
From: Michal Hocko Using kmalloc with the vmalloc fallback for larger allocations is a common pattern in the kernel code. Yet we do not have any common helper for that and so users have invented their own helpers. Some of them are really creative when doing so. Let's just add kv[mz]allo

Re: [f2fs-dev] [RFC PATCH] mm: introduce kv[mz]alloc helpers

2016-12-11 Thread Michal Hocko
On Fri 09-12-16 02:00:17, Al Viro wrote: > On Fri, Dec 09, 2016 at 12:44:17PM +1100, Dave Chinner wrote: > > On Thu, Dec 08, 2016 at 11:33:00AM +0100, Michal Hocko wrote: > > > From: Michal Hocko > > > > > > Using kmalloc with the vmalloc fallback for larger a

Re: [f2fs-dev] [RFC PATCH] mm: introduce kv[mz]alloc helpers

2016-12-11 Thread Michal Hocko
On Fri 09-12-16 06:38:04, Al Viro wrote: > On Fri, Dec 09, 2016 at 07:22:25AM +0100, Michal Hocko wrote: > > > > Easier to handle those in vmalloc() itself. > > > > I think there were some attempts in the past but some of the code paths > > are burried too dee

Re: [f2fs-dev] [RFC PATCH] mm: introduce kv[mz]alloc helpers

2016-12-11 Thread Michal Hocko
On Fri 09-12-16 12:44:17, Dave Chinner wrote: > On Thu, Dec 08, 2016 at 11:33:00AM +0100, Michal Hocko wrote: > > From: Michal Hocko > > > > Using kmalloc with the vmalloc fallback for larger allocations is a > > common pattern in the kernel code. Yet we do not hav

Re: [f2fs-dev] [RFC PATCH] mm: introduce kv[mz]alloc helpers

2016-12-14 Thread Michal Hocko
On Tue 13-12-16 14:07:33, Joe Perches wrote: > On Tue, 2016-12-13 at 11:14 +0100, Michal Hocko wrote: > > Are there any more comments or objections to this patch? Is this a good > > start or kv[mz]alloc has to provide a way to cover GFP_NOFS users as > > well in the init

Re: [f2fs-dev] [RFC PATCH] mm: introduce kv[mz]alloc helpers

2016-12-14 Thread Michal Hocko
Are there any more comments or objections to this patch? Is this a good start or kv[mz]alloc has to provide a way to cover GFP_NOFS users as well in the initial version. On Thu 08-12-16 11:33:00, Michal Hocko wrote: > From: Michal Hocko > > Using kmalloc with the vmalloc fallback f

Re: [f2fs-dev] [RFC PATCH] mm: introduce kv[mz]alloc helpers

2016-12-14 Thread Michal Hocko
On Tue 13-12-16 13:55:46, Andreas Dilger wrote: > On Dec 13, 2016, at 3:14 AM, Michal Hocko wrote: > > > > Are there any more comments or objections to this patch? Is this a good > > start or kv[mz]alloc has to provide a way to cover GFP_NOFS users as > > well in the i

[f2fs-dev] [PATCH 2/9] xfs: introduce and use KM_NOLOCKDEP to silence reclaim lockdep false positives

2016-12-16 Thread Michal Hocko
From: Michal Hocko Now that the page allocator offers __GFP_NOLOCKDEP let's introduce KM_NOLOCKDEP alias for the xfs allocation APIs. While we are at it also change KM_NOFS users introduced by b17cb364dbbb ("xfs: fix missing KM_NOFS tags to keep lockdep happy") and use the ne

[f2fs-dev] [PATCH 5/9] xfs: use memalloc_nofs_{save, restore} instead of memalloc_noio*

2016-12-16 Thread Michal Hocko
From: Michal Hocko kmem_zalloc_large and _xfs_buf_map_pages use memalloc_noio_{save,restore} API to prevent from reclaim recursion into the fs because vmalloc can invoke unconditional GFP_KERNEL allocations and these functions might be called from the NOFS contexts. The memalloc_noio_save will

[f2fs-dev] [PATCH 6/9] jbd2: mark the transaction context with the scope GFP_NOFS context

2016-12-16 Thread Michal Hocko
From: Michal Hocko now that we have memalloc_nofs_{save,restore} api we can mark the whole transaction context as implicitly GFP_NOFS. All allocations will automatically inherit GFP_NOFS this way. This means that we do not have to mark any of those requests with GFP_NOFS and moreover all the

Re: [f2fs-dev] [PATCH 0/9 v2] scope GFP_NOFS api

2016-12-16 Thread Michal Hocko
On Fri 16-12-16 16:05:58, Mike Galbraith wrote: > On Thu, 2016-12-15 at 15:07 +0100, Michal Hocko wrote: > > Hi, > > I have posted the previous version here [1]. Since then I have added a > > support to suppress reclaim lockdep warnings (__GFP_NOLOCKDEP) to allow > &

[f2fs-dev] [PATCH 8/9] Revert "ext4: avoid deadlocks in the writeback path by using sb_getblk_gfp"

2016-12-16 Thread Michal Hocko
From: Michal Hocko This reverts commit c45653c341f5c8a0ce19c8f0ad4678640849cb86 because sb_getblk_gfp is not really needed as sb_getblk __getblk_gfp __getblk_slow grow_buffers grow_dev_page gfp_mask = mapping_gfp_constraint(inode->i_mapping, ~__GFP_FS) | gfp

[f2fs-dev] [PATCH 3/9] xfs: abstract PF_FSTRANS to PF_MEMALLOC_NOFS

2016-12-16 Thread Michal Hocko
From: Michal Hocko xfs has defined PF_FSTRANS to declare a scope GFP_NOFS semantic quite some time ago. We would like to make this concept more generic and use it for other filesystems as well. Let's start by giving the flag a more genric name PF_MEMALLOC_NOFS which is in line with an ex

[f2fs-dev] [DEBUG PATCH 2/2] silent warnings which we cannot do anything about

2016-12-16 Thread Michal Hocko
From: Michal Hocko THIS PATCH IS FOR TESTING ONLY AND NOT MEANT TO HIT LINUS TREE There are some code paths used by all the filesystems which we cannot change to drop the GFP_NOFS, yet they generate a lot of warnings. Provide {disable,enable}_scope_gfp_check to silence those. alloc_page_buffers

[f2fs-dev] [PATCH 2/9 v2] xfs: introduce and use KM_NOLOCKDEP to silence reclaim lockdep false positives

2016-12-16 Thread Michal Hocko
Updated patch after Mike noticed a BUG_ON when KM_NOLOCKDEP is used. --- >From 1497e713e11639157aef21cae29052cb3dc7ab44 Mon Sep 17 00:00:00 2001 From: Michal Hocko Date: Thu, 15 Dec 2016 13:06:43 +0100 Subject: [PATCH] xfs: introduce and use KM_NOLOCKDEP to silence reclaim lockdep fa

[f2fs-dev] [PATCH 1/9] lockdep: allow to disable reclaim lockup detection

2016-12-16 Thread Michal Hocko
From: Michal Hocko The current implementation of the reclaim lockup detection can lead to false positives and those even happen and usually lead to tweak the code to silence the lockdep by using GFP_NOFS even though the context can use __GFP_FS just fine. See http://lkml.kernel.org/r

[f2fs-dev] [PATCH 4/9] mm: introduce memalloc_nofs_{save, restore} API

2016-12-16 Thread Michal Hocko
From: Michal Hocko GFP_NOFS context is used for the following 5 reasons currently - to prevent from deadlocks when the lock held by the allocation context would be needed during the memory reclaim - to prevent from stack overflows during the reclaim because

[f2fs-dev] [PATCH 0/9 v2] scope GFP_NOFS api

2016-12-16 Thread Michal Hocko
deletions(-) Shortlog: Michal Hocko (9): lockdep: allow to disable reclaim lockup detection xfs: introduce and use KM_NOLOCKDEP to silence reclaim lockdep false positives xfs: abstract PF_FSTRANS to PF_MEMALLOC_NOFS mm: introduce memalloc_nofs_{save,restore} API

[f2fs-dev] [DEBUG PATCH 0/2] debug explicit GFP_NO{FS, IO} usage from the scope context

2016-12-16 Thread Michal Hocko
Hi, I've forgot to add the following two patches which should help to identify explicit GFP_NO{FS,IO} usage from withing a scope context. Such a usage can be changed to the full GFP_KERNEL because all the calls from within the NO{FS,IO} scope will drop the __GFP_FS resp. __GFP_IO automatically and

[f2fs-dev] [PATCH 7/9] jbd2: make the whole kjournald2 kthread NOFS safe

2016-12-16 Thread Michal Hocko
From: Michal Hocko kjournald2 is central to the transaction commit processing. As such any potential allocation from this kernel thread has to be GFP_NOFS. Make sure to mark the whole kernel thread GFP_NOFS by the memalloc_nofs_save. Suggested-by: Jan Kara Signed-off-by: Michal Hocko --- fs

Re: [f2fs-dev] [PATCH 2/9 v2] xfs: introduce and use KM_NOLOCKDEP to silence reclaim lockdep false positives

2016-12-16 Thread Michal Hocko
On Fri 16-12-16 11:37:50, Brian Foster wrote: > On Fri, Dec 16, 2016 at 04:40:41PM +0100, Michal Hocko wrote: > > Updated patch after Mike noticed a BUG_ON when KM_NOLOCKDEP is used. > > --- > > From 1497e713e11639157aef21cae29052cb3dc7ab44 Mon Sep 17 00:00:00 2001 > > F

[f2fs-dev] [DEBUG PATCH 1/2] mm, debug: report when GFP_NO{FS, IO} is used explicitly from memalloc_no{fs, io}_{save, restore} context

2016-12-16 Thread Michal Hocko
From: Michal Hocko THIS PATCH IS FOR TESTING ONLY AND NOT MEANT TO HIT LINUS TREE It is desirable to reduce the direct GFP_NO{FS,IO} usage at minimum and prefer scope usage defined by memalloc_no{fs,io}_{save,restore} API. Let's help this process and add a debugging tool to catch wh

[f2fs-dev] [PATCH 5/9 v2] xfs: use memalloc_nofs_{save, restore} instead of memalloc_noio*

2016-12-16 Thread Michal Hocko
On Fri 16-12-16 11:38:11, Brian Foster wrote: > On Thu, Dec 15, 2016 at 03:07:11PM +0100, Michal Hocko wrote: [...] > > @@ -459,7 +459,7 @@ _xfs_buf_map_pages( > > break; > > vm_unmap_aliases(); > >

[f2fs-dev] [PATCH 9/9] Revert "ext4: fix wrong gfp type under transaction"

2016-12-16 Thread Michal Hocko
From: Michal Hocko This reverts commit 216553c4b7f3e3e2beb4981cddca9b2027523928. Now that the transaction context uses memalloc_nofs_save and all allocations within the this context inherit GFP_NOFS automatically, there is no reason to mark specific allocations explicitly. This patch should not

Re: [f2fs-dev] [RFC PATCH] mm: introduce kv[mz]alloc helpers

2016-12-20 Thread Michal Hocko
On Wed 14-12-16 09:59:16, Michal Hocko wrote: > On Tue 13-12-16 14:07:33, Joe Perches wrote: > > On Tue, 2016-12-13 at 11:14 +0100, Michal Hocko wrote: > > > Are there any more comments or objections to this patch? Is this a good > > > start or kv[mz]alloc has to prov

Re: [f2fs-dev] [PATCH 2/9] xfs: introduce and use KM_NOLOCKDEP to silence reclaim lockdep false positives

2016-12-20 Thread Michal Hocko
On Tue 20-12-16 08:24:13, Dave Chinner wrote: > On Thu, Dec 15, 2016 at 03:07:08PM +0100, Michal Hocko wrote: > > From: Michal Hocko > > > > Now that the page allocator offers __GFP_NOLOCKDEP let's introduce > > KM_NOLOCKDEP alias for the xfs allocation APIs. Wh

Re: [f2fs-dev] [RFC PATCH] mm: introduce kv[mz]alloc helpers

2016-12-30 Thread Michal Hocko
It seems that this email didn't get delivered due to some stupid gmail spam policy. Let me try to repost via a different relay. Sorry to those who have seen the original message and get a duplicate now. On Wed 21-12-16 08:03:53, Michal Hocko wrote: > On Tue 20-12-16 14:13:41, Andrew Mort

Re: [f2fs-dev] [PATCH 0/9 v2] scope GFP_NOFS api

2016-12-30 Thread Michal Hocko
this API. Can we find a way to use it? -- Michal Hocko SUSE Labs -- Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/sla

Re: [f2fs-dev] [PATCH] mm: introduce kv[mz]alloc helpers

2017-01-05 Thread Michal Hocko
On Tue 03-01-17 11:23:04, Vlastimil Babka wrote: > On 01/02/2017 02:37 PM, Michal Hocko wrote: > > From: Michal Hocko > > > > Using kmalloc with the vmalloc fallback for larger allocations is a > > common pattern in the kernel code. Yet we do not have any common helpe

Re: [f2fs-dev] [PATCH] mm: introduce kv[mz]alloc helpers

2017-01-05 Thread Michal Hocko
On Mon 02-01-17 07:55:22, Joe Perches wrote: > On Mon, 2017-01-02 at 14:37 +0100, Michal Hocko wrote: > > From: Michal Hocko > > > > Using kmalloc with the vmalloc fallback for larger allocations is a > > common pattern in the kernel code. Yet we do not have any common

[f2fs-dev] [PATCH] mm: introduce kv[mz]alloc helpers

2017-01-05 Thread Michal Hocko
From: Michal Hocko Using kmalloc with the vmalloc fallback for larger allocations is a common pattern in the kernel code. Yet we do not have any common helper for that and so users have invented their own helpers. Some of them are really creative when doing so. Let's just add kv[mz]allo

[f2fs-dev] [PATCH] mm: support __GFP_REPEAT in kvmalloc_node

2017-01-05 Thread Michal Hocko
about potential split ups or cleanups few more days and then repost the whole series. [1] http://lkml.kernel.org/r/20170104150800.go25...@dhcp22.suse.cz --- >From 0b92e4d2e040524b878d4e7b9ee88fbad5284b33 Mon Sep 17 00:00:00 2001 From: Michal Hocko Date: Wed, 4 Jan 2017 18:01:39 +0100 Subject: [P

Re: [f2fs-dev] [PATCH] mm: introduce kv[mz]alloc helpers

2017-01-05 Thread Michal Hocko
>From 72a8493c4f692fca8980b59e0a79d09041164203 Mon Sep 17 00:00:00 2001 From: Michal Hocko Date: Wed, 4 Jan 2017 13:30:32 +0100 Subject: [PATCH] treewide: use kv[mz]alloc* rather than opencoded variants There are many code paths opencoding kvmalloc. Let's use the helper instead. This shouldn&#x

[f2fs-dev] [PATCH 7/8] Revert "ext4: avoid deadlocks in the writeback path by using sb_getblk_gfp"

2017-01-06 Thread Michal Hocko
From: Michal Hocko This reverts commit c45653c341f5c8a0ce19c8f0ad4678640849cb86 because sb_getblk_gfp is not really needed as sb_getblk __getblk_gfp __getblk_slow grow_buffers grow_dev_page gfp_mask = mapping_gfp_constraint(inode->i_mapping, ~__GFP_FS) | gfp

Re: [f2fs-dev] [PATCH] mm: support __GFP_REPEAT in kvmalloc_node

2017-01-06 Thread Michal Hocko
On Fri 06-01-17 13:09:36, Vlastimil Babka wrote: > On 01/04/2017 07:12 PM, Michal Hocko wrote: [...] > > diff --git a/mm/util.c b/mm/util.c > > index 8e4ea6cbe379..a2bfb85e60e5 100644 > > --- a/mm/util.c > > +++ b/mm/util.c > > @@ -348,8 +348,13 @@ void *kvmalloc_n

[f2fs-dev] [PATCH 3/8] mm: introduce memalloc_nofs_{save, restore} API

2017-01-06 Thread Michal Hocko
From: Michal Hocko GFP_NOFS context is used for the following 5 reasons currently - to prevent from deadlocks when the lock held by the allocation context would be needed during the memory reclaim - to prevent from stack overflows during the reclaim because

[f2fs-dev] [DEBUG PATCH 1/2] mm, debug: report when GFP_NO{FS, IO} is used explicitly from memalloc_no{fs, io}_{save, restore} context

2017-01-06 Thread Michal Hocko
From: Michal Hocko THIS PATCH IS FOR TESTING ONLY AND NOT MEANT TO HIT LINUS TREE It is desirable to reduce the direct GFP_NO{FS,IO} usage at minimum and prefer scope usage defined by memalloc_no{fs,io}_{save,restore} API. Let's help this process and add a debugging tool to catch wh

[f2fs-dev] [PATCH 4/8] xfs: use memalloc_nofs_{save, restore} instead of memalloc_noio*

2017-01-06 Thread Michal Hocko
From: Michal Hocko kmem_zalloc_large and _xfs_buf_map_pages use memalloc_noio_{save,restore} API to prevent from reclaim recursion into the fs because vmalloc can invoke unconditional GFP_KERNEL allocations and these functions might be called from the NOFS contexts. The memalloc_noio_save will

Re: [f2fs-dev] [PATCH] mm: introduce kv[mz]alloc helpers

2017-01-06 Thread Michal Hocko
On Fri 06-01-17 14:29:33, Vlastimil Babka wrote: > On 01/02/2017 02:37 PM, Michal Hocko wrote: > > --- a/drivers/vhost/vhost.c > > +++ b/drivers/vhost/vhost.c > > @@ -514,18 +514,9 @@ long vhost_dev_set_owner(struct vhost_dev *dev) > > } > > EXPO

[f2fs-dev] [PATCH 6/8] jbd2: make the whole kjournald2 kthread NOFS safe

2017-01-06 Thread Michal Hocko
From: Michal Hocko kjournald2 is central to the transaction commit processing. As such any potential allocation from this kernel thread has to be GFP_NOFS. Make sure to mark the whole kernel thread GFP_NOFS by the memalloc_nofs_save. Suggested-by: Jan Kara Signed-off-by: Michal Hocko Reviewed

[f2fs-dev] [DEBUG PATCH 0/2] debug explicit GFP_NO{FS, IO} usage from the scope context

2017-01-06 Thread Michal Hocko
These two patches should help to identify explicit GFP_NO{FS,IO} usage from withing a scope context and reduce such a usage as a result. Such a usage can be changed to the full GFP_KERNEL because all the calls from within the NO{FS,IO} scope will drop the __GFP_FS resp. __GFP_IO automatically and i

[f2fs-dev] [PATCH 1/8] lockdep: allow to disable reclaim lockup detection

2017-01-06 Thread Michal Hocko
From: Michal Hocko The current implementation of the reclaim lockup detection can lead to false positives and those even happen and usually lead to tweak the code to silence the lockdep by using GFP_NOFS even though the context can use __GFP_FS just fine. See http://lkml.kernel.org/r

[f2fs-dev] [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-06 Thread Michal Hocko
From: Michal Hocko This reverts commit 216553c4b7f3e3e2beb4981cddca9b2027523928. Now that the transaction context uses memalloc_nofs_save and all allocations within the this context inherit GFP_NOFS automatically, there is no reason to mark specific allocations explicitly. This patch should not

[f2fs-dev] [PATCH 5/8] jbd2: mark the transaction context with the scope GFP_NOFS context

2017-01-06 Thread Michal Hocko
From: Michal Hocko now that we have memalloc_nofs_{save,restore} api we can mark the whole transaction context as implicitly GFP_NOFS. All allocations will automatically inherit GFP_NOFS this way. This means that we do not have to mark any of those requests with GFP_NOFS and moreover all the

[f2fs-dev] [PATCH 2/8] xfs: abstract PF_FSTRANS to PF_MEMALLOC_NOFS

2017-01-06 Thread Michal Hocko
From: Michal Hocko xfs has defined PF_FSTRANS to declare a scope GFP_NOFS semantic quite some time ago. We would like to make this concept more generic and use it for other filesystems as well. Let's start by giving the flag a more generic name PF_MEMALLOC_NOFS which is in line with an ex

[f2fs-dev] [PATCH 0/8 v3] scope GFP_NOFS api

2017-01-06 Thread Michal Hocko
de/linux/sched.h | 32 ++-- kernel/locking/lockdep.c | 6 +- lib/radix-tree.c | 2 ++ mm/page_alloc.c | 8 +--- mm/vmscan.c | 6 +++--- 19 files changed, 109 insertions(+), 45 deletions(-) Shortlog: Michal Hocko (8): lockdep: allow to disable reclaim l

[f2fs-dev] [DEBUG PATCH 2/2] silent warnings which we cannot do anything about

2017-01-06 Thread Michal Hocko
From: Michal Hocko THIS PATCH IS FOR TESTING ONLY AND NOT MEANT TO HIT LINUS TREE There are some code paths used by all the filesystems which we cannot change to drop the GFP_NOFS, yet they generate a lot of warnings. Provide {disable,enable}_scope_gfp_check to silence those. alloc_page_buffers

Re: [f2fs-dev] [PATCH] mm: introduce kv[mz]alloc helpers

2017-01-06 Thread Michal Hocko
On Fri 06-01-17 15:36:04, Vlastimil Babka wrote: > On 01/04/2017 03:20 PM, Michal Hocko wrote: > > diff --git a/net/netfilter/x_tables.c b/net/netfilter/x_tables.c > > index 2ff499680cc6..0a5cc1237afe 100644 > > --- a/net/netfilter/x_tables.c > > +++ b/net/netfilte

Re: [f2fs-dev] [PATCH 2/8] xfs: abstract PF_FSTRANS to PF_MEMALLOC_NOFS

2017-01-09 Thread Michal Hocko
On Mon 09-01-17 13:59:05, Vlastimil Babka wrote: > On 01/06/2017 03:11 PM, Michal Hocko wrote: > > From: Michal Hocko > > > > xfs has defined PF_FSTRANS to declare a scope GFP_NOFS semantic quite > > some time ago. We would like to make this concept more gener

Re: [f2fs-dev] [PATCH 3/8] mm: introduce memalloc_nofs_{save, restore} API

2017-01-09 Thread Michal Hocko
p_zone(gfp_mask); struct scan_control sc = { .nr_to_reclaim = max(nr_pages, SWAP_CLUSTER_MAX), - .gfp_mask = (gfp_mask = current_gfp_context(gfp_mask)), + .gfp_mask = gfp_mask, .order =

Re: [f2fs-dev] [PATCH 3/8] mm: introduce memalloc_nofs_{save, restore} API

2017-01-09 Thread Michal Hocko
On Mon 09-01-17 14:42:10, Michal Hocko wrote: > On Mon 09-01-17 14:04:21, Vlastimil Babka wrote: [...] > Now that you have opened this I have noticed that the code is wrong > here because GFP_HIGHUSER_MOVABLE & ~GFP_RECLAIM_MASK would overwrite > the removed GFP_FS. Blee, it

Re: [f2fs-dev] [PATCH 4/8] xfs: use memalloc_nofs_{save, restore} instead of memalloc_noio*

2017-01-09 Thread Michal Hocko
On Mon 09-01-17 15:08:27, Vlastimil Babka wrote: > On 01/06/2017 03:11 PM, Michal Hocko wrote: > > From: Michal Hocko > > > > kmem_zalloc_large and _xfs_buf_map_pages use memalloc_noio_{save,restore} > > API to prevent from reclaim recursion into the fs bec

Re: [f2fs-dev] [PATCH] mm: support __GFP_REPEAT in kvmalloc_node

2017-01-09 Thread Michal Hocko
On Fri 06-01-17 13:09:36, Vlastimil Babka wrote: > On 01/04/2017 07:12 PM, Michal Hocko wrote: > > While checking opencoded users I've encountered that vhost code would > > really like to use kvmalloc with __GFP_REPEAT [1] so the following patch > > adds support for __GF

Re: [f2fs-dev] [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-19 Thread Michal Hocko
On Tue 17-01-17 10:59:16, Theodore Ts'o wrote: > On Tue, Jan 17, 2017 at 04:18:17PM +0100, Michal Hocko wrote: > > > > OK, so I've been staring into the code and AFAIU current->journal_info > > can contain my stored information. I could either hijack part of t

Re: [f2fs-dev] [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-19 Thread Michal Hocko
On Tue 17-01-17 09:24:25, Michal Hocko wrote: > On Mon 16-01-17 21:56:07, Theodore Ts'o wrote: > > On Fri, Jan 06, 2017 at 03:11:07PM +0100, Michal Hocko wrote: > > > From: Michal Hocko > > > > > > This reverts commit 216553c4b7f3e3e2beb4981cddca9b2

Re: [f2fs-dev] [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-19 Thread Michal Hocko
On Tue 17-01-17 18:29:25, Jan Kara wrote: > On Tue 17-01-17 17:16:19, Michal Hocko wrote: > > > > But before going to play with that I am really wondering whether we need > > > > all this with no journal at all. AFAIU what Jack told me it is the > > > > journ

Re: [f2fs-dev] [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-19 Thread Michal Hocko
On Thu 19-01-17 10:22:36, Jan Kara wrote: > On Thu 19-01-17 09:39:56, Michal Hocko wrote: > > On Tue 17-01-17 18:29:25, Jan Kara wrote: > > > On Tue 17-01-17 17:16:19, Michal Hocko wrote: > > > > > > But before going to play with that I am really wo

Re: [f2fs-dev] [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-19 Thread Michal Hocko
On Mon 16-01-17 21:56:07, Theodore Ts'o wrote: > On Fri, Jan 06, 2017 at 03:11:07PM +0100, Michal Hocko wrote: > > From: Michal Hocko > > > > This reverts commit 216553c4b7f3e3e2beb4981cddca9b2027523928. Now that > > the transaction context uses memalloc_nofs_sav

Re: [f2fs-dev] [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-19 Thread Michal Hocko
On Tue 17-01-17 14:04:03, Andreas Dilger wrote: > On Jan 17, 2017, at 8:59 AM, Theodore Ts'o wrote: > > > > On Tue, Jan 17, 2017 at 04:18:17PM +0100, Michal Hocko wrote: > >> > >> OK, so I've been staring into the code and AFAIU current->journal_

Re: [f2fs-dev] [PATCH 7/8] Revert "ext4: avoid deadlocks in the writeback path by using sb_getblk_gfp"

2017-01-19 Thread Michal Hocko
On Mon 16-01-17 22:01:18, Theodore Ts'o wrote: > On Fri, Jan 06, 2017 at 03:11:06PM +0100, Michal Hocko wrote: > > From: Michal Hocko > > > > This reverts commit c45653c341f5c8a0ce19c8f0ad4678640849cb86 because > > sb_getblk_gfp is not really needed as

Re: [f2fs-dev] [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-27 Thread Michal Hocko
On Fri 27-01-17 01:13:18, Theodore Ts'o wrote: > On Thu, Jan 26, 2017 at 08:44:55AM +0100, Michal Hocko wrote: > > > > I'm convinced the current series is OK, only real life will tell us > > > > whether > > > > we missed something or not ;) >

  1   2   >