Re: [Cluster-devel] vmalloc with GFP_NOFS

2018-07-17 Thread Michal Hocko
On Tue 24-04-18 14:09:27, Michal Hocko wrote: > On Tue 24-04-18 20:26:23, Steven Whitehouse wrote: > [...] > > It would be good to fix this, and it has been known as an issue for a long > > time. We might well be able to make use of the new API though. It might be > > as si

Re: [Cluster-devel] vmalloc with GFP_NOFS

2018-07-17 Thread Michal Hocko
On Tue 24-04-18 14:35:36, Theodore Ts'o wrote: > On Tue, Apr 24, 2018 at 10:27:12AM -0600, Michal Hocko wrote: > > fs/ext4/xattr.c > > > > What to do about this? Well, there are two things. Firstly, it would be > > really great to double check whether the GFP_NOFS is

Re: [Cluster-devel] vmalloc with GFP_NOFS

2018-07-17 Thread Michal Hocko
On Tue 24-04-18 21:03:43, Richard Weinberger wrote: > Am Dienstag, 24. April 2018, 18:27:12 CEST schrieb Michal Hocko: > > fs/ubifs/debug.c > > This one is just for debugging. > So, preallocating + locking would not hurt much. > > > fs/ubifs/lprops.c > > Di

Re: [Cluster-devel] vmalloc with GFP_NOFS

2018-05-10 Thread Michal Hocko
On Thu 10-05-18 07:58:25, Michal Hocko wrote: > On Wed 09-05-18 15:02:31, Darrick J. Wong wrote: > > On Wed, May 09, 2018 at 11:04:47PM +0200, Michal Hocko wrote: > > > On Wed 09-05-18 08:13:51, Darrick J. Wong wrote: > [...] > > > > > FS resp. IO submittin

Re: [Cluster-devel] vmalloc with GFP_NOFS

2018-05-10 Thread Michal Hocko
On Wed 09-05-18 15:02:31, Darrick J. Wong wrote: > On Wed, May 09, 2018 at 11:04:47PM +0200, Michal Hocko wrote: > > On Wed 09-05-18 08:13:51, Darrick J. Wong wrote: [...] > > > > FS resp. IO submitting code paths have to be careful when allocating > > > > >

Re: [Cluster-devel] vmalloc with GFP_NOFS

2018-05-09 Thread Michal Hocko
On Wed 09-05-18 19:24:51, Mike Rapoport wrote: > On Wed, May 09, 2018 at 08:13:51AM -0700, Darrick J. Wong wrote: > > On Wed, May 09, 2018 at 03:42:22PM +0200, Michal Hocko wrote: [...] > > > FS/IO code then simply calls the appropriate save function right at > > > t

Re: [Cluster-devel] vmalloc with GFP_NOFS

2018-05-09 Thread Michal Hocko
On Wed 09-05-18 08:13:51, Darrick J. Wong wrote: > On Wed, May 09, 2018 at 03:42:22PM +0200, Michal Hocko wrote: > > On Tue 24-04-18 13:25:42, Michal Hocko wrote: > > [...] > > > > As a suggestion, could you take > > > > documentation about how to con

Re: [Cluster-devel] vmalloc with GFP_NOFS

2018-05-09 Thread Michal Hocko
On Tue 24-04-18 13:25:42, Michal Hocko wrote: [...] > > As a suggestion, could you take > > documentation about how to convert to the memalloc_nofs_{save,restore} > > scope api (which I think you've written about e-mails at length > > before), and put that into a file i

Re: [Cluster-devel] vmalloc with GFP_NOFS

2018-04-25 Thread Michal Hocko
On Wed 25-04-18 11:25:09, Mikulas Patocka wrote: > > > On Wed, 25 Apr 2018, Michal Hocko wrote: > > > On Wed 25-04-18 08:43:32, Mikulas Patocka wrote: > > > > > > > > > On Tue, 24 Apr 2018, Michal Hocko wrote: > > > &

Re: [Cluster-devel] vmalloc with GFP_NOFS

2018-04-25 Thread Michal Hocko
On Wed 25-04-18 08:43:32, Mikulas Patocka wrote: > > > On Tue, 24 Apr 2018, Michal Hocko wrote: > > > On Tue 24-04-18 19:17:12, Mikulas Patocka wrote: > > > > > > > > > On Tue, 24 Apr 2018, Michal Hocko wrote: > > > > > > >

Re: [Cluster-devel] vmalloc with GFP_NOFS

2018-04-24 Thread Michal Hocko
On Tue 24-04-18 19:17:12, Mikulas Patocka wrote: > > > On Tue, 24 Apr 2018, Michal Hocko wrote: > > > On Wed 25-04-18 00:18:40, Richard Weinberger wrote: > > > Am Dienstag, 24. April 2018, 21:28:03 CEST schrieb Michal Hocko: > > > > > Also only for d

Re: [Cluster-devel] vmalloc with GFP_NOFS

2018-04-24 Thread Michal Hocko
On Wed 25-04-18 00:18:40, Richard Weinberger wrote: > Am Dienstag, 24. April 2018, 21:28:03 CEST schrieb Michal Hocko: > > > Also only for debugging. > > > Getting rid of vmalloc with GFP_NOFS in UBIFS is no big problem. > > > I can prepare a patch. > > >

Re: [Cluster-devel] vmalloc with GFP_NOFS

2018-04-24 Thread Michal Hocko
ve to > check the code to be sure, Yeah, starting with annotating those locking contexts and how document how their are used in the reclaim is the great first step. This has to be done per-fs obviously. -- Michal Hocko SUSE Labs

Re: [Cluster-devel] vmalloc with GFP_NOFS

2018-04-24 Thread Michal Hocko
On Tue 24-04-18 14:35:36, Theodore Ts'o wrote: > On Tue, Apr 24, 2018 at 10:27:12AM -0600, Michal Hocko wrote: > > fs/ext4/xattr.c > > > > What to do about this? Well, there are two things. Firstly, it would be > > really great to double check whether the GFP_NOFS is

Re: [Cluster-devel] vmalloc with GFP_NOFS

2018-04-24 Thread Michal Hocko
On Tue 24-04-18 21:03:43, Richard Weinberger wrote: > Am Dienstag, 24. April 2018, 18:27:12 CEST schrieb Michal Hocko: > > fs/ubifs/debug.c > > This one is just for debugging. > So, preallocating + locking would not hurt much. > > > fs/ubifs/lprops.c > > Di

Re: [Cluster-devel] vmalloc with GFP_NOFS

2018-04-24 Thread Michal Hocko
On Tue 24-04-18 12:46:55, Mikulas Patocka wrote: > > > On Tue, 24 Apr 2018, Michal Hocko wrote: > > > Hi, > > it seems that we still have few vmalloc users who perform GFP_NOFS > > allocation: > > drivers/mtd/ubi/io.c > > fs/ext4/xattr.c > &

[Cluster-devel] vmalloc with GFP_NOFS

2018-04-24 Thread Michal Hocko
to live. Please do not hesitate to get back to me if something is not clear. Thanks! -- Michal Hocko SUSE Labs

Re: [Cluster-devel] [PATCH 4/7] mm: introduce memalloc_nofs_{save, restore} API

2017-03-07 Thread Michal Hocko
On Mon 06-03-17 13:22:14, Andrew Morton wrote: > On Mon, 6 Mar 2017 14:14:05 +0100 Michal Hocko <mho...@kernel.org> wrote: [...] > > --- a/include/linux/gfp.h > > +++ b/include/linux/gfp.h > > @@ -210,8 +210,16 @@ struct vm_area_struct; > > * > > * GFP_

[Cluster-devel] [PATCH 7/7] jbd2: make the whole kjournald2 kthread NOFS safe

2017-03-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> kjournald2 is central to the transaction commit processing. As such any potential allocation from this kernel thread has to be GFP_NOFS. Make sure to mark the whole kernel thread GFP_NOFS by the memalloc_nofs_save. Suggested-by: Jan Kara <j..

[Cluster-devel] [PATCH 2/7] lockdep: allow to disable reclaim lockup detection

2017-03-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> The current implementation of the reclaim lockup detection can lead to false positives and those even happen and usually lead to tweak the code to silence the lockdep by using GFP_NOFS even though the context can use __GFP_FS just fine. Se

[Cluster-devel] [PATCH 0/7 v5] scope GFP_NOFS api

2017-03-06 Thread Michal Hocko
6 files changed, 106 insertions(+), 37 deletions(-) Shortlog: Michal Hocko (6): lockdep: allow to disable reclaim lockup detection xfs: abstract PF_FSTRANS to PF_MEMALLOC_NOFS mm: introduce memalloc_nofs_{save,restore} API xfs: use memalloc_nofs_{save,restore} instead of memalloc_no

[Cluster-devel] [PATCH 4/7] mm: introduce memalloc_nofs_{save, restore} API

2017-03-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> GFP_NOFS context is used for the following 5 reasons currently - to prevent from deadlocks when the lock held by the allocation context would be needed during the memory reclaim - to prevent from stack overflows during the r

[Cluster-devel] [PATCH 6/7] jbd2: mark the transaction context with the scope GFP_NOFS context

2017-03-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> now that we have memalloc_nofs_{save,restore} api we can mark the whole transaction context as implicitly GFP_NOFS. All allocations will automatically inherit GFP_NOFS this way. This means that we do not have to mark any of those requests with GF

[Cluster-devel] [PATCH 1/7] lockdep: teach lockdep about memalloc_noio_save

2017-03-06 Thread Michal Hocko
[ 644.174012] SyS_lsetxattr+0x11/0x20 [ 644.174012] entry_SYSCALL_64_fastpath+0x23/0xc6 Let's fix this by making lockdep explicitly do the shaving of respective GFP flags. Fixes: 934f3072c17c ("mm: clear __GFP_FS when PF_MEMALLOC_NOIO is set") Acked-by: Michal Hocko <mho..

[Cluster-devel] [PATCH 5/7] xfs: use memalloc_nofs_{save, restore} instead of memalloc_noio*

2017-03-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> kmem_zalloc_large and _xfs_buf_map_pages use memalloc_noio_{save,restore} API to prevent from reclaim recursion into the fs because vmalloc can invoke unconditional GFP_KERNEL allocations and these functions might be called from the NOFS co

Re: [Cluster-devel] [PATCH 7/8] Revert "ext4: avoid deadlocks in the writeback path by using sb_getblk_gfp"

2017-03-06 Thread Michal Hocko
On Tue 17-01-17 08:54:50, Michal Hocko wrote: > On Mon 16-01-17 22:01:18, Theodore Ts'o wrote: > > On Fri, Jan 06, 2017 at 03:11:06PM +0100, Michal Hocko wrote: > > > From: Michal Hocko <mho...@suse.com> > > > > > > This reverts commit c45653

Re: [Cluster-devel] [PATCH 1/6] lockdep: allow to disable reclaim lockup detection

2017-02-06 Thread Michal Hocko
On Mon 06-02-17 07:24:00, Matthew Wilcox wrote: > On Mon, Feb 06, 2017 at 03:34:50PM +0100, Michal Hocko wrote: > > This part is not needed for the patch, strictly speaking but I wanted to > > make the code more future proof. > > Understood. I took an extra bit myself for m

Re: [Cluster-devel] [PATCH 1/6] lockdep: allow to disable reclaim lockup detection

2017-02-06 Thread Michal Hocko
On Mon 06-02-17 06:26:41, Matthew Wilcox wrote: > On Mon, Feb 06, 2017 at 03:07:13PM +0100, Michal Hocko wrote: > > While we are at it also make sure that the radix tree doesn't > > accidentaly override tags stored in the upper part of the gfp_mask. > > > diff --git a/lib/

[Cluster-devel] [PATCH 6/6] jbd2: make the whole kjournald2 kthread NOFS safe

2017-02-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> kjournald2 is central to the transaction commit processing. As such any potential allocation from this kernel thread has to be GFP_NOFS. Make sure to mark the whole kernel thread GFP_NOFS by the memalloc_nofs_save. Suggested-by: Jan Kara <j..

[Cluster-devel] [PATCH 3/6] mm: introduce memalloc_nofs_{save, restore} API

2017-02-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> GFP_NOFS context is used for the following 5 reasons currently - to prevent from deadlocks when the lock held by the allocation context would be needed during the memory reclaim - to prevent from stack overflows during the r

[Cluster-devel] [PATCH 4/6] xfs: use memalloc_nofs_{save, restore} instead of memalloc_noio*

2017-02-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> kmem_zalloc_large and _xfs_buf_map_pages use memalloc_noio_{save,restore} API to prevent from reclaim recursion into the fs because vmalloc can invoke unconditional GFP_KERNEL allocations and these functions might be called from the NOFS co

[Cluster-devel] [PATCH 2/6] xfs: abstract PF_FSTRANS to PF_MEMALLOC_NOFS

2017-02-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> xfs has defined PF_FSTRANS to declare a scope GFP_NOFS semantic quite some time ago. We would like to make this concept more generic and use it for other filesystems as well. Let's start by giving the flag a more generic name PF_MEMALLOC_NOFS which is i

[Cluster-devel] [PATCH 1/6] lockdep: allow to disable reclaim lockup detection

2017-02-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> The current implementation of the reclaim lockup detection can lead to false positives and those even happen and usually lead to tweak the code to silence the lockdep by using GFP_NOFS even though the context can use __GFP_FS just fine. Se

[Cluster-devel] [PATCH 0/6 v4] scope GFP_NOFS api

2017-02-06 Thread Michal Hocko
| 10 ++ mm/vmscan.c | 6 +++--- 15 files changed, 100 insertions(+), 36 deletions(-) Shortlog: Michal Hocko (6): lockdep: allow to disable reclaim lockup detection xfs: abstract PF_FSTRANS to PF_MEMALLOC_NOFS mm: introduce memalloc_nofs_{save,restore

[Cluster-devel] [PATCH 5/6] jbd2: mark the transaction context with the scope GFP_NOFS context

2017-02-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> now that we have memalloc_nofs_{save,restore} api we can mark the whole transaction context as implicitly GFP_NOFS. All allocations will automatically inherit GFP_NOFS this way. This means that we do not have to mark any of those requests with GF

Re: [Cluster-devel] [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-02-03 Thread Michal Hocko
On Mon 30-01-17 09:12:10, Michal Hocko wrote: > On Fri 27-01-17 11:40:42, Theodore Ts'o wrote: > > On Fri, Jan 27, 2017 at 10:37:35AM +0100, Michal Hocko wrote: > > > If this ever turn out to be a problem and with the vmapped stacks we > > > have good chances t

Re: [Cluster-devel] [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-30 Thread Michal Hocko
On Fri 27-01-17 11:40:42, Theodore Ts'o wrote: > On Fri, Jan 27, 2017 at 10:37:35AM +0100, Michal Hocko wrote: > > If this ever turn out to be a problem and with the vmapped stacks we > > have good chances to get a proper stack traces on a potential overflow > > we can add

Re: [Cluster-devel] [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-25 Thread Michal Hocko
On Thu 19-01-17 10:44:05, Michal Hocko wrote: > On Thu 19-01-17 10:22:36, Jan Kara wrote: > > On Thu 19-01-17 09:39:56, Michal Hocko wrote: > > > On Tue 17-01-17 18:29:25, Jan Kara wrote: > > > > On Tue 17-01-17 17:16:19, Michal Hocko wrote: > > > > >

Re: [Cluster-devel] [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-19 Thread Michal Hocko
On Thu 19-01-17 10:22:36, Jan Kara wrote: > On Thu 19-01-17 09:39:56, Michal Hocko wrote: > > On Tue 17-01-17 18:29:25, Jan Kara wrote: > > > On Tue 17-01-17 17:16:19, Michal Hocko wrote: > > > > > > But before going to play with that I am really wo

Re: [Cluster-devel] [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-19 Thread Michal Hocko
On Tue 17-01-17 18:29:25, Jan Kara wrote: > On Tue 17-01-17 17:16:19, Michal Hocko wrote: > > > > But before going to play with that I am really wondering whether we need > > > > all this with no journal at all. AFAIU what Jack told me it is the > > > > journ

Re: [Cluster-devel] [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-18 Thread Michal Hocko
On Tue 17-01-17 14:04:03, Andreas Dilger wrote: > On Jan 17, 2017, at 8:59 AM, Theodore Ts'o <ty...@mit.edu> wrote: > > > > On Tue, Jan 17, 2017 at 04:18:17PM +0100, Michal Hocko wrote: > >> > >> OK, so I've been staring into the code and AFAIU current-&

Re: [Cluster-devel] [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-17 Thread Michal Hocko
On Tue 17-01-17 10:59:16, Theodore Ts'o wrote: > On Tue, Jan 17, 2017 at 04:18:17PM +0100, Michal Hocko wrote: > > > > OK, so I've been staring into the code and AFAIU current->journal_info > > can contain my stored information. I could either hijack part of the >

Re: [Cluster-devel] [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-17 Thread Michal Hocko
On Tue 17-01-17 09:24:25, Michal Hocko wrote: > On Mon 16-01-17 21:56:07, Theodore Ts'o wrote: > > On Fri, Jan 06, 2017 at 03:11:07PM +0100, Michal Hocko wrote: > > > From: Michal Hocko <mho...@suse.com> > > > > > > This reverts commit 216

Re: [Cluster-devel] [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-17 Thread Michal Hocko
On Mon 16-01-17 21:56:07, Theodore Ts'o wrote: > On Fri, Jan 06, 2017 at 03:11:07PM +0100, Michal Hocko wrote: > > From: Michal Hocko <mho...@suse.com> > > > > This reverts commit 216553c4b7f3e3e2beb4981cddca9b2027523928. Now that > > the transaction contex

Re: [Cluster-devel] [PATCH 7/8] Revert "ext4: avoid deadlocks in the writeback path by using sb_getblk_gfp"

2017-01-16 Thread Michal Hocko
On Mon 16-01-17 22:01:18, Theodore Ts'o wrote: > On Fri, Jan 06, 2017 at 03:11:06PM +0100, Michal Hocko wrote: > > From: Michal Hocko <mho...@suse.com> > > > > This reverts commit c45653c341f5c8a0ce19c8f0ad4678640849cb86 because > > sb_getblk_gfp is n

Re: [Cluster-devel] [PATCH 4/8] xfs: use memalloc_nofs_{save, restore} instead of memalloc_noio*

2017-01-09 Thread Michal Hocko
On Mon 09-01-17 15:08:27, Vlastimil Babka wrote: > On 01/06/2017 03:11 PM, Michal Hocko wrote: > > From: Michal Hocko <mho...@suse.com> > > > > kmem_zalloc_large and _xfs_buf_map_pages use memalloc_noio_{save,restore} > > API to prevent from reclaim recursio

Re: [Cluster-devel] [PATCH 2/8] xfs: abstract PF_FSTRANS to PF_MEMALLOC_NOFS

2017-01-09 Thread Michal Hocko
On Mon 09-01-17 13:59:05, Vlastimil Babka wrote: > On 01/06/2017 03:11 PM, Michal Hocko wrote: > > From: Michal Hocko <mho...@suse.com> > > > > xfs has defined PF_FSTRANS to declare a scope GFP_NOFS semantic quite > > some time ago. We would like to make

Re: [Cluster-devel] [PATCH 3/8] mm: introduce memalloc_nofs_{save, restore} API

2017-01-09 Thread Michal Hocko
On Mon 09-01-17 14:42:10, Michal Hocko wrote: > On Mon 09-01-17 14:04:21, Vlastimil Babka wrote: [...] > Now that you have opened this I have noticed that the code is wrong > here because GFP_HIGHUSER_MOVABLE & ~GFP_RECLAIM_MASK would overwrite > the removed GFP_FS. Blee, it

[Cluster-devel] [DEBUG PATCH 1/2] mm, debug: report when GFP_NO{FS, IO} is used explicitly from memalloc_no{fs, io}_{save, restore} context

2017-01-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> THIS PATCH IS FOR TESTING ONLY AND NOT MEANT TO HIT LINUS TREE It is desirable to reduce the direct GFP_NO{FS,IO} usage at minimum and prefer scope usage defined by memalloc_no{fs,io}_{save,restore} API. Let's help this process and add a debuggin

[Cluster-devel] [DEBUG PATCH 2/2] silent warnings which we cannot do anything about

2017-01-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> THIS PATCH IS FOR TESTING ONLY AND NOT MEANT TO HIT LINUS TREE There are some code paths used by all the filesystems which we cannot change to drop the GFP_NOFS, yet they generate a lot of warnings. Provide {disable,enable}_scope_gfp_check to silence

[Cluster-devel] [DEBUG PATCH 0/2] debug explicit GFP_NO{FS, IO} usage from the scope context

2017-01-06 Thread Michal Hocko
These two patches should help to identify explicit GFP_NO{FS,IO} usage from withing a scope context and reduce such a usage as a result. Such a usage can be changed to the full GFP_KERNEL because all the calls from within the NO{FS,IO} scope will drop the __GFP_FS resp. __GFP_IO automatically and

[Cluster-devel] [PATCH 5/8] jbd2: mark the transaction context with the scope GFP_NOFS context

2017-01-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> now that we have memalloc_nofs_{save,restore} api we can mark the whole transaction context as implicitly GFP_NOFS. All allocations will automatically inherit GFP_NOFS this way. This means that we do not have to mark any of those requests with GF

[Cluster-devel] [PATCH 2/8] xfs: abstract PF_FSTRANS to PF_MEMALLOC_NOFS

2017-01-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> xfs has defined PF_FSTRANS to declare a scope GFP_NOFS semantic quite some time ago. We would like to make this concept more generic and use it for other filesystems as well. Let's start by giving the flag a more generic name PF_MEMALLOC_NOFS which is i

[Cluster-devel] [PATCH 7/8] Revert "ext4: avoid deadlocks in the writeback path by using sb_getblk_gfp"

2017-01-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> This reverts commit c45653c341f5c8a0ce19c8f0ad4678640849cb86 because sb_getblk_gfp is not really needed as sb_getblk __getblk_gfp __getblk_slow grow_buffers grow_dev_page gfp_mask = mapping_gfp_constraint(inode->

[Cluster-devel] [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> This reverts commit 216553c4b7f3e3e2beb4981cddca9b2027523928. Now that the transaction context uses memalloc_nofs_save and all allocations within the this context inherit GFP_NOFS automatically, there is no reason to mark specific allocations expl

[Cluster-devel] [PATCH 0/8 v3] scope GFP_NOFS api

2017-01-06 Thread Michal Hocko
++-- kernel/locking/lockdep.c | 6 +- lib/radix-tree.c | 2 ++ mm/page_alloc.c | 8 +--- mm/vmscan.c | 6 +++--- 19 files changed, 109 insertions(+), 45 deletions(-) Shortlog: Michal Hocko (8): lockdep: allow to disable reclaim lockup detection xfs

[Cluster-devel] [PATCH 4/8] xfs: use memalloc_nofs_{save, restore} instead of memalloc_noio*

2017-01-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> kmem_zalloc_large and _xfs_buf_map_pages use memalloc_noio_{save,restore} API to prevent from reclaim recursion into the fs because vmalloc can invoke unconditional GFP_KERNEL allocations and these functions might be called from the NOFS co

[Cluster-devel] [PATCH 6/8] jbd2: make the whole kjournald2 kthread NOFS safe

2017-01-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> kjournald2 is central to the transaction commit processing. As such any potential allocation from this kernel thread has to be GFP_NOFS. Make sure to mark the whole kernel thread GFP_NOFS by the memalloc_nofs_save. Suggested-by: Jan Kara <j...@suse.c

[Cluster-devel] [PATCH 1/8] lockdep: allow to disable reclaim lockup detection

2017-01-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> The current implementation of the reclaim lockup detection can lead to false positives and those even happen and usually lead to tweak the code to silence the lockdep by using GFP_NOFS even though the context can use __GFP_FS just fine. Se

[Cluster-devel] [PATCH 3/8] mm: introduce memalloc_nofs_{save, restore} API

2017-01-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> GFP_NOFS context is used for the following 5 reasons currently - to prevent from deadlocks when the lock held by the allocation context would be needed during the memory reclaim - to prevent from stack overflows during the r

Re: [Cluster-devel] [PATCH 0/9 v2] scope GFP_NOFS api

2016-12-22 Thread Michal Hocko
this API. Can we find a way to use it? -- Michal Hocko SUSE Labs

Re: [Cluster-devel] [PATCH 2/9] xfs: introduce and use KM_NOLOCKDEP to silence reclaim lockdep false positives

2016-12-20 Thread Michal Hocko
On Tue 20-12-16 08:24:13, Dave Chinner wrote: > On Thu, Dec 15, 2016 at 03:07:08PM +0100, Michal Hocko wrote: > > From: Michal Hocko <mho...@suse.com> > > > > Now that the page allocator offers __GFP_NOLOCKDEP let's introduce > > KM_NOLOCKDEP alias for

Re: [Cluster-devel] [PATCH 2/9 v2] xfs: introduce and use KM_NOLOCKDEP to silence reclaim lockdep false positives

2016-12-16 Thread Michal Hocko
On Fri 16-12-16 11:37:50, Brian Foster wrote: > On Fri, Dec 16, 2016 at 04:40:41PM +0100, Michal Hocko wrote: > > Updated patch after Mike noticed a BUG_ON when KM_NOLOCKDEP is used. > > --- > > From 1497e713e11639157aef21cae29052cb3dc7ab44 Mon Sep 17 00:00:00 2001 > &

[Cluster-devel] [PATCH 5/9 v2] xfs: use memalloc_nofs_{save, restore} instead of memalloc_noio*

2016-12-16 Thread Michal Hocko
On Fri 16-12-16 11:38:11, Brian Foster wrote: > On Thu, Dec 15, 2016 at 03:07:11PM +0100, Michal Hocko wrote: [...] > > @@ -459,7 +459,7 @@ _xfs_buf_map_pages( > > break; > > vm_unmap_aliases(); > >

[Cluster-devel] [PATCH 2/9 v2] xfs: introduce and use KM_NOLOCKDEP to silence reclaim lockdep false positives

2016-12-16 Thread Michal Hocko
Updated patch after Mike noticed a BUG_ON when KM_NOLOCKDEP is used. --- >From 1497e713e11639157aef21cae29052cb3dc7ab44 Mon Sep 17 00:00:00 2001 From: Michal Hocko <mho...@suse.com> Date: Thu, 15 Dec 2016 13:06:43 +0100 Subject: [PATCH] xfs: introduce and use KM_NOLOCKDEP to silenc

Re: [Cluster-devel] [PATCH 0/9 v2] scope GFP_NOFS api

2016-12-16 Thread Michal Hocko
On Fri 16-12-16 16:05:58, Mike Galbraith wrote: > On Thu, 2016-12-15 at 15:07 +0100, Michal Hocko wrote: > > Hi, > > I have posted the previous version here [1]. Since then I have added a > > support to suppress reclaim lockdep warnings (__GFP_NOLOCKDEP) to allow > &

[Cluster-devel] [DEBUG PATCH 1/2] mm, debug: report when GFP_NO{FS, IO} is used explicitly from memalloc_no{fs, io}_{save, restore} context

2016-12-16 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> THIS PATCH IS FOR TESTING ONLY AND NOT MEANT TO HIT LINUS TREE It is desirable to reduce the direct GFP_NO{FS,IO} usage at minimum and prefer scope usage defined by memalloc_no{fs,io}_{save,restore} API. Let's help this process and add a debuggin

[Cluster-devel] [DEBUG PATCH 0/2] debug explicit GFP_NO{FS, IO} usage from the scope context

2016-12-16 Thread Michal Hocko
Hi, I've forgot to add the following two patches which should help to identify explicit GFP_NO{FS,IO} usage from withing a scope context. Such a usage can be changed to the full GFP_KERNEL because all the calls from within the NO{FS,IO} scope will drop the __GFP_FS resp. __GFP_IO automatically and

[Cluster-devel] [DEBUG PATCH 2/2] silent warnings which we cannot do anything about

2016-12-16 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> THIS PATCH IS FOR TESTING ONLY AND NOT MEANT TO HIT LINUS TREE There are some code paths used by all the filesystems which we cannot change to drop the GFP_NOFS, yet they generate a lot of warnings. Provide {disable,enable}_scope_gfp_check to silence

[Cluster-devel] [PATCH 5/9] xfs: use memalloc_nofs_{save, restore} instead of memalloc_noio*

2016-12-15 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> kmem_zalloc_large and _xfs_buf_map_pages use memalloc_noio_{save,restore} API to prevent from reclaim recursion into the fs because vmalloc can invoke unconditional GFP_KERNEL allocations and these functions might be called from the NOFS co

[Cluster-devel] [PATCH 4/9] mm: introduce memalloc_nofs_{save, restore} API

2016-12-15 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> GFP_NOFS context is used for the following 5 reasons currently - to prevent from deadlocks when the lock held by the allocation context would be needed during the memory reclaim - to prevent from stack overflows during the r

[Cluster-devel] [PATCH 6/9] jbd2: mark the transaction context with the scope GFP_NOFS context

2016-12-15 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> now that we have memalloc_nofs_{save,restore} api we can mark the whole transaction context as implicitly GFP_NOFS. All allocations will automatically inherit GFP_NOFS this way. This means that we do not have to mark any of those requests with GF

[Cluster-devel] [PATCH 8/9] Revert "ext4: avoid deadlocks in the writeback path by using sb_getblk_gfp"

2016-12-15 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> This reverts commit c45653c341f5c8a0ce19c8f0ad4678640849cb86 because sb_getblk_gfp is not really needed as sb_getblk __getblk_gfp __getblk_slow grow_buffers grow_dev_page gfp_mask = mapping_gfp_constraint(inode->

[Cluster-devel] [PATCH 3/9] xfs: abstract PF_FSTRANS to PF_MEMALLOC_NOFS

2016-12-15 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> xfs has defined PF_FSTRANS to declare a scope GFP_NOFS semantic quite some time ago. We would like to make this concept more generic and use it for other filesystems as well. Let's start by giving the flag a more genric name PF_MEMALLOC_NOFS which is i

[Cluster-devel] [PATCH 0/9 v2] scope GFP_NOFS api

2016-12-15 Thread Michal Hocko
(-) Shortlog: Michal Hocko (9): lockdep: allow to disable reclaim lockup detection xfs: introduce and use KM_NOLOCKDEP to silence reclaim lockdep false positives xfs: abstract PF_FSTRANS to PF_MEMALLOC_NOFS mm: introduce memalloc_nofs_{save,restore} API xfs: use

[Cluster-devel] [PATCH 7/9] jbd2: make the whole kjournald2 kthread NOFS safe

2016-12-15 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> kjournald2 is central to the transaction commit processing. As such any potential allocation from this kernel thread has to be GFP_NOFS. Make sure to mark the whole kernel thread GFP_NOFS by the memalloc_nofs_save. Suggested-by: Jan Kara <j...@suse.c

[Cluster-devel] [PATCH 9/9] Revert "ext4: fix wrong gfp type under transaction"

2016-12-15 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> This reverts commit 216553c4b7f3e3e2beb4981cddca9b2027523928. Now that the transaction context uses memalloc_nofs_save and all allocations within the this context inherit GFP_NOFS automatically, there is no reason to mark specific allocations expl

[Cluster-devel] [PATCH 1/9] lockdep: allow to disable reclaim lockup detection

2016-12-15 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> The current implementation of the reclaim lockup detection can lead to false positives and those even happen and usually lead to tweak the code to silence the lockdep by using GFP_NOFS even though the context can use __GFP_FS just fine. Se

Re: [Cluster-devel] [PATCH 0/2] scop GFP_NOFS api

2016-05-03 Thread Michal Hocko
on to free that > memory. > > Whenever trying to free memory I think you need to do best-effort > without blocking. I agree with that. Or at least you have to wait on something that is _guaranteed_ to make a forward progress. I am not really that sure this is easy to achieve with the current code base. -- Michal Hocko SUSE Labs

Re: [Cluster-devel] [PATCH 0/2] scop GFP_NOFS api

2016-04-29 Thread Michal Hocko
On Fri 29-04-16 15:35:42, NeilBrown wrote: > On Tue, Apr 26 2016, Michal Hocko wrote: > > > Hi, > > we have discussed this topic at LSF/MM this year. There was a general > > interest in the scope GFP_NOFS allocation context among some FS > > developer

Re: [Cluster-devel] [PATCH 1.2/2] mm: introduce memalloc_nofs_{save, restore} API

2016-04-27 Thread Michal Hocko
rom 1968c0a8ace4090a9deca8f4c1a206ee546e595a Mon Sep 17 00:00:00 2001 From: Michal Hocko <mho...@suse.com> Date: Wed, 27 Apr 2016 22:32:57 +0200 Subject: [PATCH] fold me "mm: introduce memalloc_nofs_{save,restore} API" Lockdep infrastructure is hooked into early hot paths of the allocator so __lockdep_trace

Re: [Cluster-devel] [PATCH 1.2/2] mm: introduce memalloc_nofs_{save, restore} API

2016-04-27 Thread Michal Hocko
On Wed 27-04-16 22:09:27, Michal Hocko wrote: [...] > [ 53.993480] [] mark_held_locks+0x5e/0x74 > [ 53.993480] [] lockdep_trace_alloc+0xb2/0xb5 > [ 53.993480] [] kmem_cache_alloc+0x36/0x2b0 Scratch that. I got it. It is the lockdep annotation which I got wrong with my patch.

Re: [Cluster-devel] [PATCH 1.2/2] mm: introduce memalloc_nofs_{save, restore} API

2016-04-27 Thread Michal Hocko
Hi Dave, On Wed 27-04-16 13:54:35, Michal Hocko wrote: [...] > diff --git a/fs/xfs/kmem.h b/fs/xfs/kmem.h > index 0d83f332e5c2..b35688a54c9a 100644 > --- a/fs/xfs/kmem.h > +++ b/fs/xfs/kmem.h > @@ -50,7 +50,7 @@ kmem_flags_convert(xfs_km_flags_t flags) > l

Re: [Cluster-devel] [PATCH 1.1/2] xfs: abstract PF_FSTRANS to PF_MEMALLOC_NOFS

2016-04-27 Thread Michal Hocko
On Wed 27-04-16 11:41:51, Andreas Dilger wrote: > On Apr 27, 2016, at 5:54 AM, Michal Hocko <mho...@kernel.org> wrote: [...] > > --- a/fs/xfs/kmem.c > > +++ b/fs/xfs/kmem.c > > @@ -80,13 +80,13 @@ kmem_zalloc_large(size_t size, xfs_km_flags_t flags) > >

Re: [Cluster-devel] [PATCH 1.2/2] mm: introduce memalloc_nofs_{save, restore} API

2016-04-27 Thread Michal Hocko
On Wed 27-04-16 13:54:35, Michal Hocko wrote: > From: Michal Hocko <mho...@suse.com> > Ups missed Dave's note about: > GFP_NOFS context is used for the following 4 reasons currently > - to prevent from deadlocks when the lock held by the allocation > context w

[Cluster-devel] [PATCH 1.1/2] xfs: abstract PF_FSTRANS to PF_MEMALLOC_NOFS

2016-04-27 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> xfs has defined PF_FSTRANS to declare a scope GFP_NOFS semantic quite some time ago. We would like to make this concept more generic and use it for other filesystems as well. Let's start by giving the flag a more genric name PF_MEMALLOC_NOFS which is i

[Cluster-devel] [PATCH 1.2/2] mm: introduce memalloc_nofs_{save, restore} API

2016-04-27 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> GFP_NOFS context is used for the following 4 reasons currently - to prevent from deadlocks when the lock held by the allocation context would be needed during the memory reclaim - to prevent from stack overflows during the r

Re: [Cluster-devel] [PATCH 1/2] mm: add PF_MEMALLOC_NOFS

2016-04-27 Thread Michal Hocko
On Wed 27-04-16 09:07:02, Dave Chinner wrote: > On Tue, Apr 26, 2016 at 01:56:11PM +0200, Michal Hocko wrote: > > From: Michal Hocko <mho...@suse.com> > > > > GFP_NOFS context is used for the following 4 reasons currently > > - to prevent from deadlocks whe

[Cluster-devel] [PATCH 0/2] scop GFP_NOFS api

2016-04-26 Thread Michal Hocko
Hi, we have discussed this topic at LSF/MM this year. There was a general interest in the scope GFP_NOFS allocation context among some FS developers. For those who are not aware of the discussion or the issue I am trying to sort out (or at least start in that direction) please have a look at patch

[Cluster-devel] [PATCH 1/2] mm: add PF_MEMALLOC_NOFS

2016-04-26 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> GFP_NOFS context is used for the following 4 reasons currently - to prevent from deadlocks when the lock held by the allocation context would be needed during the memory reclaim - to prevent from stack overflows during the r

[Cluster-devel] [PATCH 2/2] mm, debug: report when GFP_NO{FS, IO} is used explicitly from memalloc_no{fs, io}_{save, restore} context

2016-04-26 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> THIS PATCH IS FOR TESTING ONLY AND NOT MEANT TO HIT LINUS TREE It is desirable to reduce the direct GFP_NO{FS,IO} usage at minimum and prefer scope usage defined by memalloc_no{fs,io}_{save,restore} API. Let's help this process and add a debuggin