Re: [dm-devel] [PATCH] md: simplify free_params for kmalloc vs vmalloc fallback

2016-04-29 Thread Michal Hocko
haven't checked other than quick git grep on KMALLOC_MAX_SIZE and there do not seem that many) who are sharing the same fallback strategy then why not. But I suspect that some would rather fallback earlier and even do not attempt larger than e.g. order-1 requests. -- Michal Hocko SUSE Labs -- d

Re: [dm-devel] [PATCH 18/20] dm: clean up GFP_NIO usage

2016-04-29 Thread Michal Hocko
On Thu 28-04-16 10:20:09, Mikulas Patocka wrote: > > > On Thu, 28 Apr 2016, Michal Hocko wrote: > > > From: Michal Hocko <mho...@suse.com> > > > > copy_params uses GFP_NOIO for explicit allocation requests because this > > might be called from the sus

Re: [dm-devel] [PATCH] md: simplify free_params for kmalloc vs vmalloc fallback

2016-04-29 Thread Michal Hocko
2.6.10-mm1/broken-out/alloc_large_system_hash-numa-interleaving.patch The changelog is quite verbose but no mention about this ugliness. So I do agree that the above should be fixed and a common helper might be interesting but I am afraid we are getting off topic here. Thanks! -- Michal Hocko SUSE Labs

[dm-devel] [PATCH] md: simplify free_params for kmalloc vs vmalloc fallback

2016-04-29 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> Use kvfree rather than DM_PARAMS_[KV]MALLOC specific param flags. Cc: Shaohua Li <s...@kernel.org> Cc: Mikulas Patocka <mpato...@redhat.com> Cc: dm-devel@redhat.com Signed-off-by: Michal Hocko <mho...@suse.com> --- Hi, this is a reba

[dm-devel] [PATCH 17/20] dm: get rid of superfluous gfp flags

2016-04-28 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> copy_params seems to be little bit confused about which allocation flags to use. It enforces GFP_NOIO even though it uses memalloc_noio_{save,restore} which enforces GFP_NOIO at the page allocator level automatically (via memalloc_noio_flags). It als

[dm-devel] [PATCH 19/20] md: simplify free_params for kmalloc vs vmalloc fallback

2016-04-28 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> Use kvfree rather than DM_PARAMS_[KV]MALLOC specific param flags. Cc: Shaohua Li <s...@kernel.org> Cc: Mikulas Patocka <mpato...@redhat.com> Cc: dm-devel@redhat.com Signed-off-by: Michal Hocko <mho...@suse.com> --- dr

[dm-devel] [PATCH 18/20] dm: clean up GFP_NIO usage

2016-04-28 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> copy_params uses GFP_NOIO for explicit allocation requests because this might be called from the suspend path. To quote Mikulas: : The LVM tool calls suspend and resume ioctls on device mapper block : devices. : : When a device is suspended, any bi

Re: [dm-devel] [PATCH 17/19] dm: get rid of superfluous gfp flags

2016-04-27 Thread Michal Hocko
[Adding dm-devel@redhat.com to CC] On Tue 26-04-16 13:20:04, Mikulas Patocka wrote: > On Fri, 22 Apr 2016, Michal Hocko wrote: [...] > > copy_params seems to be called only from the ioctl context which doesn't > > hold any locks which would lockup during the direct reclaim AFAICS.

Re: [dm-devel] [PATCH 17/20] dm: get rid of superfluous gfp flags

2016-05-02 Thread Michal Hocko
On Fri 29-04-16 14:54:52, Mike Snitzer wrote: > On Thu, Apr 28 2016 at 9:24am -0400, > Michal Hocko <mho...@kernel.org> wrote: > > > From: Michal Hocko <mho...@suse.com> > > > > copy_params seems to be little bit confused about which allocation flags >

Re: [dm-devel] [RFC PATCH 2/2] mm, mempool: do not throttle PF_LESS_THROTTLE tasks

2016-07-26 Thread Michal Hocko
ios for device mapper in the > past, but there were not integrated into upstream. Care to revive them? I am not an expert in dm but unbounded amount of inflight IO doesn't really sound good. [...] -- Michal Hocko SUSE Labs -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] System freezes after OOM

2016-07-13 Thread Michal Hocko
On Wed 13-07-16 10:18:35, Mikulas Patocka wrote: > > > On Wed, 13 Jul 2016, Michal Hocko wrote: > > > [CC David] > > > > > > It is caused by the commit f9054c70d28bc214b2857cf8db8269f4f45a5e23. > > > > Prior to this commit, mempool allocati

Re: [dm-devel] System freezes after OOM

2016-07-14 Thread Michal Hocko
On Thu 14-07-16 10:00:16, Mikulas Patocka wrote: > > > On Thu, 14 Jul 2016, Michal Hocko wrote: > > > On Wed 13-07-16 11:02:15, Mikulas Patocka wrote: > > > > > diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c > > > > index 4f3cb3554944.

Re: [dm-devel] System freezes after OOM

2016-07-14 Thread Michal Hocko
lusher still gets throttled because the bias it gets is not sufficient. Or does it get throttled at a different place? -- Michal Hocko SUSE Labs -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] System freezes after OOM

2016-07-14 Thread Michal Hocko
On Thu 14-07-16 19:07:52, Ondrej Kozina wrote: > On 07/14/2016 05:31 PM, Michal Hocko wrote: > > On Thu 14-07-16 16:08:28, Ondrej Kozina wrote: > > [...] > > > As Mikulas pointed out, this doesn't work. The system froze as well with > > > the > > >

Re: [dm-devel] System freezes after OOM

2016-07-14 Thread Michal Hocko
On Thu 14-07-16 19:36:59, Michal Hocko wrote: > On Thu 14-07-16 19:07:52, Ondrej Kozina wrote: > > On 07/14/2016 05:31 PM, Michal Hocko wrote: > > > On Thu 14-07-16 16:08:28, Ondrej Kozina wrote: > > > [...] > > > > As Mikulas pointed out, this

Re: [dm-devel] [RFC PATCH 2/2] mm, mempool: do not throttle PF_LESS_THROTTLE tasks

2016-07-25 Thread Michal Hocko
On Sat 23-07-16 10:12:24, NeilBrown wrote: > On Fri, Jul 22 2016, Michal Hocko wrote: [...] > > If we just back off and rely on kswapd which > > might get stuck on the writeout then the IO throughput can be reduced > > If I were king of MM,

Re: [dm-devel] [RFC PATCH 2/2] mm, mempool: do not throttle PF_LESS_THROTTLE tasks

2016-07-28 Thread Michal Hocko
On Thu 28-07-16 07:33:19, NeilBrown wrote: > On Thu, Jul 28 2016, Michal Hocko wrote: > > > On Wed 27-07-16 13:43:35, NeilBrown wrote: > >> On Mon, Jul 25 2016, Michal Hocko wrote: > >> > >> > On Sat 23-07-16 10:12:24, NeilBrown wrote: > >

Re: [dm-devel] [RFC PATCH 2/2] mm, mempool: do not throttle PF_LESS_THROTTLE tasks

2016-07-27 Thread Michal Hocko
e rare. Anyway do you see an excessive throttling with the patch posted http://lkml.kernel.org/r/20160725192344.gd2...@dhcp22.suse.cz ? Or from another side. Do you see an excessive number of dirty/writeback pages wrt. the dirty threshold or any other undesirable side effects? -- Michal Hocko SUSE Labs -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [RFC PATCH 2/2] mm, mempool: do not throttle PF_LESS_THROTTLE tasks

2016-07-27 Thread Michal Hocko
On Wed 27-07-16 13:43:35, NeilBrown wrote: > On Mon, Jul 25 2016, Michal Hocko wrote: > > > On Sat 23-07-16 10:12:24, NeilBrown wrote: [...] > >> > My thinking was that throttle_vm_writeout is there to prevent from > >> > dirtying too many pages from the re

Re: [dm-devel] [RFC PATCH 2/2] mm, mempool: do not throttle PF_LESS_THROTTLE tasks

2016-08-03 Thread Michal Hocko
On Wed 03-08-16 09:59:11, Mikulas Patocka wrote: > > > On Wed, 27 Jul 2016, Michal Hocko wrote: > > > On Wed 27-07-16 10:28:40, Mikulas Patocka wrote: [...] > > > I think that approach with PF_LESS_THROTTLE in mempool_alloc is incorrect > > > and t

Re: [dm-devel] Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage)

2016-07-12 Thread Michal Hocko
On Tue 12-07-16 13:49:20, Michal Hocko wrote: > On Tue 12-07-16 13:28:12, Matthias Dahl wrote: > > Hello Michal... > > > > On 2016-07-12 11:50, Michal Hocko wrote: > > > > > This smells like file pages are stuck in the writeback somewhere and the > > &g

Re: [dm-devel] Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage)

2016-07-12 Thread Michal Hocko
the writeback somewhere and the anon memory is not reclaimable because you do not have any swap device. -- Michal Hocko SUSE Labs -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] System freezes after OOM

2016-07-14 Thread Michal Hocko
On Wed 13-07-16 11:21:41, Mikulas Patocka wrote: > > > On Wed, 13 Jul 2016, Milan Broz wrote: > > > On 07/13/2016 02:50 PM, Michal Hocko wrote: > > > On Wed 13-07-16 13:10:06, Michal Hocko wrote: > > >> On Tue 12-07-16 19:44:11, Mikulas Patocka wrote:

Re: [dm-devel] [RFC PATCH 1/2] mempool: do not consume memory reserves from the reclaim path

2016-07-21 Thread Michal Hocko
On Thu 21-07-16 08:13:00, Johannes Weiner wrote: > On Thu, Jul 21, 2016 at 10:52:03AM +0200, Michal Hocko wrote: > > Look, there are > > $ git grep mempool_alloc | wc -l > > 304 > > > > many users of this API and we do not want to flip the default behavior &g

Re: [dm-devel] System freezes after OOM

2016-07-15 Thread Michal Hocko
On Thu 14-07-16 13:35:35, Mikulas Patocka wrote: > On Thu, 14 Jul 2016, Michal Hocko wrote: > > On Thu 14-07-16 10:00:16, Mikulas Patocka wrote: > > > But it needs other changes to honor the PF_LESS_THROTTLE flag: > > > > > > static int current_may_throttl

Re: [dm-devel] [RFC PATCH 1/2] mempool: do not consume memory reserves from the reclaim path

2016-07-20 Thread Michal Hocko
On Tue 19-07-16 17:50:29, Mikulas Patocka wrote: > > > On Mon, 18 Jul 2016, Michal Hocko wrote: > > > From: Michal Hocko <mho...@suse.com> > > > > There has been a report about OOM killer invoked when swapping out to > > a dm-crypt device. The p

Re: [dm-devel] [RFC PATCH 1/2] mempool: do not consume memory reserves from the reclaim path

2016-07-20 Thread Michal Hocko
ils about it and the patch might very well paper over a bug in mempool usage somewhere else. We definitely need more details to know that better. That being said, f9054c70d28b ("mm, mempool: only set __GFP_NOMEMALLOC if there are free elements") should be either reverted or http://lkml.ke

Re: [dm-devel] System freezes after OOM

2016-07-15 Thread Michal Hocko
On Fri 15-07-16 08:11:22, Mikulas Patocka wrote: > > > On Fri, 15 Jul 2016, Michal Hocko wrote: > > > On Thu 14-07-16 13:35:35, Mikulas Patocka wrote: > > > On Thu, 14 Jul 2016, Michal Hocko wrote: > > > > On Thu 14-07-16 10:00:16, Mikulas Patocka wrot

Re: [dm-devel] Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage)

2016-07-12 Thread Michal Hocko
On Tue 12-07-16 14:42:12, Matthias Dahl wrote: > Hello Michal... > > On 2016-07-12 13:49, Michal Hocko wrote: > > > I am not a storage expert (not even mention dm-crypt). But what those > > counters say is that the IO completion doesn't trigger so the > > P

Re: [dm-devel] Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage)

2016-07-13 Thread Michal Hocko
On Tue 12-07-16 16:56:32, Matthias Dahl wrote: > Hello Michal... > > On 2016-07-12 16:07, Michal Hocko wrote: > > > /proc/slabinfo could at least point on who is eating that memory. > > Thanks. I have made another test (and thus again put the RAID10 out of > sy

Re: [dm-devel] Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage)

2016-07-13 Thread Michal Hocko
On Wed 13-07-16 13:21:26, Michal Hocko wrote: > On Tue 12-07-16 16:56:32, Matthias Dahl wrote: [...] > > If that support is baked into the Fedora provided kernel that is. If > > you could give me a few hints or pointers, how to properly do a allocator > > trace point and get

Re: [dm-devel] System freezes after OOM

2016-07-13 Thread Michal Hocko
else if (!in_interrupt() && > - ((current->flags & PF_MEMALLOC) || > - unlikely(test_thread_flag(TIF_MEMDIE > + else if (!in_interrupt() && (current->flags & PF_MEMALLOC)) > alloc_flags |= ALLOC_NO_WATERMARKS; > } > #ifdef CONFIG_CMA > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majord...@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: mailto:"d...@kvack.org;> em...@kvack.org -- Michal Hocko SUSE Labs -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] System freezes after OOM

2016-07-13 Thread Michal Hocko
r which went off the leash. > So, it could prematurely trigger OOM killer on any slow swapping device > (including dm-crypt). Michal Hocko reworked the OOM killer in the patch > 0a0337e0d1d134465778a16f5cbea95086e8e9e0, but it still has the flaw that > it triggers OOM if there is plenty

Re: [dm-devel] System freezes after OOM

2016-07-13 Thread Michal Hocko
over to the crypt layer and finish the IO. Is it possible this might take a lot of time? > On Tue, 12 Jul 2016, Michal Hocko wrote: > > > On Mon 11-07-16 11:43:02, Mikulas Patocka wrote: > > [...] > > > The general problem is that the memory allocator does 16 retries to > &

Re: [dm-devel] System freezes after OOM

2016-07-13 Thread Michal Hocko
out earlier, when there might still be enough free memory. > > Ondrej, have you tried to increase /proc/sys/vm/watermark_scale_factor? I suspect this would just change the timing or the real problem gets hidden. -- Michal Hocko SUSE Labs -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [RFC PATCH 0/2] mempool vs. page allocator interaction

2016-07-18 Thread Michal Hocko
Hi, there have been two issues identified when investigating dm-crypt backed swap recently [1]. The first one looks like a regression from f9054c70d28b ("mm, mempool: only set __GFP_NOMEMALLOC if there are free elements") because swapout path can now deplete all the available memory reserves. The

Re: [dm-devel] System freezes after OOM

2016-07-18 Thread Michal Hocko
On Fri 15-07-16 13:02:17, Mikulas Patocka wrote: > > > On Fri, 15 Jul 2016, Michal Hocko wrote: > > > On Fri 15-07-16 08:11:22, Mikulas Patocka wrote: > > > > > > The stacktraces showed that the kcryptd process was throttled when it > > >

[dm-devel] [RFC PATCH 2/2] mm, mempool: do not throttle PF_LESS_THROTTLE tasks

2016-07-18 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> Mikulas has reported that a swap backed by dm-crypt doesn't work properly because the swapout cannot make a sufficient forward progress as the writeout path depends on dm_crypt worker which has to allocate memory to perform the encryption. In

[dm-devel] [RFC PATCH 1/2] mempool: do not consume memory reserves from the reclaim path

2016-07-18 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> There has been a report about OOM killer invoked when swapping out to a dm-crypt device. The primary reason seems to be that the swapout out IO managed to completely deplete memory reserves. Mikulas was able to bisect and explained the issue by po

Re: [dm-devel] [RFC PATCH 2/2] mm, mempool: do not throttle PF_LESS_THROTTLE tasks

2016-07-22 Thread Michal Hocko
On Fri 22-07-16 18:46:57, Neil Brown wrote: > On Mon, Jul 18 2016, Michal Hocko wrote: > > > From: Michal Hocko <mho...@suse.com> > > > > Mikulas has reported that a swap backed by dm-crypt doesn't work > > properly because the swapout cannot make a sufficient

Re: [dm-devel] [RFC PATCH] mm: introduce kv[mz]alloc helpers

2016-12-08 Thread Michal Hocko
On Thu 08-12-16 14:00:20, David Hildenbrand wrote: > Am 08.12.2016 um 11:33 schrieb Michal Hocko: > > From: Michal Hocko <mho...@suse.com> > > > > Using kmalloc with the vmalloc fallback for larger allocations is a > > common pattern in the kernel code. Yet

[dm-devel] [RFC PATCH] mm: introduce kv[mz]alloc helpers

2016-12-08 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> Using kmalloc with the vmalloc fallback for larger allocations is a common pattern in the kernel code. Yet we do not have any common helper for that and so users have invented their own helpers. Some of them are really creative when doing so. Let's just

Re: [dm-devel] [RFC PATCH] mm: introduce kv[mz]alloc helpers

2016-12-08 Thread Michal Hocko
On Fri 09-12-16 06:38:04, Al Viro wrote: > On Fri, Dec 09, 2016 at 07:22:25AM +0100, Michal Hocko wrote: > > > > Easier to handle those in vmalloc() itself. > > > > I think there were some attempts in the past but some of the code paths > > are burried too dee

Re: [dm-devel] [RFC PATCH] mm: introduce kv[mz]alloc helpers

2016-12-08 Thread Michal Hocko
On Fri 09-12-16 02:00:17, Al Viro wrote: > On Fri, Dec 09, 2016 at 12:44:17PM +1100, Dave Chinner wrote: > > On Thu, Dec 08, 2016 at 11:33:00AM +0100, Michal Hocko wrote: > > > From: Michal Hocko <mho...@suse.com> > > > > > > Using kmalloc with t

Re: [dm-devel] [RFC PATCH] mm: introduce kv[mz]alloc helpers

2016-12-14 Thread Michal Hocko
On Tue 13-12-16 13:55:46, Andreas Dilger wrote: > On Dec 13, 2016, at 3:14 AM, Michal Hocko <mho...@kernel.org> wrote: > > > > Are there any more comments or objections to this patch? Is this a good > > start or kv[mz]alloc has to provide a way to cover GFP_NOFS users

Re: [dm-devel] [RFC PATCH] mm: introduce kv[mz]alloc helpers

2016-12-13 Thread Michal Hocko
Are there any more comments or objections to this patch? Is this a good start or kv[mz]alloc has to provide a way to cover GFP_NOFS users as well in the initial version. On Thu 08-12-16 11:33:00, Michal Hocko wrote: > From: Michal Hocko <mho...@suse.com> > > Using kmalloc w

Re: [dm-devel] [RFC PATCH] mm: introduce kv[mz]alloc helpers

2016-12-21 Thread Michal Hocko
It seems that this email didn't get delivered due to some stupid gmail spam policy. Let me try to repost via a different relay. Sorry to those who have seen the original message and get a duplicate now. On Wed 21-12-16 08:03:53, Michal Hocko wrote: > On Tue 20-12-16 14:13:41, Andrew Morton wr

Re: [dm-devel] [RFC PATCH] mm: introduce kv[mz]alloc helpers

2016-12-20 Thread Michal Hocko
On Wed 14-12-16 09:59:16, Michal Hocko wrote: > On Tue 13-12-16 14:07:33, Joe Perches wrote: > > On Tue, 2016-12-13 at 11:14 +0100, Michal Hocko wrote: > > > Are there any more comments or objections to this patch? Is this a good > > > start or kv[mz]alloc has to prov

Re: [dm-devel] [PATCH v5] fault-injection: introduce kvmalloc fallback options

2018-04-27 Thread Michal Hocko
e is a risk of harm IIUC and this is not much different than other fault injecting paths. -- Michal Hocko SUSE Labs -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] dm bufio: Reduce dm_bufio_lock contention

2018-06-15 Thread Michal Hocko
On Fri 15-06-18 08:47:52, Mikulas Patocka wrote: > > > On Fri, 15 Jun 2018, Michal Hocko wrote: > > > On Fri 15-06-18 07:35:07, Mikulas Patocka wrote: > > > > > > Because mempool uses it. Mempool uses allocations with "GFP_NOIO | > > > __GFP

Re: [dm-devel] dm bufio: Reduce dm_bufio_lock contention

2018-06-15 Thread Michal Hocko
On Fri 15-06-18 07:35:07, Mikulas Patocka wrote: > > > On Fri, 15 Jun 2018, Michal Hocko wrote: > > > On Thu 14-06-18 14:34:06, Mikulas Patocka wrote: > > > > > > > > > On Thu, 14 Jun 2018, Michal Hocko wrote: > >

Re: [dm-devel] dm bufio: Reduce dm_bufio_lock contention

2018-06-14 Thread Michal Hocko
tch at ff8008593198 > #18 [ffc0282afd20] verity_prefetch_io at ff8008598384 > #19 [ffc0282afd70] process_one_work at ff80080b5b3c > #20 [ffc0282afdc0] worker_thread at ff80080b64fc > #21 [ffc0282afe20] kthread at ff80080bae34 > > >

Re: [dm-devel] dm bufio: Reduce dm_bufio_lock contention

2018-06-15 Thread Michal Hocko
On Thu 14-06-18 14:34:06, Mikulas Patocka wrote: > > > On Thu, 14 Jun 2018, Michal Hocko wrote: > > > On Thu 14-06-18 15:18:58, jing xia wrote: > > [...] > > > PID: 22920 TASK: ffc0120f1a00 CPU: 1 COMMAND: "kworker/u8:2" > > > #

Re: [dm-devel] dm bufio: Reduce dm_bufio_lock contention

2018-06-25 Thread Michal Hocko
On Mon 25-06-18 09:53:34, Mikulas Patocka wrote: > y > > On Mon, 25 Jun 2018, Michal Hocko wrote: > > > On Fri 22-06-18 14:57:10, Mikulas Patocka wrote: > > > > > > > > > On Fri, 22 Jun 2018, Michal Hocko wrote: > > > &

Re: [dm-devel] dm bufio: Reduce dm_bufio_lock contention

2018-06-25 Thread Michal Hocko
On Mon 25-06-18 10:42:30, Mikulas Patocka wrote: > > > On Mon, 25 Jun 2018, Michal Hocko wrote: > > > > And the throttling in dm-bufio prevents kswapd from making forward > > > progress, causing this situation... > > > > Which is what we have PF

Re: [dm-devel] dm bufio: Reduce dm_bufio_lock contention

2018-06-25 Thread Michal Hocko
On Fri 22-06-18 14:57:10, Mikulas Patocka wrote: > > > On Fri, 22 Jun 2018, Michal Hocko wrote: > > > On Fri 22-06-18 08:52:09, Mikulas Patocka wrote: > > > > > > > > > On Fri, 22 Jun 2018, Michal Hocko wrote: > > > > > > > On

Re: [dm-devel] dm bufio: Reduce dm_bufio_lock contention

2018-06-22 Thread Michal Hocko
rent_is_kswapd() && > +(sc->gfp_mask & (__GFP_NORETRY | __GFP_FS)) != __GFP_NORETRY > && > current_may_throttle() && pgdat_memcg_congested(pgdat, root)) > wait_iff_congested(BLK_RW_ASYNC, HZ/10); > -- Michal Hocko SUSE Labs -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] dm bufio: Reduce dm_bufio_lock contention

2018-08-05 Thread Michal Hocko
> Then, all the task who do shrink_slab() wait for dm_bufio_lock. > > Any suggestions for this?Thanks. Do not use soft limit? -- Michal Hocko SUSE Labs -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] dm bufio: Reduce dm_bufio_lock contention

2018-09-04 Thread Michal Hocko
On Tue 04-09-18 13:30:44, Mikulas Patocka wrote: > > > On Tue, 4 Sep 2018, Michal Hocko wrote: > > > Regarding other other workloads. AFAIR the problem was due to the > > wait_iff_congested in the direct reclaim. And I've been arguing that > > special casing __GF

Re: [dm-devel] dm bufio: Reduce dm_bufio_lock contention

2018-09-04 Thread Michal Hocko
On Tue 04-09-18 11:18:44, Mike Snitzer wrote: > On Tue, Sep 04 2018 at 3:08am -0400, > Michal Hocko wrote: > > > On Mon 03-09-18 18:23:17, Mikulas Patocka wrote: > > > > > > > > > On Wed, 1 Aug 2018, jing xia wrote: > > > > > > &

Re: [dm-devel] [PATCH] SLUB: Do not fallback to mininum order if __GFP_NORETRY is set

2018-04-19 Thread Michal Hocko
*allocate_slab(struct > alloc_gfp = (alloc_gfp | __GFP_NOMEMALLOC) & > ~(__GFP_RECLAIM|__GFP_NOFAIL); > > page = alloc_slab_page(s, alloc_gfp, node, oo); > - if (unlikely(!page)) { > + if (unlikely(!page) && !(flags & __GFP_NORETRY)) { >

Re: [dm-devel] [PATCH] kvmalloc: always use vmalloc if CONFIG_DEBUG_VM

2018-04-20 Thread Michal Hocko
On Fri 20-04-18 06:41:36, Matthew Wilcox wrote: > On Fri, Apr 20, 2018 at 03:08:52PM +0200, Michal Hocko wrote: > > > In order to detect these bugs reliably I submit this patch that changes > > > kvmalloc to always use vmalloc if CONFIG_DEBUG_VM is turned on. > > >

Re: [dm-devel] [PATCH v3] kvmalloc: always use vmalloc if CONFIG_DEBUG_SG

2018-04-24 Thread Michal Hocko
On Tue 24-04-18 13:00:11, Mikulas Patocka wrote: > > > On Tue, 24 Apr 2018, Michal Hocko wrote: > > > On Tue 24-04-18 11:50:30, Mikulas Patocka wrote: > > > > > > > > > On Tue, 24 Apr 2018, Michal Hocko wrote: > > > &

Re: [dm-devel] [PATCH] kvmalloc: always use vmalloc if CONFIG_DEBUG_VM

2018-04-24 Thread Michal Hocko
On Tue 24-04-18 10:12:42, Michal Hocko wrote: > On Tue 24-04-18 11:30:40, Mikulas Patocka wrote: > > > > > > On Tue, 24 Apr 2018, Michal Hocko wrote: > > > > > On Mon 23-04-18 20:25:15, Mikulas Patocka wrote: > > > > > > > Fixi

Re: [dm-devel] [PATCH v3] kvmalloc: always use vmalloc if CONFIG_DEBUG_SG

2018-04-24 Thread Michal Hocko
On Tue 24-04-18 13:28:49, Mikulas Patocka wrote: > > > On Tue, 24 Apr 2018, Michal Hocko wrote: > > > On Tue 24-04-18 13:00:11, Mikulas Patocka wrote: > > > > > > > > > On Tue, 24 Apr 2018, Michal Hocko wrote: > > > &

Re: [dm-devel] [PATCH v3] kvmalloc: always use vmalloc if CONFIG_DEBUG_SG

2018-04-24 Thread Michal Hocko
On Tue 24-04-18 11:50:30, Mikulas Patocka wrote: > > > On Tue, 24 Apr 2018, Michal Hocko wrote: > > > On Mon 23-04-18 20:06:16, Mikulas Patocka wrote: > > [...] > > > @@ -404,6 +405,12 @@ void *kvmalloc_node(size_t size, gfp_t f > > >*

Re: [dm-devel] [PATCH] kvmalloc: always use vmalloc if CONFIG_DEBUG_VM

2018-04-23 Thread Michal Hocko
The testing people won't set it up. They install the "kernel-debug" > package and run the tests in it. > > If you introduce a hidden option that no one knows about, no one will use > it. then make sure people know about it. Fuzzers already do test fault injections. -- Michal Hocko SUSE Labs -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH] kvmalloc: always use vmalloc if CONFIG_DEBUG_VM

2018-04-23 Thread Michal Hocko
04:54:53PM -0400, Mikulas Patocka wrote: > > > > > On Fri, 20 Apr 2018, Michal Hocko wrote: > > > > > > No way. This is just wrong! First of all, you will explode most > > > > > > likely > > > > > > on many allocations of small sizes. S

Re: [dm-devel] [PATCH] kvmalloc: always use vmalloc if CONFIG_DEBUG_VM

2018-04-22 Thread Michal Hocko
On Sat 21-04-18 07:47:57, Matthew Wilcox wrote: > On Fri, Apr 20, 2018 at 05:21:26PM -0400, Mikulas Patocka wrote: > > On Fri, 20 Apr 2018, Matthew Wilcox wrote: > > > On Fri, Apr 20, 2018 at 04:54:53PM -0400, Mikulas Patocka wrote: > > > > On Fri, 20 Apr 2018, Michal

Re: [dm-devel] [PATCH v3] kvmalloc: always use vmalloc if CONFIG_DEBUG_SG

2018-04-24 Thread Michal Hocko
; +#ifdef CONFIG_DEBUG_SG > +do_vmalloc: > +#endif > return __vmalloc_node_flags_caller(size, node, flags, > __builtin_return_address(0)); > } -- Michal Hocko SUSE Labs -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH] kvmalloc: always use vmalloc if CONFIG_DEBUG_VM

2018-04-24 Thread Michal Hocko
On Mon 23-04-18 20:25:15, Mikulas Patocka wrote: > > > On Mon, 23 Apr 2018, Michal Hocko wrote: > > > On Mon 23-04-18 10:06:08, Mikulas Patocka wrote: > > > > > > > He didn't want to fix vmalloc(GFP_NOIO) > > > > > > > > I do

Re: [dm-devel] [PATCH v5] fault-injection: introduce kvmalloc fallback options

2018-04-26 Thread Michal Hocko
would explain why the kernel package cannot add a boot option. Maybe there are some but I do not see them right now. -- Michal Hocko SUSE Labs -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] dm bufio: Reduce dm_bufio_lock contention

2018-06-29 Thread Michal Hocko
On Thu 28-06-18 22:43:29, Mikulas Patocka wrote: > > > On Mon, 25 Jun 2018, Michal Hocko wrote: > > > On Mon 25-06-18 10:42:30, Mikulas Patocka wrote: > > > > > > > > > On Mon, 25 Jun 2018, Michal Hocko wrote: > > > > > > > &g

Re: [dm-devel] dm bufio: Reduce dm_bufio_lock contention

2018-06-22 Thread Michal Hocko
On Fri 22-06-18 08:52:09, Mikulas Patocka wrote: > > > On Fri, 22 Jun 2018, Michal Hocko wrote: > > > On Fri 22-06-18 11:01:51, Michal Hocko wrote: > > > On Thu 21-06-18 21:17:24, Mikulas Patocka wrote: > > [...] > > > > What about this patch?

Re: [dm-devel] dm bufio: Reduce dm_bufio_lock contention

2018-06-22 Thread Michal Hocko
On Fri 22-06-18 08:44:52, Mikulas Patocka wrote: > On Fri, 22 Jun 2018, Michal Hocko wrote: [...] > > Why? How are you going to audit all the callers that the behavior makes > > sense and moreover how are you going to ensure that future usage will > > still make sense. The mor

Re: [dm-devel] [PATCH] mm: convert totalram_pages, totalhigh_pages and managed_pages to atomic.

2018-10-23 Thread Michal Hocko
s hard to review manually. -- Michal Hocko SUSE Labs -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] dm bufio: Reduce dm_bufio_lock contention

2018-09-04 Thread Michal Hocko
he soft_limit_reclaim, because of the huge > > number of memory excess of the memcg. > > Then, all the task who do shrink_slab() wait for dm_bufio_lock. > > > > Any suggestions for this?Thanks. > > There's hardly any solution because Michal Hocko r

Re: [dm-devel] [PATCH 6/6] mm: Add memalloc_nowait

2020-06-30 Thread Michal Hocko
ation. Maybe we will learn later that there is just too much unhelpful noise in the kernel log and will reconsider but I wouldn't just start with that. Also we might learn that there will be other modifiers for atomic (or should I say non-sleeping) scopes to be defined. E.g. access to memory reserves but

Re: [dm-devel] [PATCH 6/6] mm: Add memalloc_nowait

2020-06-30 Thread Michal Hocko
On Wed 01-07-20 05:12:03, Matthew Wilcox wrote: > On Tue, Jun 30, 2020 at 08:34:36AM +0200, Michal Hocko wrote: > > On Mon 29-06-20 22:28:30, Matthew Wilcox wrote: > > [...] > > > The documentation is hard to add a new case to, so I rewrote it. What > > > do

Re: [dm-devel] [PATCH 0/6] Overhaul memalloc_no*

2020-06-25 Thread Michal Hocko
ed back. But PF_MEMALLOC_NOFS needs to stay for the scoped NOFS semantic. Hope this clarifies it a bit. -- Michal Hocko SUSE Labs -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH 0/6] Overhaul memalloc_no*

2020-06-29 Thread Michal Hocko
d of many instances where "this is a fs code so it has to use NOFS gfp mask". Some of that has happened and that is really great. On the other hand many people still like to use that api as a workaround for an immediate problem because no-recursion scopes are much harder to recognize unles

Re: [dm-devel] [PATCH 6/6] mm: Add memalloc_nowait

2020-06-29 Thread Michal Hocko
ons: > memalloc_noio_save memalloc_noio_restore > Documentation/core-api/gfp_mask-from-fs-io.rst:allows nesting so it is safe > to call ``memalloc_noio_save`` or The patch is adding memalloc_nowait* and I suspect Mike had that in mind, which would be a fair request. Btw. we are missi

Re: [dm-devel] [PATCH 1/6] mm: Replace PF_MEMALLOC_NOIO with memalloc_noio

2020-06-25 Thread Michal Hocko
AL; > > if (arg2 == 1) > - current->flags |= PR_IO_FLUSHER; > + set_current_io_flusher(); > else if (!arg2) > - current->flags &= ~PR_IO_FLUSHER; > + clear_cu

Re: [dm-devel] [PATCH 2/6] mm: Add become_kswapd and restore_kswapd

2020-06-25 Thread Michal Hocko
the task_struct is about to be destroyed anyway. > > Signed-off-by: Matthew Wilcox (Oracle) Certainly better than an opencoded PF_$FOO manipulation Acked-by: Michal Hocko I would just ask for a clarification because this is rellying to have a good MM knowledge to follow > +/* > +

Re: [dm-devel] [PATCH 6/6] mm: Add memalloc_nowait

2020-06-25 Thread Michal Hocko
t_restore(nowait_flag); This looks confusing though. I am not familiar with alloc_buffer and there is quite some tweaking around __GFP_NORETRY in alloc_buffer_data which I do not follow but GFP_KERNEL just struck my eyes. So why cannot we have alloc_buffer(GFP_NOWAIT | __GFP_NOMEMALLO

Re: [dm-devel] [PATCH 1/6] mm: Replace PF_MEMALLOC_NOIO with memalloc_noio

2020-06-25 Thread Michal Hocko
On Thu 25-06-20 13:34:18, Matthew Wilcox wrote: > On Thu, Jun 25, 2020 at 02:22:39PM +0200, Michal Hocko wrote: > > On Thu 25-06-20 12:31:17, Matthew Wilcox wrote: > > > We're short on PF_* flags, so make memalloc_noio its own bit where we > > > have plenty of space. &g

Re: [dm-devel] [PATCH 6/6] mm: Add memalloc_nowait

2020-06-25 Thread Michal Hocko
On Thu 25-06-20 14:10:55, Matthew Wilcox wrote: > On Thu, Jun 25, 2020 at 02:40:17PM +0200, Michal Hocko wrote: > > On Thu 25-06-20 12:31:22, Matthew Wilcox wrote: > > > Similar to memalloc_noio() and memalloc_nofs(), memalloc_nowait() > > > guarantees we will not sl

Re: [dm-devel] [PATCH 4/6] mm: Replace PF_MEMALLOC_NOFS with memalloc_nofs

2020-06-25 Thread Michal Hocko
On Thu 25-06-20 12:31:20, Matthew Wilcox wrote: > We're short on PF_* flags, so make memalloc_nofs its own bit where we > have plenty of space. > > Signed-off-by: Matthew Wilcox (Oracle) forgot to add Acked-by: Michal Hocko > --- > fs/iomap/buffered-io.c | 2 +- >