[Devel] [PATCH v2 5/6] add user mismatch test

2013-03-12 Thread Glauber Costa
already created containers that will be owned by root:root, and will now try to run it unprivileged. Signed-off-by: Glauber Costa glom...@parallels.com --- src/lib/env.c | 13 + 1 file changed, 13 insertions(+) diff --git a/src/lib/env.c b/src/lib/env.c index 75e2dee..d3b8cd0 100644

[Devel] [PATCH v2 6/6] allow local uid and gid to be specified at container creation

2013-03-12 Thread Glauber Costa
It is a valid use case to run a container with host uid and gid different than the default. This patch provides and documents a way to do so. Signed-off-by: Glauber Costa glom...@parallels.com --- man/vzctl.8.in | 14 ++ src/vzctl-actions.c | 2 ++ src/vzctl.c | 1 + 3

[Devel] [PATCH 2/6] adjust fs_create parameter

2013-03-11 Thread Glauber Costa
We need to pass more information to fs_create. Instead of adding arguments, it is preferred to pass the whole vps_p structure and unfold it inside the callee. Signed-off-by: Glauber Costa glom...@parallels.com --- src/lib/create.c | 13 - 1 file changed, 8 insertions(+), 5 deletions

[Devel] [PATCH 3/6] run modified tar if upstream

2013-03-11 Thread Glauber Costa
If we are running upstream with user namespaces, we need to run a modified version of tar to extract the files. To avoid compatibility problems, we will resort to LD_PRELOAD if a recent tar is not available (if ever). Signed-off-by: Glauber Costa glom...@parallels.com --- scripts/vps-create.in

Re: [Devel] [PATCH 1/6] host uid and gid parameters

2013-03-11 Thread Glauber Costa
@@ -1996,6 +2006,24 @@ static int parse(envid_t veid, vps_param *vps_p, char *val, int id) case PARAM_IPTABLES: ret = parse_iptables(vps_p-res.env, val); break; + +case PARAM_LOCAL_UID: +ret = parse_ul(val, vps_p-res.misc.local_uid); 1 Are we OK

Re: [Devel] [PATCH 4/6] user namespace support for upstream containers

2013-03-11 Thread Glauber Costa
return ret; fill_container_param(arg, create_param); @@ -153,6 +158,44 @@ static int _env_create(void *data) return exec_container_init(arg, create_param); } +static int write_uid_gid_mapping(vps_handler *h, unsigned long uid, unsigned long gid, pid_t pid) +{ +

[Devel] [PATCH 0/2] namespace related fixes for upstream vzctl

2013-02-25 Thread Glauber Costa
Kir, Consider merging the following two patches. They fix two bugs I found while testing vzctl with the latest upstream kernel, which already support the final form of all namespace related technologies. Thanks Glauber Costa (2): vzctl: allow upstream to safely stop container when setns

[Devel] [PATCH 1/2] vzctl: allow upstream to safely stop container when setns is available

2013-02-25 Thread Glauber Costa
-by: Glauber Costa glom...@parallels.com --- src/lib/env.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/env.c b/src/lib/env.c index c185b3f..2da848d 100644 --- a/src/lib/env.c +++ b/src/lib/env.c @@ -797,7 +797,7 @@ static int env_stop(vps_handler *h, envid_t veid, const char

[Devel] [PATCH 2/2] hooks_ct: fix pivot_root failure with mount namespaces.

2013-02-25 Thread Glauber Costa
-off-by: Glauber Costa glom...@parallels.com --- src/lib/hooks_ct.c | 32 +--- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/src/lib/hooks_ct.c b/src/lib/hooks_ct.c index ac6d97e..6f78263 100644 --- a/src/lib/hooks_ct.c +++ b/src/lib/hooks_ct.c @@ -213,13

Re: [Devel] [PATCH 3/6] provide a upstream alternative for ub functions

2012-12-06 Thread Glauber Costa
On 12/06/2012 01:13 PM, Kir Kolyshkin wrote: On 11/26/2012 06:35 PM, Glauber Costa wrote: The ub functions will not be ever supported in upstream kernel. Still, they are called in a quite widespread fashion throughout the code. One possible course of action for this, is to skip

Re: [Devel] [PATCH 2/6] test for vz-kernel support in the target architecture

2012-12-05 Thread Glauber Costa
On 12/05/2012 04:57 PM, Kir Kolyshkin wrote: On 11/26/2012 06:35 PM, Glauber Costa wrote: The include file vzsyscalls.h have a comprehensive list of vz-syscalls and which architectures support them. If that file is included in a not supported architecture, it will trigger an #error, which

[Devel] [PATCH 2/6] test for vz-kernel support in the target architecture

2012-11-26 Thread Glauber Costa
. Signed-off-by: Glauber Costa glom...@parallels.com --- configure.ac | 18 ++ 1 file changed, 18 insertions(+) diff --git a/configure.ac b/configure.ac index 3c04300..59e7384 100644 --- a/configure.ac +++ b/configure.ac @@ -65,6 +65,14 @@ CFLAGS=${CFLAGS} -Wall -Wpointer-arith -Wcast

[Devel] [PATCH 6/6] allow vzsyscalls.h to always be called.

2012-11-26 Thread Glauber Costa
To avoid spreading ifdefs all over, we would like to keep the include of vzsyscalls.h unconditional. So after we already defined VZ_KERNEL_SUPPORTED, we can test for that. We also tweak the message a bit to make it more descriptive. Signed-off-by: Glauber Costa glom...@parallels.com --- include

[Devel] [PATCH 1/6] Don't try to initialize vz-kernel if it is not available.

2012-11-26 Thread Glauber Costa
The vz-kernel is not available in all architectures where mainline Linux can run. If we detect this to be the case, the code won't be compiled. Signed-off-by: Glauber Costa glom...@parallels.com --- src/lib/env.c | 7 +-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/lib

[Devel] [PATCH 5/6] don't compile hooks_vz.c if vz-kernel support is not enabled.

2012-11-26 Thread Glauber Costa
Signed-off-by: Glauber Costa glom...@parallels.com --- src/lib/Makefile.am | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/lib/Makefile.am b/src/lib/Makefile.am index 4502664..42a7ed6 100644 --- a/src/lib/Makefile.am +++ b/src/lib/Makefile.am @@ -40,7 +40,6

[Devel] [PATCH 0/6] Allow vzctl to build on non x86 architectures without vzkernel

2012-11-26 Thread Glauber Costa
. Glauber Costa (6): Don't try to initialize vz-kernel if it is not available. test for vz-kernel support in the target architecture provide a upstream alternative for ub functions don't support vz-specific cpu functions if vz kernel is not present don't compile hooks_vz.c if vz-kernel support

[Devel] [PATCH 3/6] provide a upstream alternative for ub functions

2012-11-26 Thread Glauber Costa
-by: Glauber Costa glom...@parallels.com --- include/ub.h| 43 +++ src/lib/Makefile.am | 5 - 2 files changed, 47 insertions(+), 1 deletion(-) diff --git a/include/ub.h b/include/ub.h index 8155f12..1882b35 100644 --- a/include/ub.h +++ b/include

[Devel] [PATCH 4/6] don't support vz-specific cpu functions if vz kernel is not present

2012-11-26 Thread Glauber Costa
a meaningful error in this case. Signed-off-by: Glauber Costa glom...@parallels.com --- src/lib/cpu.c | 29 +++-- 1 file changed, 23 insertions(+), 6 deletions(-) diff --git a/src/lib/cpu.c b/src/lib/cpu.c index f5d85d1..9ff4d20 100644 --- a/src/lib/cpu.c +++ b/src/lib/cpu.c

[Devel] Re: [PATCH v5 13/18] memcg/sl[au]b Track all the memcg children of a kmem_cache.

2012-10-30 Thread Glauber Costa
On 10/29/2012 07:26 PM, JoonSoo Kim wrote: 2012/10/19 Glauber Costa glom...@parallels.com: +void kmem_cache_destroy_memcg_children(struct kmem_cache *s) +{ + struct kmem_cache *c; + int i; + + if (!s-memcg_params) + return; + if (!s-memcg_params

[Devel] Re: [PATCH v5 11/18] sl[au]b: Allocate objects from memcg cache

2012-10-29 Thread Glauber Costa
On 10/29/2012 07:14 PM, JoonSoo Kim wrote: Hi, Glauber. 2012/10/19 Glauber Costa glom...@parallels.com: We are able to match a cache allocation to a particular memcg. If the task doesn't change groups during the allocation itself - a rare event, this will give us a good picture about who

[Devel] Re: [PATCH v5 08/18] memcg: infrastructure to match an allocation to the right cache

2012-10-25 Thread Glauber Costa
On 10/24/2012 10:10 PM, JoonSoo Kim wrote: 2012/10/19 Glauber Costa glom...@parallels.com: @@ -2930,9 +2937,188 @@ int memcg_register_cache(struct mem_cgroup *memcg, struct kmem_cache *s) void memcg_release_cache(struct kmem_cache *s) { + struct kmem_cache *root; + int id

[Devel] Re: [PATCH v5 06/18] consider a memcg parameter in kmem_create_cache

2012-10-25 Thread Glauber Costa
On 10/23/2012 09:50 PM, JoonSoo Kim wrote: -struct kmem_cache *__kmem_cache_alias(const char *name, size_t size, - size_t align, unsigned long flags, void (*ctor)(void *)) +struct kmem_cache * +__kmem_cache_alias(struct mem_cgroup *memcg, const char *name, size_t size, +

[Devel] Re: [PATCH v5 04/18] slab: don't preemptively remove element from list in cache destroy

2012-10-24 Thread Glauber Costa
On 10/24/2012 10:54 AM, Pekka Enberg wrote: On Mon, Oct 22, 2012 at 11:40 AM, Glauber Costa glom...@parallels.com wrote: On 10/19/2012 11:34 PM, Christoph Lameter wrote: On Fri, 19 Oct 2012, Glauber Costa wrote: I, however, see no reason why we need to do so, since we are now locked during

[Devel] Re: [PATCH v5 06/18] consider a memcg parameter in kmem_create_cache

2012-10-24 Thread Glauber Costa
On 10/23/2012 09:50 PM, JoonSoo Kim wrote: 2012/10/19 Glauber Costa glom...@parallels.com: diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 6a1e096..59f6d54 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -339,6 +339,12 @@ struct mem_cgroup { #if defined(CONFIG_MEMCG_KMEM

[Devel] Re: [PATCH v5 05/18] slab/slub: struct memcg_params

2012-10-24 Thread Glauber Costa
On 10/23/2012 09:25 PM, JoonSoo Kim wrote: Hi, Glauber. 2012/10/19 Glauber Costa glom...@parallels.com: For the kmem slab controller, we need to record some extra information in the kmem_cache structure. Signed-off-by: Glauber Costa glom...@parallels.com Signed-off-by: Suleiman Souhlal

[Devel] Re: [PATCH v5 14/18] memcg/sl[au]b: shrink dead caches

2012-10-22 Thread Glauber Costa
On 10/19/2012 11:47 PM, Christoph Lameter wrote: On Fri, 19 Oct 2012, Glauber Costa wrote: An unlikely branch is used to make sure this case does not affect performance in the usual slab_free path. The slab allocator has a time based reaper that would eventually get rid of the objects

[Devel] Re: [PATCH v5 16/18] slab: propagate tunables values

2012-10-22 Thread Glauber Costa
On 10/19/2012 11:51 PM, Christoph Lameter wrote: On Fri, 19 Oct 2012, Glauber Costa wrote: SLAB allows us to tune a particular cache behavior with tunables. When creating a new memcg cache copy, we'd like to preserve any tunables the parent cache already had. SLAB and SLUB allow tuning

[Devel] Re: [PATCH v5 10/18] sl[au]b: always get the cache from its page in kfree

2012-10-22 Thread Glauber Costa
On 10/19/2012 11:44 PM, Christoph Lameter wrote: On Fri, 19 Oct 2012, Glauber Costa wrote: struct page already have this information. If we start chaining caches, this information will always be more trustworthy than whatever is passed into the function Yes it does but the information

[Devel] Re: [PATCH v5 06/14] memcg: kmem controller infrastructure

2012-10-22 Thread Glauber Costa
On 10/20/2012 12:34 AM, David Rientjes wrote: On Fri, 19 Oct 2012, Glauber Costa wrote: What about gfp __GFP_FS? Do you intend to prevent or allow OOM under that flag? I personally think that anything that accepts to be OOM-killed should have GFP_WAIT set, so that ought to be enough

[Devel] Re: [PATCH v5 06/14] memcg: kmem controller infrastructure

2012-10-22 Thread Glauber Costa
On 10/22/2012 04:51 PM, Michal Hocko wrote: [Sorry for the late reply] On Mon 22-10-12 16:34:15, Glauber Costa wrote: On 10/20/2012 12:34 AM, David Rientjes wrote: On Fri, 19 Oct 2012, Glauber Costa wrote: What about gfp __GFP_FS? Do you intend to prevent or allow OOM under that flag

[Devel] Re: [PATCH v5 15/18] Aggregate memcg cache values in slabinfo

2012-10-22 Thread Glauber Costa
On 10/19/2012 11:50 PM, Christoph Lameter wrote: On Fri, 19 Oct 2012, Glauber Costa wrote: + +/* + * We use suffixes to the name in memcg because we can't have caches + * created in the system with the same name. But when we print them + * locally, better refer to them with the base name

[Devel] Re: [PATCH v5 06/14] memcg: kmem controller infrastructure

2012-10-19 Thread Glauber Costa
On 10/19/2012 02:06 AM, David Rientjes wrote: On Thu, 18 Oct 2012, Glauber Costa wrote: Do we actually need to test PF_KTHREAD when current-mm == NULL? Perhaps because of aio threads whcih temporarily adopt a userspace mm? I believe so. I remember I discussed this in the past with David

[Devel] Re: [PATCH v5 06/14] memcg: kmem controller infrastructure

2012-10-19 Thread Glauber Costa
On 10/19/2012 01:31 PM, David Rientjes wrote: On Fri, 19 Oct 2012, Glauber Costa wrote: Do we actually need to test PF_KTHREAD when current-mm == NULL? Perhaps because of aio threads whcih temporarily adopt a userspace mm? I believe so. I remember I discussed this in the past with David

[Devel] Re: [PATCH v5 06/14] memcg: kmem controller infrastructure

2012-10-19 Thread Glauber Costa
On 10/19/2012 01:59 AM, David Rientjes wrote: On Thu, 18 Oct 2012, Glauber Costa wrote: @@ -2630,6 +2634,171 @@ static void __mem_cgroup_commit_charge(struct mem_cgroup *memcg, memcg_check_events(memcg, page); } +#ifdef CONFIG_MEMCG_KMEM +static inline bool memcg_can_account_kmem

[Devel] [PATCH v5 00/18] slab accounting for memcg

2012-10-19 Thread Glauber Costa
disability and get some decent changelogs in place. For a detailed explanation about this whole effort, please refer to my previous post (https://lkml.org/lkml/2012/10/8/119) Glauber Costa (18): move slabinfo processing to slab_common.c move print_slabinfo_header to slab_common.c sl[au]b: process

[Devel] [PATCH v5 10/18] sl[au]b: always get the cache from its page in kfree

2012-10-19 Thread Glauber Costa
rightful parent. [ v3: added parent testing with VM_BUG_ON ] [ v4: make it faster when kmemcg not in use ] Signed-off-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb

[Devel] [PATCH v5 04/18] slab: don't preemptively remove element from list in cache destroy

2012-10-19 Thread Glauber Costa
are now locked during the whole deletion (which wasn't necessarily true before). I propose a simplification in which we delete it only when there is no more going back, so we don't need to add it again. Signed-off-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC: Pekka

[Devel] [PATCH v5 17/18] slub: slub-specific propagation changes.

2012-10-19 Thread Glauber Costa
, nothing will be propagated. It can also happen that a root cache has its tunables updated during normal system operation. In this case, we will propagate the change to all caches that are already active. Signed-off-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC

[Devel] [PATCH v5 06/18] consider a memcg parameter in kmem_create_cache

2012-10-19 Thread Glauber Costa
cgroup creation to fix a bug ] [ v4: no longer using the index mechanism ] Signed-off-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC: Michal Hocko mho...@suse.cz CC: Kamezawa Hiroyuki kamezawa.hir...@jp.fujitsu.com CC: Johannes

[Devel] [PATCH v5 14/18] memcg/sl[au]b: shrink dead caches

2012-10-19 Thread Glauber Costa
it explicitly, since dead caches are not a likely event. [ v2: also call verify_dead for the slab ] [ v3: use delayed_work to avoid calling verify_dead at every free] Signed-off-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC: Michal

[Devel] [PATCH v5 03/18] sl[au]b: process slabinfo_show in common code

2012-10-19 Thread Glauber Costa
and handling is done from common code. [ v2: moved objects_per_slab and cache_order to slabinfo ] Signed-off-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC: David Rientjes rient...@google.com --- mm/slab.c| 26

[Devel] [PATCH v5 01/18] move slabinfo processing to slab_common.c

2012-10-19 Thread Glauber Costa
This patch moves all the common machinery to slabinfo processing to slab_common.c. We can do better by noticing that the output is heavily common, and having the allocators to just provide finished information about this. But after this first step, this can be done easier. Signed-off-by: Glauber

[Devel] [PATCH v5 15/18] Aggregate memcg cache values in slabinfo

2012-10-19 Thread Glauber Costa
cache. Signed-off-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC: Michal Hocko mho...@suse.cz CC: Kamezawa Hiroyuki kamezawa.hir...@jp.fujitsu.com CC: Johannes Weiner han...@cmpxchg.org CC: Suleiman Souhlal sulei...@google.com

[Devel] [PATCH v5 08/18] memcg: infrastructure to match an allocation to the right cache

2012-10-19 Thread Glauber Costa
-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC: Michal Hocko mho...@suse.cz CC: Kamezawa Hiroyuki kamezawa.hir...@jp.fujitsu.com CC: Johannes Weiner han...@cmpxchg.org CC: Suleiman Souhlal sulei...@google.com CC: Tejun Heo t

[Devel] [PATCH v5 07/18] Allocate memory for memcg caches whenever a new memcg appears

2012-10-19 Thread Glauber Costa
kmem limited memcgs, a natural point for this to happen is when we write to the limit. At that point, we already have set_limit_mutex held, so that will become our natural synchronization mechanism. Signed-off-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC: Pekka

[Devel] [PATCH v5 12/18] memcg: destroy memcg caches

2012-10-19 Thread Glauber Costa
from the cache code. Caches are only destroyed in process context, so we queue them up for later processing in the general case. [ v5: removed cachep backpointer ] Signed-off-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC

[Devel] Re: [PATCH v5 00/14] kmem controller for memcg.

2012-10-18 Thread Glauber Costa
On 10/18/2012 02:11 AM, Andrew Morton wrote: On Tue, 16 Oct 2012 14:16:37 +0400 Glauber Costa glom...@parallels.com wrote: ... A general explanation of what this is all about follows: The kernel memory limitation mechanism for memcg concerns itself with disallowing potentially non

[Devel] Re: [PATCH v5 01/14] memcg: Make it possible to use the stock for more than one page.

2012-10-18 Thread Glauber Costa
On 10/18/2012 02:11 AM, Andrew Morton wrote: On Tue, 16 Oct 2012 14:16:38 +0400 Glauber Costa glom...@parallels.com wrote: From: Suleiman Souhlal ssouh...@freebsd.org We currently have a percpu stock cache scheme that charges one page at a time from memcg-res, the user counter. When

[Devel] Re: [PATCH v5 04/14] kmem accounting basic infrastructure

2012-10-18 Thread Glauber Costa
On 10/18/2012 02:08 AM, David Rientjes wrote: On Tue, 16 Oct 2012, Glauber Costa wrote: This patch adds the basic infrastructure for the accounting of kernel memory. To control that, the following files are created: * memory.kmem.usage_in_bytes * memory.kmem.limit_in_bytes

[Devel] Re: [PATCH v5 04/14] kmem accounting basic infrastructure

2012-10-18 Thread Glauber Costa
On 10/18/2012 02:12 AM, Andrew Morton wrote: On Tue, 16 Oct 2012 14:16:41 +0400 Glauber Costa glom...@parallels.com wrote: This patch adds the basic infrastructure for the accounting of kernel memory. To control that, the following files are created: * memory.kmem.usage_in_bytes

[Devel] Re: [PATCH v5 06/14] memcg: kmem controller infrastructure

2012-10-18 Thread Glauber Costa
On 10/18/2012 02:12 AM, Andrew Morton wrote: On Tue, 16 Oct 2012 14:16:43 +0400 Glauber Costa glom...@parallels.com wrote: This patch introduces infrastructure for tracking kernel memory pages to a given memcg. This will happen whenever the caller includes the flag __GFP_KMEMCG flag

[Devel] Re: [PATCH v5 06/14] memcg: kmem controller infrastructure

2012-10-18 Thread Glauber Costa
On 10/18/2012 02:37 AM, David Rientjes wrote: On Tue, 16 Oct 2012, Glauber Costa wrote: +/* If the test is dying, just let it go. */ +if (unlikely(test_thread_flag(TIF_MEMDIE) + || fatal_signal_pending(current))) +return true; This can

[Devel] Re: [PATCH v5 07/14] mm: Allocate kernel pages to the right memcg

2012-10-18 Thread Glauber Costa
On 10/18/2012 02:12 AM, Andrew Morton wrote: On Tue, 16 Oct 2012 14:16:44 +0400 Glauber Costa glom...@parallels.com wrote: When a process tries to allocate a page with the __GFP_KMEMCG flag, the page allocator will call the corresponding memcg functions to validate the allocation. Tasks

[Devel] Re: [PATCH v5 11/14] memcg: allow a memcg with kmem charges to be destructed.

2012-10-18 Thread Glauber Costa
On 10/18/2012 02:12 AM, Andrew Morton wrote: On Tue, 16 Oct 2012 14:16:48 +0400 Glauber Costa glom...@parallels.com wrote: Because the ultimate goal of the kmem tracking in memcg is to track slab pages as well, It is? For a major patchset such as this, it's pretty important to discuss

[Devel] Re: [PATCH v5 13/14] protect architectures where THREAD_SIZE = PAGE_SIZE against fork bombs

2012-10-18 Thread Glauber Costa
On 10/18/2012 02:12 AM, Andrew Morton wrote: On Tue, 16 Oct 2012 14:16:50 +0400 Glauber Costa glom...@parallels.com wrote: @@ -146,7 +146,7 @@ void __weak arch_release_thread_info(struct thread_info *ti) static struct thread_info *alloc_thread_info_node(struct task_struct *tsk

[Devel] Re: [PATCH v5 14/14] Add documentation about the kmem controller

2012-10-18 Thread Glauber Costa
On 10/18/2012 02:12 AM, Andrew Morton wrote: On Tue, 16 Oct 2012 14:16:51 +0400 Glauber Costa glom...@parallels.com wrote: +Kernel memory won't be accounted at all until limit on a group is set. This +allows for existing setups to continue working without disruption. The limit +cannot

[Devel] Re: [PATCH v5 09/14] memcg: kmem accounting lifecycle management

2012-10-18 Thread Glauber Costa
On 10/18/2012 03:28 AM, David Rientjes wrote: On Tue, 16 Oct 2012, Glauber Costa wrote: diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1182188..e24b388 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -344,6 +344,7 @@ struct mem_cgroup { /* internal only representation about

[Devel] Re: [PATCH v4 00/19] slab accounting for memcg

2012-10-18 Thread Glauber Costa
On 10/12/2012 05:40 PM, Glauber Costa wrote: This is a followup to the previous kmem series. I divided them logically so it gets easier for reviewers. But I believe they are ready to be merged together (although we can do a two-pass merge if people would prefer) Throwaway git tree found

[Devel] Re: [PATCH v5 07/14] mm: Allocate kernel pages to the right memcg

2012-10-18 Thread Glauber Costa
On 10/18/2012 02:12 AM, Andrew Morton wrote: On Tue, 16 Oct 2012 14:16:44 +0400 Glauber Costa glom...@parallels.com wrote: When a process tries to allocate a page with the __GFP_KMEMCG flag, the page allocator will call the corresponding memcg functions to validate the allocation. Tasks

[Devel] [PATCH v5] slab: Ignore internal flags in cache creation

2012-10-17 Thread Glauber Costa
any specific flag requirement should define that to mean all flags. Common code will mask out all flags not belonging to that set. [ v2: leave the mask out decision up to the allocators ] [ v3: define flags for all allocators ] [ v4: move all definitions to slab.h ] Signed-off-by: Glauber Costa

[Devel] [PATCH v5 12/14] execute the whole memcg freeing in free_worker

2012-10-16 Thread Glauber Costa
(). But because the later accesses per-zone info, free_mem_cgroup_per_zone_info() needs to be moved as well. With that, we are left with the per_cpu stats only. Better move it all. Signed-off-by: Glauber Costa glom...@parallels.com Tested-by: Greg Thelen gthe...@google.com Acked-by: Michal Hocko mho...@suse.cz

[Devel] [PATCH v5 02/14] memcg: Reclaim when more than one page needed.

2012-10-16 Thread Glauber Costa
calculation pointed out by Christoph Lameter ] Signed-off-by: Suleiman Souhlal sulei...@google.com Signed-off-by: Glauber Costa glom...@parallels.com Acked-by: Kamezawa Hiroyuki kamezawa.hir...@jp.fujitsu.com Acked-by: Michal Hocko mho...@suse.cz Acked-by: Johannes Weiner han...@cmpxchg.org CC: Tejun

[Devel] [PATCH v5 08/14] res_counter: return amount of charges after res_counter_uncharge

2012-10-16 Thread Glauber Costa
it. This is the same semantics as the atomic variables in the kernel. Since the current return value is void, we don't need to worry about anything breaking due to this change: nobody relied on that, and only users appearing from now on will be checking this value. Signed-off-by: Glauber Costa glom

[Devel] [PATCH v5 00/14] kmem controller for memcg.

2012-10-16 Thread Glauber Costa
, existing deployments will see no change in behavior. Glauber Costa (12): memcg: change defines to an enum kmem accounting basic infrastructure Add a __GFP_KMEMCG flag memcg: kmem controller infrastructure mm: Allocate kernel pages to the right memcg res_counter: return amount of charges

[Devel] [PATCH v5 03/14] memcg: change defines to an enum

2012-10-16 Thread Glauber Costa
This is just a cleanup patch for clarity of expression. In earlier submissions, people asked it to be in a separate patch, so here it is. [ v2: use named enum as type throughout the file as well ] Signed-off-by: Glauber Costa glom...@parallels.com Acked-by: Kamezawa Hiroyuki kamezawa.hir

[Devel] [PATCH v5 01/14] memcg: Make it possible to use the stock for more than one page.

2012-10-16 Thread Glauber Costa
draw from the user counter, and can be bigger than a single page, as it is the case with the stack (usually 2 pages) or some higher order slabs. [ glom...@parallels.com: added a changelog ] Signed-off-by: Suleiman Souhlal sulei...@google.com Signed-off-by: Glauber Costa glom...@parallels.com Acked

[Devel] [PATCH v5 04/14] kmem accounting basic infrastructure

2012-10-16 Thread Glauber Costa
memory) [ v4: make kmem files part of the main array; do not allow limit to be set for non-empty cgroups ] [ v5: cosmetic changes ] Signed-off-by: Glauber Costa glom...@parallels.com Acked-by: Kamezawa Hiroyuki kamezawa.hir...@jp.fujitsu.com CC: Michal Hocko mho...@suse.cz CC: Johannes Weiner

[Devel] [PATCH v5 11/14] memcg: allow a memcg with kmem charges to be destructed.

2012-10-16 Thread Glauber Costa
. Because of that, when we destroy a memcg, we only make sure the destruction will succeed by discounting the kmem charges from the user charges when we try to empty the cgroup. Signed-off-by: Glauber Costa glom...@parallels.com Acked-by: Kamezawa Hiroyuki kamezawa.hir...@jp.fujitsu.com Reviewed

[Devel] [PATCH v5 06/14] memcg: kmem controller infrastructure

2012-10-16 Thread Glauber Costa
code for kmemcg compiled out and core functions in memcontrol.c, moved kmem code to the middle to avoid forward decls ] Signed-off-by: Glauber Costa glom...@parallels.com Acked-by: Michal Hocko mho...@suse.cz CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC

[Devel] [PATCH v5 09/14] memcg: kmem accounting lifecycle management

2012-10-16 Thread Glauber Costa
better than a reference count decrease at every operation. [ v3: merged all lifecycle related patches in one ] [ v5: changed memcg_kmem_dead's name ] Signed-off-by: Glauber Costa glom...@parallels.com Acked-by: Michal Hocko mho...@suse.cz Acked-by: Kamezawa Hiroyuki kamezawa.hir...@jp.fujitsu.com

[Devel] [PATCH v5 10/14] memcg: use static branches when code not in use

2012-10-16 Thread Glauber Costa
in kmem_accounted ] Signed-off-by: Glauber Costa glom...@parallels.com Acked-by: Michal Hocko mho...@suse.cz Acked-by: Kamezawa Hiroyuki kamezawa.hir...@jp.fujitsu.com CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC: Johannes Weiner han...@cmpxchg.org CC: Suleiman Souhlal

[Devel] [PATCH v5 14/14] Add documentation about the kmem controller

2012-10-16 Thread Glauber Costa
Signed-off-by: Glauber Costa glom...@parallels.com CC: Frederic Weisbecker fweis...@redhat.com CC: Kamezawa Hiroyuki kamezawa.hir...@jp.fujitsu.com CC: Michal Hocko mho...@suse.cz CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC: Johannes Weiner han...@cmpxchg.org CC

[Devel] [PATCH v5 05/14] Add a __GFP_KMEMCG flag

2012-10-16 Thread Glauber Costa
-off-by: Glauber Costa glom...@parallels.com Acked-by: Johannes Weiner han...@cmpxchg.org Acked-by: Rik van Riel r...@redhat.com Acked-by: Mel Gorman m...@csn.ul.ie Acked-by: Kamezawa Hiroyuki kamezawa.hir...@jp.fujitsu.com CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi

[Devel] [PATCH v5 13/14] protect architectures where THREAD_SIZE = PAGE_SIZE against fork bombs

2012-10-16 Thread Glauber Costa
. For the time being, I am defining a new variant of THREADINFO_GFP, not to mess with the other path. Once the slab is also tracked by memcg, we can get rid of that flag. Tested to successfully protect against :(){ :|: };: Signed-off-by: Glauber Costa glom...@parallels.com Acked-by: Frederic Weisbecker

[Devel] [PATCH v5 07/14] mm: Allocate kernel pages to the right memcg

2012-10-16 Thread Glauber Costa
: inverted test order to avoid a memcg_get leak, free_accounted_pages simplification ] [ v4: test for TIF_MEMDIE at newpage_charge ] Signed-off-by: Glauber Costa glom...@parallels.com Acked-by: Michal Hocko mho...@suse.cz Acked-by: Mel Gorman mgor...@suse.de Acked-by: Kamezawa Hiroyuki kamezawa.hir

[Devel] Re: [PATCH v5 07/14] mm: Allocate kernel pages to the right memcg

2012-10-16 Thread Glauber Costa
On 10/16/2012 07:31 PM, Christoph Lameter wrote: On Tue, 16 Oct 2012, Glauber Costa wrote: To avoid adding markers to the page - and a kmem flag that would necessarily follow, as much as doing page_cgroup lookups for no reason, whoever is marking its allocations with __GFP_KMEMCG flag

[Devel] Re: [PATCH v4 04/14] kmem accounting basic infrastructure

2012-10-12 Thread Glauber Costa
On 10/11/2012 02:11 PM, Michal Hocko wrote: On Mon 08-10-12 14:06:10, Glauber Costa wrote: This patch adds the basic infrastructure for the accounting of the slab caches. To control that, the following files are created: * memory.kmem.usage_in_bytes * memory.kmem.limit_in_bytes

[Devel] Re: [PATCH v4 06/14] memcg: kmem controller infrastructure

2012-10-12 Thread Glauber Costa
On 10/11/2012 04:42 PM, Michal Hocko wrote: On Mon 08-10-12 14:06:12, Glauber Costa wrote: This patch introduces infrastructure for tracking kernel memory pages to a given memcg. This will happen whenever the caller includes the flag __GFP_KMEMCG flag, and the task belong to a memcg other than

[Devel] Re: [PATCH v4 09/14] memcg: kmem accounting lifecycle management

2012-10-12 Thread Glauber Costa
On 10/11/2012 05:11 PM, Michal Hocko wrote: On Mon 08-10-12 14:06:15, Glauber Costa wrote: Because kmem charges can outlive the cgroup, we need to make sure that we won't free the memcg structure while charges are still in flight. For reviewing simplicity, the charge functions will issue

[Devel] Re: [PATCH v4 10/14] memcg: use static branches when code not in use

2012-10-12 Thread Glauber Costa
On 10/11/2012 05:40 PM, Michal Hocko wrote: On Mon 08-10-12 14:06:16, Glauber Costa wrote: We can use static branches to patch the code in or out when not used. Because the _ACTIVE bit on kmem_accounted is only set after the increment is done, we guarantee that the root memcg will always

[Devel] Re: [PATCH v4 14/14] Add documentation about the kmem controller

2012-10-12 Thread Glauber Costa
On 10/11/2012 06:35 PM, Michal Hocko wrote: On Mon 08-10-12 14:06:20, Glauber Costa wrote: Signed-off-by: Glauber Costa glom...@parallels.com --- Documentation/cgroups/memory.txt | 55 +++- 1 file changed, 54 insertions(+), 1 deletion(-) diff --git

[Devel] Re: [PATCH v4 06/14] memcg: kmem controller infrastructure

2012-10-12 Thread Glauber Costa
On 10/12/2012 12:39 PM, Michal Hocko wrote: On Fri 12-10-12 11:45:46, Glauber Costa wrote: On 10/11/2012 04:42 PM, Michal Hocko wrote: On Mon 08-10-12 14:06:12, Glauber Costa wrote: [...] + /* + * Conditions under which we can wait for the oom_killer. + * __GFP_NORETRY should be masked

[Devel] Re: [PATCH v4 06/14] memcg: kmem controller infrastructure

2012-10-12 Thread Glauber Costa
On 10/12/2012 12:57 PM, Michal Hocko wrote: On Fri 12-10-12 12:44:57, Glauber Costa wrote: On 10/12/2012 12:39 PM, Michal Hocko wrote: On Fri 12-10-12 11:45:46, Glauber Costa wrote: On 10/11/2012 04:42 PM, Michal Hocko wrote: On Mon 08-10-12 14:06:12, Glauber Costa wrote

[Devel] [PATCH v4 00/19] slab accounting for memcg

2012-10-12 Thread Glauber Costa
changelogs in place. For a detailed explanation about this whole effort, please refer to my previous post (https://lkml.org/lkml/2012/10/8/119) *** BLURB HERE *** Glauber Costa (19): slab: Ignore internal flags in cache creation move slabinfo processing to slab_common.c move

[Devel] [PATCH v4 12/19] sl[au]b: Allocate objects from memcg cache

2012-10-12 Thread Glauber Costa
memcg_kmem_get_cache() before all the cache allocations. Signed-off-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC: Michal Hocko mho...@suse.cz CC: Kamezawa Hiroyuki kamezawa.hir...@jp.fujitsu.com CC: Johannes Weiner han...@cmpxchg.org

[Devel] [PATCH v4 19/19] Add slab-specific documentation about the kmem controller

2012-10-12 Thread Glauber Costa
Signed-off-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC: Michal Hocko mho...@suse.cz CC: Kamezawa Hiroyuki kamezawa.hir...@jp.fujitsu.com CC: Johannes Weiner han...@cmpxchg.org CC: Suleiman Souhlal sulei...@google.com CC

[Devel] [PATCH v4 15/19] memcg/sl[au]b: shrink dead caches

2012-10-12 Thread Glauber Costa
it explicitly, since dead caches are not a likely event. [ v2: also call verify_dead for the slab ] [ v3: use delayed_work to avoid calling verify_dead at every free] Signed-off-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC: Michal

[Devel] [PATCH v4 10/19] memcg: skip memcg kmem allocations in specified code regions

2012-10-12 Thread Glauber Costa
cache creation, when we allocate data using caches that are not necessarily created already. [ v2: wrap the whole enqueue process, INIT_WORK can alloc memory ] Signed-off-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC: Michal

[Devel] [PATCH v4 13/19] memcg: destroy memcg caches

2012-10-12 Thread Glauber Costa
from the cache code. Caches are only destroyed in process context, so we queue them up for later processing in the general case. Signed-off-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC: Michal Hocko mho...@suse.cz CC

[Devel] [PATCH v4 05/19] slab: don't preemptively remove element from list in cache destroy

2012-10-12 Thread Glauber Costa
are now locked during the whole deletion (which wasn't necessarily true before). I propose a simplification in which we delete it only when there is no more going back, so we don't need to add it again. Signed-off-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC: Pekka

[Devel] [PATCH v4 04/19] sl[au]b: process slabinfo_show in common code

2012-10-12 Thread Glauber Costa
and handling is done from common code. [ v2: moved objects_per_slab and cache_order to slabinfo ] Signed-off-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC: David Rientjes rient...@google.com --- mm/slab.c| 26

[Devel] [PATCH v4 09/19] memcg: infrastructure to match an allocation to the right cache

2012-10-12 Thread Glauber Costa
-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC: Michal Hocko mho...@suse.cz CC: Kamezawa Hiroyuki kamezawa.hir...@jp.fujitsu.com CC: Johannes Weiner han...@cmpxchg.org CC: Suleiman Souhlal sulei...@google.com CC: Tejun Heo t

[Devel] [PATCH v4 08/19] Allocate memory for memcg caches whenever a new memcg appears

2012-10-12 Thread Glauber Costa
kmem limited memcgs, a natural point for this to happen is when we write to the limit. At that point, we already have set_limit_mutex held, so that will become our natural synchronization mechanism. Signed-off-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC: Pekka

[Devel] [PATCH v4 02/19] move slabinfo processing to slab_common.c

2012-10-12 Thread Glauber Costa
This patch moves all the common machinery to slabinfo processing to slab_common.c. We can do better by noticing that the output is heavily common, and having the allocators to just provide finished information about this. But after this first step, this can be done easier. Signed-off-by: Glauber

[Devel] [PATCH v4 17/19] slab: propagate tunables values

2012-10-12 Thread Glauber Costa
this to be already set - which memcg_kmem_register_cache will do - when we reach __kmem_cache_create() Signed-off-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC: Michal Hocko mho...@suse.cz CC: Kamezawa Hiroyuki kamezawa.hir

[Devel] [PATCH v4 06/19] slab/slub: struct memcg_params

2012-10-12 Thread Glauber Costa
For the kmem slab controller, we need to record some extra information in the kmem_cache structure. Signed-off-by: Glauber Costa glom...@parallels.com Signed-off-by: Suleiman Souhlal sulei...@google.com CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC: Michal Hocko

[Devel] [PATCH v4 03/19] move print_slabinfo_header to slab_common.c

2012-10-12 Thread Glauber Costa
-visible CONFIG_DEBUG_SLAB switch, we can move the header printing to a common location. Signed-off-by: Glauber Costa glom...@parallels.com Acked-by: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC: David Rientjes rient...@google.com --- mm/slab.c| 24

[Devel] [PATCH v4 11/19] sl[au]b: always get the cache from its page in kfree

2012-10-12 Thread Glauber Costa
rightful parent. [ v3: added parent testing with VM_BUG_ON ] [ v4: make it faster when kmemcg not in use ] Signed-off-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb

[Devel] [PATCH v4 16/19] Aggregate memcg cache values in slabinfo

2012-10-12 Thread Glauber Costa
cache. Signed-off-by: Glauber Costa glom...@parallels.com CC: Christoph Lameter c...@linux.com CC: Pekka Enberg penb...@cs.helsinki.fi CC: Michal Hocko mho...@suse.cz CC: Kamezawa Hiroyuki kamezawa.hir...@jp.fujitsu.com CC: Johannes Weiner han...@cmpxchg.org CC: Suleiman Souhlal sulei...@google.com

[Devel] [PATCH v4 01/19] slab: Ignore internal flags in cache creation

2012-10-12 Thread Glauber Costa
any specific flag requirement should define that to mean all flags. Common code will mask out all flags not belonging to that set. [ v2: leave the mask out decision up to the allocators ] [ v3: define flags for all allocators ] [ v4: move all definitions to slab.h ] Signed-off-by: Glauber Costa

<    1   2   3   4   5   6   7   8   9   10   >