Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On 2015/3/2 17:12, Peter Zijlstra wrote: > On Mon, Mar 02, 2015 at 02:15:39PM +0800, Zefan Li wrote: >> Hi Rik, >> >>> Subject: cpusets,isolcpus: add file to show isolated cpus in cpuset >>> >>> The previous patch makes it so the code skips over isolcpus when >>> building scheduler load balancing domains. This makes it hard to >>> see for a user which of the CPUs in a cpuset are participating in >>> load balancing, and which ones are isolated cpus. >>> >>> Add a cpuset.isolcpus file with info on which cpus in a cpuset are >>> isolated CPUs. >>> >>> This file is read-only for now. In the future we could extend things >>> so isolcpus can be changed at run time, for the root (system wide) >>> cpuset only. >>> >> >> One Question, why not add a /sys/devices/system/cpu/isolated instead? > > It would leave userspace to calculate the result for any one cpuset > itself. It's trivial. Instead of reading cpuset.isolcpus, now we read cpuset.cpus and /sys/.../isolated. > Furthermore, is that /sys thing visible for all nested > containers? > . > Never tried nested containers, but I think so. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On 2015/3/2 17:12, Peter Zijlstra wrote: On Mon, Mar 02, 2015 at 02:15:39PM +0800, Zefan Li wrote: Hi Rik, Subject: cpusets,isolcpus: add file to show isolated cpus in cpuset The previous patch makes it so the code skips over isolcpus when building scheduler load balancing domains. This makes it hard to see for a user which of the CPUs in a cpuset are participating in load balancing, and which ones are isolated cpus. Add a cpuset.isolcpus file with info on which cpus in a cpuset are isolated CPUs. This file is read-only for now. In the future we could extend things so isolcpus can be changed at run time, for the root (system wide) cpuset only. One Question, why not add a /sys/devices/system/cpu/isolated instead? It would leave userspace to calculate the result for any one cpuset itself. It's trivial. Instead of reading cpuset.isolcpus, now we read cpuset.cpus and /sys/.../isolated. Furthermore, is that /sys thing visible for all nested containers? . Never tried nested containers, but I think so. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Mon, 2015-03-02 at 11:09 -0500, Tejun Heo wrote: > On Mon, Mar 02, 2015 at 05:02:57PM +0100, Mike Galbraith wrote: > > Well, those are the only ones we can do anything about. Dirt simple > > diddling of the workqueue default mask as sched domains are > > added/removed should do it I think. Automatically moving any existing > > unbound worker away from isolated cores at the same time would be a > > bonus, most important is that no new threads sneak in. > > Worker pools are immutable once created and configuraiton changes are > achieved by creating new pools and draining old ones but at any rate > making it follow config changes is almost trivial. Figuring out > configuration policy might take a bit of effort tho. Can you point me > to what specific configuration it should be following? For cpusets, an exclusive set should become taboo to unbound workers when load balancing is turned off. The user making sched domains go away is a not so subtle hint that he wants no random interference, as he is trying to assume full responsibility for task placement therein. In my trees, I let the user turn rt cpupri/push/pull off as well, as that further reduces jitter. -Mike -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Mon, Mar 02, 2015 at 12:01:16PM -0500, Tejun Heo wrote: > Applied 1-2 to cgroup/for-4.1. Reverted due to build failure. Looks like UP build is broken. Thanks. -- tejun -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Thu, Feb 26, 2015 at 12:12:31PM -0500, Rik van Riel wrote: > On Thu, 26 Feb 2015 19:05:57 +0800 > Zefan Li wrote: > > > Make it return -ENOMEM ? Or make it a global variable and allocate memory > > for it > > in cpuset_init(). > > Here you are. This addresses your concern, as well as the > issue David Rientjes found earlier. > > ---8<--- > > Subject: cpusets,isolcpus: add file to show isolated cpus in cpuset > > The previous patch makes it so the code skips over isolcpus when > building scheduler load balancing domains. This makes it hard to > see for a user which of the CPUs in a cpuset are participating in > load balancing, and which ones are isolated cpus. > > Add a cpuset.isolcpus file with info on which cpus in a cpuset are > isolated CPUs. > > This file is read-only for now. In the future we could extend things > so isolcpus can be changed at run time, for the root (system wide) > cpuset only. > > Acked-by: David Rientjes > Cc: Peter Zijlstra > Cc: Clark Williams > Cc: Li Zefan > Cc: Ingo Molnar > Cc: Luiz Capitulino > Cc: David Rientjes > Cc: Mike Galbraith > Cc: cgro...@vger.kernel.org > Signed-off-by: Rik van Riel Applied 1-2 to cgroup/for-4.1. Thanks. -- tejun -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Mon, Mar 02, 2015 at 05:02:57PM +0100, Mike Galbraith wrote: > Well, those are the only ones we can do anything about. Dirt simple > diddling of the workqueue default mask as sched domains are > added/removed should do it I think. Automatically moving any existing > unbound worker away from isolated cores at the same time would be a > bonus, most important is that no new threads sneak in. Worker pools are immutable once created and configuraiton changes are achieved by creating new pools and draining old ones but at any rate making it follow config changes is almost trivial. Figuring out configuration policy might take a bit of effort tho. Can you point me to what specific configuration it should be following? Thanks. -- tejun -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Mon, 2015-03-02 at 10:29 -0500, Tejun Heo wrote: > On Mon, Mar 02, 2015 at 01:44:50PM +0100, Mike Galbraith wrote: > > Hm, I'm now all system-disease-ified now (still hate the bloody thing), > > and have no problem isolating cpus via cpusets, modulo workqueues > > wanting a bat upside the head. > > It shouldn't be difficult to teach workqueue pools to follow the same > rules. This matters only for the unbound ones anyway, right? Well, those are the only ones we can do anything about. Dirt simple diddling of the workqueue default mask as sched domains are added/removed should do it I think. Automatically moving any existing unbound worker away from isolated cores at the same time would be a bonus, most important is that no new threads sneak in. -Mike -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Mon, Mar 02, 2015 at 01:44:50PM +0100, Mike Galbraith wrote: > Hm, I'm now all system-disease-ified now (still hate the bloody thing), > and have no problem isolating cpus via cpusets, modulo workqueues > wanting a bat upside the head. It shouldn't be difficult to teach workqueue pools to follow the same rules. This matters only for the unbound ones anyway, right? Thanks. -- tejun -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Mon, 2015-03-02 at 09:35 -0500, Rik van Riel wrote: > On 03/02/2015 07:44 AM, Mike Galbraith wrote: > > On Mon, 2015-03-02 at 10:09 +0100, Peter Zijlstra wrote: > >> On Thu, Feb 26, 2015 at 12:12:31PM -0500, Rik van Riel wrote: > >>> Subject: cpusets,isolcpus: add file to show isolated cpus in cpuset > >>> > >>> The previous patch makes it so the code skips over isolcpus when > >>> building scheduler load balancing domains. This makes it hard to > >>> see for a user which of the CPUs in a cpuset are participating in > >>> load balancing, and which ones are isolated cpus. > >>> > >>> Add a cpuset.isolcpus file with info on which cpus in a cpuset are > >>> isolated CPUs. > >>> > >>> This file is read-only for now. In the future we could extend things > >>> so isolcpus can be changed at run time, for the root (system wide) > >>> cpuset only. > >>> > >>> Acked-by: David Rientjes > >>> Cc: Peter Zijlstra > >>> Cc: Clark Williams > >>> Cc: Li Zefan > >>> Cc: Ingo Molnar > >>> Cc: Luiz Capitulino > >>> Cc: David Rientjes > >>> Cc: Mike Galbraith > >>> Cc: cgro...@vger.kernel.org > >>> Signed-off-by: Rik van Riel > >> > >> So let me start off by saying I hate isolcpus ;-) > >> > >> Let me further state that I had hopes we could extend cpusets to > >> natively provide the functionality isolcpus has, and kill isolcpus. > > > > +1 > > > > That's where nohz_full goop belongs too. > > Except nohz_full and isolcpus are very much global attributes of > each CPU, so I am not sure whether it would make sense to allow > configuration of this attribute anywhere other than the root > cpuset. They're attributes of exclusive sets, which excludes the root set. It'd be kinda hard to have the root set be both ticked and tickless :) -Mike -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On 03/02/2015 07:44 AM, Mike Galbraith wrote: > On Mon, 2015-03-02 at 10:09 +0100, Peter Zijlstra wrote: >> On Thu, Feb 26, 2015 at 12:12:31PM -0500, Rik van Riel wrote: >>> Subject: cpusets,isolcpus: add file to show isolated cpus in cpuset >>> >>> The previous patch makes it so the code skips over isolcpus when >>> building scheduler load balancing domains. This makes it hard to >>> see for a user which of the CPUs in a cpuset are participating in >>> load balancing, and which ones are isolated cpus. >>> >>> Add a cpuset.isolcpus file with info on which cpus in a cpuset are >>> isolated CPUs. >>> >>> This file is read-only for now. In the future we could extend things >>> so isolcpus can be changed at run time, for the root (system wide) >>> cpuset only. >>> >>> Acked-by: David Rientjes >>> Cc: Peter Zijlstra >>> Cc: Clark Williams >>> Cc: Li Zefan >>> Cc: Ingo Molnar >>> Cc: Luiz Capitulino >>> Cc: David Rientjes >>> Cc: Mike Galbraith >>> Cc: cgro...@vger.kernel.org >>> Signed-off-by: Rik van Riel >> >> So let me start off by saying I hate isolcpus ;-) >> >> Let me further state that I had hopes we could extend cpusets to >> natively provide the functionality isolcpus has, and kill isolcpus. > > +1 > > That's where nohz_full goop belongs too. Except nohz_full and isolcpus are very much global attributes of each CPU, so I am not sure whether it would make sense to allow configuration of this attribute anywhere other than the root cpuset. -- All rights reversed -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Mon, 2015-03-02 at 10:09 +0100, Peter Zijlstra wrote: > On Thu, Feb 26, 2015 at 12:12:31PM -0500, Rik van Riel wrote: > > Subject: cpusets,isolcpus: add file to show isolated cpus in cpuset > > > > The previous patch makes it so the code skips over isolcpus when > > building scheduler load balancing domains. This makes it hard to > > see for a user which of the CPUs in a cpuset are participating in > > load balancing, and which ones are isolated cpus. > > > > Add a cpuset.isolcpus file with info on which cpus in a cpuset are > > isolated CPUs. > > > > This file is read-only for now. In the future we could extend things > > so isolcpus can be changed at run time, for the root (system wide) > > cpuset only. > > > > Acked-by: David Rientjes > > Cc: Peter Zijlstra > > Cc: Clark Williams > > Cc: Li Zefan > > Cc: Ingo Molnar > > Cc: Luiz Capitulino > > Cc: David Rientjes > > Cc: Mike Galbraith > > Cc: cgro...@vger.kernel.org > > Signed-off-by: Rik van Riel > > So let me start off by saying I hate isolcpus ;-) > > Let me further state that I had hopes we could extend cpusets to > natively provide the functionality isolcpus has, and kill isolcpus. +1 That's where nohz_full goop belongs too. > The 'normal' way would be to create 2 cgroups with disjoint cpus, > disable sched_load_balance on root and one of the siblings, while moving > everything into the other group. That's what cset shield does, works fine. > The 'problem' is that we cannot move everything that is affected by > isolcpus, workqueues have grown a horrible 'new' interface outside of > the regular task interfaces and things like kthreadd are non-movable for > mostly good reasons. > > Furthermore it appears that software like system-disease and libvirt > hard assume they're lord and master of the cgroup hierarchy and do not > expect things like this. > > So while I mostly hate all of this it might be the best we can do :-( Hm, I'm now all system-disease-ified now (still hate the bloody thing), and have no problem isolating cpus via cpusets, modulo workqueues wanting a bat upside the head. -Mike -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Mon, Mar 02, 2015 at 02:15:39PM +0800, Zefan Li wrote: > Hi Rik, > > > Subject: cpusets,isolcpus: add file to show isolated cpus in cpuset > > > > The previous patch makes it so the code skips over isolcpus when > > building scheduler load balancing domains. This makes it hard to > > see for a user which of the CPUs in a cpuset are participating in > > load balancing, and which ones are isolated cpus. > > > > Add a cpuset.isolcpus file with info on which cpus in a cpuset are > > isolated CPUs. > > > > This file is read-only for now. In the future we could extend things > > so isolcpus can be changed at run time, for the root (system wide) > > cpuset only. > > > > One Question, why not add a /sys/devices/system/cpu/isolated instead? It would leave userspace to calculate the result for any one cpuset itself. Furthermore, is that /sys thing visible for all nested containers? -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Thu, Feb 26, 2015 at 12:12:31PM -0500, Rik van Riel wrote: > Subject: cpusets,isolcpus: add file to show isolated cpus in cpuset > > The previous patch makes it so the code skips over isolcpus when > building scheduler load balancing domains. This makes it hard to > see for a user which of the CPUs in a cpuset are participating in > load balancing, and which ones are isolated cpus. > > Add a cpuset.isolcpus file with info on which cpus in a cpuset are > isolated CPUs. > > This file is read-only for now. In the future we could extend things > so isolcpus can be changed at run time, for the root (system wide) > cpuset only. > > Acked-by: David Rientjes > Cc: Peter Zijlstra > Cc: Clark Williams > Cc: Li Zefan > Cc: Ingo Molnar > Cc: Luiz Capitulino > Cc: David Rientjes > Cc: Mike Galbraith > Cc: cgro...@vger.kernel.org > Signed-off-by: Rik van Riel So let me start off by saying I hate isolcpus ;-) Let me further state that I had hopes we could extend cpusets to natively provide the functionality isolcpus has, and kill isolcpus. The 'normal' way would be to create 2 cgroups with disjoint cpus, disable sched_load_balance on root and one of the siblings, while moving everything into the other group. The 'problem' is that we cannot move everything that is affected by isolcpus, workqueues have grown a horrible 'new' interface outside of the regular task interfaces and things like kthreadd are non-movable for mostly good reasons. Furthermore it appears that software like system-disease and libvirt hard assume they're lord and master of the cgroup hierarchy and do not expect things like this. So while I mostly hate all of this it might be the best we can do :-( -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Mon, Mar 02, 2015 at 02:15:39PM +0800, Zefan Li wrote: Hi Rik, Subject: cpusets,isolcpus: add file to show isolated cpus in cpuset The previous patch makes it so the code skips over isolcpus when building scheduler load balancing domains. This makes it hard to see for a user which of the CPUs in a cpuset are participating in load balancing, and which ones are isolated cpus. Add a cpuset.isolcpus file with info on which cpus in a cpuset are isolated CPUs. This file is read-only for now. In the future we could extend things so isolcpus can be changed at run time, for the root (system wide) cpuset only. One Question, why not add a /sys/devices/system/cpu/isolated instead? It would leave userspace to calculate the result for any one cpuset itself. Furthermore, is that /sys thing visible for all nested containers? -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On 03/02/2015 07:44 AM, Mike Galbraith wrote: On Mon, 2015-03-02 at 10:09 +0100, Peter Zijlstra wrote: On Thu, Feb 26, 2015 at 12:12:31PM -0500, Rik van Riel wrote: Subject: cpusets,isolcpus: add file to show isolated cpus in cpuset The previous patch makes it so the code skips over isolcpus when building scheduler load balancing domains. This makes it hard to see for a user which of the CPUs in a cpuset are participating in load balancing, and which ones are isolated cpus. Add a cpuset.isolcpus file with info on which cpus in a cpuset are isolated CPUs. This file is read-only for now. In the future we could extend things so isolcpus can be changed at run time, for the root (system wide) cpuset only. Acked-by: David Rientjes rient...@google.com Cc: Peter Zijlstra pet...@infradead.org Cc: Clark Williams willi...@redhat.com Cc: Li Zefan lize...@huawei.com Cc: Ingo Molnar mi...@redhat.com Cc: Luiz Capitulino lcapitul...@redhat.com Cc: David Rientjes rient...@google.com Cc: Mike Galbraith umgwanakikb...@gmail.com Cc: cgro...@vger.kernel.org Signed-off-by: Rik van Riel r...@redhat.com So let me start off by saying I hate isolcpus ;-) Let me further state that I had hopes we could extend cpusets to natively provide the functionality isolcpus has, and kill isolcpus. +1 That's where nohz_full goop belongs too. Except nohz_full and isolcpus are very much global attributes of each CPU, so I am not sure whether it would make sense to allow configuration of this attribute anywhere other than the root cpuset. -- All rights reversed -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Mon, 2015-03-02 at 09:35 -0500, Rik van Riel wrote: On 03/02/2015 07:44 AM, Mike Galbraith wrote: On Mon, 2015-03-02 at 10:09 +0100, Peter Zijlstra wrote: On Thu, Feb 26, 2015 at 12:12:31PM -0500, Rik van Riel wrote: Subject: cpusets,isolcpus: add file to show isolated cpus in cpuset The previous patch makes it so the code skips over isolcpus when building scheduler load balancing domains. This makes it hard to see for a user which of the CPUs in a cpuset are participating in load balancing, and which ones are isolated cpus. Add a cpuset.isolcpus file with info on which cpus in a cpuset are isolated CPUs. This file is read-only for now. In the future we could extend things so isolcpus can be changed at run time, for the root (system wide) cpuset only. Acked-by: David Rientjes rient...@google.com Cc: Peter Zijlstra pet...@infradead.org Cc: Clark Williams willi...@redhat.com Cc: Li Zefan lize...@huawei.com Cc: Ingo Molnar mi...@redhat.com Cc: Luiz Capitulino lcapitul...@redhat.com Cc: David Rientjes rient...@google.com Cc: Mike Galbraith umgwanakikb...@gmail.com Cc: cgro...@vger.kernel.org Signed-off-by: Rik van Riel r...@redhat.com So let me start off by saying I hate isolcpus ;-) Let me further state that I had hopes we could extend cpusets to natively provide the functionality isolcpus has, and kill isolcpus. +1 That's where nohz_full goop belongs too. Except nohz_full and isolcpus are very much global attributes of each CPU, so I am not sure whether it would make sense to allow configuration of this attribute anywhere other than the root cpuset. They're attributes of exclusive sets, which excludes the root set. It'd be kinda hard to have the root set be both ticked and tickless :) -Mike -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Mon, 2015-03-02 at 10:09 +0100, Peter Zijlstra wrote: On Thu, Feb 26, 2015 at 12:12:31PM -0500, Rik van Riel wrote: Subject: cpusets,isolcpus: add file to show isolated cpus in cpuset The previous patch makes it so the code skips over isolcpus when building scheduler load balancing domains. This makes it hard to see for a user which of the CPUs in a cpuset are participating in load balancing, and which ones are isolated cpus. Add a cpuset.isolcpus file with info on which cpus in a cpuset are isolated CPUs. This file is read-only for now. In the future we could extend things so isolcpus can be changed at run time, for the root (system wide) cpuset only. Acked-by: David Rientjes rient...@google.com Cc: Peter Zijlstra pet...@infradead.org Cc: Clark Williams willi...@redhat.com Cc: Li Zefan lize...@huawei.com Cc: Ingo Molnar mi...@redhat.com Cc: Luiz Capitulino lcapitul...@redhat.com Cc: David Rientjes rient...@google.com Cc: Mike Galbraith umgwanakikb...@gmail.com Cc: cgro...@vger.kernel.org Signed-off-by: Rik van Riel r...@redhat.com So let me start off by saying I hate isolcpus ;-) Let me further state that I had hopes we could extend cpusets to natively provide the functionality isolcpus has, and kill isolcpus. +1 That's where nohz_full goop belongs too. The 'normal' way would be to create 2 cgroups with disjoint cpus, disable sched_load_balance on root and one of the siblings, while moving everything into the other group. That's what cset shield does, works fine. The 'problem' is that we cannot move everything that is affected by isolcpus, workqueues have grown a horrible 'new' interface outside of the regular task interfaces and things like kthreadd are non-movable for mostly good reasons. Furthermore it appears that software like system-disease and libvirt hard assume they're lord and master of the cgroup hierarchy and do not expect things like this. So while I mostly hate all of this it might be the best we can do :-( Hm, I'm now all system-disease-ified now (still hate the bloody thing), and have no problem isolating cpus via cpusets, modulo workqueues wanting a bat upside the head. -Mike -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Thu, Feb 26, 2015 at 12:12:31PM -0500, Rik van Riel wrote: Subject: cpusets,isolcpus: add file to show isolated cpus in cpuset The previous patch makes it so the code skips over isolcpus when building scheduler load balancing domains. This makes it hard to see for a user which of the CPUs in a cpuset are participating in load balancing, and which ones are isolated cpus. Add a cpuset.isolcpus file with info on which cpus in a cpuset are isolated CPUs. This file is read-only for now. In the future we could extend things so isolcpus can be changed at run time, for the root (system wide) cpuset only. Acked-by: David Rientjes rient...@google.com Cc: Peter Zijlstra pet...@infradead.org Cc: Clark Williams willi...@redhat.com Cc: Li Zefan lize...@huawei.com Cc: Ingo Molnar mi...@redhat.com Cc: Luiz Capitulino lcapitul...@redhat.com Cc: David Rientjes rient...@google.com Cc: Mike Galbraith umgwanakikb...@gmail.com Cc: cgro...@vger.kernel.org Signed-off-by: Rik van Riel r...@redhat.com So let me start off by saying I hate isolcpus ;-) Let me further state that I had hopes we could extend cpusets to natively provide the functionality isolcpus has, and kill isolcpus. The 'normal' way would be to create 2 cgroups with disjoint cpus, disable sched_load_balance on root and one of the siblings, while moving everything into the other group. The 'problem' is that we cannot move everything that is affected by isolcpus, workqueues have grown a horrible 'new' interface outside of the regular task interfaces and things like kthreadd are non-movable for mostly good reasons. Furthermore it appears that software like system-disease and libvirt hard assume they're lord and master of the cgroup hierarchy and do not expect things like this. So while I mostly hate all of this it might be the best we can do :-( -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Mon, Mar 02, 2015 at 01:44:50PM +0100, Mike Galbraith wrote: Hm, I'm now all system-disease-ified now (still hate the bloody thing), and have no problem isolating cpus via cpusets, modulo workqueues wanting a bat upside the head. It shouldn't be difficult to teach workqueue pools to follow the same rules. This matters only for the unbound ones anyway, right? Thanks. -- tejun -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Mon, 2015-03-02 at 10:29 -0500, Tejun Heo wrote: On Mon, Mar 02, 2015 at 01:44:50PM +0100, Mike Galbraith wrote: Hm, I'm now all system-disease-ified now (still hate the bloody thing), and have no problem isolating cpus via cpusets, modulo workqueues wanting a bat upside the head. It shouldn't be difficult to teach workqueue pools to follow the same rules. This matters only for the unbound ones anyway, right? Well, those are the only ones we can do anything about. Dirt simple diddling of the workqueue default mask as sched domains are added/removed should do it I think. Automatically moving any existing unbound worker away from isolated cores at the same time would be a bonus, most important is that no new threads sneak in. -Mike -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Mon, Mar 02, 2015 at 05:02:57PM +0100, Mike Galbraith wrote: Well, those are the only ones we can do anything about. Dirt simple diddling of the workqueue default mask as sched domains are added/removed should do it I think. Automatically moving any existing unbound worker away from isolated cores at the same time would be a bonus, most important is that no new threads sneak in. Worker pools are immutable once created and configuraiton changes are achieved by creating new pools and draining old ones but at any rate making it follow config changes is almost trivial. Figuring out configuration policy might take a bit of effort tho. Can you point me to what specific configuration it should be following? Thanks. -- tejun -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Thu, Feb 26, 2015 at 12:12:31PM -0500, Rik van Riel wrote: On Thu, 26 Feb 2015 19:05:57 +0800 Zefan Li lize...@huawei.com wrote: Make it return -ENOMEM ? Or make it a global variable and allocate memory for it in cpuset_init(). Here you are. This addresses your concern, as well as the issue David Rientjes found earlier. ---8--- Subject: cpusets,isolcpus: add file to show isolated cpus in cpuset The previous patch makes it so the code skips over isolcpus when building scheduler load balancing domains. This makes it hard to see for a user which of the CPUs in a cpuset are participating in load balancing, and which ones are isolated cpus. Add a cpuset.isolcpus file with info on which cpus in a cpuset are isolated CPUs. This file is read-only for now. In the future we could extend things so isolcpus can be changed at run time, for the root (system wide) cpuset only. Acked-by: David Rientjes rient...@google.com Cc: Peter Zijlstra pet...@infradead.org Cc: Clark Williams willi...@redhat.com Cc: Li Zefan lize...@huawei.com Cc: Ingo Molnar mi...@redhat.com Cc: Luiz Capitulino lcapitul...@redhat.com Cc: David Rientjes rient...@google.com Cc: Mike Galbraith umgwanakikb...@gmail.com Cc: cgro...@vger.kernel.org Signed-off-by: Rik van Riel r...@redhat.com Applied 1-2 to cgroup/for-4.1. Thanks. -- tejun -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Mon, Mar 02, 2015 at 12:01:16PM -0500, Tejun Heo wrote: Applied 1-2 to cgroup/for-4.1. Reverted due to build failure. Looks like UP build is broken. Thanks. -- tejun -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Mon, 2015-03-02 at 11:09 -0500, Tejun Heo wrote: On Mon, Mar 02, 2015 at 05:02:57PM +0100, Mike Galbraith wrote: Well, those are the only ones we can do anything about. Dirt simple diddling of the workqueue default mask as sched domains are added/removed should do it I think. Automatically moving any existing unbound worker away from isolated cores at the same time would be a bonus, most important is that no new threads sneak in. Worker pools are immutable once created and configuraiton changes are achieved by creating new pools and draining old ones but at any rate making it follow config changes is almost trivial. Figuring out configuration policy might take a bit of effort tho. Can you point me to what specific configuration it should be following? For cpusets, an exclusive set should become taboo to unbound workers when load balancing is turned off. The user making sched domains go away is a not so subtle hint that he wants no random interference, as he is trying to assume full responsibility for task placement therein. In my trees, I let the user turn rt cpupri/push/pull off as well, as that further reduces jitter. -Mike -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
Hi Rik, > Subject: cpusets,isolcpus: add file to show isolated cpus in cpuset > > The previous patch makes it so the code skips over isolcpus when > building scheduler load balancing domains. This makes it hard to > see for a user which of the CPUs in a cpuset are participating in > load balancing, and which ones are isolated cpus. > > Add a cpuset.isolcpus file with info on which cpus in a cpuset are > isolated CPUs. > > This file is read-only for now. In the future we could extend things > so isolcpus can be changed at run time, for the root (system wide) > cpuset only. > One Question, why not add a /sys/devices/system/cpu/isolated instead? -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
Hi Rik, Subject: cpusets,isolcpus: add file to show isolated cpus in cpuset The previous patch makes it so the code skips over isolcpus when building scheduler load balancing domains. This makes it hard to see for a user which of the CPUs in a cpuset are participating in load balancing, and which ones are isolated cpus. Add a cpuset.isolcpus file with info on which cpus in a cpuset are isolated CPUs. This file is read-only for now. In the future we could extend things so isolcpus can be changed at run time, for the root (system wide) cpuset only. One Question, why not add a /sys/devices/system/cpu/isolated instead? -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
> Subject: cpusets,isolcpus: add file to show isolated cpus in cpuset > > The previous patch makes it so the code skips over isolcpus when > building scheduler load balancing domains. This makes it hard to > see for a user which of the CPUs in a cpuset are participating in > load balancing, and which ones are isolated cpus. > > Add a cpuset.isolcpus file with info on which cpus in a cpuset are > isolated CPUs. > > This file is read-only for now. In the future we could extend things > so isolcpus can be changed at run time, for the root (system wide) > cpuset only. > > Acked-by: David Rientjes > Cc: Peter Zijlstra > Cc: Clark Williams > Cc: Li Zefan > Cc: Ingo Molnar > Cc: Luiz Capitulino > Cc: David Rientjes > Cc: Mike Galbraith > Cc: cgro...@vger.kernel.org > Signed-off-by: Rik van Riel Acked-by: Zefan Li -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
Subject: cpusets,isolcpus: add file to show isolated cpus in cpuset The previous patch makes it so the code skips over isolcpus when building scheduler load balancing domains. This makes it hard to see for a user which of the CPUs in a cpuset are participating in load balancing, and which ones are isolated cpus. Add a cpuset.isolcpus file with info on which cpus in a cpuset are isolated CPUs. This file is read-only for now. In the future we could extend things so isolcpus can be changed at run time, for the root (system wide) cpuset only. Acked-by: David Rientjes rient...@google.com Cc: Peter Zijlstra pet...@infradead.org Cc: Clark Williams willi...@redhat.com Cc: Li Zefan lize...@huawei.com Cc: Ingo Molnar mi...@redhat.com Cc: Luiz Capitulino lcapitul...@redhat.com Cc: David Rientjes rient...@google.com Cc: Mike Galbraith umgwanakikb...@gmail.com Cc: cgro...@vger.kernel.org Signed-off-by: Rik van Riel r...@redhat.com Acked-by: Zefan Li lize...@huawei.com -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Thu, 26 Feb 2015 19:05:57 +0800
Zefan Li wrote:
> Make it return -ENOMEM ? Or make it a global variable and allocate memory for
> it
> in cpuset_init().
Here you are. This addresses your concern, as well as the
issue David Rientjes found earlier.
---8<---
Subject: cpusets,isolcpus: add file to show isolated cpus in cpuset
The previous patch makes it so the code skips over isolcpus when
building scheduler load balancing domains. This makes it hard to
see for a user which of the CPUs in a cpuset are participating in
load balancing, and which ones are isolated cpus.
Add a cpuset.isolcpus file with info on which cpus in a cpuset are
isolated CPUs.
This file is read-only for now. In the future we could extend things
so isolcpus can be changed at run time, for the root (system wide)
cpuset only.
Acked-by: David Rientjes
Cc: Peter Zijlstra
Cc: Clark Williams
Cc: Li Zefan
Cc: Ingo Molnar
Cc: Luiz Capitulino
Cc: David Rientjes
Cc: Mike Galbraith
Cc: cgro...@vger.kernel.org
Signed-off-by: Rik van Riel
---
kernel/cpuset.c | 22 ++
1 file changed, 22 insertions(+)
diff --git a/kernel/cpuset.c b/kernel/cpuset.c
index b544e5229d99..5462e1ca90bd 100644
--- a/kernel/cpuset.c
+++ b/kernel/cpuset.c
@@ -1563,6 +1563,7 @@ typedef enum {
FILE_MEMORY_PRESSURE,
FILE_SPREAD_PAGE,
FILE_SPREAD_SLAB,
+ FILE_ISOLCPUS,
} cpuset_filetype_t;
static int cpuset_write_u64(struct cgroup_subsys_state *css, struct cftype
*cft,
@@ -1704,6 +1705,16 @@ static ssize_t cpuset_write_resmask(struct
kernfs_open_file *of,
return retval ?: nbytes;
}
+/* protected by the lock in cpuset_common_seq_show */
+static cpumask_var_t print_isolated_cpus;
+
+static void cpuset_seq_print_isolcpus(struct seq_file *sf, struct cpuset *cs)
+{
+ cpumask_and(print_isolated_cpus, cs->cpus_allowed, cpu_isolated_map);
+
+ seq_printf(sf, "%*pbl\n", cpumask_pr_args(print_isolated_cpus));
+}
+
/*
* These ascii lists should be read in a single call, by using a user
* buffer large enough to hold the entire map. If read in smaller
@@ -1733,6 +1744,9 @@ static int cpuset_common_seq_show(struct seq_file *sf,
void *v)
case FILE_EFFECTIVE_MEMLIST:
seq_printf(sf, "%*pbl\n",
nodemask_pr_args(>effective_mems));
break;
+ case FILE_ISOLCPUS:
+ cpuset_seq_print_isolcpus(sf, cs);
+ break;
default:
ret = -EINVAL;
}
@@ -1893,6 +1907,12 @@ static struct cftype files[] = {
.private = FILE_MEMORY_PRESSURE_ENABLED,
},
+ {
+ .name = "isolcpus",
+ .seq_show = cpuset_common_seq_show,
+ .private = FILE_ISOLCPUS,
+ },
+
{ } /* terminate */
};
@@ -2070,6 +2090,8 @@ int __init cpuset_init(void)
BUG();
if (!alloc_cpumask_var(_cpuset.effective_cpus, GFP_KERNEL))
BUG();
+ if (!alloc_cpumask_var(_isolated_cpus, GFP_KERNEL))
+ BUG();
cpumask_setall(top_cpuset.cpus_allowed);
nodes_setall(top_cpuset.mems_allowed);
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[PATCH v4 2/2] cpusets,isolcpus: add file to show isolated cpus in cpuset
On Thu, 26 Feb 2015 19:05:57 +0800
Zefan Li lize...@huawei.com wrote:
Make it return -ENOMEM ? Or make it a global variable and allocate memory for
it
in cpuset_init().
Here you are. This addresses your concern, as well as the
issue David Rientjes found earlier.
---8---
Subject: cpusets,isolcpus: add file to show isolated cpus in cpuset
The previous patch makes it so the code skips over isolcpus when
building scheduler load balancing domains. This makes it hard to
see for a user which of the CPUs in a cpuset are participating in
load balancing, and which ones are isolated cpus.
Add a cpuset.isolcpus file with info on which cpus in a cpuset are
isolated CPUs.
This file is read-only for now. In the future we could extend things
so isolcpus can be changed at run time, for the root (system wide)
cpuset only.
Acked-by: David Rientjes rient...@google.com
Cc: Peter Zijlstra pet...@infradead.org
Cc: Clark Williams willi...@redhat.com
Cc: Li Zefan lize...@huawei.com
Cc: Ingo Molnar mi...@redhat.com
Cc: Luiz Capitulino lcapitul...@redhat.com
Cc: David Rientjes rient...@google.com
Cc: Mike Galbraith umgwanakikb...@gmail.com
Cc: cgro...@vger.kernel.org
Signed-off-by: Rik van Riel r...@redhat.com
---
kernel/cpuset.c | 22 ++
1 file changed, 22 insertions(+)
diff --git a/kernel/cpuset.c b/kernel/cpuset.c
index b544e5229d99..5462e1ca90bd 100644
--- a/kernel/cpuset.c
+++ b/kernel/cpuset.c
@@ -1563,6 +1563,7 @@ typedef enum {
FILE_MEMORY_PRESSURE,
FILE_SPREAD_PAGE,
FILE_SPREAD_SLAB,
+ FILE_ISOLCPUS,
} cpuset_filetype_t;
static int cpuset_write_u64(struct cgroup_subsys_state *css, struct cftype
*cft,
@@ -1704,6 +1705,16 @@ static ssize_t cpuset_write_resmask(struct
kernfs_open_file *of,
return retval ?: nbytes;
}
+/* protected by the lock in cpuset_common_seq_show */
+static cpumask_var_t print_isolated_cpus;
+
+static void cpuset_seq_print_isolcpus(struct seq_file *sf, struct cpuset *cs)
+{
+ cpumask_and(print_isolated_cpus, cs-cpus_allowed, cpu_isolated_map);
+
+ seq_printf(sf, %*pbl\n, cpumask_pr_args(print_isolated_cpus));
+}
+
/*
* These ascii lists should be read in a single call, by using a user
* buffer large enough to hold the entire map. If read in smaller
@@ -1733,6 +1744,9 @@ static int cpuset_common_seq_show(struct seq_file *sf,
void *v)
case FILE_EFFECTIVE_MEMLIST:
seq_printf(sf, %*pbl\n,
nodemask_pr_args(cs-effective_mems));
break;
+ case FILE_ISOLCPUS:
+ cpuset_seq_print_isolcpus(sf, cs);
+ break;
default:
ret = -EINVAL;
}
@@ -1893,6 +1907,12 @@ static struct cftype files[] = {
.private = FILE_MEMORY_PRESSURE_ENABLED,
},
+ {
+ .name = isolcpus,
+ .seq_show = cpuset_common_seq_show,
+ .private = FILE_ISOLCPUS,
+ },
+
{ } /* terminate */
};
@@ -2070,6 +2090,8 @@ int __init cpuset_init(void)
BUG();
if (!alloc_cpumask_var(top_cpuset.effective_cpus, GFP_KERNEL))
BUG();
+ if (!alloc_cpumask_var(print_isolated_cpus, GFP_KERNEL))
+ BUG();
cpumask_setall(top_cpuset.cpus_allowed);
nodes_setall(top_cpuset.mems_allowed);
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
