Re: [PATCH 0/2] mm, oom: fix oom_reaper fallouts
On Mon 07-08-17 13:38:37, Michal Hocko wrote: > Hi, > there are two issues this patch series attempts to fix. First one is > something that has been broken since MMF_UNSTABLE flag introduction > and I guess we should backport it stable trees (patch 1). The other > issue has been brought up by Wenwei Tao and Tetsuo Handa has created > a test case to trigger it very reliably. I am not yet sure this is a > stable material because the test case is rather artificial. If there is > a demand for the stable backport I will prepare it, of course, though. > > I hope I've done the second patch correctly but I would definitely > appreciate some more eyes on it. Hence CCing Andrea and Kirill. My > previous attempt with some more context was posted here > http://lkml.kernel.org/r/20170803135902.31977-1-mho...@kernel.org > > My testing didn't show anything unusual with these two applied on top of > the mmotm tree. unless anybody object can we have this merged? Whether to push this to the stable tree is still questionable because it requires a rather artificial workload to trigger the issue but if others think it would be better to have it backported I will prepare backports for all relevant stable trees. -- Michal Hocko SUSE Labs
Re: [PATCH 0/2] mm, oom: fix oom_reaper fallouts
On Mon 07-08-17 13:38:37, Michal Hocko wrote: > Hi, > there are two issues this patch series attempts to fix. First one is > something that has been broken since MMF_UNSTABLE flag introduction > and I guess we should backport it stable trees (patch 1). The other > issue has been brought up by Wenwei Tao and Tetsuo Handa has created > a test case to trigger it very reliably. I am not yet sure this is a > stable material because the test case is rather artificial. If there is > a demand for the stable backport I will prepare it, of course, though. > > I hope I've done the second patch correctly but I would definitely > appreciate some more eyes on it. Hence CCing Andrea and Kirill. My > previous attempt with some more context was posted here > http://lkml.kernel.org/r/20170803135902.31977-1-mho...@kernel.org > > My testing didn't show anything unusual with these two applied on top of > the mmotm tree. unless anybody object can we have this merged? Whether to push this to the stable tree is still questionable because it requires a rather artificial workload to trigger the issue but if others think it would be better to have it backported I will prepare backports for all relevant stable trees. -- Michal Hocko SUSE Labs
Re: [PATCH 0/2] mm, oom: fix oom_reaper fallouts
Michal Hocko wrote: > On Mon 07-08-17 22:28:27, Tetsuo Handa wrote: > > Michal Hocko wrote: > > > Hi, > > > there are two issues this patch series attempts to fix. First one is > > > something that has been broken since MMF_UNSTABLE flag introduction > > > and I guess we should backport it stable trees (patch 1). The other > > > issue has been brought up by Wenwei Tao and Tetsuo Handa has created > > > a test case to trigger it very reliably. I am not yet sure this is a > > > stable material because the test case is rather artificial. If there is > > > a demand for the stable backport I will prepare it, of course, though. > > > > > > I hope I've done the second patch correctly but I would definitely > > > appreciate some more eyes on it. Hence CCing Andrea and Kirill. My > > > previous attempt with some more context was posted here > > > http://lkml.kernel.org/r/20170803135902.31977-1-mho...@kernel.org > > > > > > My testing didn't show anything unusual with these two applied on top of > > > the mmotm tree. > > > > I really don't like your likely/unlikely speculation. > > Have you seen any non artificial workload triggering this? It will be 5 to 10 years away from now to know whether non artificial workload triggers this. (I mean, customers start using RHEL8.) >Look, I am > not going to argue about how likely this is or not. I've said I am > willing to do backports if there is a demand but please do realize that > this is not a trivial change to backport pre 4.9 kernels would require > MMF_UNSTABLE to be backported as well. This all can be discussed > after the merge so can we focus on the review now rather than any > distractions? 3f70dc38cec2 was not working as expected. Nobody tested that OOM situation. Then, I think we can revert 3f70dc38cec2, and then make it possible to uniformly apply MMF_UNSTABLE to all 4.6+ kernels. > > Also please note that while writing zeros is certainly bad any integrity > assumptions are basically off when an application gets killed > unexpectedly while performing an IO. I consider unexpectedly saving process image (instead of zeros) to a file is similar to fs.suid_dumpable problem (i.e. could cause a security problem). I do expect that this patch is backported to RHEL8 (I don't know which version RHEL8 will choose, but I guess it will be between 4.6 and 4.13).
Re: [PATCH 0/2] mm, oom: fix oom_reaper fallouts
Michal Hocko wrote: > On Mon 07-08-17 22:28:27, Tetsuo Handa wrote: > > Michal Hocko wrote: > > > Hi, > > > there are two issues this patch series attempts to fix. First one is > > > something that has been broken since MMF_UNSTABLE flag introduction > > > and I guess we should backport it stable trees (patch 1). The other > > > issue has been brought up by Wenwei Tao and Tetsuo Handa has created > > > a test case to trigger it very reliably. I am not yet sure this is a > > > stable material because the test case is rather artificial. If there is > > > a demand for the stable backport I will prepare it, of course, though. > > > > > > I hope I've done the second patch correctly but I would definitely > > > appreciate some more eyes on it. Hence CCing Andrea and Kirill. My > > > previous attempt with some more context was posted here > > > http://lkml.kernel.org/r/20170803135902.31977-1-mho...@kernel.org > > > > > > My testing didn't show anything unusual with these two applied on top of > > > the mmotm tree. > > > > I really don't like your likely/unlikely speculation. > > Have you seen any non artificial workload triggering this? It will be 5 to 10 years away from now to know whether non artificial workload triggers this. (I mean, customers start using RHEL8.) >Look, I am > not going to argue about how likely this is or not. I've said I am > willing to do backports if there is a demand but please do realize that > this is not a trivial change to backport pre 4.9 kernels would require > MMF_UNSTABLE to be backported as well. This all can be discussed > after the merge so can we focus on the review now rather than any > distractions? 3f70dc38cec2 was not working as expected. Nobody tested that OOM situation. Then, I think we can revert 3f70dc38cec2, and then make it possible to uniformly apply MMF_UNSTABLE to all 4.6+ kernels. > > Also please note that while writing zeros is certainly bad any integrity > assumptions are basically off when an application gets killed > unexpectedly while performing an IO. I consider unexpectedly saving process image (instead of zeros) to a file is similar to fs.suid_dumpable problem (i.e. could cause a security problem). I do expect that this patch is backported to RHEL8 (I don't know which version RHEL8 will choose, but I guess it will be between 4.6 and 4.13).
Re: [PATCH 0/2] mm, oom: fix oom_reaper fallouts
On Mon 07-08-17 22:28:27, Tetsuo Handa wrote: > Michal Hocko wrote: > > Hi, > > there are two issues this patch series attempts to fix. First one is > > something that has been broken since MMF_UNSTABLE flag introduction > > and I guess we should backport it stable trees (patch 1). The other > > issue has been brought up by Wenwei Tao and Tetsuo Handa has created > > a test case to trigger it very reliably. I am not yet sure this is a > > stable material because the test case is rather artificial. If there is > > a demand for the stable backport I will prepare it, of course, though. > > > > I hope I've done the second patch correctly but I would definitely > > appreciate some more eyes on it. Hence CCing Andrea and Kirill. My > > previous attempt with some more context was posted here > > http://lkml.kernel.org/r/20170803135902.31977-1-mho...@kernel.org > > > > My testing didn't show anything unusual with these two applied on top of > > the mmotm tree. > > I really don't like your likely/unlikely speculation. Have you seen any non artificial workload triggering this? Look, I am not going to argue about how likely this is or not. I've said I am willing to do backports if there is a demand but please do realize that this is not a trivial change to backport pre 4.9 kernels would require MMF_UNSTABLE to be backported as well. This all can be discussed after the merge so can we focus on the review now rather than any distractions? Also please note that while writing zeros is certainly bad any integrity assumptions are basically off when an application gets killed unexpectedly while performing an IO. -- Michal Hocko SUSE Labs
Re: [PATCH 0/2] mm, oom: fix oom_reaper fallouts
On Mon 07-08-17 22:28:27, Tetsuo Handa wrote: > Michal Hocko wrote: > > Hi, > > there are two issues this patch series attempts to fix. First one is > > something that has been broken since MMF_UNSTABLE flag introduction > > and I guess we should backport it stable trees (patch 1). The other > > issue has been brought up by Wenwei Tao and Tetsuo Handa has created > > a test case to trigger it very reliably. I am not yet sure this is a > > stable material because the test case is rather artificial. If there is > > a demand for the stable backport I will prepare it, of course, though. > > > > I hope I've done the second patch correctly but I would definitely > > appreciate some more eyes on it. Hence CCing Andrea and Kirill. My > > previous attempt with some more context was posted here > > http://lkml.kernel.org/r/20170803135902.31977-1-mho...@kernel.org > > > > My testing didn't show anything unusual with these two applied on top of > > the mmotm tree. > > I really don't like your likely/unlikely speculation. Have you seen any non artificial workload triggering this? Look, I am not going to argue about how likely this is or not. I've said I am willing to do backports if there is a demand but please do realize that this is not a trivial change to backport pre 4.9 kernels would require MMF_UNSTABLE to be backported as well. This all can be discussed after the merge so can we focus on the review now rather than any distractions? Also please note that while writing zeros is certainly bad any integrity assumptions are basically off when an application gets killed unexpectedly while performing an IO. -- Michal Hocko SUSE Labs
Re: [PATCH 0/2] mm, oom: fix oom_reaper fallouts
Michal Hocko wrote:
> Hi,
> there are two issues this patch series attempts to fix. First one is
> something that has been broken since MMF_UNSTABLE flag introduction
> and I guess we should backport it stable trees (patch 1). The other
> issue has been brought up by Wenwei Tao and Tetsuo Handa has created
> a test case to trigger it very reliably. I am not yet sure this is a
> stable material because the test case is rather artificial. If there is
> a demand for the stable backport I will prepare it, of course, though.
>
> I hope I've done the second patch correctly but I would definitely
> appreciate some more eyes on it. Hence CCing Andrea and Kirill. My
> previous attempt with some more context was posted here
> http://lkml.kernel.org/r/20170803135902.31977-1-mho...@kernel.org
>
> My testing didn't show anything unusual with these two applied on top of
> the mmotm tree.
I really don't like your likely/unlikely speculation.
I can trigger this problem with 8 threads using 4.13.0-rc2-next-20170728.
The written data can be random values (seems portion of a.out memory image).
I guess that unexpected information leakage is possible.
--
$ cat 0804.c
#define _GNU_SOURCE
#include
#include
#include
#include
#include
#include
#include
#include
#include
#define NUMTHREADS 8
#define STACKSIZE 8192
static int pipe_fd[2] = { EOF, EOF };
static int file_writer(void *i)
{
static char buffer[1048576];
int fd;
char buffer2[64] = { };
snprintf(buffer2, sizeof(buffer2), "/tmp/file.%lu", (unsigned long) i);
fd = open(buffer2, O_WRONLY | O_CREAT | O_APPEND, 0600);
memset(buffer, 0xFF, sizeof(buffer));
read(pipe_fd[0], buffer, 1);
while (write(fd, buffer, sizeof(buffer)) == sizeof(buffer));
return 0;
}
int main(int argc, char *argv[])
{
char *buf = NULL;
unsigned long size;
unsigned long i;
char *stack;
if (pipe(pipe_fd))
return 1;
stack = malloc(STACKSIZE * NUMTHREADS);
for (size = 1048576; size < 512UL * (1 << 30); size <<= 1) {
char *cp = realloc(buf, size);
if (!cp) {
size >>= 1;
break;
}
buf = cp;
}
for (i = 0; i < NUMTHREADS; i++)
if (clone(file_writer, stack + (i + 1) * STACKSIZE,
CLONE_THREAD | CLONE_SIGHAND | CLONE_VM | CLONE_FS |
CLONE_FILES, (void *) i) == -1)
break;
close(pipe_fd[1]);
/* Will cause OOM due to overcommit; if not use SysRq-f */
for (i = 0; i < size; i += 4096)
buf[i] = 0;
kill(-1, SIGKILL);
return 0;
}
$ gcc -Wall -O3 0804.c
$ while :; do ./a.out; cat /tmp/file.* | od -b; /bin/rm /tmp/file.*; done
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
641524
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
446100
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
376250
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
434736
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
645130
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
356401 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000
*
356402 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
505540
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
544662
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
620347
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
412756
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
522417
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
561245
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
476642 055 061 061 051 000 000 056 163 171 155 164 141 142 000 056 163
4766420020 164 162 164 141 142 000 056 163 150 163 164 162 164 141 142 000
4766420040 056 151 156 164 145 162 160 000 056 156 157 164 145 056 101 102
4766420060 111 055 164 141 147 000 056 156 157 164 145 056 147 156 165 056
4766420100 142 165 151 154 144 055 151 144 000 056 147 156 165 056 150 141
4766420120 163 150 000 056 144 171 156 163 171 155 000 056 144 171 156 163
4766420140 164 162 000 056 147 156 165 056 166 145 162 163 151 157 156 000
4766420160 056 147 156 165 056 166 145 162 163 151 157 156 137 162 000 056
4766420200 162 145 154 141 056 144 171 156 000 056 162 145 154 141 056 160
4766420220 154 164 000 056 151 156 151 164 000 056 164 145 170 164 000 056
4766420240 146 151 156 151 000 056 162 157 144 141 164 141 000 056 145 150
4766420260 137 146 162 141 155 145
Re: [PATCH 0/2] mm, oom: fix oom_reaper fallouts
Michal Hocko wrote:
> Hi,
> there are two issues this patch series attempts to fix. First one is
> something that has been broken since MMF_UNSTABLE flag introduction
> and I guess we should backport it stable trees (patch 1). The other
> issue has been brought up by Wenwei Tao and Tetsuo Handa has created
> a test case to trigger it very reliably. I am not yet sure this is a
> stable material because the test case is rather artificial. If there is
> a demand for the stable backport I will prepare it, of course, though.
>
> I hope I've done the second patch correctly but I would definitely
> appreciate some more eyes on it. Hence CCing Andrea and Kirill. My
> previous attempt with some more context was posted here
> http://lkml.kernel.org/r/20170803135902.31977-1-mho...@kernel.org
>
> My testing didn't show anything unusual with these two applied on top of
> the mmotm tree.
I really don't like your likely/unlikely speculation.
I can trigger this problem with 8 threads using 4.13.0-rc2-next-20170728.
The written data can be random values (seems portion of a.out memory image).
I guess that unexpected information leakage is possible.
--
$ cat 0804.c
#define _GNU_SOURCE
#include
#include
#include
#include
#include
#include
#include
#include
#include
#define NUMTHREADS 8
#define STACKSIZE 8192
static int pipe_fd[2] = { EOF, EOF };
static int file_writer(void *i)
{
static char buffer[1048576];
int fd;
char buffer2[64] = { };
snprintf(buffer2, sizeof(buffer2), "/tmp/file.%lu", (unsigned long) i);
fd = open(buffer2, O_WRONLY | O_CREAT | O_APPEND, 0600);
memset(buffer, 0xFF, sizeof(buffer));
read(pipe_fd[0], buffer, 1);
while (write(fd, buffer, sizeof(buffer)) == sizeof(buffer));
return 0;
}
int main(int argc, char *argv[])
{
char *buf = NULL;
unsigned long size;
unsigned long i;
char *stack;
if (pipe(pipe_fd))
return 1;
stack = malloc(STACKSIZE * NUMTHREADS);
for (size = 1048576; size < 512UL * (1 << 30); size <<= 1) {
char *cp = realloc(buf, size);
if (!cp) {
size >>= 1;
break;
}
buf = cp;
}
for (i = 0; i < NUMTHREADS; i++)
if (clone(file_writer, stack + (i + 1) * STACKSIZE,
CLONE_THREAD | CLONE_SIGHAND | CLONE_VM | CLONE_FS |
CLONE_FILES, (void *) i) == -1)
break;
close(pipe_fd[1]);
/* Will cause OOM due to overcommit; if not use SysRq-f */
for (i = 0; i < size; i += 4096)
buf[i] = 0;
kill(-1, SIGKILL);
return 0;
}
$ gcc -Wall -O3 0804.c
$ while :; do ./a.out; cat /tmp/file.* | od -b; /bin/rm /tmp/file.*; done
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
641524
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
446100
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
376250
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
434736
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
645130
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
356401 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000
*
356402 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
505540
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
544662
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
620347
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
412756
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
522417
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
561245
Killed
000 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377 377
*
476642 055 061 061 051 000 000 056 163 171 155 164 141 142 000 056 163
4766420020 164 162 164 141 142 000 056 163 150 163 164 162 164 141 142 000
4766420040 056 151 156 164 145 162 160 000 056 156 157 164 145 056 101 102
4766420060 111 055 164 141 147 000 056 156 157 164 145 056 147 156 165 056
4766420100 142 165 151 154 144 055 151 144 000 056 147 156 165 056 150 141
4766420120 163 150 000 056 144 171 156 163 171 155 000 056 144 171 156 163
4766420140 164 162 000 056 147 156 165 056 166 145 162 163 151 157 156 000
4766420160 056 147 156 165 056 166 145 162 163 151 157 156 137 162 000 056
4766420200 162 145 154 141 056 144 171 156 000 056 162 145 154 141 056 160
4766420220 154 164 000 056 151 156 151 164 000 056 164 145 170 164 000 056
4766420240 146 151 156 151 000 056 162 157 144 141 164 141 000 056 145 150
4766420260 137 146 162 141 155 145
[PATCH 0/2] mm, oom: fix oom_reaper fallouts
Hi, there are two issues this patch series attempts to fix. First one is something that has been broken since MMF_UNSTABLE flag introduction and I guess we should backport it stable trees (patch 1). The other issue has been brought up by Wenwei Tao and Tetsuo Handa has created a test case to trigger it very reliably. I am not yet sure this is a stable material because the test case is rather artificial. If there is a demand for the stable backport I will prepare it, of course, though. I hope I've done the second patch correctly but I would definitely appreciate some more eyes on it. Hence CCing Andrea and Kirill. My previous attempt with some more context was posted here http://lkml.kernel.org/r/20170803135902.31977-1-mho...@kernel.org My testing didn't show anything unusual with these two applied on top of the mmotm tree.
[PATCH 0/2] mm, oom: fix oom_reaper fallouts
Hi, there are two issues this patch series attempts to fix. First one is something that has been broken since MMF_UNSTABLE flag introduction and I guess we should backport it stable trees (patch 1). The other issue has been brought up by Wenwei Tao and Tetsuo Handa has created a test case to trigger it very reliably. I am not yet sure this is a stable material because the test case is rather artificial. If there is a demand for the stable backport I will prepare it, of course, though. I hope I've done the second patch correctly but I would definitely appreciate some more eyes on it. Hence CCing Andrea and Kirill. My previous attempt with some more context was posted here http://lkml.kernel.org/r/20170803135902.31977-1-mho...@kernel.org My testing didn't show anything unusual with these two applied on top of the mmotm tree.
