On Wed 26-08-15 15:23:07, David Rientjes wrote:
> On Wed, 26 Aug 2015, Michal Hocko wrote:
>
> > > Because the company I work for has far too many machines for that to be
> > > possible.
> >
> > OK I can see that manual intervention for hundreds of machines is not
> > practical. But not everybody is that large and there are users who might
> > want to be be able to recover.
> >
>
> If Andrew would prefer moving in a direction where all Linux users are
> required to have their admin use sysrq+f to manually trigger an oom kill,
> which may or may not resolve the livelock since there's no way to
> determine which process is holding the common mutex (or even which
> processes are currently allocating), in such situations, then we can carry
> this patch internally. I disagree with that solution for upstream Linux.
There are other possibilities than the manual sysrq intervention. E.g.
the already mentioned oom_{panic,reboot}_timeout which has a little
advantage that it allows admin to opt in into the policy rather than
having it hard coded into the kernel.
> > > If there is a holder of a mutex that then allocates gigabytes of memory,
> > > no amount of memory reserves is going to assist in resolving an oom
> > > killer
> > > livelock, whether that's partial access to memory reserves or full access
> > > to memory reserves.
> >
> > Sure, but do we have something like that in the kernel? I would argue it
> > would be terribly broken and a clear bug which should be fixed.
> >
>
> This is also why my patch dumps the stack trace of both threads: so we can
> evaluate the memory allocation of threads holding shared mutexes. If it
> is excessive, we can report that and show that it is a common offender and
> see if we can mitigate that.
>
> The scenario described, the full or partial depletion of memory reserves,
> does not need to be induced by a single user. We don't control the order
> in which the mutex is grabbed so it's multipled by the number of threads
> that grab it, allocate memory, and drop it before the victim has a chance
> to grab it. In the past, the oom killer would also increase the
> scheduling priority of a victim so it tried to resolve issues like this
> faster.
> > > Unless the oom watermark was higher than the lowest access to memory
> > > reserves other than ALLOC_NO_WATERMARKS, then no forward progress would
> > > be
> > > made in this scenario. I think it would be better to give access to that
> > > crucial last page that may solve the livelock to make forward progress,
> > > or
> > > panic as a result of complete depletion of memory reserves. That panic()
> > > is a very trivial patch that can be checked in the allocator slowpath and
> > > addresses a problem that already exists today.
> >
> > The panicing the system is really trivial, no question about that. The
> > question is whether that panic would be premature. And that is what
> > I've tried to tell you.
>
> My patch has defined that by OOM_EXPIRE_MSECS. The premise is that an oom
> victim with full access to memory reserves should never take more than 5s
> to exit, which I consider a very long time. If it's increased, we see
> userspace responsiveness issues with our processes that monitor system
> health which timeout.
Yes but it sounds very much like a policy which should better be defined
from the userspace because different users might have different
preferences.
--
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
