Greetings. I thought I would share this in hopes we can come up with ways to handle this sort of thing better. I do not blame anyone and I'd like to focus on positive steps we can take rather than pointing fingers.
Just before the Fedora 25 development cycle started, I went and upgraded our koji builders from Fedora 23 to Fedora 24. I've done this every cycle since we moved them to Fedora instances and its gone usually pretty smoothly. Not this time. ;) The kernel maintainers started getting builds that failed and got stuck in odd ways. A few other larger packages (glibc) maintainers also hit this issue. The mock logs simply showed a bunch of "Fork failed" messages and a bunch of processes stuck in D At first we thought it might be a kernel issue, so we tried various kernels on the builders without any luck. Then the kernel maintainers added some patches to see if they could get debugging from the deadlock it was hitting. That fixed the processes stuck in D, but builds still failed. I started looking around for anything else that might be related and saw there was a systemd update that mentioned task limits, which in turn pointed us to looking at systemd and seeing this (from systemd 228 NEWS): " * There's a new system.conf setting DefaultTasksMax= to control the default TasksMax= setting for services and scopes running on the system. (TasksMax= is the primary setting that exposes the "pids" cgroup controller on systemd and was introduced in the previous systemd release.) The setting now defaults to 512, which means services that are not explicitly configured otherwise will only be able to create 512 processes or threads at maximum, from this version on. Note that this means that thread- or process-heavy services might need to be reconfigured to set TasksMax= to a higher value. It is sufficient to set TasksMax= in these specific unit files to a higher value, or even "infinity". Similar, there's now a logind.conf setting UserTasksMax= that defaults to 4096 and limits the total number of processes or tasks each user may own concurrently. nspawn containers also have the TasksMax= value set by default now, to 8192. Note that all of this only has an effect if the "pids" cgroup controller is enabled in the kernel. The general benefit of these changes should be a more robust and safer system, that provides a certain amount of per-service fork() bomb protection." We had systemd 229, and this was the root cause of the issue. In systemd 231 they changed this to: " * In similar fashion TasksMax= takes percentage values now, too. The value is taken relative to the configured maximum number of processes on the system. The per-service task maximum has been changed to 15% using this functionality. (Effectively this is an increase of 512 → 4915 for service units, given the kernel's default pid_max setting.)" So, how can we do better here? * IMHO the initial upstream default didn't make sense for Fedora * The feature doesn't seem to work right (causing processes to go into D and be deadlocked). Likely this is a rcu/kernel bug(s), but systemd shouldn't enable this unless it's robust or have some kind of check for a recent enough kernel with fixes? * There isn't any logging showing that this was hit. Perhaps there's a technical limitation with cgroups? This would be very handy in tracking down things like this. * This might have been a nice thing for release notes? I try and keep up, but I don't read every systemd NEWS file as they happen or most packages NEWS files. Can we make adding release notes any easier? * Perhaps after beta but before final we ping maintainers of "important" packages asking what big changes have happened? Or someone just goes thru the release notes for them all and proposes a list of them? * Your brilliant idea here. kevin
Description: OpenPGP digital signature
_______________________________________________ devel mailing list -- email@example.com To unsubscribe send an email to devel-le...@lists.fedoraproject.org