I thought I would share this in hopes we can come up with ways to
handle this sort of thing better. I do not blame anyone and I'd like to
focus on positive steps we can take rather than pointing fingers. 

Just before the Fedora 25 development cycle started, I went and
upgraded our koji builders from Fedora 23 to Fedora 24. I've done this
every cycle since we moved them to Fedora instances and its gone
usually pretty smoothly.

Not this time. ;) The kernel maintainers started getting builds that
failed and got stuck in odd ways. A few other larger packages (glibc)
maintainers also hit this issue. The mock logs simply showed a bunch of
"Fork failed" messages and a bunch of processes stuck in D 

At first we thought it might be a kernel issue, so we tried various
kernels on the builders without any luck. Then the kernel maintainers
added some patches to see if they could get debugging from the deadlock
it was hitting. That fixed the processes stuck in D, but builds still

I started looking around for anything else that might be related and
saw there was a systemd update that mentioned task limits, which in
turn pointed us to looking at systemd and seeing this (from systemd 228

"        * There's a new system.conf setting DefaultTasksMax= to
          control the default TasksMax= setting for services and
          scopes running on the system. (TasksMax= is the primary
          setting that exposes the "pids" cgroup controller on systemd
          and was introduced in the previous systemd release.) The
          setting now defaults to 512, which means services that are
          not explicitly configured otherwise will only be able to
          create 512 processes or threads at maximum, from this
          version on. Note that this means that thread- or
          process-heavy services might need to be reconfigured to set
          TasksMax= to a higher value. It is sufficient to set
          TasksMax= in these specific unit files to a higher value, or
          even "infinity". Similar, there's now a logind.conf setting
          UserTasksMax= that defaults to 4096 and limits the total
          number of processes or tasks each user may own
          concurrently. nspawn containers also have the TasksMax=
          value set by default now, to 8192. Note that all of this
          only has an effect if the "pids" cgroup controller is
          enabled in the kernel. The general benefit of these changes
          should be a more robust and safer system, that provides a
          certain amount of per-service fork() bomb protection."

We had systemd 229, and this was the root cause of the issue. 

In systemd 231 they changed this to: 

"        * In similar fashion TasksMax= takes percentage values now,
too. The value is taken relative to the configured maximum number of
processes on the system. The per-service task maximum has been changed
to 15% using this functionality. (Effectively this is an increase of
512 → 4915 for service units, given the kernel's default pid_max

So, how can we do better here?

* IMHO the initial upstream default didn't make sense for Fedora

* The feature doesn't seem to work right (causing processes to go into
  D and be deadlocked). Likely this is a rcu/kernel bug(s), but systemd
  shouldn't enable this unless it's robust or have some kind of check
  for a recent enough kernel with fixes?

* There isn't any logging showing that this was hit. Perhaps there's a
  technical limitation with cgroups? This would be very handy in
  tracking down things like this. 

* This might have been a nice thing for release notes? I try and keep
  up, but I don't read every systemd NEWS file as they happen or most
  packages NEWS files. Can we make adding release notes any easier?

* Perhaps after beta but before final we ping maintainers of
  "important" packages asking what big changes have happened? Or
  someone just goes thru the release notes for them all and proposes a
  list of them?

* Your brilliant idea here. 


Attachment: pgpGqvnFW5UF4.pgp
Description: OpenPGP digital signature

devel mailing list --
To unsubscribe send an email to

Reply via email to