On Sun, 05.03.17 15:59, Pascal Kolijn (p.kol...@vu.nl) wrote: > Peace, > > On 28/02/2017 16:00, Lennart Poettering wrote: > > On Tue, 28.02.17 13:26, Pascal kolijn (p.kol...@vu.nl) wrote: > > > >> Hi List, > >> > >> I've subscribed to this list to ask for help in debugging a problem we > >> seem to have with the socket activated telnetd on a rhel7 system. > >> > >> A default install of telnetd collects data from some small boxes > >> deployed in the field. It works for a long time and then suddenly: > >> > >> Feb 26 17:46:53 bibr systemd: Created slice user-6006.slice. > >> Feb 26 17:46:53 bibr systemd: Starting user-6006.slice. > >> Feb 26 17:46:53 bibr systemd: Started Session 2223341 of user <USER>. > >> Feb 26 17:46:53 bibr systemd-logind: New session 2223341 of user <USER>. > >> Feb 26 17:46:53 bibr systemd: Starting Session 2223341 of user <USER>. > >> Feb 26 17:46:53 bibr systemd: Started Telnet Server (<IP>:28830). > >> Feb 26 17:46:53 pbibr001 systemd: Starting Telnet Server (<IP>:28830)... > >> Feb 26 17:46:57 bibr systemd: Failed to fork: Cannot allocate memory > > > > Hmm, Linux fork() returns ENOMEM if the maximum number of tasks on the > > system is hit (yes this is a bit misleading, but that's how it is). > > That max number of tasks is limited for example by the max number of > > assignable pids as configured in /proc/sys/kernel/pid_max? Maybe you > > hit that limit? Maybe something is leaking pids on your system? not > > reaping zombies properly? > > As far as I can determine running out of pids is not the issue, as I can > see pids being reused in a day, which will not say that some may still > go missing over time, but how do I determine if that is the case...?
Well, the "Failed to fork" log message is generated when "fork()" returns ENOMEM, i.e. this is generated by a syscall, i.e. it's an allocation of some object in the kernel that fails, not of userspace... > What I do see is that the rss of the systemd process is slowly growing > over time in the production environment. I've not been able (yet) to > reproduce the situation in a test environment, which is a pity. I think > I can simulate the telnet connects more accurately after I speak with > the developer of the said boxes, and see if I can create a reproducible > situation. Well, if it grows without bounds then this would be a leak. But if it grows only a bit during runtime that's not a problem: in-process memory management generally doesn't return all memory back to the kernel immediately, but keeps an allocation cache. In fact, our hashtable implementation in systemd (which is heavily used) implements an extra allocation cache on top. Memory in that cache is not actually leaked, even though it is not returned to the OS: it is simply kept around for next time we need a similar object. Lennart -- Lennart Poettering, Red Hat _______________________________________________ systemd-devel mailing list email@example.com https://lists.freedesktop.org/mailman/listinfo/systemd-devel