Re: [systemd-devel] systemd-journald may crash during memory pressure
Note the logs you've pasted portray a watchdog timeout which resulted in SIGABRT and a subsequent core dump. This is not really a journald "crash", and you can increase the watchdog timeout or disable it entirely to make it more tolerant of thrashing. What I presume happened is the system was thrashing and a page fault in the mapped journal took too long to complete. Regards, Vito Caputo On Thu, Feb 08, 2018 at 11:50:45PM +0100, Kai Krakow wrote: > Hello! > > During memory pressure and/or high load, journald may crash. This is > probably due to design using mmap but it should really not do this. > > On 32-bit systems, we are seeing such crashes constantly although the > available memory is still gigabytes (it's a 32-bit userland running in a > 64-bit kernel). > > > [82988.670323] systemd[1]: systemd-journald.service: Main process exited, > code=dumped, status=6/ABRT > [82988.670684] systemd[1]: systemd-journald.service: Failed with result > 'watchdog'. > [82988.685928] systemd[1]: systemd-journald.service: Service has no hold-off > time, scheduling restart. > [82988.709575] systemd[1]: systemd-journald.service: Scheduled restart job, > restart counter is at 2. > [82988.717390] systemd[1]: Stopped Flush Journal to Persistent Storage. > [82988.717411] systemd[1]: Stopping Flush Journal to Persistent Storage... > [82988.726303] systemd[1]: Stopped Journal Service. > [82988.844462] systemd[1]: Starting Journal Service... > [82993.633781] systemd-coredump[22420]: MESSAGE=Process 461 (systemd-journal) > of user 0 dumped core. > [82993.633811] systemd-coredump[22420]: Coredump diverted to > /var/lib/systemd/coredump/core.systemd-journal.0.3d492c866f254fb981f916c6c3918046.461.151812537700.lz4 > [82993.633813] systemd-coredump[22420]: Stack trace of thread 461: > [82993.633814] systemd-coredump[22420]: #0 0x7f940241d4dd > journal_file_move_to_object (libsystemd-shared-237.so) > [82993.633815] systemd-coredump[22420]: #1 0x7f940241e910 > journal_file_find_data_object_with_hash (libsystemd-shared-237.so) > [82993.633816] systemd-coredump[22420]: #2 0x7f940241fe81 > journal_file_append_data (libsystemd-shared-237.so) > [82993.633817] systemd-coredump[22420]: #3 0x556a343ae9ea > write_to_journal (systemd-journald) > [82993.633819] systemd-coredump[22420]: #4 0x556a343b0974 > server_dispatch_message (systemd-journald) > [82993.633820] systemd-coredump[22420]: #5 0x556a343b24bb > stdout_stream_log (systemd-journald) > [82993.633821] systemd-coredump[22420]: #6 0x556a343b2afe > stdout_stream_line (systemd-journald) > [82993.723157] systemd-coredum: 7 output lines suppressed due to ratelimiting > [83002.830610] systemd-journald[22424]: File > /var/log/journal/121b87ca633e8ac001665668001b/system.journal corrupted or > uncleanly shut down, renaming and replacing. > [83014.774538] systemd[1]: Started Journal Service. > [83119.277143] systemd-journald[22424]: File > /var/log/journal/121b87ca633e8ac001665668001b/user-500.journal corrupted > or uncleanly shut down, renaming and replacing. > > > -- > Regards, > Kai > > Replies to list-only preferred. > > ___ > systemd-devel mailing list > systemd-devel@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/systemd-devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] systemd-journald may crash during memory pressure
Hello! During memory pressure and/or high load, journald may crash. This is probably due to design using mmap but it should really not do this. On 32-bit systems, we are seeing such crashes constantly although the available memory is still gigabytes (it's a 32-bit userland running in a 64-bit kernel). [82988.670323] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT [82988.670684] systemd[1]: systemd-journald.service: Failed with result 'watchdog'. [82988.685928] systemd[1]: systemd-journald.service: Service has no hold-off time, scheduling restart. [82988.709575] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 2. [82988.717390] systemd[1]: Stopped Flush Journal to Persistent Storage. [82988.717411] systemd[1]: Stopping Flush Journal to Persistent Storage... [82988.726303] systemd[1]: Stopped Journal Service. [82988.844462] systemd[1]: Starting Journal Service... [82993.633781] systemd-coredump[22420]: MESSAGE=Process 461 (systemd-journal) of user 0 dumped core. [82993.633811] systemd-coredump[22420]: Coredump diverted to /var/lib/systemd/coredump/core.systemd-journal.0.3d492c866f254fb981f916c6c3918046.461.151812537700.lz4 [82993.633813] systemd-coredump[22420]: Stack trace of thread 461: [82993.633814] systemd-coredump[22420]: #0 0x7f940241d4dd journal_file_move_to_object (libsystemd-shared-237.so) [82993.633815] systemd-coredump[22420]: #1 0x7f940241e910 journal_file_find_data_object_with_hash (libsystemd-shared-237.so) [82993.633816] systemd-coredump[22420]: #2 0x7f940241fe81 journal_file_append_data (libsystemd-shared-237.so) [82993.633817] systemd-coredump[22420]: #3 0x556a343ae9ea write_to_journal (systemd-journald) [82993.633819] systemd-coredump[22420]: #4 0x556a343b0974 server_dispatch_message (systemd-journald) [82993.633820] systemd-coredump[22420]: #5 0x556a343b24bb stdout_stream_log (systemd-journald) [82993.633821] systemd-coredump[22420]: #6 0x556a343b2afe stdout_stream_line (systemd-journald) [82993.723157] systemd-coredum: 7 output lines suppressed due to ratelimiting [83002.830610] systemd-journald[22424]: File /var/log/journal/121b87ca633e8ac001665668001b/system.journal corrupted or uncleanly shut down, renaming and replacing. [83014.774538] systemd[1]: Started Journal Service. [83119.277143] systemd-journald[22424]: File /var/log/journal/121b87ca633e8ac001665668001b/user-500.journal corrupted or uncleanly shut down, renaming and replacing. -- Regards, Kai Replies to list-only preferred. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Controlling user processes with systemd+cgroups
Hello systemd-devel. Short summary from the old thread: On 06.09.15 16:14, Lennart Poettering [1]: > Ultimately our goal is that you build your tree of slices, and then > freely attach users, services, containers, VMs to these slices at the > places you want them. You can already do that nicely for services and > containers (at least for nspawn containers), but for users this is > really missing. The missing piece is thus where to store user->slice mapping (not necessarily injective as it is now). Then it would be possible to apply limits _shared in groups_ of users. Lennart also sketched such information could be in the user database, although there is no standard way how to obtain that. AFAIK this still applies and IMO it may take longer to change than comfortable. (Please enlighten me on whatever I might be missing in this regard.) I gave a thought to alternatives. They basically rely on GID -- the information that already can be obtained from a user database in standard way and it overlaps with most missing use cases. Variant 1 -- Slice.GroupId= property. Admins would create unit files for slices specifying this option and users who are members of any listed groups would have their user sessions placed into given slice instead of user-$UID.slice. Variant 2 -- group mode This would allow admins to switch how user slices are created. By switching into the group mode (e.g. pam_systemd or logind option) user sessions would be put into group-$GID-$UID.slice and cgroup configuration would be then applied to respective group-$GID.slice units. What are your thoughts on that? Do any other alternatives come to your mind? Would some of the variants be eventually acceptable to be included? Thanks, Michal [1] https://lists.freedesktop.org/archives/systemd-devel/2015-September/034131.html signature.asc Description: OpenPGP digital signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
