Re: [systemd-devel] is the watchdog useful?
On Thu, Oct 31, 2019 at 06:30:33PM +0100, Lennart Poettering wrote: > On Mo, 21.10.19 17:50, Zbigniew Jędrzejewski-Szmek (zbys...@in.waw.pl) wrote: > > > In principle, the watchdog for services is nice. But in practice it seems > > be bring only grief. The Fedora bugtracker is full of automated reports of > > ABRTs, > > and of those that were fired by the watchdog, pretty much 100% are bogus, in > > the sense that the machine was resource starved and the watchdog fired. > > > > There a few downsides to the watchdog killing the service: > > 1. if it is something like logind, it is possible that it will cause > > user-visible > > failure of other services > > 2. restarting of the service causes additional load on the machine > > 3. coredump handling causes additional load on the machine, quite > > significant > > 4. those failures are reported in bugtrackers and waste everyone's time. > > > > I had the following ideas: > > 1. disable coredumps for watchdog abrts: systemd could set some flag > > on the unit or otherwise notify systemd-coredump about this, and it could > > just > > log the occurence but not dump the core file. > > 2. generally disable watchdogs and make them opt in. We have > > 'systemd-analyze service-watchdogs', > > and we could make the default configurable to "yes|no". > > > > What do you think? > > Isn't this more a reason to substantially increase the watchdog > interval by default? i.e. 30min if needed? Yep, there was a proposal like that. I want to make it 1h in Fedora. Zbyszek ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] is the watchdog useful?
On Mo, 21.10.19 17:50, Zbigniew Jędrzejewski-Szmek (zbys...@in.waw.pl) wrote: > In principle, the watchdog for services is nice. But in practice it seems > be bring only grief. The Fedora bugtracker is full of automated reports of > ABRTs, > and of those that were fired by the watchdog, pretty much 100% are bogus, in > the sense that the machine was resource starved and the watchdog fired. > > There a few downsides to the watchdog killing the service: > 1. if it is something like logind, it is possible that it will cause > user-visible > failure of other services > 2. restarting of the service causes additional load on the machine > 3. coredump handling causes additional load on the machine, quite significant > 4. those failures are reported in bugtrackers and waste everyone's time. > > I had the following ideas: > 1. disable coredumps for watchdog abrts: systemd could set some flag > on the unit or otherwise notify systemd-coredump about this, and it could just > log the occurence but not dump the core file. > 2. generally disable watchdogs and make them opt in. We have 'systemd-analyze > service-watchdogs', > and we could make the default configurable to "yes|no". > > What do you think? Isn't this more a reason to substantially increase the watchdog interval by default? i.e. 30min if needed? Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] is the watchdog useful?
On Fri, Oct 25, 2019 at 10:26:44AM +, Zbigniew Jędrzejewski-Szmek wrote: > On Thu, Oct 24, 2019 at 02:56:55PM -0700, Vito Caputo wrote: > > On Thu, Oct 24, 2019 at 10:45:32AM +, Zbigniew Jędrzejewski-Szmek wrote: > > > On Tue, Oct 22, 2019 at 04:35:13AM -0700, Vito Caputo wrote: > > > > On Tue, Oct 22, 2019 at 10:51:49AM +, Zbigniew Jędrzejewski-Szmek > > > > wrote: > > > > > On Tue, Oct 22, 2019 at 12:34:45PM +0200, Umut Tezduyar Lindskog > > > > > wrote: > > > > > > I am curious Zbigniew of how you find out if the coredump was on a > > > > > > starved > > > > > > process? > > > > > > > > > > A very common case is systemd-journald which gets SIGABRT when in a > > > > > read() or write() or similar syscall. Another case is when > > > > > systemd-udevd workers get ABRT when doing open() on a device. > > > > > > > > > > > > > In the case of journald, is it really in read()/write() syscalls you're > > > > seeing the SIGABRTs? > > > > > > I was sloppy here — it's not read/write, but various other syscalls. > > > In particular clone(), which makes sense, because it involves memory > > > allocation. > > > > > > > That's interesting, it's not like journald calls clone() a lot. > > Hm, maybe it was udevd that was calling clone(), not journald. > All the reports are available here: > https://bugzilla.redhat.com/show_bug.cgi?id=1300212 > Yeah, nearly all the 2019 reports you marked as duplicate were udev/logind, at least of the ones I could access. Surprisingly few were journald related at all, are those the ones marked private? > I opened a pull request to make the watchdog setting configurable > for our own internal services: https://github.com/systemd/systemd/pull/13843. > It will be interesting to see how many aborts continue to come in with the timeout @ 1h. Seeing aborts in epoll_wait() suggests these processes either weren't getting scheduled to run for 3 minutes despite having watchdog events to service on an fd, or were paged out and took longer than 3 minutes to page in and actually run. If it's the latter, maybe we can lock some of these in memory in response to another build-time configuration. Desktop distros can spare the memory for logind to not get paged out for instance. I know I wouldn't object to having the drm arbiter process always resident on my machine, it's pretty close to being a kernel component in that role. Regards, Vito Caputo ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] is the watchdog useful?
On Thu, Oct 24, 2019 at 02:56:55PM -0700, Vito Caputo wrote: > On Thu, Oct 24, 2019 at 10:45:32AM +, Zbigniew Jędrzejewski-Szmek wrote: > > On Tue, Oct 22, 2019 at 04:35:13AM -0700, Vito Caputo wrote: > > > On Tue, Oct 22, 2019 at 10:51:49AM +, Zbigniew Jędrzejewski-Szmek > > > wrote: > > > > On Tue, Oct 22, 2019 at 12:34:45PM +0200, Umut Tezduyar Lindskog wrote: > > > > > I am curious Zbigniew of how you find out if the coredump was on a > > > > > starved > > > > > process? > > > > > > > > A very common case is systemd-journald which gets SIGABRT when in a > > > > read() or write() or similar syscall. Another case is when > > > > systemd-udevd workers get ABRT when doing open() on a device. > > > > > > > > > > In the case of journald, is it really in read()/write() syscalls you're > > > seeing the SIGABRTs? > > > > I was sloppy here — it's not read/write, but various other syscalls. > > In particular clone(), which makes sense, because it involves memory > > allocation. > > > > That's interesting, it's not like journald calls clone() a lot. Hm, maybe it was udevd that was calling clone(), not journald. All the reports are available here: https://bugzilla.redhat.com/show_bug.cgi?id=1300212 I opened a pull request to make the watchdog setting configurable for our own internal services: https://github.com/systemd/systemd/pull/13843. Zbyszek ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] is the watchdog useful?
On Thu, Oct 24, 2019 at 10:45:32AM +, Zbigniew Jędrzejewski-Szmek wrote: > On Tue, Oct 22, 2019 at 04:35:13AM -0700, Vito Caputo wrote: > > On Tue, Oct 22, 2019 at 10:51:49AM +, Zbigniew Jędrzejewski-Szmek wrote: > > > On Tue, Oct 22, 2019 at 12:34:45PM +0200, Umut Tezduyar Lindskog wrote: > > > > I am curious Zbigniew of how you find out if the coredump was on a > > > > starved > > > > process? > > > > > > A very common case is systemd-journald which gets SIGABRT when in a > > > read() or write() or similar syscall. Another case is when > > > systemd-udevd workers get ABRT when doing open() on a device. > > > > > > > In the case of journald, is it really in read()/write() syscalls you're > > seeing the SIGABRTs? > > I was sloppy here — it's not read/write, but various other syscalls. > In particular clone(), which makes sense, because it involves memory > allocation. > That's interesting, it's not like journald calls clone() a lot. I presume that's due to the offline thread creation? Things could be refactored a bit to create the offline thread once @ journal file open and keep it around. I'd expect the mmap-related file-backed page faults to be more problematic as it's most of what journald is doing and requires both memory and IO. Maybe clone() is getting delayed from another source of contention? It is a bit of a unique syscall. Regards, Vito Caputo ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] is the watchdog useful?
On Tue, Oct 22, 2019 at 04:35:13AM -0700, Vito Caputo wrote: > On Tue, Oct 22, 2019 at 10:51:49AM +, Zbigniew Jędrzejewski-Szmek wrote: > > On Tue, Oct 22, 2019 at 12:34:45PM +0200, Umut Tezduyar Lindskog wrote: > > > I am curious Zbigniew of how you find out if the coredump was on a starved > > > process? > > > > A very common case is systemd-journald which gets SIGABRT when in a > > read() or write() or similar syscall. Another case is when > > systemd-udevd workers get ABRT when doing open() on a device. > > > > In the case of journald, is it really in read()/write() syscalls you're > seeing the SIGABRTs? I was sloppy here — it's not read/write, but various other syscalls. In particular clone(), which makes sense, because it involves memory allocation. Zbyszek ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] is the watchdog useful?
On Tue, Oct 22, 2019 at 10:51:49AM +, Zbigniew Jędrzejewski-Szmek wrote: > On Tue, Oct 22, 2019 at 12:34:45PM +0200, Umut Tezduyar Lindskog wrote: > > I am curious Zbigniew of how you find out if the coredump was on a starved > > process? > > A very common case is systemd-journald which gets SIGABRT when in a > read() or write() or similar syscall. Another case is when > systemd-udevd workers get ABRT when doing open() on a device. > In the case of journald, is it really in read()/write() syscalls you're seeing the SIGABRTs? When I worked on making journald offlining asynchronous, back then the pimary source of problems was very slow fsync() calls. But we've now moved those into a thread, so that particular source of aborts should be removed. I'd be a little surprised to hear it's now aborting in read()/write() calls, because journald's storage IO is all done through mmap() windows. At the time I was working on it, something I latently wanted to explore was converting journald to a more database-ish Direct-IO + AIO storage engine. Partially to eliminate the nondeterministic, potentially lengthy stalls a thrashing or otherwise unresponsive backing store could cause when journald faulted on an affected page in its mappings. As long as journald is mmap()-based, it will always be suceptible to long, uncontrolled delays blocking the event loop, as it uses a single process for both its event loop and structure modify operations on those mappings. If it used AIO the IOs would all be submitted asynchronously and completed as part of the same event loop responding to watchdog timeouts. So it'd be far less likely to starve servicing the event sources, including the watchdog timer, since that'd be its principal blocking point. I realize these details are a bit orthogonal to the watchdog, but it might help appreciate the extent of non-determinism causing such false positives. Programs like journald have not been structured carefully to minimize event loop delays the watchdog perceives as hangs. Of course, if the journald process itself began faulting in and out every time it was scheduled on a thrashing system, it could still become delayed enough for an aggressive watchdog to fire. It would be a lot easier to pin its pages in memory and insulate it from those situations if its working set were a relatively small, finite set used for dispatching an upper-bound of concurrent async IO, just mlock most of what it ever needs early on. But we're nowhere near that architecture today, and that's really the kind of approach needed to make things watchdog-appropriate with a minimum of false positives. Regards, Vito Caputo ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] is the watchdog useful?
On Tue, Oct 22, 2019 at 12:34:45PM +0200, Umut Tezduyar Lindskog wrote: > I am curious Zbigniew of how you find out if the coredump was on a starved > process? A very common case is systemd-journald which gets SIGABRT when in a read() or write() or similar syscall. Another case is when systemd-udevd workers get ABRT when doing open() on a device. > This is common for our embedded devices. I didn't think it is common for > desktop too. > It is really useful for getting coredumps on deadlocked applications. For > that reason I don't think it is good to remove this functionality > completely. Yes, I never suggested removing it completely. I'm just saying that for the type of systems that Fedora targets, I don't recall any actual deadlock. For more specialized systems, where the workload is more predictable, it makes sense to have the watchdog. There might be cases where the kernel is dead-locked internally, and e.g. open() or modprobe() never returns. For those cases it might be useful to get the backtrace, but actually killing the process and/or storing the coredump is useful. Zbyszek > > Umut > > On Mon, Oct 21, 2019 at 7:51 PM Zbigniew Jędrzejewski-Szmek < > zbys...@in.waw.pl> wrote: > > > In principle, the watchdog for services is nice. But in practice it seems > > be bring only grief. The Fedora bugtracker is full of automated reports of > > ABRTs, > > and of those that were fired by the watchdog, pretty much 100% are bogus, > > in > > the sense that the machine was resource starved and the watchdog fired. > > > > There a few downsides to the watchdog killing the service: > > 1. if it is something like logind, it is possible that it will cause > > user-visible > > failure of other services > > 2. restarting of the service causes additional load on the machine > > 3. coredump handling causes additional load on the machine, quite > > significant > > 4. those failures are reported in bugtrackers and waste everyone's time. > > > > I had the following ideas: > > 1. disable coredumps for watchdog abrts: systemd could set some flag > > on the unit or otherwise notify systemd-coredump about this, and it could > > just > > log the occurence but not dump the core file. > > 2. generally disable watchdogs and make them opt in. We have > > 'systemd-analyze service-watchdogs', > > and we could make the default configurable to "yes|no". > > > > What do you think? > > Zbyszek > > ___ > > systemd-devel mailing list > > systemd-devel@lists.freedesktop.org > > https://lists.freedesktop.org/mailman/listinfo/systemd-devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] is the watchdog useful?
I am curious Zbigniew of how you find out if the coredump was on a starved process? This is common for our embedded devices. I didn't think it is common for desktop too. It is really useful for getting coredumps on deadlocked applications. For that reason I don't think it is good to remove this functionality completely. Umut On Mon, Oct 21, 2019 at 7:51 PM Zbigniew Jędrzejewski-Szmek < zbys...@in.waw.pl> wrote: > In principle, the watchdog for services is nice. But in practice it seems > be bring only grief. The Fedora bugtracker is full of automated reports of > ABRTs, > and of those that were fired by the watchdog, pretty much 100% are bogus, > in > the sense that the machine was resource starved and the watchdog fired. > > There a few downsides to the watchdog killing the service: > 1. if it is something like logind, it is possible that it will cause > user-visible > failure of other services > 2. restarting of the service causes additional load on the machine > 3. coredump handling causes additional load on the machine, quite > significant > 4. those failures are reported in bugtrackers and waste everyone's time. > > I had the following ideas: > 1. disable coredumps for watchdog abrts: systemd could set some flag > on the unit or otherwise notify systemd-coredump about this, and it could > just > log the occurence but not dump the core file. > 2. generally disable watchdogs and make them opt in. We have > 'systemd-analyze service-watchdogs', > and we could make the default configurable to "yes|no". > > What do you think? > Zbyszek > ___ > systemd-devel mailing list > systemd-devel@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/systemd-devel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] is the watchdog useful?
On Tue, 22 Oct 2019 09:17:42 + Zbigniew Jędrzejewski-Szmek wrote: > On Tue, Oct 22, 2019 at 09:54:31AM +0300, Pekka Paalanen wrote: > > On Mon, 21 Oct 2019 17:50:44 + > > Zbigniew Jędrzejewski-Szmek wrote: > > > > > In principle, the watchdog for services is nice. But in practice it seems > > > be bring only grief. The Fedora bugtracker is full of automated reports > > > of ABRTs, > > > and of those that were fired by the watchdog, pretty much 100% are bogus, > > > in > > > the sense that the machine was resource starved and the watchdog fired. > > > > Hi, > > > > just curious, is that resource starvation caused by something big, e.g. > > a browser, using too much memory which leads to the kernel reclaiming > > also pages of program text sections because they can be reloaded from > > disk at any time, however those pages are needed again immediately > > after when some CPU core switches process context, leading to something > > that looks like a hard freeze to a user, while the kernel is furiously > > loading pages from disk just to drop them again, and can take from > > minutes to hours before any progress is visible? > > I don't really know. Unfortunately, abrt in Fedora does not collect log > messages. In the old syslog days, a snippet of /var/log/messages for the > last 20 minutes or something like that before a crash would be copied > into the bug report, and this would include kernel messages about disk > errors, or kernel stalls, or other interesting hints. Unfortunately > nowadays, because of privacy concerns (?) and an effort to make things > more efficient (?), just some heavily-filtered journalctl output is > attached. In practice, usually this is at most a few lines and > completely useless. In particular, it does not give any hints to the > overall state of the system. > > I have spoken to abrt maintainers about this, but it seems that this > problem is specific to systemd, and for most other applications it is > OK to get a backtrace without any system-wide context. So I don't see > this changing any time soon ;( > > Sometimes I ask people for logs, and sometimes I get them, and in those > cases it seems that both hardware issues (e.g. a failing disk), or memory > exhaustion are often involved. In some cases there is no clear reason. > And since in the great majority we don't have any logs, it is hard to > say anything. > > > It has happened to me on Fedora in the past. I could probably dig up > > discussions about the problem in general if you want, they explain it > > better than I ever could. > > > > Does Fedora prevent that situation by tuning some kernel knobs nowadays > > for desktops? > > I don't think so. For the case I described, I don't think there are really any error messages produced, unless OOM killer finally manages to kill something. That may take half an hour to finally kick in, or even longer. The kernel is running just fine but userspace makes little to no progress. Personally I was just lucky to reproduce the issue while iotop was running and I saw 100% disk I/O used for purely reading, soon after everything just ground to halt. Maybe that could explain some of the reports. Thanks, pq pgpfOWpjVSfQw.pgp Description: OpenPGP digital signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] is the watchdog useful?
On Mon, Oct 21, 2019 at 02:32:08PM -0700, Vito Caputo wrote: > On Mon, Oct 21, 2019 at 05:50:44PM +, Zbigniew Jędrzejewski-Szmek wrote: > > In principle, the watchdog for services is nice. But in practice it seems > > be bring only grief. The Fedora bugtracker is full of automated reports of > > ABRTs, > > and of those that were fired by the watchdog, pretty much 100% are bogus, > > in > > the sense that the machine was resource starved and the watchdog fired. > > > > There a few downsides to the watchdog killing the service: > > 1. if it is something like logind, it is possible that it will cause > > user-visible > > failure of other services > > 2. restarting of the service causes additional load on the machine > > 3. coredump handling causes additional load on the machine, quite > > significant > > 4. those failures are reported in bugtrackers and waste everyone's time. > > > > I had the following ideas: > > 1. disable coredumps for watchdog abrts: systemd could set some flag > > on the unit or otherwise notify systemd-coredump about this, and it could > > just > > log the occurence but not dump the core file. > > 2. generally disable watchdogs and make them opt in. We have > > 'systemd-analyze service-watchdogs', > > and we could make the default configurable to "yes|no". > > > > What do you think? > > Zbyszek > > > I think the main issue is the watchdog timeout hasn't been tuned > appropriately for the environment it's being applied. > > It's as if the timeouts are somewhere near the hard real-time > expectations end of the spectrum, while being applied to > non-deterministically delayed and scheduled normal priority userspace > processes. It's a sort of impedance mismatch. > > I /think/ the purpose of the watchdog is to detect when processes are > permanently wedged, capture their state for debugging, and forcefully > unwedge them. > > That seems perfectly reasonable, but the timeout heuristic being used, > given our non-deterministic scheduling, should be incredibly long by > default. It's not the kind of thing you want false positives on, folks > can always shrink the timeout if they find it's desirable. It is now 3 minutes in all systemd units. Dunno, maybe we should make that 30 minutes. Zbyszek > Without having spent much time thinking about this, I'd lean towards > retaining the watchdogs but making their default timeouts so long a > program would have to be wedged for an hour+ before it triggered. > > At least that way we preserve the passive information gathering of > serious bugs which might otherwise go unnoticed with background/idle > services, improving debugging substantially, but eliminate the problems > you describe resulting from false positives. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] is the watchdog useful?
On Tue, Oct 22, 2019 at 09:54:31AM +0300, Pekka Paalanen wrote: > On Mon, 21 Oct 2019 17:50:44 + > Zbigniew Jędrzejewski-Szmek wrote: > > > In principle, the watchdog for services is nice. But in practice it seems > > be bring only grief. The Fedora bugtracker is full of automated reports of > > ABRTs, > > and of those that were fired by the watchdog, pretty much 100% are bogus, > > in > > the sense that the machine was resource starved and the watchdog fired. > > Hi, > > just curious, is that resource starvation caused by something big, e.g. > a browser, using too much memory which leads to the kernel reclaiming > also pages of program text sections because they can be reloaded from > disk at any time, however those pages are needed again immediately > after when some CPU core switches process context, leading to something > that looks like a hard freeze to a user, while the kernel is furiously > loading pages from disk just to drop them again, and can take from > minutes to hours before any progress is visible? I don't really know. Unfortunately, abrt in Fedora does not collect log messages. In the old syslog days, a snippet of /var/log/messages for the last 20 minutes or something like that before a crash would be copied into the bug report, and this would include kernel messages about disk errors, or kernel stalls, or other interesting hints. Unfortunately nowadays, because of privacy concerns (?) and an effort to make things more efficient (?), just some heavily-filtered journalctl output is attached. In practice, usually this is at most a few lines and completely useless. In particular, it does not give any hints to the overall state of the system. I have spoken to abrt maintainers about this, but it seems that this problem is specific to systemd, and for most other applications it is OK to get a backtrace without any system-wide context. So I don't see this changing any time soon ;( Sometimes I ask people for logs, and sometimes I get them, and in those cases it seems that both hardware issues (e.g. a failing disk), or memory exhaustion are often involved. In some cases there is no clear reason. And since in the great majority we don't have any logs, it is hard to say anything. > It has happened to me on Fedora in the past. I could probably dig up > discussions about the problem in general if you want, they explain it > better than I ever could. > > Does Fedora prevent that situation by tuning some kernel knobs nowadays > for desktops? I don't think so. Zbyszek ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] is the watchdog useful?
On Mon, 21 Oct 2019 17:50:44 + Zbigniew Jędrzejewski-Szmek wrote: > In principle, the watchdog for services is nice. But in practice it seems > be bring only grief. The Fedora bugtracker is full of automated reports of > ABRTs, > and of those that were fired by the watchdog, pretty much 100% are bogus, in > the sense that the machine was resource starved and the watchdog fired. Hi, just curious, is that resource starvation caused by something big, e.g. a browser, using too much memory which leads to the kernel reclaiming also pages of program text sections because they can be reloaded from disk at any time, however those pages are needed again immediately after when some CPU core switches process context, leading to something that looks like a hard freeze to a user, while the kernel is furiously loading pages from disk just to drop them again, and can take from minutes to hours before any progress is visible? It has happened to me on Fedora in the past. I could probably dig up discussions about the problem in general if you want, they explain it better than I ever could. Does Fedora prevent that situation by tuning some kernel knobs nowadays for desktops? Thanks, pq pgp5LISA40kDi.pgp Description: OpenPGP digital signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] is the watchdog useful?
On Mon, Oct 21, 2019 at 05:50:44PM +, Zbigniew Jędrzejewski-Szmek wrote: > In principle, the watchdog for services is nice. But in practice it seems > be bring only grief. The Fedora bugtracker is full of automated reports of > ABRTs, > and of those that were fired by the watchdog, pretty much 100% are bogus, in > the sense that the machine was resource starved and the watchdog fired. > > There a few downsides to the watchdog killing the service: > 1. if it is something like logind, it is possible that it will cause > user-visible > failure of other services > 2. restarting of the service causes additional load on the machine > 3. coredump handling causes additional load on the machine, quite significant > 4. those failures are reported in bugtrackers and waste everyone's time. > > I had the following ideas: > 1. disable coredumps for watchdog abrts: systemd could set some flag > on the unit or otherwise notify systemd-coredump about this, and it could just > log the occurence but not dump the core file. > 2. generally disable watchdogs and make them opt in. We have 'systemd-analyze > service-watchdogs', > and we could make the default configurable to "yes|no". > > What do you think? > Zbyszek I think the main issue is the watchdog timeout hasn't been tuned appropriately for the environment it's being applied. It's as if the timeouts are somewhere near the hard real-time expectations end of the spectrum, while being applied to non-deterministically delayed and scheduled normal priority userspace processes. It's a sort of impedance mismatch. I /think/ the purpose of the watchdog is to detect when processes are permanently wedged, capture their state for debugging, and forcefully unwedge them. That seems perfectly reasonable, but the timeout heuristic being used, given our non-deterministic scheduling, should be incredibly long by default. It's not the kind of thing you want false positives on, folks can always shrink the timeout if they find it's desirable. Without having spent much time thinking about this, I'd lean towards retaining the watchdogs but making their default timeouts so long a program would have to be wedged for an hour+ before it triggered. At least that way we preserve the passive information gathering of serious bugs which might otherwise go unnoticed with background/idle services, improving debugging substantially, but eliminate the problems you describe resulting from false positives. Regards, Vito Caputo ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
