I think I have found the issue, and I think I may have to walk back my statements on this affecting a certain version or distro.
I have 8 monitored services in an "Execution failed" state. None of the services have a timeout defined. The timeout apparently defaults to EXEC_TIMEOUT (30 seconds). monit waits the full 30 seconds for the service check to finally fail before checking the next service that is also in an "Execution failed" state. [EDT Aug 5 13:03:58] error : 'service_name' process is not running [EDT Aug 5 13:03:58] info : 'service_name' trying to restart [EDT Aug 5 13:03:58] info : 'service_name' start: /etc/init.d/ service_name [EDT Aug 5 13:03:58] info : Sleeping for 100 ms (src/control.c:127) [EDT Aug 5 13:03:58] info : Sleeping for 100 ms (src/control.c:127) [EDT Aug 5 13:03:58] info : Sleeping for 50000 ms (src/control.c:159) [EDT Aug 5 13:03:58] info : Sleeping for 100000 ms (src/control.c:159) [EDT Aug 5 13:03:58] info : Sleeping for 200000 ms (src/control.c:159) [EDT Aug 5 13:03:58] info : Sleeping for 400000 ms (src/control.c:159) [EDT Aug 5 13:03:59] info : Sleeping for 800000 ms (src/control.c:159) [EDT Aug 5 13:04:00] info : Sleeping for 1600000 ms (src/control.c:159) [EDT Aug 5 13:04:01] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:02] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:03] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:04] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:05] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:06] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:07] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:08] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:09] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:10] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:11] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:12] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:13] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:14] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:15] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:16] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:17] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:18] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:19] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:20] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:21] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:22] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:23] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:24] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:25] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:26] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:27] info : Sleeping for 1000000 ms (src/control.c:159) [EDT Aug 5 13:04:28] error : 'service_name' failed to start (exit status 1) -- /etc/init.d/service_name: Shutting down service_name: [ OK ] Starting service_name: [ OK ]^M[FAILED] 8 services at 30 seconds each = 240 seconds, this means the sleep(Run.polltime) in monit.c only gets called every 4 minutes. This is with the daemon interval set to 10 seconds. Notice ~240 seconds (4 minutes) between each occurrence: # grep 'src/monit.c' /var/log/monit [EDT Aug 5 12:56:33] info : Sleeping for 10 seconds (src/monit.c:561) [EDT Aug 5 13:00:46] info : Sleeping for 10 seconds (src/monit.c:561) [EDT Aug 5 13:05:00] info : Sleeping for 10 seconds (src/monit.c:561) So how can I control the execTimeout without having monit give up on trying to start that service? Thanks, On Fri, Aug 5, 2016 at 11:43 AM, Geoff Goas <[email protected]> wrote: > Hello, > > Thanks for the suggestions. In RHEL/CentOS 5 and 6, the default config > is /etc/monit.conf. User configs are ~/.monit.conf. This is the only > change to the source that is being applied by the package maintainer. > > Package listing: > > # rpm -ql monit > /etc/logrotate.d/monit > /etc/monit.conf > /etc/monit.d > /etc/monit.d/logging > /etc/rc.d/init.d/monit > /usr/bin/monit > /usr/share/doc/monit-5.14 > /usr/share/doc/monit-5.14/COPYING > /usr/share/doc/monit-5.14/README > /usr/share/man/man1/monit.1.gz > /var/log/monit > > From an strace of monit starting up: > > getcwd("/etc/monit.d", 4096) = 13 > stat("/root/.monit.conf", 0x7fff87cc7560) = -1 ENOENT (No such file or > directory) > stat("/etc/monit.conf", {st_mode=S_IFREG|0600, st_size=11346, ...}) = 0 > open("/etc/monit.conf", O_RDONLY) = 3 > open("/etc/monit.d", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 4 > > Showing that the set daemon directive is specified only once: > > # grep 'set daemon' /etc/monit.conf > #set daemon 30 # check services at 30 seconds intervals > > # grep 'set daemon' /etc/monit.d/* > /etc/monit.d/00base.conf:set daemon 50 > > Here is the monit log showing the 30 second interval even though it is > set to 50: > > # grep 'Aborting event' /var/log/monit | tail -n20 > [EDT Aug 4 17:05:06] error : Aborting event > [EDT Aug 4 17:05:36] error : Aborting event > [EDT Aug 4 17:06:06] error : Aborting event > [EDT Aug 4 17:06:37] error : Aborting event > [EDT Aug 4 17:06:37] error : Aborting event > [EDT Aug 4 17:07:07] error : Aborting event > [EDT Aug 4 17:07:07] error : Aborting event > [EDT Aug 4 22:18:13] error : Aborting event > [EDT Aug 4 22:18:43] error : Aborting event > [EDT Aug 4 22:19:13] error : Aborting event > [EDT Aug 4 22:19:44] error : Aborting event > [EDT Aug 4 22:20:14] error : Aborting event > [EDT Aug 4 22:20:44] error : Aborting event > [EDT Aug 4 22:21:15] error : Aborting event > [EDT Aug 4 22:21:15] error : Aborting event > [EDT Aug 4 22:21:45] error : Aborting event > [EDT Aug 4 22:21:45] error : Aborting event > [EDT Aug 5 11:19:40] error : Aborting event > [EDT Aug 5 11:20:10] error : Aborting event > [EDT Aug 5 11:23:53] error : Aborting event > > This behavior is occurring across multiple CentOS 6 hosts. All of the > CentOS 5 hosts running 4.11 and 5.2 with nearly identical > configurations ("alert...on restart" changed to "alert...on nonexist" > on the monit 5.x instances) do not have this issue. > > I'm open to more suggestions but I feel as though I will end up having > to get some more debug out of monit. > > Thanks, > > On Aug 5, 2016 9:16 AM, "Martin Pala" <[email protected]> wrote: > > > > Monit's default configuration file is /etc/monitrc ... the > /etc/monit.conf is not used, unless it was added to the search path by 3rd > party (for example package maintainer). > > > > There could be also ".monitrc" file in your home directory ... the > default search sequence for monit configuration file: > > > > ~/.monitrc > > /etc/monitrc > > @SYSCONFDIR/monitrc > > /usr/local/etc/monitrc > > ./monitrc > > > > > > > > > > > On 05 Aug 2016, at 15:07, Geoff Goas <[email protected]> wrote: > > > > > > I am setting it only in /etc/monit.conf. It is not being set in any > other configuration within /etc/monit.d. > > > > > > On Aug 5, 2016 9:03 AM, "Martin Pala" <[email protected]> wrote: > > > Hello, > > > > > > you have most probably two configuration files - the one which you > changed is different from the file used by monit. > > > > > > Best regards, > > > Martin > > > > > > > > >> On 05 Aug 2016, at 04:42, Geoff Goas <[email protected]> wrote: > > >> > > >> Hello, > > >> > > >> I'm having an issue with the CentOS 6 release of monit 5.14. I have > set the daemon interval to 5, 10, and 50 seconds - monit was fully > restarted for each adjustment of the interval - yet it still polls every 30 > seconds as if the configured value is being ignored. I also attempting > passing the interval using the -d switch to no avail. > > >> > > >> My testing consisted of having monit attempt to start a service that > could never possibly start, and without any timeout set. The log shows a 30 > second interval between service checks, and so does an strace of the monit > process. > > >> > > >> I have monit 5.2 running on CentOS 5.2 with a nearly identical > configuration. On that host, I have the daemon interval set to 10 seconds, > and it is polling at that interval just fine. > > >> > > >> Do you have any recommendations on what to check next? > > >> > > >> Thanks, > > >> > > >> -- > > >> Geoff Goas > > >> Systems Engineer > > >> -- > > >> To unsubscribe: > > >> https://lists.nongnu.org/mailman/listinfo/monit-general > > > > > > > > > -- > > > To unsubscribe: > > > https://lists.nongnu.org/mailman/listinfo/monit-general > > > -- > > > To unsubscribe: > > > https://lists.nongnu.org/mailman/listinfo/monit-general > > > > > > -- > > To unsubscribe: > > https://lists.nongnu.org/mailman/listinfo/monit-general > -- *Geoff GoasSystems Engineer*
-- To unsubscribe: https://lists.nongnu.org/mailman/listinfo/monit-general
