Hi, I have monit setup for monitoring some background workers in my rails project. Whenever I deploy new code, monit should take care of restarting them after they are shut down. But every time I deploy, monit only starts up half of them. If I ssh into the server though and run "sudo monit validate" then it correctly sees that they are not running, and spins them up. But if I don't run that command manually, then nothing happens. What could be wrong, I have no idea how to debug it further?
Here is my configs: ############################## Monitrc ################################# set daemon 10 set logfile /var/log/monit.log set idfile /var/lib/monit/id set statefile /var/lib/monit/state set eventqueue basedir /var/lib/monit/events slots 100 set eventqueue basedir /var/monit/ slots 1000 set mmonit http://monit:[email protected]:8080/collector set httpd port 2812 and use address 192.168.0.3 allow localhost allow 192.168.0.1 allow user:password check system master-worker-server if loadavg(5min) > 4 for 60 cycles then alert if memory > 75% for 60 cycles then alert if cpu(user) > 75% for 60 cycles then alert include /etc/monit/conf.d/* ############################# /etc/monit/conf.d/sidekiq.conf ###################################### # Check for Ruby sidekiq worker process check process da_workers-0.pid with pidfile /home/deployer/apps/au/shared/pids/workers/da_workers-0.pid start program = "/bin/bash -l -c 'HOME=/home/deployer PATH=$HOME/.rbenv/shims:$HOME/.rbenv/bin:$PATH RAILS_ENV=production && cd /home/deployer/apps/au/current ; nohup bundle exec sidekiq -e production -C /home/deployer/apps/au/shared/config/workers/da_workers.yml -i 0 -P /home/deployer/apps/au/shared/pids/workers/da_workers-0.pid >> /home/deployer/apps/au/current/log/sidekiq.log 2>&1 &'" as uid deployer and gid deployer with timeout 250 seconds stop program = "/bin/bash -l -c 'HOME=/home/deployer PATH=$HOME/.rbenv/shims:$HOME/.rbenv/bin:$PATH RAILS_ENV=production && if [ -d /home/deployer/apps/au/current ] && [ -f /home/deployer/apps/au/shared/pids/workers/da_workers-0.pid ] && kill -0 `cat /home/deployer/apps/au/shared/pids/workers/da_workers-0.pid`> /dev/null 2>&1; then cd /home/deployer/apps/au/current && bundle exec sidekiqctl stop /home/deployer/apps/au/shared/pids/workers/da_workers-0.pid 3 ; else echo 'Sidekiq is not running' ; fi'" as uid deployer and gid deployer with timeout 120 seconds if cpu usage > 50% for 18 cycles then restart if mem > 1200.0 MB for 18 cycles then restart if 3 restarts within 18 cycles then timeout check process da_data_maintenance_workers-0.pid with pidfile /home/deployer/apps/au/shared/pids/workers/da_data_maintenance_workers-0.pid start program = "/bin/bash -l -c 'HOME=/home/deployer PATH=$HOME/.rbenv/shims:$HOME/.rbenv/bin:$PATH RAILS_ENV=production && cd /home/deployer/apps/au/current ; nohup bundle exec sidekiq -e production -C /home/deployer/apps/au/shared/config/workers/da_data_maintenance_workers.yml -i 0 -P /home/deployer/apps/au/shared/pids/workers/da_data_maintenance_workers-0.pid >> /home/deployer/apps/au/current/log/sidekiq.log 2>&1 &'" as uid deployer and gid deployer with timeout 250 seconds stop program = "/bin/bash -l -c 'HOME=/home/deployer PATH=$HOME/.rbenv/shims:$HOME/.rbenv/bin:$PATH RAILS_ENV=production && if [ -d /home/deployer/apps/au/current ] && [ -f /home/deployer/apps/au/shared/pids/workers/da_data_maintenance_workers-0.pid ] && kill -0 `cat /home/deployer/apps/au/shared/pids/workers/da_data_maintenance_workers-0.pid`> /dev/null 2>&1; then cd /home/deployer/apps/au/current && bundle exec sidekiqctl stop /home/deployer/apps/au/shared/pids/workers/da_data_maintenance_workers-0.pid 3 ; else echo 'Sidekiq is not running' ; fi'" as uid deployer and gid deployer with timeout 120 seconds if cpu usage > 50% for 18 cycles then restart if mem > 1200.0 MB for 18 cycles then restart if 3 restarts within 18 cycles then timeout check process da_data_collecting_workers-0.pid with pidfile /home/deployer/apps/au/shared/pids/workers/da_data_collecting_workers-0.pid start program = "/bin/bash -l -c 'HOME=/home/deployer PATH=$HOME/.rbenv/shims:$HOME/.rbenv/bin:$PATH RAILS_ENV=production && cd /home/deployer/apps/au/current ; nohup bundle exec sidekiq -e production -C /home/deployer/apps/au/shared/config/workers/da_data_collecting_workers.yml -i 0 -P /home/deployer/apps/au/shared/pids/workers/da_data_collecting_workers-0.pid >> /home/deployer/apps/au/current/log/sidekiq.log 2>&1 &'" as uid deployer and gid deployer with timeout 250 seconds stop program = "/bin/bash -l -c 'HOME=/home/deployer PATH=$HOME/.rbenv/shims:$HOME/.rbenv/bin:$PATH RAILS_ENV=production && if [ -d /home/deployer/apps/au/current ] && [ -f /home/deployer/apps/au/shared/pids/workers/da_data_collecting_workers-0.pid ] && kill -0 `cat /home/deployer/apps/au/shared/pids/workers/da_data_collecting_workers-0.pid`> /dev/null 2>&1; then cd /home/deployer/apps/au/current && bundle exec sidekiqctl stop /home/deployer/apps/au/shared/pids/workers/da_data_collecting_workers-0.pid 3 ; else echo 'Sidekiq is not running' ; fi'" as uid deployer and gid deployer with timeout 120 seconds if cpu usage > 50% for 18 cycles then restart if mem > 1200.0 MB for 18 cycles then restart if 3 restarts within 18 cycles then timeout check process da_data_collecting_workers-1.pid with pidfile /home/deployer/apps/au/shared/pids/workers/da_data_collecting_workers-1.pid start program = "/bin/bash -l -c 'HOME=/home/deployer PATH=$HOME/.rbenv/shims:$HOME/.rbenv/bin:$PATH RAILS_ENV=production && cd /home/deployer/apps/au/current ; nohup bundle exec sidekiq -e production -C /home/deployer/apps/au/shared/config/workers/da_data_collecting_workers.yml -i 1 -P /home/deployer/apps/au/shared/pids/workers/da_data_collecting_workers-1.pid >> /home/deployer/apps/au/current/log/sidekiq.log 2>&1 &'" as uid deployer and gid deployer with timeout 250 seconds stop program = "/bin/bash -l -c 'HOME=/home/deployer PATH=$HOME/.rbenv/shims:$HOME/.rbenv/bin:$PATH RAILS_ENV=production && if [ -d /home/deployer/apps/au/current ] && [ -f /home/deployer/apps/au/shared/pids/workers/da_data_collecting_workers-1.pid ] && kill -0 `cat /home/deployer/apps/au/shared/pids/workers/da_data_collecting_workers-1.pid`> /dev/null 2>&1; then cd /home/deployer/apps/au/current && bundle exec sidekiqctl stop /home/deployer/apps/au/shared/pids/workers/da_data_collecting_workers-1.pid 3 ; else echo 'Sidekiq is not running' ; fi'" as uid deployer and gid deployer with timeout 120 seconds if cpu usage > 50% for 18 cycles then restart if mem > 1200.0 MB for 18 cycles then restart if 3 restarts within 18 cycles then timeout -- To unsubscribe: https://lists.nongnu.org/mailman/listinfo/monit-general
