Hi all,
We've been happily using god as part of our AppScale project (
http://github.com/AppScale/appscale) for about three years now and have
been happy with how it monitors processes and restarts them if they die for
some reason. However, we've had a lot of problems trying to get god to (1)
revive dead processes AND (2) kill processes using too much memory. We're
currently only reviving dead processes with the following god config file:
WATCH = "{0}"
START_CMD = "{1}"
STOP_CMD = "{2}"
PORTS = [{3}]
PORTS.each do |port|
God.watch do |w|
w.name = "#{5}-#{6}"
w.group = WATCH
w.interval = 30.seconds # default
w.start = START_CMD
w.stop = STOP_CMD
w.stop_signal = 'QUIT'
w.stop_timeout = 5.seconds
w.start_grace = 20.seconds
w.restart_grace = 20.seconds
w.log = "/var/log/appscale/#{5}-#{6}.log"
w.pid_file = "/var/appscale/#{5}-#{6}.pid"
w.behavior(:clean_pid_file)
w.start_if do |start|
start.condition(:process_running) do |c|
c.running = false
end
end
w.restart_if do |restart|
restart.condition(:memory_usage) do |c|
c.above = 150.megabytes
c.times = [3, 5] # 3 out of 5 intervals
end
restart.condition(:cpu_usage) do |c|
c.above = 50.percent
c.times = 5
end
end
# lifecycle
w.lifecycle do |on|
on.condition(:flapping) do |c|
c.to_state = [:start, :restart]
c.times = 5
c.within = 5.minute
c.transition = :unmonitored
c.retry_in = 10.minutes
c.retry_times = 5
c.retry_within = 2.hours
end
end
{4}
end
end
Of course, that's not complete, since we slip in some template values here
and there. This looks like it should kill processes using too much memory,
but god doesn't kill them. If I use the much more simple syntax
God.watch do |w|
w.name = "simple"
w.start = "ruby /full/path/to/simple.rb"
w.keepalive(:memory_max => 150.megabytes)end
instead, then god will kill my processes if they use too much memory, but
if they die for any other reason, god won't revive them. In that case, they
move from the init to the up state, and god complains that the process
isn't running, but takes no corrective action. Any ideas? We've been really
happy with god here but need to prevent rogue processes that take up too
much CPU or memory from hosing the system.
Thanks in advance!
--
You received this message because you are subscribed to the Google Groups
"god.rb" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/god-rb.
For more options, visit https://groups.google.com/groups/opt_out.
