Re: workers dying overnight
Update: I upgraded my stack to Ruby 1.9.1 and the crashes seem to have stopped. (Knock wood!) heroku stack:migrate bamboo-mri-1.9.1 On Jul 30, 10:28 am, Alex Chaffee ale...@gmail.com wrote: What a useful command! Thanks. Still, I don't know what I could be doing to cause DJ to crash. Doesn't it catch all exceptions raised by jobs? My jobs are just doing plain vanilla Ruby stuff... - A On Jul 29, 8:22 am, Dan Croak dcr...@thoughtbot.com wrote: Only time I've had issues with workers is when I messed up something on my side so that I was causing DJ to crash. I debugged it by doing: heroku ps On Thu, Jul 29, 2010 at 11:10 AM, Alex Chaffee a...@cohuman.com wrote: I wrote a simple Heroku app to do service monitoring. It uses DelayedJob to queue up repeating tasks like doing an HTTP ping of a server once a minute or once an hour. Twice now the worker has died overnight. heroku workers still says I'm running 1 worker, but there's no message in heroku logs -- not even the stub of logs from the DJ worker that was there before. It looks like the worker instance has been retired by the scheduler. If I run heroku workers 0; heroku workers 1 then it comes back and my jobs all start running. Aha! It just started up a worker on its own after I hit the web page for the first time today. I think the scheduler needs to be smarter about worker proceses... maybe it needs to check the database for any Delayed Jobs whose run_at is soon, and either not retire it or make sure a worker comes online in time to run it. The code is athttp://github.com/alexch/sentryifyou want to try it yourself. Has anyone else had this problem? -- Alex Chaffee - a...@cohuman.com -http://alexch.github.com Stalk me:http://friendfeed.com/alexch|http://twitter.com/alexch| http://alexch.tumblr.com -- You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/heroku?hl=en. -- You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: workers dying overnight
What a useful command! Thanks. Still, I don't know what I could be doing to cause DJ to crash. Doesn't it catch all exceptions raised by jobs? My jobs are just doing plain vanilla Ruby stuff... - A On Jul 29, 8:22 am, Dan Croak dcr...@thoughtbot.com wrote: Only time I've had issues with workers is when I messed up something on my side so that I was causing DJ to crash. I debugged it by doing: heroku ps On Thu, Jul 29, 2010 at 11:10 AM, Alex Chaffee a...@cohuman.com wrote: I wrote a simple Heroku app to do service monitoring. It uses DelayedJob to queue up repeating tasks like doing an HTTP ping of a server once a minute or once an hour. Twice now the worker has died overnight. heroku workers still says I'm running 1 worker, but there's no message in heroku logs -- not even the stub of logs from the DJ worker that was there before. It looks like the worker instance has been retired by the scheduler. If I run heroku workers 0; heroku workers 1 then it comes back and my jobs all start running. Aha! It just started up a worker on its own after I hit the web page for the first time today. I think the scheduler needs to be smarter about worker proceses... maybe it needs to check the database for any Delayed Jobs whose run_at is soon, and either not retire it or make sure a worker comes online in time to run it. The code is athttp://github.com/alexch/sentryif you want to try it yourself. Has anyone else had this problem? -- Alex Chaffee - a...@cohuman.com -http://alexch.github.com Stalk me:http://friendfeed.com/alexch|http://twitter.com/alexch| http://alexch.tumblr.com -- You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/heroku?hl=en. -- You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
workers dying overnight
I wrote a simple Heroku app to do service monitoring. It uses DelayedJob to queue up repeating tasks like doing an HTTP ping of a server once a minute or once an hour. Twice now the worker has died overnight. heroku workers still says I'm running 1 worker, but there's no message in heroku logs -- not even the stub of logs from the DJ worker that was there before. It looks like the worker instance has been retired by the scheduler. If I run heroku workers 0; heroku workers 1 then it comes back and my jobs all start running. Aha! It just started up a worker on its own after I hit the web page for the first time today. I think the scheduler needs to be smarter about worker proceses... maybe it needs to check the database for any Delayed Jobs whose run_at is soon, and either not retire it or make sure a worker comes online in time to run it. The code is at http://github.com/alexch/sentry if you want to try it yourself. Has anyone else had this problem? -- Alex Chaffee - a...@cohuman.com - http://alexch.github.com Stalk me: http://friendfeed.com/alexch | http://twitter.com/alexch | http://alexch.tumblr.com -- You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: workers dying overnight
Only time I've had issues with workers is when I messed up something on my side so that I was causing DJ to crash. I debugged it by doing: heroku ps On Thu, Jul 29, 2010 at 11:10 AM, Alex Chaffee a...@cohuman.com wrote: I wrote a simple Heroku app to do service monitoring. It uses DelayedJob to queue up repeating tasks like doing an HTTP ping of a server once a minute or once an hour. Twice now the worker has died overnight. heroku workers still says I'm running 1 worker, but there's no message in heroku logs -- not even the stub of logs from the DJ worker that was there before. It looks like the worker instance has been retired by the scheduler. If I run heroku workers 0; heroku workers 1 then it comes back and my jobs all start running. Aha! It just started up a worker on its own after I hit the web page for the first time today. I think the scheduler needs to be smarter about worker proceses... maybe it needs to check the database for any Delayed Jobs whose run_at is soon, and either not retire it or make sure a worker comes online in time to run it. The code is at http://github.com/alexch/sentry if you want to try it yourself. Has anyone else had this problem? -- Alex Chaffee - a...@cohuman.com - http://alexch.github.com Stalk me: http://friendfeed.com/alexch | http://twitter.com/alexch | http://alexch.tumblr.com -- You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en. -- You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: workers dying overnight
That's a scary subject line!! ;) -- Pito On Thu, Jul 29, 2010 at 11:10 AM, Alex Chaffee a...@cohuman.com wrote: I wrote a simple Heroku app to do service monitoring. It uses DelayedJob to queue up repeating tasks like doing an HTTP ping of a server once a minute or once an hour. Twice now the worker has died overnight. heroku workers still says I'm running 1 worker, but there's no message in heroku logs -- not even the stub of logs from the DJ worker that was there before. It looks like the worker instance has been retired by the scheduler. If I run heroku workers 0; heroku workers 1 then it comes back and my jobs all start running. Aha! It just started up a worker on its own after I hit the web page for the first time today. I think the scheduler needs to be smarter about worker proceses... maybe it needs to check the database for any Delayed Jobs whose run_at is soon, and either not retire it or make sure a worker comes online in time to run it. The code is at http://github.com/alexch/sentry if you want to try it yourself. Has anyone else had this problem? -- Alex Chaffee - a...@cohuman.com - http://alexch.github.com Stalk me: http://friendfeed.com/alexch | http://twitter.com/alexch | http://alexch.tumblr.com -- You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en. -- You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
