My name's Nick, and I work for a company called Bytemark - we use backgroundrb in a range of internal projects (all our internal apps are Ruby on Rails, so it makes sense).
Basically, and as I mentioned on IRC, I've been tasked with making backgroundrb "better" over the next month or so, and I'd like to push as much upstream as I can while I'm at it (although some stuff I can guarantee you won't want - I'll my making my own local branch use a login plugin with a different syntax, for instance). First on the agenda is this backtrace: /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:147:in `log': Mysql::Error: MySQL server has gone away: SELECT * FROM `bdrb_job_queues` WHERE ( worker_name = 'dhshell_export_worker' AND taken = 0 AND scheduled_at <= '2009-04-24 10:41:44' ) LIMIT 1 FOR UPDATE (ActiveRecord::StatementInvalid) from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/connection_adapters/mysql_adapter.rb:302:in `execute' from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/connection_adapters/mysql_adapter.rb:537:in `select' from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:7:in `select_all_without_query_cache' from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/query_cache.rb:61:in `select_all' from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/base.rb:586:in `find_by_sql' from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/base.rb:1345:in `find_every' from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/base.rb:1307:in `find_initial' from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/base.rb:538:in `find' from /home/bmrack/bmrack/releases/20090404011130/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_job_queue.rb:11:in `find_next' from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:66:in `transaction' from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/transactions.rb:79:in `transaction' from /home/bmrack/bmrack/releases/20090404011130/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_job_queue.rb:8:in `find_next' from /home/bmrack/bmrack/releases/20090404011130/vendor/plugins/backgroundrb/server/lib/meta_worker.rb:271:in `check_for_enqueued_tasks' from /home/bmrack/bmrack/releases/20090404011130/vendor/plugins/backgroundrb/server/lib/meta_worker.rb:133:in `worker_init' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_periodic_event.rb:23:in `call' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_periodic_event.rb:23:in `run' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_core.rb:301:in `check_for_timer_events' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_core.rb:301:in `each' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_core.rb:301:in `check_for_timer_events' from /home/bmrack/bmrack/releases/20090404011130/vendor/plugins/backgroundrb/server/lib/meta_worker.rb:296:in `check_for_timer_events' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_core.rb:140:in `start_reactor' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_core.rb:139:in `loop' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_core.rb:139:in `start_reactor' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_worker.rb:20:in `start_worker' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/packet_worker_runner:33:in `load_worker' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/packet_worker_runner:26:in `initialize' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/packet_worker_runner:47:in `new' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/packet_worker_runner:47 from /usr/bin/packet_worker_runner:19:in `load' from /usr/bin/packet_worker_runner:19 (not latest git, but definitely git - we're just updating to packet 0.1.15, but I doubt that'll affect this particular error) - this kills the server from time to time for us (we have our MySQL server set to go away after half a day) - it's not the full story, since for that to happen, worker requests must have stopped too (or something else), but fixing this is number 1 on my list. I've not delved much into the source yet (5pm on a Friday is *not* the time to start with that!); I was wondering if you guys have a preferred / global strategy for dealing with errors. My approach to the above would be to catch the error, and respond by attempting to re-establish the connection. If that succeeds, I'd re-do the scheduled item (so the database going away would be transparent to the client); if not... hmm. Do we have a method of reporting a failed task back to the worker? Another problem I was seeing that led me to stop using the backgroundrb cache object was that requests into a worker via the MiddleMan API would just freeze up from time to time, leaving the Rails controller method hanging on for it. That's another one for me to investigate (we weren't using memcached, mind, but it annoys me when the Hash - a far simpler system - is less reliable). I'm also tasked with redeploying the current backgroundrb setup - right now, we have one server per application using it (which is about 4,5 backgroundrb servers in total), with the backgroundrb table stored in the application's database. I'm kind of moving towards a scheme where we have a pair of backgroundrb servers (transparently load-balanced) used by all applications. Each backgroundrb server would have its own separate *SQLite* database. Requests would go to one or another of the backgroundrb servers; if one of the servers died, all the requests would go to the other server, and vice-versa. If possible (I know this is a -devel list), I'd like to get comments on whether that's a good setup or not, and suggestions for improvement. If not, well, just consider it to be a bit of context ;) Regards, -- Nick Thomas Bytemark Hosting Limited _______________________________________________ Backgroundrb-devel mailing list [email protected] http://rubyforge.org/mailman/listinfo/backgroundrb-devel
