On a weekly basis I load 5 - 600 jobs into a DelayedJob queue in seconds so I think something else is going on here. Maybe it's ActiveRecord taking a while to load a bunch of objects?
On Aug 26, 6:10 pm, Stephen Cremin <[email protected]> wrote: > I recently introuduced Resque to my application and I've started moving > background tasks into queues. In day-to-day use, I use an :after_save hook > to index articles in the background, but I also have a rake task: > > === > desc "Indexing articles" > > task :index_articles => :environment do > begin > Article.all.each { |a| Resque.enqueue(IndexArticle, a.id, a.main_text : > a.raw_main_text), a.class.name, a.publish_date.to_i) } > end > end > === > > When I ran this, I expected to see a couple of thousand articles appear > almost instantly on the relevant queue on Resque, and be able to increase > the number of workers to speed through them. Instead, I never see more than > 5-10 at a time on the queue, and the rake task takes quite some time to run. > > I have 20,000 subscribers on a mailing list that currently takes 2-3 hours > to send via SendGrid in a rake task. I was hoping that queuing these up on > Resque would allow me to send them in minutes rather than hours with enough > workers, but I'm guessing now that's not the case. > > Would Appoxy Simpleworker let me fire off 20,000 email jobs in a queue, that > could be sent off in minutes rather than hours? Or would Resque handle this > and I just haven't configured it properly. Or is it something inherent in > rake that is slowing batch jobs down in general. > > Although, it's not essential that the email goes out in minutes, those hours > waiting for the emails to go out are incredibly stressful since the mailing > list will sometimes break. > > Stephen > > PS:I noticed that SendGrid can now manage newsletter delivery for you, and I > may explore that option. -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
