I recently introuduced Resque to my application and I've started moving
background tasks into queues. In day-to-day use, I use an :after_save hook
to index articles in the background, but I also have a rake task:
===
desc "Indexing articles"
task :index_articles => :environment do
begin
Article.all.each { |a| Resque.enqueue(IndexArticle, a.id, a.main_text :
a.raw_main_text), a.class.name, a.publish_date.to_i) }
end
end
===
When I ran this, I expected to see a couple of thousand articles appear
almost instantly on the relevant queue on Resque, and be able to increase
the number of workers to speed through them. Instead, I never see more than
5-10 at a time on the queue, and the rake task takes quite some time to run.
I have 20,000 subscribers on a mailing list that currently takes 2-3 hours
to send via SendGrid in a rake task. I was hoping that queuing these up on
Resque would allow me to send them in minutes rather than hours with enough
workers, but I'm guessing now that's not the case.
Would Appoxy Simpleworker let me fire off 20,000 email jobs in a queue, that
could be sent off in minutes rather than hours? Or would Resque handle this
and I just haven't configured it properly. Or is it something inherent in
rake that is slowing batch jobs down in general.
Although, it's not essential that the email goes out in minutes, those hours
waiting for the emails to go out are incredibly stressful since the mailing
list will sometimes break.
Stephen
PS:I noticed that SendGrid can now manage newsletter delivery for you, and I
may explore that option.
--
You received this message because you are subscribed to the Google Groups
"Heroku" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/heroku?hl=en.
