On Fri, Aug 26, 2011 at 6:10 PM, Stephen Cremin <[email protected]> wrote: > I recently introuduced Resque to my application and I've started moving > background tasks into queues. In day-to-day use, I use an :after_save hook > to index articles in the background, but I also have a rake task: > === > desc "Indexing articles" > task :index_articles => :environment do > begin > Article.all.each { |a| Resque.enqueue(IndexArticle, a.id, a.main_text : > a.raw_main_text), a.class.name, a.publish_date.to_i) } > end > end > === > When I ran this, I expected to see a couple of thousand articles appear > almost instantly on the relevant queue on Resque, and be able to increase > the number of workers to speed through them. Instead, I never see more than > 5-10 at a time on the queue, and the rake task takes quite some time to run. > I have 20,000 subscribers on a mailing list that currently takes 2-3 hours > to send via SendGrid in a rake task. I was hoping that queuing these up on > Resque would allow me to send them in minutes rather than hours with enough > workers, but I'm guessing now that's not the case. > Would Appoxy Simpleworker let me fire off 20,000 email jobs in a queue, that > could be sent off in minutes rather than hours? Or would Resque handle this > and I just haven't configured it properly. Or is it something inherent in > rake that is slowing batch jobs down in general.
+1. We are using Pony mail with sendgrid and what we really need is pipelining when sending mass emails, which it doesn't support. With STARTTLS negotiation, if we could send multiple emails through the same connection (SMTP supports this) this will cut down the start up time for each email. Maybe a feature request for Pony mail? > Although, it's not essential that the email goes out in minutes, those hours > waiting for the emails to go out are incredibly stressful since the mailing > list will sometimes break. > Stephen K. --- http://blog.mudynamics.com http://blitz.io @pcapar -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
