I created another rake task that pushed jobs onto Resque, and indeed it did push thousands of jobs onto the queue and let me ramp through them by increasing the number of workers. Perhaps I was hit by the recent worker bug outlined on Heroku's status page. If nobody else has direct experience with Sendgrid on Resque vs Appoxy, I'll do my own experiment and report back what kind of throughput I can get.
Stephen On 28 August 2011 08:50, Gabriel <[email protected]> wrote: > On a weekly basis I load 5 - 600 jobs into a DelayedJob queue in > seconds so I think something else is going on here. Maybe it's > ActiveRecord taking a while to load a bunch of objects? > > On Aug 26, 6:10 pm, Stephen Cremin <[email protected]> wrote: > > I recently introuduced Resque to my application and I've started moving > > background tasks into queues. In day-to-day use, I use an :after_save > hook > > to index articles in the background, but I also have a rake task: > > > > === > > desc "Indexing articles" > > > > task :index_articles => :environment do > > begin > > Article.all.each { |a| Resque.enqueue(IndexArticle, a.id, > a.main_text : > > a.raw_main_text), a.class.name, a.publish_date.to_i) } > > end > > end > > === > > > > When I ran this, I expected to see a couple of thousand articles appear > > almost instantly on the relevant queue on Resque, and be able to increase > > the number of workers to speed through them. Instead, I never see more > than > > 5-10 at a time on the queue, and the rake task takes quite some time to > run. > > > > I have 20,000 subscribers on a mailing list that currently takes 2-3 > hours > > to send via SendGrid in a rake task. I was hoping that queuing these up > on > > Resque would allow me to send them in minutes rather than hours with > enough > > workers, but I'm guessing now that's not the case. > > > > Would Appoxy Simpleworker let me fire off 20,000 email jobs in a queue, > that > > could be sent off in minutes rather than hours? Or would Resque handle > this > > and I just haven't configured it properly. Or is it something inherent in > > rake that is slowing batch jobs down in general. > > > > Although, it's not essential that the email goes out in minutes, those > hours > > waiting for the emails to go out are incredibly stressful since the > mailing > > list will sometimes break. > > > > Stephen > > > > PS:I noticed that SendGrid can now manage newsletter delivery for you, > and I > > may explore that option. > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
