Hi Stephen, Using Resque and Sendgrid (running locally / not on heroku) with 4 resque workers, we are able to completely mail out 30K emails within an hour or two.
It took practically no time to enqueue. I may be remembering this incorrectly, as luckily we don't send out slews of emails every night. Article.all.each loads everything into memory. You may want to try find_each / find_in_batches or a pure sql route. (There are ways to do this in rails 2 and rails 3) Good Luck, Keenan On Sunday, August 28, 2011 at 5:01 AM, Stephen Cremin wrote: > I created another rake task that pushed jobs onto Resque, and indeed it did > push thousands of jobs onto the queue and let me ramp through them by > increasing the number of workers. Perhaps I was hit by the recent worker bug > outlined on Heroku's status page. If nobody else has direct experience with > Sendgrid on Resque vs Appoxy, I'll do my own experiment and report back what > kind of throughput I can get. > > Stephen > > > On 28 August 2011 08:50, Gabriel <[email protected] > (mailto:[email protected])> wrote: > > On a weekly basis I load 5 - 600 jobs into a DelayedJob queue in > > seconds so I think something else is going on here. Maybe it's > > ActiveRecord taking a while to load a bunch of objects? > > > > On Aug 26, 6:10 pm, Stephen Cremin <[email protected] > > (mailto:[email protected])> wrote: > > > I recently introuduced Resque to my application and I've started moving > > > background tasks into queues. In day-to-day use, I use an :after_save hook > > > to index articles in the background, but I also have a rake task: > > > > > > === > > > desc "Indexing articles" > > > > > > task :index_articles => :environment do > > > begin > > > Article.all.each { |a| Resque.enqueue(IndexArticle, a.id (http://a.id), > > > a.main_text : > > > a.raw_main_text), a.class.name (http://a.class.name), > > > a.publish_date.to_i) } > > > end > > > end > > > === > > > > > > When I ran this, I expected to see a couple of thousand articles appear > > > almost instantly on the relevant queue on Resque, and be able to increase > > > the number of workers to speed through them. Instead, I never see more > > > than > > > 5-10 at a time on the queue, and the rake task takes quite some time to > > > run. > > > > > > I have 20,000 subscribers on a mailing list that currently takes 2-3 hours > > > to send via SendGrid in a rake task. I was hoping that queuing these up on > > > Resque would allow me to send them in minutes rather than hours with > > > enough > > > workers, but I'm guessing now that's not the case. > > > > > > Would Appoxy Simpleworker let me fire off 20,000 email jobs in a queue, > > > that > > > could be sent off in minutes rather than hours? Or would Resque handle > > > this > > > and I just haven't configured it properly. Or is it something inherent in > > > rake that is slowing batch jobs down in general. > > > > > > Although, it's not essential that the email goes out in minutes, those > > > hours > > > waiting for the emails to go out are incredibly stressful since the > > > mailing > > > list will sometimes break. > > > > > > Stephen > > > > > > PS:I noticed that SendGrid can now manage newsletter delivery for you, > > > and I > > > may explore that option. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Heroku" group. > > To post to this group, send email to [email protected] > > (mailto:[email protected]). > > To unsubscribe from this group, send email to > > [email protected] > > (mailto:heroku%[email protected]). > > For more options, visit this group at > > http://groups.google.com/group/heroku?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to [email protected] > (mailto:[email protected]). > To unsubscribe from this group, send email to > [email protected] > (mailto:[email protected]). > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
