I've decided to process 50 emails per task. I've looked over the logs and I'm guessing that's far more than enough limit to for the task to finish in 10 minutes.The task itself checks if there's anything left (at the very top) and queues another task. It seems to be working nicely. Not sure if that's the proper way of doing parallel processing, but I'm happy with the solution.
On Sun, Jun 26, 2011 at 11:33 PM, Nischal Shetty <[email protected]>wrote: > Also remember that in a batch you can add a maximum of 100 tasks. > > > On 26 June 2011 17:00, Branko Vukelic <[email protected]> wrote: > >> Ok, thanks for all the answers. I'll do it that way. >> >> On Sun, Jun 26, 2011 at 10:16 PM, Nischal Shetty >> <[email protected]> wrote: >> > I would suggest you do it in parallel. Create n tasks with x emails in >> each >> > of them. It would be faster that way! >> > >> > >> > On 26 June 2011 15:52, Branko Vukelic <[email protected]> wrote: >> >> >> >> On Sun, Jun 26, 2011 at 9:41 PM, nischalshetty >> >> <[email protected]> wrote: >> >> > If you're using the appengine mail api be sure you want that or >> >> > something >> >> > along the lines of Amazons mail service because I remember reading a >> >> > thread >> >> > where one of the appengine devs said the email service is not meant >> for >> >> > large amount of emails (clients would block your email and many >> others >> >> > might >> >> > take it to be spam) >> >> >> >> Yes I know about this. We will use SendGrid after Beta release, but >> >> for Alpha, we plan to use the GAE mail API simply to keep things >> >> rolling. >> >> >> >> > How about querying the datastore and making use of a cursor. Pass the >> >> > cursor >> >> > value to the new tasks. Will that work for you? >> >> >> >> The mailing lists are generated dynamically. So once the list is >> >> compiled from the data in the datastore, I would not like to store it, >> >> and then repeatedly read it from the datastore each time a task is >> >> executed. What I do is: >> >> >> >> mailing_list = generate_list(from=some_data) >> >> task.add(url='/url/to/task', >> >> params={'mailing_list': mailing_list.join(',')}) >> >> >> >> and in the task handler (when I want to do a batch of 50 emails in one >> >> task): >> >> >> >> mailing_list = params.POST.get('mailing_list').split(',') >> >> to_process = mailing_list[50:] >> >> the_rest = mailing_list[:50] >> >> >> >> or in case I want to go one by one: >> >> >> >> to_process = mailing_list.pop(0) >> >> the_rest = mailing_list >> >> >> >> enqueue the_rest, and process to_process. I haven't tested this yet, >> >> but I'm wondering if it's better to go one-by-one or many-by-many. >> >> >> >> -- >> >> Branko Vukelić >> >> [email protected] >> >> >> >> Lead Developer >> >> Herd Hound (tm) - Travel that doesn't bite >> >> www.herdhound.com >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> Groups >> >> "Google App Engine" group. >> >> To post to this group, send email to [email protected] >> . >> >> To unsubscribe from this group, send email to >> >> [email protected]. >> >> For more options, visit this group at >> >> http://groups.google.com/group/google-appengine?hl=en. >> >> >> > >> > >> > >> > -- >> > -Nischal >> > twitter: NischalShetty >> > facebook: Nischal >> > >> > >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups >> > "Google App Engine" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group at >> > http://groups.google.com/group/google-appengine?hl=en. >> > >> >> >> >> -- >> Branko Vukelić >> [email protected] >> >> Lead Developer >> Herd Hound (tm) - Travel that doesn't bite >> www.herdhound.com >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> > > > -- > -Nischal > twitter: NischalShetty <http://twitter.com/nischalshetty> > facebook: Nischal <http://facebook.com/nischal> > > <http://www.justunfollow.com> <http://www.buffr.com> > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- Branko Vukelić [email protected] Lead Developer Herd Hound (tm) - Travel that doesn't bite www.herdhound.com -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
