On a related note, You should also look into using the asynchronous urlfetch api. This would allow you to send multiple requests at the same time to the rest service you are using to make push notifications.
This fits in very well with the new taskqueue pull queue api. You could still use a task per notification to send to a user, but when you are sending them (in your 'reduce' phase) you could lease multiple tasks at once and send them all at the same time. On 12 May 2011 20:20, Greg Darke (Google) <[email protected]> wrote: > By 'at a time' I mean per call to taskqueue.Queue.add, not per http > request. You can call that function twice in a row with two different > batches of tasks. > > For example, in the python api you can do this: > queue = taskqueue.Queue('default') > tasks = [taskqueue.Task(params={'id': i}) for i in xrange(100)] > queue.add(tasks) > > If you have more than 100 tasks that you want to insert, you will need > to split them into groups of 100 tasks, and call queue.add for each > group. > > For example: > queue = taskqueue.Queue('default') > tasks = [taskqueue.Task(params={'id': i}) for i in xrange(1000)] > > def _yield_in_groups(l, group_size): > for start_index in xrange(0, len(l), group_size): > yield l[start_index: start_index + group_size] > > for task_group in _yield_in_groups(tasks, 100): > queue.add(task_group) > > > > On 12 May 2011 19:11, Jeff Schnitzer <[email protected]> wrote: >> Would you please clarify that? When you say "at a time", do you mean >> "within the scope of a single HTTP request"? The docs say that 100 >> tasks is "the maximum number of tasks that can be added in a batch" >> but they don't specify what "batch" means. >> >> If this is indeed the limit, is there any talk of eliminating it for >> long-running Backend requests? It seems to seriously limit the >> utility of Backends for long-running data processing that involves >> task creation - sure, I no longer need to break my work down into 30s >> (now 10m) increments, but now I need to respawn a new task anyways >> every time I hit 100 tasks. >> >> Here's the specific issue I have right now: >> >> I have a button in one of my apps that sends push notifications to a >> few million iphone app users. A "map" task iterates through the >> users, creating a new "reduce" task that sends the relevant data for a >> single push to a REST service wired up to the APNS. The map task has >> to restart every 100 reduce tasks it creates. >> >> For technical reasons related to filtering specific users to push to, >> it would be a lot easier if the "map" task didn't have to restart. >> I'd like to fire up a Backend to run through the entire set in one go. >> But this would require millions of tasks to be created in the context >> of that long-running request. >> >> Is this just not an option? >> >> Thanks, >> Jeff >> >> On Thu, May 12, 2011 at 5:01 PM, Greg Darke (Google) >> <[email protected]> wrote: >>> Yes. >>> >>> The taskqueue.add function may only add 100 tasks at a time. >>> >>> On 12 May 2011 15:35, Jeff Schnitzer <[email protected]> wrote: >>>> With appengine frontends, each request is limited to 100 inserts into >>>> the task queue. >>>> >>>> Do long-running backend requests have the same limitation? >>>> >>>> Jeff >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Google App Engine" group. >>>> To post to this group, send email to [email protected]. >>>> To unsubscribe from this group, send email to >>>> [email protected]. >>>> For more options, visit this group at >>>> http://groups.google.com/group/google-appengine?hl=en. >>>> >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
