Hi John, Can you clarify "Mine are taking 10+ minutes"? Does that mean tasks sit in the queue for ten minutes before running?
The task queue can run tasks very fast. My tasks typically execute immediately or very close to it. However, I try hard to optimize my tasks for speed, specifically I target keeping them under 1,000ms. In my experience longer running tasks are not executed as quickly, so you can get more work done in a given task but fewer tasks will run. I was going to suggest distributing across multiple queues, but it sounds like you've already tried that. Also, you might turn the rate and concurrent exec limit up higher. Even with very fast tasks, I don't usually see anything near the rate I set. However, the concurrent execution limit is often a binding constraint -- so crank that up too. Make sure you give it at least a few minutes after increasing the settings, sometimes it takes a while to start spinning up. If some of your tasks are able to run quickly and some more slowly, I would definitely put the slow tasks in a 'special' queue and the fast tasks in a fast queue. That should at least let the fast stuff run better. Robert On Thu, Feb 24, 2011 at 10:29, John Wheeler <[email protected]> wrote: > You mock me with children's pictures!!!??? > > Just kidding. Thanks for that--it does help. > > Also, I am returning 200 successfully from each task. I changed the > queue settings to rate: 17/s, bucket: 32, and it doesn't make a > difference. I've tried several other rate/bucket combinations to round > out some edge cases. I noticed again what I said before that only 4 > tasks ever run at one time, so I tried creating multiple queues and > distributing the work between them, but no dice. If there's 5 queues, > i'll get 1 worker in 4 queues each running one task at a time, and the > other queue sits there doing nothing. Is there something I just don't > get about task execution? I was under the impression that tasks run > right away and app engine spins up all the resources to accommodate > your queue settings (at your expense of course). I know the docs > mention that the system does its best to schedule tasks and nothing is > guaranteed, but the delays I've experienced makes me think things are > really out of whack somewhere, because I see all these other posts > talking about it being faster. > > > > On Feb 24, 1:40 am, David Mora <[email protected]> wrote: >> Just checking >> >> have you checked if the task workers are not returning errors. >> >> It also seems like you are delaying the feeding a lot (100/m means >> that 100 items are going to be distributed to 50 buckets every minute >> - maybe thats your bottleneck?) >> >> here is a graph that ikai did that helped me a lot like 4 weeks a go >> >> http://twitpic.com/3y5814/full >> >> On 24 February 2011 03:14, John Wheeler <[email protected]> wrote: >> >> >> >> >> >> >> >> >> >> > I've seen posts from a few people talking about tasks taking anywhere >> > from 5-60 seconds to execute. Mine are taking 10+ minutes >> > consistently. My bucket size is 50, rate is 100/m, and I have maximum- >> > concurrent-requests set to 50. >> >> > Each one of my tasks takes about 4-5 mins to execute in itself, and I >> > have 50 lined up in 1 queue. On the admin screen, the queue appears to >> > process no more than 4 at one time. I'd ideally like it to process 50 >> > at a time, but maybe I don't understand something? >> >> > appid is hvs-lister >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "Google App Engine" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group >> > athttp://groups.google.com/group/google-appengine?hl=en. >> >> --http://about.me/david.mora > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
