Hi Brian, You need to do some experiments to figure out what will get you the best throughput. I can tell you that I've sustained some pretty high exec rates though, so it is possible. You need to figure out if you'll be able to get more work done using long tasks or lots of fast tasks. I have often found the second approach is ultimately faster, or lots of fast tasks combined with some longer tasks. You might also want to increase your rates and see if you get better performance.
Without knowing what you are doing, it is hard to say if map-reduce is a good fit or not. Robert On Fri, Mar 18, 2011 at 05:29, Brian Lim <[email protected]> wrote: > Thanks. Regardless of third-party blogs and newsgroups, I need a > solution to obtain high throughput. I will have to investigate > mapreduce. Will that also be a dead-end? > > Brian > > On Mar 18, 1:41 am, Robert Kluin <[email protected]> wrote: >> Hey Brian, >> There are a number of sources of this, and similar information, but >> if you look at the thread you cite: >> >> The volume of user-facing traffic (that is below the latency >> threshold) you serve determines how many appservers we provision for >> your application, which in turn affects the capacity available for >> running offline (task queue and cron) tasks. >> >> http://groups.google.com/group/google-appengine/msg/93922ef8458f859b >> >> I have also seen comments indicating that task-queues with lots of >> long-running requests are not given as many resources as fast task >> queues. I'm sure this was mentioned numerous times on IRC, I believe >> it has been brought up in the groups too. This also matches my >> personal observations. Here is someone else who apparently read / >> heard / was told the same thing (see Improved Background Processing): >> >> http://www.ianlewis.org/en/google-appengine-140-released >> >> Robert >> >> On Fri, Mar 18, 2011 at 00:35, Brian Lim <[email protected]> wrote: >> > The 1000ms threshold as described in this thread, seems to be >> > applicable to client facing URLs, not tasks, which have a 10 minute >> > limit. Google did not refute this when they had the opportunity. >> > Perhaps they might refute it now? >> >> >http://groups.google.com/group/google-appengine/browse_thread/thread/... >> >> > Does anybody know where I can read up on why this does not give me 72 >> > instances? Is this an undocumented bug? >> >> > On Mar 17, 12:18 pm, Robert Kluin <[email protected]> wrote: >> >> Hi Brian, >> >> From what I personally observe, if your tasks are long running you >> >> probably won't get a high throughput on that queue. You could try >> >> spreading the tasks across a couple queues, but it probably won't >> >> increase the overall throughput much (if any). If you can make the >> >> tasks run faster, un under 1000ms you'll get more instances spun-up >> >> and the queue's run rates will probably improve. >> >> >> At least that is the behavior I observe. >> >> >> Robert >> >> >> On Thu, Mar 17, 2011 at 07:35, Brian Lim <[email protected]> wrote: >> >> > I am using the default task queue with the following settings: >> >> >> > rate 10/s >> >> > bucket-size 10 >> >> > max-concurrent-requests 72 >> >> > task-age-limit 5 minutes (this is the retry age limit measured from >> >> > the first try) >> >> > min-backoff-seconds 10 >> >> > max-backoff-seconds 20 >> >> >> > The queue is used to invoke a 300 second loop (Java servlet) that >> >> > finished by writing a very small entity to the datastore. 72 >> >> > invocations are made. Billing is enabled. >> >> >> > Monitoring the console, and hitting reload, I never see more than 18 >> >> > instances running. What is the problem or why is this not working >> >> > right? >> >> >> > There are no log messages, no quota limits have been hit and at least >> >> > 12 hours of CPU time remain under the billing limit. Note 72 >> >> > invocations of 300 seconds is only 6 hours so there should be no >> >> > problem. >> >> >> > -- >> >> > You received this message because you are subscribed to the Google >> >> > Groups "Google App Engine" group. >> >> > To post to this group, send email to [email protected]. >> >> > To unsubscribe from this group, send email to >> >> > [email protected]. >> >> > For more options, visit this group >> >> > athttp://groups.google.com/group/google-appengine?hl=en. >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "Google App Engine" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group >> > athttp://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
