Thanks. Regardless of third-party blogs and newsgroups, I need a solution to obtain high throughput. I will have to investigate mapreduce. Will that also be a dead-end?
Brian On Mar 18, 1:41 am, Robert Kluin <[email protected]> wrote: > Hey Brian, > There are a number of sources of this, and similar information, but > if you look at the thread you cite: > > The volume of user-facing traffic (that is below the latency > threshold) you serve determines how many appservers we provision for > your application, which in turn affects the capacity available for > running offline (task queue and cron) tasks. > > http://groups.google.com/group/google-appengine/msg/93922ef8458f859b > > I have also seen comments indicating that task-queues with lots of > long-running requests are not given as many resources as fast task > queues. I'm sure this was mentioned numerous times on IRC, I believe > it has been brought up in the groups too. This also matches my > personal observations. Here is someone else who apparently read / > heard / was told the same thing (see Improved Background Processing): > > http://www.ianlewis.org/en/google-appengine-140-released > > Robert > > On Fri, Mar 18, 2011 at 00:35, Brian Lim <[email protected]> wrote: > > The 1000ms threshold as described in this thread, seems to be > > applicable to client facing URLs, not tasks, which have a 10 minute > > limit. Google did not refute this when they had the opportunity. > > Perhaps they might refute it now? > > >http://groups.google.com/group/google-appengine/browse_thread/thread/... > > > Does anybody know where I can read up on why this does not give me 72 > > instances? Is this an undocumented bug? > > > On Mar 17, 12:18 pm, Robert Kluin <[email protected]> wrote: > >> Hi Brian, > >> From what I personally observe, if your tasks are long running you > >> probably won't get a high throughput on that queue. You could try > >> spreading the tasks across a couple queues, but it probably won't > >> increase the overall throughput much (if any). If you can make the > >> tasks run faster, un under 1000ms you'll get more instances spun-up > >> and the queue's run rates will probably improve. > > >> At least that is the behavior I observe. > > >> Robert > > >> On Thu, Mar 17, 2011 at 07:35, Brian Lim <[email protected]> wrote: > >> > I am using the default task queue with the following settings: > > >> > rate 10/s > >> > bucket-size 10 > >> > max-concurrent-requests 72 > >> > task-age-limit 5 minutes (this is the retry age limit measured from > >> > the first try) > >> > min-backoff-seconds 10 > >> > max-backoff-seconds 20 > > >> > The queue is used to invoke a 300 second loop (Java servlet) that > >> > finished by writing a very small entity to the datastore. 72 > >> > invocations are made. Billing is enabled. > > >> > Monitoring the console, and hitting reload, I never see more than 18 > >> > instances running. What is the problem or why is this not working > >> > right? > > >> > There are no log messages, no quota limits have been hit and at least > >> > 12 hours of CPU time remain under the billing limit. Note 72 > >> > invocations of 300 seconds is only 6 hours so there should be no > >> > problem. > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups "Google App Engine" group. > >> > To post to this group, send email to [email protected]. > >> > To unsubscribe from this group, send email to > >> > [email protected]. > >> > For more options, visit this group > >> > athttp://groups.google.com/group/google-appengine?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
