Hi Waleed, Because I've found tasks to be sometimes unreliable and unpredictable performance wise, I've implemented a strategy of engaging backend instance(s) as alternative non-task-based job processors.
Basically, I now have two job processing engines that work differently and together offer redundancy and better performance. One (the traditional one) uses tasks, and the other doesn't - and I run them simultaneously. If one isn't working well for some reason, then at least the other is getting work done. In my case reliability is more of an issue than achieving very high throughput, but this could also work to increase overall throughput. It may not work quite as well for python as it does for java where I can have multiple worker threads running on each backend. And it isn't necessarily cheap. But it can pick up some slack for the task infrastructure, and if the cost allows you could always add more backend instances. With threadsafe in java land, it is even better since the backend is more fully utilized with multiple worker threads working on jobs simultaneously. Basically the implementation is to have one or more backend worker threads/instances that loop eternally looking look for jobs in the DB, grabbing batches of them and running them directly (with a short configurable pause if none are found in a cycle). Meanwhile, the other task-based processor is being driven by a cron job, which every minute essentially runs the same "find me jobs" code the backends use. Instead of running the jobs directly in the frontend, this one dispatches them out as tasks using DB cursors and task chaining to handle larger numbers of waiting tasks. It can create some contention on "job" records in the DB, though there should be ways you could minimize that if it becomes excessive. And as long as you mark newly found jobs as "assigned" within the same transaction that you query them out, you can avoid accidentally running the same job twice. Careful if you're on high-replication - you may need to account for consistency limitations while querying and assigning jobs (I'm on M/S). /Tom On Jun 21, 8:36 pm, Waleed Abdulla <[email protected]> wrote: > Thank you, Nick. > I did check your post before asking and pending_ms was missing. Also, > I'm noticing "throttle_code=2" in the log of some tasks! It would be great > to cover that one as well. > > I've spent the last 3 days trying everything I can think of to make my > app scale up, but GAE is working against me. I make the tasks longer, and > GAE starts running them less often. I make them shorter, and my queues start > to have a backlog. GAE simply doesn't want to scale up for me (started 3 > days ago). I'm running out of ideas to try. Would appreciate your help. > > More details about what I tried here: > > http://groups.google.com/group/google-appengine/browse_thread/thread/... > > Regards, > Waleed > > On Tue, Jun 21, 2011 at 5:04 PM, Nick Johnson (Google) < > > [email protected]> wrote: > > Hi Waleed, > > > You may find this blog post useful for explaining the various fields of log > > entries: > >http://blog.notdot.net/2011/06/Demystifying-the-App-Engine-request-logs > > > pending_ms is the amount of time the request spent waiting on an app server > > to become available to execute the task. > > > -Nick Johnson > > > On Wed, Jun 22, 2011 at 8:45 AM, Waleed Abdulla <[email protected]> wrote: > > >> Good catch, thanks. I'm guessing that pending_ms is the time a request > >> stays on hold before execution? I couldn't find it documented anywhere. If > >> so, then the question is: why would a request be in a pending state? Is it > >> waiting for an instance to be free? > > >> This might also be related to my other thread about the task queue not > >> scaling up [1]. > > >> Thanks, > >> Waleed > > >> [1] > >>http://groups.google.com/group/google-appengine/browse_thread/thread/... > > >> On Tue, Jun 21, 2011 at 3:33 PM, Nicholas Verne <[email protected]>wrote: > > >>> One other note: > > >>> In a large number of cases, if you the pending_ms (when it exists) to > >>> your measurement, you come close to the ms value you circled at the > >>> top. > > >>> Nick Verne > > >>> On Wed, Jun 22, 2011 at 8:18 AM, Nicholas Verne <[email protected]> > >>> wrote: > >>> > Waleed, > > >>> > I'm going to have to ask one of the other infrastructure engineers > >>> > about the log lines. The value you have circled (18386ms) is not > >>> > always greater than the cpu_ms. I'm not sure what this value > >>> > indicates. > > >>> > Nick Verne > > >>> > On Wed, Jun 22, 2011 at 6:27 AM, Waleed Abdulla <[email protected]> > >>> wrote: > >>> >> Nicholas, > >>> >> I implemented the change you suggested (making task_t0) a global > >>> >> variable, but that didn't change the calculations. I'm attaching > >>> another > >>> >> screen shot. There is a big difference between the actual processing > >>> time > >>> >> and the time reported in the logs. It doesn't happen in all tasks, but > >>> it's > >>> >> often enough that it's slowing down my application considerably and > >>> causing > >>> >> a lot of user complaints. > >>> >> I also calculated the total time to load the module (including > >>> time for > >>> >> imports). It doesn't seem to take much time (sub 1 second), and it > >>> happens > >>> >> only once right after I deploy a new version. So that is not an issue. > >>> >> I've set the processing rate, bucket size, and max concurrency on > >>> the > >>> >> task queue to high numbers (much higher than I need). > >>> >> Any explanation for why this is happening? > >>> >> Waleed > > >>> >> On Mon, Jun 20, 2011 at 6:54 PM, Brandon Wirtz <[email protected]> > >>> wrote: > > >>> >>> Use a fetch from one thread to another. > > >>> >>> Echo Time > >>> >>> Fetch > >>> >>> Echo Time > >>> >>> Do something > >>> >>> Echo Time > >>> >>> Echo Fetch > > >>> >>> Now you have a timeline of everything except the latency from User to > >>> >>> Appengine. > > >>> >>> -----Original Message----- > >>> >>> From: [email protected] > >>> >>> [mailto:[email protected]] On Behalf Of Nicholas > >>> Verne > >>> >>> Sent: Monday, June 20, 2011 6:03 PM > >>> >>> To: [email protected] > >>> >>> Subject: Re: [google-appengine] Why a big difference between > >>> execution > >>> >>> time > >>> >>> and response time? > > >>> >>> Yes, that's true. If you try this: > > >>> >>> task_t0 = None > > >>> >>> ... > > >>> >>> def main(argv): > >>> >>> global task_t0 > >>> >>> task_t0 = time.time() > > >>> >>> You'll get a measurement per request as early as we can make it. For > >>> >>> warming > >>> >>> requests, it might still be worth measuring the import time > > >>> >>> import time > >>> >>> import_start = time.time() > >>> >>> ... other imports > >>> >>> import_duration = time.time() - import_start > > >>> >>> The import_duration global should persist and tell you how long this > >>> >>> instance's imports took to process. Subsequent requests on that > >>> instance > >>> >>> don't do the imports. > > >>> >>> Nick Verne > > >>> >>> On Tue, Jun 21, 2011 at 10:49 AM, Waleed Abdulla <[email protected]> > >>> wrote: > >>> >>> >> A more useful measurement might be to set task_t0 as a global > >>> >>> >> variable at the top of your file, with only the import of "time" > >>> >>> >> preceding it > > >>> >>> > Wouldn't that cause the 2nd, 3rd, ...nth requests to the same > >>> handler > >>> >>> > to show wrong values? I see why setting task_t0 at the top of the > >>> file > >>> >>> > would apply if it was the first request that causes the handler to > >>> be > >>> >>> > loaded. But this is happening on an ongoing basis for all requests, > >>> >>> > not only the first for that handler. > > >>> >>> > -- > >>> >>> > You received this message because you are subscribed to the Google > >>> >>> > Groups "Google App Engine" group. > >>> >>> > To post to this group, send email to > >>> [email protected]. > >>> >>> > To unsubscribe from this group, send email to > >>> >>> > [email protected]. > >>> >>> > For more options, visit this group at > >>> >>> >http://groups.google.com/group/google-appengine?hl=en. > > >>> >>> -- > >>> >>> You received this message because you are subscribed to the Google > >>> Groups > >>> >>> "Google App Engine" group. > >>> >>> To post to this group, send email to > >>> [email protected]. > >>> >>> To unsubscribe from this group, send email to > >>> >>> [email protected]. > >>> >>> For more options, visit this group at > >>> >>>http://groups.google.com/group/google-appengine?hl=en. > > >>> >>> -- > >>> >>> You received this message because you are subscribed to the Google > >>> Groups > >>> >>> "Google App Engine" group. > >>> >>> To post to this group, send email to > >>> [email protected]. > >>> >>> To unsubscribe from this group, send email to > >>> >>> [email protected]. > >>> >>> For more options, visit this group at > >>> >>>http://groups.google.com/group/google-appengine?hl=en. > > >>> >> -- > >>> >> You received this message because you are subscribed to the Google > >>> Groups > >>> >> "Google App Engine" group. > >>> >> To post to this group, send email to > >>> [email protected]. > >>> >> To unsubscribe from this group, send email to > >>> >> [email protected]. > >>> >> For more options, visit this group at > >>> >>http://groups.google.com/group/google-appengine?hl=en. > > >>> -- > >>> You received this message because you are subscribed to the Google Groups > >>> "Google App Engine" group. > >>> To post to this group, send email to [email protected]. > >>> To unsubscribe from this group, send email to > >>> [email protected]. > >>> For more options, visit this group at > >>>http://groups.google.com/group/google-appengine?hl=en. > > >> -- > >> You received this message because you are subscribed to the Google Groups > >> "Google App Engine" group. > >> To post to this group, send email to [email protected]. > >> To unsubscribe from this group, send email to > >> [email protected]. > >> For more options, visit this group at > >>http://groups.google.com/group/google-appengine?hl=en. > > > -- > > Nick Johnson, Developer Programs Engineer, App Engine > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
