Yes and no. There should be more than enough capacity for us to execute task queue tasks with a reasonable latency. With delays of an hour, it's far more likely that the tasks are failing for whatever reason and being retried. There are many reasons a task may fail: deadline exceeded error, an exception is thrown etc, basically, anything that would cause a non 200 HTTP response code in the handler. One thing you can do to help us track this is to log the X-AppEngine-TaskRetryCount header - knowing how many times a task has been retried would help us identify whether it's a problem with global task queue scheduling or some other issue that may be causing execution to fail.
-- Ikai Lan Developer Programs Engineer, Google App Engine Blogger: http://googleappengine.blogspot.com Reddit: http://www.reddit.com/r/appengine Twitter: http://twitter.com/app_engine On Tue, Sep 28, 2010 at 11:28 AM, Iron Mountain Foundry < [email protected]> wrote: > I'm seeing delays of over an hour today as well (Sep 28, 2010). Are > you saying that the global task queue affects our individual queues? > That makes sense, but can we see how much the task execution is > impacted for everyone? I'd like to see an estimated delay before my > task(s) can be executed. > Brent Washburne > > > On Sep 17, 9:37 am, "Ikai Lan (Google)" > <[email protected]<ikai.l%[email protected]> > > > wrote: > > One possible explanation is that task execution was impacted and all > tasks > > were backed up, so tasks that needed to execute before your task all > needed > > to be drained first. > > > > > > > > On Fri, Sep 17, 2010 at 12:29 PM, Eli Jones <[email protected]> wrote: > > > Ikai, > > > > > I had that full hour and a half gap. > > > > > I have a chained task that runs like this (in fact, there is nothing > else > > > running in the day for my app except this single background task): > > > > > 1. After the task's primary work is done, it calculates a delay like > so: > > > > > now = datetime.today() > > > seconds = 60*(now.minute) + now.second > > > delay = 300 - seconds%300 > > > > > 2. It adds a task to the queue with a delay = the delay calculated > above. > > > > > The expected result is that the task should run at the next 5 minute > mark. > > > > > A task successfully ran at 10:30 AM PST, and presumably added a task > with a > > > delay that would have the next one start at 10:35 AM PST. > > > > > Instead, nothing at all happens in my logs or on my dashboard or > anywhere > > > until 11:53 AM PST when the task runs successfully and resumes the once > > > every 5 minutes chained process. > > > > > I know that the 11:53 AM task that ran is the expected 10:35 AM task > since > > > they use numeric task names and it is the next task in the sequence. > > > > > So, there are two probable explanations: > > > > > 1. The task was sitting in the queue and was scheduled to run at 10:35 > AM > > > PST.. but was not executed until 11:53 AM PST for unknown reasons. > > > 2. Somehow the delay ended up being equal to about 4800 seconds. > > > > > Pretty much, option 2 is impossible. Even if datetime.now() returned > some > > > wildly wrong value.. seconds%300 will always return some value less > than > > > 300. thus the delay is going to be between 300 and 1. > > > > > So, the task should always start within 300 seconds (or 5 minutes) if > the > > > delay setting works for task.add(). > > > > > Even though I did not view the Task Queue during this delay, I feel > fairly > > > confident that.. there was a task sitting in my default queue (and, in > fact, > > > was the only task scheduled for my entire app during that time) for > over > > > 4,800 seconds that had a delay setting that was less than 300 seconds. > > > > > On Fri, Sep 17, 2010 at 11:45 AM, Ikai Lan (Google) < > > > [email protected] <ikai.l%[email protected]> < > ikai.l%[email protected] <ikai.l%[email protected]>>> wrote: > > > > >> It looks like there was a period of time when tasks were not > executing. It > > >> shouldn't have been an hour and a half, but I'll double check. > > > > >> On Fri, Sep 17, 2010 at 11:18 AM, Tom Wu <[email protected]> > wrote: > > > > >>> +1, me too. > > > > >>> 2010/9/17 Francois Masurel <[email protected]> > > > > >>> Yep, I can confirm this tasks and cron gap shows in my dashboard. > > > > >>>> On Sep 16, 11:50 pm, Eli Jones <[email protected]> wrote: > > >>>> > I second this report. > > > > >>>> > I have an app that has a chained task that executes every 5 > minutes. > > >>>> Each > > >>>> > time the task completes it enqueues the next task to be executed > (the > > >>>> next > > >>>> > task is always scheduled for the next 5 minute mark). > > > > >>>> > From looking at the logs.. the task ran fine at 10:30 AM PST.. > then > > >>>> just did > > >>>> > nothing (no errors no nothing) until 11:53 AM PST when it just > started > > >>>> > running again. > > > > >>>> > So, there was pretty much an hour and a half gap.. with nothing > > >>>> happening. > > > > >>>> > This chained task has been running fine for the past 9 months with > no > > >>>> > mysterious gaps like this. > > > > >>>> > On Thu, Sep 16, 2010 at 2:57 PM, Chris Vaughn <[email protected]> > > >>>> wrote: > > >>>> > > According to the app engine system stats page everything is > running > > >>>> > > fine but I'm seeing an issue that I can't figure out. > > > > >>>> > > I have an app that relies heavily on task queues. Every minute > or > > >>>> so > > >>>> > > a payload comes in and is broken into 1 or many (depending on > the > > >>>> size > > >>>> > > of the payload) schedule tasks and added to the > "schedule-queue". > > >>>> For > > >>>> > > the last 1h 15m nothing in my schedule-queue has been executed. > I > > >>>> see > > >>>> > > no errors in the logs and it has stayed at 0 run in the last > minute. > > >>>> > > Here's a screenshot from my admin page of that one queue. > > >>>> > >http://skitch.com/chris.vaughn/dabtu/task-queues-appspot.com > > > > >>>> > > I've found another user on Twitter reporting similar behavior at > the > > >>>> > > same time. > > >>>> > >http://twitter.com/bjschnei/status/24685776803 > > > > >>>> > > Chris > > > > >>>> > > -- > > >>>> > > You received this message because you are subscribed to the > Google > > >>>> Groups > > >>>> > > "Google App Engine" group. > > >>>> > > To post to this group, send email to > > >>>> [email protected]. > > >>>> > > To unsubscribe from this group, send email to > > >>>> > > [email protected]<google-appengine%[email protected]><google-appengine%2Bunsubscrib > [email protected]><google-appengine%2Bunsubscrib > > >>>> [email protected]> > > >>>> > > . > > >>>> > > For more options, visit this group at > > >>>> > >http://groups.google.com/group/google-appengine?hl=en. > > > > >>>> -- > > >>>> You received this message because you are subscribed to the Google > > >>>> Groups "Google App Engine" group. > > >>>> To post to this group, send email to > [email protected]. > > >>>> To unsubscribe from this group, send email to > > >>>> [email protected]<google-appengine%[email protected]><google-appengine%2Bunsubscrib > [email protected]> > > >>>> . > > >>>> For more options, visit this group at > > >>>>http://groups.google.com/group/google-appengine?hl=en. > > > > >>> -- > > >>> You received this message because you are subscribed to the Google > Groups > > >>> "Google App Engine" group. > > >>> To post to this group, send email to > [email protected]. > > >>> To unsubscribe from this group, send email to > > >>> [email protected]<google-appengine%[email protected]><google-appengine%2Bunsubscrib > [email protected]> > > >>> . > > >>> For more options, visit this group at > > >>>http://groups.google.com/group/google-appengine?hl=en. > > > > >> -- > > >> You received this message because you are subscribed to the Google > Groups > > >> "Google App Engine" group. > > >> To post to this group, send email to > [email protected]. > > >> To unsubscribe from this group, send email to > > >> [email protected]<google-appengine%[email protected]><google-appengine%2Bunsubscrib > [email protected]> > > >> . > > >> For more options, visit this group at > > >>http://groups.google.com/group/google-appengine?hl=en. > > > > > -- > > > You received this message because you are subscribed to the Google > Groups > > > "Google App Engine" group. > > > To post to this group, send email to [email protected] > . > > > To unsubscribe from this group, send email to > > > [email protected]<google-appengine%[email protected]><google-appengine%2Bunsubscrib > [email protected]> > > > . > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
