We're still seeing some on one of our apps 'vendastaweb', but the
problem seems to have gone away on our other apps.
vendastaweb uses django 1.0 (the Google-provided lib) and
appengine_django helper project (r90). An example stack trace follows,
though the stack traces are always different (depending on when the
30s limit is hit):
09-27 01:14AM 52.788
<class 'google.appengine.runtime.DeadlineExceededError'>:
Traceback (most recent call last):
File "/base/data/home/apps/vendastaweb/trunk-8081.335730273874884367/
main.py", line 111, in <module>
main()
File "/base/data/home/apps/vendastaweb/trunk-8081.335730273874884367/
main.py", line 77, in real_main
util.run_wsgi_app(application)
File "/base/python_lib/versions/1/google/appengine/ext/webapp/
util.py", line 76, in run_wsgi_app
result = application(env, _start_response)
File "/base/python_lib/versions/third_party/django-1.0/django/core/
handlers/wsgi.py", line 228, in __call__
self.load_middleware()
File "/base/python_lib/versions/third_party/django-1.0/django/core/
handlers/base.py", line 38, in load_middleware
mod = __import__(mw_module, {}, {}, [''])
File "/base/python_lib/versions/third_party/django-1.0/django/
middleware/http.py", line 1, in <module>
from django.utils.http import http_date
On Sep 25, 2:36 pm, "Jeff S (Google)" <[email protected]> wrote:
> Hi all,
>
> Thank you for all of the detailed reports. We've made some changes
> which should have improved the deadline exception situation,
> especially when related to a large number of imports when spinning up
> a new instance of an app. Since the fix has gone out we've been
> closely monitoring these apps but if you are still seeing issues
> please post here with your app ID, imports, and stack traces. Thanks
> again and apologies for the latency induced errors.
>
> Happy coding,
>
> Jeff
>
> On Sep 23, 2:10 pm, Jason C <[email protected]> wrote:
>
>
>
> > We are using Django 1.0.2 via zipimport and the most recent
> > appengine_django (r90), and are still seeing the
> > DeadlineExceededErrors.
>
> > Taro, please repost here again and update us in a day or two.
>
> > j
>
> > On Sep 23, 2:59 am, Taro Morimoto <[email protected]> wrote:
>
> > > Hi!
>
> > > I had the same problem with very high latencies and
> > > DeadlineExceededErrors. I don't know if you have the same problem as I
> > > did but I hope this helps for some of you.
>
> > > I'm using django 1.0.2 with zip import. So I just updated my google-
> > > app-engine-django helper library and the latencies are now gone. :)
>
> > > Best regards,
> > > Taro
>
> > > On Sep 19, 2:17 am, Michael <[email protected]> wrote:
>
> > > > I'm having the same problem (app: sovpad)
>
> > > > Except from my logs can be found
> > > > here:http://groups.google.com/group/google-appengine-python/browse_thread/...
>
> > > > On Sep 19, 1:05 am, bvelasquez <[email protected]> wrote:
>
> > > > > I'm experiencing this also (app: jacob-6). 2 Sept. is when I started
> > > > > noticing the issue. Mostly in the morning (PST). Later in the day,
> > > > > the app is very fast, and responses come back after 200ms. In the
> > > > > morning, these same requests can take up 20000ms.
>
> > > > > Barry
>
> > > > > On Sep 18, 9:28 am, johnP <[email protected]> wrote:
>
> > > > > > Dozens of people have been starring the following
> > > > > > item:http://code.google.com/p/googleappengine/issues/detail?id=1695
>
> > > > > > Something happened after the 2 Sept deployment.
>
> > > > > > On Sep 18, 7:42 am, Mahmoud <[email protected]> wrote:
>
> > > > > > > Actually, our app (spont.appspot.com) has been seeing elevated
> > > > > > > latency
> > > > > > > as well in the last two weeks. The code has not changed at all in
> > > > > > > those two weeks.
>
> > > > > > > -M
>
> > > > > > > On Sep 18, 4:54 am, "Nick Johnson (Google)"
> > > > > > > <[email protected]>
> > > > > > > wrote:
>
> > > > > > > > Hi Brandon,
> > > > > > > > What is your App ID?
>
> > > > > > > > On Fri, Sep 18, 2009 at 8:46 AM, Brandon Thomson
> > > > > > > > <[email protected]> wrote:
>
> > > > > > > > > I get bursty periods of very high request latency, with
> > > > > > > > > increasing
> > > > > > > > > frequency over the last few weeks, which last from seconds to
> > > > > > > > > hours.
> > > > > > > > > Usually the elevated latencies are in the 3-6s range but
> > > > > > > > > sometimes it
> > > > > > > > > goes up into the 20-30s range. For example, here are some
> > > > > > > > > logs from a
> > > > > > > > > few minutes ago:
>
> > > > > > > > > 09-18 12:23AM 46.511 /checkStatus 500 2906ms 105cpu_ms
> > > > > > > > > 8api_cpu_ms 0kb Pingdom.com_bot_version_1.4_(http://
> > > > > > > > >www.pingdom.com/),gzip(gfe),gzip(gfe)
> > > > > > > > > 09-18 12:23AM 46.511 /checkStatus 500 2906ms 105cpu_ms
> > > > > > > > > 8api_cpu_ms 0kb Pingdom.com_bot_version_1.4_(http://
> > > > > > > > >www.pingdom.com/),gzip(gfe),gzip(gfe)
> > > > > > > > > 09-18 12:23AM 46.360 /stats 200 2771ms 105cpu_ms
> > > > > > > > > 8api_cpu_ms 5kb
> > > > > > > > > w3m/0.5.2,gzip(gfe),gzip(gfe)
> > > > > > > > > 09-18 12:23AM 46.118 /admin/tick 200 2706ms 113cpu_ms
> > > > > > > > > 16api_cpu_ms 0kb Python-urllib/2.5,gzip(gfe),gzip(gfe)
> > > > > > > > > 09-18 12:23AM 43.878 /admin/tick 200 3825ms 113cpu_ms
> > > > > > > > > 16api_cpu_ms 0kb Python-urllib/2.5,gzip(gfe),gzip(gfe)
> > > > > > > > > 09-18 12:23AM 41.639 /admin/tick 200 4983ms 113cpu_ms
> > > > > > > > > 16api_cpu_ms 0kb Python-urllib/2.5,gzip(gfe),gzip(gfe)
> > > > > > > > > 09-18 12:23AM 41.603 /stats 200 4602ms 105cpu_ms
> > > > > > > > > 8api_cpu_ms 5kb
> > > > > > > > > w3m/0.5.2,gzip(gfe),gzip(gfe)
> > > > > > > > > 09-18 12:23AM 41.603 /stats 200 4602ms 105cpu_ms
> > > > > > > > > 8api_cpu_ms 5kb
> > > > > > > > > w3m/0.5.2,gzip(gfe),gzip(gfe)
> > > > > > > > > 09-18 12:23AM 39.401 /admin/tick 200 4826ms 113cpu_ms
> > > > > > > > > 16api_cpu_ms 0kb Python-urllib/2.5,gzip(gfe),gzip(gfe)
> > > > > > > > > 09-18 12:23AM 39.401 /admin/tick 200 4826ms 113cpu_ms
> > > > > > > > > 16api_cpu_ms 0kb Python-urllib/2.5,gzip(gfe),gzip(gfe)
> > > > > > > > > 09-18 12:23AM 37.158 /admin/tick 200 4800ms 113cpu_ms
> > > > > > > > > 16api_cpu_ms 0kb Python-urllib/2.5,gzip(gfe),gzip(gfe)
>
> > > > > > > > > Pingdom is pinging from all over the globe and sees the
> > > > > > > > > problem so it
> > > > > > > > > is definitely on the App Engine side.
>
> > > > > > > > Can you elaborate on this?
>
> > > > > > > > > Drilling down into my logs: Even when the requests are taking
> > > > > > > > > 6s+ to
> > > > > > > > > return, from my first logging statement at the very top of my
> > > > > > > > > python
> > > > > > > > > file to the last logging statement at the end of my request
> > > > > > > > > handler is
> > > > > > > > > only 500ms of real time. So, I don't know where exactly the
> > > > > > > > > issue is,
> > > > > > > > > but it doesn't seem to be inside my code.
>
> > > > > > > > What imports, other than 'logging' are you doing before your
> > > > > > > > first logging
> > > > > > > > statement?
>
> > > > > > > > > These requests all have < 200ms of latency most of the time
> > > > > > > > > (although
> > > > > > > > > the bad behavior has gotten increasingly frequent lately).
>
> > > > > > > > > I thought this might be related to the problems the Django
> > > > > > > > > folks are
> > > > > > > > > having with DeadlineExceeded errors... See
> > > > > > > > >http://code.google.com/p/googleappengine/issues/detail?id=1695and
> > > > > > > > > many other discussion threads linked therein.
>
> > > > > > > > --
> > > > > > > > Nick Johnson, Developer Programs Engineer, App Engine
> > > > > > > > Google Ireland Ltd. :: Registered in Dublin, Ireland,
> > > > > > > > Registration Number:
> > > > > > > > 368047
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~----------~----~----~----~------~----~------~--~---
