AppId ID: geognos version:5 (default) - or any version if that matters language:python QPS : < 0.1 live instances at error time: 0 or 1-3 live instances
I have posted already about my application's Errors 500 and time out problems since "Short service outage on October 27." see : http://groups.google.com/group/google-appengine-python/browse_thread/thread/1ee7d07d55ab543c http://groups.google.com/group/google-appengine/browse_thread/thread/fda9cd3dfec6f6e6/12f225a9bc36d593?lnk=raot http://groups.google.com/group/google-appengine/msg/9567a52f187e31da but I got no answers. I am desperate because around 30% of Requests result in Error 500 and/ or time outs while for the rest the latency has quadrupled to unacceptable levels, making the application useless. I was pretty sure it has nothing to do with my appl, since I changed nothing on my code. Now I can prove it : Take a look at the error trace and logs below where you can see it returning Error 500 /or time outs while on a Datastore Admin requests which have nothing to do with my code. Also notice that the appl was idling at the time, no other requests except mine trying to access Data store Admin page. Also it is not something intermittent it repeats it self almost all the time. >From this and from other stack traces and debugging i have done I believe the problem has nothing to do with any App Engine service since it happens on handlers that are not using the datastore or mcashe. Most - if not all - of the errors and time outs happen while application tries to access the file system for a read (module imports or plain file reads - no huge files involved just some Kbytes) Usually happens on cold starts but I have seen it also happening on hot instances. So please somebody take a look at this. T R A C E : ____________________________________________ Last record searched: 10-26 03:32PM 43.618. 10-29 08:40AM 48.122 /_ah/datastore_admin/static/img/help.gif 500 31892ms 180cpu_ms 1kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/ 534.7,gzip(gfe) 195.97.105.127 - nickmilon [29/Oct/2010:08:41:20 -0700] "GET /_ah/ datastore_admin/static/img/help.gif HTTP/1.1" 500 1268 "http:// 5.latest.geognos.appspot.com/_ah/datastore_admin/? app_id=geognos&kind=AppSetingsDic&kind=NM_DScashe&kind=_AE_MR_MapreduceControl&kind=_AE_MR_MapreduceState&kind=_AE_MR_ShardState&kind=city&kind=cntr&kind=cntrAux&kind=countries&kind=country_list&kind=ipv4&namespace=" "Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe)" "5.latest.geognos.appspot.com" ms=31893 cpu_ms=180 api_cpu_ms=0 cpm_usd=0.005299 E 10-29 08:41AM 20.008 <class 'google.appengine.runtime.DeadlineExceededError'>: Traceback (most recent call last): File "/base/python_runtime/python_lib/versions/1/google/appengine/ ext/datastore_admin/main.py", line 204, in <module> main() File "/base/python_runtime/python_lib/versions/1/google/appengine/ ext/datastore_admin/main.py", line 200, in main util.run_wsgi_app(APP) File "/base/python_runtime/python_lib/versions/1/google/appengine/ ext/webapp/util.py", line 97, in run_wsgi_app run_bare_wsgi_app(add_wsgi_middleware(application)) File "/base/python_runtime/python_lib/versions/1/google/appengine/ ext/webapp/util.py", line 115, in run_bare_wsgi_app result = application(env, _start_response) File "/base/python_runtime/python_lib/versions/1/google/appengine/ ext/webapp/__init__.py", line 511, in __call__ handler.get(*groups) File "/base/python_runtime/python_lib/versions/1/google/appengine/ ext/datastore_admin/main.py", line 171, in get self.response.out.write(open(path).read()) 10-29 08:30AM 58.405 /_ah/datastore_admin/? app_id=geognos&kind=AppSetingsDic&kind=NM_DScashe&kind=_AE_MR_MapreduceControl&kind=_AE_MR_MapreduceState&kind=_AE_MR_ShardState&kind=city&kind=cntr&kind=cntrAux&kind=countries&kind=country_list&kind=ipv4&namespace= 500 36561ms 220cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/ 534.7,gzip(gfe) 195.97.105.127 - nickmilon [29/Oct/2010:08:31:34 -0700] "GET /_ah/ datastore_admin/? app_id=geognos&kind=AppSetingsDic&kind=NM_DScashe&kind=_AE_MR_MapreduceControl&kind=_AE_MR_MapreduceState&kind=_AE_MR_ShardState&kind=city&kind=cntr&kind=cntrAux&kind=countries&kind=country_list&kind=ipv4&namespace= HTTP/1.1" 500 850 - "Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/ 534.7,gzip(gfe)" "5.latest.geognos.appspot.com" ms=36561 cpu_ms=220 api_cpu_ms=0 cpm_usd=0.006364 E 10-29 08:31AM 34.802 <class 'google.appengine.runtime.DeadlineExceededError'>: Traceback (most recent call last): File "/base/python_runtime/python_lib/versions/1/google/appengine/ ext/datastore_admin/main.py", line 30, in <module> from google.appengine.ext.datastore_admin import delete_handler File "/base/python_runtime/python_lib/versions/1/google/appengine/ ext/datastore_admin/delete_handler.py", line 29, in <module> from google.appengine.ext.mapreduce import control File "/base/python_runtime/python_lib/versions/1/google/appengine/ ext/mapreduce/__init__.py", line 1, in <module> #!/usr/bin/env python 10-29 07:56AM 00.302 /_ah/datastore_admin/? app_id=geognos&kind=AppSetingsDic&kind=NM_DScashe&kind=_AE_MR_MapreduceControl&kind=_AE_MR_MapreduceState&kind=_AE_MR_ShardState&kind=city&kind=cntr&kind=cntrAux&kind=countries&kind=country_list&kind=ipv4&namespace= 500 29736ms 180cpu_ms 1kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/ 534.7,gzip(gfe) 195.97.105.127 - nickmilon [29/Oct/2010:07:56:30 -0700] "GET /_ah/ datastore_admin/? app_id=geognos&kind=AppSetingsDic&kind=NM_DScashe&kind=_AE_MR_MapreduceControl&kind=_AE_MR_MapreduceState&kind=_AE_MR_ShardState&kind=city&kind=cntr&kind=cntrAux&kind=countries&kind=country_list&kind=ipv4&namespace= HTTP/1.1" 500 1471 - "Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/ 534.7,gzip(gfe)" "5.latest.geognos.appspot.com" ms=29737 cpu_ms=180 api_cpu_ms=0 cpm_usd=0.005325 E 10-29 07:56AM 30.035 <class 'google.appengine.runtime.DeadlineExceededError'>: Traceback (most recent call last): File "/base/python_runtime/python_lib/versions/1/google/appengine/ ext/datastore_admin/main.py", line 30, in <module> from google.appengine.ext.datastore_admin import delete_handler File "/base/python_runtime/python_lib/versions/1/google/appengine/ ext/datastore_admin/delete_handler.py", line 29, in <module> from google.appengine.ext.mapreduce import control File "/base/python_runtime/python_lib/versions/1/google/appengine/ ext/mapreduce/control.py", line 26, in <module> from google.appengine.ext.mapreduce import handlers File "/base/python_runtime/python_lib/versions/1/google/appengine/ ext/mapreduce/handlers.py", line 34, in <module> from google.appengine.ext.mapreduce import model File "/base/python_runtime/python_lib/versions/1/google/appengine/ ext/mapreduce/model.py", line 44, in <module> from google.appengine.ext.mapreduce import util File "/base/python_runtime/python_lib/versions/1/google/appengine/ ext/mapreduce/util.py", line 18, in <module> """Utility functions for use with the mapreduce library.""" __________________________________________________________________________________________________ EXAMPLES FROM L O G S 10-29 09:33AM 18.326 /_ah/datastore_admin/static/img/help.gif 200 64ms 0cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 09:33AM 16.600 /_ah/datastore_admin/static/js/compiled.js 200 1359ms 19cpu_ms 5kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/ 534.7,gzip(gfe) 10-29 09:33AM 16.502 /_ah/datastore_admin/static/css/compiled.css 200 1391ms 291cpu_ms 9kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/ 534.7,gzip(gfe) 10-29 09:33AM 09.952 /_ah/datastore_admin/? app_id=geognos&kind=AppSetingsDic&kind=NM_DScashe&kind=_AE_MR_MapreduceControl&kind=_AE_MR_MapreduceState&kind=_AE_MR_ShardState&kind=city&kind=cntr&kind=cntrAux&kind=countries&kind=country_list&kind=ipv4&namespace= 200 6252ms 279cpu_ms 199api_cpu_ms 1kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 09:10AM 24.983 /_ah/datastore_admin/static/img/help.gif 200 13ms 0cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 09:10AM 21.001 /_ah/datastore_admin/static/js/compiled.js 200 3679ms 20cpu_ms 5kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/ 534.7,gzip(gfe) 10-29 09:10AM 20.976 /_ah/datastore_admin/static/css/compiled.css 200 12ms 0cpu_ms 9kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/ 534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 09:10AM 20.613 /_ah/datastore_admin/? app_id=geognos&kind=AppSetingsDic&kind=NM_DScashe&kind=_AE_MR_MapreduceControl&kind=_AE_MR_MapreduceState&kind=_AE_MR_ShardState&kind=city&kind=cntr&kind=cntrAux&kind=countries&kind=country_list&kind=ipv4&namespace= 200 71ms 259cpu_ms 199api_cpu_ms 1kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/ 534.7,gzip(gfe) 10-29 09:09AM 49.572 /_ah/datastore_admin/static/img/help.gif 200 6159ms 0cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/ 534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 09:09AM 38.293 /_ah/datastore_admin/static/css/compiled.css 200 10897ms 20cpu_ms 9kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/ 534.7,gzip(gfe) 10-29 09:09AM 38.420 /_ah/datastore_admin/static/js/compiled.js 500 10349ms 0cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/ 534.7,gzip(gfe) 10-29 09:09AM 35.818 /_ah/datastore_admin/? app_id=geognos&kind=AppSetingsDic&kind=NM_DScashe&kind=_AE_MR_MapreduceControl&kind=_AE_MR_MapreduceState&kind=_AE_MR_ShardState&kind=city&kind=cntr&kind=cntrAux&kind=countries&kind=country_list&kind=ipv4&namespace= 200 2227ms 318cpu_ms 199api_cpu_ms 1kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 08:55AM 17.898 /_ah/datastore_admin/static/img/help.gif 200 3703ms 0cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/ 534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 08:55AM 07.836 /_ah/datastore_admin/static/js/compiled.js 200 9790ms 20cpu_ms 5kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/ 534.7,gzip(gfe) 10-29 08:55AM 07.718 /_ah/datastore_admin/static/css/compiled.css 200 4720ms 0cpu_ms 9kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/ 534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 08:55AM 01.629 /_ah/datastore_admin/? app_id=geognos&kind=AppSetingsDic&kind=NM_DScashe&kind=_AE_MR_MapreduceControl&kind=_AE_MR_MapreduceState&kind=_AE_MR_ShardState&kind=city&kind=cntr&kind=cntrAux&kind=countries&kind=country_list&kind=ipv4&namespace= 200 5857ms 418cpu_ms 199api_cpu_ms 1kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 08:48AM 08.314 /_ah/datastore_admin/static/img/help.gif 200 10ms 0cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 08:48AM 08.026 /_ah/datastore_admin/static/js/compiled.js 200 13ms 20cpu_ms 5kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/ 534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 08:48AM 08.003 /_ah/datastore_admin/static/css/compiled.css 200 12ms 0cpu_ms 9kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/ 534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 08:48AM 06.663 /_ah/datastore_admin/? app_id=geognos&kind=AppSetingsDic&kind=NM_DScashe&kind=_AE_MR_MapreduceControl&kind=_AE_MR_MapreduceState&kind=_AE_MR_ShardState&kind=city&kind=cntr&kind=cntrAux&kind=countries&kind=country_list&kind=ipv4&namespace= 200 1076ms 279cpu_ms 199api_cpu_ms 1kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 08:46AM 59.158 /_ah/datastore_admin/static/img/help.gif 200 414ms 20cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/ 534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 08:46AM 56.802 /_ah/datastore_admin/static/js/compiled.js 200 1878ms 160cpu_ms 5kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/ 534.7,gzip(gfe) 10-29 08:46AM 56.712 /_ah/datastore_admin/static/css/compiled.css 200 1961ms 140cpu_ms 9kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/ 534.7,gzip(gfe) 10-29 08:46AM 53.037 /_ah/datastore_admin/? app_id=geognos&kind=AppSetingsDic&kind=NM_DScashe&kind=_AE_MR_MapreduceControl&kind=_AE_MR_MapreduceState&kind=_AE_MR_ShardState&kind=city&kind=cntr&kind=cntrAux&kind=countries&kind=country_list&kind=ipv4&namespace= 200 3429ms 568cpu_ms 199api_cpu_ms 1kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 08:42AM 09.065 /_ah/datastore_admin/static/img/help.gif 200 10ms 0cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 08:41AM 57.384 /_ah/datastore_admin/static/css/compiled.css 200 11282ms 199cpu_ms 9kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/ 534.7,gzip(gfe) 10-29 08:41AM 57.363 /_ah/datastore_admin/static/js/compiled.js 200 11215ms 0cpu_ms 5kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/ 534.7,gzip(gfe) 10-29 08:41AM 49.673 /_ah/datastore_admin/? app_id=geognos&kind=AppSetingsDic&kind=NM_DScashe&kind=_AE_MR_MapreduceControl&kind=_AE_MR_MapreduceState&kind=_AE_MR_ShardState&kind=city&kind=cntr&kind=cntrAux&kind=countries&kind=country_list&kind=ipv4&namespace= 200 7420ms 298cpu_ms 199api_cpu_ms 1kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 08:41AM 46.929 /_ah/datastore_admin/static/img/help.gif 200 48ms 19cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 08:41AM 46.411 /_ah/datastore_admin/static/js/compiled.js 200 17ms 0cpu_ms 5kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/ 534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 08:41AM 46.379 /_ah/datastore_admin/static/css/compiled.css 200 28ms 19cpu_ms 9kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/ 534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 08:41AM 46.008 /_ah/datastore_admin/? app_id=geognos&kind=AppSetingsDic&kind=NM_DScashe&kind=_AE_MR_MapreduceControl&kind=_AE_MR_MapreduceState&kind=_AE_MR_ShardState&kind=city&kind=cntr&kind=cntrAux&kind=countries&kind=country_list&kind=ipv4&namespace= 200 127ms 257cpu_ms 199api_cpu_ms 1kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/ 534.7,gzip(gfe) 10-29 08:40AM 48.122 /_ah/datastore_admin/static/img/help.gif 500 31892ms 180cpu_ms 1kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/ 534.7,gzip(gfe) 10-29 08:40AM 47.705 /_ah/datastore_admin/static/js/compiled.js 200 56ms 19cpu_ms 5kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/ 534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 08:40AM 47.533 /_ah/datastore_admin/static/css/compiled.css 200 54ms 19cpu_ms 9kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/ 534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 08:40AM 45.089 /_ah/datastore_admin/? app_id=geognos&kind=AppSetingsDic&kind=NM_DScashe&kind=_AE_MR_MapreduceControl&kind=_AE_MR_MapreduceState&kind=_AE_MR_ShardState&kind=city&kind=cntr&kind=cntrAux&kind=countries&kind=country_list&kind=ipv4&namespace= 200 2109ms 587cpu_ms 199api_cpu_ms 1kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7,gzip(gfe) 10-29 08:38AM 09.683 /_ah/datastore_admin/? app_id=geognos&kind=AppSetingsDic&kind=NM_DScashe&kind=_AE_MR_MapreduceControl&kind=_AE_MR_MapreduceState&kind=_AE_MR_ShardState&kind=city&kind=cntr&kind=cntrAux&kind=countries&kind=country_list&kind=ipv4&namespace= 500 45686ms 0cpu_ms 0kb Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/ 534.7,gzip(gfe) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
