I've noticed over the past two days that my application has been
getting a lot of deadline exceeded errors during warmup. This wasn't
happening the previous week, and I pay for "always on". Google really
needs to respond to this and tell us that they know about the problem,
know how to fix it, and are going to fix it. I've noticed other posts
about this same problem, so I'm not alone. People come to the site and
the page shows a Google server error, because of the timeout.
Stack trace example:
2011-05-25 15:37:01.368 /_ah/warmup 500 525386ms 1633cpu_ms 0kb
0.1.0.3 - - [25/May/2011:15:37:01 -0700] "GET /_ah/warmup HTTP/
1.1" 500 0 - - "0-2.350650477797125837.MY_APP.appspot.com" ms=525386
cpu_ms=1633 api_cpu_ms=0 cpm_usd=0.045389 loading_request=1
exit_code=104 throttle_code=1
W 2011-05-25 15:31:08.987
EXCEPTION
com.google.apphosting.api.DeadlineExceededException: This request
(5baa7c00da50fc7b) started at 2011/05/25 22:28:31.276 UTC and was
still executing at 2011/05/25 22:31:08.808 UTC.
at
com.google.appengine.runtime.Request.process-5baa7c00da50fc7b(Request.java)
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:231)
at sun.misc.Resource.getBytes(Resource.java:124)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:273)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
43)
at java.lang.reflect.Method.invoke(Method.java:616)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
at org.mortbay.util.Loader.loadClass(Loader.java:91)
at org.mortbay.util.Loader.loadClass(Loader.java:71)
at org.mortbay.jetty.servlet.Holder.doStart(Holder.java:73)
at
org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:
242)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
at
org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:
685)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:
140)
at
org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:
1250)
at
org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:
517)
at
org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
at
com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:439)
at com.google.net.rpc.impl.Server
$RpcTask.runInContext(Server.java:573)
at com.google.tracing.TraceContext$TraceContextRunnable
$1.run(TraceContext.java:448)
at com.google.tracing.TraceContext.runInContext(TraceContext.java:
688)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:
326)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:
318)
at com.google.tracing.TraceContext
$TraceContextRunnable.run(TraceContext.java:446)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
E 2011-05-25 15:31:08.994
javax.servlet.ServletContext log: unavailable
javax.servlet.UnavailableException: This request
(5baa7c00da50fc7b) started at 2011/05/25 22:28:31.276 UTC and was
still executing at 2011/05/25 22:31:08.808 UTC.
at org.mortbay.jetty.servlet.Holder.doStart(Holder.java:79)
at
org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:
242)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
at
org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:
685)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:
140)
at
org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:
1250)
at
org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:
517)
at
org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:
191)
at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:
168)
at
com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:
123)
at
com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java:
260)
at com.google.apphosting.base.RuntimePb$EvaluationRuntime
$2.handleRequest(RuntimePb.java:9669)
at
com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:439)
at com.google.net.rpc.impl.Server
$RpcTask.runInContext(Server.java:573)
at com.google.tracing.TraceContext$TraceContextRunnable
$1.run(TraceContext.java:448)
at com.google.tracing.TraceContext.runInContext(TraceContext.java:
688)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:
326)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:
318)
at com.google.tracing.TraceContext
$TraceContextRunnable.run(TraceContext.java:446)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
W 2011-05-25 15:37:01.256
Failed startup of context
com.google.apphosting.utils.jetty.RuntimeAppEngineWebAppContext@70c38c{/,/
base/data/home/apps/MY_APP/0-2.350650477797125837}
org.mortbay.util.MultiException[java.lang.NullPointerException,
com.google.apphosting.runtime.HardDeadlineExceededError: This request
(5baa7c00da50fc7b) started at 2011/05/25 22:28:31.276 UTC and was
still executing at 2011/05/25 22:31:12.625 UTC.]
at
org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:
656)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:
140)
at
org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:
1250)
at
org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:
517)
at
org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:
191)
at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:
168)
at
com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:
123)
at
com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java:
260)
at com.google.apphosting.base.RuntimePb$EvaluationRuntime
$2.handleRequest(RuntimePb.java:9669)
at
com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:439)
at com.google.net.rpc.impl.Server
$RpcTask.runInContext(Server.java:573)
at com.google.tracing.TraceContext$TraceContextRunnable
$1.run(TraceContext.java:448)
at com.google.tracing.TraceContext.runInContext(TraceContext.java:
688)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:
326)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:
318)
at com.google.tracing.TraceContext
$TraceContextRunnable.run(TraceContext.java:446)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
java.lang.NullPointerException
at java.lang.Class.isAssignableFrom(Native Method)
at
org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:
256)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
at
org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:
685)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:
140)
at
org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:
1250)
at
org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:
517)
at
org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:
191)
at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:
168)
at
com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:
123)
at
com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java:
260)
at com.google.apphosting.base.RuntimePb$EvaluationRuntime
$2.handleRequest(RuntimePb.java:9669)
at
com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:439)
at com.google.net.rpc.impl.Server
$RpcTask.runInContext(Server.java:573)
at com.google.tracing.TraceContext$TraceContextRunnable
$1.run(TraceContext.java:448)
at com.google.tracing.TraceContext.runInContext(TraceContext.java:
688)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:
326)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:
318)
at com.google.tracing.TraceContext
$TraceContextRunnable.run(TraceContext.java:446)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
com.google.apphosting.runtime.HardDeadlineExceededError: This
request (5baa7c00da50fc7b) started at 2011/05/25 22:28:31.276 UTC and
was still executing at 2011/05/25 22:31:12.625 UTC.
at
com.google.appengine.runtime.Request.process-5baa7c00da50fc7b(Request.java)
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:231)
at sun.misc.Resource.getBytes(Resource.java:124)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:273)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
43)
at java.lang.reflect.Method.invoke(Method.java:616)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
at org.mortbay.util.Loader.loadClass(Loader.java:91)
at org.mortbay.util.Loader.loadClass(Loader.java:71)
at org.mortbay.jetty.servlet.Holder.doStart(Holder.java:73)
at
org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:
242)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
at
org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:
685)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:
140)
at
org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:
1250)
at
org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:
517)
at
org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
at
com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:439)
at com.google.net.rpc.impl.Server
$RpcTask.runInContext(Server.java:573)
at com.google.tracing.TraceContext$TraceContextRunnable
$1.run(TraceContext.java:448)
at com.google.tracing.TraceContext.runInContext(TraceContext.java:
688)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:
326)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:
318)
at com.google.tracing.TraceContext
$TraceContextRunnable.run(TraceContext.java:446)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
C 2011-05-25 15:37:01.266
Uncaught exception from servlet
javax.servlet.UnavailableException: Initialization failed.
at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:
200)
at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:
168)
at
com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:
123)
at
com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java:
260)
at com.google.apphosting.base.RuntimePb$EvaluationRuntime
$2.handleRequest(RuntimePb.java:9669)
at
com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:439)
at com.google.net.rpc.impl.Server
$RpcTask.runInContext(Server.java:573)
at com.google.tracing.TraceContext$TraceContextRunnable
$1.run(TraceContext.java:448)
at com.google.tracing.TraceContext.runInContext(TraceContext.java:
688)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:
326)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:
318)
at com.google.tracing.TraceContext
$TraceContextRunnable.run(TraceContext.java:446)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
I 2011-05-25 15:37:01.334
This request caused a new process to be started for your
application, and thus caused your application code to be loaded for
the first time. This request may thus take longer and use more CPU
than a typical request for your application.
W 2011-05-25 15:37:01.334
A serious problem was encountered with the process that handled
this request, causing it to exit. This is likely to cause a new
process to be used for the next request to your application. If you
see this message frequently, you may be throwing exceptions during the
initialization of your application. (Error code 104)
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
