Ops my mistake, our HR id is cmsevobg-hr. Normally we get initialization
time for ~10 seconds, now the time is 38 seconds using F4 CPU model.
Here are some exception traces that might help
1.
Failed startup of context
com.google.apphosting.utils.jetty.RuntimeAppEngineWebAppContext@199f8e6{/,/base/data/home/apps/s~cmsevobg-hr/production.357324755677494831}
java.lang.NullPointerException
at
com.google.inject.servlet.GuiceServletContextListener.contextInitialized(GuiceServletContextListener.java:46)
at
xxx.xxx.xxx.xxx.AppBootstrap.contextInitialized(AdmBootstrap.java:189)
at
org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:548)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:136)
at
org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250)
at
org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
at
org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:202)
at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:171)
at
com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:123)
at
com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:422)
at
com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:449)
at
com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:455)
at com.google.tracing.TraceContext.runInContext(TraceContext.java:695)
at
com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:333)
at
com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:325)
at
com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:453)
at
com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251)
at java.lang.Thread.run(Thread.java:679)
2. C2012-03-13 11:02:31.883
Uncaught exception from servlet
javax.servlet.UnavailableException: Initialization failed.
at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:211)
at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:171)
at
com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:123)
at
com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:422)
at
com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:449)
at
com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:455)
at com.google.tracing.TraceContext.runInContext(TraceContext.java:695)
at
com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:333)
at
com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:325)
at
com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:453)
at
com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251)
at java.lang.Thread.run(Thread.java:679)
On Tuesday, March 13, 2012 10:59:18 AM UTC+2, Miroslav Genov wrote:
>
> I'm encountering the same issue with HR app. The spike got started in
> about ~1 hour from now.
>
> AppID: cmsevobg
> Datastore: HR
>
> On Tuesday, March 13, 2012 10:36:38 AM UTC+2, Richard Watson wrote:
>>
>> In case you're keeping track of issues thinking it's generally cleared up:
>>
>> I'm on HR and have noticed higher latencies (couple seconds instead of
>> e.g. 300ms) lately and sometimes higher error rates (a few instead of 0-3).
>> Yesterday over about 6 hours I got a ton of 60-second requests that threw
>> 500's with accompanying messages [1], usually on memcache sets hitting a
>> deadline exceeded. Also, over the last couple weeks I've been running 3
>> instances permanently despite sometimes shutting them down manually.
>> Usually I get by on one just fine with bursts of 2 or 3. I've noticed
>> that one instance serves the majority of traffic with the other two serving
>> maybe 50 requests over many hours, so the shutdown isn't aggressive enough.
>>
>> "This request caused a new process to be started for your application..."
>> +
>> "A problem was encountered with the process that handled this request,
>> causing it to exit" in the same request.
>>
>> App-id: 2dumo-hr
>>
>
> On Tuesday, March 13, 2012 10:36:38 AM UTC+2, Richard Watson wrote:
>>
>> In case you're keeping track of issues thinking it's generally cleared up:
>>
>> I'm on HR and have noticed higher latencies (couple seconds instead of
>> e.g. 300ms) lately and sometimes higher error rates (a few instead of 0-3).
>> Yesterday over about 6 hours I got a ton of 60-second requests that threw
>> 500's with accompanying messages [1], usually on memcache sets hitting a
>> deadline exceeded. Also, over the last couple weeks I've been running 3
>> instances permanently despite sometimes shutting them down manually.
>> Usually I get by on one just fine with bursts of 2 or 3. I've noticed
>> that one instance serves the majority of traffic with the other two serving
>> maybe 50 requests over many hours, so the shutdown isn't aggressive enough.
>>
>> "This request caused a new process to be started for your application..."
>> +
>> "A problem was encountered with the process that handled this request,
>> causing it to exit" in the same request.
>>
>> App-id: 2dumo-hr
>>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/7jyamzUUvKYJ.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
