Hi, thanks for the updates, it looked similar for me yestarday, fortunately it is OK now, or maybe I should say - for now.
It is a little fishy for me, there obviously was a problem, but none of the system status tools did show that something bad is happening... The other thing is that I cannot understand one thing, and you seem to have the same issue: You have 3 resident instances, why new dynamic ones are created? Why don't the resident instances take the requests? Looking at the table attached by you: Sum of requests - dynamic instances: 47 Sum of requests - resident instances: 15 It's the same for me, even if I add 5 resident instances, only one of them is serving the requests, 4 are idle and new dynamic instances are created to serve those requests that should go to additional resident ones. This is odd and makes reserving instances pointless. Marcin W dniu piątek, 8 lutego 2013 14:39:34 UTC+1 użytkownik John V Denley napisał: > > Sorry for all the updates, hopefully someone will find it useful though!! > > Latency is through the roof! - what is causing this?! can I fix it?... > > Total number of instancesAverage QPS*Average Latency*Average Memory7 > total (3 Resident)0.00233671.0 ms68.9 MBytes*Instances *[image: > help]<https://developers.google.com/appengine/docs/adminconsole/instances> > QPS*Latency*RequestsErrorsAgeMemoryLogsAvailabilityShutdown0.0000.0 ms30 > 0:04:2067.0 MBytesView > Logs<https://appengine.google.com/logs?app_id=idebanet&version_id=6-20.362344675424379244&severity_level_override=1&severity_level=3&filter=instance%3D00c61b117c8b437629ba0681cf452275258982&filter_type=regex&date_type=now&limit=20&view=Search>[image: > > Resident Icon]Resident0.0000.0 ms1200:04:2067.8 MBytesView > Logs<https://appengine.google.com/logs?app_id=idebanet&version_id=6-20.362344675424379244&severity_level_override=1&severity_level=3&filter=instance%3D00c61b117c3a9330bdcf5e1074ecba490b5935&filter_type=regex&date_type=now&limit=20&view=Search>[image: > > Resident Icon]Resident0.01733671.0 ms100:01:2852.0 MBytesView > Logs<https://appengine.google.com/logs?app_id=idebanet&version_id=6-20.362344675424379244&severity_level_override=1&severity_level=3&filter=instance%3D00c61b117c157a0c8723723a486140193eb8&filter_type=regex&date_type=now&limit=20&view=Search>[image: > > Resident Icon]Resident0.0000.0 ms200:02:2355.3 MBytesView > Logs<https://appengine.google.com/logs?app_id=idebanet&version_id=6-20.362344675424379244&severity_level_override=1&severity_level=3&filter=instance%3D00c61b117c9bb59efbad64506f2b0498baa5&filter_type=regex&date_type=now&limit=20&view=Search>[image: > > Dynamic Icon]Dynamic0.0000.0 ms1400:04:1068.5 MBytesView > Logs<https://appengine.google.com/logs?app_id=idebanet&version_id=6-20.362344675424379244&severity_level_override=1&severity_level=3&filter=instance%3D00c61b117cf91ad57f164b7c2e5d22806846b1&filter_type=regex&date_type=now&limit=20&view=Search>[image: > > Dynamic Icon]Dynamic0.0000.0 ms200:02:0167.4 MBytesView > Logs<https://appengine.google.com/logs?app_id=idebanet&version_id=6-20.362344675424379244&severity_level_override=1&severity_level=3&filter=instance%3D00c61b117cc3d4aff4f08dc57b18c11ecc0f34bf&filter_type=regex&date_type=now&limit=20&view=Search>[image: > > Dynamic Icon]Dynamic0.0000.0 ms2900:07:19104.1 MBytesView > Logs<https://appengine.google.com/logs?app_id=idebanet&version_id=6-20.362344675424379244&severity_level_override=1&severity_level=3&filter=instance%3D00c61b117cb72247c66e582eded2d6293f502db2&filter_type=regex&date_type=now&limit=20&view=Search>[image: > > Dynamic Icon]Dynamic > On Friday, 8 February 2013 13:27:21 UTC, John V Denley wrote: >> >> I just adjusted my instance setting so the number of idle instances was >> 3-24 and the pending latency was 0-500ms >> >> and suddenly everything seemed to kick into action... >> >> I changed my settings back to auto-auto and auto-auto its not working >> again.... >> >> more tests... >> >> On Friday, 8 February 2013 13:09:32 UTC, John V Denley wrote: >>> >>> Also Just noticed this which REALLY doesn't seem right!! >>> >>> Total number of instancesAverage QPS*Average Latency*Average Memory0 >>> totalUnknownUnknown msUnknown MBytes >>> On Friday, 8 February 2013 01:04:45 UTC, Marcin wrote: >>>> >>>> Hi, >>>> >>>> I am experiencing extremely long response times for a couple of hours >>>> now. I am using GAE/J. >>>> >>>> I can provide app id via email. >>>> >>>> My app has billing enabled and is using SSL VIP. >>>> >>>> This affects both requests with API calls and static resources like >>>> images: >>>> >>>> Static image: >>>> >>>> ms=2655 cpu_ms=0 >>>> >>>> Controller with API calls displaying dashboard: >>>> >>>> ms=8430 cpu_ms=131 >>>> >>>> >>>> For a list of activities 131 cpu_ms seems like a good result, but why >>>> does the whole request last for 8.5secs? >>>> >>>> Other odd things that I have noticed: >>>> - My resident / reserved instance is missing - I have only dynamic >>>> ones, there was always a permanent instance with green shield icon. >>>> - After disabling and reenabling the application I always get >>>> /_ah/warmup 500 87610ms 0kb - 90 seconds to start an instance? I'm not >>>> using Spring, it usually took about 7 secs to spin a new one. >>>> >>>> com.google.apphosting.api.DeadlineExceededException: This request >>>> (18f5b04d255c672c) started at 2013/02/08 00:28:56.487 UTC and was still >>>> executing at 2013/02/08 00:29:56.229 UTC. >>>> at java.util.zip.ZipFile.read(Native Method) >>>> at java.util.zip.ZipFile.access$1200(ZipFile.java:57) >>>> at java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:476) >>>> at java.util.zip.ZipFile$1.fill(ZipFile.java:259) >>>> at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158) >>>> at java.io.DataInputStream.readFully(DataInputStream.java:195) >>>> at java.util.jar.JarFile.hasClassPathAttribute(JarFile.java:482) >>>> at >>>> java.util.jar.JavaUtilJarAccessImpl.jarFileHasClassPathAttribute(JavaUtilJarAccessImpl.java:33) >>>> at sun.misc.URLClassPath$JarLoader.getClassPath(URLClassPath.java:1176) >>>> at sun.misc.URLClassPath.getLoader(URLClassPath.java:374) >>>> at sun.misc.URLClassPath.findResource(URLClassPath.java:201) >>>> at java.net.URLClassLoader$2.run(URLClassLoader.java:379) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at java.net.URLClassLoader.findResource(URLClassLoader.java:376) >>>> at >>>> com.google.apphosting.runtime.security.UserClassLoader.findResource(UserClassLoader.java:723) >>>> at java.lang.ClassLoader.getResource(ClassLoader.java:977) >>>> at org.mortbay.resource.Resource.newSystemResource(Resource.java:203) >>>> at >>>> org.mortbay.jetty.webapp.WebXmlConfiguration.configureDefaults(WebXmlConfiguration.java:159) >>>> at >>>> org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1230) >>>> at >>>> org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517) >>>> at >>>> org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467) >>>> at >>>> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) >>>> at >>>> com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:219) >>>> at >>>> com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:194) >>>> at >>>> com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:134) >>>> at >>>> com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:447) >>>> at >>>> com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:454) >>>> at >>>> com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:461) >>>> at com.google.tracing.TraceContext.runInContext(TraceContext.java:703) >>>> at >>>> com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:338) >>>> at >>>> com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:330) >>>> at >>>> com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:458) >>>> at >>>> com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251) >>>> at java.lang.Thread.run(Thread.java:679) >>>> >>>> Uncaught exception from servlet >>>> javax.servlet.UnavailableException: Initialization failed. >>>> at >>>> com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:228) >>>> at >>>> com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:194) >>>> at >>>> com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:134) >>>> at >>>> com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:447) >>>> at >>>> com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:454) >>>> at >>>> com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:461) >>>> at com.google.tracing.TraceContext.runInContext(TraceContext.java:703) >>>> at >>>> com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:338) >>>> at >>>> com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:330) >>>> at >>>> com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:458) >>>> at >>>> com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251) >>>> at java.lang.Thread.run(Thread.java:679) >>>> >>>> >>>> We are starting to consider moving out of App Engine. This issue and the >>>> thing that our users tend to hit cold instances every n requests makes it >>>> not suitable for our need >>>> >>>> >>>> Marcin >>>> >>>> >>>> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
