Oh, and if you haven't already, star these issues: http://code.google.com/p/googleappengine/issues/detail?id=7706 http://code.google.com/p/googleappengine/issues/detail?id=7865
Jeff Random not-so-amusing anecdote: My app, voo.st, is also down right now - not because of appengine, but because the .st domain registry started screwing up dns delegation this morning. Half of requests fail to delegate to the correct 'voo' nameservers. And their support channels are doing nothing. I've never seen a nic fail before... and it's not like I can switch to a different provider! On Mon, Aug 20, 2012 at 2:36 PM, Jeff Schnitzer <[email protected]> wrote: > This looks like a startup request that is taking too long - is that the issue? > > If so, it could be yet another blip in startup times - as I (and > others) have complained many times in the past, there seems to be a > lot of variance in startup times. 20s today could be 60s deadline > failures tomorrow. > > One thing that may help: Start a cron job that hits your app as often > as possible. Cron has a 10m deadline so you're almost guaranteed to > get an instance off the ground even when it blows the limit. > > Jeff > > On Mon, Aug 20, 2012 at 12:55 PM, Mos <[email protected]> wrote: >>> Is that the full stacktrace?!? >> >> No, here is one example: >> >> com.google.apphosting.api.DeadlineExceededException: This request >> (681ae7cef438e16b) started at 2012/08/20 16:30:31.376 UTC and was >> still executing at 2012/08/20 16:31:30.903 UTC. >> at >> com.google.appengine.runtime.Request.process-681ae7cef438e16b(Request.java) >> at java.util.zip.ZipFile.read(Native Method) >> at java.util.zip.ZipFile.access$1200(ZipFile.java:57) >> at java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:476) >> at java.util.zip.ZipFile$1.fill(ZipFile.java:259) >> at >> java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158) >> at sun.misc.Resource.getBytes(Resource.java:124) >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:273) >> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:616) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:266) >> at >> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:188) >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1003) >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:907) >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485) >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) >> at >> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:270) >> at >> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:125) >> at >> org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:616) >> at >> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:148) >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1003) >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:907) >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485) >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) >> at >> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293) >> at >> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) >> at >> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290) >> at >> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:196) >> at >> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:400) >> at >> org.springframework.context.support.AbstractApplicationContext.getBeansOfType(AbstractApplicationContext.java:1146) >> at >> org.springframework.beans.factory.BeanFactoryUtils.beansOfTypeIncludingAncestors(BeanFactoryUtils.java:275) >> at >> org.springframework.web.servlet.handler.AbstractUrlHandlerMapping.initInterceptors(AbstractUrlHandlerMapping.java:168) >> at >> org.springframework.web.servlet.handler.AbstractHandlerMapping.initApplicationContext(AbstractHandlerMapping.java:110) >> at >> org.springframework.web.servlet.handler.AbstractDetectingUrlHandlerMapping.initApplicationContext(AbstractDetectingUrlHandlerMapping.java:57) >> at >> org.springframework.context.support.ApplicationObjectSupport.initApplicationContext(ApplicationObjectSupport.java:119) >> at >> org.springframework.web.context.support.WebApplicationObjectSupport.initApplicationContext(WebApplicationObjectSupport.java:72) >> >> >> On Mon, Aug 20, 2012 at 6:32 PM, Jeff Schnitzer <[email protected]> wrote: >>> Is that the full stacktrace?!? >>> >>> Jeff >>> >>> On Mon, Aug 20, 2012 at 12:15 PM, Mos <[email protected]> wrote: >>>> The last two days the instances kept unstable. Today I kill all >>>> instances hopping this helps. But 3 instances spanned up again and now >>>> every second request to the application fails with the known GAE >>>> problem: >>>> >>>> com.google.apphosting.api.DeadlineExceededException: This request >>>> (9706ce3068c95802) started at 2012/08/20 16:05:11.636 UTC and was >>>> still executing at 2012/08/20 16:06:11.214 UTC. >>>> at >>>> com.google.appengine.runtime.Request.process-9706ce3068c95802(Request.java) >>>> >>>> PLEASE GOOGLE - SOME FEEDBACK / EVALUATIONS WOULD BE NICE ! >>>> >>>> see also: http://code.google.com/p/googleappengine/issues/detail?id=7910 >>>> >>>> >>>> latest Pingdom History: >>>> [email protected] UP alert: krisentalk (www.krisentalk.de) is UP >>>> 18:02 >>>> [email protected] DOWN alert: krisentalk (www.krisentalk.de) is DOWN >>>> 18:01 >>>> [email protected] UP alert: krisentalk (www.krisentalk.de) is UP >>>> 17:59 >>>> [email protected] DOWN alert: krisentalk (www.krisentalk.de) is DOWN >>>> 17:58 >>>> [email protected] UP alert: krisentalk (www.krisentalk.de) is UP >>>> 17:53 >>>> [email protected] DOWN alert: krisentalk (www.krisentalk.de) is DOWN >>>> 17:52 >>>> [email protected] UP alert: krisentalk (www.krisentalk.de) is UP >>>> 17:50 >>>> [email protected] DOWN alert: krisentalk (www.krisentalk.de) is DOWN >>>> 17:47 >>>> [email protected] UP alert: krisentalk (www.krisentalk.de) is UP >>>> 17:45 >>>> [email protected] DOWN alert: krisentalk (www.krisentalk.de) is DOWN >>>> 17:44 >>>> [email protected] UP alert: krisentalk (www.krisentalk.de) is UP >>>> 17:42 >>>> [email protected] DOWN alert: krisentalk (www.krisentalk.de) is DOWN >>>> 17:41 >>>> [email protected] DOWN alert: krisentalk (www.krisentalk.de) is DOWN >>>> 17:35 >>>> [email protected] UP alert: krisentalk (www.krisentalk.de) is UP >>>> 16:52 >>>> [email protected] DOWN alert: krisentalk (www.krisentalk.de) is DOWN >>>> 16:52 >>>> [email protected] UP alert: krisentalk (www.krisentalk.de) is UP >>>> 04:24 >>>> [email protected] DOWN alert: krisentalk (www.krisentalk.de) is DOWN >>>> 04:23 >>>> [email protected] UP alert: krisentalk (www.krisentalk.de) is UP >>>> 02:39 >>>> [email protected] DOWN alert: krisentalk (www.krisentalk.de) is DOWN >>>> 02:38 >>>> [email protected] UP alert: krisentalk (www.krisentalk.de) is UP So >>>> 19:16 >>>> [email protected] DOWN alert: krisentalk (www.krisentalk.de) is DOWN >>>> So 19:15 >>>> [email protected] UP alert: krisentalk (www.krisentalk.de) is UP So >>>> 17:19 >>>> [email protected] DOWN alert: krisentalk (www.krisentalk.de) is DOWN >>>> So 17:19 >>>> [email protected] UP alert: krisentalk (www.krisentalk.de) is UP So >>>> 16:03 >>>> [email protected] DOWN alert: krisentalk (www.krisentalk.de) is DOWN >>>> So 16:02 >>>> [email protected] UP alert: krisentalk (www.krisentalk.de) is UP So >>>> 5:15 >>>> [email protected] DOWN alert: krisentalk (www.krisentalk.de) is DOWN >>>> So 5:14 >>>> >>>> >>>> On Sat, Aug 18, 2012 at 2:56 PM, Mos <[email protected]> wrote: >>>>> Add again. The last 48 hours the scheduler creates and closes >>>>> instances without reason (traffic as usual, no software updates, no >>>>> special tasks) >>>>> >>>>> That's definitely not a "WorkAsIntended" issue! >>>>> >>>>> As a result instance hours go up and we have to pay again for GAE's >>>>> fault. Great business-model: Let customers pay for bugs and underline >>>>> it as "WorkAsIntended"! >>>>> >>>>> Check: http://code.google.com/p/googleappengine/issues/detail?id=7910 >>>>> >>>>> On Sat, Aug 4, 2012 at 9:00 PM, Mos <[email protected]> wrote: >>>>>> Again a not very reliable GAE month: >>>>>> >>>>>> Pingdom July >>>>>> >>>>>> Uptime Downtime Outages Response time >>>>>> 99.90% 0h 43m 35s 27 466 ms >>>>>> >>>>>> >>>>>> On Wed, Aug 1, 2012 at 11:50 PM, Mos <[email protected]> wrote: >>>>>>> And again I have to pay for GAE issues: >>>>>>> >>>>>>> On Jul 30 the Frontend Instance Hours goes beyond the free limit. >>>>>>> First time this week in my application history. Remember: The >>>>>>> application was >>>>>>> unusable on this day because GAE starts instances like crazy (on low >>>>>>> traffic). >>>>>>> Frontent Instance Hours were consumed of the buggy instance scheduler. >>>>>>> >>>>>>> But to rescue there is another bug in GAE. The billing does not work. >>>>>>> It's stuck for months and I still read the following message on my >>>>>>> billing history: >>>>>>> "We were unable to process your last payment. If the account balance >>>>>>> ($2.32) is not paid in full by 06/01/2012, this application's quotas >>>>>>> may be reset to the free levels." >>>>>>> >>>>>>> That's ingenious, Google. ;) >>>>>>> >>>>>>> >>>>>>> On Mon, Jul 30, 2012 at 5:47 PM, Mos <[email protected]> wrote: >>>>>>>> Anyone else seeing issue with GAE instance management? >>>>>>>> In one minute - 10 requests - around 5 instances are started. Old ones >>>>>>>> do not response. New instances are created again and again... >>>>>>>> >>>>>>>> http://code.google.com/p/googleappengine/issues/detail?id=7910 >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Google App Engine" group. >>>> To post to this group, send email to [email protected]. >>>> To unsubscribe from this group, send email to >>>> [email protected]. >>>> For more options, visit this group at >>>> http://groups.google.com/group/google-appengine?hl=en. >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
