I don't appreciate Google taking down my entire company for 2 hours because they are doing testing in production. We had one customer send out 500 emails today saying "hey, check out this site" and all they got was an error page. Really poor timing.
Last time Google took my entire company down, it was for days, because they decided to block CloudFlare and couldn't just roll back quickly. At least when I was hosting things myself, I had myself to blame... now I just have to hold my breath and hope someone is paying attention and decides to respond at some point in the future. I love the promise of not having to carry a pager, but it is also fear inducing at the same time. Of course I have the option of paying $500/month to have a phone number to call, but that seems kind of outrageous for an issue that looks like it shouldn't have made it to production in the first place. Really, 4+ hours of silence isn't good business. A simple response of 'Hey, we are looking into this, we'll let you know more when we know more' would have been super helpful for my blood pressure. At least I could then respond back to my client saying that I know it is being worked on. Christina, is something like that really out of the question? thanks, jon On Wed, Sep 12, 2012 at 3:12 PM, Kaan Soral <[email protected]> wrote: > This is why I love App Engine, when a problem occurs instead of having a > heart attack or committing suicide, you can just wait for it to be > resolved. I remember downtime's that lasted nearly a day when I was on a > custom server/php platform. App Engine really is a breeze, at the worst > case this happens :) > > However new release related bugs are very frequent > > > On Thursday, September 13, 2012 12:56:28 AM UTC+3, Christina Ilvento wrote: > >> *Hi All, >> >> Beginning yesterday, September 11, Google App Engine experienced two >> periods of serving degradation for a subset of Java applications due to a >> gradual roll-out of a new version of the Java runtime. Affected >> applications would have seen errors related to class loading. We have >> resolved the first issue by fixing the underlying bug. We are still >> investigating the cause of the second issue but have rolled back the >> problematic update and all affected applications should now be returned to >> normal serving behavior. >> >> No changes to your code or application configuration are needed at this >> time. We apologize for any inconvenience this issue has caused, and we’ll >> follow up with more details on the underlying incident and resolution soon. >> >> >> Regards, >> Christina Ilvento, App Engine PM* >> >> >> >> On Wed, Sep 12, 2012 at 1:34 PM, Kaan Soral <[email protected]> wrote: >> >>> I wonder what happened, subscribed to this topic to be updated, hope >>> someone explains what happened and additionally hope this never happens to >>> me (python) :) >>> >>> On Wednesday, September 12, 2012 11:00:49 PM UTC+3, Jeff Schnitzer wrote: >>> >>>> We are back up and running now after 2 hrs of downtime. >>>> >>>> To whoever fixed it: THANK YOU!!! >>>> >>>> To whoever broke it in the first place: SPANKINGS!!! >>>> >>>> Jeff >>>> >>>> On Wed, Sep 12, 2012 at 12:11 PM, Jeff Schnitzer <[email protected]> >>>> wrote: >>>> > More information: >>>> > >>>> > * The failure began at 10:54am (pacific). >>>> > * Same app on different appid has the same problem. >>>> > >>>> > (as listed in the stacktraces, the appid is voost0) >>>> > >>>> > Jeff >>>> > >>>> > On Wed, Sep 12, 2012 at 11:53 AM, Jeff Schnitzer < >>>> [email protected]> wrote: >>>> >> HEEEEEEEEEEEEELP! >>>> >> >>>> >> We have tried everything at this point. Shut down instances, tried >>>> to >>>> >> deploy a new version, even tried old versions. We've reported a >>>> >> production issue. Something is broken inside of GAE. The Guice >>>> error >>>> >> must be a symptom; the smoking gun seems to be: >>>> >> >>>> >> java.lang.**ClassNotFoundExcepti**on: java.io.FileOutputStream >>>> >> >>>> >> We've been down for 30 mins and getting complaints from our clients. >>>> >> This looks really, really bad. It's my nightmare scenario - an >>>> outage >>>> >> in GAE that is small enough not to raise major alarm bells, but >>>> >> nevertheless cripples my business. It is not the first time this >>>> has >>>> >> happened. It is shaking my faith in GAE. >>>> >> >>>> >> Jeff >>>> >> >>>> >> On Wed, Sep 12, 2012 at 11:11 AM, Jeff Schnitzer < >>>> [email protected]> wrote: >>>> >>> Our app has been running fine on the same version, no code deploys >>>> >>> (our versions are timestamped so this is 100% certain), for two >>>> days. >>>> >>> All of a sudden (10 mins ago) our app stopped running. Every >>>> attempt >>>> >>> to run a request produces this cryptic message: >>>> >>> >>>> >>> -------- >>>> >>> >>>> >>> 2012-09-12 10:56:21.791 >>>> >>> com.google.inject.servlet.**Guic**eFilter setPipeline: Multiple >>>> Servlet >>>> >>> injectors detected. This is a warning indicating that you have more >>>> >>> than one GuiceFilter running in your web application. If this is >>>> >>> deliberate, you may safely ignore this message. If this is NOT >>>> >>> deliberate however, your application may not work as expected. >>>> >>> D 2012-09-12 10:56:21.792 >>>> >>> st.voo.tick.util.cambridge.**Cam**bridgeSetup <init>: Establishing >>>> >>> cambridge view resolver >>>> >>> I 2012-09-12 10:56:21.792 >>>> >>> st.voo.tick.GuiceConfig contextInitialized: Guice initialization >>>> took 514 millis >>>> >>> W 2012-09-12 10:56:21.888 >>>> >>> Failed startup of context >>>> >>> com.google.apphosting.utils.**je**tty.**RuntimeAppEngineWebAppCont* >>>> *ext@**1479784{/,/base/data/home/****apps/s~voost0/2012-09-10-1715.****361669184733923098} >>>> >>>> >>> java.lang.RuntimeException: java.lang.RuntimeException: Unable to >>>> >>> instantiate MessageBodyReader >>>> >>> at org.jboss.resteasy.plugins.**pro** >>>> viders.RegisterBuiltin.**registe**r(RegisterBuiltin.java:**35) >>>> >>> at org.jboss.resteasy.spi.**Resteas**yDeployment.start(** >>>> ResteasyDepl**oyment.java:211) >>>> >>> at org.jboss.resteasy.plugins.**ser**ver.servlet.** >>>> ServletContainerDi**spatcher.**init(**ServletContainer**Dispatcher.**java:67) >>>> >>>> >>> at org.jboss.resteasy.plugins.**ser**ver.servlet.** >>>> FilterDispatcher.**init(**FilterDispatcher.java:39) >>>> >>> at st.voo.tick.util.**GuiceResteasy**FilterDispatcher.** >>>> init(**GuiceRes**teasyFilterDispatcher.**java:48) >>>> >>> at com.google.inject.servlet.**Filt**erDefinition.init(** >>>> FilterDefini**tion.java:114) >>>> >>> at com.google.inject.servlet.**Mana**gedFilterPipeline.** >>>> initPipeline**(**ManagedFilterPipeline.java:**98) >>>> >>> at com.google.inject.servlet.**Guic** >>>> eFilter.init(GuiceFilter.**java:**172) >>>> >>> at org.mortbay.jetty.servlet.**Filt**erHolder.doStart(** >>>> FilterHolder.**java:97) >>>> >>> at org.mortbay.component.**Abstract**LifeCycle.start(** >>>> AbstractLifeCy**cle.java:50) >>>> >>> at org.mortbay.jetty.servlet.**Serv**letHandler.initialize( >>>> **ServletH**andler.java:662) >>>> >>> at org.mortbay.jetty.servlet.**Cont** >>>> ext.startContext(Context.**java:**140) >>>> >>> at org.mortbay.jetty.webapp.**WebAp**pContext.startContext( >>>> **WebAppCo**ntext.java:1250) >>>> >>> at org.mortbay.jetty.handler.**Cont**extHandler.doStart(** >>>> ContextHand**ler.java:517) >>>> >>> at org.mortbay.jetty.webapp.**WebAp**pContext.doStart(** >>>> WebAppContext**.java:467) >>>> >>> at org.mortbay.component.**Abstract**LifeCycle.start(** >>>> AbstractLifeCy**cle.java:50) >>>> >>> at com.google.apphosting.runtime.**** >>>> jetty.AppVersionHandlerMap.**cre**ateHandler(**AppVersionHandlerMa**p.java:219) >>>> >>>> >>> at com.google.apphosting.runtime.**** >>>> jetty.AppVersionHandlerMap.**get**Handler(**AppVersionHandlerMap.**java:194) >>>> >>>> >>> at com.google.apphosting.runtime.****jetty.** >>>> JettyServletEngineAdapte**r.**serviceRequest(**JettyServletE** >>>> ngineAdapter.**java:134) >>>> >>> at com.google.apphosting.runtime.**** >>>> JavaRuntime$RequestRunnable.**ru**n(JavaRuntime.java:447) >>>> >>> at com.google.tracing.**TraceContex**t$** >>>> TraceContextRunnable.**runInCo**ntext(TraceContext.**java:452) >>>> >>> at com.google.tracing.**TraceContex**t$** >>>> TraceContextRunnable$1.run(**T**raceContext.java:459) >>>> >>> at com.google.tracing.**TraceContex**t.runInContext(** >>>> TraceContext.**java:701) >>>> >>> at com.google.tracing.**TraceContex**t$** >>>> AbstractTraceContextCallback**.**runInInheritedContextNoUnref(****TraceContext.java:336) >>>> >>>> >>> at com.google.tracing.**TraceContex**t$** >>>> AbstractTraceContextCallback**.**runInInheritedContext(**TraceCo**ntext.java:328) >>>> >>>> >>> at com.google.tracing.**TraceContex**t$** >>>> TraceContextRunnable.run(**Tra**ceContext.java:456) >>>> >>> at com.google.apphosting.runtime.**** >>>> ThreadGroupPool$PoolEntry.run(****ThreadGroupPool.java:251) >>>> >>> at java.lang.Thread.run(Thread.**ja**va:679) >>>> >>> Caused by: java.lang.RuntimeException: Unable to instantiate >>>> MessageBodyReader >>>> >>> at org.jboss.resteasy.spi.**Resteas**yProviderFactory.** >>>> registerProvi**der(**ResteasyProviderFactory.**java:**761) >>>> >>> at org.jboss.resteasy.plugins.**pro** >>>> viders.RegisterBuiltin.**registe**rProviders(**RegisterBuiltin.**java:70) >>>> >>>> >>> at org.jboss.resteasy.plugins.**pro** >>>> viders.RegisterBuiltin.**registe**r(RegisterBuiltin.java:**31) >>>> >>> ... 27 more >>>> >>> Caused by: java.lang.SecurityException: Unable to get members for >>>> >>> class org.jboss.resteasy.plugins.**pro**viders.DataSourceProvider >>>> >>> at com.google.appengine.runtime.**R**equest.process-** >>>> b6ca2b194d66ed2**3(Request.java) >>>> >>> at java.lang.Class.**getConstructor**s(Class.java:**291) >>>> >>> at org.jboss.resteasy.util.**PickCo**nstructor.** >>>> pickSingletonConstru**ctor(**PickConstructor.java:27) >>>> >>> at org.jboss.resteasy.spi.**Resteas**yProviderFactory.** >>>> getProviderIn**stance(**ResteasyProviderFactory**.java:**1032) >>>> >>> at org.jboss.resteasy.spi.**Resteas**yProviderFactory.** >>>> addMessageBod**yReader(**ResteasyProviderFactor**y.java:**478) >>>> >>> at org.jboss.resteasy.spi.**Resteas**yProviderFactory.** >>>> registerProvi**der(**ResteasyProviderFactory.**java:**757) >>>> >>> at org.jboss.resteasy.plugins.**pro** >>>> viders.RegisterBuiltin.**registe**rProviders(**RegisterBuiltin.**java:70) >>>> >>>> >>> at org.jboss.resteasy.plugins.**pro** >>>> viders.RegisterBuiltin.**registe**r(RegisterBuiltin.java:**31) >>>> >>> at org.jboss.resteasy.spi.**Resteas**yDeployment.start(** >>>> ResteasyDepl**oyment.java:211) >>>> >>> at org.jboss.resteasy.plugins.**ser**ver.servlet.** >>>> ServletContainerDi**spatcher.**init(**ServletContainer**Dispatcher.**java:67) >>>> >>>> >>> at org.jboss.resteasy.plugins.**ser**ver.servlet.** >>>> FilterDispatcher.**init(**FilterDispatcher.java:39) >>>> >>> at st.voo.tick.util.**GuiceResteasy**FilterDispatcher.** >>>> init(**GuiceRes**teasyFilterDispatcher.**java:48) >>>> >>> at com.google.inject.servlet.**Filt**erDefinition.init(** >>>> FilterDefini**tion.java:114) >>>> >>> at com.google.inject.servlet.**Mana**gedFilterPipeline.** >>>> initPipeline**(**ManagedFilterPipeline.java:**98) >>>> >>> at com.google.inject.servlet.**Guic** >>>> eFilter.init(GuiceFilter.**java:**172) >>>> >>> at org.mortbay.jetty.servlet.**Filt**erHolder.doStart(** >>>> FilterHolder.**java:97) >>>> >>> at org.mortbay.component.**Abstract**LifeCycle.start(** >>>> AbstractLifeCy**cle.java:50) >>>> >>> at org.mortbay.jetty.servlet.**Serv**letHandler.initialize( >>>> **ServletH**andler.java:662) >>>> >>> at org.mortbay.jetty.servlet.**Cont** >>>> ext.startContext(Context.**java:**140) >>>> >>> at org.mortbay.jetty.webapp.**WebAp**pContext.startContext( >>>> **WebAppCo**ntext.java:1250) >>>> >>> at org.mortbay.jetty.handler.**Cont**extHandler.doStart(** >>>> ContextHand**ler.java:517) >>>> >>> at org.mortbay.jetty.webapp.**WebAp**pContext.doStart(** >>>> WebAppContext**.java:467) >>>> >>> at org.mortbay.component.**Abstract**LifeCycle.start(** >>>> AbstractLifeCy**cle.java:50) >>>> >>> at com.google.tracing.**TraceContex**t$** >>>> TraceContextRunnable.**runInCo**ntext(TraceContext.**java:452) >>>> >>> at com.google.tracing.**TraceContex**t$** >>>> TraceContextRunnable$1.run(**T**raceContext.java:459) >>>> >>> at com.google.tracing.**TraceContex**t.runInContext(** >>>> TraceContext.**java:701) >>>> >>> at com.google.tracing.**TraceContex**t$** >>>> AbstractTraceContextCallback**.**runInInheritedContextNoUnref(****TraceContext.java:336) >>>> >>>> >>> at com.google.tracing.**TraceContex**t$** >>>> AbstractTraceContextCallback**.**runInInheritedContext(**TraceCo**ntext.java:328) >>>> >>>> >>> at com.google.tracing.**TraceContex**t$** >>>> TraceContextRunnable.run(**Tra**ceContext.java:456) >>>> >>> ... 1 more >>>> >>> Caused by: java.lang.reflect.**InvocationTa**rgetException >>>> >>> at sun.reflect.**GeneratedMethodAcc**essor11.**invoke(Unknown >>>> Source) >>>> >>> at sun.reflect.**DelegatingMethodAc**cessorImpl.**invoke(** >>>> DelegatingMe**thodAccessorImpl.**java:43) >>>> >>> at java.lang.reflect.Method.**invok**e(Method.java:616) >>>> >>> ... 29 more >>>> >>> Caused by: java.lang.**NoClassDefFoundError**: >>>> java/io/FileOutputStream >>>> >>> at java.lang.Class.**getDeclaredCon**structors0(**Native >>>> Method) >>>> >>> at java.lang.Class.**privateGetDecl**aredConstructors** >>>> (Class.java:**2406) >>>> >>> ... 29 more >>>> >>> Caused by: java.lang.**ClassNotFoundExcepti**on: >>>> java.io.FileOutputStream >>>> >>> ... 29 more >>>> >>> C 2012-09-12 10:56:21.889 >>>> >>> Uncaught exception from servlet >>>> >>> javax.servlet.**UnavailableExcep**tion: Initialization failed. >>>> >>> at com.google.apphosting.runtime.**** >>>> jetty.AppVersionHandlerMap.**cre**ateHandler(**AppVersionHandlerMa**p.java:228) >>>> >>>> >>> at com.google.apphosting.runtime.**** >>>> jetty.AppVersionHandlerMap.**get**Handler(**AppVersionHandlerMap.**java:194) >>>> >>>> >>> at com.google.apphosting.runtime.****jetty.** >>>> JettyServletEngineAdapte**r.**serviceRequest(**JettyServletE** >>>> ngineAdapter.**java:134) >>>> >>> at com.google.apphosting.runtime.**** >>>> JavaRuntime$RequestRunnable.**ru**n(JavaRuntime.java:447) >>>> >>> at com.google.tracing.**TraceContex**t$** >>>> TraceContextRunnable.**runInCo**ntext(TraceContext.**java:452) >>>> >>> at com.google.tracing.**TraceContex**t$** >>>> TraceContextRunnable$1.run(**T**raceContext.java:459) >>>> >>> at com.google.tracing.**TraceContex**t.runInContext(** >>>> TraceContext.**java:701) >>>> >>> at com.google.tracing.**TraceContex**t$** >>>> AbstractTraceContextCallback**.**runInInheritedContextNoUnref(****TraceContext.java:336) >>>> >>>> >>> at com.google.tracing.**TraceContex**t$** >>>> AbstractTraceContextCallback**.**runInInheritedContext(**TraceCo**ntext.java:328) >>>> >>>> >>> at com.google.tracing.**TraceContex**t$** >>>> TraceContextRunnable.run(**Tra**ceContext.java:456) >>>> >>> at com.google.apphosting.runtime.**** >>>> ThreadGroupPool$PoolEntry.run(****ThreadGroupPool.java:251) >>>> >>> at java.lang.Thread.run(Thread.**ja**va:679) >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To view this discussion on the web visit https://groups.google.com/d/** >>> msg/google-appengine/-/**EJUrxhiFMp4J<https://groups.google.com/d/msg/google-appengine/-/EJUrxhiFMp4J> >>> . >>> >>> To post to this group, send email to google-a...@googlegroups.**com. >>> To unsubscribe from this group, send email to google-appengi...@** >>> googlegroups.com. >>> >>> For more options, visit this group at http://groups.google.com/** >>> group/google-appengine?hl=en<http://groups.google.com/group/google-appengine?hl=en> >>> . >>> >> >> >> >> -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/r5WgDSWVVvcJ. > > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
