You probably won't disclose more details, but I'm curious whether you hint at disk I/O or at network I/O? In the past I've dealt (successfully) with some problems which bear similarity to the GAE stability issues. In my case the root cause were the TCP protocol peculiarities - namely that TCP connections are reliable once established, but the handshake itself can fail, and we were seeing batches of such failures when under load at seemingly random times. So instead of creating new connections all the time we had to introduce pools of e.g. keep-alive HTTP connections at a few points. I wonder if something similar is happening here.
- Peter On Mon, Nov 22, 2010 at 9:34 PM, Ikai Lan (Google) <[email protected]<ikai.l%[email protected]> > wrote: > I laughed, because sometimes divining the root cause of technical > issues at scale feels like occult work. Let's just say ... dealing > with I/O is hard. > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > Blogger: http://googleappengine.blogspot.com > Reddit: http://www.reddit.com/r/appengine > Twitter: http://twitter.com/app_engine > > > > On Fri, Nov 19, 2010 at 1:13 PM, Kenneth <[email protected]> wrote: > > I'd sacrifice a chicken to know the technical details of what's been > > going on. > > > > It's been more than 2 days since I got a deadline error so I'm a very > > happy camper (excepting the blobstore bugs). > > > > Thanks > > > > > > On Nov 19, 6:46 pm, "Ikai Lan (Google)" > > <[email protected]<ikai.l%[email protected]> > > > > wrote: > >> Well, our expectation was that the performance right after the > maintenance > >> wouldn't be sustainable for various reasons, however, our expectation is > >> that performance should *not* revert to the state it was right before > the > >> maintenance, or even close. > >> > >> -- > >> Ikai Lan > >> Developer Programs Engineer, Google App Engine > >> Blogger:http://googleappengine.blogspot.com > >> Reddit:http://www.reddit.com/r/appengine > >> Twitter:http://twitter.com/app_engine > >> > >> On Wed, Nov 17, 2010 at 1:57 AM, Francois Masurel <[email protected] > >wrote: > >> > >> > >> > >> > >> > >> > >> > >> > Yep, getting more error 500 recently, mostly caused by datastore > >> > timeouts and longer cold starts. > >> > >> > I didn't have them before last maintenance. > >> > >> > And yes, performance seems to be slipping back to the level it was > >> > before the maintenance. > >> > >> > Francois > >> > >> > On Nov 16, 9:33 pm, Robert Lancer <[email protected]> wrote: > >> > > Yup, just look at the spikeshttp:// > >> > code.google.com/status/appengine/detail/datastore/2010/11/16#a... > >> > >> > > Google whats going on with Cloud Portability? > >> >http://code.google.com/cloudportability/ > >> > >> > > From the article: > >> > > Using the Eclipse-based SpringSource Tool Suite, developers can now > >> > > choose to deploy their application on Google App Engine for > Business, > >> > > a VMware environment (your vSphere infrastructure, your choice of > >> > > vCloud partners, or VMforce), or other infrastructure such as Amazon > >> > > EC2. We call this cloud portability. > >> > >> > > When will we be able to deploy our apps to alternate > infrastructures? > >> > > Our businesses are suffering, I know Google as a business isnt > suffer > >> > > and they also dont have any core products running on App Engine, > maybe > >> > > you guys could put a core product product on Google App Engine, > (maybe > >> > > Ad Words?) so you can suffer along with us. > >> > >> > > On Nov 16, 3:27 pm, Joshua Smith <[email protected]> wrote: > >> > >> > > > Anecdotally, yes. It definitely does not seem as chipper. > >> > >> > > > On Nov 16, 2010, at 2:51 PM, Jamie H wrote: > >> > >> > > > > Everything was great after maintenance! But is it just me, or > do > >> > > > > things seem to be slipping away? I am seeing timeouts on > several > >> > > > > apps, along with slower cold starts and increased latency since > the > >> > > > > maintenance... anyone else? > >> > >> > > > > -- > >> > > > > You received this message because you are subscribed to the > Google > >> > Groups "Google App Engine" group. > >> > > > > To post to this group, send email to > >> > [email protected]. > >> > > > > To unsubscribe from this group, send email to > >> > [email protected]<google-appengine%[email protected]><google-appengine%2Bunsubscrib > [email protected]> > >> > . > >> > > > > For more options, visit this group athttp:// > >> > groups.google.com/group/google-appengine?hl=en. > >> > >> > -- > >> > You received this message because you are subscribed to the Google > Groups > >> > "Google App Engine" group. > >> > To post to this group, send email to > [email protected]. > >> > To unsubscribe from this group, send email to > >> > [email protected]<google-appengine%[email protected]><google-appengine%2Bunsubscrib > [email protected]> > >> > . > >> > For more options, visit this group at > >> >http://groups.google.com/group/google-appengine?hl=en. > > > > -- > > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > [email protected]<google-appengine%[email protected]> > . > > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
