Hi Adrin, I feel your pain as my costs are similar and my servers are crashing every day at 11pm-9am PST due to this bug: https://code.google.com/p/googleappengine/issues/detail?id=8788
Today I have migrated to HDR and the whole process was much easier then I was expecting. If you're still on master/slave, migrate to HDR before doing any architectural changes. I have seen that the servers are more stable and the warmups are going under 60 seconds. It was over 60s before the migration. Although I still have to see if they will be ok on the 11pm-9am gap. I do understand that migration is the least of the things you want to do right now, but there's no other way. There's anything you can do to decrease the need for 40GB? In my case I have a triple layer of caching, where it goes memory -> memcache -> disk. You may consider having a backend as memory database if you must ensure that your objects are long-lived. good luck rafa On Thu, Mar 21, 2013 at 1:19 PM, Bryce Cutt <[email protected]> wrote: > I have noticed this as well. Going into the instances control panel for > the old version and manually stopping them fixes it for me but it can take > a bit of time for new version instances to be used. > > Also, I have become very used to doing everything I can to not break the > schema across versions and if I have to I do it in two steps. First deploy > supports both old and new and second deploy removes old once I am happy > everything has switched over. Not always possible though. > > > On Thursday, March 21, 2013 8:29:12 AM UTC-7, Andrin von Rechenberg wrote: >> >> On March 5th Google must have updated the AppEngine infrastructure and >> ever since, AppEngine is quite annoying. >> >> *Memcache Size shrunk 10x* >> Before we had about 40 GB constantly. Now we have on average 4GB, but it >> changes >> greatly during the day. Sometimes we only have a few seconds in memcache. >> Shrinking memcache by a factor of 10x from 1 day to another without a >> headsup to >> the devs is really bad. >> >> *Releases don't update all instances immediately* >> We get "sticky" instances running old code (about 10%) forever. >> This is super dangerous if you change for example database schemas. >> There is no easy way to resolve the issue. Changing the default version >> doesn't help either. You can't run update and ruin MapReduces to analyse >> data because you first need to make sure all instances run the new code >> which is tricky. >> >> Google did those changes without giving us a heads up. >> Decisions like this really screw us developers. I'm extremely >> dissatisfied with Google. >> We spend well over $100'000 on server costs and would expect Google >> to make sure our experience is great. But it's not. >> >> Anybody else having trouble lately? >> >> app-id: miumeet-hr >> >> Cheers, >> -Andrin >> > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/google-appengine?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
