I don't think this is related to M/R datastore spikes. The app has been running stable since months and I'm suffering of this problems since 3-4 weeks now.
FYI, I have also created an issue in the production tracker: http://code.google.com/p/googleappengine/issues/detail?id=5790 - The whole app is down, even Servlets that doesn't use the datastore respond with an HTTP 500 error - When I do nothing the app recoveres itself after a few hours - I once managed to fix it when I changed the Max Idle Instance setting back to automatic. So this could be an hint that all my instances were somehow blocked and app engine wasn't able to spin up more instances. - The usual fix is to clear all data from the memcache. If the memcache is empty everything wents back to normal immediately. The memcache is constantly filled with 12 items, each has a size <1MB. I agree one could think it is an code issue, but as I said the whole app is down, even Servlets that doesn't use Memcache/Datastor or vice versa are not accessible. I've refactored my code 10 times already. I don't know what to do anymore. All I get is "move to HRD", but I think I have to move away from App Engine. To make this clear: I will move to HRD when the tool is released. Yeah I know there is a beta. But anyway what has that todo with my problem? The whole app is down, when the system status of M/R says okie dokie. On 28 Sep., 22:51, "Ikai Lan (Google)" <[email protected]> wrote: > Could these be related to datastore latency spikes? Those have been known to > occur regularly on master/slave datastore, and we are encouraging developers > to move to high replication. You can sign up for migration tool access here: > > https://docs.google.com/a/google.com/spreadsheet/viewform?authkey=CLX... > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > plus.ikailan.com | twitter.com/ikai > > > > > > > > On Tue, Sep 27, 2011 at 4:39 AM, Bernd F <[email protected]> wrote: > > Tonight it happened again, but after ~1 hour the app recovered on its > > own. There must be something wrong with app engine. > > > On 20 Sep., 00:08, Bernd Final <[email protected]> wrote: > > > I've had this issue for 5 or 6 times already. My app is running for a > > > long time and I've had no problems so far, except this one: > > > > Out of nowhere, 20-30 seconds latency and only serving HTTP 500 errors > > > for 99,9% of all requests. In most Servlets (yeah I'm using Java if > > > that matters) I'm accessing the memcache, sometimes the datastore > > > (Master/Slave). As you can imagine the app is totally unusable when > > > this happens. When it occured the first time I didn't to any special, > > > just waited for a couple of hours until the app recovered on it's own. > > > > Dashboard Milliseconds/Request Chart: > > > Yesterday:http://bit.ly/pSF66E > > > Last 30 Days:http://bit.ly/nYdZ2S > > > > When it happend again and again I tried to figure out what's going on, > > > luckily I had some kind of Admin-Servlet deployed which allowed me to > > > clear the entire memcache. Tried it, and voila, everything back to > > > normal. > > > > - The memcache size was up to 1.3-1.5 MB in total (I did a rough > > > estimation with the ByteArrayOutputStream trick). 12 Objects are > > > stored in the memcache. Each Object is POJO which implements > > > Serializable and contains some Strings, Integers and ArrayLists. > > > - My first idea was to minimize the memory usage, because I though I > > > was hitting the 1MB limit. At this time I didn't know the limit is for > > > each object not for the whole cache, anyway I was able to reduce the > > > size by 30-40% so the size is now about 800kb - 1MB > > > - I'm running a cron job which cleans out garbage of the memcache. > > > - This is how I access the memcache on every request: > > > CacheManager.getInstance().getCacheFactory().createCache(Collections.emptyMap()); > > > - I'm using put and get Methods for storing / retrieving, > > > - The key length is about 160 bytes > > > > Unfortunately I was not able to solve this issue yet, I would be very > > > happy for ANY input. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
