Is there any chance you're hitting per minute quotas? http://code.google.com/appengine/docs/quotas.html
Image transforms have this quota, for instance: 14,000 transforms/minute Memcache has a limit of 56/mb of writes and 284/mb of reads. On Fri, Mar 5, 2010 at 1:49 PM, Jon Blower <[email protected]> wrote: > Hi Ikai, > > I'm afraid I don't have any unit test code easily available, but I > could make a code sample when I have a little more time. In the > meantime I have some more information. It's not just the memcache put > operation that fails in this way. I've also seen similar failures of > other API operations like datastore get: > > com.google.apphosting.api.ApiProxy$OverQuotaException: The API > call datastore_v3.Get() required more quota than is available > > and calling image transform operations (to convert PNG to JPEG): > > com.google.apphosting.api.ApiProxy$OverQuotaException: The API > call images.Transform() required more quota than is available > > In all cases I'm well within my daily quota for my app. During stress > testing (using JMeter and ramping up the client threads) I see a > consistent pattern of thirty or so successes, followed by thirty or so > failures (of memcache put, datastore get or image transform). Then I > see successes again, then failures and so on. > > My interpretation of these data is as follows: > 1) Stress test starts, all is well, all requests succeed for a while > 2) GAE monitors the number of API calls the application makes in a 10- > second sliding window. (I'm guessing here.) > 3) After a while GAE detects that the app is calling the various APIs > at a rate above a certain limit. So requests start failing. > 4) The failed requests take longer to process than successful ones. > So when failures start happening, the rate of calling the APIs goes > down. > 5) At some point the 10-second average (or whatever) dips below the > limit, so we get successes again. > 6) And so on... with successful requests the API calling rate goes up > above the limit and requests start failing. > > These are complete guesses, but would seem to fit the pattern I > observe. Is there likely to be any truth in this? > > Thanks, > Jon > > On 5 Mar, 19:40, "Ikai L (Google)" <[email protected]> wrote: >> Jon, do you happen to have unit test code that reproduces this that I >> can plug in to an app? I'd like to reproduce this. >> >> >> >> On Thu, Mar 4, 2010 at 1:31 PM, Jon Blower <[email protected]> wrote: >> > Hi, >> >> > I have a GAE-J application that involves requesting dynamically- >> > generated images. The application has fairly high latency (~1s per >> > request). I'm stress-testing the application using JMeter, hitting >> > GAE with four simultaneous threads, each looping through a fixed set >> > of requests. I get a consistent pattern, in which the first twenty or >> > thirty requests succeed, then every subsequent request fails. The >> > server log shows that the failures occur during a memcache put >> > operation: >> >> > "com.google.appengine.api.memcache.stdimpl.GCacheException: Policy >> > prevented put operation >> > at com.google.appengine.api.memcache.stdimpl.GCache.put(GCache.java: >> > 165)" >> >> > If I stop the stress test for a couple of minutes, then restart it I >> > get the same pattern: success for the first 20-30 requests, then this >> > failure thereafter. It's not the same requests that fail each time. >> > Also, the data I'm storing in the memcache is always 1MB or less. My >> > memcache set policy is the default of SET_ALWAYS. >> >> > Is there some policy restricting the number of memcache puts I can do >> > per second perhaps? Or is there a thread safety issue? >> >> > Here are a few more details about my application. I have stored high- >> > resolution images in the persistent store by breaking them up into >> > chunks of size 1MB or less. I have layered memcache above the >> > persistent store. So I have code that looks like this: >> >> > public byte[] requestChunk(String chunkId) { >> > byte[] chunk = searchMemcache(chunkId); >> > if (chunk != null) return chunk; >> > chunk = searchPersistentStore(chunkId); >> > if (chunk != null) { >> > putChunkInMemcache(chunkId, chunk); // *** This is where the >> > errors come from! *** >> > return chunk; >> > } >> > return null; >> > } >> >> > It's the putChunkInMemcache() method that fails during the stress >> > test. >> >> > Any guidance much appreciated! >> > Jon >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "Google App Engine for Java" group. >> > To post to this group, send email to >> > [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group >> > athttp://groups.google.com/group/google-appengine-java?hl=en. >> >> -- >> Ikai Lan >> Developer Programs Engineer, Google App >> Enginehttp://googleappengine.blogspot.com|http://twitter.com/app_engine > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- Ikai Lan Developer Programs Engineer, Google App Engine http://googleappengine.blogspot.com | http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
