Hi Ikai,
I'm afraid I don't have any unit test code easily available, but I
could make a code sample when I have a little more time. In the
meantime I have some more information. It's not just the memcache put
operation that fails in this way. I've also seen similar failures of
other API operations like datastore get:
com.google.apphosting.api.ApiProxy$OverQuotaException: The API
call datastore_v3.Get() required more quota than is available
and calling image transform operations (to convert PNG to JPEG):
com.google.apphosting.api.ApiProxy$OverQuotaException: The API
call images.Transform() required more quota than is available
In all cases I'm well within my daily quota for my app. During stress
testing (using JMeter and ramping up the client threads) I see a
consistent pattern of thirty or so successes, followed by thirty or so
failures (of memcache put, datastore get or image transform). Then I
see successes again, then failures and so on.
My interpretation of these data is as follows:
1) Stress test starts, all is well, all requests succeed for a while
2) GAE monitors the number of API calls the application makes in a 10-
second sliding window. (I'm guessing here.)
3) After a while GAE detects that the app is calling the various APIs
at a rate above a certain limit. So requests start failing.
4) The failed requests take longer to process than successful ones.
So when failures start happening, the rate of calling the APIs goes
down.
5) At some point the 10-second average (or whatever) dips below the
limit, so we get successes again.
6) And so on... with successful requests the API calling rate goes up
above the limit and requests start failing.
These are complete guesses, but would seem to fit the pattern I
observe. Is there likely to be any truth in this?
Thanks,
Jon
On 5 Mar, 19:40, "Ikai L (Google)" <[email protected]> wrote:
> Jon, do you happen to have unit test code that reproduces this that I
> can plug in to an app? I'd like to reproduce this.
>
>
>
> On Thu, Mar 4, 2010 at 1:31 PM, Jon Blower <[email protected]> wrote:
> > Hi,
>
> > I have a GAE-J application that involves requesting dynamically-
> > generated images. The application has fairly high latency (~1s per
> > request). I'm stress-testing the application using JMeter, hitting
> > GAE with four simultaneous threads, each looping through a fixed set
> > of requests. I get a consistent pattern, in which the first twenty or
> > thirty requests succeed, then every subsequent request fails. The
> > server log shows that the failures occur during a memcache put
> > operation:
>
> > "com.google.appengine.api.memcache.stdimpl.GCacheException: Policy
> > prevented put operation
> > at com.google.appengine.api.memcache.stdimpl.GCache.put(GCache.java:
> > 165)"
>
> > If I stop the stress test for a couple of minutes, then restart it I
> > get the same pattern: success for the first 20-30 requests, then this
> > failure thereafter. It's not the same requests that fail each time.
> > Also, the data I'm storing in the memcache is always 1MB or less. My
> > memcache set policy is the default of SET_ALWAYS.
>
> > Is there some policy restricting the number of memcache puts I can do
> > per second perhaps? Or is there a thread safety issue?
>
> > Here are a few more details about my application. I have stored high-
> > resolution images in the persistent store by breaking them up into
> > chunks of size 1MB or less. I have layered memcache above the
> > persistent store. So I have code that looks like this:
>
> > public byte[] requestChunk(String chunkId) {
> > byte[] chunk = searchMemcache(chunkId);
> > if (chunk != null) return chunk;
> > chunk = searchPersistentStore(chunkId);
> > if (chunk != null) {
> > putChunkInMemcache(chunkId, chunk); // *** This is where the
> > errors come from! ***
> > return chunk;
> > }
> > return null;
> > }
>
> > It's the putChunkInMemcache() method that fails during the stress
> > test.
>
> > Any guidance much appreciated!
> > Jon
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine for Java" group.
> > To post to this group, send email to [email protected].
> > To unsubscribe from this group, send email to
> > [email protected].
> > For more options, visit this group
> > athttp://groups.google.com/group/google-appengine-java?hl=en.
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App
> Enginehttp://googleappengine.blogspot.com|http://twitter.com/app_engine
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
