Thanks for the info, we tried ramping up the requests and it seemed to help somewhat but we still saw quite a lot of high pending_ms times. When we set threadsafe=false we get more instances allocated and lower pending times which seems odd.
On May 27, 5:01 am, Robert Kluin <[email protected]> wrote: > Hi, > Did you try ramping up your request ratea little more slowly? If > your app has a long startup time (and it liekly does since it is Java) > and services most requests very fast, then I've sometimes seen the > scheduler be a bit too resistent to spin up new instances. In that > situation I sometimes start seeing those nasty pending_ms times, which > causes increased latency, which then causes the app not to get more > instances, etc.... To me it appears to be a very nasty feedback loop. > > Watch the instance info in your app's dashboard, paying particular > attention to the latency. You want to try keeping that number well > under 1000ms if possible. I find that by ramping up the traffic a bit > more slowly I get more instnances. Note that by slowly I'm not > talking about hours, but at least over a minute or two. Another > option is to 'prime' the app with a couple smaller bursts of traffic > to get a few instances spun up. > > Robert > > On Thu, May 26, 2011 at 07:36, Hamish <[email protected]> wrote: > > Hi, > > > We are developing a Java app for Google App Engine. We did some > > testing today using "ab" to hit a particular URL. We had 1000 > > concurrent threads in ab hitting the URL 1 million times. > > > We were surprised by the response times. A lot of the responses were > > taking well over a second to return with some even taking up to 4 > > seconds. This is a lot higher than we would like. From looking at the > > logs it seems the majority of the time is in "pending_ms". At the peak > > of the traffic GAE had a total of 7 instances with 4 active > > instances. > > > Could someone explain what "pending_ms" actually means? What can we do > > to improve it? > > > Some application details: > > - <threadsafe>true</threadsafe> in the appengine-web.xml file. > > - the app uses the Restlet framework > > - the URL we hit returns JSON data less than 2kb in size > > - the app increments a counter in memcache for each request > > - the datastore is _not_ hit for a typical request > > - we have a cron that runs every couple minutes to persist the > > counters to the datastore > > > Here is a typical log entry: > > > 50.16.218.117 - - [26/May/2011:02:51:26 -0700] "GET /v1/{my_url} HTTP/ > > 1.1" 200 370 - "ApacheBench/2.3,gzip(gfe)" "{appname}.appspot.com" > > ms=1262 cpu_ms=19 api_cpu_ms=0 cpm_usd=0.000604 pending_ms=1179 > > > note: I have removed the actual app name and URL. > > > Thanks for your help. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
