Thanks Gary. I'll run this code through my mental debugger to see if anything pops out.
I'm going to write my own more simplified test where I just create some python code that hits the request handler.. and once it gets a response.. it just tries again (For maybe 10 - 20 times in a row.).. Then, just see if I can run 30 instances (or at least more than 10) of that code at once. On Tue, Mar 2, 2010 at 12:21 PM, Gary Orser <[email protected]> wrote: > Eli, > > You have the python request server. > Here is the java client: > You'll have to get the libraries yourself. > > Cheers, Gary > > import java.util.ArrayList; > import java.util.concurrent.Callable; > import java.util.concurrent.ExecutionException; > import java.util.concurrent.ExecutorService; > import java.util.concurrent.Executors; > import java.util.concurrent.Future; > > import org.apache.commons.io.IOUtils; > import org.apache.http.HttpHost; > import org.apache.http.HttpResponse; > import org.apache.http.HttpVersion; > import org.apache.http.client.HttpClient; > import org.apache.http.client.methods.HttpGet; > import org.apache.http.conn.ClientConnectionManager; > import org.apache.http.conn.params.ConnManagerParams; > import org.apache.http.conn.params.ConnPerRouteBean; > import org.apache.http.conn.scheme.PlainSocketFactory; > import org.apache.http.conn.scheme.Scheme; > import org.apache.http.conn.scheme.SchemeRegistry; > import org.apache.http.conn.ssl.SSLSocketFactory; > import org.apache.http.impl.client.DefaultHttpClient; > import org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager; > import org.apache.http.params.BasicHttpParams; > import org.apache.http.params.HttpParams; > import org.apache.http.params.HttpProtocolParams; > > > import org.apache.commons.logging.Log; > import org.apache.commons.logging.LogFactory; > > public class Main > { > private Log log = LogFactory.getLog(Main.class); > // ADJUST: number of threads to make requests on > public static int NUM_PARALLEL_SECTION_REQUESTS = 20; > public static HttpParams httpParams = new BasicHttpParams(); > { > httpParams.setBooleanParameter("http.protocol.expect- > continue", false); > // ADJUST: if this is included, will use 8888 as a > proxy port. Charles Proxy defaults to this port. > //httpParams.setParameter("http.route.default-proxy", > new HttpHost("localhost", 8888)); > } > > protected class GetSection implements Callable<String> > { > protected int index; > protected HttpClient client; > protected String URL; > public GetSection(int index, HttpClient client, String > URL) > { > this.index = index; > this.client = client; > this.URL = URL; } > public String call() throws Exception > { > HttpGet getSection = new > HttpGet(URL); HttpResponse respSection = > client.execute(getSection); > String foo = > IOUtils.toString(respSection.getEntity().getContent(), "UTF-8"); > return foo; > } } > > public static void main(String[] args) throws Exception > { new Main().maint(args); > } > > public void maint(String[] args) throws Exception { > SchemeRegistry schemeRegistry = new SchemeRegistry(); > schemeRegistry.register(new Scheme("http", > PlainSocketFactory.getSocketFactory(), 80)); > schemeRegistry.register(new Scheme("https", > SSLSocketFactory.getSocketFactory(), 443)); > HttpParams params = new BasicHttpParams(); > ConnManagerParams.setMaxTotalConnections(params, > NUM_PARALLEL_SECTION_REQUESTS); > ConnManagerParams.setMaxConnectionsPerRoute(params, > new ConnPerRouteBean(NUM_PARALLEL_SECTION_REQUESTS)); > HttpProtocolParams.setVersion(params, > HttpVersion.HTTP_1_1); > ClientConnectionManager cm = new > ThreadSafeClientConnManager(params, schemeRegistry); > HttpClient client = new DefaultHttpClient(cm, > httpParams); > > ExecutorService es = > Executors.newFixedThreadPool(NUM_PARALLEL_SECTION_REQUESTS); > // ADJUST: total number of requests to make. > int numSections = 100; > ArrayList<Future<String>> futures = new > ArrayList<Future<String>>(numSections); > log.info("queuing requests"); > for (int i = 0; i < numSections; i++) > { > // ADJUST: set a real hostname here > futures.add(es.submit(new GetSection(i, > client, "http://yourappid.appspot.com/sit/"; + Integer.toString(i)))); > // ADJUST: stagger initial requests with this > sleep > //Thread.sleep(200); > } > > es.shutdown(); > > log.info("waiting for thread pool to finish"); > while (!es.isTerminated()) > Thread.sleep(500); > > log.info("all requests queued"); > > try > { > for (Future<String> future: futures) > future.get(); > log.info("got all futures"); > } > catch (ExecutionException e) > { > // TODO: not really sure what to do if cause > is Throwable but not Exception > if (e.getCause() instanceof Exception) > throw (Exception)e.getCause(); > } > } > > On Mar 2, 9:50 am, Eli Jones <[email protected]> wrote: > > What I'm suggesting is.. You need to create a simple test setup that > > recreates this dynamic request limit error.. (It definitely should not > > take 8mb of code). > > > > I will see if I can create a handler like the one you posted, deploy > > it, and then run 30 seperate processes that keep getting from that > > handler.. (I can write this up in less than 10kb or python code)... > > > > My guess is this will work. Without seeing sample code.. I can't tell > > where you may be going wrong (or where GAE may be breaking) > > > > On 3/2/10, Gary Orser <[email protected]> wrote: > > > > > > > > > Actually, 4 threads was before we optimized server side, and set up > > > the test environment. > > > > > I have a tarball, about 8mb, with the test environment. (django and > > > libraries, grrr) > > > What is the best way to post this? I don't see any file attachments > > > on groups. > > > > > Cheers, Gary > > > > > On Mar 2, 8:23 am, Eli Jones <[email protected]> wrote: > > >> Are these threads you're using (at this point, it really seems like > you > > >> should post some simplified code to illustrate the issue at hand) > waiting > > >> for their response before trying to get again? > > > > >> Posting some code to help recreate this issue will lead to a much > faster > > >> resolution.. as it stands.. I just know that someone on the internet > has > > >> "10 > > >> threads" that are hitting a dynamic request limit. > > > > >> I also know that in the initial e-mail, when the request took longer > to > > >> return.. these "threads" were hitting a lower dynamic request limit > (only > > >> 4 > > >> could run). This suggest that there is an important detail to how > your > > >> "threads" are doing their work.. and we would need that to provide > useful > > >> help. > > > > >> Thanks for info. > > > > >> On Tue, Mar 2, 2010 at 10:01 AM, Gary Orser <[email protected]> > wrote: > > > > >> > But that's the point. I can not reach 30 active requests. > > >> > I can only reach 10 active requests without error. > > > > >> > Any ideas on how I can debug this? > > > > >> > Cheers, Gary. > > > > >> > On Mar 2, 7:05 am, "Nick Johnson (Google)" <[email protected] > > > > >> > wrote: > > >> > > Hi, > > > > >> > > On Tue, Mar 2, 2010 at 1:54 PM, Wooble <[email protected]> > wrote: > > >> > > > The 500 requests per second number relies on the > > >> > > > probably-unreasonable > > >> > > > assumption that each request can complete in ~75ms. > Deliberately > > >> > > > making your requests take a whole 3 seconds each is, obviously, > not > > >> > > > going to work. You can only have 10 instances active at a time > by > > >> > > > default; if the pages you're serving actually take 3 seconds to > > >> > > > complete you'll need to optimize things a whole lot or be stuck > with > > >> > > > a > > >> > > > 3.33 request/sec maximum. > > > > >> > > Actually, the default limit is 30 active requests. > > > > >> > > -Nick Johnson > > > > >> > > > On Mar 1, 11:33 pm, Gary Orser <[email protected]> wrote: > > >> > > > > Hi Nick, > > > > >> > > > > Hmm, I was running tests on a billing enabled appspot today. > 100 > > >> > > > > requests/test. > > > > >> > > > > 10 threads getting a URL with a 3 second sleep (to emulate > > >> > > > > computation) on appspot, was the most I could get without > getting > > >> > > > > 500 > > >> > > > > errors. > > >> > > > > If I raised the thread pool beyond 10, I started getting > errors?? > > > > >> > > > > That doesn't reconcile very well with this statement from the > > >> > > > > appengine website. > > >> > > > > "Requests > > >> > > > > The total number of requests to the app. The per-minute > quotas > > >> > for > > >> > > > > application with billing enabled allow for up to 500 requests > per > > >> > > > > second--more than one billion requests per month. If your > > >> > > > > application > > >> > > > > requires even higher quotas than the "billing-enabled" values > > >> > > > > listed > > >> > > > > below, you can request an increase in these limits here. > > >> > > > > " > > > > >> > > > > Is there some billing setting that affects this? > > > > >> > > > > Cheers, Gary > > > > >> > > > > PS. dead simple request handler. > > > > >> > > > > import time > > >> > > > > from django import http > > >> > > > > def sit(req): > > >> > > > > time.sleep(3) > > >> > > > > return http.HttpResponse('foo') > > > > >> > > > > errors are: > > > > >> > > > > 03-01 04:15PM 48.177 /sit/91 500 10019ms 0cpu_ms 0kb gzip(gfe) > > >> > > > > 153.90.236.210 - - [01/Mar/2010:16:15:58 -0800] "GET /sit/91 > > >> > HTTP/1.1" > > >> > > > > 500 0 - "gzip(gfe)" ".appspot.com" > > >> > > > > W 03-01 04:15PM 58.197 > > >> > > > > Request was aborted after waiting too long to attempt to > service > > >> > > > > your > > >> > > > > request. Most likely, this indicates that you have reached > your > > >> > > > > simultaneous dynamic request limit. This is almost always due > to > > >> > > > > excessively high latency in your app. Please seehttp:// > > >> > > > code.google.com/appengine/docs/quotas.htmlfor more details. > > > > >> > > > > On Mar 1, 2:36 pm, Michael Wesner <[email protected] > > > > >> > wrote: > > > > >> > > > > > Correction/addition to my last email. > > > > >> > > > > > It turns out that our requests for this EC2 pull thing are > > >> > > > > > actually > > >> > > > much faster now. Gary and our other devs have reworked it. I > need > > >> > updated > > >> > > > numbers, but they don't take 10s, probably more like 2s. We > still > > >> > > > have > > >> > some > > >> > > > heavy ~5s services though, so the same issue exists with the > > >> > > > simul-req > > >> > > > stuff, just to less extent. We don't actually hit this limit > much > > >> > > > now > > >> > with > > >> > > > the current beta that is in production, but it is low traffic at > the > > >> > moment. > > >> > > > We are just getting ready to ramp up heavily. > > > > >> > > > > > I asked Nick what we should do, well just today after my > last > > >> > email, I > > >> > > > have made contact with a Developer Advocate and whatnot, which > is > > >> > fantastic. > > >> > > > It looks like we, as a business, will be able to have better > > >> > > > contact > > >> > with > > >> > > > the GAE team. We would very much like to continue working with > you > > >> > > > to > > >> > figure > > >> > > > out what actions we can take and what provisioning we can do to > make > > >> > our > > >> > > > product successful and scale it as we grow in the near future. > Gary > > >> > Orser > > >> > > > will be replying to this thread soon with more findings from > both > > >> > > > our > > >> > real > > >> > > > app code and a little test app we are using and which he will > share > > >> > with > > >> > > > you. > > > > >> > > > > > We plan on having a presence at Google I/O this year as we > did > > >> > > > > > at > > >> > > > PyCon. Hopefully we can even get setup in the demonstration > area at > > >> > I/O. > > > > >> > > > > > Thanks Nick for your help. Could we possibly setup a quick > > >> > > > > > skype > > >> > conf > > >> > > > call at some point? > > > > >> > > > > > -Mike Wesner > > > > >> > > > > > On Mar 1, 2010, at 1:13 PM, Michael Wesner wrote: > > > > >> > > > > > > Nick, > > > > >> > > > > > > If we (I work with Gary) require fairly heavy requests > which > > >> > > > > > > run > > >> > for > > >> > > > multiple seconds then it is not possible to get anywhere near > 400 > > >> > > > QPS. > > >> > The > > >> > > > math used on the docs page only applies to 75ms requests. > > > > >> > > > > > > (1000 ms/second / 75 ms/request) * 30 = 400 > requests/second > > > > >> > > > > > > so lets say each request takes 10 seconds (and ours, > pulling > > >> > > > > > > data > > >> > to > > >> > > > EC2 for a heavy operation that we can't do on GAE could take > that > > >> > > > much > > >> > since > > >> > > > we have to process and update some XML before sending it) > > > > >> > > > > > > (1000 ms/second / 10000 ms/request) * 30 = 3 > requests/second > > > > >> > > > > > > And that does not even take into account all the other > traffic > > >> > > > > > > to > > >> > our > > >> > > > application, nor the fact that many users could be doing this > same > > >> > heavy > > >> > > > operation at the same time. Our application will see spikes in > this > > >> > type of > > >> > > > activity also. The docs also mention that CPU heavy apps incur > > >> > penalties, > > >> > > > which is vague and scary. > > > > >> > > > > > > Great effort is put into doing things in the most > efficient > > >> > > > > > > way > > >> > > > possible, but not everyones apps can do everything in 75ms. Most > all > > >> > > > of > > >> > our > > >> > > > service calls are under 250ms. We do have a little overhead from > our > > >> > > > framework which we are constantly working on improving. Our > > >> > application is > > >> > > > AMF service/object based which is inherently heavy compared to > > >> > > > simple > > >> > web > > >> > > > requests. It limits the amount of memcache work we can do also, > but > > >> > > > we > > >> > are > > >> > > > also working on improving our use of that. > > > > >> > > > > > > We easily hit these boundaries during testing so I think > we > > >> > really > > >> > > > need much higher simultaneous dynamic request limits for not > only > > >> > > > our > > >> > > > production instance but our dev/qa instances so we can test and > load > > >> > them to > > >> > > > some degree. Our QA team could easily bust this limit 20 times > > >> > > > over. > > > > >> > > > > > > So, Nick Johnson... I ask your advice. We are running a > > >> > > > company/product on GAE. We are more than happy to pay for > > >> > > > quota/service/extra assistance in these matters. What do you > suggest > > >> > > > we > > >> > do? > > > > >> > > > > > > I should also mention that I spoke with Brett Slatkin at > PyCon > > >> > and he > > >> > > > is now at least semi-familiar with the scope of product we have > > >> > developed. > > >> > > > I have exchanged contact info with him but have not heard > anything > > >> > back > > >> > > > from him yet. We would really appreciate contact or even a > brief > > >> > meeting at > > >> > > > some point (in person or otherwise). > > > > >> > > > > > > Thanks, > > > > >> > > > > > > -Mike Wesner > > > > >> > > > > > > On Mar 1, 2010, at 7:40 AM, Nick Johnson (Google) wrote: > > > > >> > > > > > >> Hi Gary, > > > > >> > > > > > >> Practically speaking, for an app that hasn't been given > > >> > > > > > >> elevated > > >> > > > permissions, you should be able to have at least 30 concurrent > > >> > > > requests > > >> > - > > >> > > > equating to around 400 QPS if your app is fairly efficient. What > > >> > problems > > >> > > > are you running into that lead you to conclude you're hitting a > > >> > > > limit > > >> > at 4 > > >> > > > QPS, and that the problem is at App Engine's end? > > > > ... > > > > read more ยป > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
