bump On Mar 3, 7:23 pm, Eli Jones <[email protected]> wrote: > Heh, yes.. I printed out your Java and it seemed perfectly fine after > reading through it several times... But Java is just wayy to busy > looking for my tastes. > > Good luck getting some clarification on what's happening with this. > > On 3/3/10, Gary Orser <[email protected]> wrote: > > > > > Eli, > > > Python as usual is sooo much more elegant than java. > > > I confirmed your test results on a billing enabled appspot. > > > Cheers, Gary > > > On Mar 2, 9:57 pm, Eli Jones <[email protected]> wrote: > >> I did my own testing in python. And, I definitely couldn't get 30 > >> simultaneous requests to work. > > >> This was done on a test app that does not have Billing Enabled.. so not > >> sure > >> if that affects the Dynamic Request Limit. > > >> Either way, it seems that for this test, the effective Limit was 16 > >> simultaneous accesses. As soon as I modified my code to try 17, it > >> started > >> throwing a few Dynamic Request Limit errors.. though, the Logs page > >> AppEngine just refers to it as another Request (with code 500 and a time > >> of > >> 10 seconds). > > >> Here is the code for the handler.. after receiving a GET, it sleeps for 3 > >> seconds and the responds with a 'hello': > > >> from google.appengine.ext import webapp > >> from google.appengine.ext.webapp.util import run_wsgi_app > >> import time > > >> class meTest(webapp.RequestHandler): > >> def get(self): > >> time.sleep(3) > >> meid = self.request.get('id') > >> self.response.out.write('hi! foo%s'%meid) > > >> application = webapp.WSGIApplication([('/test/meTest',meTest)], > >> debug = True) > > >> def main(): > >> run_wsgi_app(application) > > >> if __name__ == "__main__": > >> main() > > >> And here is they python code I am running on my local machine to test. It > >> fires off 17 threads. Each thread requests a GET from the handler and > >> prints the response, three times in a row: > > >> import httplib > >> import threading > > >> class meThread(threading.Thread): > >> def run(self): > >> for i in range(3): > >> conn = httplib.HTTPConnection('datastoretester.appspot.com') > >> conn.request("GET","/test/meTest?id=%s" % self.getName()) > >> response = conn.getresponse() > >> print response.read() > >> conn.close() > > >> for i in range(17): > >> meThread(name=str(i+1)).start() > > >> As soon as I scale the range back to (16), no more Dynamic Request Error. > > >> Special notes: I am not sure if there is some sort of built in > >> rate-limiting for requests from the same IP address.. but when sending 17 > >> threads at the handler.. it is not responding in 3 seconds to their GETs.. > >> It takes about 11 seconds. > > >> If you look in the Logs, for a run with 17 threads, you see this: > > >> 03-02 08:31PM 00.384 /test/meTest?id=16 200 12855ms 19cpu_ms 0kb gzip(gfe) > >> 03-02 08:31PM 01.266 /test/meTest?id=7 200 11973ms 19cpu_ms 0kb gzip(gfe) > >> 03-02 08:31PM 01.283 /test/meTest?id=15 500 10017ms 0cpu_ms 0kb gzip(gfe) > >> 03-02 08:30PM 58.280 /test/meTest?id=4 200 11952ms 0cpu_ms 0kb gzip(gfe) > > >> (Notice the 500 error in there as well.. that's the one that mentions the > >> Dynamic Request Limit). > > >> Now, if I run the test with only 4 threads, it looks nice and quick like > >> this: > > >> 03-02 08:35PM 25.602 /test/meTest?id=1 200 3009ms 0cpu_ms 0kb gzip(gfe) > >> 03-02 08:35PM 22.637 /test/meTest?id=2 200 3008ms 0cpu_ms 0kb gzip(gfe) > >> 03-02 08:35PM 22.623 /test/meTest?id=3 200 3009ms 0cpu_ms 0kb gzip(gfe) > >> 03-02 08:35PM 22.603 /test/meTest?id=4 200 3008ms 0cpu_ms 0kb gzip(gfe) > > >> Once I start testing with more than 4 threads, it starts to slow down in > >> its > >> response time... > > >> So, I'd guess something would need to be clarified.. is there some > >> internal > >> limiting going on per ip address? Does a "long" running process have a > >> lower simultaneous request limit? > > >> On Tue, Mar 2, 2010 at 12:21 PM, Gary Orser <[email protected]> wrote: > >> > Eli, > > >> > You have the python request server. > >> > Here is the java client: > >> > You'll have to get the libraries yourself. > > >> > Cheers, Gary > > >> > import java.util.ArrayList; > >> > import java.util.concurrent.Callable; > >> > import java.util.concurrent.ExecutionException; > >> > import java.util.concurrent.ExecutorService; > >> > import java.util.concurrent.Executors; > >> > import java.util.concurrent.Future; > > >> > import org.apache.commons.io.IOUtils; > >> > import org.apache.http.HttpHost; > >> > import org.apache.http.HttpResponse; > >> > import org.apache.http.HttpVersion; > >> > import org.apache.http.client.HttpClient; > >> > import org.apache.http.client.methods.HttpGet; > >> > import org.apache.http.conn.ClientConnectionManager; > >> > import org.apache.http.conn.params.ConnManagerParams; > >> > import org.apache.http.conn.params.ConnPerRouteBean; > >> > import org.apache.http.conn.scheme.PlainSocketFactory; > >> > import org.apache.http.conn.scheme.Scheme; > >> > import org.apache.http.conn.scheme.SchemeRegistry; > >> > import org.apache.http.conn.ssl.SSLSocketFactory; > >> > import org.apache.http.impl.client.DefaultHttpClient; > >> > import org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager; > >> > import org.apache.http.params.BasicHttpParams; > >> > import org.apache.http.params.HttpParams; > >> > import org.apache.http.params.HttpProtocolParams; > > >> > import org.apache.commons.logging.Log; > >> > import org.apache.commons.logging.LogFactory; > > >> > public class Main > >> > { > >> > private Log log = LogFactory.getLog(Main.class); > >> > // ADJUST: number of threads to make requests on > >> > public static int NUM_PARALLEL_SECTION_REQUESTS = 20; > >> > public static HttpParams httpParams = new BasicHttpParams(); > >> > { > >> > httpParams.setBooleanParameter("http.protocol.expect- > >> > continue", false); > >> > // ADJUST: if this is included, will use 8888 as a > >> > proxy port. Charles Proxy defaults to this port. > >> > //httpParams.setParameter("http.route.default-proxy", > >> > new HttpHost("localhost", 8888)); > >> > } > > >> > protected class GetSection implements Callable<String> > >> > { > >> > protected int index; > >> > protected HttpClient client; > >> > protected String URL; > >> > public GetSection(int index, HttpClient client, String > >> > URL) > >> > { > >> > this.index = index; > >> > this.client = client; > >> > this.URL = URL; } > >> > public String call() throws Exception > >> > { > >> > HttpGet getSection = new > >> > HttpGet(URL); HttpResponse respSection = > >> > client.execute(getSection); > >> > String foo = > >> > IOUtils.toString(respSection.getEntity().getContent(), "UTF-8"); > >> > return foo; > >> > } } > > >> > public static void main(String[] args) throws Exception > >> > { new Main().maint(args); > >> > } > > >> > public void maint(String[] args) throws Exception { > >> > SchemeRegistry schemeRegistry = new SchemeRegistry(); > >> > schemeRegistry.register(new Scheme("http", > >> > PlainSocketFactory.getSocketFactory(), 80)); > >> > schemeRegistry.register(new Scheme("https", > >> > SSLSocketFactory.getSocketFactory(), 443)); > >> > HttpParams params = new BasicHttpParams(); > >> > ConnManagerParams.setMaxTotalConnections(params, > >> > NUM_PARALLEL_SECTION_REQUESTS); > >> > ConnManagerParams.setMaxConnectionsPerRoute(params, > >> > new ConnPerRouteBean(NUM_PARALLEL_SECTION_REQUESTS)); > >> > HttpProtocolParams.setVersion(params, > >> > HttpVersion.HTTP_1_1); > >> > ClientConnectionManager cm = new > >> > ThreadSafeClientConnManager(params, schemeRegistry); > >> > HttpClient client = new DefaultHttpClient(cm, > >> > httpParams); > > >> > ExecutorService es = > >> > Executors.newFixedThreadPool(NUM_PARALLEL_SECTION_REQUESTS); > >> > // ADJUST: total number of requests to make. > >> > int numSections = 100; > >> > ArrayList<Future<String>> futures = new > >> > ArrayList<Future<String>>(numSections); > >> > log.info("queuing requests"); > >> > for (int i = 0; i < numSections; i++) > >> > { > >> > // ADJUST: set a real hostname here > >> > futures.add(es.submit(new GetSection(i, > >> > client, "http://yourappid.appspot.com/sit/"; + Integer.toString(i)))); > >> > // ADJUST: stagger initial requests with this > >> > sleep > >> > //Thread.sleep(200); > >> > } > > >> > es.shutdown(); > > >> > log.info("waiting for thread pool to finish"); > >> > while (!es.isTerminated()) > >> > Thread.sleep(500); > > >> > log.info("all requests queued"); > > >> > try > >> > { > >> > for (Future<String> future: futures) > >> > future.get(); > >> > log.info("got all futures"); > >> > > > ... > > read more »
-- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
