On Tue, 2008-06-03 at 21:42 +0800, maomaode wrote: > Hi Oleg, > > Thanks for the quick response, I'm dev an intranet web > crawling/indexing, previous I used the httpclient 3-x. > we use the nio feature in the new httpclient to increase the performance. >
Please note NIO is actually _significantly_ slow compared to classic I/O _unless_ you are dealing with thousands of simultaneous high latency connections. If the number of simultaneous connections is less than 500 NIO is almost certainly bound to perform worse than the classic blocking I/O. > I don't know how much of simultaneous conn httpclient-nio can handle? > for me, I hope as much as the system can handle. > Really a lot. BUT every connection usually takes from 100 to 150KB, so you have to ensure there is enough memory of the heap to support a great number of simultaneous connections. > I don't know the max length of the content in our system, I guess 300KB max? > > Since the ThrottlingHttpClientHandler can use the ThreadPool, I guess > the workerCount in the DefaultConnectingIOReactor will take NO effect, > is that right? > Pretty much. But I would still recommend to use the same number of I/O dispatch workers as CPU cores on the system to ensure better CPU utilization. Could you please create a test app that reproduces the problem you have been seeing and post it to this list or to the dev list? Cheers Oleg > > Thanks, > James > > http://people.apache.org/~mmao > > > > On Tue, 2008-06-03 at 20:48 +0800, maomaode wrote: > > > >> Hi, > >> > >> > >> I'm hitting a Out Of Memory error when using ThrottlingHttpClientHandler > >> <http://hc.apache.org/httpcomponents-core/httpcore-nio/apidocs/org/apache/http/nio/protocol/ThrottlingHttpClientHandler.html> > >> > >> I gave a Executors.|*newCachedThreadPool > >> <cid:[email protected]>*(), should I give a fixed thread > >> pool? > >> > >> > > > > James, > > > > What is it exactly you are trying to do? How many simultaneous > > connections are you trying to open? What is the size of the content > > buffer? it is 20480 (default)? > > > > > > > >> And BTW, what's the workerCount means in the |DefaultConnectingIOReactor? > >> > >> > > > > The number of threads I/O reactor uses to dispatch I/O events to the > > protocol handlers. This number should be equal to that of CPU cores of > > the system. > > > > Hope this helps > > > > Oleg > > > > > > > >> Thanks in advance, > >> James > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [EMAIL PROTECTED] > >> For additional commands, e-mail: [EMAIL PROTECTED] > >> > >> > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
