Oleg, You saved my day. Thank you very much. Zhaohua Meng WebSphere Consultant
"Oleg Kalnichevski" <[EMAIL PROTECTED]> 11/03/2005 04:13 AM Please respond to "Jakarta Commons Users List" <[email protected]> To [email protected] cc Subject Re: HttpClient: can it cause "too many open files" ? On Wed, Nov 02, 2005 at 06:03:03PM -0500, Zhaohua Meng wrote: > HttpClient gugus, > > > I have a HttpClientEngine (see code listing at the bottom ) which wraps > commons HttpClient. It is used as following in my code deployed on SuSe > Linux. > > for (int i=0; i<requests.length; i++) } > HttpClientEngine engine = new HttpClientEngine (); > response = engine.doPost(requests[i]); > //code to process response > } > > In real situation t he request loop can be as big as 500+. After the code > running for a while I got "too many open files" exception. I know in Linux > a stream is considered a file. Please note that > "method.releaseConnection()" is called before doPost() returns a response > object. My questions are following: > > 1. Will the following code work? In other words, can HttpClient object be > reused to post with different HttpBaseMethod? > > HttpClientEngine engine = new HttpClientEngine (); > for (int i=0; i<requests.length; i++) } > response = engine.doPost(requests[i]); > //code to process response > } > Actually it is strongly recommended to reuse the HttpClient instance. For details see the HttpClient optimization guide: http://jakarta.apache.org/commons/httpclient/performance.html > 2. Will my code leave some streams open thus causes "too many open files"? > Yes it will. HttpMethod#releaseConnection() call does not close the underlying socket if the connection can be kept alive. This method merely returns connection back to the connection manager > > 3. Besides "method.releaseConnection()", what other methods should I call > to close all streams? > MultiThreadedHttpConnectionManager#shutdown() SimpleHttpConnectionManager#dhutdown() > Thanks, > Zhaohua > > > public class HttpClientEngine { > private HttpClient httpClient; //this is the commons HttpClient > private HttpResponse response; //my proprietary class > private HttpRequest request; //my proprietary class > private URL url; > private Credentials credentials; > public HttpClientEngine() { > httpClient = new HttpClient(); > } > public HttpResponse doPost(HttpRequest request) { > return > doPost(request,request.getURL(),request.getCredentials()); > } > public HttpResponse doPost(HttpRequest request, URL url) { > return doPost(request,url,request.getCredentials()); > } > private static Logger logger = Logger.getInstance(); > public HttpResponse doPost(HttpRequest request, URL url, > Credentials credentials) { > this.request = request; > this.url = url; > this.credentials = credentials; > return service(request,new PostMethod(url.toString())); > } > > public HttpResponse doGet(HttpRequest request) { > return > doGet(request,request.getURL(),request.getCredentials()); > } > public HttpResponse doGet(HttpRequest request, URL url) { > return doGet(request,url,request.getCredentials()); > } > public HttpResponse doGet(HttpRequest request, URL url, > Credentials credentials) { > this.request = request; > this.url = url; > this.credentials = credentials; > return service(request,new GetMethod(url.toString())); > } > > > private HttpResponse service(HttpRequest request, HttpMethodBase > method) { > HttpHeader[] requestHeaders = request.getHeaders(); > if ( requestHeaders != null) { > for (int i=0; i<requestHeaders.length; i++) { > //debug("header: > ["+requestHeaders[i].getName()+"]["+requestHeaders[i].getValue()+"]"); > > method.setRequestHeader(requestHeaders[i].getName(),requestHeaders[i].getValue()); > } > } > if (credentials != null) { > UsernamePasswordCredentials login = > new > UsernamePasswordCredentials(credentials.getName(),credentials.getPassword()); > > AuthScope auth= new AuthScope(url.getHost(), > url.getPort(), null); > httpClient.getState().setCredentials(auth, login); > method.setDoAuthentication(true); > } > if (method instanceof PostMethod) { > try { > ((PostMethod)method).setRequestEntity(new > ByteArrayRequestEntity(request.getBody())); > } catch (Exception e) { > method.releaseConnection(); > return null; > } > } > try { > int statusCode = httpClient.executeMethod(method); > Header[] responseHeaders = > method.getResponseHeaders(); > HttpHeader[] headers = null; > if (responseHeaders != null) { > headers = new > HttpHeader[responseHeaders.length]; > for (int i=0; i<responseHeaders.length; > i++) { > headers[i] = new HttpHeader(); > headers[i].setName(responseHeaders[i].getName()); > headers[i].setValue(responseHeaders[i].getValue()); > } > } > > response = > handler.process(method.getResponseBody(),statusCode, headers); > } catch (HttpException e) { > throw new RuntimeException(e.getMessage()); > } catch (IOException e) { > throw new RuntimeException(e.getMessage()); > } finally { > method.releaseConnection(); > } > return response; > } > private HttpResponseHandler handler; > public void setHttpResponseHandler(HttpResponseHandler handler) { > this.handler = handler; > } > public static void debug(Object obj) { > System.out.println(HttpClientEngine.class.getName()+"::"+obj.toString()); > } > > Zhaohua Meng > WebSphere Consultant > This message is being sent from a law firm and may contain confidential or privileged information. If you are not the intended recipient, please advise the sender immediately by reply e-mail and delete this message and any attachments without retaining a copy. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This message is being sent from a law firm and may contain confidential or privileged information. If you are not the intended recipient, please advise the sender immediately by reply e-mail and delete this message and any attachments without retaining a copy.
