I would suggest enabling wire-level debugging to get a clearer picture of what is taking place (see http://hc.apache.org/httpcomponents-client-ga/logging.html)
That said, the symptoms lead me to think that you're running into Java's DNS caching behaviour. Amazon elastic load balancers frequently change IP address, and have a low TTL in DNS accordingly. Depending on your Java version and configuration, you may be caching DNS records indefinitely. If this was the case, then your JVM may have cached an old IP address for your Amazon ELB and HttpClient is dutifully trying to connect to this old IP address that is no longer running your remote service. You could verify this on your production server (in a non-invasive way) by running tcpdump to look at what IP address your application is trying to connect to. Java 6 DNS options are detailed at http://docs.oracle.com/javase/6/docs/technotes/guides/net/properties.html (networkaddress.cache.ttl is the property in particular I'm referring to). Hope this helps, Sam On 7 May 2012 19:42, Eugene Dvorkin <[email protected]> wrote: > I have a client application that makes http calls to another servers > using http client library (httpclient-4.1.3.jar) . Application was > deployed to 5 servers and it was working for a while. All those 5 > servers are behind load balancer and firewall in data center. This > application is a web application using Spring framework. The end point, > to where the application is making http requests, are deployed on Amazon > EC2 instances behind amazon load balancer. > Today I start experience problems: > > 12/05/07 10:03:48 java.net.ConnectException: Connection timed out > 12/05/07 10:03:48 at java.net.PlainSocketImpl.socketConnect(Native > Method) > 12/05/07 10:03:48 at > java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333) > 12/05/07 10:03:48 at > java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195) > 12/05/07 10:03:48 at > java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182) > 12/05/07 10:03:48 at > java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) > 12/05/07 10:03:48 at java.net.Socket.connect(Socket.java:507) > 12/05/07 10:03:48 at java.net.Socket.connect(Socket.java:457) > 12/05/07 10:03:48 at java.net.Socket.<init>(Socket.java:365) > 12/05/07 10:03:48 at java.net.Socket.<init>(Socket.java:238) > 12/05/07 10:03:48 at > org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java > :79) > 12/05/07 10:03:48 at > org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java > :121) > 12/05/07 10:03:48 at > org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:706) > 12/05/07 10:03:48 at > org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386) > 12/05/07 10:03:48 at > org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170) > 12/05/07 10:03:48 at > org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396) > 12/05/07 10:03:48 at > org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324) > 12/05/07 10:03:48 at > org.artstor.security.acegi.ui.SamlAuthenticationProcessingFilter.makeHttpsRequest(SamlAuthenticationProcessingFilt > er.java:377) > > On receiving end, I don't see any log in Apache access logs. > If I do curl to the same URL, It works fine. Nobody changed any > configuration or anything like that. > How can I drill down and debug the issue? It makes more complicated > because it is on production server but this functionality is not > released yet. > Any ideas? > httpclient-4.1.3.jar > httpclient-cache-4.1.3.jar > httpcore-4.1.4.jar > httpmime-4.1.3.jar > > > Our code: > public byte[] makeHttpsRequest(String url) { > byte[] responseBody = null; > > > HttpClient httpClient=new HttpClient(); > > HttpMethod method = new GetMethod(url); > > HttpMethodRetryHandler myretryhandler = new > HttpMethodRetryHandler() { > public boolean retryMethod( > final HttpMethod method, > final IOException exception, > int executionCount) { > if (executionCount >= 3) { > // Do not retry if over max retry count > return false; > } > if (exception instanceof NoHttpResponseException) { > // Retry if the server dropped connection on us > return true; > } > if (exception instanceof IOException) { > // Retry if read timeout happens > return true; > } > if (!method.isRequestSent()) { > // Retry if the request has not been sent fully or > // if it's OK to retry methods that have been sent > return true; > } > // otherwise do not retry > return false; > } > }; > > > > > // Provide custom retry handler is necessary > > method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER,myretryhandler); > > try { > HttpClientParams param=new HttpClientParams(); > param.setSoTimeout(3000); > httpClient.setParams(param); > method.addRequestHeader("Accept", > "text/html,application/xhtml+xml,application/xml,application/json"); > // Provide custom retry handler is necessary > > method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, > new DefaultHttpMethodRetryHandler(3, false)); > > int statusCode = httpClient.executeMethod(method); > if (statusCode != HttpStatus.SC_OK) { > System.err.println("Method failed: " + > method.getStatusLine()); > } > > > > responseBody = method.getResponseBody(); > > } catch (SocketTimeoutException e) { > > System.err.println("can't connect to shibboleth server: " > + e.getMessage()); > e.printStackTrace(); > } catch (IOException e) { > > System.err.println("can't connect to shibboleth server: " > + e.getMessage()); > e.printStackTrace(); > > > > } finally { > // When HttpClient instance is no longer needed, > // shut down the connection manager to ensure > // immediate deallocation of all system resources > > method.releaseConnection(); > } > > > return responseBody; > } > > > > > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
