In my recent work to stabilize the tests after the Jetty 9 upgrade I struggled
with a number of issues. The most insidious of which was perhaps caused by
what must be some poor configuration on one of the Jenkins slaves ubuntu3 that
results in very poor performance during (ie hangs) reverse DNS lookups of
0.0.0.0 in particular.
The offending Knox code is/was in
gateway-provider-rewrite/src/main/java/org/apache/hadoop/gateway/filter/rewrite/impl/UrlRewriteResponse.java
Note that this is the previous code. The code in master not is much more
complex in support of a timeout that I'm not sure we should keep.
More on that later.
// KNOX-464: Doing this because Jetty only returns the string version of the IP
address for request.getLocalName().
// Hopefully the local hostname will be cached so this will not be a
significant performance hit.
// Previously this was an inline request.getServerName() but this ended up
mixing the hostname from the Host header
// and the local port which was making load balancer configuration difficult if
not impossible.
private String getRequestLocalHostName() {
String hostName = request.getLocalName();
try {
hostName = InetAddress.getByName( hostName ).getHostName();
} catch( UnknownHostException e ) {
// Ignore it and use the original hostname.
}
return hostName;
}
The other half of the puzzle is the use of Jetty's ServletTester. It turns out
that from Jetty 8 to 9 this went through some non-trivial changes. The bottom
line is that by default it now only waits 5 seconds for a request to complete.
It just to wait indefinitely. So between these two things this has exposed
this potential issue.
Here BTW is a reference to one of the tests that uses Jetty's ServletTester.
gateway-provider-rewrite/src/test/java/org/apache/hadoop/gateway/filter/rewrite/api/UrlRewriteServletFilterTest.java
So there is a choice to be made.
A) Should we continue down the path I used as a workaround to get the tests to
pass in master and try an compensate for Java not having terrible timeout
behavior on DNS reverse lookups?
B) Should we increase the timeout in the tests from the default of 5 seconds to
something like 30 seconds and put the DNS reverse lookup code back the way it
was?
response = HttpTester.parseResponse( server.getResponses( request.generate(),
30, TimeUnit.SECONDS ) );
C) Should we be solving KNOX-464 in some other way that does not involve a DNS
lookup at all?
