Hi there,I have recently discovered a problem with McAfee Security Center that interferes with HttpClient. When downloading a file from a web server, it appears that HTTPClient is downloading the entire request while the executeMethod() call is in progress, rather than quickly establishing a connection and returning an InputStream from getResponseBodyAsStream. This means in my application, where I want to be streaming sound files, that the entire file has to download before I can start reading any part of it. My investigation shows that this seems to be a feature that only applies to port 80: SSL on 443 and plain HTTP on other ports (e.g. 9000) seem to be unaffected.
Originally I thought this was a problem with HttpClient, and spent some time on Google looking for things like 'executeMethod loads all data' or 'HTTPClient buffers entire request', because that's what I thought was happening. Instead, if you go into the Services administrative tool (run services.msc) and find the McAfee Redirector service and disable it, then restart (as your internet will be stuffed), you should see a return to normal behaviour. In looking around for solutions, it seems it might be possible to add a special port to the Ignore list under HKEY_LOCAL_MACHINE\SOFTWARE\McAfee\Redirector\Ports, but I think it's a hard sell to tell customers they need to mess with their antivirus to use your product.
So I'm hoping someone knows some alternatives! Things I tried with HTTPClient were setting the socket receive buffers to a small size and turning on the tcp.nodelay flag, neither of which had any effect.
In the interests of making this a useful hit in the archives, here's the tester script that demonstrates the bug:
--- JAVA
import java.io.IOException;
import java.io.InputStream;
import java.text.MessageFormat;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.HttpMethod;
import org.apache.commons.httpclient.methods.GetMethod;
public class HttpClientTest {
public static void main(String[] args) {
// Very simple code to fetch a resource and capture some timing
information
long times[] = new long[16384];
String tasks[] = new String[times.length];
int log_count = 0;
tasks[log_count] = "Start";
times[log_count++] = System.currentTimeMillis();
HttpClient client = new HttpClient();
HttpMethod get = new
GetMethod("http://home.zip-it.org:9000/stream.mp3";);
try {
tasks[log_count] = "Executing GET method";
times[log_count++] = System.currentTimeMillis();
client.executeMethod(get);
tasks[log_count] = "GET method complete";
times[log_count++] = System.currentTimeMillis();
tasks[log_count] = "Status code " + get.getStatusCode() + ":
" + get.getStatusText();
times[log_count++] = System.currentTimeMillis();
tasks[log_count] = "Retrieving response body";
times[log_count++] = System.currentTimeMillis();
InputStream in = get.getResponseBodyAsStream();
tasks[log_count] = "Reading response body";
times[log_count++] = System.currentTimeMillis();
byte buffer[] = new byte[65536];
int bytes;
while((bytes = in.read(buffer)) > 0) {
tasks[log_count] = "Reading " + bytes + " bytes";
times[log_count++] = System.currentTimeMillis();
if(log_count > 100)
break;
}
in.close();
tasks[log_count] = "Closed";
times[log_count++] = System.currentTimeMillis();
} catch (HttpException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
tasks[log_count] = "Finish";
times[log_count++] = System.currentTimeMillis();
for(int i = 0; i < log_count; ++i) {
Double offset = new Double(((double)(times[i] - times[0])) /
1000.0);
System.out.println(MessageFormat.format("{0,number,0.000}:
{1}", new Object[] { offset, tasks[i] }));
}
}
}
--- JAVA
On my machine, here's a sample of the output for a small (600K) file,
being downloaded at 60K/s:
--- OUTPUT -- with filtering 0.000: Start 0.234: Executing GET method 11.359: GET method complete 11.359: Status code 200: OK 11.359: Retrieving response body 11.359: Reading response body 11.359: Reading 65536 bytes 11.359: Reading 65536 bytes 11.359: Reading 65536 bytes 11.375: Reading 65536 bytes 11.375: Reading 28174 bytes 11.375: Reading 17520 bytes 11.375: Reading 14600 bytes 11.375: Reading 2920 bytes 11.375: Reading 14600 bytes 11.375: Reading 17520 bytes (-- this line repeats x19) 11.375: Reading 15226 bytes 11.375: Closed 11.375: FinishThere is an obvious pause while the file is downloaded by McAfee and then the data arrives all at once, and in bigger chunks than expected for the socket buffer. When downloading the file on my linux box, the result is much more like the next example, which was generated on my Windows box, for the same file as above, on the same server, but where the URL was changed to https:
0.000: Start 0.110: Executing GET method 1.750: GET method complete 1.750: Status code 200: OK 1.750: Retrieving response body 1.750: Reading response body 1.750: Reading 3874 bytes 2.438: Reading 16384 bytes 2.719: Reading 12510 bytes 2.953: Reading 16384 bytes 3.203: Reading 16384 bytes 3.453: Reading 16384 bytes 3.703: Reading 16384 bytes 4.125: Reading 16384 bytes 4.125: Reading 16384 bytes 4.438: Reading 16384 bytes 4.610: Reading 16384 bytes 4.860: Reading 16384 bytes 5.172: Reading 16384 bytes 5.344: Reading 16384 bytes 5.594: Reading 16384 bytes 5.907: Reading 16384 bytes 6.078: Reading 16384 bytes 6.328: Reading 16384 bytes 6.391: Reading 16384 bytes 6.641: Reading 16384 bytes 7.063: Reading 16384 bytes 7.297: Reading 16384 bytes 7.547: Reading 16384 bytes 7.797: Reading 16384 bytes 8.047: Reading 16384 bytes 8.282: Reading 16384 bytes 8.344: Reading 16384 bytes 8.594: Reading 16384 bytes 9.032: Reading 16384 bytes 9.094: Reading 16384 bytes 9.328: Reading 16384 bytes 9.766: Reading 16384 bytes 9.828: Reading 16384 bytes 10.078: Reading 16384 bytes 10.250: Reading 16384 bytes 10.500: Reading 16384 bytes 10.813: Reading 16384 bytes 11.000: Reading 16384 bytes 11.297: Reading 16384 bytes 11.719: Reading 16384 bytes 11.953: Reading 16384 bytes 12.219: Reading 16384 bytes 12.219: Reading 16320 bytes 12.219: Closed 12.219: FinishSo clearly there is not a problem with HTTPClient, or Java sockets, or anything else except McAfee Internet Security.
I hope this information saves someone some time in the future. cheers, Will
smime.p7s
Description: S/MIME Cryptographic Signature
