Yes, it appears that the server is the cause. I'm not sure how familiar you are with packet captures, but I've shown the final four packets in the exchange at http://pastebin.com/EDE4KhZr
You'll see that www.ncbi.nlm.nih.gov sends a TCP FIN on line 5, and that the chunked response is ended within the same packet on line 57 (the "0" indicates the end of a chunked response). The client (curl, on 192.168.2.100) correctly responds with a FIN as well, and then the connection is closed. The TCP FIN occurred almost exactly 7 minutes after the HTTP request first began. I strongly suspect that the server side script that's processing this data has a maximum execution time of 7 minutes set. Hope this helps, Sam On 19 August 2011 16:32, Sam Crawford <[email protected]> wrote: > Hrmmm, I've not run your Java code, but I think the server may be at > fault here. I'm seeing the response being cut short in curl as well. > It takes a very long time to complete the transaction, but I'm running > it again with a packet capture right now so should be able to see what > is taking place shortly. > > > On 19 August 2011 15:58, Mark Southern <[email protected]> wrote: >> Further details from the HttpResponse Entity: >> >> Response encoding: null >> Response content length: -1 >> Chunked?: true >> Content-Type: text/xml; charset=UTF-8 >> >> >> -----Original Message----- >> From: kim young ill [mailto:[email protected]] >> Sent: Friday, August 19, 2011 10:02 AM >> To: HttpClient User Discussion >> Subject: Re: Truncated output with HttpPost but works with cURL >> >> could be that response is gzipped without specifying content-encoding which >> leads to wrong content-length ? >> >> On Fri, Aug 19, 2011 at 3:32 PM, Mark Southern <[email protected]> wrote: >> >>> Dear Peter, >>> >>> Unfortunately that's not it. The problem occurs whether or not I >>> include the step to dump to the local file (and also with an explicit >>> flush/close added to the BufferedOutputStream). >>> >>> I have made available the java source, ids text file and cUrl example >>> if case anyone cares for a further look: >>> >>> http://chemutils.florida.scripps.edu/httpclient/curl.sh >>> http://chemutils.florida.scripps.edu/httpclient/ids.txt >>> http://chemutils.florida.scripps.edu/httpclient/HttpClientTest.java >>> >>> Thank you, >>> ~Mark. >>> >>> -----Original Message----- >>> From: Pete Keyes [mailto:] >>> Sent: Thursday, August 18, 2011 9:29 PM >>> To: HttpClient User Discussion >>> Subject: RE: Truncated output with HttpPost but works with cURL >>> >>> This is just a guess, but I don't see where you've closed the >>> BufferedOutputStream passed to the writeTo() method. >>> >>> ...Pete Keyes >>> Starbucks Coffee Co. >>> >>> ________________________________________ >>> From: Mark Southern [[email protected]] >>> Sent: Thursday, August 18, 2011 05:48 PM >>> To: [email protected] >>> Subject: Truncated output with HttpPost but works with cURL >>> >>> To whomever can help me! >>> >>> I am submitting to a web service and receiving an xml file in return. >>> I am using POST b/c this is too large for a GET. If I use cURL then this >>> works. >>> However with HttpClient (4.1.2) and the same parameters, I am getting >>> a truncated xml file returned. >>> >>> Does anyone have an idea what might be wrong? I include the test file >>> but not the whole list of parameters I am submitting as this is a further >>> 350Kb. >>> >>> import java.io.BufferedInputStream; >>> import java.io.BufferedOutputStream; >>> import java.io.BufferedReader; >>> import java.io.File; >>> import java.io.FileInputStream; >>> import java.io.FileOutputStream; >>> import java.io.InputStream; >>> import java.io.InputStreamReader; >>> import java.util.ArrayList; >>> import java.util.List; >>> >>> import javax.xml.parsers.SAXParser; >>> import javax.xml.parsers.SAXParserFactory; >>> >>> import org.apache.http.HttpResponse; >>> import org.apache.http.NameValuePair; >>> import org.apache.http.client.HttpClient; >>> import org.apache.http.client.entity.UrlEncodedFormEntity; >>> import org.apache.http.client.methods.HttpPost; >>> import org.apache.http.impl.client.DefaultHttpClient; >>> import org.apache.http.message.BasicNameValuePair; >>> import org.xml.sax.helpers.DefaultHandler; >>> >>> public class HttpClientTest { >>> >>> public static void main(String[] args) throws Exception { >>> new HttpClientTest().runTest(); >>> } >>> >>> public void runTest() throws Exception { >>> List<NameValuePair> list = new ArrayList(50000); >>> list.add(new BasicNameValuePair("db", "pcassay")); >>> list.add(new BasicNameValuePair("dbfrom", "pcassay")); >>> // read in ~50000 ids from file. These are existing >>> uids in the pcassay >>> // database. >>> BufferedReader reader = new BufferedReader(new >>> InputStreamReader(getClass().getClassLoader().getResourceAsStream("ids.txt"))); >>> String line = null; >>> while (null != (line = reader.readLine())) { >>> list.add(new BasicNameValuePair("id", line)); >>> } >>> list.add(new BasicNameValuePair("term", >>> "summary[activityoutcomemethod]")); >>> list.add(new BasicNameValuePair("version", "2.0")); >>> list.add(new BasicNameValuePair("email", >>> "[email protected] ")); >>> list.add(new BasicNameValuePair("tool", >>> "[email protected] ")); >>> >>> System.out.println("#NameValuePairs: " + list.size()); >>> >>> HttpPost post = new HttpPost(" >>> http://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi";); >>> UrlEncodedFormEntity entity = new >>> UrlEncodedFormEntity(list); >>> post.setEntity(entity); >>> HttpClient client = new DefaultHttpClient(); >>> HttpResponse response = client.execute(post); >>> >>> // dump to local file so we can refer to it. >>> File file = File.createTempFile("eutils", ".xml"); >>> System.out.println("Copying eUtils stream to: " + file); >>> response.getEntity().writeTo(new >>> BufferedOutputStream(new FileOutputStream(file))); >>> InputStream is = new BufferedInputStream(new >>> FileInputStream(file)); >>> >>> // attempt to parse with sax. There will be an error if >>> only a fragment >>> // of the xml is returned. >>> SAXParserFactory factory = SAXParserFactory.newInstance(); >>> SAXParser saxParser = factory.newSAXParser(); >>> saxParser.parse(is, new DefaultHandler() { >>> }); >>> } >>> } >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
