Hi Sam, Thank you. That does seem definitive. I had tried several times with cURL yesterday that all completed but I never got it to work with HttpClient! I repeated it today and I see truncated files with cURL as well.
I'll follow up on the server side. Thanks again! -----Original Message----- From: Sam Crawford [mailto:[email protected]] Sent: Friday, August 19, 2011 11:48 AM To: HttpClient User Discussion Subject: Re: Truncated output with HttpPost but works with cURL Yes, it appears that the server is the cause. I'm not sure how familiar you are with packet captures, but I've shown the final four packets in the exchange at http://pastebin.com/EDE4KhZr You'll see that www.ncbi.nlm.nih.gov sends a TCP FIN on line 5, and that the chunked response is ended within the same packet on line 57 (the "0" indicates the end of a chunked response). The client (curl, on 192.168.2.100) correctly responds with a FIN as well, and then the connection is closed. The TCP FIN occurred almost exactly 7 minutes after the HTTP request first began. I strongly suspect that the server side script that's processing this data has a maximum execution time of 7 minutes set. Hope this helps, Sam On 19 August 2011 16:32, Sam Crawford <[email protected]> wrote: > Hrmmm, I've not run your Java code, but I think the server may be at > fault here. I'm seeing the response being cut short in curl as well. > It takes a very long time to complete the transaction, but I'm running > it again with a packet capture right now so should be able to see what > is taking place shortly. > > > On 19 August 2011 15:58, Mark Southern <[email protected]> wrote: >> Further details from the HttpResponse Entity: >> >> Response encoding: null >> Response content length: -1 >> Chunked?: true >> Content-Type: text/xml; charset=UTF-8 >> >> >> -----Original Message----- >> From: kim young ill [mailto:[email protected]] >> Sent: Friday, August 19, 2011 10:02 AM >> To: HttpClient User Discussion >> Subject: Re: Truncated output with HttpPost but works with cURL >> >> could be that response is gzipped without specifying content-encoding which >> leads to wrong content-length ? >> >> On Fri, Aug 19, 2011 at 3:32 PM, Mark Southern <[email protected]> wrote: >> >>> Dear Peter, >>> >>> Unfortunately that's not it. The problem occurs whether or not I >>> include the step to dump to the local file (and also with an >>> explicit flush/close added to the BufferedOutputStream). >>> >>> I have made available the java source, ids text file and cUrl >>> example if case anyone cares for a further look: >>> >>> http://chemutils.florida.scripps.edu/httpclient/curl.sh >>> http://chemutils.florida.scripps.edu/httpclient/ids.txt >>> http://chemutils.florida.scripps.edu/httpclient/HttpClientTest.java >>> >>> Thank you, >>> ~Mark. >>> >>> -----Original Message----- >>> From: Pete Keyes [mailto:] >>> Sent: Thursday, August 18, 2011 9:29 PM >>> To: HttpClient User Discussion >>> Subject: RE: Truncated output with HttpPost but works with cURL >>> >>> This is just a guess, but I don't see where you've closed the >>> BufferedOutputStream passed to the writeTo() method. >>> >>> ...Pete Keyes >>> Starbucks Coffee Co. >>> >>> ________________________________________ >>> From: Mark Southern [[email protected]] >>> Sent: Thursday, August 18, 2011 05:48 PM >>> To: [email protected] >>> Subject: Truncated output with HttpPost but works with cURL >>> >>> To whomever can help me! >>> >>> I am submitting to a web service and receiving an xml file in return. >>> I am using POST b/c this is too large for a GET. If I use cURL then this >>> works. >>> However with HttpClient (4.1.2) and the same parameters, I am >>> getting a truncated xml file returned. >>> >>> Does anyone have an idea what might be wrong? I include the test >>> file but not the whole list of parameters I am submitting as this is a >>> further 350Kb. >>> >>> import java.io.BufferedInputStream; >>> import java.io.BufferedOutputStream; import java.io.BufferedReader; >>> import java.io.File; import java.io.FileInputStream; import >>> java.io.FileOutputStream; import java.io.InputStream; import >>> java.io.InputStreamReader; import java.util.ArrayList; import >>> java.util.List; >>> >>> import javax.xml.parsers.SAXParser; >>> import javax.xml.parsers.SAXParserFactory; >>> >>> import org.apache.http.HttpResponse; import >>> org.apache.http.NameValuePair; import >>> org.apache.http.client.HttpClient; >>> import org.apache.http.client.entity.UrlEncodedFormEntity; >>> import org.apache.http.client.methods.HttpPost; >>> import org.apache.http.impl.client.DefaultHttpClient; >>> import org.apache.http.message.BasicNameValuePair; >>> import org.xml.sax.helpers.DefaultHandler; >>> >>> public class HttpClientTest { >>> >>> public static void main(String[] args) throws Exception { >>> new HttpClientTest().runTest(); >>> } >>> >>> public void runTest() throws Exception { >>> List<NameValuePair> list = new ArrayList(50000); >>> list.add(new BasicNameValuePair("db", "pcassay")); >>> list.add(new BasicNameValuePair("dbfrom", >>> "pcassay")); >>> // read in ~50000 ids from file. These are existing >>> uids in the pcassay >>> // database. >>> BufferedReader reader = new BufferedReader(new >>> InputStreamReader(getClass().getClassLoader().getResourceAsStream("i >>> ds.txt"))); >>> String line = null; >>> while (null != (line = reader.readLine())) { >>> list.add(new BasicNameValuePair("id", line)); >>> } >>> list.add(new BasicNameValuePair("term", >>> "summary[activityoutcomemethod]")); >>> list.add(new BasicNameValuePair("version", "2.0")); >>> list.add(new BasicNameValuePair("email", >>> "[email protected] ")); >>> list.add(new BasicNameValuePair("tool", >>> "[email protected] ")); >>> >>> System.out.println("#NameValuePairs: " + >>> list.size()); >>> >>> HttpPost post = new HttpPost(" >>> http://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi";); >>> UrlEncodedFormEntity entity = new >>> UrlEncodedFormEntity(list); >>> post.setEntity(entity); >>> HttpClient client = new DefaultHttpClient(); >>> HttpResponse response = client.execute(post); >>> >>> // dump to local file so we can refer to it. >>> File file = File.createTempFile("eutils", ".xml"); >>> System.out.println("Copying eUtils stream to: " + >>> file); >>> response.getEntity().writeTo(new >>> BufferedOutputStream(new FileOutputStream(file))); >>> InputStream is = new BufferedInputStream(new >>> FileInputStream(file)); >>> >>> // attempt to parse with sax. There will be an error >>> if only a fragment >>> // of the xml is returned. >>> SAXParserFactory factory = >>> SAXParserFactory.newInstance(); >>> SAXParser saxParser = factory.newSAXParser(); >>> saxParser.parse(is, new DefaultHandler() { >>> }); >>> } >>> } >>> >>> -------------------------------------------------------------------- >>> - To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >>> -------------------------------------------------------------------- >>> - To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >>> -------------------------------------------------------------------- >>> - To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
