I am hitting a 3rd party server IIS. I have written a lot of variations of my code and what I see is that when I take a long time to process the InputStream, it gets closed down on me, as you said. I did a simple experiment. I read the whole buffer with no outputting and it worked. I then added a sleep of 1/2 a second between reading each line and it closed before I finished. (usu 7+ minutes)
What I am doing now, is stopping to read line by line and read a bunch of lines at a time and process the lines a bunch at a time. This is much faster and seems to get around the problem and still is scalalable to the larger files. If this does not work, I would use some kind of concurrent queue. At any rate, I wonder if there is a way to force the web server to keep the connection open while we read the InputStream. -----Original Message----- From: Oleg Kalnichevski [mailto:[EMAIL PROTECTED] Sent: Monday, August 07, 2006 1:34 PM To: HttpClient User Discussion Subject: RE: Reading line by line using outputstream On Sun, 2006-08-06 at 16:58 +0300, David Rosenstark wrote: > Oleg, > There is no exception. I get a -1 on the read even the file is not finished. > (I have even removed my code that uses a BufferedReader) > > Here is the log. > > 2006-08-06 15:51:43,191 [main] DEBUG httpclient.wire.content - << > ",,_G2G Jhook $100 Standard,06/11/2006,******0064580557,, ,,,,," > > At this point, I get a -1. David, A socket input stream is meant to return -1 to indicate an 'end of stream' condition. To me that means that the server for some reason chose to close the connection in the middle of the process. This does not seem like a client side issue to me, because otherwise an I/O exception should have been thrown. What server are you targeting? Are you using a proxy? Oleg > > 2006-08-06 15:51:50,181 [main] DEBUG > org.apache.commons.httpclient.HttpMethodBase - Resorting to protocol > version default close connection policy > 2006-08-06 15:51:50,181 [main] DEBUG > org.apache.commons.httpclient.HttpMethodBase - Should NOT close > connection, using HTTP/1.1 > 2006-08-06 15:51:50,181 [main] DEBUG > org.apache.commons.httpclient.HttpConnection - enter > HttpConnection.isResponseAvailable() > 2006-08-06 15:51:50,181 [main] DEBUG > org.apache.commons.httpclient.HttpConnection - enter > HttpConnection.releaseConnection() > 2006-08-06 15:51:50,181 [main] DEBUG > org.apache.commons.httpclient.HttpConnection - Releasing connection > back to connection manager. > > > When I read the file with no actions taken, then it works, since it > manages in under 5 minutes. > When I have I/0, as in the case above where I am spitting out each > line to standard out as I read it, it dies after 7 minutes 14 seconds. > The problem is that I would rather not read such a large file in, yet > I do not see a way to process lines as I read them. > > Any ideas -- some sort of connection timeout? > > > -----Original Message----- > From: Oleg Kalnichevski [mailto:[EMAIL PROTECTED] > Sent: Wednesday, August 02, 2006 1:51 PM > To: HttpClient User Discussion > Subject: RE: Reading line by line using outputstream > > On Wed, 2006-08-02 at 13:49 +0300, David Rosenstark wrote: > > Not sure if this matters, but I am wrapping the InputStream in a > > bufferedReader so I can read a line at a time. Could this be the problem? > > > > Thanks, > > David > > > > Please post the exception stack trace and a wire/context log of the > session > > Oleg > > > -----Original Message----- > > From: David Rosenstark [mailto:[EMAIL PROTECTED] > > Sent: Wednesday, August 02, 2006 12:58 PM > > To: 'HttpClient User Discussion' > > Subject: Reading line by line using outputstream > > > > I am trying to switch my application to be more efficient and > > process line by line rather than reading the whole buffer in first, > > since we are retrieiving very large files. > > However, at some point the connection is closing. Is there some > > parameter > I > > need to set to keep it open while I am reading? I do not see any > > long > delay > > in the reading -- the code ran for about 8 mintues and then the > > connection close. This all worked fine when read in at once. > > > > > > > > -------------------------------------------------------------------- > > - To unsubscribe, e-mail: > > [EMAIL PROTECTED] > > For additional commands, e-mail: > > [EMAIL PROTECTED] > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: > [EMAIL PROTECTED] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: > [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
