I'll give it a shot. It may be a bit, we've solved the problem by moving the Apache's HTTPClient.
I'll also create an alternate test. The one I wrote uses our internal services; obviously not a good candidate for a standalone test. Thanks for looking into this From: Kevin Conaway [mailto:[EMAIL PROTECTED] Sent: Tuesday, March 25, 2008 2:23 PM To: discuss@restlet.tigris.org Subject: Re: StreamClientCall and FH exhaustion Thanks Jerome. Matt, could you see if that solves your problem as well? If possible, can you attach your test case to 439? On Tue, Mar 25, 2008 at 5:20 PM, Jerome Louvel <[EMAIL PROTECTED]> wrote: Hi Kevin, Very nice work (as always). The patch is in SVN trunk with some slight improvements (more getContentLength() refactoring). Added complete() handling on SimpleCall as well. Best regards, Jerome > -----Message d'origine----- > De : Kevin Conaway [mailto:[EMAIL PROTECTED] > Envoyé : mardi 25 mars 2008 21:18 > À : discuss@restlet.tigris.org > Objet : Re: StreamClientCall and FH exhaustion > > Yes. If the response is chunked, the stream won't get > closed. In general, its not a great idea to close a socket > *stream with the intention of closing the socket. > > I'm going to attach a patch to 439 with the following fixes: > > 1.) In StreamClientCall, the response entity representation > is now wrapped with a WrapperRepresentation that will close > the associated socket when release() is called > > 2.) In HttpServerCall, a new method called complete() was > added that is called after commit(). StreamServerCall > implements this by: > A.) Flushing the response output stream > B.) Exhausting the request input stream > C.) Closing the request socket > > We exhaust the request input stream in case the client is > still writing a request entity to the socket. If we close > the socket without reading the entire request, the client > would get IOExceptions while it is still writing > > 3.) I also modified > StreamServerHelper.getRequestEntityStream() and > StreamClientCall.getResponseEntityStream() to always return a > KeepAliveInputStream. The "raw" stream should never be > exposed to callers as they can (and most likely will) close > it at will. Closing the input/output stream of a socket > closes the entire socket so it should be protected as much as > possible. > > Kevin > > > On Tue, Mar 25, 2008 at 4:08 PM, Jerome Louvel > <[EMAIL PROTECTED]> wrote: > > > > Hi Kevin, > > The InputRepresentation.release() method currently > closes the underlying > stream. As we currently don't support persistent > connections, the closing of > the input stream should actually close the socket input > stream and as a > consequence the socket. > > Do you obtain results that suggest it isn't happening like this? > > > Best regards, > Jerome > > > -----Message d'origine----- > > De : Kevin Conaway [mailto:[EMAIL PROTECTED] > > > Envoyé : lundi 24 mars 2008 18:47 > > > À : discuss@restlet.tigris.org > > Objet : Re: StreamClientCall and FH exhaustion > > > > Assuming the response entity is NOT pre-fetched, what about > > wrapping the InputRepresentation used to store the response > > so that release() closes the associated client socket. > > > > Is it safe to assume that the socket is no longer needed once > > a user calls release() on the Response entity? > > > > > > On 3/23/08, Kevin Conaway <[EMAIL PROTECTED]> wrote: > > > > Jerome, I think you misunderstood me. I was suggesting > > it might be useful to allow the client to tell Restlet to > > read and store the response entity to be consumed later. > > > > If the response entity is read right away and stored, > > there would be no need to keep the connection open and the > > entire request/response cycle could be completed in > short order. > > > > > > > > On Sun, Mar 23, 2008 at 2:28 PM, Jerome Louvel > > <[EMAIL PROTECTED]> wrote: > > > > > > > > Hi Kevin, > > > > Why not, but what would be the use case for > > making such a request? If a > > client application isn't interested in the > > entity, it can always issue a > > HEAD request. > > > > > > Best regards, > > Jerome > > > > > -----Message d'origine----- > > > De : Kevin Conaway > [mailto:[EMAIL PROTECTED] > > > > > Envoyé : samedi 22 mars 2008 13:33 > > > > > À : discuss@restlet.tigris.org > > > Objet : Re: StreamClientCall and FH exhaustion > > > > > > Would it make sense to add an option to the > > Client or Request > > > that specifies whether Restlet should > > prefetch the Response > > > entity? If the response entity was > > prefetched, the socket > > > could be closed quite easily. > > > > > > > > > On Tue, Mar 18, 2008 at 12:42 PM, > Jerome Louvel > > > <[EMAIL PROTECTED]> wrote: > > > > > > > > > > > > Hi all, > > > > > > The client should indeed make sure it > > either consumes > > > the response entity > > > (via getText(), getStream(), etc.) or > > that it calls the > > > release() method. > > > > > > Otherwise, there indeed seem to be a > > bug related to the > > > keep alive > > > connections not being closed (after a > > given timeout?). > > > We have a bug report > > > pending: > > > > > http://restlet.tigris.org/issues/show_bug.cgi?id=439 > > > > > > We hope to have that fixed in 1.1 M3 at > > the end of the month. > > > > > > Best regards, > > > Jerome > > > > > > > -----Message d'origine----- > > > > De : Kevin Conaway > > [mailto:[EMAIL PROTECTED] > > > > Envoyé : mardi 18 mars 2008 00:36 > > > > À : discuss@restlet.tigris.org > > > > Objet : Re: StreamClientCall > and FH exhaustion > > > > > > > > > > > Its my understanding that the socket > > can't be automatically > > > > closed by Restlet because the > > response entity body isn't > > > > ready until the caller > decides to access it. > > > > > > > > If this is correct, I think its up > > the client to some how > > > > alert the Client or the Response that > > he is through > > > with the entity > > > > > > > > > > > > On Mon, Mar 17, 2008 at 7:20 > PM, Matt Reynolds > > > > <[EMAIL PROTECTED]> wrote: > > > > > > > > > > > > I have a question around the > > StreamClientCall code and > > > > socket use. > > > > > > > > The setup : > > > > We're using a single client > > (org.restlet.Client) > > > > instance through a > > > > number of threads (more than 10 > > or so), and calling a > > > > service repeatedly > > > > that has a 50ms round-trip > > time. We're using M1 > > > > (moving to M2, but more > > > > on that later). > > > > > > > > The symptoms : > > > > During a performance test, > > after 20 minutes or so, we > > > > run out of file > > > > handles. We used lsof and > > /proc to isolate the number > > > > of handles used, > > > > and noticed they're > mostly sockets. > > > > > > > > I created a small test harness > > around our usage of > > > > Client to see what > > > > happens. When I run this > > client very fast (30 threads, > > > > no waiting), I > > > > can reproduce the problem in > > less than 10 seconds (on > > > > my local linux > > > > machine, albeit not the same). > > > > > > > > Anyway, the exception thrown > > comes through > > > > > > StreamClientCall.createSocket:80/sendRequest:211 . In > > > > that code, I > > > > notice that the socket is > > created, but never closed. > > > > Following the code > > > > paths, I notice that an > > OutputStream and InputStream > > > > and "gotten" and > > > > passed around to other methods, > > that end up > > > calling OS.close and > > > > IS.close. However, the code > > paths appear to be calling > > > > close on the > > > > wrapper instances of > the OS/IS, either > > > > ChunkedOutputStream(which doesn't > > > > call the > > > > > > destination.close)/KeepAliveOutputStream(which, seemingly > > > > obviously, has no close) or > > > ChunkedInputStream(no underlying > > > > close)/InputEntityStream(same). > > > > > > > > So, it simply appears that the > > Client socket isn't > > > > being closed. I'm > > > > guessing it should be closed as > > part of the finally > > > > block, with a null > > > > check and then a call to > > isClosed? Anyway, please let > > > > me know if I > > > > debugged this path correctly > > and, if possible, what I > > > > should do to get > > > > around this. > > > > > > > > Thanks, > > > > Matt > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
