[ https://issues.apache.org/jira/browse/HTTPCORE-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17095843#comment-17095843 ]
Kristof Jozsa commented on HTTPCORE-629: ---------------------------------------- Meanwhile I built 4.1.5-SNAPSHOT from your git repository and deployed for testing. The smoke test is ok, we'll see how it behaves under load on our staging environment. If you have a more official snapshot out on some public repository, I'd be happy to switch to it :) > IOReactor shutdowns on ConnectionClosedException > ------------------------------------------------ > > Key: HTTPCORE-629 > URL: https://issues.apache.org/jira/browse/HTTPCORE-629 > Project: HttpComponents HttpCore > Issue Type: Bug > Affects Versions: 4.4.12 > Reporter: Kristof Jozsa > Priority: Major > > We run a Camel/CXFRS project with httpcore-nio 4.4.12 and we experience > IOReactor shutdowns which seem to happen on ConnectionClosedExceptions in our > production environment. > We found this spamming in our logs: > {{Caused by: java.lang.IllegalStateException: Request cannot be executed; I/O > reactor status: STOPPED}} > {{ at org.apache.http.util.Asserts.check(Asserts.java:46)}} > {{ at > org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)}} > {{ at > org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)}} > {{ at > org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.connect(AsyncHTTPConduit.java:596)}} > {{ at > org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.setupWrappedStream(AsyncHTTPConduit.java:442)}} > {{ at > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleHeadersTrustCaching(HTTPConduit.java:1343)}} > {{ at > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.onFirstWrite(HTTPConduit.java:1304)}} > {{ at > org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.onFirstWrite(AsyncHTTPConduit.java:435)}} > > Hunting for reasons we first found `ConnectionClosedException`s - these are > completely fine: > {{}}{{Caused by: org.apache.http.ConnectionClosedException: > ConnectionClosedException invoking > [https://target-host-masked.com/api/rest/v1/products/21110363:] Connection > closed unexpectedly at > java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) at > java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at > java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) > at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481) > at > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1400) > at > org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1384) > at > org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.close(AsyncHTTPConduit.java:429) > at > org.apache.cxf.io.AbstractWrappedOutputStream.close(AbstractWrappedOutputStream.java:77) > at > org.apache.cxf.io.CacheAndWriteOutputStream.postClose(CacheAndWriteOutputStream.java:56) > at org.apache.cxf.io.CachedOutputStream.close(CachedOutputStream.java:228) > at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) at > org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:671) at > org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:63)}} > > but these following traces: > {{Caused by: org.apache.http.ConnectionClosedException: Connection closed > unexpectedlyCaused by: org.apache.http.ConnectionClosedException: Connection > closed unexpectedly at > org.apache.http.nio.protocol.HttpAsyncRequestExecutor.closed(HttpAsyncRequestExecutor.java:146) > at > org.apache.http.impl.nio.client.InternalIODispatch.onClosed(InternalIODispatch.java:71) > at > org.apache.http.impl.nio.client.InternalIODispatch.onClosed(InternalIODispatch.java:39) > at > org.apache.http.impl.nio.reactor.AbstractIODispatch.disconnected(AbstractIODispatch.java:100) > at > org.apache.http.impl.nio.reactor.BaseIOReactor.sessionClosed(BaseIOReactor.java:277) > at > org.apache.http.impl.nio.reactor.AbstractIOReactor.processClosedSessions(AbstractIOReactor.java:449) > at > org.apache.http.impl.nio.reactor.AbstractIOReactor.hardShutdown(AbstractIOReactor.java:590) > at > org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:305) > at > org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) > at > org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)}} > > seem to show that the IOReactor went to hardShutdown upon the > ConnectionClosedException? > > We tried digging into the source to understand how this can happen and we > believe that if the {{processClosedSessions()}} method in > {{AbstractIOReactor}} class shows any uncaught exceptions (like a > ConnectionClosedException?), it will indeed cause the reactor to stop outside > its infinite loop in the finally block. > > Can you please double check and confirm this behaviour? How can we avoid > getting this problem? I would assume if a certain connection gets broken on > the server side, it's handled gracefully. The fact that the whole reactor > shuts down leaves us no other option then to restart our service - clearly we > need a proper way to manage this somehow. > > I hope I'm not horribly mistaken with the drawn consequences above - if so, > please point out my errors. Thanks really much in advance! > EDIT: fixed formatting of stacktraces -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org For additional commands, e-mail: dev-h...@hc.apache.org