[
https://issues.apache.org/jira/browse/HTTPCORE-629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristof Jozsa updated HTTPCORE-629:
-----------------------------------
Description:
We run a Camel/CXFRS project with httpcore-nio 4.4.12 and we experience
IOReactor shutdowns which seem to happen on ConnectionClosedExceptions in our
production environment.
We found this spamming in our logs:
{{Caused by: java.lang.IllegalStateException: Request cannot be executed; I/O
reactor status: STOPPED}}
{{ at org.apache.http.util.Asserts.check(Asserts.java:46)}}
{{ at
org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)}}
{{ at
org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)}}
{{ at
org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.connect(AsyncHTTPConduit.java:596)}}
{{ at
org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.setupWrappedStream(AsyncHTTPConduit.java:442)}}
{{ at
org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleHeadersTrustCaching(HTTPConduit.java:1343)}}
{{ at
org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.onFirstWrite(HTTPConduit.java:1304)}}
{{ at
org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.onFirstWrite(AsyncHTTPConduit.java:435)}}
Hunting for reasons we first found `ConnectionClosedException`s - these are
completely fine:
{{}}{{Caused by: org.apache.http.ConnectionClosedException:
ConnectionClosedException invoking
[https://target-host-masked.com/api/rest/v1/products/21110363:] Connection
closed unexpectedly at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at
org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1400)
at
org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1384)
at
org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.close(AsyncHTTPConduit.java:429)
at
org.apache.cxf.io.AbstractWrappedOutputStream.close(AbstractWrappedOutputStream.java:77)
at
org.apache.cxf.io.CacheAndWriteOutputStream.postClose(CacheAndWriteOutputStream.java:56)
at org.apache.cxf.io.CachedOutputStream.close(CachedOutputStream.java:228) at
org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) at
org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:671) at
org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:63)}}
but these following traces:
{{Caused by: org.apache.http.ConnectionClosedException: Connection closed
unexpectedlyCaused by: org.apache.http.ConnectionClosedException: Connection
closed unexpectedly at
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.closed(HttpAsyncRequestExecutor.java:146)
at
org.apache.http.impl.nio.client.InternalIODispatch.onClosed(InternalIODispatch.java:71)
at
org.apache.http.impl.nio.client.InternalIODispatch.onClosed(InternalIODispatch.java:39)
at
org.apache.http.impl.nio.reactor.AbstractIODispatch.disconnected(AbstractIODispatch.java:100)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.sessionClosed(BaseIOReactor.java:277)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processClosedSessions(AbstractIOReactor.java:449)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.hardShutdown(AbstractIOReactor.java:590)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:305)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)}}
seem to show that the IOReactor went to hardShutdown upon the
ConnectionClosedException?
We tried digging into the source to understand how this can happen and we
believe that if the {{processClosedSessions()}} method in {{AbstractIOReactor}}
class shows any uncaught exceptions (like a ConnectionClosedException?), it
will indeed cause the reactor to stop outside its infinite loop in the finally
block.
Can you please double check and confirm this behaviour? How can we avoid
getting this problem? I would assume if a certain connection gets broken on the
server side, it's handled gracefully. The fact that the whole reactor shuts
down leaves us no other option then to restart our service - clearly we need a
proper way to manage this somehow.
I hope I'm not horribly mistaken with the drawn consequences above - if so,
please point out my errors. Thanks really much in advance!
EDIT: fixed formatting of stacktraces
was:
We run a Camel/CXFRS project with httpcore-nio 4.4.12 and we experience
IOReactor shutdowns which seem to happen on ConnectionClosedExceptions in our
production environment.
We found this spamming in our logs:
{{Caused by: java.lang.IllegalStateException: Request cannot be executed; I/O
reactor status: STOPPED}}
{{ at org.apache.http.util.Asserts.check(Asserts.java:46)}}
{{ at
org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)}}
{{ at
org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)}}
{{ at
org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.connect(AsyncHTTPConduit.java:596)}}
{{ at
org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.setupWrappedStream(AsyncHTTPConduit.java:442)}}
{{ at
org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleHeadersTrustCaching(HTTPConduit.java:1343)}}
{{ at
org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.onFirstWrite(HTTPConduit.java:1304)}}
{{ at
org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.onFirstWrite(AsyncHTTPConduit.java:435)}}
Hunting for reasons we first found `ConnectionClosedException`s - these are
completely fine:
{{Caused by: org.apache.http.ConnectionClosedException:
ConnectionClosedException invoking
https://target-host-masked.com/api/rest/v1/products/21110363: Connection closed
unexpectedlyCaused by: org.apache.http.ConnectionClosedException:
ConnectionClosedException invoking
https://target-host-masked.com/api/rest/v1/products/21110363: Connection closed
unexpectedly at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at
org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1400)
at
org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1384)
at
org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.close(AsyncHTTPConduit.java:429)
at
org.apache.cxf.io.AbstractWrappedOutputStream.close(AbstractWrappedOutputStream.java:77)
at
org.apache.cxf.io.CacheAndWriteOutputStream.postClose(CacheAndWriteOutputStream.java:56)
at org.apache.cxf.io.CachedOutputStream.close(CachedOutputStream.java:228) at
org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) at
org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:671) at
org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:63)}}
but these following traces:
{{Caused by: org.apache.http.ConnectionClosedException: Connection closed
unexpectedlyCaused by: org.apache.http.ConnectionClosedException: Connection
closed unexpectedly at
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.closed(HttpAsyncRequestExecutor.java:146)
at
org.apache.http.impl.nio.client.InternalIODispatch.onClosed(InternalIODispatch.java:71)
at
org.apache.http.impl.nio.client.InternalIODispatch.onClosed(InternalIODispatch.java:39)
at
org.apache.http.impl.nio.reactor.AbstractIODispatch.disconnected(AbstractIODispatch.java:100)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.sessionClosed(BaseIOReactor.java:277)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processClosedSessions(AbstractIOReactor.java:449)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.hardShutdown(AbstractIOReactor.java:590)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:305)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)}}
seem to show that the IOReactor went to hardShutdown upon the
ConnectionClosedException?
We tried digging into the source to understand how this can happen and we
believe that if the {{processClosedSessions()}} method in {{AbstractIOReactor}}
class shows any uncaught exceptions (like a ConnectionClosedException?), it
will indeed cause the reactor to stop outside its infinite loop in the finally
block.
Can you please double check and confirm this behaviour? How can we avoid
getting this problem? I would assume if a certain connection gets broken on the
server side, it's handled gracefully. The fact that the whole reactor shuts
down leaves us no other option then to restart our service - clearly we need a
proper way to manage this somehow.
I hope I'm not horribly mistaken with the drawn consequences above - if so,
please point out my errors. Thanks really much in advance!
> IOReactor shutdowns on ConnectionClosedException
> ------------------------------------------------
>
> Key: HTTPCORE-629
> URL: https://issues.apache.org/jira/browse/HTTPCORE-629
> Project: HttpComponents HttpCore
> Issue Type: Bug
> Affects Versions: 4.4.12
> Reporter: Kristof Jozsa
> Priority: Critical
>
> We run a Camel/CXFRS project with httpcore-nio 4.4.12 and we experience
> IOReactor shutdowns which seem to happen on ConnectionClosedExceptions in our
> production environment.
> We found this spamming in our logs:
> {{Caused by: java.lang.IllegalStateException: Request cannot be executed; I/O
> reactor status: STOPPED}}
> {{ at org.apache.http.util.Asserts.check(Asserts.java:46)}}
> {{ at
> org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)}}
> {{ at
> org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)}}
> {{ at
> org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.connect(AsyncHTTPConduit.java:596)}}
> {{ at
> org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.setupWrappedStream(AsyncHTTPConduit.java:442)}}
> {{ at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleHeadersTrustCaching(HTTPConduit.java:1343)}}
> {{ at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.onFirstWrite(HTTPConduit.java:1304)}}
> {{ at
> org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.onFirstWrite(AsyncHTTPConduit.java:435)}}
>
> Hunting for reasons we first found `ConnectionClosedException`s - these are
> completely fine:
> {{}}{{Caused by: org.apache.http.ConnectionClosedException:
> ConnectionClosedException invoking
> [https://target-host-masked.com/api/rest/v1/products/21110363:] Connection
> closed unexpectedly at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at
> java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
> at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1400)
> at
> org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1384)
> at
> org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.close(AsyncHTTPConduit.java:429)
> at
> org.apache.cxf.io.AbstractWrappedOutputStream.close(AbstractWrappedOutputStream.java:77)
> at
> org.apache.cxf.io.CacheAndWriteOutputStream.postClose(CacheAndWriteOutputStream.java:56)
> at org.apache.cxf.io.CachedOutputStream.close(CachedOutputStream.java:228)
> at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) at
> org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:671) at
> org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:63)}}
>
> but these following traces:
> {{Caused by: org.apache.http.ConnectionClosedException: Connection closed
> unexpectedlyCaused by: org.apache.http.ConnectionClosedException: Connection
> closed unexpectedly at
> org.apache.http.nio.protocol.HttpAsyncRequestExecutor.closed(HttpAsyncRequestExecutor.java:146)
> at
> org.apache.http.impl.nio.client.InternalIODispatch.onClosed(InternalIODispatch.java:71)
> at
> org.apache.http.impl.nio.client.InternalIODispatch.onClosed(InternalIODispatch.java:39)
> at
> org.apache.http.impl.nio.reactor.AbstractIODispatch.disconnected(AbstractIODispatch.java:100)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.sessionClosed(BaseIOReactor.java:277)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processClosedSessions(AbstractIOReactor.java:449)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.hardShutdown(AbstractIOReactor.java:590)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:305)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)}}
>
> seem to show that the IOReactor went to hardShutdown upon the
> ConnectionClosedException?
>
> We tried digging into the source to understand how this can happen and we
> believe that if the {{processClosedSessions()}} method in
> {{AbstractIOReactor}} class shows any uncaught exceptions (like a
> ConnectionClosedException?), it will indeed cause the reactor to stop outside
> its infinite loop in the finally block.
>
> Can you please double check and confirm this behaviour? How can we avoid
> getting this problem? I would assume if a certain connection gets broken on
> the server side, it's handled gracefully. The fact that the whole reactor
> shuts down leaves us no other option then to restart our service - clearly we
> need a proper way to manage this somehow.
>
> I hope I'm not horribly mistaken with the drawn consequences above - if so,
> please point out my errors. Thanks really much in advance!
> EDIT: fixed formatting of stacktraces
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@hc.apache.org
For additional commands, e-mail: dev-h...@hc.apache.org
