[
https://issues.apache.org/jira/browse/CAMEL-26696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on CAMEL-26696 started by Guillaume Nodet.
-----------------------------------------------
> camel-cxf: async producer may never complete the exchange when the connection
> fails
> -----------------------------------------------------------------------------------
>
> Key: CAMEL-26696
> URL: https://issues.apache.org/jira/browse/CAMEL-26696
> Project: Camel
> Issue Type: Bug
> Components: camel-cxf
> Affects Versions: 4.24.0
> Reporter: Aurélien Pupier
> Assignee: Guillaume Nodet
> Priority: Major
>
> When a cxf:// producer (default, non-synchronous) calls an endpoint where the
> connection fails, such as a closed port, the Camel exchange sometimes never
> completes. The calling thread then waits on the
> DefaultAsyncProcessorAwaitManager latch indefinitely. A route calling a SOAP
> service that is down could hang instead of failing with an exception.
> This was observed as a flaky test, CxfProducerTest.testInvokingAWrongServer
> (CAMEL-26600). It hits the 30s @Timeout even after connection and receive
> timeouts were configured on the HTTP conduit (PR #29219). Thread dump of the
> stuck test thread:
> java.util.concurrent.TimeoutException: testInvokingAWrongServer() timed out
> after 30 seconds
> Caused by: ...ExecutionTimeoutException: Execution timed out in thread
> junit-timeout-thread-10
> at
> java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)
> at
> org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager.await(DefaultAsyncProcessorAwaitManager.java:112)
> at
> org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager.process(DefaultAsyncProcessorAwaitManager.java:89)
> at
> org.apache.camel.impl.engine.SharedCamelInternalProcessor.process(SharedCamelInternalProcessor.java:86)
> at
> org.apache.camel.support.cache.DefaultProducerCache.send(DefaultProducerCache.java:177)
> at
> org.apache.camel.impl.engine.DefaultProducerTemplate.request(DefaultProducerTemplate.java:372)
> at
> org.apache.camel.component.cxf.jaxws.CxfProducerTest.sendSimpleMessageWithPayloadMessage(CxfProducerTest.java:276)
> at
> org.apache.camel.component.cxf.jaxws.CxfProducerTest.testInvokingAWrongServer(CxfProducerTest.java:201)
> The test thread is parked in the await manager. So the exchange went through
> CxfProducer.process(Exchange, AsyncCallback), which returned false, and
> AsyncCallback.done(...) was never called.
> Analysis so far (not confirmed)
> The async path completes the exchange only through
> CxfClientCallback.handleResponse or handleException calling
> camelAsyncCallback.done(false). Possible causes:
> CXF's HTTPConduit or async client never invokes
> ClientCallback.handleException for some connection failures. This might be a
> CXF bug, or a missing timeout or executor configuration.
> handleException is invoked, but code before camelAsyncCallback.done(false) in
> the finally block throws, so the callback is skipped. Candidates in
> CxfClientCallback:
> endpoint.getCookieHandler().storeCookies(...) (only IOException is caught).
> endpoint.getCxfBinding().populateExchangeFromCxfResponse(...).
> conduitSelector.complete(cxfExchange).
> Wrapping the Camel callback in its own try/finally would make this robust.
> A race or thread issue in the CXF async client when the failure happens very
> quickly.
> The flakiness increased around 2 Nov. The camel-cxf changes of that day
> (CAMEL-26143, CAMEL-26144) don't touch the async callback, so they don't seem
> to be the cause. A CXF or JDK change, or the CI environment, may be involved.
> Current workaround
> PR #29339 sets synchronous=true on the "wrong server" endpoint of
> CxfProducerTest. The test is stable, but the async error path is no longer
> covered by any test.
> Expected behavior
> When the connection to the target fails, the exchange completes with an
> exception. The error is reported via the callback, and the thread never waits
> indefinitely.
> Suggested next steps
> Reproduce outside the flaky test, for example by looping a producer call to a
> closed port with the default async mode.
> Log or debug whether CxfClientCallback.handleException is reached and whether
> done() is called.
> Make CxfClientCallback always call camelAsyncCallback.done(false), even if
> the preceding steps throw.
> If the cause is in CXF, report it upstream or add a safe configuration or
> timeout.
> Add a test covering the async path with a failing connection, then remove the
> synchronous=true workaround from CxfProducerTest.
> Drafted by IBM Bob on behalf of apupier.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
