Federico Mariani created CXF-9272:
-------------------------------------
Summary: HttpClientHTTPConduit: async call to a refused connection
sometimes never calls the ClientCallback (regression from CXF-9250)
Key: CXF-9272
URL: https://issues.apache.org/jira/browse/CXF-9272
Project: CXF
Issue Type: Bug
Components: Transports
Affects Versions: 4.2.4
Reporter: Federico Mariani
Since 4.2.4, an asynchronous invocation through {{HttpClientHTTPConduit}} to an
address that refuses the connection sometimes never completes. Neither
{{ClientCallback.handleResponse}} nor {{handleException}} is called, so a
caller waiting on the callback (or on the future) waits forever. The receive
timeout does not help, because no response handling is ever scheduled.
We found this through Apache Camel's camel-cxf async producer, where an
exchange hangs forever when the target SOAP service is down (CAMEL-26696,
CAMEL-26600). The reproducer below uses plain CXF only.
*Reproducer* (JUnit 5, plain JAX-WS client, a fresh client per call, nothing
listening on the port):
{code:java}
@WebService(targetNamespace = "http://example.org/echo")
public interface Echo {
String echo(String text);
}
@Test
void asyncCallToClosedPortAlwaysCallsBack() throws Exception {
int port;
try (ServerSocket s = new ServerSocket(0)) {
port = s.getLocalPort(); // nothing listens on it after close
}
String address = "http://localhost:" + port + "/echo";
int calls = 2400;
AtomicInteger stuck = new AtomicInteger();
ExecutorService pool = Executors.newFixedThreadPool(16);
CountDownLatch all = new CountDownLatch(calls);
for (int i = 0; i < calls; i++) {
pool.execute(() -> {
try {
JaxWsProxyFactoryBean factory = new JaxWsProxyFactoryBean();
factory.setServiceClass(Echo.class);
factory.setAddress(address);
Client client = ClientProxy.getClient(factory.create());
HTTPClientPolicy policy = new HTTPClientPolicy();
policy.setConnectionTimeout(3000);
policy.setReceiveTimeout(3000);
((HTTPConduit) client.getConduit()).setClient(policy);
CountDownLatch callback = new CountDownLatch(1);
client.invoke(new ClientCallback() {
@Override
public void handleResponse(Map<String, Object> ctx,
Object[] res) {
super.handleResponse(ctx, res);
callback.countDown();
}
@Override
public void handleException(Map<String, Object> ctx,
Throwable ex) {
super.handleException(ctx, ex);
callback.countDown();
}
}, "echo", "hello");
if (!callback.await(8, TimeUnit.SECONDS)) {
stuck.incrementAndGet();
}
client.destroy();
} catch (Exception e) {
// a synchronous failure from invoke() counts as completed
} finally {
all.countDown();
}
});
}
all.await();
pool.shutdown();
System.out.println(Version.getCurrentVersion() + ": stuck (no callback
within 8s) = " + stuck.get());
}
{code}
*Results* (2 runs of 2400 calls per version, 16 threads, JDK 25.0.2 Temurin,
macOS aarch64):
||CXF||calls||no callback within 8 s||
|4.2.3|4800|0|
|4.2.4|4800|22 (18 + 4)|
|4.2.5-SNAPSHOT (20261110.085644)|4800|42 (22 + 20)|
Sequential calls rarely hit it locally (0 in 3300 calls through the Camel
producer). Concurrency widens the race window, as does a loaded CI machine.
*What the stuck exchanges look like*
We instrumented the Camel-based test, which uses the same async
{{ClientImpl.invoke(ClientCallback, ...)}} path, with an out interceptor at
{{Phase.SETUP}} that keeps the out message. Every stuck exchange shows the same
state:
* {{outMessage.getContent(Exception.class)}} is
{{java.util.concurrent.CompletionException: java.net.ConnectException}}
* {{exchange.getInMessage()}} is null
* {{exchange.get(ClientCallback.class)}} is still set, so {{ClientImpl}} never
dispatched it
The calls that do complete report a {{Fault}} caused by {{ConnectException}}
through {{handleException}}.
*Suspected cause* (our reading of the code; we did not instrument inside CXF)
CXF-9250 added this to the {{future.exceptionally(...)}} callback in
{{HttpClientHTTPConduit.HttpClientWrappedOutputStream#setProtocolHeaders}}:
{code:java}
outMessage.setContent(Exception.class, ex instanceof Exception ? (Exception) ex
: new IOException(ex));
...
close();
{code}
{{HTTPConduit.WrappedOutputStream#handleResponseOnWorkqueue}} only schedules
the runnable that calls {{handleResponseInternal()}}, and the fault observer if
that fails, when no exception is recorded:
{code:java}
boolean exceptionSet = outMessage.getContent(Exception.class) != null;
if (!exceptionSet) {
... execute(runnable) ...
}
{code}
If the {{exceptionally}} callback records the {{ConnectException}} before the
caller thread's {{close()}} reaches {{handleResponse()}}, both {{close()}}
calls (the caller's and the one from {{exceptionally}}) skip scheduling.
Nothing then calls the fault observer, so the {{ClientCallback}} is never
invoked. The raw {{CompletionException}} left on the out message of the stuck
exchanges matches this path. The 4.2.3 to 4.2.4 difference also points at
CXF-9250: between the {{cxf-4.2.3}} and {{cxf-4.2.4}} tags it is the only
commit touching {{HttpClientHTTPConduit}} or {{HTTPConduit}}.
CXF-9250 also went into 4.1.9, 4.0.12 and 3.6.13. We did not test those
versions.
Possibly related, but a different cause: CXF-9257 (the callback throws) and
CXF-9267 (a race in {{HttpClientBodyPublisher}} hides the connect timeout).
Both are fixed on main, and this issue still reproduces on the 4.2.5-SNAPSHOT
above.
_Claude Code on behalf of Croway_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)