https://bz.apache.org/bugzilla/show_bug.cgi?id=70236
Bug ID: 70236
Summary: TLS handshake stalls when the server's first flight is
bigger than the BIO pair buffer
Product: Tomcat 11
Version: unspecified
Hardware: PC
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: Connectors
Assignee: [email protected]
Reporter: [email protected]
Target Milestone: -------
If the first burst of handshake messages the server sends is bigger than 17408
bytes, the OpenSSL-backed `SSLEngine` writes out as much as fits, drops the
rest on the floor, and then waits for the client. The client is waiting for
that rest, and this causes deadlock. Post-quantum (ML-DSA) certificates hit
this with a completely ordinary two-certificate chain.
---
## 1. Reproducer
Two certificate sets, ML-DSA-87 and ML-DSA-65. Each is a self-signed CA plus a
server certificate signed by it. The chain of two is required, one self-signed
certificate is too small.
```bash
mkdir -p /etc/pki/tomcat-pq && cd /etc/pki/tomcat-pq
for ALG in ML-DSA-87 ML-DSA-65; do
openssl req -x509 -newkey $ALG -noenc -keyout ca-$ALG.key -out ca-$ALG.crt \
-subj "/CN=PQ Test CA $ALG" -days 365
openssl req -new -newkey $ALG -noenc -keyout srv-$ALG.key -out srv-$ALG.csr \
-subj "/CN=localhost"
openssl x509 -req -in srv-$ALG.csr -CA ca-$ALG.crt -CAkey ca-$ALG.key \
-out srv-$ALG.crt -days 365
done
```
Point a connector at the ML-DSA-87 set, with `OpenSSLLifecycleListener` enabled
so the FFM engine is actually used:
```xml
<Listener className="org.apache.catalina.core.OpenSSLLifecycleListener" />
...
<Connector port="9443" protocol="org.apache.coyote.http11.Http11NioProtocol"
SSLEnabled="true" scheme="https" secure="true">
<SSLHostConfig protocols="TLSv1.3">
<Certificate certificateFile="/etc/pki/tomcat-pq/srv-ML-DSA-87.crt"
certificateKeyFile="/etc/pki/tomcat-pq/srv-ML-DSA-87.key"
certificateChainFile="/etc/pki/tomcat-pq/ca-ML-DSA-87.crt" />
</SSLHostConfig>
</Connector>
```
`certificateChainFile` matters. Without it Tomcat sends only the leaf, the
handshake stays small and nothing goes wrong.
Then:
```bash
# curl -k https://localhost:9443/
curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 0
```
It hangs first, then dies. Tomcat logs `java.io.EOFException` out of
`SecureNioChannel.handshakeUnwrap`, which is the server noticing the client
gave up.
Change nothing except swapping the three paths to the ML-DSA-65 set and it
serves fine.
How much the server sends in each case:
```bash
timeout 20 openssl s_client -connect localhost:9443 -tls1_3 </dev/null 2>&1 \
| grep -oE 'handshake has read [0-9]+ bytes'
```
```
ML-DSA-87, leaf only (drop certificateChainFile) => 13477 B => HTTP 200
ML-DSA-65, leaf + CA => 15738 B => HTTP 200
ML-DSA-87, leaf + CA => 20900 B => hangs
```
The cutoff is 17408 bytes, which is `BIO_new_bio_pair()`'s default buffer size.
Tomcat asks for that default by passing `0`:
```java
BIO_new_bio_pair(internalBIOPointer, 0, networkBIOPointer, 0);
```
OpenSSL documents that default as sufficient for a maximum size TLS *record*. A
handshake flight is many records.
---
## 2. Real life example: ML-DSA certificates
This was found while enabling ML-DSA server certificates for Candlepin, on RHEL
10.2 with `tomcat-10.1.49-3.el10_2`, `java-25-openjdk-headless-25.0.4.1.1` and
`openssl-3.5.8`, using the OpenSSL FFM connector.
ML-DSA certificates are roughly an order of magnitude bigger than RSA or EC
ones. A leaf cert plus the cert of the CA that issued it is already about 15 KB
for ML-DSA-87, and CertificateVerify adds another 4627-byte signature after
that. You go over 17408 bytes with nothing exotic in the config: no
cross-signing, no deep PKI, just a server certificate and its issuer.
A long RSA chain reproduces exactly the same thing, which is what the attached
test uses, because committing ML-DSA fixtures would require every test
environment to have OpenSSL 3.5+ with the right providers. Still, with PQC it
is the normal case, and it is going to start showing up as more deployments
turn PQC on.
---
## 3. What is actually going wrong
OpenSSL writes a whole handshake flight to the network BIO in one go. If it
does not fit, OpenSSL fills the buffer, keeps the rest internally, and returns
`SSL_ERROR_WANT_WRITE`.
`wrap()` copies out whatever the BIO holds, and then works out what to do next
from nothing more than "is the BIO empty?":
```java
if (sendHandshakeError || BIO_ctrl_pending(state.networkBIO) != 0) {
return SSLEngineResult.HandshakeStatus.NEED_WRAP; // more to send
}
...
return SSLEngineResult.HandshakeStatus.NEED_UNWRAP; // client's turn
```
The BIO is empty, because `wrap()` just emptied it. That does not mean OpenSSL
is done. `wrap()` hands back `NEED_UNWRAP` without ever giving OpenSSL a chance
to write the rest.
There is no explicit re-drive anywhere. There is, however, an accidental one:
`unwrap()` calls `pendingReadableBytesInSSL()`, which does a priming read:
```java
// NOTE: Calling a fake read is necessary before calling
pendingReadableBytesInSSL because
// SSL_pending will return 0 if OpenSSL has not started the current TLS record
int lastPrimingReadResult = SSL_read(state.ssl, MemorySegment.NULL, 0); //
priming read
```
That `SSL_read()` re-enters the state machine and flushes the rest of the
flight as a side effect. So today the engine can only finish sending its own
flight if the peer sends it something first. When the peer is sitting there
waiting for that exact flight, `unwrap()` never gets called and both ends wait
forever.
---
## 4. Fix
Once `wrap()` has drained the network BIO, and while a handshake is still going
on, drive OpenSSL again so it can write whatever did not fit before:
```java
// The network BIO has just been drained. Give OpenSSL the opportunity to write
out any part of the
// current handshake flight that it did not previously have space for, before
the handshake status is
// calculated below.
if (!handshakeFinished && !engineClosed) {
continueHandshake();
}
```
Applied to both OpenSSL engines. A few notes on why it looks like this:
- When there is nothing left to write the call costs nothing.
`SSL_do_handshake()` returns `WANT_READ`, produces no bytes, and leaves the
handshake status alone.
- It is a separate `continueHandshake()` rather than the existing `handshake()`
because `handshake()` opens with `currentHandshake = state.handshakeCount`, and
completion is detected by comparing exactly those two values. Re-snapshotting
the counter halfway through a flight would suppress `FINISHED` for good.
- Ideally this would be conditional on the previous call having returned
`SSL_ERROR_WANT_WRITE`. That is not possible right now: neither engine binds
`SSL_get_error()`. The JNI engine has the `SSL_ERROR_*` constants sitting in
`org.apache.tomcat.jni.SSL` but no accessor for them, and only exposes the
error *queue* through `ERR_error_string()`, which `SSL_ERROR_WANT_WRITE` does
not put anything into. Adding the binding needs a Tomcat Native release. See
bug 67609.
- Just making the BIO pair buffer bigger is not a fix. It moves the threshold
and leaves the failure mode in place, and any fixed size can be exceeded.
---
## 5. Why this may not be the fix you want
The `SSLEngine` javadoc says:
> The SSLEngine produces/consumes **complete** SSL/TLS packets only, and does
> not store application data internally between calls to wrap()/unwrap().
The OpenSSL engines do not obey that today, and this patch does not make them
obey it either. The BIO pair fills right up to its 17408 byte capacity, which
will not line up with a record boundary, so `wrap()` hands the caller a partial
TLS record and OpenSSL holds on to the rest. All the patch does is make sure
the rest comes out on the next call. It does not stop the engine emitting
partial records, and it cannot: a BIO pair backed engine always has an internal
buffer that can run out, and no caller side buffer sizing can help, because
that buffer is not the caller's.
So this clears the deadlock without making the engine contract-conforming. If
you would rather fix the contract, that is a much bigger change, and I am happy
for this patch to be thrown away.
---
## 6. Regression test
`TestSsl.testLargeCertificateChain`, plus new PEM fixtures: an RSA-8192 leaf,
six intermediates and a root, so the certificate message on its own is a bit
over 20 KiB. Eight certificates is the ceiling because
`jdk.tls.maxCertificateChainLength` defaults to 10 on the client.
The test pins the client to TLS 1.2. This matters: with a TLS 1.3 JSSE client
it passes against unpatched code and therefore tests nothing. The client is
what decides whether you see the bug, not the protocol version. The first 17408
bytes do get written, so the client has the ServerHello, and a JSSE TLS 1.3
client then sends something back straight away rather than waiting for the rest
of the flight. That gives `unwrap()` something to read, the priming read
flushes the remainder, and the handshake completes. An OpenSSL TLS 1.3 client
stays silent until it has the whole flight, which is why curl deadlocks in
section 1 while JSSE does not. A TLS 1.2 client sends nothing at all between
ClientHello and ServerHelloDone, so it deadlocks every time and makes a
reliable test.
It also asserts that the chain the server really presented is over 17408 bytes,
so it cannot quietly stop exercising the bug if the fixtures are ever
regenerated smaller.
Verified failing against unpatched trunk and passing with the patch, with JSSE
passing throughout as a control.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
