mwashburn2 opened a new issue, #2744:
URL: https://github.com/apache/activemq/issues/2744
`NIOTransport.serviceRead()` allocates a new direct `ByteBuffer` for every
inbound frame larger than the I/O buffer (8 KB by default):
```java
if (nextFrameSize > inputBuffer.capacity()) {
currentBuffer = ByteBuffer.allocateDirect(nextFrameSize);
```
This was a heap buffer before AMQ-6184. Direct buffers are only released
when their `Cleaner` runs after a GC, so with steady large-message traffic,
direct memory fills up to `MaxDirectMemorySize`. `Bits.reserveMemory` then
calls `System.gc()` and the NIO selector thread blocks until it completes. All
connections served by that selector stall.
**Observed:** a production broker (nio+ transport, ZGC, 64 GB heap) showed
periodic 2–3 s throughput dips.
**Test setup:** 1 publisher host, 5 subscriber hosts, about 17.5k msgs/s, 2%
large messages plus 2 huge messages per second. The 40 Gbit NIC was near
saturation. Three runs per variant, with a fresh broker each run. Median across
runs:
| | stock | heap buffer for oversize frames |
|---|---|---|
| explicit `System.gc()` per 5-min run | 68 | 0 |
| peak direct memory | 2.15 GB (the cap) | 3.3 MB |
| producer send p99 / p999 | 19.8 / 95.7 ms | 10.4 / 76.6 ms |
| sends stalled >100 ms | 4,804 | 2,747 |
| huge-message latency p99 / max | 803 / 1,417 ms | 288 / 477 ms |
**Proposal:** read oversize frames into a heap buffer. The frame is
unmarshalled from a byte stream anyway, so a direct buffer brings no benefit
here. The 8 KB `inputBuffer`, which is reused, stays direct.
Alternatives tested:
- A reusable per-connection direct buffer up to 1 MB: similar results, but
more code.
- A shared bounded direct pool: CPU spikes, and worse tail latency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
For further information, visit: https://activemq.apache.org/contact
