John Leacox created HTTPCORE-726:
------------------------------------
Summary: SharedInputBuffer expands unbounded on large responses
Key: HTTPCORE-726
URL: https://issues.apache.org/jira/browse/HTTPCORE-726
Project: HttpComponents HttpCore
Issue Type: Bug
Components: HttpCore
Affects Versions: 5.1.3
Environment: MacOS 12, Debian 9, Debian 10
Reporter: John Leacox
Attachments: SharedInputBuffer_Unbounded_Growth_Example.patch,
SharedInputBuffer_Unbounded_Growth_Possible_Fix.patch
Large responses can result in the {{SharedInputBuffer}} expanding unbounded,
but with small intervals. Each expansion is the minimum 1024 byte aligned size
bigger than the current buffer that is filling it beyond its current capacity.
This leads to a ton of heap allocations and garbage collection each time is
expands.
It looks like the capacity updates to {{CapacityChannel}} are reporting current
capacity rather than deltas/increments, so that even when small reads occur,
the whole capacity is reported again to the channel and the capacity window
also grows unbounded.
I've attached a patch that includes some {{println}} statements for the
capacity window size and for buffer being expanded along with the integration
test {{testLargeResponseConsumer}} that shows the buffer and window growing
quite large (the test passes, but the println statements show how large they
are growing).
For a fix I was thinking of updating the capacity tracking to be similar to
{{ReactiveDataConsumer}} in tracking the number of bytes read and then
reporting that delta to {{CapacityChannel}}. This does seem to prevent the
buffer from expanding to huge sizes, but it still expands more than I would
expect because after each read from the {{SharedInputBuffer}}. This remaining
expansion appears to be from {{CapacityWindow}} always setting the IO sessions
with {{OP_READ}} even when the window is still negative after the update.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
