[
https://issues.apache.org/jira/browse/HTTPCORE-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619089#comment-17619089
]
ASF subversion and git services commented on HTTPCORE-726:
----------------------------------------------------------
Commit 791dcd25df0fd40b0f14e8a02d6c8f1bc1b9b27b in httpcomponents-core's branch
refs/heads/HTTPCORE-726 from John Leacox
[ https://gitbox.apache.org/repos/asf?p=httpcomponents-core.git;h=791dcd25d ]
HTTPCORE-726: Improved capacity management in SharedInputBuffer
> SharedInputBuffer expands unbounded on large responses
> ------------------------------------------------------
>
> Key: HTTPCORE-726
> URL: https://issues.apache.org/jira/browse/HTTPCORE-726
> Project: HttpComponents HttpCore
> Issue Type: Bug
> Components: HttpCore
> Affects Versions: 5.1.3
> Environment: MacOS 12, Debian 9, Debian 10
> Reporter: John Leacox
> Priority: Major
> Fix For: 5.1.5, 5.2-beta3
>
> Attachments: SharedInputBuffer_Unbounded_Growth_Example.patch,
> SharedInputBuffer_Unbounded_Growth_Possible_Fix.patch
>
>
> Large responses can result in the {{SharedInputBuffer}} expanding unbounded,
> but with small intervals. Each expansion is the minimum 1024 byte aligned
> size bigger than the current buffer that is filling it beyond its current
> capacity. This leads to a ton of heap allocations and garbage collection each
> time is expands.
>
> It looks like the capacity updates to {{CapacityChannel}} are reporting
> current capacity rather than deltas/increments, so that even when small reads
> occur, the whole capacity is reported again to the channel and the capacity
> window also grows unbounded.
>
> I've attached a patch that includes some {{println}} statements for the
> capacity window size and for buffer being expanded along with the integration
> test {{testLargeResponseConsumer}} that shows the buffer and window growing
> quite large (the test passes, but the println statements show how large they
> are growing).
>
> For a fix I was thinking of updating the capacity tracking to be similar to
> {{ReactiveDataConsumer}} in tracking the number of bytes read and then
> reporting that delta to {{CapacityChannel}}. This does seem to prevent the
> buffer from expanding to huge sizes, but it still expands more than I would
> expect because after each read from the {{SharedInputBuffer}}. This remaining
> expansion appears to be from {{CapacityWindow}} always setting the IO
> sessions with {{OP_READ}} even when the window is still negative after the
> update.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
