[
https://issues.apache.org/jira/browse/TS-1457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454574#comment-13454574
]
Alan M. Carroll commented on TS-1457:
-------------------------------------
I have a testing patch available. This patch does not, IMHO, actually fix the
problem but if I understand the issue it should noticeably ameliorate it. It
does so for me, I would like feedback if it helps for others.
Patch Theory;
The problem is in the chunked handling. When performing transforms on chunked
streams the stream is dechunked, transformed, then chunked. As part of this
instances of the ChunkedHandler class (HttpTunnel.cc) are created. These
contain MIOBuffers used to buffer the data. These are initialized such that
they by default create maximal (2M) sized buffers. Internally the maximum chunk
size is 4096. While the presumption is that using large buffers would reduce
buffer creation overhead it appears that the downstream client is sufficiently
aggressive that the buffers are consumed before being filled, in many cases
with a single chunk in them. Hence ATS uses a 2M buffer per 4K chunk which
leads to the huge memory footprint. Because the buffers generally transferred
between MIOBuffers instead of copied these large and mostly empty buffers
travel through the transform chain. In my testing, effectively almost every
time the chunking logic wrote the trailing CR LF, it would allocate a new 2M
buffer.
What this patch does is two things. It sets the chunking buffer default size to
the maximum chunk size, and adjusts the maximum chunk size so that it, plus the
header and trailer, are exactly the original maximum chunk size so that the
actual chunks of output are a nice 4096 bytes long.
This is just a stopgap, though. Depending on the concurrency we could do better
by, when writing chunks, checking the write availability and adjusting it to be
as big as the pending data rather than just a single chunk. This helps if the
producer and client run in the same thread so that buffers cannot be consumed
while the pending data is being transferred.
Similarly we should look at the transfer case on the consumption size and be
careful there to optimize buffer allocation use and size. That is less of an
issue because it uses primarily transfer to move data and only does buffer
allocation in the MIOBuffer when the byte count is small. It might be
reasonable to set the default buffer size down to that min transfer byte value.
If you want to try that change HttpTunnel.cc:169 to
{code}
dechunked_buffer = new_MIOBuffer(BUFFER_SIZE_INDEX_256);
{code}
The costs of setting the buffer sizes smaller is more overhead and potentially
too many buffers (there is a warning somewhere about stack overflow if there
are too many buffers).
> transforms during video requests cause massive memory growth
> ------------------------------------------------------------
>
> Key: TS-1457
> URL: https://issues.apache.org/jira/browse/TS-1457
> Project: Traffic Server
> Issue Type: Bug
> Components: Core
> Affects Versions: 3.2.0
> Environment: Linux RHEL6.2
> Reporter: Aidan McGurn
> Priority: Critical
> Attachments: 3.0.3-null-massif.txt, 3.0.5-null-massif.txt,
> no-null-3.2.0-massif.txt, null-3.0.2-massif.txt, null-3.2.0-massif.txt,
> ts-1457-patch-1.diff
>
>
> Running the null-transform example object over 3.2.0, shows massive memory
> growth -
> This could be seen on our system since using 3.2.0 with custom plugin which
> invokes transforms - under load ~6K and with video content (only Video -
> small files are ok), a huge growth was seen (GB's) such that it crashes the
> load test machine.
> heap profiles are attached for comparison:
> -null transform example so over 3.2.0
> -NO null transform over 3.2.0
> -null transform example so over 3.0.2
> As can be seen no issue in 3.0.2 -
> The profiles show clearly the call chain is different for both releases
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
