[
https://issues.apache.org/jira/browse/HDFS-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14185710#comment-14185710
]
Tsz Wo Nicholas Sze commented on HDFS-7276:
-------------------------------------------
bq. Thinking on it, the patch only limits the outstanding number of Packets if
Packets always allocate same size buffers, right? If the objective is a limit
on Packet count, perhaps a more direct choke on Packet allocations would be
more effective?
The objective is to limit the memory usage. Limiting Packet count may not be
desirable since packets can be small. They occupy small amount of space even
there is a large number of them.
Ideally, we should limit the total size of all arrays. However, it is
unnecessary since non-full packets are usually small and not frequent. So, we
only limit full packets.
bq. Meantime, I like the buffer reuse that this patch brings.
Thanks.
> Limit the number of byte arrays used by DFSOutputStream
> -------------------------------------------------------
>
> Key: HDFS-7276
> URL: https://issues.apache.org/jira/browse/HDFS-7276
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
> Reporter: Tsz Wo Nicholas Sze
> Assignee: Tsz Wo Nicholas Sze
> Attachments: h7276_20141021.patch, h7276_20141022.patch,
> h7276_20141023.patch, h7276_20141024.patch
>
>
> When there are a lot of DFSOutputStream's writing concurrently, the number of
> outstanding packets could be large. The byte arrays created by those packets
> could occupy a lot of memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
