[
https://issues.apache.org/jira/browse/HDFS-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183623#comment-14183623
]
Tsz Wo Nicholas Sze commented on HDFS-7276:
-------------------------------------------
[~stack], thanks for taking a look.
bq. The patch does not seem to be concerned with the amount of memory allocated
since it only counts buffers, not how much has been actually allocated. Or is
the idea throttle array allocations as means of throttling number of Packets
outstanding?
Yes, limiting the number of arrays also limits the number of outstanding
packets. A full packet is around 64kB. We indeed only limit the outstanding
full packets.
bq. Any idea how much the synchronize on allocation/recycle and count as well
as the wait/notifyAll changes write perf?
I will do some performance tests.
bq. Why volatile in below when doesn't seem to ever change (Should it be
configurable? Would be good if MAX_ARRAYS was configurable at least for clients
who'd like to postpone blocking): ...
It needs volatile. Otherwise, other threads may not see the change done by
reset(..). Configurable is a good idea. Let me think about how to do it.
bq. Nit: IMO 1 << 11 is cryptic. Suggest instead you write out 2048?
Sure, will do.
> Limit the number of byte arrays used by DFSOutputStream
> -------------------------------------------------------
>
> Key: HDFS-7276
> URL: https://issues.apache.org/jira/browse/HDFS-7276
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
> Reporter: Tsz Wo Nicholas Sze
> Assignee: Tsz Wo Nicholas Sze
> Attachments: h7276_20141021.patch, h7276_20141022.patch,
> h7276_20141023.patch, h7276_20141024.patch
>
>
> When there are a lot of DFSOutputStream's writing concurrently, the number of
> outstanding packets could be large. The byte arrays created by those packets
> could occupy a lot of memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
