[
https://issues.apache.org/jira/browse/HDFS-8792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659302#comment-14659302
]
Yi Liu edited comment on HDFS-8792 at 8/6/15 1:32 AM:
------------------------------------------------------
Thanks [~cmccabe] for the review.
{quote}
This JavaDoc comment is stale. It seems like it should be "the current
modification epoch" or similar.
{quote}
Will update it in the new patch right now.
{quote}
Can you put this in a separate JIRA? It's less obvious to me that this is a
win. Java HashMaps don't ever shrink, whereas TreeMap uses less memory when
elements are removed.
{quote}
Sure, I will put it in a separate JIRA.
That's right HashMap don't ever shrink when elements are removed, but TreeMap
entry needs to store more (memory) references (left, right, parent) than
HashMap entry (only one reference next), even when there is element removing
and cause some entry empty, the empty HashMap entry is just a {{null}}
reference (4 bytes), so they are close at this point. On the other hand, the
key of {{excessReplicateMap}} is datanode uuid, so the entries number is almost
fixed, so HashMap memory is good than TreeMap memory in this case. I think
the most important is the search/insert/remove performance, HashMap is
absolutely better than TreeMap. Because we don't need to sort, we should use
HashMap instead of TreeMap.
was (Author: hitliuyi):
Thanks [~cmccabe] for the review.
{quote}
This JavaDoc comment is stale. It seems like it should be "the current
modification epoch" or similar.
{quote}
Will update it in the new patch right now.
{quote}
Can you put this in a separate JIRA? It's less obvious to me that this is a
win. Java HashMaps don't ever shrink, whereas TreeMap uses less memory when
elements are removed.
{quote}
Sure, I will put it in a separate JIRA.
That's right HashMap don't ever shrink when elements are removed, but TreeMap
entry needs to store more (memory) references (left, right, parent) than
HashMap entry (only one reference next), even when there is element removing,
the empty HashMap entry is just a {{null}} reference (4 bytes), so they are
nearly at this point. On the other hand, the key of {{excessReplicateMap}} is
datanode uuid, so the entries number is almost fixed, so HashMap memory is good
than TreeMap memory in this case. I think the most important is the
search/insert/remove performance, HashMap is absolutely better than TreeMap.
Because we don't need to sort, we should use HashMap instead of TreeMap.
> Improve BlockManager#postponedMisreplicatedBlocks
> -------------------------------------------------
>
> Key: HDFS-8792
> URL: https://issues.apache.org/jira/browse/HDFS-8792
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Yi Liu
> Assignee: Yi Liu
> Attachments: HDFS-8792.001.patch, HDFS-8792.002.patch,
> HDFS-8792.003.patch
>
>
> {{LightWeightHashSet}} requires fewer memory than java hashset.
> Furthermore, for {{excessReplicateMap}}, we can use {{HashMap}} instead of
> {{TreeMap}} instead, since no need to sort.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
