[
https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344735#comment-14344735
]
Yi Liu commented on HDFS-7836:
------------------------------
Good proposal, Colin and Charles.
{quote}
Reduce lock contention for the FSNamesystem lock
Enable concurrent processing of block reports
{quote}
Maybe at the first phase, we can separate locks for block reports from FSN
lock. Even so, It's a good improvement and include bulk of work.
Then we can do further improvement to have multiple locks (stripes) for block
reports .
{quote}
Use offheap to reduce the Java heap size of the NameNode
{quote}
Generally we have some ways to use offheap, 1) directbuffer, it's not fit to
our case. 2) Sun Unsafe. 3) Write native code and allocate memory ourselves. It
seems we are going to use #3?
{quote}
We probably want to support dynamically growing the hash table, to avoid
putting too much of a
burden on the administrator to configure the size
{quote}
If so, we still need to have an initial capacity for the hash table, otherwise
there are lots of rehash when the load factor reaches the threshold.
How about we do the same thing as what we currently do in java blockmap, using
2% of totoal memory? Then we don't need the load factor and assume the table
rows number are enough?
> BlockManager Scalability Improvements
> -------------------------------------
>
> Key: HDFS-7836
> URL: https://issues.apache.org/jira/browse/HDFS-7836
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Charles Lamb
> Assignee: Charles Lamb
> Attachments: BlockManagerScalabilityImprovementsDesign.pdf
>
>
> Improvements to BlockManager scalability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
