[
https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14338865#comment-14338865
]
Chris Nauroth commented on HDFS-7836:
-------------------------------------
+1 for the proposal overall. Thanks for the great write-up, Charles and Colin.
I have questions on a few details:
bq. The Java heap size will be reduced, because the BlockManager data will be
offÂ-heap.
Do you intend to enforce an upper limit on growth of the off-heap allocation?
If so, do you see this as a new configuration property or as a function of an
existing parameter (i.e. equal to max heap with the consideration that the
block map takes ~50% of the heap now)? {{-Xmx}} alone will no longer be
sufficient to define a ceiling for RAM utilization by the NameNode process.
This can be important in deployments that choose to co-locate other Hadoop
daemons on the same hosts as the NameNode.
bq. malloc is also accessible without the use of JNI via the Unsafe package.
Can I take this to mean that there will be no new native code written as part
of this project? Of course, we can always do a native code implementation
later if use of a private Sun API becomes problematic, but I wanted to
understand the code footprint for the current proposal. Avoiding native code
entirely would be nice, because it reduces the scope of testing efforts across
multiple platforms.
bq. The offÂheaping code should have the option to use on-heap memory.
Are you proposing that off-heaping is an opt-in feature that must be explicitly
enabled in configuration, or are you proposing that off-heaping will be the new
default behavior? Arguably, jumping to off-heaping as the default could be
seen as a backwards-incompatibility, because it might be unsafe to deploy the
feature without simultaneous down-tuning the NameNode max heap size. Some
might see that as backwards-incompatible with existing configurations.
> BlockManager Scalability Improvements
> -------------------------------------
>
> Key: HDFS-7836
> URL: https://issues.apache.org/jira/browse/HDFS-7836
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Charles Lamb
> Assignee: Charles Lamb
> Attachments: BlockManagerScalabilityImprovementsDesign.pdf
>
>
> Improvements to BlockManager scalability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
