[
https://issues.apache.org/jira/browse/HDFS-16220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17422417#comment-17422417
]
Konstantin Shvachko edited comment on HDFS-16220 at 9/29/21, 11:22 PM:
-----------------------------------------------------------------------
# I don't think we should make Key depth configurable. We should generalize the
long[] into a Key class, then we will be able to configure any Key class and
load it using a Key factory. Different Key classes then can have different
depths. The problem here is that one should be able to construct the keys while
loading INodes from the image, so that they could be placed into the right
partitions.
# Number of partitions should in the end be configurable. It should depend on
the number of cores on your server. Increasing the number of partitions does
not necessarily increase the parallelism because at any moment the CPU cannot
support more threads than the number of cores. So this change is useful, but
not critical.
And the main problem here is to be able to rebuild new partitions while
reloading the fsimage. If you upgraded your NameNode to a server with more
cores you should be able to adjust the number of partitions.
was (Author: shv):
# I don't think we should make Key depth configurable. We should generalize the
long[] into a Key class, then we will be able to configure any Key class and
load it using a Key factory. Different Key classes then can have different
depths. The problem here is that one should be able to construct the keys while
loading INodes from the image, so that they could be placed into the right
partitions.
# Number of partitions should in the end be configurable. It should depend on
the number of cores on your server. Increasing the number of partitions does
not necessarily increase the parallelism because at any moment the CPU cannot
support more threads than the number of cores. So this change is useful, but
not critical.
> [FGL]Configurable INodeMap#NAMESPACE_KEY_DEPTH&NUM_RANGES_STATIC
> ----------------------------------------------------------------
>
> Key: HDFS-16220
> URL: https://issues.apache.org/jira/browse/HDFS-16220
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs, namenode
> Reporter: JiangHua Zhu
> Assignee: JiangHua Zhu
> Priority: Major
> Labels: pull-request-available
> Attachments: debug1.jpg, debug2.jpg
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> In INodeMap, NAMESPACE_KEY_DEPTH and NUM_RANGES_STATIC are a fixed value, we
> should make it configurable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
