[
https://issues.apache.org/jira/browse/HDFS-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295013#comment-14295013
]
Leitao Guo commented on HDFS-7692:
----------------------------------
When we upgrade HDFS from 2.0.0 to 2.5.0 with datanodes have 12 dataDirs and
each dataDir has 2.5TB data, upgrade of block pool at each dataDir cost us
about 25 minutes and the total time is 5 hours. This is really time consuming,
especially when datanodes have more dataDirs and more data.
I will submit the patch later after some tests.
> BlockPoolSliceStorage#loadBpStorageDirectories(...) should support
> MultiThread to speedup the upgrade of block pool at multi storage directories.
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-7692
> URL: https://issues.apache.org/jira/browse/HDFS-7692
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Affects Versions: 2.5.2
> Reporter: Leitao Guo
>
> {code:title=BlockPoolSliceStorage#loadBpStorageDirectories(...)|borderStyle=solid}
> for (File dataDir : dataDirs) {
> if (containsStorageDir(dataDir)) {
> throw new IOException(
> "BlockPoolSliceStorage.recoverTransitionRead: " +
> "attempt to load an used block storage: " + dataDir);
> }
> StorageDirectory sd =
> loadStorageDirectory(datanode, nsInfo, dataDir, startOpt);
> succeedDirs.add(sd);
> }
> {code}
> In the above code the storage directories will be analyzed one by one, which
> is really time consuming when upgrading HDFS with datanodes have dozens of
> large volumes. MultiThread dataDirs analyzing should be supported here to
> speedup upgrade.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
