[
https://issues.apache.org/jira/browse/HDFS-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590910#comment-14590910
]
Colin Patrick McCabe commented on HDFS-8578:
--------------------------------------------
bq. Any concerns about overloading the controller?
In general, the upgrade workload is creating a bunch of hardlinks, often one
per block file. This is not a large amount of I/O in terms of bandwidth.
Normally it completes in a second or two. The only cases we have seen problems
are where write caching is turned off on the hard disks, forcing a lot of
non-sequential I/O to update the inode entries. I would also argue that it is
Linux's responsibility to manage sending commands to the disk controller and
backing off (putting the user mode process to sleep) if there are too many in
the pipe. So I don't see any concerns here about overloading the disk
controller controller.
> On upgrade, Datanode should process all storage/data dirs in parallel
> ---------------------------------------------------------------------
>
> Key: HDFS-8578
> URL: https://issues.apache.org/jira/browse/HDFS-8578
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Reporter: Raju Bairishetti
> Priority: Critical
> Attachments: HDFS-8578-01.patch, HDFS-8578-02.patch
>
>
> Right now, during upgrades datanode is processing all the storage dirs
> sequentially. Assume it takes ~20 mins to process a single storage dir then
> datanode which has ~10 disks will take around 3hours to come up.
> *BlockPoolSliceStorage.java*
> {code}
> for (int idx = 0; idx < getNumStorageDirs(); idx++) {
> doTransition(datanode, getStorageDir(idx), nsInfo, startOpt);
> assert getCTime() == nsInfo.getCTime()
> : "Data-node and name-node CTimes must be the same.";
> }
> {code}
> It would save lots of time during major upgrades if datanode process all
> storagedirs/disks parallelly.
> Can we make datanode to process all storage dirs parallelly?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
