[
https://issues.apache.org/jira/browse/HDFS-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041929#comment-15041929
]
Vinayakumar B commented on HDFS-8578:
-------------------------------------
Thanks [~jrottinghuis] for review.
bq. With 12 disks and 3 namespaces that would mean 36 parallel threads right?
Parallelism is within blockpool/namespace. Not all blockpools will be in
parallel. {{Datanode#initStorage()}} have the synchronization.
{code} synchronized (this) {
storage.recoverTransitionRead(this, nsInfo, dataDirs, startOpt);
}{code}
bq. Perhaps it would be better to make this configurable. It can default to 0,
meaning as parallel as possible,
Earlier versions patch of patch had configurable value. But it was changed to #
of storage dirs on suggestion of [~wheat9].
bq. I can imagine that 12 parallel threads (1 per disk in the above example)
might turn out to be a reasonable compromise for some use cases.
Yes, this is what happens.
{quote} Wrt. catching InterruptedException, even though the newer patches do
distinguish between ExecutionException | CancellationException and others, it
is still good form to leave the interrupt status intact by re-interupting the
current thread before returning:
Or in this case perhaps yield and break out of the for loop, because the
interrupt indicated that the thread should clean up and wrap up asap.{quote}
Yes, Will upload updated patch to re-interrupt thread.
Its would be good to wait till all the threads finish instead of leaving fs in
bad state.
> On upgrade, Datanode should process all storage/data dirs in parallel
> ---------------------------------------------------------------------
>
> Key: HDFS-8578
> URL: https://issues.apache.org/jira/browse/HDFS-8578
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Reporter: Raju Bairishetti
> Assignee: Vinayakumar B
> Priority: Critical
> Attachments: HDFS-8578-01.patch, HDFS-8578-02.patch,
> HDFS-8578-03.patch, HDFS-8578-04.patch, HDFS-8578-05.patch,
> HDFS-8578-06.patch, HDFS-8578-07.patch, HDFS-8578-08.patch,
> HDFS-8578-09.patch, HDFS-8578-10.patch, HDFS-8578-11.patch,
> HDFS-8578-branch-2.6.0.patch
>
>
> Right now, during upgrades datanode is processing all the storage dirs
> sequentially. Assume it takes ~20 mins to process a single storage dir then
> datanode which has ~10 disks will take around 3hours to come up.
> *BlockPoolSliceStorage.java*
> {code}
> for (int idx = 0; idx < getNumStorageDirs(); idx++) {
> doTransition(datanode, getStorageDir(idx), nsInfo, startOpt);
> assert getCTime() == nsInfo.getCTime()
> : "Data-node and name-node CTimes must be the same.";
> }
> {code}
> It would save lots of time during major upgrades if datanode process all
> storagedirs/disks parallelly.
> Can we make datanode to process all storage dirs parallelly?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
