[
https://issues.apache.org/jira/browse/HDFS-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15073904#comment-15073904
]
Vinayakumar B commented on HDFS-8578:
-------------------------------------
For h8578_20151213.patch
1. I think executor service should be shutdown once the loading is over.
{code} SubmissionService(int numThreads) {
super(Executors.newFixedThreadPool(numThreads));
this.numThreads = numThreads;
}
{code}
2. I think there is some change in behavior, in case of any upgrade failure for
any one of the disk.
{code}
final int taskCount = service.getCount();
LOG.info("loadBlockPoolSliceStorage: processing " + service);
for(int i = 0; i < taskCount; i++) {
success.add(service.takeResult());
}
{code}
In above part of the code after patch, if any disk fails to upgrade, entire
upgrade will fail, even though other upgrades in-progress.. But earlier, only
that disk was excluded and upgrade of other disks were continued. Need to
handle exceptions here and continue with other disks.
Nit:
3. One checkstyle comment, line length > 80 also can be corrected.
Other changes looks fine.
+1 once addressed.
> On upgrade, Datanode should process all storage/data dirs in parallel
> ---------------------------------------------------------------------
>
> Key: HDFS-8578
> URL: https://issues.apache.org/jira/browse/HDFS-8578
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Reporter: Raju Bairishetti
> Assignee: Vinayakumar B
> Priority: Critical
> Attachments: HDFS-8578-01.patch, HDFS-8578-02.patch,
> HDFS-8578-03.patch, HDFS-8578-04.patch, HDFS-8578-05.patch,
> HDFS-8578-06.patch, HDFS-8578-07.patch, HDFS-8578-08.patch,
> HDFS-8578-09.patch, HDFS-8578-10.patch, HDFS-8578-11.patch,
> HDFS-8578-12.patch, HDFS-8578-13.patch, HDFS-8578-14.patch,
> HDFS-8578-15.patch, HDFS-8578-16.patch, HDFS-8578-17.patch,
> HDFS-8578-branch-2.6.0.patch, HDFS-8578-branch-2.7-001.patch,
> HDFS-8578-branch-2.7-002.patch, HDFS-8578-branch-2.7-003.patch,
> h8578_20151210.patch, h8578_20151211.patch, h8578_20151211b.patch,
> h8578_20151212.patch, h8578_20151213.patch
>
>
> Right now, during upgrades datanode is processing all the storage dirs
> sequentially. Assume it takes ~20 mins to process a single storage dir then
> datanode which has ~10 disks will take around 3hours to come up.
> *BlockPoolSliceStorage.java*
> {code}
> for (int idx = 0; idx < getNumStorageDirs(); idx++) {
> doTransition(datanode, getStorageDir(idx), nsInfo, startOpt);
> assert getCTime() == nsInfo.getCTime()
> : "Data-node and name-node CTimes must be the same.";
> }
> {code}
> It would save lots of time during major upgrades if datanode process all
> storagedirs/disks parallelly.
> Can we make datanode to process all storage dirs parallelly?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
