[
https://issues.apache.org/jira/browse/HDFS-9730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15127613#comment-15127613
]
Vinayakumar B commented on HDFS-9730:
-------------------------------------
Thanks [~kihwal] for reporting this. Yes, indeed {{createStorageId(..)}} is not
called during layout upgrade.
bq. We still call createStorageID(..) the upgrade path. The call is moved to
doUgrade(..) as shown below (line 802 in DataStorage).
Yes [~szetszwo], its called. But it will hit only during layout upgrade from
pre-federation layouts. Otherwise it will just write latest layoutVersion to
VERSION file and return. does nothing else.
{code} if (DataNodeLayoutVersion.supports(
LayoutVersion.Feature.FEDERATION, layoutVersion)) {
// The VERSION file is already read in. Override the layoutVersion
// field and overwrite the file. The upgrade work is handled by
// {@link BlockPoolSliceStorage#doUpgrade}
LOG.info("Updating layout version from " + layoutVersion + " to "
+ HdfsServerConstants.DATANODE_LAYOUT_VERSION + " for storage "
+ sd.getRoot());
layoutVersion = HdfsServerConstants.DATANODE_LAYOUT_VERSION;
writeProperties(sd);
return;
}{code}
Change created this is
{code} // do upgrade
if (this.layoutVersion > HdfsServerConstants.DATANODE_LAYOUT_VERSION) {
- doUpgrade(datanode, sd, nsInfo); // upgrade
- createStorageID(sd, !haveValidStorageId);
- return;
+ doUpgrade(sd, nsInfo, conf); // upgrade
+ return true; // doUgrade already has written properties
}{code}
My Apologies for missing this during review :)
The fix would be as below.
{code}---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
@@ -743,6 +743,8 @@ void doUpgrade(final StorageDirectory sd, final
NamespaceInfo nsInfo,
LOG.info("Updating layout version from " + layoutVersion + " to "
+ HdfsServerConstants.DATANODE_LAYOUT_VERSION + " for storage "
+ sd.getRoot());
+ // check and regenerate storageId if required.
+ createStorageID(sd, layoutVersion);
layoutVersion = HdfsServerConstants.DATANODE_LAYOUT_VERSION;
writeProperties(sd);
return;
{code}
But, I have tried to run tests by changing layout version in my eclipse, didnt
hit failures though ;)
> Storage ID update does not happen when there is a layout change
> ---------------------------------------------------------------
>
> Key: HDFS-9730
> URL: https://issues.apache.org/jira/browse/HDFS-9730
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Kihwal Lee
> Assignee: Tsz Wo Nicholas Sze
>
> HDFS-9654 will cause test failures when we increment the datanode layout
> version next time.
> {noformat}
> TestDatanodeStartupFixesLegacyStorageIDs#testUpgradeFrom22via26FixesStorageIDs
> TestDatanodeStartupFixesLegacyStorageIDs#testUpgradeFrom22FixesStorageIDs
> {noformat}
> This is because createStorageID() is no longer called when it goes through
> the layout upgrade path.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
