Eric Sirianni created HDFS-5505:
-----------------------------------
Summary: DataStorage.format() should ignore NamespaceID
Key: HDFS-5505
URL: https://issues.apache.org/jira/browse/HDFS-5505
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, federation
Affects Versions: 2.2.0
Reporter: Eric Sirianni
Priority: Minor
I'm observing inconsistencies in the {{DataStorage.namespaceID}} value passed
in to my {{FsDatasetSpi}} plugin:
* Upon _initial_ startup (when the DataNode is formatted), the storage
{{DataStorage.namespaceID}} value is populated with the namespace ID from the
NameNode (or an arbitrary namenode in a federated setup).
* Upon subsequent startups the {{DataStorage.namespaceID}} value is {{NULL}}.
The issue appears to be different handling of the {{DataStorage.namespaceID}}
value in {{DataStorage.format()}} vs. {{DataStorage.setFieldsFromProperties()}}:
{code:title=DataStorage.java}
void format(StorageDirectory sd, NamespaceInfo nsInfo, String datanodeUuid)
throws IOException {
sd.clearDirectory(); // create directory
this.layoutVersion = HdfsConstants.LAYOUT_VERSION;
this.clusterID = nsInfo.getClusterID();
this.namespaceID = nsInfo.getNamespaceID();
}
....
protected void setFieldsFromProperties(Properties props, StorageDirectory sd)
// Read NamespaceID in version before federation
if (!LayoutVersion.supports(Feature.FEDERATION, layoutVersion)) {
setNamespaceID(props, sd);
}
{code}
If I understand correctly, the proper fix is to leave namespaceID *unset* in
{{DataStorage.format()}}. The namespaceID here is invalid anyway since it is
susceptible to race conditions in federated NameNode setups -- it will happen
to match the namespaceID of the first {{BPOfferService}} thread that
successfully handshakes.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
