[jira] [Commented] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13850530#comment-13850530
]
Hudson commented on HDFS-5454:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1641 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1641/])
HDFS-5454. DataNode UUID should be assigned prior to FsDataset initialization.
(Arpit Agarwal) (arp:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551296)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeInitStorage.java
> DataNode UUID should be assigned prior to FsDataset initialization
> --
>
> Key: HDFS-5454
> URL: https://issues.apache.org/jira/browse/HDFS-5454
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: Eric Sirianni
>Assignee: Arpit Agarwal
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HDFS-5454.01.patch, HDFS-5454.02.patch,
> HDFS-5454.03.patch
>
>
> The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
> point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
> As the {{DataStorage}} object is an input to the {{FsDataset}} factory
> method, it is desirable for it to be fully populated with a UUID at this
> point. In particular, our {{FsDatasetSpi}} implementation relies upon the
> DataNode UUID as a key to access our underlying block storage device.
> This also appears to be a regression compared to Hadoop 1.x - our 1.x
> {{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
> fully traced through the code, but I suspect this came from the
> {{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
> namenodes.
> With HDFS-5448, the DataNode is now responsible for generating its own UUID.
> This greatly simplifies the fix. Move the UUID check/generation in from
> {{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
> naturally co-locates UUID generation immediately subsequent to the read of
> the UUID from the {{DataStorage}} properties file.
> {code}
> private void initStorage(final NamespaceInfo nsInfo) throws IOException {
> // ...
> final String bpid = nsInfo.getBlockPoolID();
> //read storage info, lock data dirs and transition fs state if necessary
> storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
>
> // SUGGESTED NEW PLACE TO CHECK DATANODE UUID
> checkDatanodeUuid();
> // ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13850442#comment-13850442
]
Hudson commented on HDFS-5454:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1615 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1615/])
HDFS-5454. DataNode UUID should be assigned prior to FsDataset initialization.
(Arpit Agarwal) (arp:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551296)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeInitStorage.java
> DataNode UUID should be assigned prior to FsDataset initialization
> --
>
> Key: HDFS-5454
> URL: https://issues.apache.org/jira/browse/HDFS-5454
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: Eric Sirianni
>Assignee: Arpit Agarwal
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HDFS-5454.01.patch, HDFS-5454.02.patch,
> HDFS-5454.03.patch
>
>
> The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
> point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
> As the {{DataStorage}} object is an input to the {{FsDataset}} factory
> method, it is desirable for it to be fully populated with a UUID at this
> point. In particular, our {{FsDatasetSpi}} implementation relies upon the
> DataNode UUID as a key to access our underlying block storage device.
> This also appears to be a regression compared to Hadoop 1.x - our 1.x
> {{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
> fully traced through the code, but I suspect this came from the
> {{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
> namenodes.
> With HDFS-5448, the DataNode is now responsible for generating its own UUID.
> This greatly simplifies the fix. Move the UUID check/generation in from
> {{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
> naturally co-locates UUID generation immediately subsequent to the read of
> the UUID from the {{DataStorage}} properties file.
> {code}
> private void initStorage(final NamespaceInfo nsInfo) throws IOException {
> // ...
> final String bpid = nsInfo.getBlockPoolID();
> //read storage info, lock data dirs and transition fs state if necessary
> storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
>
> // SUGGESTED NEW PLACE TO CHECK DATANODE UUID
> checkDatanodeUuid();
> // ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13850346#comment-13850346
]
Hudson commented on HDFS-5454:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #424 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/424/])
HDFS-5454. DataNode UUID should be assigned prior to FsDataset initialization.
(Arpit Agarwal) (arp:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551296)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeInitStorage.java
> DataNode UUID should be assigned prior to FsDataset initialization
> --
>
> Key: HDFS-5454
> URL: https://issues.apache.org/jira/browse/HDFS-5454
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: Eric Sirianni
>Assignee: Arpit Agarwal
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HDFS-5454.01.patch, HDFS-5454.02.patch,
> HDFS-5454.03.patch
>
>
> The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
> point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
> As the {{DataStorage}} object is an input to the {{FsDataset}} factory
> method, it is desirable for it to be fully populated with a UUID at this
> point. In particular, our {{FsDatasetSpi}} implementation relies upon the
> DataNode UUID as a key to access our underlying block storage device.
> This also appears to be a regression compared to Hadoop 1.x - our 1.x
> {{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
> fully traced through the code, but I suspect this came from the
> {{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
> namenodes.
> With HDFS-5448, the DataNode is now responsible for generating its own UUID.
> This greatly simplifies the fix. Move the UUID check/generation in from
> {{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
> naturally co-locates UUID generation immediately subsequent to the read of
> the UUID from the {{DataStorage}} properties file.
> {code}
> private void initStorage(final NamespaceInfo nsInfo) throws IOException {
> // ...
> final String bpid = nsInfo.getBlockPoolID();
> //read storage info, lock data dirs and transition fs state if necessary
> storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
>
> // SUGGESTED NEW PLACE TO CHECK DATANODE UUID
> checkDatanodeUuid();
> // ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849433#comment-13849433
]
Hudson commented on HDFS-5454:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #4889 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/4889/])
HDFS-5454. DataNode UUID should be assigned prior to FsDataset initialization.
(Arpit Agarwal) (arp:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1551296)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeInitStorage.java
> DataNode UUID should be assigned prior to FsDataset initialization
> --
>
> Key: HDFS-5454
> URL: https://issues.apache.org/jira/browse/HDFS-5454
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: Eric Sirianni
>Assignee: Arpit Agarwal
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HDFS-5454.01.patch, HDFS-5454.02.patch,
> HDFS-5454.03.patch
>
>
> The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
> point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
> As the {{DataStorage}} object is an input to the {{FsDataset}} factory
> method, it is desirable for it to be fully populated with a UUID at this
> point. In particular, our {{FsDatasetSpi}} implementation relies upon the
> DataNode UUID as a key to access our underlying block storage device.
> This also appears to be a regression compared to Hadoop 1.x - our 1.x
> {{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
> fully traced through the code, but I suspect this came from the
> {{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
> namenodes.
> With HDFS-5448, the DataNode is now responsible for generating its own UUID.
> This greatly simplifies the fix. Move the UUID check/generation in from
> {{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
> naturally co-locates UUID generation immediately subsequent to the read of
> the UUID from the {{DataStorage}} properties file.
> {code}
> private void initStorage(final NamespaceInfo nsInfo) throws IOException {
> // ...
> final String bpid = nsInfo.getBlockPoolID();
> //read storage info, lock data dirs and transition fs state if necessary
> storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
>
> // SUGGESTED NEW PLACE TO CHECK DATANODE UUID
> checkDatanodeUuid();
> // ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848820#comment-13848820
]
Junping Du commented on HDFS-5454:
--
+1. Patch looks good to me. Thanks for this refactoring. Aript!
> DataNode UUID should be assigned prior to FsDataset initialization
> --
>
> Key: HDFS-5454
> URL: https://issues.apache.org/jira/browse/HDFS-5454
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: Eric Sirianni
>Priority: Minor
> Attachments: HDFS-5454.01.patch, HDFS-5454.02.patch,
> HDFS-5454.03.patch
>
>
> The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
> point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
> As the {{DataStorage}} object is an input to the {{FsDataset}} factory
> method, it is desirable for it to be fully populated with a UUID at this
> point. In particular, our {{FsDatasetSpi}} implementation relies upon the
> DataNode UUID as a key to access our underlying block storage device.
> This also appears to be a regression compared to Hadoop 1.x - our 1.x
> {{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
> fully traced through the code, but I suspect this came from the
> {{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
> namenodes.
> With HDFS-5448, the DataNode is now responsible for generating its own UUID.
> This greatly simplifies the fix. Move the UUID check/generation in from
> {{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
> naturally co-locates UUID generation immediately subsequent to the read of
> the UUID from the {{DataStorage}} properties file.
> {code}
> private void initStorage(final NamespaceInfo nsInfo) throws IOException {
> // ...
> final String bpid = nsInfo.getBlockPoolID();
> //read storage info, lock data dirs and transition fs state if necessary
> storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
>
> // SUGGESTED NEW PLACE TO CHECK DATANODE UUID
> checkDatanodeUuid();
> // ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848512#comment-13848512
]
Hadoop QA commented on HDFS-5454:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618793/HDFS-5454.03.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5723//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5723//console
This message is automatically generated.
> DataNode UUID should be assigned prior to FsDataset initialization
> --
>
> Key: HDFS-5454
> URL: https://issues.apache.org/jira/browse/HDFS-5454
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: Eric Sirianni
>Priority: Minor
> Attachments: HDFS-5454.01.patch, HDFS-5454.02.patch,
> HDFS-5454.03.patch
>
>
> The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
> point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
> As the {{DataStorage}} object is an input to the {{FsDataset}} factory
> method, it is desirable for it to be fully populated with a UUID at this
> point. In particular, our {{FsDatasetSpi}} implementation relies upon the
> DataNode UUID as a key to access our underlying block storage device.
> This also appears to be a regression compared to Hadoop 1.x - our 1.x
> {{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
> fully traced through the code, but I suspect this came from the
> {{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
> namenodes.
> With HDFS-5448, the DataNode is now responsible for generating its own UUID.
> This greatly simplifies the fix. Move the UUID check/generation in from
> {{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
> naturally co-locates UUID generation immediately subsequent to the read of
> the UUID from the {{DataStorage}} properties file.
> {code}
> private void initStorage(final NamespaceInfo nsInfo) throws IOException {
> // ...
> final String bpid = nsInfo.getBlockPoolID();
> //read storage info, lock data dirs and transition fs state if necessary
> storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
>
> // SUGGESTED NEW PLACE TO CHECK DATANODE UUID
> checkDatanodeUuid();
> // ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13847985#comment-13847985
]
Hadoop QA commented on HDFS-5454:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618671/HDFS-5454.02.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5713//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5713//console
This message is automatically generated.
> DataNode UUID should be assigned prior to FsDataset initialization
> --
>
> Key: HDFS-5454
> URL: https://issues.apache.org/jira/browse/HDFS-5454
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: Eric Sirianni
>Priority: Minor
> Attachments: HDFS-5454.01.patch, HDFS-5454.02.patch
>
>
> The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
> point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
> As the {{DataStorage}} object is an input to the {{FsDataset}} factory
> method, it is desirable for it to be fully populated with a UUID at this
> point. In particular, our {{FsDatasetSpi}} implementation relies upon the
> DataNode UUID as a key to access our underlying block storage device.
> This also appears to be a regression compared to Hadoop 1.x - our 1.x
> {{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
> fully traced through the code, but I suspect this came from the
> {{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
> namenodes.
> With HDFS-5448, the DataNode is now responsible for generating its own UUID.
> This greatly simplifies the fix. Move the UUID check/generation in from
> {{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
> naturally co-locates UUID generation immediately subsequent to the read of
> the UUID from the {{DataStorage}} properties file.
> {code}
> private void initStorage(final NamespaceInfo nsInfo) throws IOException {
> // ...
> final String bpid = nsInfo.getBlockPoolID();
> //read storage info, lock data dirs and transition fs state if necessary
> storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
>
> // SUGGESTED NEW PLACE TO CHECK DATANODE UUID
> checkDatanodeUuid();
> // ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13846955#comment-13846955
]
Hadoop QA commented on HDFS-5454:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618477/HDFS-5454.01.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage
org.apache.hadoop.hdfs.TestPread
org.apache.hadoop.hdfs.TestReplication
org.apache.hadoop.hdfs.TestSmallBlock
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
org.apache.hadoop.hdfs.TestFileCreation
org.apache.hadoop.hdfs.TestSetrepIncreasing
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
org.apache.hadoop.hdfs.server.namenode.TestFileLimit
org.apache.hadoop.hdfs.server.balancer.TestBalancer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5708//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5708//console
This message is automatically generated.
> DataNode UUID should be assigned prior to FsDataset initialization
> --
>
> Key: HDFS-5454
> URL: https://issues.apache.org/jira/browse/HDFS-5454
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: Eric Sirianni
>Priority: Minor
> Attachments: HDFS-5454.01.patch
>
>
> The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
> point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
> As the {{DataStorage}} object is an input to the {{FsDataset}} factory
> method, it is desirable for it to be fully populated with a UUID at this
> point. In particular, our {{FsDatasetSpi}} implementation relies upon the
> DataNode UUID as a key to access our underlying block storage device.
> This also appears to be a regression compared to Hadoop 1.x - our 1.x
> {{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
> fully traced through the code, but I suspect this came from the
> {{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
> namenodes.
> With HDFS-5448, the DataNode is now responsible for generating its own UUID.
> This greatly simplifies the fix. Move the UUID check/generation in from
> {{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
> naturally co-locates UUID generation immediately subsequent to the read of
> the UUID from the {{DataStorage}} properties file.
> {code}
> private void initStorage(final NamespaceInfo nsInfo) throws IOException {
> // ...
> final String bpid = nsInfo.getBlockPoolID();
> //read storage info, lock data dirs and transition fs state if necessary
> storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
>
> // SUGGESTED NEW PLACE TO CHECK DATANODE UUID
> checkDatanodeUuid();
> // ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13846773#comment-13846773
]
Arpit Agarwal commented on HDFS-5454:
-
Thanks for reviewing Eric.
I apologize for not asking whether you were already working on it. :-(
> DataNode UUID should be assigned prior to FsDataset initialization
> --
>
> Key: HDFS-5454
> URL: https://issues.apache.org/jira/browse/HDFS-5454
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: Eric Sirianni
>Priority: Minor
> Attachments: HDFS-5454.01.patch
>
>
> The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
> point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
> As the {{DataStorage}} object is an input to the {{FsDataset}} factory
> method, it is desirable for it to be fully populated with a UUID at this
> point. In particular, our {{FsDatasetSpi}} implementation relies upon the
> DataNode UUID as a key to access our underlying block storage device.
> This also appears to be a regression compared to Hadoop 1.x - our 1.x
> {{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
> fully traced through the code, but I suspect this came from the
> {{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
> namenodes.
> With HDFS-5448, the DataNode is now responsible for generating its own UUID.
> This greatly simplifies the fix. Move the UUID check/generation in from
> {{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
> naturally co-locates UUID generation immediately subsequent to the read of
> the UUID from the {{DataStorage}} properties file.
> {code}
> private void initStorage(final NamespaceInfo nsInfo) throws IOException {
> // ...
> final String bpid = nsInfo.getBlockPoolID();
> //read storage info, lock data dirs and transition fs state if necessary
> storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
>
> // SUGGESTED NEW PLACE TO CHECK DATANODE UUID
> checkDatanodeUuid();
> // ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13846763#comment-13846763
]
Eric Sirianni commented on HDFS-5454:
-
+1 for the patch. I've made a similar change internally and can confirm that
it fixes the issue we are seeing in our {{FsDatasetSpi}} plugin.
> DataNode UUID should be assigned prior to FsDataset initialization
> --
>
> Key: HDFS-5454
> URL: https://issues.apache.org/jira/browse/HDFS-5454
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: Eric Sirianni
>Priority: Minor
> Attachments: HDFS-5454.01.patch
>
>
> The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
> point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
> As the {{DataStorage}} object is an input to the {{FsDataset}} factory
> method, it is desirable for it to be fully populated with a UUID at this
> point. In particular, our {{FsDatasetSpi}} implementation relies upon the
> DataNode UUID as a key to access our underlying block storage device.
> This also appears to be a regression compared to Hadoop 1.x - our 1.x
> {{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
> fully traced through the code, but I suspect this came from the
> {{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
> namenodes.
> With HDFS-5448, the DataNode is now responsible for generating its own UUID.
> This greatly simplifies the fix. Move the UUID check/generation in from
> {{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
> naturally co-locates UUID generation immediately subsequent to the read of
> the UUID from the {{DataStorage}} properties file.
> {code}
> private void initStorage(final NamespaceInfo nsInfo) throws IOException {
> // ...
> final String bpid = nsInfo.getBlockPoolID();
> //read storage info, lock data dirs and transition fs state if necessary
> storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
>
> // SUGGESTED NEW PLACE TO CHECK DATANODE UUID
> checkDatanodeUuid();
> // ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13846762#comment-13846762
]
Arpit Agarwal commented on HDFS-5454:
-
Patch attached. I made this a sub-task of HDFS-2832 since the fix depends on
those changes, and to make it easier to track for any subsequent merge work.
> DataNode UUID should be assigned prior to FsDataset initialization
> --
>
> Key: HDFS-5454
> URL: https://issues.apache.org/jira/browse/HDFS-5454
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: 3.0.0
>Reporter: Eric Sirianni
>Priority: Minor
> Attachments: HDFS-5454.01.patch
>
>
> The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
> point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
> As the {{DataStorage}} object is an input to the {{FsDataset}} factory
> method, it is desirable for it to be fully populated with a UUID at this
> point. In particular, our {{FsDatasetSpi}} implementation relies upon the
> DataNode UUID as a key to access our underlying block storage device.
> This also appears to be a regression compared to Hadoop 1.x - our 1.x
> {{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
> fully traced through the code, but I suspect this came from the
> {{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
> namenodes.
> With HDFS-5448, the DataNode is now responsible for generating its own UUID.
> This greatly simplifies the fix. Move the UUID check/generation in from
> {{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
> naturally co-locates UUID generation immediately subsequent to the read of
> the UUID from the {{DataStorage}} properties file.
> {code}
> private void initStorage(final NamespaceInfo nsInfo) throws IOException {
> // ...
> final String bpid = nsInfo.getBlockPoolID();
> //read storage info, lock data dirs and transition fs state if necessary
> storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
>
> // SUGGESTED NEW PLACE TO CHECK DATANODE UUID
> checkDatanodeUuid();
> // ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13813182#comment-13813182
]
Arpit Agarwal commented on HDFS-5454:
-
The same issue exists in 2.2 except that the Datanode identifier is not a UUID.
I changed the affected version back to 2.2 which Eric used when filing.
> DataNode UUID should be assigned prior to FsDataset initialization
> --
>
> Key: HDFS-5454
> URL: https://issues.apache.org/jira/browse/HDFS-5454
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.2.0
>Reporter: Eric Sirianni
>Priority: Minor
>
> The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
> point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
> As the {{DataStorage}} object is an input to the {{FsDataset}} factory
> method, it is desirable for it to be fully populated with a UUID at this
> point. In particular, our {{FsDatasetSpi}} implementation relies upon the
> DataNode UUID as a key to access our underlying block storage device.
> This also appears to be a regression compared to Hadoop 1.x - our 1.x
> {{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
> fully traced through the code, but I suspect this came from the
> {{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
> namenodes.
> With HDFS-5448, the DataNode is now responsible for generating its own UUID.
> This greatly simplifies the fix. Move the UUID check/generation in from
> {{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
> naturally co-locates UUID generation immediately subsequent to the read of
> the UUID from the {{DataStorage}} properties file.
> {code}
> private void initStorage(final NamespaceInfo nsInfo) throws IOException {
> // ...
> final String bpid = nsInfo.getBlockPoolID();
> //read storage info, lock data dirs and transition fs state if necessary
> storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
>
> // SUGGESTED NEW PLACE TO CHECK DATANODE UUID
> checkDatanodeUuid();
> // ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13813117#comment-13813117
]
Arpit Agarwal commented on HDFS-5454:
-
The javadoc for {{DataNode#initStorage}} does not appear to match the code.
{code}
/**
* Initializes the {@link #data}. The initialization is done only once, when
* handshake with the the first namenode is completed.
*/
private void initStorage(final NamespaceInfo nsInfo) throws IOException {
{code}
However initStorage is invoked before the handshake completes, in
{{BPServiceActor#connectToNNAndHandshake}}
{code}
// Verify that this matches the other NN in this HA pair.
// This also initializes our block pool in the DN if we are
// the first NN connection for this BP.
bpos.verifyAndSetNamespaceInfo(nsInfo);<<<--- Calls initStorage.
// Second phase of the handshake with the NN.
register();
{code}
I am not sure if we need to reorder the calls. Would need to look at this
further.
> DataNode UUID should be assigned prior to FsDataset initialization
> --
>
> Key: HDFS-5454
> URL: https://issues.apache.org/jira/browse/HDFS-5454
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: Heterogeneous Storage (HDFS-2832)
>Reporter: Eric Sirianni
>Priority: Minor
>
> The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
> point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
> As the {{DataStorage}} object is an input to the {{FsDataset}} factory
> method, it is desirable for it to be fully populated with a UUID at this
> point. In particular, our {{FsDatasetSpi}} implementation relies upon the
> DataNode UUID as a key to access our underlying block storage device.
> This also appears to be a regression compared to Hadoop 1.x - our 1.x
> {{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
> fully traced through the code, but I suspect this came from the
> {{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
> namenodes.
> With HDFS-5448, the DataNode is now responsible for generating its own UUID.
> This greatly simplifies the fix. Move the UUID check/generation in from
> {{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
> naturally co-locates UUID generation immediately subsequent to the read of
> the UUID from the {{DataStorage}} properties file.
> {code}
> private void initStorage(final NamespaceInfo nsInfo) throws IOException {
> // ...
> final String bpid = nsInfo.getBlockPoolID();
> //read storage info, lock data dirs and transition fs state if necessary
> storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
>
> // SUGGESTED NEW PLACE TO CHECK DATANODE UUID
> checkDatanodeUuid();
> // ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
