[jira] [Commented] (HDFS-7035) Refactor DataStorage and BlockSlicePoolStorage

2014-09-15 Thread Lei (Eddy) Xu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14134467#comment-14134467
 ] 

Lei (Eddy) Xu commented on HDFS-7035:
-

Hi, all

I've run 
TestHDFSTrash,TestDistributedFileSystem,TestFileCreation,TestDataStorage,TestDataNodeRollingUpgrade,TestDatanodeStorageBase,TestBlockPoolSliceStorage,TestBPOfferService,TestFileAppend4,TestDataTransferKeepalive
 locally without failures. 

I think that the above failures are not related to this patch. 

 Refactor DataStorage and BlockSlicePoolStorage 
 ---

 Key: HDFS-7035
 URL: https://issues.apache.org/jira/browse/HDFS-7035
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: datanode
Affects Versions: 2.5.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
 Attachments: HDFS-7035.000.combo.patch, HDFS-7035.000.patch, 
 HDFS-7035.001.combo.patch, HDFS-7035.001.patch, HDFS-7035.002.patch, 
 HDFS-7035.003.patch, HDFS-7035.003.patch


 {{DataStorage}} and {{BlockPoolSliceStorage}} share many similar code path. 
 This jira extracts the common part of these two classes to simplify the logic 
 for both.
 This is the ground work for handling partial failures during hot swapping 
 volumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7035) Refactor DataStorage and BlockSlicePoolStorage

2014-09-13 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132694#comment-14132694
 ] 

Hadoop QA commented on HDFS-7035:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12668558/HDFS-7035.003.patch
  against trunk revision 98588cf.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-hdfs-project/hadoop-hdfs:

  org.apache.hadoop.hdfs.server.datanode.TestBPOfferService
  
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
  
org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
  org.apache.hadoop.hdfs.TestFileAppend4
  org.apache.hadoop.hdfs.TestDataTransferKeepalive

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/8016//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8016//console

This message is automatically generated.

 Refactor DataStorage and BlockSlicePoolStorage 
 ---

 Key: HDFS-7035
 URL: https://issues.apache.org/jira/browse/HDFS-7035
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: datanode
Affects Versions: 2.5.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
 Attachments: HDFS-7035.000.combo.patch, HDFS-7035.000.patch, 
 HDFS-7035.001.combo.patch, HDFS-7035.001.patch, HDFS-7035.002.patch, 
 HDFS-7035.003.patch, HDFS-7035.003.patch


 {{DataStorage}} and {{BlockPoolSliceStorage}} share many similar code path. 
 This jira extracts the common part of these two classes to simplify the logic 
 for both.
 This is the ground work for handling partial failures during hot swapping 
 volumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7035) Refactor DataStorage and BlockSlicePoolStorage

2014-09-12 Thread Lei (Eddy) Xu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131189#comment-14131189
 ] 

Lei (Eddy) Xu commented on HDFS-7035:
-

{{TestDataNodeVolumeFailureToleration}} is a related failure. I will address 
the failure soon.

 Refactor DataStorage and BlockSlicePoolStorage 
 ---

 Key: HDFS-7035
 URL: https://issues.apache.org/jira/browse/HDFS-7035
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: datanode
Affects Versions: 2.5.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
 Attachments: HDFS-7035.000.combo.patch, HDFS-7035.000.patch, 
 HDFS-7035.001.combo.patch, HDFS-7035.001.patch, HDFS-7035.002.patch


 {{DataStorage}} and {{BlockPoolSliceStorage}} share many similar code path. 
 This jira extracts the common part of these two classes to simplify the logic 
 for both.
 This is the ground work for handling partial failures during hot swapping 
 volumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7035) Refactor DataStorage and BlockSlicePoolStorage

2014-09-12 Thread Arpit Agarwal (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131198#comment-14131198
 ] 

Arpit Agarwal commented on HDFS-7035:
-

Hi Lei, could you please provide a summary or overview of the changes? Thanks.

 Refactor DataStorage and BlockSlicePoolStorage 
 ---

 Key: HDFS-7035
 URL: https://issues.apache.org/jira/browse/HDFS-7035
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: datanode
Affects Versions: 2.5.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
 Attachments: HDFS-7035.000.combo.patch, HDFS-7035.000.patch, 
 HDFS-7035.001.combo.patch, HDFS-7035.001.patch, HDFS-7035.002.patch


 {{DataStorage}} and {{BlockPoolSliceStorage}} share many similar code path. 
 This jira extracts the common part of these two classes to simplify the logic 
 for both.
 This is the ground work for handling partial failures during hot swapping 
 volumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7035) Refactor DataStorage and BlockSlicePoolStorage

2014-09-12 Thread Lei (Eddy) Xu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131845#comment-14131845
 ] 

Lei (Eddy) Xu commented on HDFS-7035:
-

Hi, [~arpitagarwal] Thanks very much for looking into this.

This patch serves three purposes: 

# For the volume management functionality, it refactors the loading storage 
directories, so that if there is a failure during the loading _a storage 
directory_,  the partial loaded metadata belonging to this storage directory is 
completed removed from {{DataStorage}} and {{BlockStorage}},  all partial 
loaded directories are unlocked, and etc. It is basically for more gracefully 
to handle the failures of loading volumes when {{DataNode}} is running.
# It eliminates the duplicated logic in {{DataStorage}} and 
{{BlockSlicePoolStorage}} to simplify the failure recovery logic needed to be 
handle from {{DataNode}}.
# Also it changes the order to add storage directories. In the original code, 
all storage directories for one namespace is loading in batch. This behavior is 
changed to add one storage directory for one namespace at a time, also for 
easier failure handling for upper level (e.g., HDFS-6727) in the hot swap 
volume scenarios.

[~arpitagarwal] do the above answer your questions? It would be great to if you 
could give me some suggestions on this.


 Refactor DataStorage and BlockSlicePoolStorage 
 ---

 Key: HDFS-7035
 URL: https://issues.apache.org/jira/browse/HDFS-7035
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: datanode
Affects Versions: 2.5.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
 Attachments: HDFS-7035.000.combo.patch, HDFS-7035.000.patch, 
 HDFS-7035.001.combo.patch, HDFS-7035.001.patch, HDFS-7035.002.patch


 {{DataStorage}} and {{BlockPoolSliceStorage}} share many similar code path. 
 This jira extracts the common part of these two classes to simplify the logic 
 for both.
 This is the ground work for handling partial failures during hot swapping 
 volumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7035) Refactor DataStorage and BlockSlicePoolStorage

2014-09-12 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132526#comment-14132526
 ] 

Hadoop QA commented on HDFS-7035:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12668501/HDFS-7035.003.patch
  against trunk revision a0ad975.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-hdfs-project/hadoop-hdfs:

  org.apache.hadoop.hdfs.server.datanode.TestDataStorage
  org.apache.hadoop.tracing.TestTracing
  org.apache.hadoop.hdfs.TestCrcCorruption

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/8012//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8012//console

This message is automatically generated.

 Refactor DataStorage and BlockSlicePoolStorage 
 ---

 Key: HDFS-7035
 URL: https://issues.apache.org/jira/browse/HDFS-7035
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: datanode
Affects Versions: 2.5.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
 Attachments: HDFS-7035.000.combo.patch, HDFS-7035.000.patch, 
 HDFS-7035.001.combo.patch, HDFS-7035.001.patch, HDFS-7035.002.patch, 
 HDFS-7035.003.patch


 {{DataStorage}} and {{BlockPoolSliceStorage}} share many similar code path. 
 This jira extracts the common part of these two classes to simplify the logic 
 for both.
 This is the ground work for handling partial failures during hot swapping 
 volumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7035) Refactor DataStorage and BlockSlicePoolStorage

2014-09-11 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131028#comment-14131028
 ] 

Hadoop QA commented on HDFS-7035:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12668197/HDFS-7035.002.patch
  against trunk revision 6c08339.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-hdfs-project/hadoop-hdfs:

  org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
  
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
  
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/8002//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8002//console

This message is automatically generated.

 Refactor DataStorage and BlockSlicePoolStorage 
 ---

 Key: HDFS-7035
 URL: https://issues.apache.org/jira/browse/HDFS-7035
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: datanode
Affects Versions: 2.5.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
 Attachments: HDFS-7035.000.combo.patch, HDFS-7035.000.patch, 
 HDFS-7035.001.combo.patch, HDFS-7035.001.patch, HDFS-7035.002.patch


 {{DataStorage}} and {{BlockPoolSliceStorage}} share many similar code path. 
 This jira extracts the common part of these two classes to simplify the logic 
 for both.
 This is the ground work for handling partial failures during hot swapping 
 volumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7035) Refactor DataStorage and BlockSlicePoolStorage

2014-09-10 Thread Lei (Eddy) Xu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14128896#comment-14128896
 ] 

Lei (Eddy) Xu commented on HDFS-7035:
-

I will update the patch to address those failures

 Refactor DataStorage and BlockSlicePoolStorage 
 ---

 Key: HDFS-7035
 URL: https://issues.apache.org/jira/browse/HDFS-7035
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: datanode
Affects Versions: 2.5.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
 Attachments: HDFS-7035.000.combo.patch, HDFS-7035.000.patch, 
 HDFS-7035.001.combo.patch, HDFS-7035.001.patch


 {{DataStorage}} and {{BlockPoolSliceStorage}} share many similar code path. 
 This jira extracts the common part of these two classes to simplify the logic 
 for both.
 This is the ground work for handling partial failures during hot swapping 
 volumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7035) Refactor DataStorage and BlockSlicePoolStorage

2014-09-09 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14127590#comment-14127590
 ] 

Hadoop QA commented on HDFS-7035:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12667494/HDFS-7035.000.combo.patch
  against trunk revision 3e8f353.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7974//console

This message is automatically generated.

 Refactor DataStorage and BlockSlicePoolStorage 
 ---

 Key: HDFS-7035
 URL: https://issues.apache.org/jira/browse/HDFS-7035
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: datanode
Affects Versions: 2.5.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
 Attachments: HDFS-7035.000.combo.patch, HDFS-7035.000.patch


 {{DataStorage}} and {{BlockPoolSliceStorage}} share many similar code path. 
 This jira extracts the common part of these two classes to simplify the logic 
 for both.
 This is the ground work for handling partial failures during hot swapping 
 volumes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-7035) Refactor DataStorage and BlockSlicePoolStorage

2014-09-09 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14127917#comment-14127917
 ] 

Hadoop QA commented on HDFS-7035:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12667501/HDFS-7035.001.combo.patch
  against trunk revision 3e8f353.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

  org.apache.hadoop.hdfs.TestLeaseRecovery
  org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot
  org.apache.hadoop.hdfs.TestSafeMode
  
org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
  org.apache.hadoop.hdfs.TestDatanodeRegistration
  org.apache.hadoop.hdfs.TestHFlush
  org.apache.hadoop.hdfs.TestSetTimes
  org.apache.hadoop.hdfs.TestPersistBlocks
  org.apache.hadoop.hdfs.TestEncryptedTransfer
  org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs
  org.apache.hadoop.net.TestNetworkTopology
  org.apache.hadoop.hdfs.server.namenode.TestFSImage
  org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
  org.apache.hadoop.hdfs.tools.TestDFSAdmin
  org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead
  org.apache.hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
  org.apache.hadoop.hdfs.TestDecommission
  org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite
  
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
  org.apache.hadoop.fs.permission.TestStickyBit
  org.apache.hadoop.hdfs.server.namenode.TestAclConfigFlag
  
org.apache.hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot
  
org.apache.hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot
  org.apache.hadoop.hdfs.server.namenode.TestXAttrConfigFlag
  org.apache.hadoop.hdfs.web.TestWebHDFSAcl
  org.apache.hadoop.hdfs.TestHDFSTrash
  org.apache.hadoop.hdfs.TestDistributedFileSystem
  org.apache.hadoop.hdfs.TestFileCreation
  org.apache.hadoop.hdfs.TestDataTransferKeepalive
  
org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus
  org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyIsHot
  org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
  
org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
  org.apache.hadoop.hdfs.server.namenode.TestFileContextAcl
  org.apache.hadoop.hdfs.TestRenameWhileOpen
  org.apache.hadoop.hdfs.server.namenode.TestNameNodeXAttr
  org.apache.hadoop.fs.TestHDFSFileContextMainOperations
  org.apache.hadoop.hdfs.web.TestWebHDFSXAttr
  org.apache.hadoop.hdfs.server.namenode.TestFsck
  
org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
  
org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
  org.apache.hadoop.hdfs.server.namenode.TestNameNodeAcl
  org.apache.hadoop.hdfs.server.namenode.TestStartup
  
org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
  org.apache.hadoop.hdfs.server.namenode.TestFileContextXAttr

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/7976//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HDFS-Build/7976//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7976//console

This message is automatically generated.

 Refactor DataStorage and