[jira] [Updated] (HDFS-9868) Add ability for DistCp to run between 2 clusters

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HDFS-9868:

Issue Type: Improvement  (was: New Feature)

Updating jira type to improvement since this is just adding a new option to 
distcp command.

> Add ability for DistCp to run between 2 clusters
> 
>
> Key: HDFS-9868
> URL: https://issues.apache.org/jira/browse/HDFS-9868
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: distcp
>Affects Versions: 2.7.1
>Reporter: NING DING
>Assignee: NING DING
> Attachments: HDFS-9868.05.patch, HDFS-9868.06.patch, 
> HDFS-9868.07.patch, HDFS-9868.1.patch, HDFS-9868.2.patch, HDFS-9868.3.patch, 
> HDFS-9868.4.patch
>
>
> Normally the HDFS cluster is HA enabled. It could take a long time when 
> coping huge data by distp. If the source cluster changes active namenode, the 
> distp will run failed. This patch supports the DistCp can read source cluster 
> files in HA access mode. A source cluster configuration file needs to be 
> specified (via the -sourceClusterConf option).
>   The following is an example of the contents of a source cluster 
> configuration
>   file:
> {code:xml}
> 
>   
>   fs.defaultFS
>   hdfs://mycluster
> 
> 
>   dfs.nameservices
>   mycluster
> 
> 
>   dfs.ha.namenodes.mycluster
>   nn1,nn2
> 
> 
>   dfs.namenode.rpc-address.mycluster.nn1
>   host1:9000
> 
> 
>   dfs.namenode.rpc-address.mycluster.nn2
>   host2:9000
> 
> 
>   dfs.namenode.http-address.mycluster.nn1
>   host1:50070
> 
> 
>   dfs.namenode.http-address.mycluster.nn2
>   host2:50070
> 
> 
>   dfs.client.failover.proxy.provider.mycluster
>   
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> 
>   
> {code}
>   The invocation of DistCp is as below:
> {code}
> bash$ hadoop distcp -sourceClusterConf sourceCluster.xml /foo/bar 
> hdfs://nn2:8020/bar/foo
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-9868) Add ability for DistCp to run between 2 clusters

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HDFS-9868:

Attachment: HDFS-9868.07.patch

> Add ability for DistCp to run between 2 clusters
> 
>
> Key: HDFS-9868
> URL: https://issues.apache.org/jira/browse/HDFS-9868
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: distcp
>Affects Versions: 2.7.1
>Reporter: NING DING
>Assignee: NING DING
> Attachments: HDFS-9868.05.patch, HDFS-9868.06.patch, 
> HDFS-9868.07.patch, HDFS-9868.1.patch, HDFS-9868.2.patch, HDFS-9868.3.patch, 
> HDFS-9868.4.patch
>
>
> Normally the HDFS cluster is HA enabled. It could take a long time when 
> coping huge data by distp. If the source cluster changes active namenode, the 
> distp will run failed. This patch supports the DistCp can read source cluster 
> files in HA access mode. A source cluster configuration file needs to be 
> specified (via the -sourceClusterConf option).
>   The following is an example of the contents of a source cluster 
> configuration
>   file:
> {code:xml}
> 
>   
>   fs.defaultFS
>   hdfs://mycluster
> 
> 
>   dfs.nameservices
>   mycluster
> 
> 
>   dfs.ha.namenodes.mycluster
>   nn1,nn2
> 
> 
>   dfs.namenode.rpc-address.mycluster.nn1
>   host1:9000
> 
> 
>   dfs.namenode.rpc-address.mycluster.nn2
>   host2:9000
> 
> 
>   dfs.namenode.http-address.mycluster.nn1
>   host1:50070
> 
> 
>   dfs.namenode.http-address.mycluster.nn2
>   host2:50070
> 
> 
>   dfs.client.failover.proxy.provider.mycluster
>   
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> 
>   
> {code}
>   The invocation of DistCp is as below:
> {code}
> bash$ hadoop distcp -sourceClusterConf sourceCluster.xml /foo/bar 
> hdfs://nn2:8020/bar/foo
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-9868) Add ability for DistCp to run between 2 clusters

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HDFS-9868:

Attachment: (was: HDFS-9868.07.patch)

> Add ability for DistCp to run between 2 clusters
> 
>
> Key: HDFS-9868
> URL: https://issues.apache.org/jira/browse/HDFS-9868
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: distcp
>Affects Versions: 2.7.1
>Reporter: NING DING
>Assignee: NING DING
> Attachments: HDFS-9868.05.patch, HDFS-9868.06.patch, 
> HDFS-9868.07.patch, HDFS-9868.1.patch, HDFS-9868.2.patch, HDFS-9868.3.patch, 
> HDFS-9868.4.patch
>
>
> Normally the HDFS cluster is HA enabled. It could take a long time when 
> coping huge data by distp. If the source cluster changes active namenode, the 
> distp will run failed. This patch supports the DistCp can read source cluster 
> files in HA access mode. A source cluster configuration file needs to be 
> specified (via the -sourceClusterConf option).
>   The following is an example of the contents of a source cluster 
> configuration
>   file:
> {code:xml}
> 
>   
>   fs.defaultFS
>   hdfs://mycluster
> 
> 
>   dfs.nameservices
>   mycluster
> 
> 
>   dfs.ha.namenodes.mycluster
>   nn1,nn2
> 
> 
>   dfs.namenode.rpc-address.mycluster.nn1
>   host1:9000
> 
> 
>   dfs.namenode.rpc-address.mycluster.nn2
>   host2:9000
> 
> 
>   dfs.namenode.http-address.mycluster.nn1
>   host1:50070
> 
> 
>   dfs.namenode.http-address.mycluster.nn2
>   host2:50070
> 
> 
>   dfs.client.failover.proxy.provider.mycluster
>   
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> 
>   
> {code}
>   The invocation of DistCp is as below:
> {code}
> bash$ hadoop distcp -sourceClusterConf sourceCluster.xml /foo/bar 
> hdfs://nn2:8020/bar/foo
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-10927) Lease Recovery: File not getting closed on HDFS when block write operation fails

[Chen Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871310#comment-15871310
 ] 

Chen Zhang edited comment on HDFS-10927 at 2/17/17 7:25 AM:


Hey guys, I just found an issue like this.
HBase RegionServer got an DiskFull exception while writing WAL files, and 
client failed after several times retry. When Master trying to use recoverLease 
to recover these file, we got almost same logs as [~ngoswami] attached
{quote}
java.io.IOException: File length mismatched.  The length of 
/home/work/ssd11/hdfs//datanode/current/BP-228094273-10.136.5.10-1486630815208/current/rbw/blk_1073970099
 is 174432256 but r=ReplicaBeingWritten, blk_1073970099_229357, RBW
  getNumBytes() = 174437376
  getBytesOnDisk()  = 174429752
  getVisibleLength()= 174429752
  getVolume()   = /home/work/ssd11/hdfs/x/datanode/current
  getBlockFile()= 
/home/work/ssd11/hdfs/x/datanode/current/BP-228094273-10.136.5.10-1486630815208/current/rbw/blk_1073970099
  bytesAcked=174429752
  bytesOnDisk=174429752
{quote}

In my case, it's caused by the exception while out.write() in receivePacket() 
of BlockReceiver. 
receivePacket() first update numbytes in replicaInfo, then write data to disk, 
and update bytesOnDisk at last, the DiskFull exception makes the length not 
consistent.



was (Author: zhangchen):
Hey guys, I just found an issue like this.
HBase RegionServer got an DiskFull exception while writing WAL files, and 
client failed after several times retry. When Master trying to use recoverLease 
to recover these file, we got almost same logs as [~ngoswami] attached
{quote}
java.io.IOException: File length mismatched.  The length of 
/home/work/ssd11/hdfs//datanode/current/BP-228094273-10.136.5.10-1486630815208/current/rbw/blk_1073970099
 is 174432256 but r=ReplicaBeingWritten, blk_1073970099_229357, RBW
  getNumBytes() = 174437376
  getBytesOnDisk()  = 174429752
  getVisibleLength()= 174429752
  getVolume()   = /home/work/ssd11/hdfs/x/datanode/current
  getBlockFile()= 
/home/work/ssd11/hdfs/x/datanode/current/BP-228094273-10.136.5.10-1486630815208/current/rbw/blk_1073970099
  bytesAcked=174429752
  bytesOnDisk=174429752
{quote}

It's caused by the exception while out.write() in receivePacket() of 
BlockReceiver. 
receivePacket() first update numbytes in replicaInfo, then write data to disk, 
and update bytesOnDisk at last, the DiskFull exception makes the length not 
consistent.


> Lease Recovery: File not getting closed on HDFS when block write operation 
> fails
> 
>
> Key: HDFS-10927
> URL: https://issues.apache.org/jira/browse/HDFS-10927
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 2.7.1
>Reporter: Nitin Goswami
>
> HDFS was unable to close a file when block write operation failed because of 
> too high disk usage.
> Scenario:
> HBase was writing WAL logs on HDFS and the disk usage was too high at that 
> time. While writing these WAL logs, one of the blocks writes operation failed 
> with the following exception:
> 2016-09-13 10:00:49,978 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Exception for 
> BP-337226066-192.168.193.217-1468912147102:blk_1074859607_1160899
> java.net.SocketTimeoutException: 6 millis timeout while waiting for 
> channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
> local=/192.168.194.144:50010 remote=/192.168.192.162:43105]
> at 
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
> at 
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at 
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at java.io.BufferedInputStream.fill(Unknown Source)
> at java.io.BufferedInputStream.read1(Unknown Source)
> at java.io.BufferedInputStream.read(Unknown Source)
> at java.io.DataInputStream.read(Unknown Source)
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472)
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:849)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:807)
> at 
> 

[jira] [Commented] (HDFS-10927) Lease Recovery: File not getting closed on HDFS when block write operation fails

[Chen Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871310#comment-15871310
 ] 

Chen Zhang commented on HDFS-10927:
---

Hey guys, I just found an issue like this.
HBase RegionServer got an DiskFull exception while writing WAL files, and 
client failed after several times retry. When Master trying to use recoverLease 
to recover these file, we got almost same logs as [~ngoswami] attached
{quote}
java.io.IOException: File length mismatched.  The length of 
/home/work/ssd11/hdfs//datanode/current/BP-228094273-10.136.5.10-1486630815208/current/rbw/blk_1073970099
 is 174432256 but r=ReplicaBeingWritten, blk_1073970099_229357, RBW
  getNumBytes() = 174437376
  getBytesOnDisk()  = 174429752
  getVisibleLength()= 174429752
  getVolume()   = /home/work/ssd11/hdfs/x/datanode/current
  getBlockFile()= 
/home/work/ssd11/hdfs/x/datanode/current/BP-228094273-10.136.5.10-1486630815208/current/rbw/blk_1073970099
  bytesAcked=174429752
  bytesOnDisk=174429752
{quote}

It's caused by the exception while out.write() in receivePacket() of 
BlockReceiver. 
receivePacket() first update numbytes in replicaInfo, then write data to disk, 
and update bytesOnDisk at last, the DiskFull exception makes the length not 
consistent.


> Lease Recovery: File not getting closed on HDFS when block write operation 
> fails
> 
>
> Key: HDFS-10927
> URL: https://issues.apache.org/jira/browse/HDFS-10927
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 2.7.1
>Reporter: Nitin Goswami
>
> HDFS was unable to close a file when block write operation failed because of 
> too high disk usage.
> Scenario:
> HBase was writing WAL logs on HDFS and the disk usage was too high at that 
> time. While writing these WAL logs, one of the blocks writes operation failed 
> with the following exception:
> 2016-09-13 10:00:49,978 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Exception for 
> BP-337226066-192.168.193.217-1468912147102:blk_1074859607_1160899
> java.net.SocketTimeoutException: 6 millis timeout while waiting for 
> channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
> local=/192.168.194.144:50010 remote=/192.168.192.162:43105]
> at 
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
> at 
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at 
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at java.io.BufferedInputStream.fill(Unknown Source)
> at java.io.BufferedInputStream.read1(Unknown Source)
> at java.io.BufferedInputStream.read(Unknown Source)
> at java.io.DataInputStream.read(Unknown Source)
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472)
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:849)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:807)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:251)
> at java.lang.Thread.run(Unknown Source)
> After this exception, HBase tried to close/rollover the WAL file but that 
> call also failed and WAL file couldn't be closed. After this HBase closed the 
> region server
> After some time, Lease Recovery got triggered for this file and following 
> exceptions starts occurring:
> 2016-09-13 11:51:11,743 WARN 
> org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol: Failed to 
> obtain replica info for block 
> (=BP-337226066-192.168.193.217-1468912147102:blk_1074859607_1161187) from 
> datanode (=DatanodeInfoWithStorage[192.168.192.162:50010,null,null])
> java.io.IOException: THIS IS NOT SUPPOSED TO HAPPEN: getBytesOnDisk() < 
> getVisibleLength(), rip=ReplicaBeingWritten, blk_1074859607_1161187, RBW
>   getNumBytes() = 45524696
>   getBytesOnDisk()  = 45483527
>   getVisibleLength()= 45511557
>   getVolume()   = /opt/reflex/data/yarn/datanode/current
>   getBlockFile()= 
> 

[jira] [Commented] (HDFS-11420) Edit file should not be processed by same type processor in OfflineEditViewer

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871254#comment-15871254
 ] 

Hadoop QA commented on HDFS-11420:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
28s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
 6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 42s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
25s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}120m 16s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11420 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12853210/HDFS-11420.002.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 9b374236f1a1 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 02c5494 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/18394/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/18394/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/18394/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Edit file should not be processed by same type processor in OfflineEditViewer
> -
>
> Key: HDFS-11420
> URL: https://issues.apache.org/jira/browse/HDFS-11420
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0-alpha2
>  

[jira] [Commented] (HDFS-11420) Edit file should not be processed by same type processor in OfflineEditViewer

[Yiqun Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871241#comment-15871241
 ] 

Yiqun Lin commented on HDFS-11420:
--

Thanks [~surendrasingh] for the review!
{quote}
Can you add same check in OfflineImageViewer ?
{quote}
I am thinking this may be not suitable in OfflineImageViewer. Currently 
OfflineImageViewer doesn't use the suffix name to judge one file is an xml file 
or binary file. For example, if we use the processor {{ReverseXML}} to 
reconstructor the fsimage from an xml file, it will run well whether the input 
file has {{.xml}} suffix or not. It only cares if the content of the input file 
is XML format.

> Edit file should not be processed by same type processor in OfflineEditViewer
> -
>
> Key: HDFS-11420
> URL: https://issues.apache.org/jira/browse/HDFS-11420
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11420.001.patch, HDFS-11420.002.patch
>
>
> XML/Binary format edit file should not be processed by XML/Binary processor 
> since it's no meaning. The result is complete same with the input file. We 
> should catch this special case and print the error info to let users know



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11420) Edit file should not be processed by same type processor in OfflineEditViewer

[Surendra Singh Lilhore (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871216#comment-15871216
 ] 

Surendra Singh Lilhore commented on HDFS-11420:
---

Can you add same check in {{OfflineImageViewer}} ?

> Edit file should not be processed by same type processor in OfflineEditViewer
> -
>
> Key: HDFS-11420
> URL: https://issues.apache.org/jira/browse/HDFS-11420
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11420.001.patch, HDFS-11420.002.patch
>
>
> XML/Binary format edit file should not be processed by XML/Binary processor 
> since it's no meaning. The result is complete same with the input file. We 
> should catch this special case and print the error info to let users know



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11420) Edit file should not be processed by same type processor in OfflineEditViewer

[Surendra Singh Lilhore (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871213#comment-15871213
 ] 

Surendra Singh Lilhore commented on HDFS-11420:
---

Hi [~linyiqun], Patch LGTM

> Edit file should not be processed by same type processor in OfflineEditViewer
> -
>
> Key: HDFS-11420
> URL: https://issues.apache.org/jira/browse/HDFS-11420
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11420.001.patch, HDFS-11420.002.patch
>
>
> XML/Binary format edit file should not be processed by XML/Binary processor 
> since it's no meaning. The result is complete same with the input file. We 
> should catch this special case and print the error info to let users know



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11298) Add storage policy info in FileStatus

[Yiqun Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871197#comment-15871197
 ] 

Yiqun Lin commented on HDFS-11298:
--

Hi [~surendrasingh], I have taken a look for this and your patch. I'm not so 
sure if we need to add this storage policy info in {{FileStatus}}. From the 
definition of class {{FileStatus}}, "Interface that represents the client side 
information for a file.", the client need to know the storage policy info of 
one file? Currently the field infos in {{FileStatus}} are all base info like 
blocksize, modification_time etc. And some other info like storage policy info 
is defined in {{HdfsFileStatus}}. So I'm +0 for this.

In addition, one comment for the patch
{code}
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java
@@ -44,6 +44,7 @@
   private String owner;
   private String group;
   private Path symlink;
+  private byte storagePolicy;
   
   public FileStatus() { this(0, false, 0, 0, 0, 0, null, null, null, null); }
   
@@ -73,6 +74,15 @@ public FileStatus(long length, boolean isdir,
 FsPermission permission, String owner, String group, 
 Path symlink,
 Path path) {
+this(length, isdir, block_replication, blocksize, modification_time,
+access_time, permission, owner, group, symlink, path, (byte) 0);
{code}
Here the default policy id is not {{(byte) 0}},  it should be byte 7({{ 
HdfsConstants.HOT_STORAGE_POLICY_ID}}). You may use this var to replace with 
bytes 0.

> Add storage policy info in FileStatus
> -
>
> Key: HDFS-11298
> URL: https://issues.apache.org/jira/browse/HDFS-11298
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 2.7.2
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-11298.001.patch
>
>
> Its good to add storagePolicy field in FileStatus. We no need to call 
> getStoragePolicy() API to get the policy.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-11298) Add storage policy info in FileStatus

[Yiqun Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871197#comment-15871197
 ] 

Yiqun Lin edited comment on HDFS-11298 at 2/17/17 5:24 AM:
---

Hi [~surendrasingh], I have taken a look for this and your patch. I'm not so 
sure if we need to add this storage policy info in {{FileStatus}}. From the 
definition of class {{FileStatus}}, "Interface that represents the client side 
information for a file.", the client need to know the storage policy info of 
one file? Currently the field infos in {{FileStatus}} are all base info like 
blocksize, modification_time etc. And some other info like storage policy info 
is defined in {{HdfsFileStatus}}. So I'm +0 for this.

In addition, one comment for the patch
{code}
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java
@@ -44,6 +44,7 @@
   private String owner;
   private String group;
   private Path symlink;
+  private byte storagePolicy;
   
   public FileStatus() { this(0, false, 0, 0, 0, 0, null, null, null, null); }
   
@@ -73,6 +74,15 @@ public FileStatus(long length, boolean isdir,
 FsPermission permission, String owner, String group, 
 Path symlink,
 Path path) {
+this(length, isdir, block_replication, blocksize, modification_time,
+access_time, permission, owner, group, symlink, path, (byte) 0);
{code}
Here the default policy id is not {{(byte) 0}},  it should be byte 
7({{HdfsConstants.HOT_STORAGE_POLICY_ID}}). You may use this var to replace 
with bytes 0.


was (Author: linyiqun):
Hi [~surendrasingh], I have taken a look for this and your patch. I'm not so 
sure if we need to add this storage policy info in {{FileStatus}}. From the 
definition of class {{FileStatus}}, "Interface that represents the client side 
information for a file.", the client need to know the storage policy info of 
one file? Currently the field infos in {{FileStatus}} are all base info like 
blocksize, modification_time etc. And some other info like storage policy info 
is defined in {{HdfsFileStatus}}. So I'm +0 for this.

In addition, one comment for the patch
{code}
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java
@@ -44,6 +44,7 @@
   private String owner;
   private String group;
   private Path symlink;
+  private byte storagePolicy;
   
   public FileStatus() { this(0, false, 0, 0, 0, 0, null, null, null, null); }
   
@@ -73,6 +74,15 @@ public FileStatus(long length, boolean isdir,
 FsPermission permission, String owner, String group, 
 Path symlink,
 Path path) {
+this(length, isdir, block_replication, blocksize, modification_time,
+access_time, permission, owner, group, symlink, path, (byte) 0);
{code}
Here the default policy id is not {{(byte) 0}},  it should be byte 7({{ 
HdfsConstants.HOT_STORAGE_POLICY_ID}}). You may use this var to replace with 
bytes 0.

> Add storage policy info in FileStatus
> -
>
> Key: HDFS-11298
> URL: https://issues.apache.org/jira/browse/HDFS-11298
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 2.7.2
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-11298.001.patch
>
>
> Its good to add storagePolicy field in FileStatus. We no need to call 
> getStoragePolicy() API to get the policy.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11402) HDFS Snapshots should capture point-in-time copies of OPEN files

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871185#comment-15871185
 ] 

Hadoop QA commented on HDFS-11402:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m  
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
54s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 63m 
41s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m  7s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11402 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12853200/HDFS-11402.02.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux c91ae938b25f 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 02c5494 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/18393/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client 
hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/18393/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> HDFS Snapshots should 

[jira] [Updated] (HDFS-11420) Edit file should not be processed by same type processor in OfflineEditViewer

[Yiqun Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated HDFS-11420:
-
Description: XML/Binary format edit file should not be processed by 
XML/Binary processor since it's no meaning. The result is complete same with 
the input file. We should catch this special case and print the error info to 
let users know  (was: XML format edit file should not be processed by XML 
processor since it's no meaning. The result is complete same with the input XML 
file. We should catch this case and print the error info to let users know)

> Edit file should not be processed by same type processor in OfflineEditViewer
> -
>
> Key: HDFS-11420
> URL: https://issues.apache.org/jira/browse/HDFS-11420
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11420.001.patch, HDFS-11420.002.patch
>
>
> XML/Binary format edit file should not be processed by XML/Binary processor 
> since it's no meaning. The result is complete same with the input file. We 
> should catch this special case and print the error info to let users know



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11420) Edit file should not be processed by same type processor in OfflineEditViewer

[Yiqun Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated HDFS-11420:
-
Summary: Edit file should not be processed by same type processor in 
OfflineEditViewer  (was: XML format edit file should not be processed by XML 
processor in OfflineEditViewer)

> Edit file should not be processed by same type processor in OfflineEditViewer
> -
>
> Key: HDFS-11420
> URL: https://issues.apache.org/jira/browse/HDFS-11420
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11420.001.patch, HDFS-11420.002.patch
>
>
> XML format edit file should not be processed by XML processor since it's no 
> meaning. The result is complete same with the input XML file. We should catch 
> this case and print the error info to let users know



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11420) XML format edit file should not be processed by XML processor in OfflineEditViewer

[Yiqun Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated HDFS-11420:
-
Attachment: HDFS-11420.002.patch

Found the same problem for binary processor. Upload the new patch.

> XML format edit file should not be processed by XML processor in 
> OfflineEditViewer
> --
>
> Key: HDFS-11420
> URL: https://issues.apache.org/jira/browse/HDFS-11420
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11420.001.patch, HDFS-11420.002.patch
>
>
> XML format edit file should not be processed by XML processor since it's no 
> meaning. The result is complete same with the input XML file. We should catch 
> this case and print the error info to let users know



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11402) HDFS Snapshots should capture point-in-time copies of OPEN files

[Manoj Govindassamy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manoj Govindassamy updated HDFS-11402:
--
Attachment: HDFS-11402.02.patch

Attaching v02 patch to address the following
1. {{LeaseManager#getINodeWithLeases()}} now runs the open files INode 
filtering task in a thread pool of 4 workers. So, at clusters having lots of 
open files, this should improve the performance
2. TestLeaseManager unit tests updated to verify various corner cases around 
open files count and thread pool execution.
3. Update hdfs-default.xml with the new config param.

[~yzhangal], [~linyiqun], [~jingzhao] and others, can you please take a look at 
the patch ?

> HDFS Snapshots should capture point-in-time copies of OPEN files
> 
>
> Key: HDFS-11402
> URL: https://issues.apache.org/jira/browse/HDFS-11402
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 2.6.0
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11402.01.patch, HDFS-11402.02.patch
>
>
> *Problem:*
> 1. When there are files being written and when HDFS Snapshots are taken in 
> parallel, Snapshots do capture all these files, but these being written files 
> in Snapshots do not have the point-in-time file length captured. That is, 
> these open files are not frozen in HDFS Snapshots. These open files 
> grow/shrink in length, just like the original file, even after the snapshot 
> time.
> 2. At the time of File close or any other meta data modification operation on 
> these files, HDFS reconciles the file length and records the modification in 
> the last taken Snapshot. All the previously taken Snapshots continue to have 
> those open Files with no modification recorded. So, all those previous 
> snapshots end up using the final modification record in the last snapshot. 
> Thus after the file close, file lengths in all those snapshots will end up 
> same.
> Assume File1 is opened for write and a total of 1MB written to it. While the 
> writes are happening, snapshots are taken in parallel.
> {noformat}
> |---Time---T1---T2-T3T4-->
> |---Snap1--Snap2-Snap3--->
> |---File1.open---write-write---close->
> {noformat}
> Then at time,
> T2:
> Snap1.File1.length = 0
> T3:
> Snap1.File1.length = 0
> Snap2.File1.length = 0
> 
> T4:
> Snap1.File1.length = 1MB
> Snap2.File1.length = 1MB
> Snap3.File1.length = 1MB
> *Proposal*
> 1. At the time of taking Snapshot, {{SnapshotManager#createSnapshot}} can 
> optionally request {{DirectorySnapshottableFeature#addSnapshot}} to freeze 
> open files. 
> 2. {{DirectorySnapshottableFeature#addSnapshot}} can consult with 
> {{LeaseManager}} and get a list INodesInPath for all open files under the 
> snapshot dir. 
> 3. {{DirectorySnapshottableFeature#addSnapshot}} after the Snapshot creation, 
> Diff creation and updating modification time, can invoke 
> {{INodeFile#recordModification}} for each of the open files. This way, the 
> Snapshot just taken will have a {{FileDiff}} with {{fileSize}} captured for 
> each of the open files. 
> 4. Above model follows the current Snapshot and Diff protocols and doesn't 
> introduce any any disk formats. So, I don't think we will be needing any new 
> FSImage Loader/Saver changes for Snapshots.
> 5. One of the design goals of HDFS Snapshot was ability to take any number of 
> snapshots in O(1) time. LeaseManager though has all the open files with 
> leases in-memory map, an iteration is still needed to prune the needed open 
> files and then run recordModification on each of them. So, it will not be a 
> strict O(1) with the above proposal. But, its going be a marginal increase 
> only as the new order will be of O(open_files_under_snap_dir). In order to 
> avoid HDFS Snapshots change in behavior for open files and avoid change in 
> time complexity, this improvement can be made under a new config 
> {{"dfs.namenode.snapshot.freeze.openfiles"}} which by default can be 
> {{false}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11401) Reduce verbosity of logs with favored nodes and block pinning enabled

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871119#comment-15871119
 ] 

Hadoop QA commented on HDFS-11401:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
 2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 43s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 32s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11401 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12853188/HDFS-11401.03.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux a4a6e9c8ee7e 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 658702e |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/18392/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/18392/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/18392/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Reduce verbosity of logs with favored nodes and block pinning enabled
> -
>
> Key: HDFS-11401
> URL: https://issues.apache.org/jira/browse/HDFS-11401
> Project: Hadoop HDFS
>  

[jira] [Commented] (HDFS-11422) Need a method to refresh the list of NN StorageDirectories after removal

[Harsh J (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871117#comment-15871117
 ] 

Harsh J commented on HDFS-11422:


IIRC, you can also use the command {{hdfs dfsadmin -restoreFailedStorage}} to 
control this.

> Need a method to refresh the list of NN StorageDirectories after removal
> 
>
> Key: HDFS-11422
> URL: https://issues.apache.org/jira/browse/HDFS-11422
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Affects Versions: 2.6.0
>Reporter: Attila Bukor
>
> If a NN storage directory is removed due to an error, the NameNode will fail 
> to write the image even if the issue was intermittent. It would be good to 
> have a way to make the NameNode try writing again after the issue is fixed - 
> and maybe even try it automatically every certain amount of time 
> (configurable).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11422) Need a method to refresh the list of NN StorageDirectories after removal

[Harsh J (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871115#comment-15871115
 ] 

Harsh J commented on HDFS-11422:


Does the feature offered by {{dfs.namenode.name.dir.restore}} help, even 
partially? For NNs with multiple directories, a reattempt of an earlier failed 
directory gets tried at every checkpoint trigger. Its not purely time based 
however.

We'd considered making it default to true (HDFS-3560), but that was not done. 
If this fits, perhaps we can turn it on by default as the feature has existed 
for a while now.

> Need a method to refresh the list of NN StorageDirectories after removal
> 
>
> Key: HDFS-11422
> URL: https://issues.apache.org/jira/browse/HDFS-11422
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Affects Versions: 2.6.0
>Reporter: Attila Bukor
>
> If a NN storage directory is removed due to an error, the NameNode will fail 
> to write the image even if the issue was intermittent. It would be good to 
> have a way to make the NameNode try writing again after the issue is fixed - 
> and maybe even try it automatically every certain amount of time 
> (configurable).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9705) Refine the behaviour of getFileChecksum when length = 0

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1587#comment-1587
 ] 

Hadoop QA commented on HDFS-9705:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
 4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m  
8s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 13s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}112m 52s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHDFS |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
|   | hadoop.hdfs.server.datanode.TestDataNodeUUID |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-9705 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12796250/HDFS-9705-v5.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 6462f7d752dc 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 658702e |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/18389/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/18389/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client 
hadoop-hdfs-project/hadoop-hdfs U: 

[jira] [Commented] (HDFS-9868) Add ability for DistCp to run between 2 clusters

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871110#comment-15871110
 ] 

Hadoop QA commented on HDFS-9868:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
31s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m  
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  9m  9s{color} 
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
20s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m  4s{color} 
| {color:red} hadoop-distcp in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
39s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 87m 48s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
|   | hadoop.tools.TestOptionsParser |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-9868 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12853186/HDFS-9868.07.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux db8802e6e9c4 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 658702e |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/18391/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/18391/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp.txt
 |
|  Test Results | 

[jira] [Commented] (HDFS-9868) Add ability for DistCp to run between 2 clusters

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871100#comment-15871100
 ] 

Hadoop QA commented on HDFS-9868:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m  
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
 0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
39s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
 0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
28s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m  
8s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 26s{color} 
| {color:red} hadoop-distcp in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
36s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 12s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.tools.TestDistCpWithSourceClusterConf |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-9868 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12838264/HDFS-9868.06.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 1e875c7061e5 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 658702e |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/18390/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/18390/testReport/ |
| modules | C: hadoop-common-project/hadoop-common 
hadoop-hdfs-project/hadoop-hdfs-client hadoop-tools/hadoop-distcp U: . |
| 

[jira] [Commented] (HDFS-11410) Use the cached instance when edit logging SetAclOp, SetXAttrOp and RemoveXAttrOp

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871075#comment-15871075
 ] 

Hudson commented on HDFS-11410:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11272 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/11272/])
HDFS-11410. Use the cached instance when edit logging SetAclOp, (xiao: rev 
02c549484a4fe6215c7f1a18d89389dbba6ea723)
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java


> Use the cached instance when edit logging SetAclOp, SetXAttrOp and 
> RemoveXAttrOp
> 
>
> Key: HDFS-11410
> URL: https://issues.apache.org/jira/browse/HDFS-11410
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.5.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: HDFS-11410.01.patch
>
>
> [~andrew.wang] recently had a comment on HDFS-10899:
> {quote}
> Looks like we aren't using the op cache in FSEditLog SetXAttrOp / 
> RemoveXAttrOp. I think this is accidental, could you do some research? 
> Particularly since we'll be doing a lot of SetXAttrOps, avoiding all that 
> object allocation would be nice. This could be a separate JIRA.
> {quote}
> i.e. 
> {code}
> static SetXAttrOp getInstance() {
>   return new SetXAttrOp();
> }
> {code}
> v.s.
> {code}
> static AddOp getInstance(OpInstanceCache cache) {
>   return (AddOp) cache.get(OP_ADD);
> }
> {code}
> Seems we should fix these non-caching usages.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11410) Use the cached instance when edit logging SetAclOp, SetXAttrOp and RemoveXAttrOp

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HDFS-11410:
-
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha3
   2.9.0
   Status: Resolved  (was: Patch Available)

Committed to trunk and branch-2.

Thanks Andrew for identifying the issue and reviewing!

> Use the cached instance when edit logging SetAclOp, SetXAttrOp and 
> RemoveXAttrOp
> 
>
> Key: HDFS-11410
> URL: https://issues.apache.org/jira/browse/HDFS-11410
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.5.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: HDFS-11410.01.patch
>
>
> [~andrew.wang] recently had a comment on HDFS-10899:
> {quote}
> Looks like we aren't using the op cache in FSEditLog SetXAttrOp / 
> RemoveXAttrOp. I think this is accidental, could you do some research? 
> Particularly since we'll be doing a lot of SetXAttrOps, avoiding all that 
> object allocation would be nice. This could be a separate JIRA.
> {quote}
> i.e. 
> {code}
> static SetXAttrOp getInstance() {
>   return new SetXAttrOp();
> }
> {code}
> v.s.
> {code}
> static AddOp getInstance(OpInstanceCache cache) {
>   return (AddOp) cache.get(OP_ADD);
> }
> {code}
> Seems we should fix these non-caching usages.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11401) Reduce verbosity of logs with favored nodes and block pinning enabled

[Weiwei Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871037#comment-15871037
 ] 

Weiwei Yang commented on HDFS-11401:


Hi [~rakeshr]

Thanks for your comments, it makes sense to me. I just uploaded the patch with 
the change you suggested, thank you!

> Reduce verbosity of logs with favored nodes and block pinning enabled
> -
>
> Key: HDFS-11401
> URL: https://issues.apache.org/jira/browse/HDFS-11401
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: balancer & mover, datanode
>Affects Versions: 2.8.0
>Reporter: Thiruvel Thirumoolan
>Assignee: Weiwei Yang
>Priority: Minor
> Attachments: HDFS-11401.01.patch, HDFS-11401.02.patch, 
> HDFS-11401.03.patch, testBalancerWithPinnedBlocks.output.txt
>
>
> I am working on enabling favored nodes for HBase (HBASE-15531). Was trying 
> out what happens if favored nodes is used and HDFS balancer is enabled. Ran 
> the unit test TestBalancer#testBalancerWithPinnedBlocks from branch-2.8.  
> There were too many exceptions and error messages with this (output attached) 
> since pinned blocks can't be moved.
> Is there any way to reduce this logging since block pinning is intentional? 
> On a real cluster, this could be too much. HDFS-6133 enabled block pinning.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11401) Reduce verbosity of logs with favored nodes and block pinning enabled

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated HDFS-11401:
---
Attachment: HDFS-11401.03.patch

> Reduce verbosity of logs with favored nodes and block pinning enabled
> -
>
> Key: HDFS-11401
> URL: https://issues.apache.org/jira/browse/HDFS-11401
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: balancer & mover, datanode
>Affects Versions: 2.8.0
>Reporter: Thiruvel Thirumoolan
>Assignee: Weiwei Yang
>Priority: Minor
> Attachments: HDFS-11401.01.patch, HDFS-11401.02.patch, 
> HDFS-11401.03.patch, testBalancerWithPinnedBlocks.output.txt
>
>
> I am working on enabling favored nodes for HBase (HBASE-15531). Was trying 
> out what happens if favored nodes is used and HDFS balancer is enabled. Ran 
> the unit test TestBalancer#testBalancerWithPinnedBlocks from branch-2.8.  
> There were too many exceptions and error messages with this (output attached) 
> since pinned blocks can't be moved.
> Is there any way to reduce this logging since block pinning is intentional? 
> On a real cluster, this could be too much. HDFS-6133 enabled block pinning.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-9868) Add ability for DistCp to run between 2 clusters

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HDFS-9868:

Attachment: HDFS-9868.07.patch

Reviving this jira, attaching patch 7, which is after a new review of patch 6, 
with mostly cosmetic fixes.

The only limitation on the current approach is it has to be run from target 
cluster. Ideally we should support both source and target (i.e. distcp from 
either one, as long as you have the remote conf). But as commented above 
couldn't think of a way to generalized the 'remote' concept. We could add a 
{{-targetClusterConf}} later, and reuse most of the conf code, but let's focus 
on the current patch now.

[~yzhangal], could you please review? Thanks much.

> Add ability for DistCp to run between 2 clusters
> 
>
> Key: HDFS-9868
> URL: https://issues.apache.org/jira/browse/HDFS-9868
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: distcp
>Affects Versions: 2.7.1
>Reporter: NING DING
>Assignee: NING DING
> Attachments: HDFS-9868.05.patch, HDFS-9868.06.patch, 
> HDFS-9868.07.patch, HDFS-9868.1.patch, HDFS-9868.2.patch, HDFS-9868.3.patch, 
> HDFS-9868.4.patch
>
>
> Normally the HDFS cluster is HA enabled. It could take a long time when 
> coping huge data by distp. If the source cluster changes active namenode, the 
> distp will run failed. This patch supports the DistCp can read source cluster 
> files in HA access mode. A source cluster configuration file needs to be 
> specified (via the -sourceClusterConf option).
>   The following is an example of the contents of a source cluster 
> configuration
>   file:
> {code:xml}
> 
>   
>   fs.defaultFS
>   hdfs://mycluster
> 
> 
>   dfs.nameservices
>   mycluster
> 
> 
>   dfs.ha.namenodes.mycluster
>   nn1,nn2
> 
> 
>   dfs.namenode.rpc-address.mycluster.nn1
>   host1:9000
> 
> 
>   dfs.namenode.rpc-address.mycluster.nn2
>   host2:9000
> 
> 
>   dfs.namenode.http-address.mycluster.nn1
>   host1:50070
> 
> 
>   dfs.namenode.http-address.mycluster.nn2
>   host2:50070
> 
> 
>   dfs.client.failover.proxy.provider.mycluster
>   
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> 
>   
> {code}
>   The invocation of DistCp is as below:
> {code}
> bash$ hadoop distcp -sourceClusterConf sourceCluster.xml /foo/bar 
> hdfs://nn2:8020/bar/foo
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-9868) Add ability for DistCp to run between 2 clusters

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HDFS-9868:

Summary: Add ability for DistCp to run between 2 clusters  (was: Add 
ability to read remote cluster configuration for DistCp)

> Add ability for DistCp to run between 2 clusters
> 
>
> Key: HDFS-9868
> URL: https://issues.apache.org/jira/browse/HDFS-9868
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: distcp
>Affects Versions: 2.7.1
>Reporter: NING DING
>Assignee: NING DING
> Attachments: HDFS-9868.05.patch, HDFS-9868.06.patch, 
> HDFS-9868.1.patch, HDFS-9868.2.patch, HDFS-9868.3.patch, HDFS-9868.4.patch
>
>
> Normally the HDFS cluster is HA enabled. It could take a long time when 
> coping huge data by distp. If the source cluster changes active namenode, the 
> distp will run failed. This patch supports the DistCp can read source cluster 
> files in HA access mode. A source cluster configuration file needs to be 
> specified (via the -sourceClusterConf option).
>   The following is an example of the contents of a source cluster 
> configuration
>   file:
> {code:xml}
> 
>   
>   fs.defaultFS
>   hdfs://mycluster
> 
> 
>   dfs.nameservices
>   mycluster
> 
> 
>   dfs.ha.namenodes.mycluster
>   nn1,nn2
> 
> 
>   dfs.namenode.rpc-address.mycluster.nn1
>   host1:9000
> 
> 
>   dfs.namenode.rpc-address.mycluster.nn2
>   host2:9000
> 
> 
>   dfs.namenode.http-address.mycluster.nn1
>   host1:50070
> 
> 
>   dfs.namenode.http-address.mycluster.nn2
>   host2:50070
> 
> 
>   dfs.client.failover.proxy.provider.mycluster
>   
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> 
>   
> {code}
>   The invocation of DistCp is as below:
> {code}
> bash$ hadoop distcp -sourceClusterConf sourceCluster.xml /foo/bar 
> hdfs://nn2:8020/bar/foo
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-8430) Erasure coding: compute file checksum for striped files (stripe by stripe)

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated HDFS-8430:
--
  Labels:   (was: hdfs-ec-3.0-must-do)
Priority: Major  (was: Blocker)

Thanks for the historical perspective Kai, very helpful! I'm quite happy to 
drop this from the blocker list, less work for us to do :)

> Erasure coding: compute file checksum for striped files (stripe by stripe)
> --
>
> Key: HDFS-8430
> URL: https://issues.apache.org/jira/browse/HDFS-8430
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: erasure-coding
>Affects Versions: HDFS-7285
>Reporter: Walter Su
>Assignee: Kai Zheng
> Attachments: HDFS-8430-poc1.patch
>
>
> HADOOP-3981 introduces a  distributed file checksum algorithm. It's designed 
> for replicated block.
> {{DFSClient.getFileChecksum()}} need some updates, so it can work for striped 
> block group.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-8430) Erasure coding: compute file checksum for striped files (stripe by stripe)

[Andrew Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated HDFS-8430:
--
Issue Type: Improvement  (was: Sub-task)
Parent: (was: HDFS-8031)

> Erasure coding: compute file checksum for striped files (stripe by stripe)
> --
>
> Key: HDFS-8430
> URL: https://issues.apache.org/jira/browse/HDFS-8430
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: erasure-coding
>Affects Versions: HDFS-7285
>Reporter: Walter Su
>Assignee: Kai Zheng
> Attachments: HDFS-8430-poc1.patch
>
>
> HADOOP-3981 introduces a  distributed file checksum algorithm. It's designed 
> for replicated block.
> {{DFSClient.getFileChecksum()}} need some updates, so it can work for striped 
> block group.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9705) Refine the behaviour of getFileChecksum when length = 0

[Kai Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870998#comment-15870998
 ] 

Kai Zheng commented on HDFS-9705:
-

[~andrew.wang], could we pick this up? Would be good to have for the 3.0 
release. Thanks.

> Refine the behaviour of getFileChecksum when length = 0
> ---
>
> Key: HDFS-9705
> URL: https://issues.apache.org/jira/browse/HDFS-9705
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Kai Zheng
>Priority: Minor
> Attachments: HDFS-9705-v1.patch, HDFS-9705-v2.patch, 
> HDFS-9705-v3.patch, HDFS-9705-v4.patch, HDFS-9705-v5.patch
>
>
> {{FileSystem#getFileChecksum}} may accept {{length}} parameter and 0 is a 
> valid value. Currently it will return {{null}} when length is 0, in the 
> following code block:
> {code}
> //compute file MD5
> final MD5Hash fileMD5 = MD5Hash.digest(md5out.getData());
> switch (crcType) {
> case CRC32:
>   return new MD5MD5CRC32GzipFileChecksum(bytesPerCRC,
>   crcPerBlock, fileMD5);
> case CRC32C:
>   return new MD5MD5CRC32CastagnoliFileChecksum(bytesPerCRC,
>   crcPerBlock, fileMD5);
> default:
>   // If there is no block allocated for the file,
>   // return one with the magic entry that matches what previous
>   // hdfs versions return.
>   if (locatedblocks.size() == 0) {
> return new MD5MD5CRC32GzipFileChecksum(0, 0, fileMD5);
>   }
>   // we should never get here since the validity was checked
>   // when getCrcType() was called above.
>   return null;
> }
> {code}
> The comment says "we should never get here since the validity was checked" 
> but it does. As we're using the MD5-MD5-X approach, and {{EMPTY--CONTENT}} 
> actually is a valid case in which the MD5 value is 
> {{d41d8cd98f00b204e9800998ecf8427e}}, so suggest we return a reasonable value 
> other than null. At least some useful information in the returned value can 
> be seen, like values from block checksum header.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8430) Erasure coding: compute file checksum for striped files (stripe by stripe)

[Kai Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870992#comment-15870992
 ] 

Kai Zheng commented on HDFS-8430:
-

Hi Andrew,

Sorry for the late response.

Quite some time ago [~szetszwo] and I sorted out two approaches for this thru a 
long discussion:
{quote}
First, add a new API like getFileChecksum(int cell) using the New Algorithm 2. 
Using this API users can compare a replicated file with a striped file, and if 
the file content are the same, the file checksums will be the same. This 
version may incur bigger network traffic as it needs to collect cells into 
client side for the computing.

Second, still change the existing API getFileChecksum() (no args) for striped 
files, using the algorithm that specific to striped files, but similar to 
existing one for the replicated files. No CRCs data will be collected centrally 
so no bigger network traffic involved as the new API does. As the block layouts 
are different, the results will differ if it's used to compare a striped file 
against a replicated file. It can be used to compare two files that are of the 
same layout, either replicated or striped.
{quote}

Sub-tasks of this, HDFS-9694 and HDFS-9833 implemented the {{2nd}} approach, 
enhancing the existing API getFileChecksum() (no args) to support striped 
files. It can be used to compare two files that are of the same layout, either 
replicated or striped. I thought this is good enough so far, for example, the 
distcp usage.

The {{1st}} approach can be used to compare a replicated file against a striped 
file. It needs non-trivial development work and also involves big network 
traffic to centrally compute an aggregate checksum results for a block group. 
IMO, we could continue to pend this for explicit user requirement for the 
target behavior ({{compare a replicated file against a striped file}}).

So what's your thoughts?

> Erasure coding: compute file checksum for striped files (stripe by stripe)
> --
>
> Key: HDFS-8430
> URL: https://issues.apache.org/jira/browse/HDFS-8430
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: erasure-coding
>Affects Versions: HDFS-7285
>Reporter: Walter Su
>Assignee: Kai Zheng
>Priority: Blocker
>  Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-8430-poc1.patch
>
>
> HADOOP-3981 introduces a  distributed file checksum algorithm. It's designed 
> for replicated block.
> {{DFSClient.getFileChecksum()}} need some updates, so it can work for striped 
> block group.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11424) Block Storage : add container representation to meta data

[Chen Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen Liang updated HDFS-11424:
--
Attachment: HDFS-11424-HDFS-7240.002.patch

> Block Storage : add container representation to meta data
> -
>
> Key: HDFS-11424
> URL: https://issues.apache.org/jira/browse/HDFS-11424
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11424-HDFS-7240.001.patch, 
> HDFS-11424-HDFS-7240.002.patch
>
>
> This JIRA adds the reference of containers to cblock. A reference of 
> container is the handler to some container, that can be used to *actually* 
> talk to an existing container. 
> HDFS-7240 already provides a container reference class, which is 
> {{org.apache.hadoop.scm.container.common.helpers.Pipeline}}. This JIRA adds 
> references of this class to relevant meta data classes in cblock.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work started] (HDFS-11425) Ozone : add client-facing container APIs

[Chen Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HDFS-11425 started by Chen Liang.
-
> Ozone : add client-facing container APIs
> 
>
> Key: HDFS-11425
> URL: https://issues.apache.org/jira/browse/HDFS-11425
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11425-HDFS-7240.001.patch
>
>
> This JIRA adds the container APIs, such as create container, delete container 
> etc exposed to users.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11425) Ozone : add client-facing container APIs

[Chen Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen Liang updated HDFS-11425:
--
Attachment: HDFS-11425-HDFS-7240.001.patch

> Ozone : add client-facing container APIs
> 
>
> Key: HDFS-11425
> URL: https://issues.apache.org/jira/browse/HDFS-11425
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11425-HDFS-7240.001.patch
>
>
> This JIRA adds the container APIs, such as create container, delete container 
> etc exposed to users.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-11425) Ozone : add client-facing container APIs

[Chen Liang (JIRA)]

Chen Liang created HDFS-11425:
-

 Summary: Ozone : add client-facing container APIs
 Key: HDFS-11425
 URL: https://issues.apache.org/jira/browse/HDFS-11425
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Chen Liang
Assignee: Chen Liang


This JIRA adds the container APIs, such as create container, delete container 
etc exposed to users.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10899) Add functionality to re-encrypt EDEKs.

[Andrew Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870903#comment-15870903
 ] 

Andrew Wang commented on HDFS-10899:


Hi Xiao, this latest rev looks like a big improvement in readability, and 
thanks for the speedy work getting through all the previous comments!

Nit:
* FSEditLog, rename logCompleteReencryptZone to logReencryptZoneComplete to 
match name of the op and the Status method?

ReencryptionHandler:
* processCurrentBatch, checkReadyForWrite can throw an exception, and it's not 
guarded by the try/finally for the writeLock.
* processCurrentBatch, it looks like we ride over a GeneralSecurityException or 
null return value when talking to the KMS. Shouldn't this either throw, or be 
retried? We don't want to miss reencrypting an EDEKs.
* High-level, it'd be good to think about the behavior when the KMS is 
slow/flaky/down. It looks like right now an IOException will throw all the way 
up to {{run}} which has a fixed sleep, but we might want some backoff for 
connectivity issues.
* In reencryptEncryptionZone, I think restoreFromLastProcessedFile also needs 
the read lock held when calling? Sprinkling additional paranoia {{hasReadLock}} 
/ {{hasWriteLock}} checks everywhere would be appreciated.
* {{restoreFromLPF}}, javadoc looks out of date since there's no longer a 
{{subdirs}}
* The new throttling mechanism could use some code comments
* Would still prefer this if statement to be in the constructor, have it throw 
an exception all the way up to the FSDirectory constructor:

{code}
if (ezManager.getProvider() == null) {
  LOG.info("No provider set, cannot re-encrypt");
  return;
}
{code}

* Similarly, we should do some bounds checking on all the config keys in the 
constructor (like > 0). I think {{isEnabled}} becomes extraneous afterwards. I 
don't think enable/disable is necessary since reencryption is already "opt-in" 
in that an admin has to trigger it, but if we really do want a flag, I'd prefer 
we do it with a new config key rather than with the throttling configs.
* I still find the call hierarchy complicated to reason about, since it has two 
entry points and is bi-recursive (run -> reencryptEZ -> restoreFromLPF or 
reencryptDir -> reencryptDirInt <-> reencryptINode). Rather than recursion, 
have you considered using a stack or queue to hold unprocessed items? I think 
this gets us towards a single reencryptDir call in reencryptEZ, since 
restoreFromLPF would be responsible for repopulating the stack/queue.
* reencryptDirInt, I see there's a fixup for {{i}} if things have changed, but 
I don't think this is sufficient. Imagine if the dir's children have been 
deleted. {{INodeDirectory#nextChild}} can return a negative index, which will 
always be less than the for loop termination condition {{children.size()}}. 
{{children.get(i)}} won't behave correctly with a negative index.
* reencryptINode, needs documentation on the return value
* If the # of items is an exact multiple of {{reencryptBatchSize}}, will the 
final call to {{processCurrentBatch}} with {{lastBatch = true}} in 
{{reencryptEncryptionZone}} correctly log the completion? It returns early if 
{{currentBatch}} is empty. IMO we should separate batch processing from logging 
the completion, there doesn't seem like much code sharing.
* Could use some class-level documentation about the iteration order (appears 
to be depth-first) and how we handle deletes and creates.
* Add a Precondition check in ReencryptTracker that passed inode is not null? 
This is important so the {{Objects.equals}} in processCurrentBatch behaves 
correctly.

EZManager:
* Rename {{getDir}} to {{getFSDirectory}} for clarity?

Follow-on:
* This relates to the status follow-on you're planning. I noticed cancelling a 
reencrypt reuses the complete edit log op. It'd be nice to store some 
additional info in the EZ root xattr admins can differentiate between 
completion and cancellation.

> Add functionality to re-encrypt EDEKs.
> --
>
> Key: HDFS-10899
> URL: https://issues.apache.org/jira/browse/HDFS-10899
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: encryption, kms
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: editsStored, HDFS-10899.01.patch, HDFS-10899.02.patch, 
> HDFS-10899.03.patch, HDFS-10899.04.patch, HDFS-10899.05.patch, 
> HDFS-10899.06.patch, HDFS-10899.07.patch, HDFS-10899.08.patch, 
> HDFS-10899.09.patch, HDFS-10899.wip.2.patch, HDFS-10899.wip.patch, Re-encrypt 
> edek design doc.pdf
>
>
> Currently when an encryption zone (EZ) key is rotated, it only takes effect 
> on new EDEKs. We should provide a way to re-encrypt EDEKs after the EZ key 
> rotation, for improved security.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HDFS-11379) DFSInputStream may infinite loop requesting block locations

[Xiao Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870830#comment-15870830
 ] 

Xiao Chen commented on HDFS-11379:
--

Ping, does the 2.7 backport look good?
(Also not sure if backports require a +1 to commit. Conflicts are minor and 
please let me know if +1 isn't required)

> DFSInputStream may infinite loop requesting block locations
> ---
>
> Key: HDFS-11379
> URL: https://issues.apache.org/jira/browse/HDFS-11379
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs-client
>Affects Versions: 2.7.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Fix For: 2.8.0, 3.0.0-alpha3
>
> Attachments: HDFS-11379.branch-2.7.patch, HDFS-11379.branch-2.patch, 
> HDFS-11379.trunk.patch
>
>
> DFSInputStream creation caches file size and initial range of locations.  If 
> the file is truncated (or replaced) and the client attempts to read outside 
> the initial range, the client goes into a tight infinite looping requesting 
> locations for the nonexistent range.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11424) Block Storage : add container representation to meta data

[Chen Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen Liang updated HDFS-11424:
--
Attachment: HDFS-11424-HDFS-7240.001.patch

> Block Storage : add container representation to meta data
> -
>
> Key: HDFS-11424
> URL: https://issues.apache.org/jira/browse/HDFS-11424
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11424-HDFS-7240.001.patch
>
>
> This JIRA adds the reference of containers to cblock. A reference of 
> container is the handler to some container, that can be used to *actually* 
> talk to an existing container. 
> HDFS-7240 already provides a container reference class, which is 
> {{org.apache.hadoop.scm.container.common.helpers.Pipeline}}. This JIRA adds 
> references of this class to relevant meta data classes in cblock.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-11424) Block Storage : add container representation to meta data

[Chen Liang (JIRA)]

Chen Liang created HDFS-11424:
-

 Summary: Block Storage : add container representation to meta data
 Key: HDFS-11424
 URL: https://issues.apache.org/jira/browse/HDFS-11424
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Chen Liang
Assignee: Chen Liang


This JIRA adds the reference of containers to cblock. A reference of container 
is the handler to some container, that can be used to *actually* talk to an 
existing container. 

HDFS-7240 already provides a container reference class, which is 
{{org.apache.hadoop.scm.container.common.helpers.Pipeline}}. This JIRA adds 
references of this class to relevant meta data classes in cblock.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8498) Blocks can be committed with wrong size

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870794#comment-15870794
 ] 

Hadoop QA commented on HDFS-8498:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 21m 
45s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 5s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
21s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
27s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
35s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
28s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
31s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
39s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
16s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
1s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
14s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 32s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new + 
89 unchanged - 2 fixed = 91 total (was 91) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
54s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m  
2s{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK 
v1.7.0_121. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 46s{color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
25s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}196m  6s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_121 Failed junit tests | 
hadoop.hdfs.server.datanode.TestDataNodeUUID |
|   | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | 

[jira] [Updated] (HDFS-11414) Ozone : move StorageContainerLocation protocol to hdfs-client

[Chen Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen Liang updated HDFS-11414:
--
Status: In Progress  (was: Patch Available)

> Ozone : move StorageContainerLocation protocol to hdfs-client
> -
>
> Key: HDFS-11414
> URL: https://issues.apache.org/jira/browse/HDFS-11414
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11414-HDFS-7240.001.patch, 
> HDFS-11414-HDFS-7240.002.patch, HDFS-11414-HDFS-7240.003.patch, 
> HDFS-11414-HDFS-7240.004.patch
>
>
> {{StorageContainerLocation}} classes are client-facing classes of containers, 
> similar to {{XceiverClient}}, so they should be moved to hadoop-hdfs-client.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-11414) Ozone : move StorageContainerLocation protocol to hdfs-client

[Chen Liang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870759#comment-15870759
 ] 

Chen Liang edited comment on HDFS-11414 at 2/16/17 9:57 PM:


v003 patch will have some merge conflicts when merging with HDFS-11184. upload 
v004 patch to fix it, and mark HDFS-11184 as a blocker


was (Author: vagarychen):
v003 patch will have some merge conflicts when merging with HDFS-11182. upload 
v004 patch to fix it, and mark HDFS-11182 as a blocker

> Ozone : move StorageContainerLocation protocol to hdfs-client
> -
>
> Key: HDFS-11414
> URL: https://issues.apache.org/jira/browse/HDFS-11414
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11414-HDFS-7240.001.patch, 
> HDFS-11414-HDFS-7240.002.patch, HDFS-11414-HDFS-7240.003.patch, 
> HDFS-11414-HDFS-7240.004.patch
>
>
> {{StorageContainerLocation}} classes are client-facing classes of containers, 
> similar to {{XceiverClient}}, so they should be moved to hadoop-hdfs-client.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11414) Ozone : move StorageContainerLocation protocol to hdfs-client

[Chen Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen Liang updated HDFS-11414:
--
Attachment: HDFS-11414-HDFS-7240.004.patch

v003 patch will have some merge conflicts when merging with HDFS-11182. upload 
v004 patch to fix it, and mark HDFS-11182 as a blocker

> Ozone : move StorageContainerLocation protocol to hdfs-client
> -
>
> Key: HDFS-11414
> URL: https://issues.apache.org/jira/browse/HDFS-11414
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11414-HDFS-7240.001.patch, 
> HDFS-11414-HDFS-7240.002.patch, HDFS-11414-HDFS-7240.003.patch, 
> HDFS-11414-HDFS-7240.004.patch
>
>
> {{StorageContainerLocation}} classes are client-facing classes of containers, 
> similar to {{XceiverClient}}, so they should be moved to hadoop-hdfs-client.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11423) Allow ReplicaInfo to be subclassed outside of package

[Joe Pallas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Pallas updated HDFS-11423:
--
Attachment: HDFS-11423.001.patch

> Allow ReplicaInfo to be subclassed outside of package
> -
>
> Key: HDFS-11423
> URL: https://issues.apache.org/jira/browse/HDFS-11423
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 3.0.0-alpha2
>Reporter: Joe Pallas
>Priority: Minor
> Attachments: HDFS-11423.001.patch
>
>
> The constructor for {{ReplicaInfo}} is package-scope instead of protected.  
> This means that an alternative dataset implementation in its own package 
> cannot create Replica classes that inherit from ReplicaInfo the way 
> {{LocalReplica}} does (or the way {{ProvidedReplica}} does in HDFS-9806).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-11423) Allow ReplicaInfo to be subclassed outside of package

[Joe Pallas (JIRA)]

Joe Pallas created HDFS-11423:
-

 Summary: Allow ReplicaInfo to be subclassed outside of package
 Key: HDFS-11423
 URL: https://issues.apache.org/jira/browse/HDFS-11423
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: datanode
Affects Versions: 3.0.0-alpha2
Reporter: Joe Pallas
Priority: Minor


The constructor for {{ReplicaInfo}} is package-scope instead of protected.  
This means that an alternative dataset implementation in its own package cannot 
create Replica classes that inherit from ReplicaInfo the way {{LocalReplica}} 
does (or the way {{ProvidedReplica}} does in HDFS-9806).





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11402) HDFS Snapshots should capture point-in-time copies of OPEN files

[Manoj Govindassamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870619#comment-15870619
 ] 

Manoj Govindassamy commented on HDFS-11402:
---

Thanks for the review [~linyiqun]. 

I can split {{LeaseManager#leasesById}} Map entries into few parts and spin off 
the {{LeaseManager#getINodeWithLeases()}} work in fixed thread pool Executor 
and have them run in parallel. Each _Task_ will filter the open files from its 
list and we can benefit from many such tasks running in parallel. Your thoughts 
on this please ?

> HDFS Snapshots should capture point-in-time copies of OPEN files
> 
>
> Key: HDFS-11402
> URL: https://issues.apache.org/jira/browse/HDFS-11402
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 2.6.0
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11402.01.patch
>
>
> *Problem:*
> 1. When there are files being written and when HDFS Snapshots are taken in 
> parallel, Snapshots do capture all these files, but these being written files 
> in Snapshots do not have the point-in-time file length captured. That is, 
> these open files are not frozen in HDFS Snapshots. These open files 
> grow/shrink in length, just like the original file, even after the snapshot 
> time.
> 2. At the time of File close or any other meta data modification operation on 
> these files, HDFS reconciles the file length and records the modification in 
> the last taken Snapshot. All the previously taken Snapshots continue to have 
> those open Files with no modification recorded. So, all those previous 
> snapshots end up using the final modification record in the last snapshot. 
> Thus after the file close, file lengths in all those snapshots will end up 
> same.
> Assume File1 is opened for write and a total of 1MB written to it. While the 
> writes are happening, snapshots are taken in parallel.
> {noformat}
> |---Time---T1---T2-T3T4-->
> |---Snap1--Snap2-Snap3--->
> |---File1.open---write-write---close->
> {noformat}
> Then at time,
> T2:
> Snap1.File1.length = 0
> T3:
> Snap1.File1.length = 0
> Snap2.File1.length = 0
> 
> T4:
> Snap1.File1.length = 1MB
> Snap2.File1.length = 1MB
> Snap3.File1.length = 1MB
> *Proposal*
> 1. At the time of taking Snapshot, {{SnapshotManager#createSnapshot}} can 
> optionally request {{DirectorySnapshottableFeature#addSnapshot}} to freeze 
> open files. 
> 2. {{DirectorySnapshottableFeature#addSnapshot}} can consult with 
> {{LeaseManager}} and get a list INodesInPath for all open files under the 
> snapshot dir. 
> 3. {{DirectorySnapshottableFeature#addSnapshot}} after the Snapshot creation, 
> Diff creation and updating modification time, can invoke 
> {{INodeFile#recordModification}} for each of the open files. This way, the 
> Snapshot just taken will have a {{FileDiff}} with {{fileSize}} captured for 
> each of the open files. 
> 4. Above model follows the current Snapshot and Diff protocols and doesn't 
> introduce any any disk formats. So, I don't think we will be needing any new 
> FSImage Loader/Saver changes for Snapshots.
> 5. One of the design goals of HDFS Snapshot was ability to take any number of 
> snapshots in O(1) time. LeaseManager though has all the open files with 
> leases in-memory map, an iteration is still needed to prune the needed open 
> files and then run recordModification on each of them. So, it will not be a 
> strict O(1) with the above proposal. But, its going be a marginal increase 
> only as the new order will be of O(open_files_under_snap_dir). In order to 
> avoid HDFS Snapshots change in behavior for open files and avoid change in 
> time complexity, this improvement can be made under a new config 
> {{"dfs.namenode.snapshot.freeze.openfiles"}} which by default can be 
> {{false}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11419) BlockPlacementPolicyDefault is choosing datanode in an inefficient way

[Chen Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen Liang updated HDFS-11419:
--
Description: 
Currently in {{BlockPlacementPolicyDefault}}, {{chooseTarget}} will end up 
calling into {{chooseRandom}}, which will first find a random datanode by 
calling
{code}DatanodeDescriptor chosenNode = chooseDataNode(scope, 
excludedNodes);{code}, then it checks whether that returned datanode satisfies 
storage type requirement
{code}storage = chooseStorage4Block(
  chosenNode, blocksize, results, entry.getKey());{code}
If yes, {{numOfReplicas--;}}, otherwise, the node is added to excluded nodes, 
and runs the loop again until {{numOfReplicas}} is down to 0.

A problem here is that, storage type is not being considered only until after a 
random node is already returned.  We've seen a case where a cluster has a large 
number of datanodes, while only a few satisfy the storage type condition. So, 
for the most part, this code blindly picks random datanodes that do not satisfy 
the storage type requirement.

To make matters worse, the way {{NetworkTopology#chooseRandom}} works is that, 
given a set of excluded nodes, it first finds a random datanodes, then if it is 
in excluded nodes set, try find another random nodes. So the more excluded 
nodes there are, the more likely a random node will be in the excluded set, in 
which case we basically wasted one iteration.

Therefore, this JIRA proposes to augment/modify the relevant classes in a way 
that datanodes can be found more efficiently. There are currently two different 
high level solutions we are considering:

1. add some field to Node base types to describe the storage type info, and 
when searching for a node, we take into account such field(s), and do not 
return node that does not meet the storage type requirement.

2. change {{NetworkTopology}} class to be aware of storage types, e.g. for one 
storage type, there is one tree subset that connects all the nodes with that 
type. And one search happens on only one such subset. So unexpected storage 
types are simply not in the search space. 

Thanks [~szetszwo] for the offline discussion, and thanks [~linyiqun] for 
pointing out a wrong statement (corrected now) in the description. Any further 
comments are more than welcome.

  was:
Currently in {{BlockPlacementPolicyDefault}}, {{chooseTarget}} will end up 
calling into {{chooseRandom}}, which will first find a random datanode by 
calling
{code}DatanodeDescriptor chosenNode = chooseDataNode(scope, 
excludedNodes);{code}, then it checks whether that returned datanode satisfies 
storage type requirement
{code}storage = chooseStorage4Block(
  chosenNode, blocksize, results, entry.getKey());{code}
If yes, {{numOfReplicas--;}}, otherwise, the node is added to excluded nodes, 
and runs the loop again until {{numOfReplicas}} is down to 0.

A problem here is that, storage type is not being considered only until after a 
random node is already returned.  We've seen a case where a cluster has a large 
number of datanodes, while only a few satisfy the storage type condition. So, 
for the most part, this code blindly picks random datanodes that do not satisfy 
the storage type requirement.

To make matters worse, the way {{NetworkTopology#chooseRandom}} works is that, 
given a set of excluded nodes, it first finds a random datanodes, then if it is 
in excluded nodes set, try find another random nodes. So the more excluded 
nodes there are, the more likely a random node will be in the excluded set, in 
which case we basically wasted one iteration.

Therefore, this JIRA proposes to augment/modify the relevant classes in a way 
that datanodes can be found more efficiently. There are currently two different 
high level solutions we are considering:

1. add some field to Node base types to describe the storage type info, and 
when searching for a node, we take such field(s) in to account, and do not 
return node that does not meet the storage type requirement.

2. change {{NetworkTopology}} class to be aware of storage types: for one 
storage type, there is one tree subset that connects all the nodes with that 
type. And one search happens on only one such subset. So unexpected storage 
types are simply in the search space. 

Thanks [~szetszwo] for the offline discussion, and thanks [~linyiqun] for 
pointing out a wrong statement in the description, and any comments are more 
than welcome.


> BlockPlacementPolicyDefault is choosing datanode in an inefficient way
> --
>
> Key: HDFS-11419
> URL: https://issues.apache.org/jira/browse/HDFS-11419
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Reporter: Chen Liang
>Assignee: Chen Liang
>
> Currently in 

[jira] [Reopened] (HDFS-8498) Blocks can be committed with wrong size

[Jing Zhao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jing Zhao reopened HDFS-8498:
-

Reopen for the branch-2 patch.

> Blocks can be committed with wrong size
> ---
>
> Key: HDFS-8498
> URL: https://issues.apache.org/jira/browse/HDFS-8498
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.5.0
>Reporter: Daryn Sharp
>Assignee: Jing Zhao
>Priority: Critical
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-8498.000.patch, HDFS-8498.001.patch, 
> HDFS-8498.branch-2.patch
>
>
> When an IBR for a UC block arrives, the NN updates the expected location's 
> block and replica state _only_ if it's on an unexpected storage for an 
> expected DN.  If it's for an expected storage, only the genstamp is updated.  
> When the block is committed, and the expected locations are verified, only 
> the genstamp is checked.  The size is not checked but it wasn't updated in 
> the expected locations anyway.
> A faulty client may misreport the size when committing the block.  The block 
> is effectively corrupted.  If the NN issues replications, the received IBR is 
> considered corrupt, the NN invalidates the block, immediately issues another 
> replication.  The NN eventually realizes all the original replicas are 
> corrupt after full BRs are received from the original DNs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-8498) Blocks can be committed with wrong size

[Jing Zhao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jing Zhao updated HDFS-8498:

Attachment: HDFS-8498.branch-2.patch

Upload the patch for branch-2

> Blocks can be committed with wrong size
> ---
>
> Key: HDFS-8498
> URL: https://issues.apache.org/jira/browse/HDFS-8498
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.5.0
>Reporter: Daryn Sharp
>Assignee: Jing Zhao
>Priority: Critical
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-8498.000.patch, HDFS-8498.001.patch, 
> HDFS-8498.branch-2.patch
>
>
> When an IBR for a UC block arrives, the NN updates the expected location's 
> block and replica state _only_ if it's on an unexpected storage for an 
> expected DN.  If it's for an expected storage, only the genstamp is updated.  
> When the block is committed, and the expected locations are verified, only 
> the genstamp is checked.  The size is not checked but it wasn't updated in 
> the expected locations anyway.
> A faulty client may misreport the size when committing the block.  The block 
> is effectively corrupted.  If the NN issues replications, the received IBR is 
> considered corrupt, the NN invalidates the block, immediately issues another 
> replication.  The NN eventually realizes all the original replicas are 
> corrupt after full BRs are received from the original DNs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11419) BlockPlacementPolicyDefault is choosing datanode in an inefficient way

[Chen Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen Liang updated HDFS-11419:
--
Description: 
Currently in {{BlockPlacementPolicyDefault}}, {{chooseTarget}} will end up 
calling into {{chooseRandom}}, which will first find a random datanode by 
calling
{code}DatanodeDescriptor chosenNode = chooseDataNode(scope, 
excludedNodes);{code}, then it checks whether that returned datanode satisfies 
storage type requirement
{code}storage = chooseStorage4Block(
  chosenNode, blocksize, results, entry.getKey());{code}
If yes, {{numOfReplicas--;}}, otherwise, the node is added to excluded nodes, 
and runs the loop again until {{numOfReplicas}} is down to 0.

A problem here is that, storage type is not being considered only until after a 
random node is already returned.  We've seen a case where a cluster has a large 
number of datanodes, while only a few satisfy the storage type condition. So, 
for the most part, this code blindly picks random datanodes that do not satisfy 
the storage type requirement.

To make matters worse, the way {{NetworkTopology#chooseRandom}} works is that, 
given a set of excluded nodes, it first finds a random datanodes, then if it is 
in excluded nodes set, try find another random nodes. So the more excluded 
nodes there are, the more likely a random node will be in the excluded set, in 
which case we basically wasted one iteration.

Therefore, this JIRA proposes to augment/modify the relevant classes in a way 
that datanodes can be found more efficiently. There are currently two different 
high level solutions we are considering:

1. add some field to Node base types to describe the storage type info, and 
when searching for a node, we take such field(s) in to account, and do not 
return node that does not meet the storage type requirement.

2. change {{NetworkTopology}} class to be aware of storage types: for one 
storage type, there is one tree subset that connects all the nodes with that 
type. And one search happens on only one such subset. So unexpected storage 
types are simply in the search space. 

Thanks [~szetszwo] for the offline discussion, and thanks [~linyiqun] for 
pointing out a wrong statement in the description, and any comments are more 
than welcome.

  was:
Currently in {{BlockPlacementPolicyDefault}}, {{chooseTarget}} will end up 
calling into {{chooseRandom}}, which will first find a random datanode by 
calling
{code}DatanodeDescriptor chosenNode = chooseDataNode(scope, 
excludedNodes);{code}, then it checks whether that returned datanode satisfies 
storage type requirement
{code}storage = chooseStorage4Block(
  chosenNode, blocksize, results, entry.getKey());{code}
If yes, {{numOfReplicas--;}}, otherwise, the node is added to excluded nodes, 
and runs the loop again until {{numOfReplicas}} is down to 0.

A problem here is that, storage type is not being considered only until after a 
random node is already returned.  We've seen a case where a cluster has a large 
number of datanodes, while only a few satisfy the storage type condition. So, 
for the most part, this code blindly picks random datanodes that do not satisfy 
the storage type, and adds the node to excluded and tries again and again.

To make matters worse, the way {{NetworkTopology#chooseRandom}} works is that, 
given a set of excluded nodes, it first finds a random datanodes, then if it is 
in excluded nodes set, try find another random nodes. So the more excluded 
nodes there are, the more likely a random node will be in the excluded set, in 
which case we basically wasted one iteration.

Therefore, this JIRA proposes to augment/modify the relevant classes in a way 
that datanodes can be found more efficiently. There are currently two different 
high level solutions we are considering:

1. add some field to Node base types to describe the storage type info, and 
when searching for a node, we take such field(s) in to account, and do not 
return node that does not meet the storage type requirement.

2. change {{NetworkTopology}} class to be aware of storage types: for one 
storage type, there is one tree subset that connects all the nodes with that 
type. And one search happens on only one such subset. So unexpected storage 
types are simply in the search space. 

Thanks [~szetszwo] for the offline discussion, and any comments are more than 
welcome.


> BlockPlacementPolicyDefault is choosing datanode in an inefficient way
> --
>
> Key: HDFS-11419
> URL: https://issues.apache.org/jira/browse/HDFS-11419
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Reporter: Chen Liang
>Assignee: Chen Liang
>
> Currently in {{BlockPlacementPolicyDefault}}, {{chooseTarget}} will end up 
> calling into 

[jira] [Commented] (HDFS-11419) BlockPlacementPolicyDefault is choosing datanode in an inefficient way

[Chen Liang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870528#comment-15870528
 ] 

Chen Liang commented on HDFS-11419:
---

[~linyiqun] thanks for the comments!

I think you are right about the excluded nodes part. Now it appears to me that 
if the randomly chosen node could not satisfy storage type requirement, there 
is nothing done to that node...(no {{else}} branch for {{storage == null}} 
case) which means it could be randomly picked again...In which case it works in 
a similar way like excluded node set: when a node is chosen, it could be 
satisfying storage type policy, but it is already picked (in excluded nodes 
set), OR it simply does not satisfy storage type. Either way we ignore it and 
try again. Only a node not in either of these two situations can pass.

Will change the description to make this correction. Thanks for the catch!

> BlockPlacementPolicyDefault is choosing datanode in an inefficient way
> --
>
> Key: HDFS-11419
> URL: https://issues.apache.org/jira/browse/HDFS-11419
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Reporter: Chen Liang
>Assignee: Chen Liang
>
> Currently in {{BlockPlacementPolicyDefault}}, {{chooseTarget}} will end up 
> calling into {{chooseRandom}}, which will first find a random datanode by 
> calling
> {code}DatanodeDescriptor chosenNode = chooseDataNode(scope, 
> excludedNodes);{code}, then it checks whether that returned datanode 
> satisfies storage type requirement
> {code}storage = chooseStorage4Block(
>   chosenNode, blocksize, results, entry.getKey());{code}
> If yes, {{numOfReplicas--;}}, otherwise, the node is added to excluded nodes, 
> and runs the loop again until {{numOfReplicas}} is down to 0.
> A problem here is that, storage type is not being considered only until after 
> a random node is already returned.  We've seen a case where a cluster has a 
> large number of datanodes, while only a few satisfy the storage type 
> condition. So, for the most part, this code blindly picks random datanodes 
> that do not satisfy the storage type, and adds the node to excluded and tries 
> again and again.
> To make matters worse, the way {{NetworkTopology#chooseRandom}} works is 
> that, given a set of excluded nodes, it first finds a random datanodes, then 
> if it is in excluded nodes set, try find another random nodes. So the more 
> excluded nodes there are, the more likely a random node will be in the 
> excluded set, in which case we basically wasted one iteration.
> Therefore, this JIRA proposes to augment/modify the relevant classes in a way 
> that datanodes can be found more efficiently. There are currently two 
> different high level solutions we are considering:
> 1. add some field to Node base types to describe the storage type info, and 
> when searching for a node, we take such field(s) in to account, and do not 
> return node that does not meet the storage type requirement.
> 2. change {{NetworkTopology}} class to be aware of storage types: for one 
> storage type, there is one tree subset that connects all the nodes with that 
> type. And one search happens on only one such subset. So unexpected storage 
> types are simply in the search space. 
> Thanks [~szetszwo] for the offline discussion, and any comments are more than 
> welcome.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-11422) Need a method to refresh the list of NN StorageDirectories after removal

[Attila Bukor (JIRA)]

Attila Bukor created HDFS-11422:
---

 Summary: Need a method to refresh the list of NN 
StorageDirectories after removal
 Key: HDFS-11422
 URL: https://issues.apache.org/jira/browse/HDFS-11422
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: namenode
Affects Versions: 2.6.0
Reporter: Attila Bukor


If a NN storage directory is removed due to an error, the NameNode will fail to 
write the image even if the issue was intermittent. It would be good to have a 
way to make the NameNode try writing again after the issue is fixed - and maybe 
even try it automatically every certain amount of time (configurable).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8196) Erasure Coding related information on NameNode UI

[Andrew Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870478#comment-15870478
 ] 

Andrew Wang commented on HDFS-8196:
---

Thanks for the rev [~lewuathe], did another pass, one nit and one more major 
thing I missed before (apologies):

Nit:

{code}
  Total EC Striped Block Groups  Size 
(bytes){StripedBlocksTotalSize}
{code}

I think this should read:

{code}
  Total EC Striped Block Groups 
Size{StripedBlocksTotalSize|fmt_bytes}
{code}

Major:

{code}
4032  public long getTotalStripedBlocks() {
4033return StreamSupport.stream(blocksMap.getBlocks().spliterator(), 
false)
4034.filter(BlockInfo::isStriped)
4035.count();
4036  }
4037
4038  public long getTotalStripedBlockSize() {
4039return StreamSupport.stream(blocksMap.getBlocks().spliterator(), 
false)
4040.filter(BlockInfo::isStriped)
4041.mapToLong(BlockInfo::getNumBytes)
4042.sum();
4043  }
{code}

Looks like these new methods iterate the entire block map to calculate the 
totals, which is inefficient. It also looks not thread safe.

Instead, what do you think about maintaining these counts in {{BlocksMap}}?

Another small nit, maybe we rename {{getTotalStripedBlocks}} to 
{{getTotalStripedBlockGroups}} for clarity?

> Erasure Coding related information on NameNode UI
> -
>
> Key: HDFS-8196
> URL: https://issues.apache.org/jira/browse/HDFS-8196
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
>  Labels: NameNode, WebUI, hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-8196.01.patch, HDFS-8196.02.patch, 
> HDFS-8196.03.patch, Screen Shot 2017-02-06 at 22.30.40.png, Screen Shot 
> 2017-02-12 at 20.21.42.png, Screen Shot 2017-02-14 at 22.43.57.png
>
>
> NameNode WebUI shows EC related information and metrics. 
> This is depend on [HDFS-7674|https://issues.apache.org/jira/browse/HDFS-7674].



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11401) Reduce verbosity of logs with favored nodes and block pinning enabled

[Rakesh R (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870467#comment-15870467
 ] 

Rakesh R commented on HDFS-11401:
-

Thank you [~cheersyang] for the patch. I've one comment.

With the patch, it is logging all exceptions with DEBUG level, right?
{code}
-  errMsg = "opReplaceBlock " + block + " received exception " + ioe; 
-  LOG.info(errMsg);
+  errMsg = "opReplaceBlock " + block + " received exception " + ioe;
+  if (LOG.isDebugEnabled()) {
+LOG.debug(errMsg);
+  }
{code}
Instead, how about logging only the pinning err with DEBUG level and all other 
errs continue with INFO, like below?
{code}
  errMsg = "opReplaceBlock " + block + " received exception " + ioe; 
  if (ioe instanceof BlockPinningException) {
opStatus = Status.ERROR_BLOCK_PINNED;
if (LOG.isDebugEnabled()) {
  LOG.debug(errMsg);
}
  } else {
LOG.info(errMsg);
  }
{code}

> Reduce verbosity of logs with favored nodes and block pinning enabled
> -
>
> Key: HDFS-11401
> URL: https://issues.apache.org/jira/browse/HDFS-11401
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: balancer & mover, datanode
>Affects Versions: 2.8.0
>Reporter: Thiruvel Thirumoolan
>Assignee: Weiwei Yang
>Priority: Minor
> Attachments: HDFS-11401.01.patch, HDFS-11401.02.patch, 
> testBalancerWithPinnedBlocks.output.txt
>
>
> I am working on enabling favored nodes for HBase (HBASE-15531). Was trying 
> out what happens if favored nodes is used and HDFS balancer is enabled. Ran 
> the unit test TestBalancer#testBalancerWithPinnedBlocks from branch-2.8.  
> There were too many exceptions and error messages with this (output attached) 
> since pinned blocks can't be moved.
> Is there any way to reduce this logging since block pinning is intentional? 
> On a real cluster, this could be too much. HDFS-6133 enabled block pinning.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11100) Recursively deleting file protected by sticky bit should fail

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HDFS-11100:
---
Release Note: Changed the behavior of removing directories with sticky 
bits, so that it is closer to what most Unix/Linux users would expect.

> Recursively deleting file protected by sticky bit should fail
> -
>
> Key: HDFS-11100
> URL: https://issues.apache.org/jira/browse/HDFS-11100
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Critical
>  Labels: permissions
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11100.001.patch, HDFS-11100.002.patch, 
> HDFS-11100.003.patch, HDFS-11100.004.patch, HDFS-11100.005.patch, hdfs_cmds
>
>
> Recursively deleting a directory that contains files or directories protected 
> by sticky bit should fail but it doesn't in HDFS. In the case below, 
> {{/tmp/test/sticky_dir/f2}} is protected by sticky bit, thus recursive 
> deleting {{/tmp/test/sticky_dir}} should fail.
> {noformat}
> + hdfs dfs -ls -R /tmp/test
> drwxrwxrwt   - jzhuge supergroup  0 2016-11-03 18:08 
> /tmp/test/sticky_dir
> -rwxrwxrwx   1 jzhuge supergroup  0 2016-11-03 18:08 
> /tmp/test/sticky_dir/f2
> + sudo -u hadoop hdfs dfs -rm -skipTrash /tmp/test/sticky_dir/f2
> rm: Permission denied by sticky bit: user=hadoop, 
> path="/tmp/test/sticky_dir/f2":jzhuge:supergroup:-rwxrwxrwx, 
> parent="/tmp/test/sticky_dir":jzhuge:supergroup:drwxrwxrwt
> + sudo -u hadoop hdfs dfs -rm -r -skipTrash /tmp/test/sticky_dir
> Deleted /tmp/test/sticky_dir
> {noformat}
> Centos 6.4 behavior:
> {noformat}
> $ ls -lR /tmp/test
> /tmp/test: 
> total 4
> drwxrwxrwt 2 systest systest 4096 Nov  3 18:36 sbit
> /tmp/test/sbit:
> total 0
> -rw-rw-rw- 1 systest systest 0 Nov  2 13:45 f2
> $ sudo -u mapred rm -fr /tmp/test/sbit
> rm: cannot remove `/tmp/test/sbit/f2': Operation not permitted
> $ chmod -t /tmp/test/sbit
> $ sudo -u mapred rm -fr /tmp/test/sbit
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11100) Recursively deleting file protected by sticky bit should fail

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870385#comment-15870385
 ] 

Wei-Chiu Chuang commented on HDFS-11100:


[~jzhuge] Since this is an incompatible change, if you can, please 
review/update my release note. 

> Recursively deleting file protected by sticky bit should fail
> -
>
> Key: HDFS-11100
> URL: https://issues.apache.org/jira/browse/HDFS-11100
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Critical
>  Labels: permissions
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11100.001.patch, HDFS-11100.002.patch, 
> HDFS-11100.003.patch, HDFS-11100.004.patch, HDFS-11100.005.patch, hdfs_cmds
>
>
> Recursively deleting a directory that contains files or directories protected 
> by sticky bit should fail but it doesn't in HDFS. In the case below, 
> {{/tmp/test/sticky_dir/f2}} is protected by sticky bit, thus recursive 
> deleting {{/tmp/test/sticky_dir}} should fail.
> {noformat}
> + hdfs dfs -ls -R /tmp/test
> drwxrwxrwt   - jzhuge supergroup  0 2016-11-03 18:08 
> /tmp/test/sticky_dir
> -rwxrwxrwx   1 jzhuge supergroup  0 2016-11-03 18:08 
> /tmp/test/sticky_dir/f2
> + sudo -u hadoop hdfs dfs -rm -skipTrash /tmp/test/sticky_dir/f2
> rm: Permission denied by sticky bit: user=hadoop, 
> path="/tmp/test/sticky_dir/f2":jzhuge:supergroup:-rwxrwxrwx, 
> parent="/tmp/test/sticky_dir":jzhuge:supergroup:drwxrwxrwt
> + sudo -u hadoop hdfs dfs -rm -r -skipTrash /tmp/test/sticky_dir
> Deleted /tmp/test/sticky_dir
> {noformat}
> Centos 6.4 behavior:
> {noformat}
> $ ls -lR /tmp/test
> /tmp/test: 
> total 4
> drwxrwxrwt 2 systest systest 4096 Nov  3 18:36 sbit
> /tmp/test/sbit:
> total 0
> -rw-rw-rw- 1 systest systest 0 Nov  2 13:45 f2
> $ sudo -u mapred rm -fr /tmp/test/sbit
> rm: cannot remove `/tmp/test/sbit/f2': Operation not permitted
> $ chmod -t /tmp/test/sbit
> $ sudo -u mapred rm -fr /tmp/test/sbit
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11298) Add storage policy info in FileStatus

[Surendra Singh Lilhore (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870380#comment-15870380
 ] 

Surendra Singh Lilhore commented on HDFS-11298:
---

[~linyiqun] would you review this ?

> Add storage policy info in FileStatus
> -
>
> Key: HDFS-11298
> URL: https://issues.apache.org/jira/browse/HDFS-11298
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 2.7.2
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-11298.001.patch
>
>
> Its good to add storagePolicy field in FileStatus. We no need to call 
> getStoragePolicy() API to get the policy.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11100) Recursively deleting file protected by sticky bit should fail

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870370#comment-15870370
 ] 

John Zhuge commented on HDFS-11100:
---

Thanks [~jojochuang] and [~yzhangal] for the review and commit!

> Recursively deleting file protected by sticky bit should fail
> -
>
> Key: HDFS-11100
> URL: https://issues.apache.org/jira/browse/HDFS-11100
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Critical
>  Labels: permissions
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11100.001.patch, HDFS-11100.002.patch, 
> HDFS-11100.003.patch, HDFS-11100.004.patch, HDFS-11100.005.patch, hdfs_cmds
>
>
> Recursively deleting a directory that contains files or directories protected 
> by sticky bit should fail but it doesn't in HDFS. In the case below, 
> {{/tmp/test/sticky_dir/f2}} is protected by sticky bit, thus recursive 
> deleting {{/tmp/test/sticky_dir}} should fail.
> {noformat}
> + hdfs dfs -ls -R /tmp/test
> drwxrwxrwt   - jzhuge supergroup  0 2016-11-03 18:08 
> /tmp/test/sticky_dir
> -rwxrwxrwx   1 jzhuge supergroup  0 2016-11-03 18:08 
> /tmp/test/sticky_dir/f2
> + sudo -u hadoop hdfs dfs -rm -skipTrash /tmp/test/sticky_dir/f2
> rm: Permission denied by sticky bit: user=hadoop, 
> path="/tmp/test/sticky_dir/f2":jzhuge:supergroup:-rwxrwxrwx, 
> parent="/tmp/test/sticky_dir":jzhuge:supergroup:drwxrwxrwt
> + sudo -u hadoop hdfs dfs -rm -r -skipTrash /tmp/test/sticky_dir
> Deleted /tmp/test/sticky_dir
> {noformat}
> Centos 6.4 behavior:
> {noformat}
> $ ls -lR /tmp/test
> /tmp/test: 
> total 4
> drwxrwxrwt 2 systest systest 4096 Nov  3 18:36 sbit
> /tmp/test/sbit:
> total 0
> -rw-rw-rw- 1 systest systest 0 Nov  2 13:45 f2
> $ sudo -u mapred rm -fr /tmp/test/sbit
> rm: cannot remove `/tmp/test/sbit/f2': Operation not permitted
> $ chmod -t /tmp/test/sbit
> $ sudo -u mapred rm -fr /tmp/test/sbit
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-11375) Display the volume storage type in datanode UI

[Surendra Singh Lilhore (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870266#comment-15870266
 ] 

Surendra Singh Lilhore edited comment on HDFS-11375 at 2/16/17 4:50 PM:


Thanks [~liuml07] for review and commit..


was (Author: surendrasingh):
Thanks [~liuml07] for review and commite

> Display the volume storage type in datanode UI
> --
>
> Key: HDFS-11375
> URL: https://issues.apache.org/jira/browse/HDFS-11375
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode, ui
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: DN_UI_Aftrerfix.png, HDFS-11375.01.patch, 
> HDFS-11375.02.patch
>
>
> Volume storage info is useful for debugging the issue related to policy... 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11375) Display the volume storage type in datanode UI

[Surendra Singh Lilhore (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870266#comment-15870266
 ] 

Surendra Singh Lilhore commented on HDFS-11375:
---

Thanks [~liuml07] for review and commite

> Display the volume storage type in datanode UI
> --
>
> Key: HDFS-11375
> URL: https://issues.apache.org/jira/browse/HDFS-11375
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode, ui
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: DN_UI_Aftrerfix.png, HDFS-11375.01.patch, 
> HDFS-11375.02.patch
>
>
> Volume storage info is useful for debugging the issue related to policy... 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11100) Recursively deleting file protected by sticky bit should fail

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869995#comment-15869995
 ] 

Hudson commented on HDFS-11100:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11265 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/11265/])
HDFS-11100. Recursively deleting file protected by sticky bit should (weichiu: 
rev 5690b51ef7c708c0a71162ddaff04466bc71cdcc)
* (edit) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSExceptionMessages.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/permission/TestStickyBit.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java


> Recursively deleting file protected by sticky bit should fail
> -
>
> Key: HDFS-11100
> URL: https://issues.apache.org/jira/browse/HDFS-11100
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Critical
>  Labels: permissions
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11100.001.patch, HDFS-11100.002.patch, 
> HDFS-11100.003.patch, HDFS-11100.004.patch, HDFS-11100.005.patch, hdfs_cmds
>
>
> Recursively deleting a directory that contains files or directories protected 
> by sticky bit should fail but it doesn't in HDFS. In the case below, 
> {{/tmp/test/sticky_dir/f2}} is protected by sticky bit, thus recursive 
> deleting {{/tmp/test/sticky_dir}} should fail.
> {noformat}
> + hdfs dfs -ls -R /tmp/test
> drwxrwxrwt   - jzhuge supergroup  0 2016-11-03 18:08 
> /tmp/test/sticky_dir
> -rwxrwxrwx   1 jzhuge supergroup  0 2016-11-03 18:08 
> /tmp/test/sticky_dir/f2
> + sudo -u hadoop hdfs dfs -rm -skipTrash /tmp/test/sticky_dir/f2
> rm: Permission denied by sticky bit: user=hadoop, 
> path="/tmp/test/sticky_dir/f2":jzhuge:supergroup:-rwxrwxrwx, 
> parent="/tmp/test/sticky_dir":jzhuge:supergroup:drwxrwxrwt
> + sudo -u hadoop hdfs dfs -rm -r -skipTrash /tmp/test/sticky_dir
> Deleted /tmp/test/sticky_dir
> {noformat}
> Centos 6.4 behavior:
> {noformat}
> $ ls -lR /tmp/test
> /tmp/test: 
> total 4
> drwxrwxrwt 2 systest systest 4096 Nov  3 18:36 sbit
> /tmp/test/sbit:
> total 0
> -rw-rw-rw- 1 systest systest 0 Nov  2 13:45 f2
> $ sudo -u mapred rm -fr /tmp/test/sbit
> rm: cannot remove `/tmp/test/sbit/f2': Operation not permitted
> $ chmod -t /tmp/test/sbit
> $ sudo -u mapred rm -fr /tmp/test/sbit
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11100) Recursively deleting file protected by sticky bit should fail

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HDFS-11100:
---
   Resolution: Fixed
 Hadoop Flags: Incompatible change,Reviewed  (was: Incompatible change)
Fix Version/s: 3.0.0-alpha3
   Status: Resolved  (was: Patch Available)

Committed 005 patch to trunk.
Much thanks to [~jzhuge] for the patch and [~yzhangal] for the initial review!

> Recursively deleting file protected by sticky bit should fail
> -
>
> Key: HDFS-11100
> URL: https://issues.apache.org/jira/browse/HDFS-11100
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Critical
>  Labels: permissions
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11100.001.patch, HDFS-11100.002.patch, 
> HDFS-11100.003.patch, HDFS-11100.004.patch, HDFS-11100.005.patch, hdfs_cmds
>
>
> Recursively deleting a directory that contains files or directories protected 
> by sticky bit should fail but it doesn't in HDFS. In the case below, 
> {{/tmp/test/sticky_dir/f2}} is protected by sticky bit, thus recursive 
> deleting {{/tmp/test/sticky_dir}} should fail.
> {noformat}
> + hdfs dfs -ls -R /tmp/test
> drwxrwxrwt   - jzhuge supergroup  0 2016-11-03 18:08 
> /tmp/test/sticky_dir
> -rwxrwxrwx   1 jzhuge supergroup  0 2016-11-03 18:08 
> /tmp/test/sticky_dir/f2
> + sudo -u hadoop hdfs dfs -rm -skipTrash /tmp/test/sticky_dir/f2
> rm: Permission denied by sticky bit: user=hadoop, 
> path="/tmp/test/sticky_dir/f2":jzhuge:supergroup:-rwxrwxrwx, 
> parent="/tmp/test/sticky_dir":jzhuge:supergroup:drwxrwxrwt
> + sudo -u hadoop hdfs dfs -rm -r -skipTrash /tmp/test/sticky_dir
> Deleted /tmp/test/sticky_dir
> {noformat}
> Centos 6.4 behavior:
> {noformat}
> $ ls -lR /tmp/test
> /tmp/test: 
> total 4
> drwxrwxrwt 2 systest systest 4096 Nov  3 18:36 sbit
> /tmp/test/sbit:
> total 0
> -rw-rw-rw- 1 systest systest 0 Nov  2 13:45 f2
> $ sudo -u mapred rm -fr /tmp/test/sbit
> rm: cannot remove `/tmp/test/sbit/f2': Operation not permitted
> $ chmod -t /tmp/test/sbit
> $ sudo -u mapred rm -fr /tmp/test/sbit
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11421) Make WebHDFS' ACLs RegEx configurable

[Harsh J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J updated HDFS-11421:
---
Affects Version/s: 2.6.0
 Target Version/s: 2.9.0
   Status: Patch Available  (was: Open)

> Make WebHDFS' ACLs RegEx configurable
> -
>
> Key: HDFS-11421
> URL: https://issues.apache.org/jira/browse/HDFS-11421
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: webhdfs
>Affects Versions: 2.6.0
>Reporter: Harsh J
>Assignee: Harsh J
>
> Part of HDFS-5608 added support for GET/SET ACLs over WebHDFS. This currently 
> identifies the passed arguments via a hard-coded regex that mandates certain 
> group and user naming styles.
> A similar limitation had existed before for CHOWN and other User/Group set 
> related operations of WebHDFS, where it was then made configurable via 
> HDFS-11391 + HDFS-4983.
> Such configurability should be allowed for the ACL operations too.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11421) Make WebHDFS' ACLs RegEx configurable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869876#comment-15869876
 ] 

ASF GitHub Bot commented on HDFS-11421:
---

GitHub user QwertyManiac opened a pull request:

https://github.com/apache/hadoop/pull/195

HDFS-11421. Make WebHDFS' ACLs RegEx configurable. (harsh)

- Introduced a new config key `dfs.webhdfs.acl.provider.permission.pattern` 
with value set to existing default ACL regex
- Added pattern-from-config setting points for `AclPermissionParam` to 
`NameNodeHttpServer`, `WebHdfsFileSystem` and `WebHdfsHandler`, akin to how 
`UserParam` is set today at these three places
- Added setters/getters for custom pattern to `AclPermissionParam`
- Added a new test for `AclPermissionParam` custom patterns that exercises 
numeric usernames and group names with `@` characters
- Extended existing `UserParam` configurability tests to cover ACL 
modification with numeric usernames and group names with `@` characters over a 
custom pattern
- Ran existing relevant tests, which continue to pass

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/QwertyManiac/hadoop HDFS-11421

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hadoop/pull/195.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #195


commit f80343d972342033db7157dcf39a89bdb98ebf16
Author: Harsh J 
Date:   2017-02-16T12:47:07Z

HDFS-11421. Make WebHDFS' ACLs RegEx configurable. (harsh)




> Make WebHDFS' ACLs RegEx configurable
> -
>
> Key: HDFS-11421
> URL: https://issues.apache.org/jira/browse/HDFS-11421
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: webhdfs
>Reporter: Harsh J
>Assignee: Harsh J
>
> Part of HDFS-5608 added support for GET/SET ACLs over WebHDFS. This currently 
> identifies the passed arguments via a hard-coded regex that mandates certain 
> group and user naming styles.
> A similar limitation had existed before for CHOWN and other User/Group set 
> related operations of WebHDFS, where it was then made configurable via 
> HDFS-11391 + HDFS-4983.
> Such configurability should be allowed for the ACL operations too.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-6804) race condition between transferring block and appending block causes "Unexpected checksum mismatch exception"

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-6804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869871#comment-15869871
 ] 

Wei-Chiu Chuang commented on HDFS-6804:
---

[~brahmareddy]
I like the approach in the patch. In fact the test case in HDFS-11160 is 
fragile.
Two nits:
* can you rebase the code to trunk? There seems to be minor conflict.
* Would you be able to remove the hard coded sleep time? Like 
{code}
// STEP 2: Wait till the transfer happens.
Thread.sleep(5000);
{code}
Use the existing test apis such as {{GenericTestUtils#waitFor}} might help.

Thanks!

> race condition between transferring block and appending block causes 
> "Unexpected checksum mismatch exception" 
> --
>
> Key: HDFS-6804
> URL: https://issues.apache.org/jira/browse/HDFS-6804
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.2.0
>Reporter: Gordon Wang
>Assignee: Brahma Reddy Battula
> Attachments: Testcase_append_transfer_block.patch
>
>
> We found some error log in the datanode. like this
> {noformat}
> 2014-07-22 01:49:51,338 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Ex
> ception for BP-2072804351-192.168.2.104-1406008383435:blk_1073741997_9248
> java.io.IOException: Terminating due to a checksum error.java.io.IOException: 
> Unexpected checksum mismatch while writing 
> BP-2072804351-192.168.2.104-1406008383435:blk_1073741997_9248 from 
> /192.168.2.101:39495
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:536)
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:703)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:575)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:115)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:68)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> While on the source datanode, the log says the block is transmitted.
> {noformat}
> 2014-07-22 01:49:50,805 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Da
> taTransfer: Transmitted 
> BP-2072804351-192.168.2.104-1406008383435:blk_1073741997
> _9248 (numBytes=16188152) to /192.168.2.103:50010
> {noformat}
> When the destination datanode gets the checksum mismatch, it reports bad 
> block to NameNode and NameNode marks the replica on the source datanode as 
> corrupt. But actually, the replica on the source datanode is valid. Because 
> the replica can pass the checksum verification.
> In all, the replica on the source data is wrongly marked as corrupted.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11420) XML format edit file should not be processed by XML processor in OfflineEditViewer

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869846#comment-15869846
 ] 

Hadoop QA commented on HDFS-11420:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 44s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}135m 34s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestBlocksScheduledCounter |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| Timed out junit tests | 
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11420 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12853021/HDFS-11420.001.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux e55d8cda0437 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 74dd142 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/18387/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/18387/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/18387/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> XML format edit file should not be processed by XML processor in 
> OfflineEditViewer
> --
>
> Key: 

[jira] [Commented] (HDFS-11402) HDFS Snapshots should capture point-in-time copies of OPEN files

[Yiqun Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869832#comment-15869832
 ] 

Yiqun Lin commented on HDFS-11402:
--

Hi, [~manojg], thanks for the work on this. 
I just take a quick look on this, some comments from me. I am also concerning 
the run time problem for this. Yes, you have mentioned that 
{{LeaseManager#getINodeWithLeases()}} will return set of open files in 
INodesInPath format under given {{INodeDirectory}}. And it will not deal with 
all the open files. But I looked into this method, it seems this operation is 
still expensive. The {{getInode}} and {{ INodesInPath#fromINode}} are still 
executed whether the file is under given directory or not. So if the open files 
is much enough, this method looks still expensive operation, right? How do you 
think?
{code}
public Set getINodeWithLeases(final INodeDirectory
+  restrictFilesFromDir) {
+Set iNodeSet = new HashSet<>();
+for (final long iNodeId : leasesById.keySet()) {
+  INode inode = fsnamesystem.getFSDirectory().getInode(iNodeId);
+  assert inode.isFile();
+  INodesInPath iNodesInPath = INodesInPath.fromINode(
+  fsnamesystem.getFSDirectory().getRoot(), inode.asFile());
+  if (restrictFilesFromDir != null &&
+  !iNodesInPath.isAncestor(restrictFilesFromDir)) {
+continue;
+  }
+  iNodeSet.add(iNodesInPath);
+}
+return iNodeSet;
+  }
{code}

> HDFS Snapshots should capture point-in-time copies of OPEN files
> 
>
> Key: HDFS-11402
> URL: https://issues.apache.org/jira/browse/HDFS-11402
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 2.6.0
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11402.01.patch
>
>
> *Problem:*
> 1. When there are files being written and when HDFS Snapshots are taken in 
> parallel, Snapshots do capture all these files, but these being written files 
> in Snapshots do not have the point-in-time file length captured. That is, 
> these open files are not frozen in HDFS Snapshots. These open files 
> grow/shrink in length, just like the original file, even after the snapshot 
> time.
> 2. At the time of File close or any other meta data modification operation on 
> these files, HDFS reconciles the file length and records the modification in 
> the last taken Snapshot. All the previously taken Snapshots continue to have 
> those open Files with no modification recorded. So, all those previous 
> snapshots end up using the final modification record in the last snapshot. 
> Thus after the file close, file lengths in all those snapshots will end up 
> same.
> Assume File1 is opened for write and a total of 1MB written to it. While the 
> writes are happening, snapshots are taken in parallel.
> {noformat}
> |---Time---T1---T2-T3T4-->
> |---Snap1--Snap2-Snap3--->
> |---File1.open---write-write---close->
> {noformat}
> Then at time,
> T2:
> Snap1.File1.length = 0
> T3:
> Snap1.File1.length = 0
> Snap2.File1.length = 0
> 
> T4:
> Snap1.File1.length = 1MB
> Snap2.File1.length = 1MB
> Snap3.File1.length = 1MB
> *Proposal*
> 1. At the time of taking Snapshot, {{SnapshotManager#createSnapshot}} can 
> optionally request {{DirectorySnapshottableFeature#addSnapshot}} to freeze 
> open files. 
> 2. {{DirectorySnapshottableFeature#addSnapshot}} can consult with 
> {{LeaseManager}} and get a list INodesInPath for all open files under the 
> snapshot dir. 
> 3. {{DirectorySnapshottableFeature#addSnapshot}} after the Snapshot creation, 
> Diff creation and updating modification time, can invoke 
> {{INodeFile#recordModification}} for each of the open files. This way, the 
> Snapshot just taken will have a {{FileDiff}} with {{fileSize}} captured for 
> each of the open files. 
> 4. Above model follows the current Snapshot and Diff protocols and doesn't 
> introduce any any disk formats. So, I don't think we will be needing any new 
> FSImage Loader/Saver changes for Snapshots.
> 5. One of the design goals of HDFS Snapshot was ability to take any number of 
> snapshots in O(1) time. LeaseManager though has all the open files with 
> leases in-memory map, an iteration is still needed to prune the needed open 
> files and then run recordModification on each of them. So, it will not be a 
> strict O(1) with the above proposal. But, its going be a marginal increase 
> only as the new order will be of O(open_files_under_snap_dir). In order to 
> avoid HDFS Snapshots change in behavior for open files and avoid change in 
> time complexity, this improvement can be made under a new config 
> {{"dfs.namenode.snapshot.freeze.openfiles"}} which 

[jira] [Created] (HDFS-11421) Make WebHDFS' ACLs RegEx configurable

[Harsh J (JIRA)]

Harsh J created HDFS-11421:
--

 Summary: Make WebHDFS' ACLs RegEx configurable
 Key: HDFS-11421
 URL: https://issues.apache.org/jira/browse/HDFS-11421
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: webhdfs
Reporter: Harsh J
Assignee: Harsh J


Part of HDFS-5608 added support for GET/SET ACLs over WebHDFS. This currently 
identifies the passed arguments via a hard-coded regex that mandates certain 
group and user naming styles.

A similar limitation had existed before for CHOWN and other User/Group set 
related operations of WebHDFS, where it was then made configurable via 
HDFS-11391 + HDFS-4983.

Such configurability should be allowed for the ACL operations too.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11420) XML format edit file should not be processed by XML processor in OfflineEditViewer

[Yiqun Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated HDFS-11420:
-
Attachment: HDFS-11420.001.patch

> XML format edit file should not be processed by XML processor in 
> OfflineEditViewer
> --
>
> Key: HDFS-11420
> URL: https://issues.apache.org/jira/browse/HDFS-11420
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11420.001.patch
>
>
> XML format edit file should not be processed by XML processor since it's no 
> meaning. The result is complete same with the input XML file. We should catch 
> this case and print the error info to let users know



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11420) XML format edit file should not be processed by XML processor in OfflineEditViewer

[Yiqun Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated HDFS-11420:
-
Status: Patch Available  (was: Open)

Initial patch attached. Kindly review.

> XML format edit file should not be processed by XML processor in 
> OfflineEditViewer
> --
>
> Key: HDFS-11420
> URL: https://issues.apache.org/jira/browse/HDFS-11420
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-11420.001.patch
>
>
> XML format edit file should not be processed by XML processor since it's no 
> meaning. The result is complete same with the input XML file. We should catch 
> this case and print the error info to let users know



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11420) XML format edit file should not be processed by XML processor in OfflineEditViewer

[Yiqun Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated HDFS-11420:
-
Description: XML format edit file should not be processed by XML processor 
since it's no meaning. The result is complete same with the input XML file. We 
should catch this case and print the error info to let users know  (was: XML 
format edit file should not be processed by XML processor since it's no 
meaning. The result is complete same with the input XML file. We should catch 
this case and print the error info to let uses know)

> XML format edit file should not be processed by XML processor in 
> OfflineEditViewer
> --
>
> Key: HDFS-11420
> URL: https://issues.apache.org/jira/browse/HDFS-11420
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>
> XML format edit file should not be processed by XML processor since it's no 
> meaning. The result is complete same with the input XML file. We should catch 
> this case and print the error info to let users know



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11420) XML format edit file should not be processed by XML processor in OfflineEditViewer

[Yiqun Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiqun Lin updated HDFS-11420:
-
Description: XML format edit file should not be processed by XML processor 
since it's no meaning. The result is complete same with the input XML file. We 
should catch this case and print the error info to let uses know

> XML format edit file should not be processed by XML processor in 
> OfflineEditViewer
> --
>
> Key: HDFS-11420
> URL: https://issues.apache.org/jira/browse/HDFS-11420
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>
> XML format edit file should not be processed by XML processor since it's no 
> meaning. The result is complete same with the input XML file. We should catch 
> this case and print the error info to let uses know



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-11420) XML format edit file should not be processed by XML processor in OfflineEditViewer

[Yiqun Lin (JIRA)]

Yiqun Lin created HDFS-11420:


 Summary: XML format edit file should not be processed by XML 
processor in OfflineEditViewer
 Key: HDFS-11420
 URL: https://issues.apache.org/jira/browse/HDFS-11420
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: tools
Affects Versions: 3.0.0-alpha2
Reporter: Yiqun Lin
Assignee: Yiqun Lin






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-11413) HDFS fsck command shows health as corrupt for '/'

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved HDFS-11413.

Resolution: Not A Bug

> HDFS fsck command shows health as corrupt for '/'
> -
>
> Key: HDFS-11413
> URL: https://issues.apache.org/jira/browse/HDFS-11413
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Nishant Verma
>
> I have open source hadoop version 2.7.3 cluster (2 Masters + 3 Slaves) 
> installed on AWS EC2 instances. I am using the cluster to integrate it with 
> Kafka Connect. 
> The setup of cluster was done last month and setup of kafka connect was 
> completed last fortnight. Since then, we were able to operate the kafka topic 
> records on our HDFS and do various operations.
> Since last afternoon, I find that any kafka topic is not getting committed to 
> the cluster. When I tried to open the older files, I started getting below 
> error. When I copy a new file to the cluster from local, it comes and gets 
> opened but after some time, again starts showing similar IOException:
> ==
> 17/02/14 07:57:55 INFO hdfs.DFSClient: No node available for 
> BP-1831277630-10.16.37.124-1484306078618:blk_1073793876_55013 
> file=/test/inputdata/derby.log
> 17/02/14 07:57:55 INFO hdfs.DFSClient: Could not obtain 
> BP-1831277630-10.16.37.124-1484306078618:blk_1073793876_55013 from any node: 
> java.io.IOException: No live nodes contain block 
> BP-1831277630-10.16.37.124-1484306078618:blk_1073793876_55013 after checking 
> nodes = [], ignoredNodes = null No live nodes contain current block Block 
> locations: Dead nodes: . Will get new block locations from namenode and 
> retry...
> 17/02/14 07:57:55 WARN hdfs.DFSClient: DFS chooseDataNode: got # 1 
> IOException, will wait for 499.3472970548959 msec.
> 17/02/14 07:57:55 INFO hdfs.DFSClient: No node available for 
> BP-1831277630-10.16.37.124-1484306078618:blk_1073793876_55013 
> file=/test/inputdata/derby.log
> 17/02/14 07:57:55 INFO hdfs.DFSClient: Could not obtain 
> BP-1831277630-10.16.37.124-1484306078618:blk_1073793876_55013 from any node: 
> java.io.IOException: No live nodes contain block 
> BP-1831277630-10.16.37.124-1484306078618:blk_1073793876_55013 after checking 
> nodes = [], ignoredNodes = null No live nodes contain current block Block 
> locations: Dead nodes: . Will get new block locations from namenode and 
> retry...
> 17/02/14 07:57:55 WARN hdfs.DFSClient: DFS chooseDataNode: got # 2 
> IOException, will wait for 4988.873277172643 msec.
> 17/02/14 07:58:00 INFO hdfs.DFSClient: No node available for 
> BP-1831277630-10.16.37.124-1484306078618:blk_1073793876_55013 
> file=/test/inputdata/derby.log
> 17/02/14 07:58:00 INFO hdfs.DFSClient: Could not obtain 
> BP-1831277630-10.16.37.124-1484306078618:blk_1073793876_55013 from any node: 
> java.io.IOException: No live nodes contain block 
> BP-1831277630-10.16.37.124-1484306078618:blk_1073793876_55013 after checking 
> nodes = [], ignoredNodes = null No live nodes contain current block Block 
> locations: Dead nodes: . Will get new block locations from namenode and 
> retry...
> 17/02/14 07:58:00 WARN hdfs.DFSClient: DFS chooseDataNode: got # 3 
> IOException, will wait for 8598.311122824263 msec.
> 17/02/14 07:58:09 WARN hdfs.DFSClient: Could not obtain block: 
> BP-1831277630-10.16.37.124-1484306078618:blk_1073793876_55013 
> file=/test/inputdata/derby.log No live nodes contain current block Block 
> locations: Dead nodes: . Throwing a BlockMissingException
> 17/02/14 07:58:09 WARN hdfs.DFSClient: Could not obtain block: 
> BP-1831277630-10.16.37.124-1484306078618:blk_1073793876_55013 
> file=/test/inputdata/derby.log No live nodes contain current block Block 
> locations: Dead nodes: . Throwing a BlockMissingException
> 17/02/14 07:58:09 WARN hdfs.DFSClient: DFS Read
> org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: 
> BP-1831277630-10.16.37.124-1484306078618:blk_1073793876_55013 
> file=/test/inputdata/derby.log
> at 
> org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:983)
> at 
> org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:642)
> at 
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)
> at java.io.DataInputStream.read(DataInputStream.java:100)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
> at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
> at 
> org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:107)
> at 
>