[jira] [Commented] (HDFS-9311) Support optional offload of NameNode HA service health checks to a separate RPC server.

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978342#comment-14978342
 ] 

Hudson commented on HDFS-9311:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #545 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/545/])
HDFS-9311. Support optional offload of NameNode HA service health checks 
(cnauroth: rev bf8e45298218f70e38838152f69c7705d8606bd6)
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitorWithDedicatedHealthAddress.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRespectsBindHostKeys.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/NNHAServiceTarget.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestNNHealthCheck.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java


> Support optional offload of NameNode HA service health checks to a separate 
> RPC server.
> ---
>
> Key: HDFS-9311
> URL: https://issues.apache.org/jira/browse/HDFS-9311
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: ha, namenode
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Fix For: 2.8.0
>
> Attachments: HDFS-9311.001.patch, HDFS-9311.002.patch, 
> HDFS-9311.003.patch
>
>
> When a NameNode is overwhelmed with load, it can lead to resource exhaustion 
> of the RPC handler pools (both client-facing and service-facing).  
> Eventually, this blocks the health check RPC issued from ZKFC, which triggers 
> a failover.  Depending on fencing configuration, the former active NameNode 
> may be killed.  In an overloaded situation, the new active NameNode is likely 
> to suffer the same fate, because client load patterns don't change after the 
> failover.  This can degenerate into flapping between the 2 NameNodes without 
> real recovery.  If a NameNode had been killed by fencing, then it would have 
> to transition through safe mode, further delaying time to recovery.
> This issue proposes a separate, optional RPC server at the NameNode for 
> isolating the HA health checks.  These health checks are lightweight 
> operations that do not suffer from contention issues on the namesystem lock 
> or other shared resources.  Isolating the RPC handlers is sufficient to avoid 
> this situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9317) Document fsck -blockId and -storagepolicy options in branch-2.7

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978340#comment-14978340
 ] 

Hudson commented on HDFS-9317:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #545 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/545/])
Add HDFS-9317 to CHANGES.txt. (aajisaka: rev 
eca51b13af1951b18ab7a5a92aaa0b9fe07d9ef8)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> Document fsck -blockId and -storagepolicy options in branch-2.7
> ---
>
> Key: HDFS-9317
> URL: https://issues.apache.org/jira/browse/HDFS-9317
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.7.1
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Fix For: 2.7.2
>
> Attachments: HDFS-9317-branch-2.7.00.patch
>
>
> fsck -blockId and -storagepolicy options are implemented by HDFS-6663 and 
> HDFS-7467 but the options are not documented in 2.7.1 release.
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#fsck



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9302) WebHDFS throws NullPointerException if newLength is not provided

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978339#comment-14978339
 ] 

Hudson commented on HDFS-9302:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #545 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/545/])
HDFS-9302. WebHDFS throws NullPointerException if newLength is not (yliu: rev 
6ff6663f64476eab5612ae9eb409104f44c6e6c7)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> WebHDFS throws NullPointerException if newLength is not provided
> 
>
> Key: HDFS-9302
> URL: https://issues.apache.org/jira/browse/HDFS-9302
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: HDFS
>Affects Versions: 2.7.1
> Environment: Centos6
>Reporter: Karthik Palaniappan
>Assignee: Jagadesh Kiran N
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9302_00.patch, HDFS-9302_01.patch
>
>
> $ curl -X POST "http://namenode:50070/webhdfs/v1/foo?op=truncate;
> {"RemoteException":{"exception":"NullPointerException","javaClassName":"java.lang.NullPointerException","message":null}}
> We should change newLength to be a required parameter in the webhdfs 
> documentation 
> (https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#New_Length),
>  and throw an IllegalArgumentException if isn't provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9231) fsck doesn't list correct file path when Bad Replicas/Blocks are in a snapshot

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978343#comment-14978343
 ] 

Hudson commented on HDFS-9231:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #545 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/545/])
HDFS-9231. fsck doesn't list correct file path when Bad Replicas/Blocks 
(yzhang: rev 97913f430cbe3f82ac866ae6ab8f42754102f6c0)
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSnapshotOp.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java


> fsck doesn't list correct file path when Bad Replicas/Blocks are in a snapshot
> --
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 2.8.0
>
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch, 
> HDFS-9231.003.patch, HDFS-9231.004.patch, HDFS-9231.005.patch, 
> HDFS-9231.006.patch, HDFS-9231.007.patch, HDFS-9231.008.patch, 
> HDFS-9231.009.patch
>
>
> Currently for snapshot files, {{fsck -list-corruptfileblocks}} shows corrupt 
> blocks with the original file dir instead of the snapshot dir, and {{fsck 
> -list-corruptfileblocks -includeSnapshots}} behave the same.
> This can be confusing because even when the original file is deleted, fsck 
> will still show that deleted file as corrupted, although what's actually 
> corrupted is the snapshot. 
> As a side note, {{fsck -files -includeSnapshots}} shows the snapshot dirs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8766) Implement a libhdfs(3) compatible API

[Bob Hansen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978366#comment-14978366
 ] 

Bob Hansen commented on HDFS-8766:
--

Yow, [~wheat9]!  That's a good catch on the promise/future bug.  The 
implementation of promise/future is a total failure of the Principle of Least 
Astonishment.  Thank you for linking to the reference.

> Implement a libhdfs(3) compatible API
> -
>
> Key: HDFS-8766
> URL: https://issues.apache.org/jira/browse/HDFS-8766
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
> Attachments: HDFS-8766.HDFS-8707.000.patch, 
> HDFS-8766.HDFS-8707.001.patch, HDFS-8766.HDFS-8707.002.patch, 
> HDFS-8766.HDFS-8707.003.patch, HDFS-8766.HDFS-8707.004.patch, 
> HDFS-8766.HDFS-8707.005.patch, HDFS-8766.HDFS-8707.006.patch, 
> HDFS-8766.HDFS-8707.007.patch, HDFS-8766.HDFS-8707.008.patch, 
> HDFS-8766.HDFS-8707.009.patch, HDFS-8766.HDFS-8707.010.patch
>
>
> Add a synchronous API that is compatible with the hdfs.h header used in 
> libhdfs and libhdfs3.  This will make it possible for projects using 
> libhdfs/libhdfs3 to relink against libhdfspp with minimal changes.
> This also provides a pure C interface that can be linked against projects 
> that aren't built in C++11 mode for various reasons but use the same 
> compiler.  It also allows many other programming languages to access 
> libhdfspp through builtin FFI interfaces.
> The libhdfs API is very similar to the posix file API which makes it easier 
> for programs built using posix filesystem calls to be modified to access HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9231) fsck doesn't list correct file path when Bad Replicas/Blocks are in a snapshot

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978246#comment-14978246
 ] 

Hudson commented on HDFS-9231:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #593 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/593/])
HDFS-9231. fsck doesn't list correct file path when Bad Replicas/Blocks 
(yzhang: rev 97913f430cbe3f82ac866ae6ab8f42754102f6c0)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSnapshotOp.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java


> fsck doesn't list correct file path when Bad Replicas/Blocks are in a snapshot
> --
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 2.8.0
>
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch, 
> HDFS-9231.003.patch, HDFS-9231.004.patch, HDFS-9231.005.patch, 
> HDFS-9231.006.patch, HDFS-9231.007.patch, HDFS-9231.008.patch, 
> HDFS-9231.009.patch
>
>
> Currently for snapshot files, {{fsck -list-corruptfileblocks}} shows corrupt 
> blocks with the original file dir instead of the snapshot dir, and {{fsck 
> -list-corruptfileblocks -includeSnapshots}} behave the same.
> This can be confusing because even when the original file is deleted, fsck 
> will still show that deleted file as corrupted, although what's actually 
> corrupted is the snapshot. 
> As a side note, {{fsck -files -includeSnapshots}} shows the snapshot dirs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9317) Document fsck -blockId and -storagepolicy options in branch-2.7

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978242#comment-14978242
 ] 

Hudson commented on HDFS-9317:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #593 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/593/])
Add HDFS-9317 to CHANGES.txt. (aajisaka: rev 
eca51b13af1951b18ab7a5a92aaa0b9fe07d9ef8)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> Document fsck -blockId and -storagepolicy options in branch-2.7
> ---
>
> Key: HDFS-9317
> URL: https://issues.apache.org/jira/browse/HDFS-9317
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.7.1
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Fix For: 2.7.2
>
> Attachments: HDFS-9317-branch-2.7.00.patch
>
>
> fsck -blockId and -storagepolicy options are implemented by HDFS-6663 and 
> HDFS-7467 but the options are not documented in 2.7.1 release.
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#fsck



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-7163) WebHdfsFileSystem should retry reads according to the configured retry policy.

[Eric Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Payne updated HDFS-7163:
-
Attachment: (was: HDFS-7163.001.patch)

> WebHdfsFileSystem should retry reads according to the configured retry policy.
> --
>
> Key: HDFS-7163
> URL: https://issues.apache.org/jira/browse/HDFS-7163
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.0.0, 2.5.1
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: WebHDFS Read Retry.pdf
>
>
> In the current implementation of WebHdfsFileSystem, opens are retried 
> according to the configured retry policy, but not reads. Therefore, if a 
> connection goes down while data is being read, the read will fail and the 
> read will have to be retried by the client code.
> Also, after a connection has been established, the next read (or seek/read) 
> will fail and the read will have to be restarted by the client code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-7163) WebHdfsFileSystem should retry reads according to the configured retry policy.

[Eric Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Payne updated HDFS-7163:
-
Status: Open  (was: Patch Available)

> WebHdfsFileSystem should retry reads according to the configured retry policy.
> --
>
> Key: HDFS-7163
> URL: https://issues.apache.org/jira/browse/HDFS-7163
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 2.5.1, 3.0.0
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: WebHDFS Read Retry.pdf
>
>
> In the current implementation of WebHdfsFileSystem, opens are retried 
> according to the configured retry policy, but not reads. Therefore, if a 
> connection goes down while data is being read, the read will fail and the 
> read will have to be retried by the client code.
> Also, after a connection has been established, the next read (or seek/read) 
> will fail and the read will have to be restarted by the client code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-7163) WebHdfsFileSystem should retry reads according to the configured retry policy.

[Eric Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Payne updated HDFS-7163:
-
Attachment: HDFS-7163.001.patch

> WebHdfsFileSystem should retry reads according to the configured retry policy.
> --
>
> Key: HDFS-7163
> URL: https://issues.apache.org/jira/browse/HDFS-7163
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.0.0, 2.5.1
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: HDFS-7163.001.patch, WebHDFS Read Retry.pdf
>
>
> In the current implementation of WebHdfsFileSystem, opens are retried 
> according to the configured retry policy, but not reads. Therefore, if a 
> connection goes down while data is being read, the read will fail and the 
> read will have to be retried by the client code.
> Also, after a connection has been established, the next read (or seek/read) 
> will fail and the read will have to be restarted by the client code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-7163) WebHdfsFileSystem should retry reads according to the configured retry policy.

[Eric Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Payne updated HDFS-7163:
-
Status: Patch Available  (was: Open)

> WebHdfsFileSystem should retry reads according to the configured retry policy.
> --
>
> Key: HDFS-7163
> URL: https://issues.apache.org/jira/browse/HDFS-7163
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 2.5.1, 3.0.0
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: HDFS-7163.001.patch, WebHDFS Read Retry.pdf
>
>
> In the current implementation of WebHdfsFileSystem, opens are retried 
> according to the configured retry policy, but not reads. Therefore, if a 
> connection goes down while data is being read, the read will fail and the 
> read will have to be retried by the client code.
> Also, after a connection has been established, the next read (or seek/read) 
> will fail and the read will have to be restarted by the client code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9083) Replication violates block placement policy.

[Rushabh S Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978452#comment-14978452
 ] 

Rushabh S Shah commented on HDFS-9083:
--

[~jingzhao] [~mingma] [~brahmareddy]: Thanks for the reviews.
I ran all the hdfs tests since jenkins failed to run the tests.
The following tests failed:
{noformat}
TestSecureNNWithQJM#testSecureMode
TestSecureNNWithQJM#testSecondaryNameNodeHttpAddressNotNeeded
TestAppendSnapshotTruncate#testAST
TestBalancer#testTwoReplicaShouldNotInSameDN
TestBalancer#testBalancerWithPinnedBlocks
TestBalancer#testBalancerWithZeroThreadsForMove
TestBalancerWithSaslDataTransfer#testBalancer0Integrity
TestBalancerWithSaslDataTransfer#testBalancer0Authentication
TestBalancerWithSaslDataTransfer#testBalancer0Privacy
TestBalancerWithNodeGroup#testBalancerWithNodeGroup
TestBalancerWithNodeGroup#testBalancerEndInNoMoveProgress
TestSaslDataTransfer#testServerSaslNoClientSasl
TestSaslDataTransfer#testClientAndServerDoNotHaveCommonQop
TestSaslDataTransfer#testAuthentication
TestSaslDataTransfer#testPrivacy
TestSaslDataTransfer#testNoSaslAndSecurePortsIgnored
TestSaslDataTransfer#testIntegrity
{noformat}
I ran all these tests multiple times.
All these tests failed always except TestAppendSnapshotTruncate#testAST, which 
failed intermittently.

I ran all the failed tests without my patch also and they failed.
So none of the test failures are related to my patch.
I will start the test-patch.sh on my machine  and upload the results shortly.

> Replication violates block placement policy.
> 
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: HDFS, namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Attachments: HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are 
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in 
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is 
> dependent on the config (net.topology.script.file.name)
> {noformat}
>  if (!this.shouldCheckForEnoughRacks) {
>   return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation) 
> via net.topology.node.switch.mapping.impl and no longer use 
> net.topology.script.file.name config.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9311) Support optional offload of NameNode HA service health checks to a separate RPC server.

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978245#comment-14978245
 ] 

Hudson commented on HDFS-9311:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #593 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/593/])
HDFS-9311. Support optional offload of NameNode HA service health checks 
(cnauroth: rev bf8e45298218f70e38838152f69c7705d8606bd6)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/NNHAServiceTarget.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitorWithDedicatedHealthAddress.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestNNHealthCheck.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRespectsBindHostKeys.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java


> Support optional offload of NameNode HA service health checks to a separate 
> RPC server.
> ---
>
> Key: HDFS-9311
> URL: https://issues.apache.org/jira/browse/HDFS-9311
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: ha, namenode
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Fix For: 2.8.0
>
> Attachments: HDFS-9311.001.patch, HDFS-9311.002.patch, 
> HDFS-9311.003.patch
>
>
> When a NameNode is overwhelmed with load, it can lead to resource exhaustion 
> of the RPC handler pools (both client-facing and service-facing).  
> Eventually, this blocks the health check RPC issued from ZKFC, which triggers 
> a failover.  Depending on fencing configuration, the former active NameNode 
> may be killed.  In an overloaded situation, the new active NameNode is likely 
> to suffer the same fate, because client load patterns don't change after the 
> failover.  This can degenerate into flapping between the 2 NameNodes without 
> real recovery.  If a NameNode had been killed by fencing, then it would have 
> to transition through safe mode, further delaying time to recovery.
> This issue proposes a separate, optional RPC server at the NameNode for 
> isolating the HA health checks.  These health checks are lightweight 
> operations that do not suffer from contention issues on the namesystem lock 
> or other shared resources.  Isolating the RPC handlers is sufficient to avoid 
> this situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9302) WebHDFS throws NullPointerException if newLength is not provided

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978241#comment-14978241
 ] 

Hudson commented on HDFS-9302:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #593 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/593/])
HDFS-9302. WebHDFS throws NullPointerException if newLength is not (yliu: rev 
6ff6663f64476eab5612ae9eb409104f44c6e6c7)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> WebHDFS throws NullPointerException if newLength is not provided
> 
>
> Key: HDFS-9302
> URL: https://issues.apache.org/jira/browse/HDFS-9302
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: HDFS
>Affects Versions: 2.7.1
> Environment: Centos6
>Reporter: Karthik Palaniappan
>Assignee: Jagadesh Kiran N
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9302_00.patch, HDFS-9302_01.patch
>
>
> $ curl -X POST "http://namenode:50070/webhdfs/v1/foo?op=truncate;
> {"RemoteException":{"exception":"NullPointerException","javaClassName":"java.lang.NullPointerException","message":null}}
> We should change newLength to be a required parameter in the webhdfs 
> documentation 
> (https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#New_Length),
>  and throw an IllegalArgumentException if isn't provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8766) Implement a libhdfs(3) compatible API

[James Clampffer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978378#comment-14978378
 ] 

James Clampffer commented on HDFS-8766:
---

Thanks for the review Haohui!

Good catch on the "using namespace" in the header, that must have ended up 
there while I was splitting the files apart.  I'll fix that, the comment issue, 
and go back to using shared_ptrs to the promises to fix the race condition 
similar to your fix on the old github branch.

> Implement a libhdfs(3) compatible API
> -
>
> Key: HDFS-8766
> URL: https://issues.apache.org/jira/browse/HDFS-8766
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
> Attachments: HDFS-8766.HDFS-8707.000.patch, 
> HDFS-8766.HDFS-8707.001.patch, HDFS-8766.HDFS-8707.002.patch, 
> HDFS-8766.HDFS-8707.003.patch, HDFS-8766.HDFS-8707.004.patch, 
> HDFS-8766.HDFS-8707.005.patch, HDFS-8766.HDFS-8707.006.patch, 
> HDFS-8766.HDFS-8707.007.patch, HDFS-8766.HDFS-8707.008.patch, 
> HDFS-8766.HDFS-8707.009.patch, HDFS-8766.HDFS-8707.010.patch
>
>
> Add a synchronous API that is compatible with the hdfs.h header used in 
> libhdfs and libhdfs3.  This will make it possible for projects using 
> libhdfs/libhdfs3 to relink against libhdfspp with minimal changes.
> This also provides a pure C interface that can be linked against projects 
> that aren't built in C++11 mode for various reasons but use the same 
> compiler.  It also allows many other programming languages to access 
> libhdfspp through builtin FFI interfaces.
> The libhdfs API is very similar to the posix file API which makes it easier 
> for programs built using posix filesystem calls to be modified to access HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9255) Consolidate block recovery related implementation into a single class

[Walter Su (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Walter Su updated HDFS-9255:

Attachment: HDFS-9255-branch-2.06.patch

Thanks [~zhz], [~jingzhao] for review!
And sorry for the trouble, [~zhz]. Uploaded HDFS-9255-branch-2.06.patch.

> Consolidate block recovery related implementation into a single class
> -
>
> Key: HDFS-9255
> URL: https://issues.apache.org/jira/browse/HDFS-9255
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Reporter: Walter Su
>Assignee: Walter Su
>Priority: Minor
> Attachments: HDFS-9255-branch-2.06.patch, HDFS-9255.01.patch, 
> HDFS-9255.02.patch, HDFS-9255.03.patch, HDFS-9255.04.patch, 
> HDFS-9255.05.patch, HDFS-9255.06.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9229) Expose size of NameNode directory as a metric

[Surendra Singh Lilhore (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978289#comment-14978289
 ] 

Surendra Singh Lilhore commented on HDFS-9229:
--

Thanks [~zhz] for review..

bq. The behavior of shared directories is worth more discussions. The current 
patch returns 0 if the directory is shared. Since the purpose of this new 
metric is for local storage planning / provisioning, shall we report size of 
shared dirs as well?

Do we have any interface to get size of shared dirs in namenode ?.

bq. The temporary nnDirSizeMap is not necessary. I think we can directly clear 
and add to nameDirSizeMap.

This I did purposefully. Getting size for all the dir and updating in map will 
take some time because it is I/O call. Suppose we updated size for one dir and 
in-between some one read that map then he will get incomplete info.


> Expose size of NameNode directory as a metric
> -
>
> Key: HDFS-9229
> URL: https://issues.apache.org/jira/browse/HDFS-9229
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Affects Versions: 2.7.1
>Reporter: Zhe Zhang
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Attachments: HDFS-9229.001.patch, HDFS-9229.002.patch, 
> HDFS-9229.003.patch, HDFS-9229.004.patch, HDFS-9229.005.patch
>
>
> Useful for admins in reserving / managing NN local file system space. Also 
> useful when transferring NN backups.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9302) WebHDFS throws NullPointerException if newLength is not provided

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978331#comment-14978331
 ] 

Hudson commented on HDFS-9302:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk #2483 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2483/])
HDFS-9302. WebHDFS throws NullPointerException if newLength is not (yliu: rev 
6ff6663f64476eab5612ae9eb409104f44c6e6c7)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> WebHDFS throws NullPointerException if newLength is not provided
> 
>
> Key: HDFS-9302
> URL: https://issues.apache.org/jira/browse/HDFS-9302
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: HDFS
>Affects Versions: 2.7.1
> Environment: Centos6
>Reporter: Karthik Palaniappan
>Assignee: Jagadesh Kiran N
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9302_00.patch, HDFS-9302_01.patch
>
>
> $ curl -X POST "http://namenode:50070/webhdfs/v1/foo?op=truncate;
> {"RemoteException":{"exception":"NullPointerException","javaClassName":"java.lang.NullPointerException","message":null}}
> We should change newLength to be a required parameter in the webhdfs 
> documentation 
> (https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#New_Length),
>  and throw an IllegalArgumentException if isn't provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9311) Support optional offload of NameNode HA service health checks to a separate RPC server.

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978335#comment-14978335
 ] 

Hudson commented on HDFS-9311:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk #2483 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2483/])
HDFS-9311. Support optional offload of NameNode HA service health checks 
(cnauroth: rev bf8e45298218f70e38838152f69c7705d8606bd6)
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRespectsBindHostKeys.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestNNHealthCheck.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitorWithDedicatedHealthAddress.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/NNHAServiceTarget.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java


> Support optional offload of NameNode HA service health checks to a separate 
> RPC server.
> ---
>
> Key: HDFS-9311
> URL: https://issues.apache.org/jira/browse/HDFS-9311
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: ha, namenode
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Fix For: 2.8.0
>
> Attachments: HDFS-9311.001.patch, HDFS-9311.002.patch, 
> HDFS-9311.003.patch
>
>
> When a NameNode is overwhelmed with load, it can lead to resource exhaustion 
> of the RPC handler pools (both client-facing and service-facing).  
> Eventually, this blocks the health check RPC issued from ZKFC, which triggers 
> a failover.  Depending on fencing configuration, the former active NameNode 
> may be killed.  In an overloaded situation, the new active NameNode is likely 
> to suffer the same fate, because client load patterns don't change after the 
> failover.  This can degenerate into flapping between the 2 NameNodes without 
> real recovery.  If a NameNode had been killed by fencing, then it would have 
> to transition through safe mode, further delaying time to recovery.
> This issue proposes a separate, optional RPC server at the NameNode for 
> isolating the HA health checks.  These health checks are lightweight 
> operations that do not suffer from contention issues on the namesystem lock 
> or other shared resources.  Isolating the RPC handlers is sufficient to avoid 
> this situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9231) fsck doesn't list correct file path when Bad Replicas/Blocks are in a snapshot

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978336#comment-14978336
 ] 

Hudson commented on HDFS-9231:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk #2483 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2483/])
HDFS-9231. fsck doesn't list correct file path when Bad Replicas/Blocks 
(yzhang: rev 97913f430cbe3f82ac866ae6ab8f42754102f6c0)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSnapshotOp.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java


> fsck doesn't list correct file path when Bad Replicas/Blocks are in a snapshot
> --
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 2.8.0
>
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch, 
> HDFS-9231.003.patch, HDFS-9231.004.patch, HDFS-9231.005.patch, 
> HDFS-9231.006.patch, HDFS-9231.007.patch, HDFS-9231.008.patch, 
> HDFS-9231.009.patch
>
>
> Currently for snapshot files, {{fsck -list-corruptfileblocks}} shows corrupt 
> blocks with the original file dir instead of the snapshot dir, and {{fsck 
> -list-corruptfileblocks -includeSnapshots}} behave the same.
> This can be confusing because even when the original file is deleted, fsck 
> will still show that deleted file as corrupted, although what's actually 
> corrupted is the snapshot. 
> As a side note, {{fsck -files -includeSnapshots}} shows the snapshot dirs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9317) Document fsck -blockId and -storagepolicy options in branch-2.7

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978332#comment-14978332
 ] 

Hudson commented on HDFS-9317:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk #2483 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2483/])
Add HDFS-9317 to CHANGES.txt. (aajisaka: rev 
eca51b13af1951b18ab7a5a92aaa0b9fe07d9ef8)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> Document fsck -blockId and -storagepolicy options in branch-2.7
> ---
>
> Key: HDFS-9317
> URL: https://issues.apache.org/jira/browse/HDFS-9317
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.7.1
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Fix For: 2.7.2
>
> Attachments: HDFS-9317-branch-2.7.00.patch
>
>
> fsck -blockId and -storagepolicy options are implemented by HDFS-6663 and 
> HDFS-7467 but the options are not documented in 2.7.1 release.
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#fsck



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9117) Config file reader / options classes for libhdfs++

[Bob Hansen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bob Hansen updated HDFS-9117:
-
Attachment: HDFS-9117.HDFS-8707.008.patch

> Config file reader / options classes for libhdfs++
> --
>
> Key: HDFS-9117
> URL: https://issues.apache.org/jira/browse/HDFS-9117
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Affects Versions: HDFS-8707
>Reporter: Bob Hansen
>Assignee: Bob Hansen
> Attachments: HDFS-9117.HDFS-8707.001.patch, 
> HDFS-9117.HDFS-8707.002.patch, HDFS-9117.HDFS-8707.003.patch, 
> HDFS-9117.HDFS-8707.004.patch, HDFS-9117.HDFS-8707.005.patch, 
> HDFS-9117.HDFS-8707.006.patch, HDFS-9117.HDFS-8707.008.patch, 
> HDFS-9117.HDFS-9288.007.patch
>
>
> For environmental compatability with HDFS installations, libhdfs++ should be 
> able to read the configurations from Hadoop XML files and behave in line with 
> the Java implementation.
> Most notably, machine names and ports should be readable from Hadoop XML 
> configuration files.
> Similarly, an internal Options architecture for libhdfs++ should be developed 
> to efficiently transport the configuration information within the system.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9325) Allow the location of hdfs.h to be passed to CMake during a build.

[James Clampffer (JIRA)]

James Clampffer created HDFS-9325:
-

 Summary: Allow the location of hdfs.h to be passed to CMake during 
a build.
 Key: HDFS-9325
 URL: https://issues.apache.org/jira/browse/HDFS-9325
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: James Clampffer
Assignee: James Clampffer


It would be nice if CMake could take an optional parameter with the location of 
hdfs.h that typically lives at libhdfs/includes/hdfs/hdfs.h, otherwise it would 
default to this location.  This would be useful for projects using libhdfs++ 
that gather headers defining library APIs in a single location.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-8766) Implement a libhdfs(3) compatible API

[James Clampffer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Clampffer updated HDFS-8766:
--
Attachment: HDFS-8766.HDFS-8707.011.patch

Should have all the fixes for your last comment Haohui:
-Updated the style for block comments.
-Moved the hdfs_internal and hdfsFile_internal implementations into the same 
file as the C API functions because they shouldn't be visible elsewhere anyway. 
 This also got the "include namespace hdfs" out of the header.
-All promises are made using make_shared, the lambda captures the shared_ptr by 
value to bump the reference count to prevent the race condition.
 

> Implement a libhdfs(3) compatible API
> -
>
> Key: HDFS-8766
> URL: https://issues.apache.org/jira/browse/HDFS-8766
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
> Attachments: HDFS-8766.HDFS-8707.000.patch, 
> HDFS-8766.HDFS-8707.001.patch, HDFS-8766.HDFS-8707.002.patch, 
> HDFS-8766.HDFS-8707.003.patch, HDFS-8766.HDFS-8707.004.patch, 
> HDFS-8766.HDFS-8707.005.patch, HDFS-8766.HDFS-8707.006.patch, 
> HDFS-8766.HDFS-8707.007.patch, HDFS-8766.HDFS-8707.008.patch, 
> HDFS-8766.HDFS-8707.009.patch, HDFS-8766.HDFS-8707.010.patch, 
> HDFS-8766.HDFS-8707.011.patch
>
>
> Add a synchronous API that is compatible with the hdfs.h header used in 
> libhdfs and libhdfs3.  This will make it possible for projects using 
> libhdfs/libhdfs3 to relink against libhdfspp with minimal changes.
> This also provides a pure C interface that can be linked against projects 
> that aren't built in C++11 mode for various reasons but use the same 
> compiler.  It also allows many other programming languages to access 
> libhdfspp through builtin FFI interfaces.
> The libhdfs API is very similar to the posix file API which makes it easier 
> for programs built using posix filesystem calls to be modified to access HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9117) Config file reader / options classes for libhdfs++

[Bob Hansen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bob Hansen updated HDFS-9117:
-
Attachment: HDFS-9117.HDFS-8707.009.patch

Patch added that implements the above changes.  Please review at your leisure.

> Config file reader / options classes for libhdfs++
> --
>
> Key: HDFS-9117
> URL: https://issues.apache.org/jira/browse/HDFS-9117
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Affects Versions: HDFS-8707
>Reporter: Bob Hansen
>Assignee: Bob Hansen
> Attachments: HDFS-9117.HDFS-8707.001.patch, 
> HDFS-9117.HDFS-8707.002.patch, HDFS-9117.HDFS-8707.003.patch, 
> HDFS-9117.HDFS-8707.004.patch, HDFS-9117.HDFS-8707.005.patch, 
> HDFS-9117.HDFS-8707.006.patch, HDFS-9117.HDFS-8707.008.patch, 
> HDFS-9117.HDFS-8707.009.patch, HDFS-9117.HDFS-9288.007.patch
>
>
> For environmental compatability with HDFS installations, libhdfs++ should be 
> able to read the configurations from Hadoop XML files and behave in line with 
> the Java implementation.
> Most notably, machine names and ports should be readable from Hadoop XML 
> configuration files.
> Similarly, an internal Options architecture for libhdfs++ should be developed 
> to efficiently transport the configuration information within the system.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8333) Create EC zone should not need superuser privilege

[Yong Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978574#comment-14978574
 ] 

Yong Zhang commented on HDFS-8333:
--

Hi [~szetszwo], currently we have user cases of multi-tenancy on hdfs, if only 
superuser/supergroup has right to set ec zone, it will be so limitation.
In my opinion, it is better for each tenant to have its own administrator, and 
manage everything belong to him.

> Create EC zone should not need superuser privilege
> --
>
> Key: HDFS-8333
> URL: https://issues.apache.org/jira/browse/HDFS-8333
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Yong Zhang
>Assignee: Yong Zhang
> Attachments: HDFS-8333-HDFS-7285.000.patch
>
>
> create EC zone should not need superuser privilege, for example, in multiple 
> tenant scenario, common users only manage their own directory and 
> subdirectory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9117) Config file reader / options classes for libhdfs++

[Bob Hansen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978590#comment-14978590
 ] 

Bob Hansen commented on HDFS-9117:
--

bq. 1. It's beneficial to further separate the patch, such as substituting 
variables, and all the TODO items to a separate jira.
Removed the variable substitution, and re-cast the TODOs as a comment capturing 
"Here's the things you might expect this class to do, that it doesn't do yet"

bq. 2. The code should not use C++ exceptions.
I think the code was cleaner using the std::string parsers (which had 
exceptions that were caught), but I moved them to the strtoXXX functions.

bq. 3. That current interface is insufficient in terms of not every 
configuration has a default value.
It's a good point; I was unaware that we had use cases where there would be no 
default.  Unfortunately, we don't have boost::optional available, so the API is 
a bit muddier now.

{quote}
4. It's much cleaner to implement the final attribute with priority. The 
configuration follow the follow rules:

The value reads later overrides the earlier values.
Values that are tagged as final overrides the previous value.

It essentially defines a total order which can be mapped to an integer.
{quote}
The implementation follows the rules outlined (and all the tests over them).  I 
implemented it as a final flag on all the values which prevents future 
resources from changing it, which seems a very natural implementation.  Is the 
current implementation wrong in any way?

bq. 6. I suggest leaving environment substitutions and other platform-specific 
functionality out (at least for now).
It has been moved over with the other substitutions.  It's implemented, tested, 
and part of the spec of the Java config files, so I don't see any reason to 
excise it.

bq. 7. I suggest putting the code along with the libhdfs compatibility layer.
I disagree; the code is required for a pure C++ implementation that satisfies 
the use case of "Read the local config and connect to HDFS."  The libhdfs 
compatability layer should be for the glue for the C API.


> Config file reader / options classes for libhdfs++
> --
>
> Key: HDFS-9117
> URL: https://issues.apache.org/jira/browse/HDFS-9117
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Affects Versions: HDFS-8707
>Reporter: Bob Hansen
>Assignee: Bob Hansen
> Attachments: HDFS-9117.HDFS-8707.001.patch, 
> HDFS-9117.HDFS-8707.002.patch, HDFS-9117.HDFS-8707.003.patch, 
> HDFS-9117.HDFS-8707.004.patch, HDFS-9117.HDFS-8707.005.patch, 
> HDFS-9117.HDFS-8707.006.patch, HDFS-9117.HDFS-8707.008.patch, 
> HDFS-9117.HDFS-9288.007.patch
>
>
> For environmental compatability with HDFS installations, libhdfs++ should be 
> able to read the configurations from Hadoop XML files and behave in line with 
> the Java implementation.
> Most notably, machine names and ports should be readable from Hadoop XML 
> configuration files.
> Similarly, an internal Options architecture for libhdfs++ should be developed 
> to efficiently transport the configuration information within the system.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9255) Consolidate block recovery related implementation into a single class

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-9255:

  Resolution: Fixed
Hadoop Flags: Reviewed
   Fix Version/s: 2.8.0
Target Version/s: 2.8.0
  Status: Resolved  (was: Patch Available)

Thanks Walter! I just committed to both trunk and branch-2. Thanks [~rakeshr] 
and [~jingzhao] for the helpful reviews as well.

> Consolidate block recovery related implementation into a single class
> -
>
> Key: HDFS-9255
> URL: https://issues.apache.org/jira/browse/HDFS-9255
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Reporter: Walter Su
>Assignee: Walter Su
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9255-branch-2.06.patch, HDFS-9255.01.patch, 
> HDFS-9255.02.patch, HDFS-9255.03.patch, HDFS-9255.04.patch, 
> HDFS-9255.05.patch, HDFS-9255.06.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9326) Create a generic function to synchronize async functions and methods.

[James Clampffer (JIRA)]

James Clampffer created HDFS-9326:
-

 Summary: Create a generic function to synchronize async functions 
and methods. 
 Key: HDFS-9326
 URL: https://issues.apache.org/jira/browse/HDFS-9326
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: James Clampffer


The majority of the functionality in libhdfs++ is asynchronous, but some 
applications need synchronous operations.  At the time of filing this only 
happens in 3 places in the C API, however that number is going to grow a lot 
once the C and high level C++ APIs expose all of the namenode functions.

This synchronization is typically implemented like this:
auto promise = std::make_shared()
std::future = future(promise->get_future());

auto async_callback = [promise] () {promise->set_value(val);};

SomeClass::AsyncMethod(async_callback); 

auto result = future.get()

Ideally this could all be pushed into a templated function so that the promise 
and future don't need to be defined at the call site.  This would probably take 
the form of doing a std::bind to get all the arguments in place at the call 
site and then passing that to the synchronize function.

This appears to require some template magic that isn't always well supported; 
see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51979.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9317) Document fsck -blockId and -storagepolicy options in branch-2.7

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977946#comment-14977946
 ] 

Hudson commented on HDFS-9317:
--

FAILURE: Integrated in Hadoop-trunk-Commit #8717 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/8717/])
Add HDFS-9317 to CHANGES.txt. (aajisaka: rev 
eca51b13af1951b18ab7a5a92aaa0b9fe07d9ef8)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> Document fsck -blockId and -storagepolicy options in branch-2.7
> ---
>
> Key: HDFS-9317
> URL: https://issues.apache.org/jira/browse/HDFS-9317
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.7.1
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Fix For: 2.7.2
>
> Attachments: HDFS-9317-branch-2.7.00.patch
>
>
> fsck -blockId and -storagepolicy options are implemented by HDFS-6663 and 
> HDFS-7467 but the options are not documented in 2.7.1 release.
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#fsck



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9311) Support optional offload of NameNode HA service health checks to a separate RPC server.

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977948#comment-14977948
 ] 

Hudson commented on HDFS-9311:
--

FAILURE: Integrated in Hadoop-trunk-Commit #8717 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/8717/])
HDFS-9311. Support optional offload of NameNode HA service health checks 
(cnauroth: rev bf8e45298218f70e38838152f69c7705d8606bd6)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/NNHAServiceTarget.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestNNHealthCheck.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRespectsBindHostKeys.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitorWithDedicatedHealthAddress.java


> Support optional offload of NameNode HA service health checks to a separate 
> RPC server.
> ---
>
> Key: HDFS-9311
> URL: https://issues.apache.org/jira/browse/HDFS-9311
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: ha, namenode
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Fix For: 2.8.0
>
> Attachments: HDFS-9311.001.patch, HDFS-9311.002.patch, 
> HDFS-9311.003.patch
>
>
> When a NameNode is overwhelmed with load, it can lead to resource exhaustion 
> of the RPC handler pools (both client-facing and service-facing).  
> Eventually, this blocks the health check RPC issued from ZKFC, which triggers 
> a failover.  Depending on fencing configuration, the former active NameNode 
> may be killed.  In an overloaded situation, the new active NameNode is likely 
> to suffer the same fate, because client load patterns don't change after the 
> failover.  This can degenerate into flapping between the 2 NameNodes without 
> real recovery.  If a NameNode had been killed by fencing, then it would have 
> to transition through safe mode, further delaying time to recovery.
> This issue proposes a separate, optional RPC server at the NameNode for 
> isolating the HA health checks.  These health checks are lightweight 
> operations that do not suffer from contention issues on the namesystem lock 
> or other shared resources.  Isolating the RPC handlers is sufficient to avoid 
> this situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-8545) Refactor FS#getUsed() to use ContentSummary and add an API to fetch the total file length from a specific path

[J.Andreina (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

J.Andreina updated HDFS-8545:
-
Attachment: HDFS-8545.6.patch

Thanks [~vinayrpet] for the review comments.
I have updated the patch. Please review.

> Refactor FS#getUsed() to use ContentSummary and add an API to fetch the total 
> file length from a specific path
> --
>
> Key: HDFS-8545
> URL: https://issues.apache.org/jira/browse/HDFS-8545
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Minor
> Attachments: HDFS-8545.1.patch, HDFS-8545.2.patch, HDFS-8545.3.patch, 
> HDFS-8545.4.patch, HDFS-8545.5.patch, HDFS-8545.6.patch
>
>
> Currently by default in FileSystem#getUsed() returns the total file size from 
> root. 
> It is good to have an api to return the total file size from specified path 
> ,same as we specify the path in "./hdfs dfs -du -s /path" .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group

[J.Andreina (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

J.Andreina updated HDFS-9044:
-
Attachment: HDFS-9044.6.patch

Thanks [~vinayrpet] for the review comments.
Updated the patch please review

> Give Priority to FavouredNodes , before selecting nodes from FavouredNode's 
> Node Group
> --
>
> Key: HDFS-9044
> URL: https://issues.apache.org/jira/browse/HDFS-9044
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9044.1.patch, HDFS-9044.2.patch, HDFS-9044.3.patch, 
> HDFS-9044.4.patch, HDFS-9044.5.patch, HDFS-9044.6.patch
>
>
> Passing Favored nodes intention is to place replica among the favored node
> Current behavior in Node group is 
>   If favored node is not available it goes to one among favored 
> nodegroup. 
> {noformat}
> Say for example:
>   1)I need 3 replicas and passed 5 favored nodes.
>   2)Out of 5 favored nodes 3 favored nodes are not good.
>   3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node 
> returned , 3 will be random node from 3 bad FavoredNode's nodegroup. 
>   4)Then there is a probability that all my 3 replicas are placed on 
> Random node from FavoredNodes's nodegroup , instead of giving priority to 2 
> favored nodes returned as target.
> {noformat}
> *Instead of returning 5 targets on 3rd step above , we can return 2 good 
> favored nodes as target*
> *And remaining 1 needed replica can be chosen from Random node of bad 
> FavoredNodes's nodegroup.*
> This will make sure that the FavoredNodes are given priority.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9322) Keep order of addresses to nameservice mapping from configuration file

[Rafal Wojdyla (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rafal Wojdyla updated HDFS-9322:

Attachment: HDFS-9322.patch_1

> Keep order of addresses to nameservice mapping from configuration file
> --
>
> Key: HDFS-9322
> URL: https://issues.apache.org/jira/browse/HDFS-9322
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Rafal Wojdyla
>Assignee: Rafal Wojdyla
>  Labels: has-patch
> Attachments: HDFS-9322.patch_1
>
>
> getAddressesForNameserviceId which is used by ConfiguredFailoverProxyProvider 
> does not keep order of namenodes/addresses from configuration file - instead 
> it relays on order given by HashMap (key is service id) which is misaligned 
> with comment/doc in ConfiguredFailoverProxyProvider that says:
> {code}
> /**
>  * A FailoverProxyProvider implementation which allows one to configure two 
> URIs
>  * to connect to during fail-over. The first configured address is tried 
> first,
>  * and on a fail-over event the other address is tried.
>  */
> {code}
> One solution is to use LinkedHashMap which is insertion-ordered. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9322) Keep order of addresses to nameservice mapping from configuration file

[Rafal Wojdyla (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rafal Wojdyla updated HDFS-9322:

Labels: has-patch  (was: )
Status: Patch Available  (was: Open)

> Keep order of addresses to nameservice mapping from configuration file
> --
>
> Key: HDFS-9322
> URL: https://issues.apache.org/jira/browse/HDFS-9322
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Rafal Wojdyla
>Assignee: Rafal Wojdyla
>  Labels: has-patch
> Attachments: HDFS-9322.patch_1
>
>
> getAddressesForNameserviceId which is used by ConfiguredFailoverProxyProvider 
> does not keep order of namenodes/addresses from configuration file - instead 
> it relays on order given by HashMap (key is service id) which is misaligned 
> with comment/doc in ConfiguredFailoverProxyProvider that says:
> {code}
> /**
>  * A FailoverProxyProvider implementation which allows one to configure two 
> URIs
>  * to connect to during fail-over. The first configured address is tried 
> first,
>  * and on a fail-over event the other address is tried.
>  */
> {code}
> One solution is to use LinkedHashMap which is insertion-ordered. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9302) WebHDFS throws NullPointerException if newLength is not provided

[Yi Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977821#comment-14977821
 ] 

Yi Liu commented on HDFS-9302:
--

OK, thanks Jagadesh, Akira, will commit shortly.

> WebHDFS throws NullPointerException if newLength is not provided
> 
>
> Key: HDFS-9302
> URL: https://issues.apache.org/jira/browse/HDFS-9302
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: HDFS
>Affects Versions: 2.7.1
> Environment: Centos6
>Reporter: Karthik Palaniappan
>Assignee: Jagadesh Kiran N
>Priority: Minor
> Attachments: HDFS-9302_00.patch, HDFS-9302_01.patch
>
>
> $ curl -X POST "http://namenode:50070/webhdfs/v1/foo?op=truncate;
> {"RemoteException":{"exception":"NullPointerException","javaClassName":"java.lang.NullPointerException","message":null}}
> We should change newLength to be a required parameter in the webhdfs 
> documentation 
> (https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#New_Length),
>  and throw an IllegalArgumentException if isn't provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Moved] (HDFS-9323) Randomize the DFSStripedOutputStreamWithFailure tests

[Tsz Wo Nicholas Sze (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo Nicholas Sze moved HADOOP-12524 to HDFS-9323:


Component/s: (was: test)
 test
Key: HDFS-9323  (was: HADOOP-12524)
Project: Hadoop HDFS  (was: Hadoop Common)

> Randomize the DFSStripedOutputStreamWithFailure tests
> -
>
> Key: HDFS-9323
> URL: https://issues.apache.org/jira/browse/HDFS-9323
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: test
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>Priority: Minor
>
> Currently, the DFSStripedOutputStreamWithFailure tests are run with fixed 
> indices #0 - #19 and #59, and also a test with random index 
> (testDatanodeFailureRandomLength).  We should randomize all the tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9323) Randomize the DFSStripedOutputStreamWithFailure tests

[Tsz Wo Nicholas Sze (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo Nicholas Sze updated HDFS-9323:
--
Attachment: h9323_20151028.patch

h9323_20151028.patch: randomize all the tests.

> Randomize the DFSStripedOutputStreamWithFailure tests
> -
>
> Key: HDFS-9323
> URL: https://issues.apache.org/jira/browse/HDFS-9323
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: test
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>Priority: Minor
> Attachments: h9323_20151028.patch
>
>
> Currently, the DFSStripedOutputStreamWithFailure tests are run with fixed 
> indices #0 - #19 and #59, and also a test with random index 
> (testDatanodeFailureRandomLength).  We should randomize all the tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9231) fsck doesn't list correct file path when Bad Replicas/Blocks are in a snapshot

[Yongjun Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yongjun Zhang updated HDFS-9231:

Summary: fsck doesn't list correct file path when Bad Replicas/Blocks are 
in a snapshot  (was: fsck doesn't explicitly list when Bad Replicas/Blocks are 
in a snapshot)

> fsck doesn't list correct file path when Bad Replicas/Blocks are in a snapshot
> --
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch, 
> HDFS-9231.003.patch, HDFS-9231.004.patch, HDFS-9231.005.patch, 
> HDFS-9231.006.patch, HDFS-9231.007.patch, HDFS-9231.008.patch, 
> HDFS-9231.009.patch
>
>
> Currently for snapshot files, {{fsck -list-corruptfileblocks}} shows corrupt 
> blocks with the original file dir instead of the snapshot dir, and {{fsck 
> -list-corruptfileblocks -includeSnapshots}} behave the same.
> This can be confusing because even when the original file is deleted, fsck 
> will still show that deleted file as corrupted, although what's actually 
> corrupted is the snapshot. 
> As a side note, {{fsck -files -includeSnapshots}} shows the snapshot dirs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9129) Move the safemode block count into BlockManager

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HDFS-9129:

Attachment: HDFS-9129.012.patch

> Move the safemode block count into BlockManager
> ---
>
> Key: HDFS-9129
> URL: https://issues.apache.org/jira/browse/HDFS-9129
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Mingliang Liu
> Attachments: HDFS-9129.000.patch, HDFS-9129.001.patch, 
> HDFS-9129.002.patch, HDFS-9129.003.patch, HDFS-9129.004.patch, 
> HDFS-9129.005.patch, HDFS-9129.006.patch, HDFS-9129.007.patch, 
> HDFS-9129.008.patch, HDFS-9129.009.patch, HDFS-9129.010.patch, 
> HDFS-9129.011.patch, HDFS-9129.012.patch
>
>
> The {{SafeMode}} needs to track whether there are enough blocks so that the 
> NN can get out of the safemode. These fields can moved to the 
> {{BlockManager}} class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9311) Support optional offload of NameNode HA service health checks to a separate RPC server.

[Chris Nauroth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth updated HDFS-9311:

Issue Type: Improvement  (was: Bug)

> Support optional offload of NameNode HA service health checks to a separate 
> RPC server.
> ---
>
> Key: HDFS-9311
> URL: https://issues.apache.org/jira/browse/HDFS-9311
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: ha, namenode
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-9311.001.patch, HDFS-9311.002.patch, 
> HDFS-9311.003.patch
>
>
> When a NameNode is overwhelmed with load, it can lead to resource exhaustion 
> of the RPC handler pools (both client-facing and service-facing).  
> Eventually, this blocks the health check RPC issued from ZKFC, which triggers 
> a failover.  Depending on fencing configuration, the former active NameNode 
> may be killed.  In an overloaded situation, the new active NameNode is likely 
> to suffer the same fate, because client load patterns don't change after the 
> failover.  This can degenerate into flapping between the 2 NameNodes without 
> real recovery.  If a NameNode had been killed by fencing, then it would have 
> to transition through safe mode, further delaying time to recovery.
> This issue proposes a separate, optional RPC server at the NameNode for 
> isolating the HA health checks.  These health checks are lightweight 
> operations that do not suffer from contention issues on the namesystem lock 
> or other shared resources.  Isolating the RPC handlers is sufficient to avoid 
> this situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9311) Support optional offload of NameNode HA service health checks to a separate RPC server.

[Chris Nauroth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth updated HDFS-9311:

   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
   Status: Resolved  (was: Patch Available)

Jitendra, thank you for the code review.  I cannot repro the test failures from 
the last Jenkins run.  The whitespace check flagged on lines that are unchanged 
in my patch, but I took its advice anyway and used {{git apply 
--whitespace=fix}}.

I have committed this to trunk and branch-2.

> Support optional offload of NameNode HA service health checks to a separate 
> RPC server.
> ---
>
> Key: HDFS-9311
> URL: https://issues.apache.org/jira/browse/HDFS-9311
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: ha, namenode
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Fix For: 2.8.0
>
> Attachments: HDFS-9311.001.patch, HDFS-9311.002.patch, 
> HDFS-9311.003.patch
>
>
> When a NameNode is overwhelmed with load, it can lead to resource exhaustion 
> of the RPC handler pools (both client-facing and service-facing).  
> Eventually, this blocks the health check RPC issued from ZKFC, which triggers 
> a failover.  Depending on fencing configuration, the former active NameNode 
> may be killed.  In an overloaded situation, the new active NameNode is likely 
> to suffer the same fate, because client load patterns don't change after the 
> failover.  This can degenerate into flapping between the 2 NameNodes without 
> real recovery.  If a NameNode had been killed by fencing, then it would have 
> to transition through safe mode, further delaying time to recovery.
> This issue proposes a separate, optional RPC server at the NameNode for 
> isolating the HA health checks.  These health checks are lightweight 
> operations that do not suffer from contention issues on the namesystem lock 
> or other shared resources.  Isolating the RPC handlers is sufficient to avoid 
> this situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9168) Move client side unit test to hadoop-hdfs-client

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977836#comment-14977836
 ] 

Mingliang Liu commented on HDFS-9168:
-

Thanks for working on this.

+1 (non-binding)

> Move client side unit test to hadoop-hdfs-client
> 
>
> Key: HDFS-9168
> URL: https://issues.apache.org/jira/browse/HDFS-9168
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: build
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-9168.000.patch, HDFS-9168.001.patch, 
> HDFS-9168.002.patch, HDFS-9168.003.patch, HDFS-9168.004.patch
>
>
> We need to identify and move the unit tests on the client of hdfs to the 
> hdfs-client module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9323) Randomize the DFSStripedOutputStreamWithFailure tests

[Tsz Wo Nicholas Sze (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo Nicholas Sze updated HDFS-9323:
--
Status: Patch Available  (was: Open)

The patch also decrease the total running time from ~9 minutes to 7 minutes in 
my machine.

> Randomize the DFSStripedOutputStreamWithFailure tests
> -
>
> Key: HDFS-9323
> URL: https://issues.apache.org/jira/browse/HDFS-9323
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: test
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>Priority: Minor
> Attachments: h9323_20151028.patch
>
>
> Currently, the DFSStripedOutputStreamWithFailure tests are run with fixed 
> indices #0 - #19 and #59, and also a test with random index 
> (testDatanodeFailureRandomLength).  We should randomize all the tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9231) fsck doesn't list correct file path when Bad Replicas/Blocks are in a snapshot

[Yongjun Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yongjun Zhang updated HDFS-9231:

   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
   Status: Resolved  (was: Patch Available)

Committed to trunk and branch-2. Thanks Xiao for the contribution.


> fsck doesn't list correct file path when Bad Replicas/Blocks are in a snapshot
> --
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 2.8.0
>
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch, 
> HDFS-9231.003.patch, HDFS-9231.004.patch, HDFS-9231.005.patch, 
> HDFS-9231.006.patch, HDFS-9231.007.patch, HDFS-9231.008.patch, 
> HDFS-9231.009.patch
>
>
> Currently for snapshot files, {{fsck -list-corruptfileblocks}} shows corrupt 
> blocks with the original file dir instead of the snapshot dir, and {{fsck 
> -list-corruptfileblocks -includeSnapshots}} behave the same.
> This can be confusing because even when the original file is deleted, fsck 
> will still show that deleted file as corrupted, although what's actually 
> corrupted is the snapshot. 
> As a side note, {{fsck -files -includeSnapshots}} shows the snapshot dirs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9302) WebHDFS throws NullPointerException if newLength is not provided

[Yi Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Liu updated HDFS-9302:
-
Target Version/s: 2.8.0  (was: 2.8.0, 2.7.2)

> WebHDFS throws NullPointerException if newLength is not provided
> 
>
> Key: HDFS-9302
> URL: https://issues.apache.org/jira/browse/HDFS-9302
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: HDFS
>Affects Versions: 2.7.1
> Environment: Centos6
>Reporter: Karthik Palaniappan
>Assignee: Jagadesh Kiran N
>Priority: Minor
> Attachments: HDFS-9302_00.patch, HDFS-9302_01.patch
>
>
> $ curl -X POST "http://namenode:50070/webhdfs/v1/foo?op=truncate;
> {"RemoteException":{"exception":"NullPointerException","javaClassName":"java.lang.NullPointerException","message":null}}
> We should change newLength to be a required parameter in the webhdfs 
> documentation 
> (https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#New_Length),
>  and throw an IllegalArgumentException if isn't provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9302) WebHDFS throws NullPointerException if newLength is not provided

[Yi Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Liu updated HDFS-9302:
-
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
   Status: Resolved  (was: Patch Available)

Committed to trunk and branch-2.

> WebHDFS throws NullPointerException if newLength is not provided
> 
>
> Key: HDFS-9302
> URL: https://issues.apache.org/jira/browse/HDFS-9302
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: HDFS
>Affects Versions: 2.7.1
> Environment: Centos6
>Reporter: Karthik Palaniappan
>Assignee: Jagadesh Kiran N
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9302_00.patch, HDFS-9302_01.patch
>
>
> $ curl -X POST "http://namenode:50070/webhdfs/v1/foo?op=truncate;
> {"RemoteException":{"exception":"NullPointerException","javaClassName":"java.lang.NullPointerException","message":null}}
> We should change newLength to be a required parameter in the webhdfs 
> documentation 
> (https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#New_Length),
>  and throw an IllegalArgumentException if isn't provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group

[Vinayakumar B (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977941#comment-14977941
 ] 

Vinayakumar B commented on HDFS-9044:
-

Patch looks good.
checkstyle comments can be cleared by formatting  below line 
{code}super.chooseFavouredNodes(src, numOfReplicas, favoredNodes,
  favoriteAndExcludedNodes, blocksize,
  maxNodesPerRack, results, avoidStaleNodes, storageTypes);{code}

Whitespaces line endigs also could be removed.

+1 once addresed.

> Give Priority to FavouredNodes , before selecting nodes from FavouredNode's 
> Node Group
> --
>
> Key: HDFS-9044
> URL: https://issues.apache.org/jira/browse/HDFS-9044
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9044.1.patch, HDFS-9044.2.patch, HDFS-9044.3.patch, 
> HDFS-9044.4.patch, HDFS-9044.5.patch
>
>
> Passing Favored nodes intention is to place replica among the favored node
> Current behavior in Node group is 
>   If favored node is not available it goes to one among favored 
> nodegroup. 
> {noformat}
> Say for example:
>   1)I need 3 replicas and passed 5 favored nodes.
>   2)Out of 5 favored nodes 3 favored nodes are not good.
>   3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node 
> returned , 3 will be random node from 3 bad FavoredNode's nodegroup. 
>   4)Then there is a probability that all my 3 replicas are placed on 
> Random node from FavoredNodes's nodegroup , instead of giving priority to 2 
> favored nodes returned as target.
> {noformat}
> *Instead of returning 5 targets on 3rd step above , we can return 2 good 
> favored nodes as target*
> *And remaining 1 needed replica can be chosen from Random node of bad 
> FavoredNodes's nodegroup.*
> This will make sure that the FavoredNodes are given priority.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9231) fsck doesn't list correct file path when Bad Replicas/Blocks are in a snapshot

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977949#comment-14977949
 ] 

Hudson commented on HDFS-9231:
--

FAILURE: Integrated in Hadoop-trunk-Commit #8717 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/8717/])
HDFS-9231. fsck doesn't list correct file path when Bad Replicas/Blocks 
(yzhang: rev 97913f430cbe3f82ac866ae6ab8f42754102f6c0)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSnapshotOp.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java


> fsck doesn't list correct file path when Bad Replicas/Blocks are in a snapshot
> --
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 2.8.0
>
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch, 
> HDFS-9231.003.patch, HDFS-9231.004.patch, HDFS-9231.005.patch, 
> HDFS-9231.006.patch, HDFS-9231.007.patch, HDFS-9231.008.patch, 
> HDFS-9231.009.patch
>
>
> Currently for snapshot files, {{fsck -list-corruptfileblocks}} shows corrupt 
> blocks with the original file dir instead of the snapshot dir, and {{fsck 
> -list-corruptfileblocks -includeSnapshots}} behave the same.
> This can be confusing because even when the original file is deleted, fsck 
> will still show that deleted file as corrupted, although what's actually 
> corrupted is the snapshot. 
> As a side note, {{fsck -files -includeSnapshots}} shows the snapshot dirs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9302) WebHDFS throws NullPointerException if newLength is not provided

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977945#comment-14977945
 ] 

Hudson commented on HDFS-9302:
--

FAILURE: Integrated in Hadoop-trunk-Commit #8717 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/8717/])
HDFS-9302. WebHDFS throws NullPointerException if newLength is not (yliu: rev 
6ff6663f64476eab5612ae9eb409104f44c6e6c7)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java


> WebHDFS throws NullPointerException if newLength is not provided
> 
>
> Key: HDFS-9302
> URL: https://issues.apache.org/jira/browse/HDFS-9302
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: HDFS
>Affects Versions: 2.7.1
> Environment: Centos6
>Reporter: Karthik Palaniappan
>Assignee: Jagadesh Kiran N
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9302_00.patch, HDFS-9302_01.patch
>
>
> $ curl -X POST "http://namenode:50070/webhdfs/v1/foo?op=truncate;
> {"RemoteException":{"exception":"NullPointerException","javaClassName":"java.lang.NullPointerException","message":null}}
> We should change newLength to be a required parameter in the webhdfs 
> documentation 
> (https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#New_Length),
>  and throw an IllegalArgumentException if isn't provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group

[Vinayakumar B (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978094#comment-14978094
 ] 

Vinayakumar B commented on HDFS-9044:
-

+1, pending jenkins

> Give Priority to FavouredNodes , before selecting nodes from FavouredNode's 
> Node Group
> --
>
> Key: HDFS-9044
> URL: https://issues.apache.org/jira/browse/HDFS-9044
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9044.1.patch, HDFS-9044.2.patch, HDFS-9044.3.patch, 
> HDFS-9044.4.patch, HDFS-9044.5.patch, HDFS-9044.6.patch
>
>
> Passing Favored nodes intention is to place replica among the favored node
> Current behavior in Node group is 
>   If favored node is not available it goes to one among favored 
> nodegroup. 
> {noformat}
> Say for example:
>   1)I need 3 replicas and passed 5 favored nodes.
>   2)Out of 5 favored nodes 3 favored nodes are not good.
>   3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node 
> returned , 3 will be random node from 3 bad FavoredNode's nodegroup. 
>   4)Then there is a probability that all my 3 replicas are placed on 
> Random node from FavoredNodes's nodegroup , instead of giving priority to 2 
> favored nodes returned as target.
> {noformat}
> *Instead of returning 5 targets on 3rd step above , we can return 2 good 
> favored nodes as target*
> *And remaining 1 needed replica can be chosen from Random node of bad 
> FavoredNodes's nodegroup.*
> This will make sure that the FavoredNodes are given priority.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8545) Refactor FS#getUsed() to use ContentSummary and add an API to fetch the total file length from a specific path

[Vinayakumar B (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978102#comment-14978102
 ] 

Vinayakumar B commented on HDFS-8545:
-

+1 lgtm,

Pending jenkins

> Refactor FS#getUsed() to use ContentSummary and add an API to fetch the total 
> file length from a specific path
> --
>
> Key: HDFS-8545
> URL: https://issues.apache.org/jira/browse/HDFS-8545
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Minor
> Attachments: HDFS-8545.1.patch, HDFS-8545.2.patch, HDFS-8545.3.patch, 
> HDFS-8545.4.patch, HDFS-8545.5.patch, HDFS-8545.6.patch
>
>
> Currently by default in FileSystem#getUsed() returns the total file size from 
> root. 
> It is good to have an api to return the total file size from specified path 
> ,same as we specify the path in "./hdfs dfs -du -s /path" .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9324) DFSClient got deadlock when close file and failed to renew lease

[Vinayakumar B (JIRA)]

Vinayakumar B created HDFS-9324:
---

 Summary: DFSClient got deadlock when close file and failed to 
renew lease
 Key: HDFS-9324
 URL: https://issues.apache.org/jira/browse/HDFS-9324
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs-client
Reporter: Vinayakumar B
Assignee: Vinayakumar B


this issue is from one of the user mailing list query from daniedeng(邓飞)

there is possibility of dead lock between lease renewer and DFSOutputstream 

when file is getting closed and lease renewal also failed.

stack trace will be attached.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9057) allow/disallow snapshots via webhdfs

[Brahma Reddy Battula (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brahma Reddy Battula updated HDFS-9057:
---
Attachment: HDFS-9057-003.patch

> allow/disallow snapshots via webhdfs
> 
>
> Key: HDFS-9057
> URL: https://issues.apache.org/jira/browse/HDFS-9057
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: webhdfs
>Affects Versions: 3.0.0
>Reporter: Allen Wittenauer
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9057-002.patch, HDFS-9057-003.patch, HDFS-9057.patch
>
>
> We should be able to allow and disallow directories for snapshotting via 
> WebHDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9057) allow/disallow snapshots via webhdfs

[Brahma Reddy Battula (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978128#comment-14978128
 ] 

Brahma Reddy Battula commented on HDFS-9057:


[~vinayrpet] uploaded patch based on your review comments.. Kindly Review.

> allow/disallow snapshots via webhdfs
> 
>
> Key: HDFS-9057
> URL: https://issues.apache.org/jira/browse/HDFS-9057
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: webhdfs
>Affects Versions: 3.0.0
>Reporter: Allen Wittenauer
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9057-002.patch, HDFS-9057-003.patch, HDFS-9057.patch
>
>
> We should be able to allow and disallow directories for snapshotting via 
> WebHDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9324) DFSClient got deadlock when close file and failed to renew lease

[Vinayakumar B (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B updated HDFS-9324:

Attachment: deadlock.log

Attached stackrace

> DFSClient got deadlock when close file and failed to renew lease
> 
>
> Key: HDFS-9324
> URL: https://issues.apache.org/jira/browse/HDFS-9324
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs-client
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: deadlock.log
>
>
> this issue is from one of the user mailing list query from daniedeng(邓飞)
> there is possibility of dead lock between lease renewer and DFSOutputstream 
> when file is getting closed and lease renewal also failed.
> stack trace will be attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-9324) DFSClient got deadlock when close file and failed to renew lease

[Vinayakumar B (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HDFS-9324.
-
Resolution: Duplicate

duplicate of HDFS-9294

> DFSClient got deadlock when close file and failed to renew lease
> 
>
> Key: HDFS-9324
> URL: https://issues.apache.org/jira/browse/HDFS-9324
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs-client
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: deadlock.log
>
>
> this issue is from one of the user mailing list query from daniedeng(邓飞)
> there is possibility of dead lock between lease renewer and DFSOutputstream 
> when file is getting closed and lease renewal also failed.
> stack trace will be attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9324) DFSClient got deadlock when close file and failed to renew lease

[Brahma Reddy Battula (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978134#comment-14978134
 ] 

Brahma Reddy Battula commented on HDFS-9324:


Dupe of HDFS-9294..?

> DFSClient got deadlock when close file and failed to renew lease
> 
>
> Key: HDFS-9324
> URL: https://issues.apache.org/jira/browse/HDFS-9324
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs-client
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: deadlock.log
>
>
> this issue is from one of the user mailing list query from daniedeng(邓飞)
> there is possibility of dead lock between lease renewer and DFSOutputstream 
> when file is getting closed and lease renewal also failed.
> stack trace will be attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9311) Support optional offload of NameNode HA service health checks to a separate RPC server.

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978143#comment-14978143
 ] 

Hudson commented on HDFS-9311:
--

FAILURE: Integrated in Hadoop-Yarn-trunk #1329 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1329/])
HDFS-9311. Support optional offload of NameNode HA service health checks 
(cnauroth: rev bf8e45298218f70e38838152f69c7705d8606bd6)
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthMonitor.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestNNHealthCheck.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRespectsBindHostKeys.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/NNHAServiceTarget.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitorWithDedicatedHealthAddress.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestHealthMonitor.java


> Support optional offload of NameNode HA service health checks to a separate 
> RPC server.
> ---
>
> Key: HDFS-9311
> URL: https://issues.apache.org/jira/browse/HDFS-9311
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: ha, namenode
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Fix For: 2.8.0
>
> Attachments: HDFS-9311.001.patch, HDFS-9311.002.patch, 
> HDFS-9311.003.patch
>
>
> When a NameNode is overwhelmed with load, it can lead to resource exhaustion 
> of the RPC handler pools (both client-facing and service-facing).  
> Eventually, this blocks the health check RPC issued from ZKFC, which triggers 
> a failover.  Depending on fencing configuration, the former active NameNode 
> may be killed.  In an overloaded situation, the new active NameNode is likely 
> to suffer the same fate, because client load patterns don't change after the 
> failover.  This can degenerate into flapping between the 2 NameNodes without 
> real recovery.  If a NameNode had been killed by fencing, then it would have 
> to transition through safe mode, further delaying time to recovery.
> This issue proposes a separate, optional RPC server at the NameNode for 
> isolating the HA health checks.  These health checks are lightweight 
> operations that do not suffer from contention issues on the namesystem lock 
> or other shared resources.  Isolating the RPC handlers is sufficient to avoid 
> this situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9302) WebHDFS throws NullPointerException if newLength is not provided

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978139#comment-14978139
 ] 

Hudson commented on HDFS-9302:
--

FAILURE: Integrated in Hadoop-Yarn-trunk #1329 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1329/])
HDFS-9302. WebHDFS throws NullPointerException if newLength is not (yliu: rev 
6ff6663f64476eab5612ae9eb409104f44c6e6c7)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java


> WebHDFS throws NullPointerException if newLength is not provided
> 
>
> Key: HDFS-9302
> URL: https://issues.apache.org/jira/browse/HDFS-9302
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: HDFS
>Affects Versions: 2.7.1
> Environment: Centos6
>Reporter: Karthik Palaniappan
>Assignee: Jagadesh Kiran N
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9302_00.patch, HDFS-9302_01.patch
>
>
> $ curl -X POST "http://namenode:50070/webhdfs/v1/foo?op=truncate;
> {"RemoteException":{"exception":"NullPointerException","javaClassName":"java.lang.NullPointerException","message":null}}
> We should change newLength to be a required parameter in the webhdfs 
> documentation 
> (https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#New_Length),
>  and throw an IllegalArgumentException if isn't provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9231) fsck doesn't list correct file path when Bad Replicas/Blocks are in a snapshot

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978144#comment-14978144
 ] 

Hudson commented on HDFS-9231:
--

FAILURE: Integrated in Hadoop-Yarn-trunk #1329 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1329/])
HDFS-9231. fsck doesn't list correct file path when Bad Replicas/Blocks 
(yzhang: rev 97913f430cbe3f82ac866ae6ab8f42754102f6c0)
* hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeMXBean.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSnapshotOp.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> fsck doesn't list correct file path when Bad Replicas/Blocks are in a snapshot
> --
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 2.8.0
>
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch, 
> HDFS-9231.003.patch, HDFS-9231.004.patch, HDFS-9231.005.patch, 
> HDFS-9231.006.patch, HDFS-9231.007.patch, HDFS-9231.008.patch, 
> HDFS-9231.009.patch
>
>
> Currently for snapshot files, {{fsck -list-corruptfileblocks}} shows corrupt 
> blocks with the original file dir instead of the snapshot dir, and {{fsck 
> -list-corruptfileblocks -includeSnapshots}} behave the same.
> This can be confusing because even when the original file is deleted, fsck 
> will still show that deleted file as corrupted, although what's actually 
> corrupted is the snapshot. 
> As a side note, {{fsck -files -includeSnapshots}} shows the snapshot dirs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9317) Document fsck -blockId and -storagepolicy options in branch-2.7

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978140#comment-14978140
 ] 

Hudson commented on HDFS-9317:
--

FAILURE: Integrated in Hadoop-Yarn-trunk #1329 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1329/])
Add HDFS-9317 to CHANGES.txt. (aajisaka: rev 
eca51b13af1951b18ab7a5a92aaa0b9fe07d9ef8)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> Document fsck -blockId and -storagepolicy options in branch-2.7
> ---
>
> Key: HDFS-9317
> URL: https://issues.apache.org/jira/browse/HDFS-9317
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.7.1
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Fix For: 2.7.2
>
> Attachments: HDFS-9317-branch-2.7.00.patch
>
>
> fsck -blockId and -storagepolicy options are implemented by HDFS-6663 and 
> HDFS-7467 but the options are not documented in 2.7.1 release.
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#fsck



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9327) WebHDFS throws html trace if Source is not provided

[Jagadesh Kiran N (JIRA)]

Jagadesh Kiran N created HDFS-9327:
--

 Summary: WebHDFS throws html trace if Source is not provided
 Key: HDFS-9327
 URL: https://issues.apache.org/jira/browse/HDFS-9327
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Jagadesh Kiran N
Assignee: Jagadesh Kiran N


Execute the following command in webhdfs with out entering Sources
{code}
curl -i -X POST "http://10.19.92.127:50070/webhdfs/v1/kiran/1?op=CONCAT;
{code}
the html trace will be disclosed with jetty version 
{code}
HTTP/1.1 500 Absolute path required
Cache-Control: must-revalidate,no-cache,no-store
Date: Wed, 28 Oct 2015 16:10:44 GMT
Pragma: no-cache
Date: Wed, 28 Oct 2015 16:10:44 GMT
Pragma: no-cache
Content-Type: text/html; charset=iso-8859-1
Content-Length: 7082
Server: Jetty(6.1.26)




Error 500 Absolute path required

HTTP ERROR 500
Problem accessing /webhdfs/v1/kiran/1. Reason:
Absolute path requiredCaused 
by:java.lang.AssertionError: Absolute path required
at 
org.apache.hadoop.hdfs.server.namenode.INode.getPathNames(INode.java:745)
at 
org.apache.hadoop.hdfs.server.namenode.INode.getPathComponents(INode.java:724)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath4Write(FSDirectory.java:1456)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINodesInPath4Write(FSDirectory.java:1417)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirConcatOp.verifySrcFiles(FSDirConcatOp.java:112)
at 
org.apache.hadoop.hdfs.server.namenode.FSDirConcatOp.concat(FSDirConcatOp.java:71)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.concat(FSNamesystem.java:1790)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.concat(NameNodeRpcServer.java:880)
at 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.post(NamenodeWebHdfsMethods.java:705)
at 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.access$200(NamenodeWebHdfsMethods.java:101)
at 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$2.run(NamenodeWebHdfsMethods.java:670)
at 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$2.run(NamenodeWebHdfsMethods.java:666)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1667)
at 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.post(NamenodeWebHdfsMethods.java:666)
at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at 
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at 
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at 
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at 
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at 
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at 
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at 
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at 
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at 
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at 

[jira] [Commented] (HDFS-9231) fsck doesn't list correct file path when Bad Replicas/Blocks are in a snapshot

[Xiao Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978717#comment-14978717
 ] 

Xiao Chen commented on HDFS-9231:
-

Thanks Yongjun for the reviews and commit!

> fsck doesn't list correct file path when Bad Replicas/Blocks are in a snapshot
> --
>
> Key: HDFS-9231
> URL: https://issues.apache.org/jira/browse/HDFS-9231
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: snapshots
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 2.8.0
>
> Attachments: HDFS-9231.001.patch, HDFS-9231.002.patch, 
> HDFS-9231.003.patch, HDFS-9231.004.patch, HDFS-9231.005.patch, 
> HDFS-9231.006.patch, HDFS-9231.007.patch, HDFS-9231.008.patch, 
> HDFS-9231.009.patch
>
>
> Currently for snapshot files, {{fsck -list-corruptfileblocks}} shows corrupt 
> blocks with the original file dir instead of the snapshot dir, and {{fsck 
> -list-corruptfileblocks -includeSnapshots}} behave the same.
> This can be confusing because even when the original file is deleted, fsck 
> will still show that deleted file as corrupted, although what's actually 
> corrupted is the snapshot. 
> As a side note, {{fsck -files -includeSnapshots}} shows the snapshot dirs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity

[Kihwal Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee updated HDFS-9279:
-
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
   3.0.0
   Status: Resolved  (was: Patch Available)

> Decomissioned capacity should not be considered for configured/used capacity
> 
>
> Key: HDFS-9279
> URL: https://issues.apache.org/jira/browse/HDFS-9279
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Fix For: 3.0.0, 2.8.0
>
> Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, 
> HDFS-9279-v3.patch, HDFS-9279-v4.patch
>
>
> Capacity of a decommissioned node is being accounted as configured and used 
> capacity metrics. This gives incorrect perception of cluster usage.
> Once a node is decommissioned, its capacity should be considered similar to a 
> dead node.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9255) Consolidate block recovery related implementation into a single class

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978676#comment-14978676
 ] 

Hudson commented on HDFS-9255:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #8719 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/8719/])
HDFS-9255. Consolidate block recovery related implementation into a (zhz: rev 
e287e7d14b838a866ba03d895fa3581d7c09)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java


> Consolidate block recovery related implementation into a single class
> -
>
> Key: HDFS-9255
> URL: https://issues.apache.org/jira/browse/HDFS-9255
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Reporter: Walter Su
>Assignee: Walter Su
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9255-branch-2.06.patch, HDFS-9255.01.patch, 
> HDFS-9255.02.patch, HDFS-9255.03.patch, HDFS-9255.04.patch, 
> HDFS-9255.05.patch, HDFS-9255.06.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity

[Kihwal Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978664#comment-14978664
 ] 

Kihwal Lee commented on HDFS-9279:
--

The jira was originally to exclude "live" decommissioned nodes from stat, but 
it seems decommissioning nodes are excluded as well.

> Decomissioned capacity should not be considered for configured/used capacity
> 
>
> Key: HDFS-9279
> URL: https://issues.apache.org/jira/browse/HDFS-9279
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, 
> HDFS-9279-v3.patch, HDFS-9279-v4.patch
>
>
> Capacity of a decommissioned node is being accounted as configured and used 
> capacity metrics. This gives incorrect perception of cluster usage.
> Once a node is decommissioned, its capacity should be considered similar to a 
> dead node.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity

[Kihwal Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978744#comment-14978744
 ] 

Kihwal Lee commented on HDFS-9279:
--

+1 the patch looks good.

> Decomissioned capacity should not be considered for configured/used capacity
> 
>
> Key: HDFS-9279
> URL: https://issues.apache.org/jira/browse/HDFS-9279
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, 
> HDFS-9279-v3.patch, HDFS-9279-v4.patch
>
>
> Capacity of a decommissioned node is being accounted as configured and used 
> capacity metrics. This gives incorrect perception of cluster usage.
> Once a node is decommissioned, its capacity should be considered similar to a 
> dead node.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9289) check genStamp when complete file

[Chang Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978617#comment-14978617
 ] 

Chang Li commented on HDFS-9289:


[~zhz], no we don't have this log because we didn't enable the 
blockStateChangeLog. How do you propose we should proceed with this jira?

> check genStamp when complete file
> -
>
> Key: HDFS-9289
> URL: https://issues.apache.org/jira/browse/HDFS-9289
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Chang Li
>Assignee: Chang Li
>Priority: Critical
> Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch
>
>
> we have seen a case of corrupt block which is caused by file complete after a 
> pipelineUpdate, but the file complete with the old block genStamp. This 
> caused the replicas of two datanodes in updated pipeline to be viewed as 
> corrupte. Propose to check genstamp when commit block



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity

[Kihwal Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978708#comment-14978708
 ] 

Kihwal Lee commented on HDFS-9279:
--

Actually, I think this makes more sense. Assuming roughly even distribution of 
data, excluding everything from dicomm'ing nodes makes the used/total unchanged 
initially. As blocks are replicated used/total will increase. I think this 
tracks the reality better.

> Decomissioned capacity should not be considered for configured/used capacity
> 
>
> Key: HDFS-9279
> URL: https://issues.apache.org/jira/browse/HDFS-9279
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, 
> HDFS-9279-v3.patch, HDFS-9279-v4.patch
>
>
> Capacity of a decommissioned node is being accounted as configured and used 
> capacity metrics. This gives incorrect perception of cluster usage.
> Once a node is decommissioned, its capacity should be considered similar to a 
> dead node.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity

[Kihwal Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978758#comment-14978758
 ] 

Kihwal Lee commented on HDFS-9279:
--

I've committed this to trunk and branch-2. Thank you for working on the issue, 
Kuhu.

> Decomissioned capacity should not be considered for configured/used capacity
> 
>
> Key: HDFS-9279
> URL: https://issues.apache.org/jira/browse/HDFS-9279
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Fix For: 3.0.0, 2.8.0
>
> Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, 
> HDFS-9279-v3.patch, HDFS-9279-v4.patch
>
>
> Capacity of a decommissioned node is being accounted as configured and used 
> capacity metrics. This gives incorrect perception of cluster usage.
> Once a node is decommissioned, its capacity should be considered similar to a 
> dead node.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8766) Implement a libhdfs(3) compatible API

[James Clampffer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979012#comment-14979012
 ] 

James Clampffer commented on HDFS-8766:
---

"Please wrap the statement with braces though it is a single line. There are 
several places like the above code needs to be fixed."

It looks like this is something the google coding style explicitly allows: 
"Short conditional statements may be written on one line if this enhances 
readability. You may use this only when the line is brief and the statement 
does not use the else clause.".  If we are going to be very strict about those 
rules everywhere else I'd like to keep them the same here.  I don't feel 
strongly about it either way but any exceptions or additional restrictions need 
to be easily available for everyone to see. Even without any extra stuff we at 
least need a README that says "all code must follow google's standards".  
Currently the only way people can find out that this project sticks to the 
google style guide is by reading all of the comments and even they will only 
catch the additional rules that have been pointed out.  This will save everyone 
a significant amount of time because we'll have fewer code reviews fail due to 
style issues.

If you have time today or tomorrow would you mind dropping a README into 
/libhdfspp file with your coding standards?  I'd be happy to add one but I 
don't know what else I'd be missing.  I'll file a jira for adding a file with 
coding standards to the source tree.

Let me know what you prefer with the simple if..cond..something and I'll 
conform to that.  I'll get the debug helpers out of the header.

> Implement a libhdfs(3) compatible API
> -
>
> Key: HDFS-8766
> URL: https://issues.apache.org/jira/browse/HDFS-8766
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
> Attachments: HDFS-8766.HDFS-8707.000.patch, 
> HDFS-8766.HDFS-8707.001.patch, HDFS-8766.HDFS-8707.002.patch, 
> HDFS-8766.HDFS-8707.003.patch, HDFS-8766.HDFS-8707.004.patch, 
> HDFS-8766.HDFS-8707.005.patch, HDFS-8766.HDFS-8707.006.patch, 
> HDFS-8766.HDFS-8707.007.patch, HDFS-8766.HDFS-8707.008.patch, 
> HDFS-8766.HDFS-8707.009.patch, HDFS-8766.HDFS-8707.010.patch, 
> HDFS-8766.HDFS-8707.011.patch
>
>
> Add a synchronous API that is compatible with the hdfs.h header used in 
> libhdfs and libhdfs3.  This will make it possible for projects using 
> libhdfs/libhdfs3 to relink against libhdfspp with minimal changes.
> This also provides a pure C interface that can be linked against projects 
> that aren't built in C++11 mode for various reasons but use the same 
> compiler.  It also allows many other programming languages to access 
> libhdfspp through builtin FFI interfaces.
> The libhdfs API is very similar to the posix file API which makes it easier 
> for programs built using posix filesystem calls to be modified to access HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9117) Config file reader / options classes for libhdfs++

[Haohui Mai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978780#comment-14978780
 ] 

Haohui Mai commented on HDFS-9117:
--

bq. I disagree; the code is required for a pure C++ implementation that 
satisfies the use case of "Read the local config and connect to HDFS." The 
libhdfs compatability layer should be for the glue for the C API.

The principle is that the C++ APIs should be able to solely rely on the 
{{Options}} class for all configurations. The class is strongly-typed and has 
clear schema. From the dependency point of view there are strong desires and 
use cases that running libhdfspp without depending XML configuration. I'll be 
-1 if the C++ APIs are bundled with XML configurations.

bq. It has been moved over with the other substitutions. It's implemented, 
tested, and part of the spec of the Java config files, so I don't see any 
reason to excise it.

Environment variables and substitutions are clearly platform-dependent. There 
are use cases that want a platform-independent implementation for minimal 
dependency. The implementation needs to address the use case.

> Config file reader / options classes for libhdfs++
> --
>
> Key: HDFS-9117
> URL: https://issues.apache.org/jira/browse/HDFS-9117
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Affects Versions: HDFS-8707
>Reporter: Bob Hansen
>Assignee: Bob Hansen
> Attachments: HDFS-9117.HDFS-8707.001.patch, 
> HDFS-9117.HDFS-8707.002.patch, HDFS-9117.HDFS-8707.003.patch, 
> HDFS-9117.HDFS-8707.004.patch, HDFS-9117.HDFS-8707.005.patch, 
> HDFS-9117.HDFS-8707.006.patch, HDFS-9117.HDFS-8707.008.patch, 
> HDFS-9117.HDFS-8707.009.patch, HDFS-9117.HDFS-9288.007.patch
>
>
> For environmental compatability with HDFS installations, libhdfs++ should be 
> able to read the configurations from Hadoop XML files and behave in line with 
> the Java implementation.
> Most notably, machine names and ports should be readable from Hadoop XML 
> configuration files.
> Similarly, an internal Options architecture for libhdfs++ should be developed 
> to efficiently transport the configuration information within the system.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8766) Implement a libhdfs(3) compatible API

[Haohui Mai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978816#comment-14978816
 ] 

Haohui Mai commented on HDFS-8766:
--

Looks good to me overall. A few nitpicks:

{code}
+  if (!s.ok()) return -1;
{code}

Please wrap the statement with braces though it is a single line. There are 
several places like the above code needs to be fixed.

{code}

+/** error reporting helpers **/
+extern thread_local std::string errstr;
+void ReportError(int errnum, std::string msg);
+
+/**
+ *client supplies buffer and length of buffer
+ *fill it and add trailing null
+ **/
+void getErrorStr(char *buf, size_t len);
{code}

Looks like they can be removed from the header file.

> Implement a libhdfs(3) compatible API
> -
>
> Key: HDFS-8766
> URL: https://issues.apache.org/jira/browse/HDFS-8766
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
> Attachments: HDFS-8766.HDFS-8707.000.patch, 
> HDFS-8766.HDFS-8707.001.patch, HDFS-8766.HDFS-8707.002.patch, 
> HDFS-8766.HDFS-8707.003.patch, HDFS-8766.HDFS-8707.004.patch, 
> HDFS-8766.HDFS-8707.005.patch, HDFS-8766.HDFS-8707.006.patch, 
> HDFS-8766.HDFS-8707.007.patch, HDFS-8766.HDFS-8707.008.patch, 
> HDFS-8766.HDFS-8707.009.patch, HDFS-8766.HDFS-8707.010.patch, 
> HDFS-8766.HDFS-8707.011.patch
>
>
> Add a synchronous API that is compatible with the hdfs.h header used in 
> libhdfs and libhdfs3.  This will make it possible for projects using 
> libhdfs/libhdfs3 to relink against libhdfspp with minimal changes.
> This also provides a pure C interface that can be linked against projects 
> that aren't built in C++11 mode for various reasons but use the same 
> compiler.  It also allows many other programming languages to access 
> libhdfspp through builtin FFI interfaces.
> The libhdfs API is very similar to the posix file API which makes it easier 
> for programs built using posix filesystem calls to be modified to access HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9255) Consolidate block recovery related implementation into a single class

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978849#comment-14978849
 ] 

Hudson commented on HDFS-9255:
--

FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #608 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/608/])
HDFS-9255. Consolidate block recovery related implementation into a (zhz: rev 
e287e7d14b838a866ba03d895fa3581d7c09)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java


> Consolidate block recovery related implementation into a single class
> -
>
> Key: HDFS-9255
> URL: https://issues.apache.org/jira/browse/HDFS-9255
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Reporter: Walter Su
>Assignee: Walter Su
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9255-branch-2.06.patch, HDFS-9255.01.patch, 
> HDFS-9255.02.patch, HDFS-9255.03.patch, HDFS-9255.04.patch, 
> HDFS-9255.05.patch, HDFS-9255.06.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-7163) WebHdfsFileSystem should retry reads according to the configured retry policy.

[Eric Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Payne updated HDFS-7163:
-
Status: Open  (was: Patch Available)

> WebHdfsFileSystem should retry reads according to the configured retry policy.
> --
>
> Key: HDFS-7163
> URL: https://issues.apache.org/jira/browse/HDFS-7163
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 2.5.1, 3.0.0
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: WebHDFS Read Retry.pdf
>
>
> In the current implementation of WebHdfsFileSystem, opens are retried 
> according to the configured retry policy, but not reads. Therefore, if a 
> connection goes down while data is being read, the read will fail and the 
> read will have to be retried by the client code.
> Also, after a connection has been established, the next read (or seek/read) 
> will fail and the read will have to be restarted by the client code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-7163) WebHdfsFileSystem should retry reads according to the configured retry policy.

[Eric Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Payne updated HDFS-7163:
-
Attachment: (was: HDFS-7163.001.patch)

> WebHdfsFileSystem should retry reads according to the configured retry policy.
> --
>
> Key: HDFS-7163
> URL: https://issues.apache.org/jira/browse/HDFS-7163
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.0.0, 2.5.1
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: WebHDFS Read Retry.pdf
>
>
> In the current implementation of WebHdfsFileSystem, opens are retried 
> according to the configured retry policy, but not reads. Therefore, if a 
> connection goes down while data is being read, the read will fail and the 
> read will have to be retried by the client code.
> Also, after a connection has been established, the next read (or seek/read) 
> will fail and the read will have to be restarted by the client code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-8766) Implement a libhdfs(3) compatible API

[Haohui Mai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978819#comment-14978819
 ] 

Haohui Mai commented on HDFS-8766:
--

Things that be a
{code}
+#include "libhdfspp/hdfs.h"

+  auto callback = [stat, ](const Status , FileSystem *f) {
+fs = f;
+stat->set_value(s);
+  };
{code}

It might make sense to rename the {{libhdfspp/hdfs.h}} to 
{{libhdfspp/libhdfspp.h}}. It is also beneficial to change the way how the 
{{FileSystem}} works to only make the connection when calling {{Connect()}}. 
They can be addressed in follow-up jiras.

> Implement a libhdfs(3) compatible API
> -
>
> Key: HDFS-8766
> URL: https://issues.apache.org/jira/browse/HDFS-8766
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
> Attachments: HDFS-8766.HDFS-8707.000.patch, 
> HDFS-8766.HDFS-8707.001.patch, HDFS-8766.HDFS-8707.002.patch, 
> HDFS-8766.HDFS-8707.003.patch, HDFS-8766.HDFS-8707.004.patch, 
> HDFS-8766.HDFS-8707.005.patch, HDFS-8766.HDFS-8707.006.patch, 
> HDFS-8766.HDFS-8707.007.patch, HDFS-8766.HDFS-8707.008.patch, 
> HDFS-8766.HDFS-8707.009.patch, HDFS-8766.HDFS-8707.010.patch, 
> HDFS-8766.HDFS-8707.011.patch
>
>
> Add a synchronous API that is compatible with the hdfs.h header used in 
> libhdfs and libhdfs3.  This will make it possible for projects using 
> libhdfs/libhdfs3 to relink against libhdfspp with minimal changes.
> This also provides a pure C interface that can be linked against projects 
> that aren't built in C++11 mode for various reasons but use the same 
> compiler.  It also allows many other programming languages to access 
> libhdfspp through builtin FFI interfaces.
> The libhdfs API is very similar to the posix file API which makes it easier 
> for programs built using posix filesystem calls to be modified to access HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-9087) Add some jitter to DataNode.checkDiskErrorThread

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HDFS-9087:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

This was pushed a while ago.

> Add some jitter to DataNode.checkDiskErrorThread
> 
>
> Key: HDFS-9087
> URL: https://issues.apache.org/jira/browse/HDFS-9087
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.6.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HDFS-9087-v0.patch, HDFS-9087-v1.patch, 
> HDFS-9087-v2.patch, HDFS-9087-v3.patch
>
>
> If all datanodes are started across a cluster at the same time (or errors in 
> the network cause ioexceptions) there can be storms where lots of datanodes 
> check their disks at the exact same time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978926#comment-14978926
 ] 

Hudson commented on HDFS-9279:
--

FAILURE: Integrated in Hadoop-trunk-Commit #8720 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/8720/])
HDFS-9279. Decomissioned capacity should not be considered for (kihwal: rev 
19a77f546657b086af8f41fa631099bdde7e010c)
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


> Decomissioned capacity should not be considered for configured/used capacity
> 
>
> Key: HDFS-9279
> URL: https://issues.apache.org/jira/browse/HDFS-9279
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Fix For: 3.0.0, 2.8.0
>
> Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, 
> HDFS-9279-v3.patch, HDFS-9279-v4.patch
>
>
> Capacity of a decommissioned node is being accounted as configured and used 
> capacity metrics. This gives incorrect perception of cluster usage.
> Once a node is decommissioned, its capacity should be considered similar to a 
> dead node.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9255) Consolidate block recovery related implementation into a single class

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978929#comment-14978929
 ] 

Hudson commented on HDFS-9255:
--

SUCCESS: Integrated in Hadoop-Hdfs-trunk #2484 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2484/])
HDFS-9255. Consolidate block recovery related implementation into a (zhz: rev 
e287e7d14b838a866ba03d895fa3581d7c09)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java


> Consolidate block recovery related implementation into a single class
> -
>
> Key: HDFS-9255
> URL: https://issues.apache.org/jira/browse/HDFS-9255
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Reporter: Walter Su
>Assignee: Walter Su
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9255-branch-2.06.patch, HDFS-9255.01.patch, 
> HDFS-9255.02.patch, HDFS-9255.03.patch, HDFS-9255.04.patch, 
> HDFS-9255.05.patch, HDFS-9255.06.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9255) Consolidate block recovery related implementation into a single class

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978796#comment-14978796
 ] 

Hudson commented on HDFS-9255:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #595 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/595/])
HDFS-9255. Consolidate block recovery related implementation into a (zhz: rev 
e287e7d14b838a866ba03d895fa3581d7c09)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java


> Consolidate block recovery related implementation into a single class
> -
>
> Key: HDFS-9255
> URL: https://issues.apache.org/jira/browse/HDFS-9255
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Reporter: Walter Su
>Assignee: Walter Su
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9255-branch-2.06.patch, HDFS-9255.01.patch, 
> HDFS-9255.02.patch, HDFS-9255.03.patch, HDFS-9255.04.patch, 
> HDFS-9255.05.patch, HDFS-9255.06.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9117) Config file reader / options classes for libhdfs++

[Bob Hansen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978833#comment-14978833
 ] 

Bob Hansen commented on HDFS-9117:
--

The current design has the core of the C++ engine (FileSystem, InputStream, 
etc.) dependent solely on the Options class, as we would all like.   I believe 
that the Configuration class should be part of the larger C++ API, however, 
since it is a requirement for interoperability with deployed installations.

I propose we leave the base Configuration class in lib/common, and move the 
hdfs_configuration class to lib/config.  The HDFSConfiguration class will 
continue to be responsible for mapping keys to hdfs::Option members and 
producing the hdfs::Option object that the FileSystem and friends will consume. 
 Does that seem an appropriate approach?

bq. Environment variables and substitutions are clearly platform-dependent.
getenv is part of the C standard library, and the substitutions are entirely 
the purview of the Java and C++ Configuration implementation, so I don't see 
how they are at all platform-dependent.  We can wrap them in configuration 
#ifdefs to support platforms that for some reason don't support the C standard 
library, but I don't see that as being a great service to our consumer base.

The configuration tests use setenv, which is available on all unix-y platforms. 
 While I hope this library will eventually support Windows clients, we don't 
currently build or test on that platform, and I expect the setenv/_putenv 
refactoring will be the smallest part of the work.  Again, to guard against 
that, we can #ifdef out those tests based on the available capabilities.



> Config file reader / options classes for libhdfs++
> --
>
> Key: HDFS-9117
> URL: https://issues.apache.org/jira/browse/HDFS-9117
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Affects Versions: HDFS-8707
>Reporter: Bob Hansen
>Assignee: Bob Hansen
> Attachments: HDFS-9117.HDFS-8707.001.patch, 
> HDFS-9117.HDFS-8707.002.patch, HDFS-9117.HDFS-8707.003.patch, 
> HDFS-9117.HDFS-8707.004.patch, HDFS-9117.HDFS-8707.005.patch, 
> HDFS-9117.HDFS-8707.006.patch, HDFS-9117.HDFS-8707.008.patch, 
> HDFS-9117.HDFS-8707.009.patch, HDFS-9117.HDFS-9288.007.patch
>
>
> For environmental compatability with HDFS installations, libhdfs++ should be 
> able to read the configurations from Hadoop XML files and behave in line with 
> the Java implementation.
> Most notably, machine names and ports should be readable from Hadoop XML 
> configuration files.
> Similarly, an internal Options architecture for libhdfs++ should be developed 
> to efficiently transport the configuration information within the system.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HDFS-8766) Implement a libhdfs(3) compatible API

[Haohui Mai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978819#comment-14978819
 ] 

Haohui Mai edited comment on HDFS-8766 at 10/28/15 5:27 PM:


{code}
+#include "libhdfspp/hdfs.h"

+  auto callback = [stat, ](const Status , FileSystem *f) {
+fs = f;
+stat->set_value(s);
+  };
{code}

It might make sense to rename the {{libhdfspp/hdfs.h}} to 
{{libhdfspp/libhdfspp.h}}. It is also beneficial to change the way how the 
{{FileSystem}} works to only make the connection when calling {{Connect()}}.

However, I think both of them can be addressed in follow-up jiras.


was (Author: wheat9):
Things that be a
{code}
+#include "libhdfspp/hdfs.h"

+  auto callback = [stat, ](const Status , FileSystem *f) {
+fs = f;
+stat->set_value(s);
+  };
{code}

It might make sense to rename the {{libhdfspp/hdfs.h}} to 
{{libhdfspp/libhdfspp.h}}. It is also beneficial to change the way how the 
{{FileSystem}} works to only make the connection when calling {{Connect()}}. 
They can be addressed in follow-up jiras.

> Implement a libhdfs(3) compatible API
> -
>
> Key: HDFS-8766
> URL: https://issues.apache.org/jira/browse/HDFS-8766
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
> Attachments: HDFS-8766.HDFS-8707.000.patch, 
> HDFS-8766.HDFS-8707.001.patch, HDFS-8766.HDFS-8707.002.patch, 
> HDFS-8766.HDFS-8707.003.patch, HDFS-8766.HDFS-8707.004.patch, 
> HDFS-8766.HDFS-8707.005.patch, HDFS-8766.HDFS-8707.006.patch, 
> HDFS-8766.HDFS-8707.007.patch, HDFS-8766.HDFS-8707.008.patch, 
> HDFS-8766.HDFS-8707.009.patch, HDFS-8766.HDFS-8707.010.patch, 
> HDFS-8766.HDFS-8707.011.patch
>
>
> Add a synchronous API that is compatible with the hdfs.h header used in 
> libhdfs and libhdfs3.  This will make it possible for projects using 
> libhdfs/libhdfs3 to relink against libhdfspp with minimal changes.
> This also provides a pure C interface that can be linked against projects 
> that aren't built in C++11 mode for various reasons but use the same 
> compiler.  It also allows many other programming languages to access 
> libhdfspp through builtin FFI interfaces.
> The libhdfs API is very similar to the posix file API which makes it easier 
> for programs built using posix filesystem calls to be modified to access HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9289) check genStamp when complete file

[Jing Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978896#comment-14978896
 ] 

Jing Zhao commented on HDFS-9289:
-

Making DataStreamer#block volatile is a good change, the GS validation on the 
NN side also looks good to me. Maybe we do not need a new type of 
{{InvalidGenStampException}} though. And to log the detailed information of the 
block with mismatching GS on the NN side will also be useful.

> check genStamp when complete file
> -
>
> Key: HDFS-9289
> URL: https://issues.apache.org/jira/browse/HDFS-9289
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Chang Li
>Assignee: Chang Li
>Priority: Critical
> Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch
>
>
> we have seen a case of corrupt block which is caused by file complete after a 
> pipelineUpdate, but the file complete with the old block genStamp. This 
> caused the replicas of two datanodes in updated pipeline to be viewed as 
> corrupte. Propose to check genstamp when commit block



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9255) Consolidate block recovery related implementation into a single class

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978910#comment-14978910
 ] 

Hudson commented on HDFS-9255:
--

SUCCESS: Integrated in Hadoop-Yarn-trunk #1331 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1331/])
HDFS-9255. Consolidate block recovery related implementation into a (zhz: rev 
e287e7d14b838a866ba03d895fa3581d7c09)
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java


> Consolidate block recovery related implementation into a single class
> -
>
> Key: HDFS-9255
> URL: https://issues.apache.org/jira/browse/HDFS-9255
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Reporter: Walter Su
>Assignee: Walter Su
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9255-branch-2.06.patch, HDFS-9255.01.patch, 
> HDFS-9255.02.patch, HDFS-9255.03.patch, HDFS-9255.04.patch, 
> HDFS-9255.05.patch, HDFS-9255.06.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-7163) WebHdfsFileSystem should retry reads according to the configured retry policy.

[Eric Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Payne updated HDFS-7163:
-
Status: Patch Available  (was: Open)

> WebHdfsFileSystem should retry reads according to the configured retry policy.
> --
>
> Key: HDFS-7163
> URL: https://issues.apache.org/jira/browse/HDFS-7163
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 2.5.1, 3.0.0
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: HDFS-7163.001.patch, WebHDFS Read Retry.pdf
>
>
> In the current implementation of WebHdfsFileSystem, opens are retried 
> according to the configured retry policy, but not reads. Therefore, if a 
> connection goes down while data is being read, the read will fail and the 
> read will have to be retried by the client code.
> Also, after a connection has been established, the next read (or seek/read) 
> will fail and the read will have to be restarted by the client code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-7163) WebHdfsFileSystem should retry reads according to the configured retry policy.

[Eric Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Payne updated HDFS-7163:
-
Attachment: HDFS-7163.001.patch

> WebHdfsFileSystem should retry reads according to the configured retry policy.
> --
>
> Key: HDFS-7163
> URL: https://issues.apache.org/jira/browse/HDFS-7163
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 3.0.0, 2.5.1
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: HDFS-7163.001.patch, WebHDFS Read Retry.pdf
>
>
> In the current implementation of WebHdfsFileSystem, opens are retried 
> according to the configured retry policy, but not reads. Therefore, if a 
> connection goes down while data is being read, the read will fail and the 
> read will have to be retried by the client code.
> Also, after a connection has been established, the next read (or seek/read) 
> will fail and the read will have to be restarted by the client code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979002#comment-14979002
 ] 

Hudson commented on HDFS-9279:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #596 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/596/])
HDFS-9279. Decomissioned capacity should not be considered for (kihwal: rev 
19a77f546657b086af8f41fa631099bdde7e010c)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java


> Decomissioned capacity should not be considered for configured/used capacity
> 
>
> Key: HDFS-9279
> URL: https://issues.apache.org/jira/browse/HDFS-9279
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Fix For: 3.0.0, 2.8.0
>
> Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, 
> HDFS-9279-v3.patch, HDFS-9279-v4.patch
>
>
> Capacity of a decommissioned node is being accounted as configured and used 
> capacity metrics. This gives incorrect perception of cluster usage.
> Once a node is decommissioned, its capacity should be considered similar to a 
> dead node.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9117) Config file reader / options classes for libhdfs++

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979090#comment-14979090
 ] 

Hadoop QA commented on HDFS-9117:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s 
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 5m 32s 
{color} | {color:red} root in HDFS-8707 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 12s 
{color} | {color:red} hadoop-hdfs-native-client in HDFS-8707 failed with JDK 
v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 7s 
{color} | {color:red} hadoop-hdfs-native-client in HDFS-8707 failed with JDK 
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} mvneclipse {color} | {color:red} 0m 13s 
{color} | {color:red} hadoop-hdfs-native-client in HDFS-8707 failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 7s 
{color} | {color:red} hadoop-hdfs-native-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 7s 
{color} | {color:red} hadoop-hdfs-native-client in the patch failed with JDK 
v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 7s {color} | 
{color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.8.0_60. 
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 7s {color} 
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.8.0_60. 
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 7s 
{color} | {color:red} hadoop-hdfs-native-client in the patch failed with JDK 
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 7s {color} | 
{color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.7.0_79. 
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 7s {color} 
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.7.0_79. 
{color} |
| {color:red}-1{color} | {color:red} mvneclipse {color} | {color:red} 0m 8s 
{color} | {color:red} hadoop-hdfs-native-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 11s {color} 
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.8.0_60. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 10s {color} 
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.7.0_79. 
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 19s 
{color} | {color:red} Patch generated 425 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 8m 54s {color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.1 Server=1.7.1 
Image:test-patch-base-hadoop-date2015-10-28 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12769308/HDFS-9117.HDFS-8707.009.patch
 |
| JIRA Issue | HDFS-9117 |
| Optional Tests |  asflicense  cc  unit  javac  compile  |
| uname | Linux c11fd3a34c45 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-67f42f1/precommit/personality/hadoop.sh
 |
| git revision | HDFS-8707 / 6d68759 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions |  /usr/lib/jvm/java-8-oracle:1.8.0_60 
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 |
| mvninstall | 
https://builds.apache.org/job/PreCommit-HDFS-Build/13257/artifact/patchprocess/branch-mvninstall-root.txt
 |
| compile | 
https://builds.apache.org/job/PreCommit-HDFS-Build/13257/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.8.0_60.txt
 |
| compile | 
https://builds.apache.org/job/PreCommit-HDFS-Build/13257/artifact/patchprocess/branch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.7.0_79.txt
 |
| mvneclipse | 

[jira] [Updated] (HDFS-8660) Slow write to packet mirror should log which mirror and which block

[Hazem Mahmoud (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hazem Mahmoud updated HDFS-8660:

Affects Version/s: (was: 2.7.0)

> Slow write to packet mirror should log which mirror and which block
> ---
>
> Key: HDFS-8660
> URL: https://issues.apache.org/jira/browse/HDFS-8660
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Hazem Mahmoud
>Assignee: Hazem Mahmoud
> Attachments: HDFS-8660.001.patch
>
>
> Currently, log format states something similar to: 
> "Slow BlockReceiver write packet to mirror took 468ms (threshold=300ms)"
> For troubleshooting purposes, it would be good to have it mention which block 
> ID it's writing as well as the mirror (DN) that it's writing it to.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9328) Formalize coding standards for libhdfs++ and put them in a README.txt

[James Clampffer (JIRA)]

James Clampffer created HDFS-9328:
-

 Summary: Formalize coding standards for libhdfs++ and put them in 
a README.txt
 Key: HDFS-9328
 URL: https://issues.apache.org/jira/browse/HDFS-9328
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: James Clampffer
Assignee: Haohui Mai
Priority: Blocker


We have 2-3 people working on this project full time and hopefully more people 
will start contributing.  In order to efficiently scale we need a single, easy 
to find, place where developers can check to make sure they are following the 
coding standards of this project to both save their time and save the time of 
people doing code reviews.

The most practical place to do this seems like a README file in libhdfspp/. 

The foundation of the standards is google's C++ guide found here: 
https://google-styleguide.googlecode.com/svn/trunk/cppguide.html

Any exceptions to google's standards or additional restrictions need to be 
explicitly enumerated so there is one single point of reference for all 
libhdfs++ code standards.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-9329) TestBootstrapStandby#testRateThrottling is flaky because fsimage size is smaller than IO buffer size

[Zhe Zhang (JIRA)]

Zhe Zhang created HDFS-9329:
---

 Summary: TestBootstrapStandby#testRateThrottling is flaky because 
fsimage size is smaller than IO buffer size
 Key: HDFS-9329
 URL: https://issues.apache.org/jira/browse/HDFS-9329
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Affects Versions: 2.7.1
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Minor


{{testRateThrottling}} verifies that bootstrap transfer should timeout with a 
very small {{DFS_IMAGE_TRANSFER_BOOTSTRAP_STANDBY_RATE_KEY}} value. However, 
throttling on the image sender only happens after sending each IO buffer. 
Therefore, the test sometimes fails if the receiver receives the full fsimage 
(which is smaller than IO buffer size) before throttling begins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9255) Consolidate block recovery related implementation into a single class

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979056#comment-14979056
 ] 

Hudson commented on HDFS-9255:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #546 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/546/])
HDFS-9255. Consolidate block recovery related implementation into a (zhz: rev 
e287e7d14b838a866ba03d895fa3581d7c09)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java


> Consolidate block recovery related implementation into a single class
> -
>
> Key: HDFS-9255
> URL: https://issues.apache.org/jira/browse/HDFS-9255
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Reporter: Walter Su
>Assignee: Walter Su
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9255-branch-2.06.patch, HDFS-9255.01.patch, 
> HDFS-9255.02.patch, HDFS-9255.03.patch, HDFS-9255.04.patch, 
> HDFS-9255.05.patch, HDFS-9255.06.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-8660) Slow write to packet mirror should log which mirror and which block

[Hazem Mahmoud (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hazem Mahmoud updated HDFS-8660:

Status: Patch Available  (was: Open)

> Slow write to packet mirror should log which mirror and which block
> ---
>
> Key: HDFS-8660
> URL: https://issues.apache.org/jira/browse/HDFS-8660
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Hazem Mahmoud
>Assignee: Hazem Mahmoud
> Attachments: HDFS-8660.001.patch
>
>
> Currently, log format states something similar to: 
> "Slow BlockReceiver write packet to mirror took 468ms (threshold=300ms)"
> For troubleshooting purposes, it would be good to have it mention which block 
> ID it's writing as well as the mirror (DN) that it's writing it to.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-8660) Slow write to packet mirror should log which mirror and which block

[Hazem Mahmoud (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hazem Mahmoud updated HDFS-8660:

Status: Open  (was: Patch Available)

Canceling to resubmit after removing Affects Version to see if that gets me 
passed failed test (due to DataStreamer.java not existing in that version)

> Slow write to packet mirror should log which mirror and which block
> ---
>
> Key: HDFS-8660
> URL: https://issues.apache.org/jira/browse/HDFS-8660
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Hazem Mahmoud
>Assignee: Hazem Mahmoud
> Attachments: HDFS-8660.001.patch
>
>
> Currently, log format states something similar to: 
> "Slow BlockReceiver write packet to mirror took 468ms (threshold=300ms)"
> For troubleshooting purposes, it would be good to have it mention which block 
> ID it's writing as well as the mirror (DN) that it's writing it to.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9279) Decomissioned capacity should not be considered for configured/used capacity

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979081#comment-14979081
 ] 

Hudson commented on HDFS-9279:
--

FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #609 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/609/])
HDFS-9279. Decomissioned capacity should not be considered for (kihwal: rev 
19a77f546657b086af8f41fa631099bdde7e010c)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStats.java


> Decomissioned capacity should not be considered for configured/used capacity
> 
>
> Key: HDFS-9279
> URL: https://issues.apache.org/jira/browse/HDFS-9279
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Fix For: 3.0.0, 2.8.0
>
> Attachments: HDFS-9279-v1.patch, HDFS-9279-v2.patch, 
> HDFS-9279-v3.patch, HDFS-9279-v4.patch
>
>
> Capacity of a decommissioned node is being accounted as configured and used 
> capacity metrics. This gives incorrect perception of cluster usage.
> Once a node is decommissioned, its capacity should be considered similar to a 
> dead node.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9083) Replication violates block placement policy.

[Rushabh S Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979094#comment-14979094
 ] 

Rushabh S Shah commented on HDFS-9083:
--

Ran the findbugs on hadoop-hdfs-project and there is one findbugs warning in 
PBImageTextWriter.java.
I haven't made any changes in that file.

> Replication violates block placement policy.
> 
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: HDFS, namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Attachments: HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are 
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in 
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is 
> dependent on the config (net.topology.script.file.name)
> {noformat}
>  if (!this.shouldCheckForEnoughRacks) {
>   return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation) 
> via net.topology.node.switch.mapping.impl and no longer use 
> net.topology.script.file.name config.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9255) Consolidate block recovery related implementation into a single class

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979118#comment-14979118
 ] 

Hudson commented on HDFS-9255:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk #2538 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2538/])
HDFS-9255. Consolidate block recovery related implementation into a (zhz: rev 
e287e7d14b838a866ba03d895fa3581d7c09)
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java


> Consolidate block recovery related implementation into a single class
> -
>
> Key: HDFS-9255
> URL: https://issues.apache.org/jira/browse/HDFS-9255
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Reporter: Walter Su
>Assignee: Walter Su
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9255-branch-2.06.patch, HDFS-9255.01.patch, 
> HDFS-9255.02.patch, HDFS-9255.03.patch, HDFS-9255.04.patch, 
> HDFS-9255.05.patch, HDFS-9255.06.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-9083) Replication violates block placement policy.

[Rushabh S Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979126#comment-14979126
 ] 

Rushabh S Shah commented on HDFS-9083:
--

[~kihwal]: Thanks for committing.

> Replication violates block placement policy.
> 
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: HDFS, namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Fix For: 2.7.2
>
> Attachments: HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are 
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in 
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is 
> dependent on the config (net.topology.script.file.name)
> {noformat}
>  if (!this.shouldCheckForEnoughRacks) {
>   return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation) 
> via net.topology.node.switch.mapping.impl and no longer use 
> net.topology.script.file.name config.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)