[jira] [Commented] (HDFS-8093) BP does not exist or is not under Constructionnull
[ https://issues.apache.org/jira/browse/HDFS-8093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986929#comment-14986929 ] Xiaoyu Yao commented on HDFS-8093: -- [~Alexandre LINTE], thanks for the confirmation!. This one can be resolved as a dup of HDFS-9364 [~szetszwo] is working on. > BP does not exist or is not under Constructionnull > -- > > Key: HDFS-8093 > URL: https://issues.apache.org/jira/browse/HDFS-8093 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover >Affects Versions: 2.6.0 > Environment: Centos 6.5 >Reporter: LINTE > > HDFS balancer run during several hours blancing blocs beetween datanode, it > ended by failing with the following error. > getStoredBlock function return a null BlockInfo. > java.io.IOException: Bad response ERROR for block > BP-970443206-192.168.0.208-1397583979378:blk_1086729930_13046030 from > datanode 192.168.0.18:1004 > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:897) > 15/04/08 05:52:51 WARN hdfs.DFSClient: Error Recovery for block > BP-970443206-192.168.0.208-1397583979378:blk_1086729930_13046030 in pipeline > 192.168.0.63:1004, 192.168.0.1:1004, 192.168.0.18:1004: bad datanode > 192.168.0.18:1004 > 15/04/08 05:52:51 WARN hdfs.DFSClient: DataStreamer Exception > org.apache.hadoop.ipc.RemoteException(java.io.IOException): > BP-970443206-192.168.0.208-1397583979378:blk_1086729930_13046030 does not > exist or is not under Constructionnull > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6913) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6980) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:717) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:931) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) > at org.apache.hadoop.ipc.Client.call(Client.java:1468) > at org.apache.hadoop.ipc.Client.call(Client.java:1399) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) > at com.sun.proxy.$Proxy11.updateBlockForPipeline(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:877) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy12.updateBlockForPipeline(Unknown Source) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1266) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1004) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:548) > 15/04/08 05:52:51 ERROR hdfs.DFSClient: Failed to close inode 19801755 > org.apache.hadoop.ipc.RemoteException(java.io.IOException): > BP-970443206-192.168.0.208-1397583979378:blk_1086729930_13046030 does not > exist or is not under Constructionnull > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6913) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6980) > at >
[jira] [Commented] (HDFS-9007) Fix HDFS Balancer to honor upgrade domain policy
[
https://issues.apache.org/jira/browse/HDFS-9007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986940#comment-14986940
]
Hadoop QA commented on HDFS-9007:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 5s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
7s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 50s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk cannot run
convertXmlToText from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 46s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s
{color} | {color:red} Patch generated 3 new checkstyle issues in
hadoop-hdfs-project/hadoop-hdfs (total was 109, now 104). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 46s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 51m 52s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 49m 53s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 20s
{color} | {color:red} Patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 121m 1s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_60 Failed junit tests |
hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| JDK v1.7.0_79 Failed junit tests |
hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.1 Server=1.7.1
Image:test-patch-base-hadoop-date2015-11-03 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12770257/HDFS-9007-2.patch |
| JIRA Issue | HDFS-9007 |
| Optional Tests | asflicense javac javadoc mvninstall unit findbugs
checkstyle compile |
| uname | Linux efb30c1dc716 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
|
[jira] [Commented] (HDFS-8093) BP does not exist or is not under Constructionnull
[ https://issues.apache.org/jira/browse/HDFS-8093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986927#comment-14986927 ] LINTE commented on HDFS-8093: - No more mistakes for me with secure hadoop 2.7.1 and namenode HA settings for balancer. > BP does not exist or is not under Constructionnull > -- > > Key: HDFS-8093 > URL: https://issues.apache.org/jira/browse/HDFS-8093 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover >Affects Versions: 2.6.0 > Environment: Centos 6.5 >Reporter: LINTE > > HDFS balancer run during several hours blancing blocs beetween datanode, it > ended by failing with the following error. > getStoredBlock function return a null BlockInfo. > java.io.IOException: Bad response ERROR for block > BP-970443206-192.168.0.208-1397583979378:blk_1086729930_13046030 from > datanode 192.168.0.18:1004 > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:897) > 15/04/08 05:52:51 WARN hdfs.DFSClient: Error Recovery for block > BP-970443206-192.168.0.208-1397583979378:blk_1086729930_13046030 in pipeline > 192.168.0.63:1004, 192.168.0.1:1004, 192.168.0.18:1004: bad datanode > 192.168.0.18:1004 > 15/04/08 05:52:51 WARN hdfs.DFSClient: DataStreamer Exception > org.apache.hadoop.ipc.RemoteException(java.io.IOException): > BP-970443206-192.168.0.208-1397583979378:blk_1086729930_13046030 does not > exist or is not under Constructionnull > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6913) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6980) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:717) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:931) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) > at org.apache.hadoop.ipc.Client.call(Client.java:1468) > at org.apache.hadoop.ipc.Client.call(Client.java:1399) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) > at com.sun.proxy.$Proxy11.updateBlockForPipeline(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:877) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy12.updateBlockForPipeline(Unknown Source) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1266) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1004) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:548) > 15/04/08 05:52:51 ERROR hdfs.DFSClient: Failed to close inode 19801755 > org.apache.hadoop.ipc.RemoteException(java.io.IOException): > BP-970443206-192.168.0.208-1397583979378:blk_1086729930_13046030 does not > exist or is not under Constructionnull > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6913) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6980) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:717) > at >
[jira] [Created] (HDFS-9366) Webhdfs support set result size and cursor value in op=LISTSTATUS
Yong Zhang created HDFS-9366: Summary: Webhdfs support set result size and cursor value in op=LISTSTATUS Key: HDFS-9366 URL: https://issues.apache.org/jira/browse/HDFS-9366 Project: Hadoop HDFS Issue Type: Improvement Reporter: Yong Zhang Assignee: Yong Zhang webhdfs will return large result if HDFS directory with large number of children. Some webui will not work or very slow in this scnario such as HUE. webhdfs could support LISTSTATUS operation to get all children via more than 1 request, each request only return a small size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-8093) BP does not exist or is not under Constructionnull
[ https://issues.apache.org/jira/browse/HDFS-8093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDFS-8093. -- Resolution: Duplicate > BP does not exist or is not under Constructionnull > -- > > Key: HDFS-8093 > URL: https://issues.apache.org/jira/browse/HDFS-8093 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover >Affects Versions: 2.6.0 > Environment: Centos 6.5 >Reporter: LINTE > > HDFS balancer run during several hours blancing blocs beetween datanode, it > ended by failing with the following error. > getStoredBlock function return a null BlockInfo. > java.io.IOException: Bad response ERROR for block > BP-970443206-192.168.0.208-1397583979378:blk_1086729930_13046030 from > datanode 192.168.0.18:1004 > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:897) > 15/04/08 05:52:51 WARN hdfs.DFSClient: Error Recovery for block > BP-970443206-192.168.0.208-1397583979378:blk_1086729930_13046030 in pipeline > 192.168.0.63:1004, 192.168.0.1:1004, 192.168.0.18:1004: bad datanode > 192.168.0.18:1004 > 15/04/08 05:52:51 WARN hdfs.DFSClient: DataStreamer Exception > org.apache.hadoop.ipc.RemoteException(java.io.IOException): > BP-970443206-192.168.0.208-1397583979378:blk_1086729930_13046030 does not > exist or is not under Constructionnull > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6913) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6980) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:717) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:931) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) > at org.apache.hadoop.ipc.Client.call(Client.java:1468) > at org.apache.hadoop.ipc.Client.call(Client.java:1399) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) > at com.sun.proxy.$Proxy11.updateBlockForPipeline(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:877) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy12.updateBlockForPipeline(Unknown Source) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1266) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1004) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:548) > 15/04/08 05:52:51 ERROR hdfs.DFSClient: Failed to close inode 19801755 > org.apache.hadoop.ipc.RemoteException(java.io.IOException): > BP-970443206-192.168.0.208-1397583979378:blk_1086729930_13046030 does not > exist or is not under Constructionnull > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6913) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6980) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:717) > at >
[jira] [Commented] (HDFS-9354) Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows
[
https://issues.apache.org/jira/browse/HDFS-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986872#comment-14986872
]
Hadoop QA commented on HDFS-9354:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 6s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
2s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 52s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk cannot run
convertXmlToText from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 48s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 47s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 38s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 23s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 22s
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 147m 26s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_60 Failed junit tests |
hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot |
| JDK v1.7.0_79 Failed junit tests |
hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot |
| | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.TestParallelShortCircuitReadUnCached |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.1 Server=1.7.1
Image:test-patch-base-hadoop-date2015-11-03 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12770211/HDFS-9354.01.patch |
| JIRA Issue | HDFS-9354 |
| Optional Tests | asflicense javac javadoc mvninstall unit findbugs
checkstyle compile |
| uname | Linux a40ec487e282 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HDFS-9362) TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings, fails on Windows.
[
https://issues.apache.org/jira/browse/HDFS-9362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986909#comment-14986909
]
Hadoop QA commented on HDFS-9362:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 8s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
57s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 56s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk cannot run
convertXmlToText from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 49s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 48s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 30s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 25s
{color} | {color:red} Patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 158m 31s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.TestSafeMode |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength |
| JDK v1.8.0_66 Timed out junit tests |
org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary
|
| JDK v1.7.0_79 Failed junit tests |
hadoop.hdfs.TestReadStripedFileWithDecoding |
| | hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.0 Server=1.7.0
Image:test-patch-base-hadoop-date2015-11-03 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12770187/HDFS-9362.001.patch |
| JIRA Issue | HDFS-9362 |
| Optional Tests | asflicense javac javadoc mvninstall unit findbugs
checkstyle compile |
| uname | Linux 353dae7e4a40 3.13.0-36-lowlatency
[jira] [Commented] (HDFS-8093) BP does not exist or is not under Constructionnull
[
https://issues.apache.org/jira/browse/HDFS-8093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986922#comment-14986922
]
Xiaoyu Yao commented on HDFS-8093:
--
[~fborchers], do you have namenode HA setup? If yes, does it still repro when
you explicitly specify active NN as follows?
{code}
hdfs balancer -fs http://activeNN:8020 -threshold 5
{code}
> BP does not exist or is not under Constructionnull
> --
>
> Key: HDFS-8093
> URL: https://issues.apache.org/jira/browse/HDFS-8093
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Affects Versions: 2.6.0
> Environment: Centos 6.5
>Reporter: LINTE
>
> HDFS balancer run during several hours blancing blocs beetween datanode, it
> ended by failing with the following error.
> getStoredBlock function return a null BlockInfo.
> java.io.IOException: Bad response ERROR for block
> BP-970443206-192.168.0.208-1397583979378:blk_1086729930_13046030 from
> datanode 192.168.0.18:1004
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:897)
> 15/04/08 05:52:51 WARN hdfs.DFSClient: Error Recovery for block
> BP-970443206-192.168.0.208-1397583979378:blk_1086729930_13046030 in pipeline
> 192.168.0.63:1004, 192.168.0.1:1004, 192.168.0.18:1004: bad datanode
> 192.168.0.18:1004
> 15/04/08 05:52:51 WARN hdfs.DFSClient: DataStreamer Exception
> org.apache.hadoop.ipc.RemoteException(java.io.IOException):
> BP-970443206-192.168.0.208-1397583979378:blk_1086729930_13046030 does not
> exist or is not under Constructionnull
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6913)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6980)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:717)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:931)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
> at org.apache.hadoop.ipc.Client.call(Client.java:1468)
> at org.apache.hadoop.ipc.Client.call(Client.java:1399)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy11.updateBlockForPipeline(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:877)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy12.updateBlockForPipeline(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1266)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1004)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:548)
> 15/04/08 05:52:51 ERROR hdfs.DFSClient: Failed to close inode 19801755
> org.apache.hadoop.ipc.RemoteException(java.io.IOException):
> BP-970443206-192.168.0.208-1397583979378:blk_1086729930_13046030 does not
> exist or is not under Constructionnull
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6913)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6980)
> at
>
[jira] [Commented] (HDFS-8093) BP does not exist or is not under Constructionnull
[ https://issues.apache.org/jira/browse/HDFS-8093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986930#comment-14986930 ] Xiaoyu Yao commented on HDFS-8093: -- It should be HDFS-9365. I can't edit my previous comments after it is posted. > BP does not exist or is not under Constructionnull > -- > > Key: HDFS-8093 > URL: https://issues.apache.org/jira/browse/HDFS-8093 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover >Affects Versions: 2.6.0 > Environment: Centos 6.5 >Reporter: LINTE > > HDFS balancer run during several hours blancing blocs beetween datanode, it > ended by failing with the following error. > getStoredBlock function return a null BlockInfo. > java.io.IOException: Bad response ERROR for block > BP-970443206-192.168.0.208-1397583979378:blk_1086729930_13046030 from > datanode 192.168.0.18:1004 > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:897) > 15/04/08 05:52:51 WARN hdfs.DFSClient: Error Recovery for block > BP-970443206-192.168.0.208-1397583979378:blk_1086729930_13046030 in pipeline > 192.168.0.63:1004, 192.168.0.1:1004, 192.168.0.18:1004: bad datanode > 192.168.0.18:1004 > 15/04/08 05:52:51 WARN hdfs.DFSClient: DataStreamer Exception > org.apache.hadoop.ipc.RemoteException(java.io.IOException): > BP-970443206-192.168.0.208-1397583979378:blk_1086729930_13046030 does not > exist or is not under Constructionnull > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6913) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6980) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:717) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:931) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) > at org.apache.hadoop.ipc.Client.call(Client.java:1468) > at org.apache.hadoop.ipc.Client.call(Client.java:1399) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) > at com.sun.proxy.$Proxy11.updateBlockForPipeline(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:877) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy12.updateBlockForPipeline(Unknown Source) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1266) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1004) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:548) > 15/04/08 05:52:51 ERROR hdfs.DFSClient: Failed to close inode 19801755 > org.apache.hadoop.ipc.RemoteException(java.io.IOException): > BP-970443206-192.168.0.208-1397583979378:blk_1086729930_13046030 does not > exist or is not under Constructionnull > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6913) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6980) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:717) > at >
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987027#comment-14987027
]
Hudson commented on HDFS-9313:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #566 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/566/])
HDFS-9313. Possible NullPointerException in BlockManager if no excess (mingma:
rev d565480da2f646b40c3180e1ccb2935c9863dfef)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Fix For: 2.8.0
>
> Attachments: HDFS-9313-2.patch, HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9333) Some tests using MiniDFSCluster errored complaining port in use
[
https://issues.apache.org/jira/browse/HDFS-9333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987168#comment-14987168
]
Kai Zheng commented on HDFS-9333:
-
Sounds good to me. Thank you for looking at this!
> Some tests using MiniDFSCluster errored complaining port in use
> ---
>
> Key: HDFS-9333
> URL: https://issues.apache.org/jira/browse/HDFS-9333
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Kai Zheng
>Assignee: Masatake Iwasaki
>Priority: Minor
>
> Ref. the following:
> {noformat}
> Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 30.483 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
> testRead(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped)
> Time elapsed: 11.021 sec <<< ERROR!
> java.net.BindException: Port in use: localhost:49333
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at
> org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
> at
> org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:884)
> at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:826)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:821)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:675)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1555)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:2015)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1996)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS.doTestRead(TestBlockTokenWithDFS.java:539)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped.testRead(TestBlockTokenWithDFSStriped.java:62)
> {noformat}
> Another one:
> {noformat}
> Tests run: 5, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 9.859 sec <<<
> FAILURE! - in org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController
> testFailoverAndBackOnNNShutdown(org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController)
> Time elapsed: 0.41 sec <<< ERROR!
> java.net.BindException: Problem binding to [localhost:10021]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:469)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:695)
> at org.apache.hadoop.ipc.Server.(Server.java:2464)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:399)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1555)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1245)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1014)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:889)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:821)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:480)
> at
>
[jira] [Commented] (HDFS-9129) Move the safemode block count into BlockManager
[
https://issues.apache.org/jira/browse/HDFS-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987196#comment-14987196
]
Hadoop QA commented on HDFS-9129:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 6s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 7s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk cannot run
convertXmlToText from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 8s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 17s
{color} | {color:red} Patch generated 4 new checkstyle issues in
hadoop-hdfs-project/hadoop-hdfs (total was 808, now 755). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 3s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 107m 2s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 100m 7s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 23s
{color} | {color:red} Patch generated 60 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 229m 19s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_60 Failed junit tests |
hadoop.hdfs.TestReadStripedFileWithDecoding |
| | hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.server.blockmanagement.TestNodeCount |
| | hadoop.hdfs.server.mover.TestStorageMover |
| | hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks |
| | hadoop.hdfs.server.namenode.TestMetaSave |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.mover.TestMover |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks |
| | hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| |
[jira] [Updated] (HDFS-9357) NN UI is not showing which DN is "Decommissioned "and "Decommissioned & dead".
[ https://issues.apache.org/jira/browse/HDFS-9357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-9357: - Attachment: HDFS-9357.001.patch > NN UI is not showing which DN is "Decommissioned "and "Decommissioned & dead". > -- > > Key: HDFS-9357 > URL: https://issues.apache.org/jira/browse/HDFS-9357 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Archana T >Assignee: Surendra Singh Lilhore >Priority: Critical > Attachments: HDFS-9357.001.patch, decommisioned_n_dead_.png, > decommissioned_.png > > > NN UI is not showing which DN is "Decommissioned "and "Decommissioned & dead" > Root Cause -- > "Decommissioned" and "Decommissioned & dead" icon not reflected on NN UI > When DN is in Decommissioned status or in "Decommissioned & dead" status, > same status is not reflected on NN UI > DN status is as below -- > hdfs dfsadmin -report > Name: 10.xx.xx.xx1:50076 (host-xx1) > Hostname: host-xx > Decommission Status : Decommissioned > Configured Capacity: 230501634048 (214.67 GB) > DFS Used: 36864 (36 KB) > Dead datanodes (1): > Name: 10.xx.xx.xx2:50076 (host-xx2) > Hostname: host-xx > Decommission Status : Decommissioned > Same is not reflected on NN UI. > Attached NN UI snapshots for the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8898) Create API and command-line argument to get quota without need to get file and directory counts
[
https://issues.apache.org/jira/browse/HDFS-8898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987097#comment-14987097
]
Vinayakumar B commented on HDFS-8898:
-
Thanks [~mingma] for the latest patch.
It looks really nice.
Below are some minor commments
1. {{DFSClient#getQuotaUsage(..)}} should call NN RPC inside Trace span.
2. {{DFSClient#getQuotaUsage(..)}} should call {{checkOpen()}} before going for
NN RPC call
3. Option added to Count command needs document update.
4. Following code too is expected to be inside fsd lock.
{code}src = fsd.resolvePath(pc, src, pathComponents);
final INodesInPath iip = fsd.getINodesInPath(src, false);
if (fsd.isPermissionEnabled()) {
fsd.checkPermission(pc, iip, false, null, null, null,
FsAction.READ_EXECUTE);
}{code}
5. {{FSNamesystem#getQuotaUsage(..)}} needs formatting.
> Create API and command-line argument to get quota without need to get file
> and directory counts
> ---
>
> Key: HDFS-8898
> URL: https://issues.apache.org/jira/browse/HDFS-8898
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs
>Reporter: Joep Rottinghuis
>Assignee: Ming Ma
> Attachments: HDFS-8898-2.patch, HDFS-8898.patch
>
>
> On large directory structures it takes significant time to iterate through
> the file and directory counts recursively to get a complete ContentSummary.
> When you want to just check for the quota on a higher level directory it
> would be good to have an option to skip the file and directory counts.
> Moreover, currently one can only check the quota if you have access to all
> the directories underneath. For example, if I have a large home directory
> under /user/joep and I host some files for another user in a sub-directory,
> the moment they create an unreadable sub-directory under my home I can no
> longer check what my quota is. Understood that I cannot check the current
> file counts unless I can iterate through all the usage, but for
> administrative purposes it is nice to be able to get the current quota
> setting on a directory without the need to iterate through and run into
> permission issues on sub-directories.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9364) Unnecessary DNS resolution attempts when NameNodeProxies
[
https://issues.apache.org/jira/browse/HDFS-9364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987125#comment-14987125
]
Hadoop QA commented on HDFS-9364:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 6s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 8s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk cannot run
convertXmlToText from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 44s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 30s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 27s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 26s {color}
| {color:red} hadoop-hdfs-project-jdk1.8.0_60 with JDK v1.8.0_60 has problems.
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 9m 30s {color}
| {color:red} hadoop-hdfs-project-jdk1.7.0_79 with JDK v1.7.0_79 has problems.
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 29s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 21s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 1s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 39s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 2s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 24s
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 158m 1s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_60 Failed junit tests |
hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| JDK v1.7.0_79 Failed junit tests |
hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| |
[jira] [Commented] (HDFS-8898) Create API and command-line argument to get quota without need to get file and directory counts
[ https://issues.apache.org/jira/browse/HDFS-8898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987107#comment-14987107 ] Vinayakumar B commented on HDFS-8898: - Seems like precommit didnt detect this. triggered one run now. > Create API and command-line argument to get quota without need to get file > and directory counts > --- > > Key: HDFS-8898 > URL: https://issues.apache.org/jira/browse/HDFS-8898 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Joep Rottinghuis >Assignee: Ming Ma > Attachments: HDFS-8898-2.patch, HDFS-8898.patch > > > On large directory structures it takes significant time to iterate through > the file and directory counts recursively to get a complete ContentSummary. > When you want to just check for the quota on a higher level directory it > would be good to have an option to skip the file and directory counts. > Moreover, currently one can only check the quota if you have access to all > the directories underneath. For example, if I have a large home directory > under /user/joep and I host some files for another user in a sub-directory, > the moment they create an unreadable sub-directory under my home I can no > longer check what my quota is. Understood that I cannot check the current > file counts unless I can iterate through all the usage, but for > administrative purposes it is nice to be able to get the current quota > setting on a directory without the need to iterate through and run into > permission issues on sub-directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9357) NN UI is not showing which DN is "Decommissioned "and "Decommissioned & dead".
[
https://issues.apache.org/jira/browse/HDFS-9357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-9357:
-
Status: Patch Available (was: Open)
This issue came because of typo in {{hadoop.css}} and {{dfshealth.html}}
Attached patch, please review
> NN UI is not showing which DN is "Decommissioned "and "Decommissioned & dead".
> --
>
> Key: HDFS-9357
> URL: https://issues.apache.org/jira/browse/HDFS-9357
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Archana T
>Assignee: Surendra Singh Lilhore
>Priority: Critical
> Attachments: HDFS-9357.001.patch, decommisioned_n_dead_.png,
> decommissioned_.png
>
>
> NN UI is not showing which DN is "Decommissioned "and "Decommissioned & dead"
> Root Cause --
> "Decommissioned" and "Decommissioned & dead" icon not reflected on NN UI
> When DN is in Decommissioned status or in "Decommissioned & dead" status,
> same status is not reflected on NN UI
> DN status is as below --
> hdfs dfsadmin -report
> Name: 10.xx.xx.xx1:50076 (host-xx1)
> Hostname: host-xx
> Decommission Status : Decommissioned
> Configured Capacity: 230501634048 (214.67 GB)
> DFS Used: 36864 (36 KB)
> Dead datanodes (1):
> Name: 10.xx.xx.xx2:50076 (host-xx2)
> Hostname: host-xx
> Decommission Status : Decommissioned
> Same is not reflected on NN UI.
> Attached NN UI snapshots for the same.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9357) NN UI is not showing which DN is "Decommissioned "and "Decommissioned & dead".
[ https://issues.apache.org/jira/browse/HDFS-9357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-9357: - Attachment: Decommissioned_Fixed.PNG Decommissioned_Dead_Fixed.PNG > NN UI is not showing which DN is "Decommissioned "and "Decommissioned & dead". > -- > > Key: HDFS-9357 > URL: https://issues.apache.org/jira/browse/HDFS-9357 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Archana T >Assignee: Surendra Singh Lilhore >Priority: Critical > Attachments: Decommissioned_Dead_Fixed.PNG, Decommissioned_Fixed.PNG, > HDFS-9357.001.patch, decommisioned_n_dead_.png, decommissioned_.png > > > NN UI is not showing which DN is "Decommissioned "and "Decommissioned & dead" > Root Cause -- > "Decommissioned" and "Decommissioned & dead" icon not reflected on NN UI > When DN is in Decommissioned status or in "Decommissioned & dead" status, > same status is not reflected on NN UI > DN status is as below -- > hdfs dfsadmin -report > Name: 10.xx.xx.xx1:50076 (host-xx1) > Hostname: host-xx > Decommission Status : Decommissioned > Configured Capacity: 230501634048 (214.67 GB) > DFS Used: 36864 (36 KB) > Dead datanodes (1): > Name: 10.xx.xx.xx2:50076 (host-xx2) > Hostname: host-xx > Decommission Status : Decommissioned > Same is not reflected on NN UI. > Attached NN UI snapshots for the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9308) Add truncateMeta() and deleteMeta() to MiniDFSCluster
[
https://issues.apache.org/jira/browse/HDFS-9308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987076#comment-14987076
]
Hudson commented on HDFS-9308:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2502 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2502/])
HDFS-9308. Add truncateMeta() and deleteMeta() to MiniDFSCluster. (Tony (lei:
rev 8e05dbf2bddce95d5f5a5bae5df61acabf0ba7c5)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRecovery.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestCrcCorruption.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
> Add truncateMeta() and deleteMeta() to MiniDFSCluster
> -
>
> Key: HDFS-9308
> URL: https://issues.apache.org/jira/browse/HDFS-9308
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: HDFS, test
>Affects Versions: 2.7.1
>Reporter: Tony Wu
>Assignee: Tony Wu
>Priority: Minor
> Fix For: 3.0.0, 2.8.0
>
> Attachments: HDFS-9308.001.patch, HDFS-9308.002.patch,
> HDFS-9308.003.patch
>
>
> HDFS-9188 introduced {{corruptMeta()}} method to make corrupting the metadata
> file filesystem agnostic. There should also be a {{truncateMeta()}} and
> {{deleteMeta()}} method in MiniDFSCluster to allow truncation of metadata
> files on DataNodes without writing code that's specific to underling file
> system. {{FsDatasetTestUtils#truncateMeta()}} is already implemented by
> HDFS-9188 and cam be exposed easily in {{MiniDFSCluster}}.
> This will be useful for tests such as
> {{TestLeaseRecovery#testBlockRecoveryWithLessMetafile}} and
> {{TestCrcCorruption#testCrcCorruption}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9313) Possible NullPointerException in BlockManager if no excess replica can be chosen
[
https://issues.apache.org/jira/browse/HDFS-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987075#comment-14987075
]
Hudson commented on HDFS-9313:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2502 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2502/])
HDFS-9313. Possible NullPointerException in BlockManager if no excess (mingma:
rev d565480da2f646b40c3180e1ccb2935c9863dfef)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
> Possible NullPointerException in BlockManager if no excess replica can be
> chosen
>
>
> Key: HDFS-9313
> URL: https://issues.apache.org/jira/browse/HDFS-9313
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Fix For: 2.8.0
>
> Attachments: HDFS-9313-2.patch, HDFS-9313.patch
>
>
> HDFS-8647 makes it easier to reason about various block placement scenarios.
> Here is one possible case where BlockManager won't be able to find the excess
> replica to delete: when storage policy changes around the same time balancer
> moves the block. When this happens, it will cause NullPointerException.
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:156)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseReplicasToDelete(BlockPlacementPolicyDefault.java:978)
> {noformat}
> Note that it isn't found in any production clusters. Instead, it is found
> from new unit tests. In addition, the issue has been there before HDFS-8647.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9275) Wait previous ErasureCodingWork to finish before schedule another one
[
https://issues.apache.org/jira/browse/HDFS-9275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987077#comment-14987077
]
Hudson commented on HDFS-9275:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2502 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2502/])
HDFS-9275. Wait previous ErasureCodingWork to finish before schedule (yliu: rev
5ba2b98d0fe29603e136fc43a14f853e820cf7e2)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/StripedFileTestUtil.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFSStriped.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteStripedFileWithFailure.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithMissingBlocks.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeModeWithStripedFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteReadStripedFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestRecoverStripedBlocks.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRecoverStripedFile.java
> Wait previous ErasureCodingWork to finish before schedule another one
> -
>
> Key: HDFS-9275
> URL: https://issues.apache.org/jira/browse/HDFS-9275
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Walter Su
>Assignee: Walter Su
> Fix For: 3.0.0
>
> Attachments: HDFS-9275.01.patch, HDFS-9275.02.patch,
> HDFS-9275.03.patch, HDFS-9275.04.patch, HDFS-9275.05.patch
>
>
> In {{ErasureCodingWorker}}, for the same block group, one task doesn't know
> which internal blocks is in recovering by other tasks. We could end up with
> recovering 2 identical block with same index. So, {{ReplicationMonitor}}
> should wait previous work to finish before schedule another one.
> This is related to the occasional failure of {{TestRecoverStripedFile}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9312) Fix TestReplication to be FsDataset-agnostic.
[
https://issues.apache.org/jira/browse/HDFS-9312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987078#comment-14987078
]
Hudson commented on HDFS-9312:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2502 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2502/])
HDFS-9312. Fix TestReplication to be FsDataset-agnostic. (lei) (lei: rev
7632409482aaf06ecc6fe370a9f519afb969ad30)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/FsDatasetTestUtils.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReplication.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImplTestUtils.java
> Fix TestReplication to be FsDataset-agnostic.
> -
>
> Key: HDFS-9312
> URL: https://issues.apache.org/jira/browse/HDFS-9312
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.7.1
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Fix For: 3.0.0, 2.8.0
>
> Attachments: HDFS-9312.00.patch, HDFS-9312.01.patch
>
>
> {{TestReplication}} uses raw file system access to inject dummy replica
> files. It makes {{TestReplication}} not compatible to non-fs dataset
> implementation.
> We can fix it by using existing {{FsDatasetTestUtils}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9328) Formalize coding standards for libhdfs++ and put them in a README.txt
[ https://issues.apache.org/jira/browse/HDFS-9328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987448#comment-14987448 ] James Clampffer commented on HDFS-9328: --- Right now Haohui wants to use google's C++ style guide with some extra bits from sun's style guide. I just want to have all the exceptions/extensions to google's guide in one place to reduce the amount of failed code reviews due to style issues. That way the reviewers have less patches to look at and contributors know how they should be doing things. Is there a common c++ style guide already used for hadoop projects? > Formalize coding standards for libhdfs++ and put them in a README.txt > - > > Key: HDFS-9328 > URL: https://issues.apache.org/jira/browse/HDFS-9328 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Haohui Mai >Priority: Blocker > > We have 2-3 people working on this project full time and hopefully more > people will start contributing. In order to efficiently scale we need a > single, easy to find, place where developers can check to make sure they are > following the coding standards of this project to both save their time and > save the time of people doing code reviews. > The most practical place to do this seems like a README file in libhdfspp/. > The foundation of the standards is google's C++ guide found here: > https://google-styleguide.googlecode.com/svn/trunk/cppguide.html > Any exceptions to google's standards or additional restrictions need to be > explicitly enumerated so there is one single point of reference for all > libhdfs++ code standards. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9367) Get libhdfs++ gmock tests running with CI
[ https://issues.apache.org/jira/browse/HDFS-9367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-9367: -- Assignee: Haohui Mai (was: James Clampffer) > Get libhdfs++ gmock tests running with CI > - > > Key: HDFS-9367 > URL: https://issues.apache.org/jira/browse/HDFS-9367 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Haohui Mai > > The gmock tests build with maven but there's no 'make test' target exposed to > maven/antrun. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7984) webhdfs:// needs to support provided delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-7984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] HeeSoo Kim updated HDFS-7984: - Attachment: HDFS-7984.006.patch > webhdfs:// needs to support provided delegation tokens > -- > > Key: HDFS-7984 > URL: https://issues.apache.org/jira/browse/HDFS-7984 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: HeeSoo Kim >Priority: Blocker > Attachments: HDFS-7984.001.patch, HDFS-7984.002.patch, > HDFS-7984.003.patch, HDFS-7984.004.patch, HDFS-7984.005.patch, > HDFS-7984.006.patch, HDFS-7984.patch > > > When using the webhdfs:// filesystem (especially from distcp), we need the > ability to inject a delegation token rather than webhdfs initialize its own. > This would allow for cross-authentication-zone file system accesses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[
https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987944#comment-14987944
]
Hadoop QA commented on HDFS-9289:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 8s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 5s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk cannot run
convertXmlToText from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 39s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 19s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 24s
{color} | {color:red} Patch generated 1 new checkstyle issues in
hadoop-hdfs-project (total was 247, now 247). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 22s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 29s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 2s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 38s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 1s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 22s
{color} | {color:red} Patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 178m 46s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits |
| | hadoop.hdfs.server.blockmanagement.TestNodeCount |
| JDK v1.7.0_79 Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock |
| | hadoop.hdfs.server.namenode.ha.TestDNFencing |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.TestFileCreationDelete |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
\\
\\
|| Subsystem
[jira] [Commented] (HDFS-9249) NPE thrown if an IOException is thrown in NameNode.
[
https://issues.apache.org/jira/browse/HDFS-9249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987840#comment-14987840
]
Yongjun Zhang commented on HDFS-9249:
-
BTW, it's a good find of yours about the try/catch code could run into the null
pointer situation. Thanks.
> NPE thrown if an IOException is thrown in NameNode.
> -
>
> Key: HDFS-9249
> URL: https://issues.apache.org/jira/browse/HDFS-9249
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Labels: supportability
> Attachments: HDFS-9249.001.patch, HDFS-9249.002.patch,
> HDFS-9249.003.patch, HDFS-9249.004.patch
>
>
> This issue was found when running test case
> TestBackupNode.testCheckpointNode, but upon closer look, the problem is not
> due to the test case.
> Looks like an IOException was thrown in
> try {
> initializeGenericKeys(conf, nsId, namenodeId);
> initialize(conf);
> try {
> haContext.writeLock();
> state.prepareToEnterState(haContext);
> state.enterState(haContext);
> } finally {
> haContext.writeUnlock();
> }
> causing the namenode to stop, but the namesystem was not yet properly
> instantiated, causing NPE.
> I tried to reproduce locally, but to no avail.
> Because I could not reproduce the bug, and the log does not indicate what
> caused the IOException, I suggest make this a supportability JIRA to log the
> exception for future improvement.
> Stacktrace
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getFSImage(NameNode.java:906)
> at org.apache.hadoop.hdfs.server.namenode.BackupNode.stop(BackupNode.java:210)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:827)
> at
> org.apache.hadoop.hdfs.server.namenode.BackupNode.(BackupNode.java:89)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1474)
> at
> org.apache.hadoop.hdfs.server.namenode.TestBackupNode.startBackupNode(TestBackupNode.java:102)
> at
> org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpoint(TestBackupNode.java:298)
> at
> org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpointNode(TestBackupNode.java:130)
> The last few lines of log:
> 2015-10-14 19:45:07,807 INFO namenode.NameNode
> (NameNode.java:createNameNode(1422)) - createNameNode [-checkpoint]
> 2015-10-14 19:45:07,807 INFO impl.MetricsSystemImpl
> (MetricsSystemImpl.java:init(158)) - CheckpointNode metrics system started
> (again)
> 2015-10-14 19:45:07,808 INFO namenode.NameNode
> (NameNode.java:setClientNamenodeAddress(402)) - fs.defaultFS is
> hdfs://localhost:37835
> 2015-10-14 19:45:07,808 INFO namenode.NameNode
> (NameNode.java:setClientNamenodeAddress(422)) - Clients are to use
> localhost:37835 to access this namenode/service.
> 2015-10-14 19:45:07,810 INFO hdfs.MiniDFSCluster
> (MiniDFSCluster.java:shutdown(1708)) - Shutting down the Mini HDFS Cluster
> 2015-10-14 19:45:07,810 INFO namenode.FSNamesystem
> (FSNamesystem.java:stopActiveServices(1298)) - Stopping services started for
> active state
> 2015-10-14 19:45:07,811 INFO namenode.FSEditLog
> (FSEditLog.java:endCurrentLogSegment(1228)) - Ending log segment 1
> 2015-10-14 19:45:07,811 INFO namenode.FSNamesystem
> (FSNamesystem.java:run(5306)) - NameNodeEditLogRoller was interrupted, exiting
> 2015-10-14 19:45:07,811 INFO namenode.FSEditLog
> (FSEditLog.java:printStatistics(703)) - Number of transactions: 3 Total time
> for transactions(ms): 0 Number of transactions batched in Syncs: 0 Number of
> syncs: 4 SyncTimes(ms): 2 1
> 2015-10-14 19:45:07,811 INFO namenode.FSNamesystem
> (FSNamesystem.java:run(5373)) - LazyPersistFileScrubber was interrupted,
> exiting
> 2015-10-14 19:45:07,822 INFO namenode.FileJournalManager
> (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file
> /data/jenkins/workspace/CDH5.5.0-Hadoop-HDFS-2.6.0/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/name1/current/edits_inprogress_001
> ->
> /data/jenkins/workspace/CDH5.5.0-Hadoop-HDFS-2.6.0/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/name1/current/edits_001-003
> 2015-10-14 19:45:07,835 INFO namenode.FileJournalManager
> (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file
> /data/jenkins/workspace/CDH5.5.0-Hadoop-HDFS-2.6.0/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/name2/current/edits_inprogress_001
> ->
>
[jira] [Updated] (HDFS-9364) Unnecessary DNS resolution attempts when creating NameNodeProxies
[ https://issues.apache.org/jira/browse/HDFS-9364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-9364: Summary: Unnecessary DNS resolution attempts when creating NameNodeProxies (was: Unnecessary DNS resolution attempts when NameNodeProxies) > Unnecessary DNS resolution attempts when creating NameNodeProxies > - > > Key: HDFS-9364 > URL: https://issues.apache.org/jira/browse/HDFS-9364 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HDFS-9364.001.patch > > > When creating NameNodeProxies, we always try to DNS-resolve namenode URIs. > This is unnecessary if the URI is logical, and may be significantly slow if > the DNS is having problems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9351) No checkNNStartup() is called when fsck calls FSNamesystem.getSnapshottableDirs()
[
https://issues.apache.org/jira/browse/HDFS-9351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987727#comment-14987727
]
Yongjun Zhang commented on HDFS-9351:
-
Thanks [~xiaochen] for the patch, +1 and will commit soon.
> No checkNNStartup() is called when fsck calls
> FSNamesystem.getSnapshottableDirs()
> -
>
> Key: HDFS-9351
> URL: https://issues.apache.org/jira/browse/HDFS-9351
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Reporter: Yongjun Zhang
>Assignee: Xiao Chen
> Attachments: HDFS-9351.patch
>
>
> I looked further at HDFS-9231 change I reviewed earlier, and realize I missed
> one thing.
> The patch changed
> {code}
> if (snapshottableDirs != null) {
> SnapshottableDirectoryStatus[] snapshotDirs = namenode.getRpcServer()
> .getSnapshottableDirListing();
> if (snapshotDirs != null) {
> for (SnapshottableDirectoryStatus dir : snapshotDirs) {
> snapshottableDirs.add(dir.getFullPath().toString());
> }
> }
> }
> {code}
> to
> {code}
> if (snapshottableDirs != null) {
> snapshottableDirs = namenode.getNamesystem().getSnapshottableDirs();
> }
> {code}
> In old code, namenode.getRpcServer().getSnapshottableDirListing() calls
> checkNNStartup(), however, this is not done with the new code. This is a hole
> of the earlier HDFS-9231 patch.
> Create this jira to fix the hole. Suggest to revert this portion of the
> change, and add code to do what we need at NamenodeFsck.
> Sorry for missing this in my earlier review of HDFS-9231.
> Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8820) Enable RPC Congestion control by default
[ https://issues.apache.org/jira/browse/HDFS-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987848#comment-14987848 ] Arpit Agarwal commented on HDFS-8820: - Thanks for the feedback [~mingma]. I didn't get a chance to look at this again until now. bq. Thanks Arpit Agarwal. Should we enable this for communication between DN and NN? Good point, this should not be enabled for the service RPC port. I'll fix that part of the patch. bq. For the configuration part, I wonder if we should use the pattern similar to RPC's setProtocolEngine,or ipc.server.read.threadpool.size where NN or other services can call RPC.Builder#setnumReaders to override the value. In that way, the NN doesn't need to know the format of the configuration key name. Agreed that would be best. However I am not sure changing the RPCEngine interface will be a compatible change. > Enable RPC Congestion control by default > > > Key: HDFS-8820 > URL: https://issues.apache.org/jira/browse/HDFS-8820 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-8820.01.patch, HDFS-8820.02.patch > > > We propose enabling RPC congestion control introduced by HADOOP-10597 by > default. > We enabled it on a couple of large clusters a few weeks ago and it has helped > keep the namenodes responsive under load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8820) Simplify enabling NameNode RPC congestion control and FairCallQueue
[ https://issues.apache.org/jira/browse/HDFS-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-8820: Summary: Simplify enabling NameNode RPC congestion control and FairCallQueue (was: Enable RPC Congestion control by default) > Simplify enabling NameNode RPC congestion control and FairCallQueue > --- > > Key: HDFS-8820 > URL: https://issues.apache.org/jira/browse/HDFS-8820 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-8820.01.patch, HDFS-8820.02.patch > > > We propose enabling RPC congestion control introduced by HADOOP-10597 by > default. > We enabled it on a couple of large clusters a few weeks ago and it has helped > keep the namenodes responsive under load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9351) No checkNNStartup() is called when fsck calls FSNamesystem.getSnapshottableDirs()
[
https://issues.apache.org/jira/browse/HDFS-9351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987757#comment-14987757
]
Xiao Chen commented on HDFS-9351:
-
Thanks Yongjun!
> No checkNNStartup() is called when fsck calls
> FSNamesystem.getSnapshottableDirs()
> -
>
> Key: HDFS-9351
> URL: https://issues.apache.org/jira/browse/HDFS-9351
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Reporter: Yongjun Zhang
>Assignee: Xiao Chen
> Attachments: HDFS-9351.patch
>
>
> I looked further at HDFS-9231 change I reviewed earlier, and realize I missed
> one thing.
> The patch changed
> {code}
> if (snapshottableDirs != null) {
> SnapshottableDirectoryStatus[] snapshotDirs = namenode.getRpcServer()
> .getSnapshottableDirListing();
> if (snapshotDirs != null) {
> for (SnapshottableDirectoryStatus dir : snapshotDirs) {
> snapshottableDirs.add(dir.getFullPath().toString());
> }
> }
> }
> {code}
> to
> {code}
> if (snapshottableDirs != null) {
> snapshottableDirs = namenode.getNamesystem().getSnapshottableDirs();
> }
> {code}
> In old code, namenode.getRpcServer().getSnapshottableDirListing() calls
> checkNNStartup(), however, this is not done with the new code. This is a hole
> of the earlier HDFS-9231 patch.
> Create this jira to fix the hole. Suggest to revert this portion of the
> change, and add code to do what we need at NamenodeFsck.
> Sorry for missing this in my earlier review of HDFS-9231.
> Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9249) NPE thrown if an IOException is thrown in NameNode.
[
https://issues.apache.org/jira/browse/HDFS-9249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987826#comment-14987826
]
Yongjun Zhang commented on HDFS-9249:
-
Hi [~jojochuang],
Thanks for the explanation. So {{namesystem}} is {{null}} only when the
constructor failed, so the way you check in rev 003 makes sense to me.
I think we can merge 003 (we still check whether {{namesystem}} is {{null}}
in {{stop}}) and 004 (call {{stop}} and report exception if there is any)
together with the following changes:
{code}
private void stopAtException(Exception e) {
try {
this.stop();
} catch (Exception ex) {
LOG.warn("Encountered exception when handling exception ("
+ e.getMessage() + "):", ex);
}
}
..
} catch (IOException e) {
stopAtException(e);
throw e;
} catch (HadoopIllegalArgumentException e) {
stopAtException(e);
throw e;
}
{code}
Notice that above rev 004, there is another call to {[this.stop()}} right below
the one you modified, so my suggested change included both.
Thanks.
> NPE thrown if an IOException is thrown in NameNode.
> -
>
> Key: HDFS-9249
> URL: https://issues.apache.org/jira/browse/HDFS-9249
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Labels: supportability
> Attachments: HDFS-9249.001.patch, HDFS-9249.002.patch,
> HDFS-9249.003.patch, HDFS-9249.004.patch
>
>
> This issue was found when running test case
> TestBackupNode.testCheckpointNode, but upon closer look, the problem is not
> due to the test case.
> Looks like an IOException was thrown in
> try {
> initializeGenericKeys(conf, nsId, namenodeId);
> initialize(conf);
> try {
> haContext.writeLock();
> state.prepareToEnterState(haContext);
> state.enterState(haContext);
> } finally {
> haContext.writeUnlock();
> }
> causing the namenode to stop, but the namesystem was not yet properly
> instantiated, causing NPE.
> I tried to reproduce locally, but to no avail.
> Because I could not reproduce the bug, and the log does not indicate what
> caused the IOException, I suggest make this a supportability JIRA to log the
> exception for future improvement.
> Stacktrace
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getFSImage(NameNode.java:906)
> at org.apache.hadoop.hdfs.server.namenode.BackupNode.stop(BackupNode.java:210)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:827)
> at
> org.apache.hadoop.hdfs.server.namenode.BackupNode.(BackupNode.java:89)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1474)
> at
> org.apache.hadoop.hdfs.server.namenode.TestBackupNode.startBackupNode(TestBackupNode.java:102)
> at
> org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpoint(TestBackupNode.java:298)
> at
> org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpointNode(TestBackupNode.java:130)
> The last few lines of log:
> 2015-10-14 19:45:07,807 INFO namenode.NameNode
> (NameNode.java:createNameNode(1422)) - createNameNode [-checkpoint]
> 2015-10-14 19:45:07,807 INFO impl.MetricsSystemImpl
> (MetricsSystemImpl.java:init(158)) - CheckpointNode metrics system started
> (again)
> 2015-10-14 19:45:07,808 INFO namenode.NameNode
> (NameNode.java:setClientNamenodeAddress(402)) - fs.defaultFS is
> hdfs://localhost:37835
> 2015-10-14 19:45:07,808 INFO namenode.NameNode
> (NameNode.java:setClientNamenodeAddress(422)) - Clients are to use
> localhost:37835 to access this namenode/service.
> 2015-10-14 19:45:07,810 INFO hdfs.MiniDFSCluster
> (MiniDFSCluster.java:shutdown(1708)) - Shutting down the Mini HDFS Cluster
> 2015-10-14 19:45:07,810 INFO namenode.FSNamesystem
> (FSNamesystem.java:stopActiveServices(1298)) - Stopping services started for
> active state
> 2015-10-14 19:45:07,811 INFO namenode.FSEditLog
> (FSEditLog.java:endCurrentLogSegment(1228)) - Ending log segment 1
> 2015-10-14 19:45:07,811 INFO namenode.FSNamesystem
> (FSNamesystem.java:run(5306)) - NameNodeEditLogRoller was interrupted, exiting
> 2015-10-14 19:45:07,811 INFO namenode.FSEditLog
> (FSEditLog.java:printStatistics(703)) - Number of transactions: 3 Total time
> for transactions(ms): 0 Number of transactions batched in Syncs: 0 Number of
> syncs: 4 SyncTimes(ms): 2 1
> 2015-10-14 19:45:07,811 INFO namenode.FSNamesystem
> (FSNamesystem.java:run(5373)) - LazyPersistFileScrubber was interrupted,
> exiting
> 2015-10-14 19:45:07,822 INFO namenode.FileJournalManager
> (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing
[jira] [Updated] (HDFS-8820) Simplify enabling NameNode RPC congestion control and FairCallQueue
[ https://issues.apache.org/jira/browse/HDFS-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-8820: Description: Enabling RPC Congestion control and FairCallQueue settings can be simplified with HDFS-specific configuration keys. Currently the configuration requires knowing the exact RPC port number and also whether the service RPC port is enabled or not separately. If a separate service RPC endpoint is not defined then RPC congestion control must be enabled. (was: We propose enabling RPC congestion control introduced by HADOOP-10597 by default. We enabled it on a couple of large clusters a few weeks ago and it has helped keep the namenodes responsive under load.) > Simplify enabling NameNode RPC congestion control and FairCallQueue > --- > > Key: HDFS-8820 > URL: https://issues.apache.org/jira/browse/HDFS-8820 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-8820.01.patch, HDFS-8820.02.patch > > > Enabling RPC Congestion control and FairCallQueue settings can be simplified > with HDFS-specific configuration keys. Currently the configuration requires > knowing the exact RPC port number and also whether the service RPC port is > enabled or not separately. If a separate service RPC endpoint is not defined > then RPC congestion control must be enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8820) Simplify enabling NameNode RPC congestion control and FairCallQueue
[ https://issues.apache.org/jira/browse/HDFS-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-8820: Description: Enabling RPC Congestion control and FairCallQueue settings can be simplified with HDFS-specific configuration keys. Currently the configuration requires knowing the exact RPC port number and also whether the service RPC port is enabled or not separately. If a separate service RPC endpoint is not defined then RPC congestion control must be enabled ([see comment|https://issues.apache.org/jira/browse/HDFS-8820?focusedCommentId=14987848=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14987848] from [~mingma] below. (was: Enabling RPC Congestion control and FairCallQueue settings can be simplified with HDFS-specific configuration keys. Currently the configuration requires knowing the exact RPC port number and also whether the service RPC port is enabled or not separately. If a separate service RPC endpoint is not defined then RPC congestion control must be enabled. ) > Simplify enabling NameNode RPC congestion control and FairCallQueue > --- > > Key: HDFS-8820 > URL: https://issues.apache.org/jira/browse/HDFS-8820 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-8820.01.patch, HDFS-8820.02.patch > > > Enabling RPC Congestion control and FairCallQueue settings can be simplified > with HDFS-specific configuration keys. Currently the configuration requires > knowing the exact RPC port number and also whether the service RPC port is > enabled or not separately. If a separate service RPC endpoint is not defined > then RPC congestion control must be enabled ([see > comment|https://issues.apache.org/jira/browse/HDFS-8820?focusedCommentId=14987848=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14987848] > from [~mingma] below. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9198) Coalesce IBR processing in the NN
[
https://issues.apache.org/jira/browse/HDFS-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987804#comment-14987804
]
Hadoop QA commented on HDFS-9198:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 7s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 3s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk cannot run
convertXmlToText from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 56s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 16s
{color} | {color:red} Patch generated 5 new checkstyle issues in
hadoop-hdfs-project/hadoop-hdfs (total was 410, now 410). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 59s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 1s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 45s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 20s
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 143m 55s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_60 Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestNodeCount |
| | hadoop.hdfs.server.namenode.TestNNThroughputBenchmark |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
| | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| JDK v1.7.0_79 Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestNodeCount |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.namenode.TestNNThroughputBenchmark |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.1 Server=1.7.1
Image:test-patch-base-hadoop-date2015-11-03 |
| JIRA Patch URL |
[jira] [Commented] (HDFS-9362) TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings, fails on Windows.
[
https://issues.apache.org/jira/browse/HDFS-9362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987846#comment-14987846
]
Chris Nauroth commented on HDFS-9362:
-
Something is going wrong with the Findbugs pre-check. I'll need to follow up
on that separately.
I cannot reproduce any of the test failures, and they are not related to the
code changed in this patch.
The license check warning is a known issue with HDFS tests writing output files
outside of the Maven build root. This is tracked elsewhere.
> TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings,
> fails on Windows.
> ---
>
> Key: HDFS-9362
> URL: https://issues.apache.org/jira/browse/HDFS-9362
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-9362.001.patch
>
>
> {{TestAuditLogger#testAuditLoggerWithCallContext}} was added recently to
> exercise the new audit logging with caller context functionality. The tests
> assume Unix line endings by hard-coding "\n" in asserts. These tests fail on
> Windows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9354) Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows
[ https://issues.apache.org/jira/browse/HDFS-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987845#comment-14987845 ] Chris Nauroth commented on HDFS-9354: - Something is going wrong with the Findbugs pre-check. I'll need to follow up on that separately. I cannot reproduce any of the test failures, and they are not related to the code changed in this patch. The license check warning is a known issue with HDFS tests writing output files outside of the Maven build root. This is tracked elsewhere. > Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows > -- > > Key: HDFS-9354 > URL: https://issues.apache.org/jira/browse/HDFS-9354 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-9354.00.patch, HDFS-9354.01.patch > > > This negative test expect HadoopIllegalArgumentException on illegal > configuration. It uses JUnit (expected=HadoopIllegalArgumentException.class) > and passed fine on Linux. > On windows, this test passes as well. But it left open handles on NN metadata > directories used by MiniDFSCluster. As a result, quite a few of subsequent > TestBalancer unit tests can't start MiniDFSCluster. The open handles prevents > them from cleaning up NN metadata directories on Windows. > This JIRA is opened to explicitly catch the Exception and ensure the test > cluster is properly shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[
https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987673#comment-14987673
]
Zhe Zhang commented on HDFS-9289:
-
Thanks Chang. +1 on the latest patch pending Jenkins.
I'll file a follow-on JIRA to cleanup {{addBlockToFile}} a bit. Maybe we can
consolidate {{numStripes}} and {{len}}.
> Make DataStreamer#block thread safe and verify genStamp in commitBlock
> --
>
> Key: HDFS-9289
> URL: https://issues.apache.org/jira/browse/HDFS-9289
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Chang Li
>Assignee: Chang Li
>Priority: Critical
> Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch,
> HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch
>
>
> we have seen a case of corrupt block which is caused by file complete after a
> pipelineUpdate, but the file complete with the old block genStamp. This
> caused the replicas of two datanodes in updated pipeline to be viewed as
> corrupte. Propose to check genstamp when commit block
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9354) Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows
[ https://issues.apache.org/jira/browse/HDFS-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988226#comment-14988226 ] Hudson commented on HDFS-9354: -- SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #633 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/633/]) HDFS-9354. Fix TestBalancer#testBalancerWithZeroThreadsForMove on (cnauroth: rev 095ac834022df6136b42961c507ec745c6cf8f97) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows > -- > > Key: HDFS-9354 > URL: https://issues.apache.org/jira/browse/HDFS-9354 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Fix For: 2.8.0 > > Attachments: HDFS-9354.00.patch, HDFS-9354.01.patch > > > This negative test expect HadoopIllegalArgumentException on illegal > configuration. It uses JUnit (expected=HadoopIllegalArgumentException.class) > and passed fine on Linux. > On windows, this test passes as well. But it left open handles on NN metadata > directories used by MiniDFSCluster. As a result, quite a few of subsequent > TestBalancer unit tests can't start MiniDFSCluster. The open handles prevents > them from cleaning up NN metadata directories on Windows. > This JIRA is opened to explicitly catch the Exception and ensure the test > cluster is properly shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9362) TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings, fails on Windows.
[
https://issues.apache.org/jira/browse/HDFS-9362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988225#comment-14988225
]
Hudson commented on HDFS-9362:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #633 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/633/])
HDFS-9362. TestAuditLogger#testAuditLoggerWithCallContext assumes Unix
(cnauroth: rev 7e2829662b4c4bf33ebaf2fa09312d0bed3d6f92)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLogger.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings,
> fails on Windows.
> ---
>
> Key: HDFS-9362
> URL: https://issues.apache.org/jira/browse/HDFS-9362
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9362.001.patch
>
>
> {{TestAuditLogger#testAuditLoggerWithCallContext}} was added recently to
> exercise the new audit logging with caller context functionality. The tests
> assume Unix line endings by hard-coding "\n" in asserts. These tests fail on
> Windows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-9289: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Fixing the JIRA for 3.0 for now. Will reopen and update fix version once branch-2 patch is done. > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 3.0.0 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-9289: Fix Version/s: (was: 3.0.0) 2.8.0 > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 2.8.0 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988292#comment-14988292 ] Zhe Zhang commented on HDFS-9289: - Thanks Chang. I just committed to branch-2 as well. Changed fix version to 2.8.0. > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 2.8.0 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988304#comment-14988304 ] Hudson commented on HDFS-9289: -- FAILURE: Integrated in Hadoop-trunk-Commit #8750 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8750/]) HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in (zhz: rev dac0463a4e20dfb3a802355919fc22b8e017a4e1) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockWithInvalidGenStamp.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaWithStripedBlocks.java > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 2.8.0 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9354) Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows
[ https://issues.apache.org/jira/browse/HDFS-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988336#comment-14988336 ] Xiaoyu Yao commented on HDFS-9354: -- Thank you, [~cnauroth] for reviewing and committing the patch! > Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows > -- > > Key: HDFS-9354 > URL: https://issues.apache.org/jira/browse/HDFS-9354 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Fix For: 2.8.0 > > Attachments: HDFS-9354.00.patch, HDFS-9354.01.patch > > > This negative test expect HadoopIllegalArgumentException on illegal > configuration. It uses JUnit (expected=HadoopIllegalArgumentException.class) > and passed fine on Linux. > On windows, this test passes as well. But it left open handles on NN metadata > directories used by MiniDFSCluster. As a result, quite a few of subsequent > TestBalancer unit tests can't start MiniDFSCluster. The open handles prevents > them from cleaning up NN metadata directories on Windows. > This JIRA is opened to explicitly catch the Exception and ensure the test > cluster is properly shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9369) Use ctest to run tests for hadoop-hdfs-native-client
[
https://issues.apache.org/jira/browse/HDFS-9369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988388#comment-14988388
]
Hadoop QA commented on HDFS-9369:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
4s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK
v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 53s
{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK
v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 24s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.1 Server=1.7.1
Image:test-patch-base-hadoop-date2015-11-03 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12770407/HDFS-9369.000.patch |
| JIRA Issue | HDFS-9369 |
| Optional Tests | asflicense javac javadoc mvninstall unit xml compile
cc |
| uname | Linux eee2e9392894 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-1a9afee/precommit/personality/hadoop.sh
|
| git revision | trunk / dac0463 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_60
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 |
| JDK v1.7.0_79 Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/13366/testReport/
[jira] [Comment Edited] (HDFS-4937) ReplicationMonitor can infinite-loop in BlockPlacementPolicyDefault#chooseRandom()
[
https://issues.apache.org/jira/browse/HDFS-4937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988363#comment-14988363
]
Kihwal Lee edited comment on HDFS-4937 at 11/3/15 11:14 PM:
Actually the patch I wrote 2 years ago and the rebased one are correct. My
subsequent attempt to "improve" it was based on my recent incorrect
understanding of the patch. Now I remember why I did that way.
After a sufficient number of random, potentially duplicate picking is tried,
the total candidate node count is refreshed. The refreshed number will not
include what is already tried and excluded, so it is truly the remaining
candidate node count based on the list of nodes that it already tried and the
latest network topology. The loop will continue until all candidate nodes are
exhausted or enough number of replicas are picked.
Resubmitting v1, the rebased original patch.
was (Author: kihwal):
Actually the patch I wrote 2 years ago and the rebased one are correct. My
subsequent attempt to "improve" it was based on my recent incorrect
misunderstanding of the patch. Now I remember why I did that way.
After a sufficient number of random, potentially duplicate picking is tried,
the total candidate node count is refreshed. The refreshed number will not
include what is already tried and excluded, so it is truly the remaining
candidate node count based on the list of nodes that it already tried and the
latest network topology. The loop will continue until all candidate nodes are
exhausted or enough number of replicas are picked.
Resubmitting v1, the rebased original patch.
> ReplicationMonitor can infinite-loop in
> BlockPlacementPolicyDefault#chooseRandom()
> --
>
> Key: HDFS-4937
> URL: https://issues.apache.org/jira/browse/HDFS-4937
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.0.4-alpha, 0.23.8
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Labels: BB2015-05-TBR
> Attachments: HDFS-4937.patch, HDFS-4937.v1.patch, HDFS-4937.v1.patch,
> HDFS-4937.v2.patch, HDFS-4937.v3.patch
>
>
> When a large number of nodes are removed by refreshing node lists, the
> network topology is updated. If the refresh happens at the right moment, the
> replication monitor thread may stuck in the while loop of {{chooseRandom()}}.
> This is because the cached cluster size is used in the terminal condition
> check of the loop. This usually happens when a block with a high replication
> factor is being processed. Since replicas/rack is also calculated beforehand,
> no node choice may satisfy the goodness criteria if refreshing removed racks.
> All nodes will end up in the excluded list, but the size will still be less
> than the cached cluster size, so it will loop infinitely. This was observed
> in a production environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-9289: Affects Version/s: 2.7.1 > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9362) TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings, fails on Windows.
[
https://issues.apache.org/jira/browse/HDFS-9362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988247#comment-14988247
]
Hudson commented on HDFS-9362:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #1356 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1356/])
HDFS-9362. TestAuditLogger#testAuditLoggerWithCallContext assumes Unix
(cnauroth: rev 7e2829662b4c4bf33ebaf2fa09312d0bed3d6f92)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLogger.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings,
> fails on Windows.
> ---
>
> Key: HDFS-9362
> URL: https://issues.apache.org/jira/browse/HDFS-9362
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9362.001.patch
>
>
> {{TestAuditLogger#testAuditLoggerWithCallContext}} was added recently to
> exercise the new audit logging with caller context functionality. The tests
> assume Unix line endings by hard-coding "\n" in asserts. These tests fail on
> Windows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9354) Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows
[ https://issues.apache.org/jira/browse/HDFS-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988248#comment-14988248 ] Hudson commented on HDFS-9354: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1356 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1356/]) HDFS-9354. Fix TestBalancer#testBalancerWithZeroThreadsForMove on (cnauroth: rev 095ac834022df6136b42961c507ec745c6cf8f97) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows > -- > > Key: HDFS-9354 > URL: https://issues.apache.org/jira/browse/HDFS-9354 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Fix For: 2.8.0 > > Attachments: HDFS-9354.00.patch, HDFS-9354.01.patch > > > This negative test expect HadoopIllegalArgumentException on illegal > configuration. It uses JUnit (expected=HadoopIllegalArgumentException.class) > and passed fine on Linux. > On windows, this test passes as well. But it left open handles on NN metadata > directories used by MiniDFSCluster. As a result, quite a few of subsequent > TestBalancer unit tests can't start MiniDFSCluster. The open handles prevents > them from cleaning up NN metadata directories on Windows. > This JIRA is opened to explicitly catch the Exception and ensure the test > cluster is properly shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9260) Improve performance and GC friendliness of startup and FBRs
[ https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HDFS-9260: -- Attachment: FBR processing.png I've read the doc now. Sorry I commented before doing so. The results are interesting until the the final details about a 4x reduction in block updates. Here are some basic specs to consider: * 10-80k adds/min * job submissions increasing replication factor to 10 * at least 1 node/day decommissioning or going dead with 100k-400k blocks * every few weeks entire racks (40 nodes) are decommissioned for refresh or reallocation * balancer is constantly churning to populate recommissioned dead nodes That's a lot of IBRs which is why a 4x degradation is quite concerning. The block report processing times seem a bit high in the tests. :) I'll attach an image of the BR processing times for some of our busiest clusters. They span the gamut from 100M-300M blocks with roughly the same number of files. We got a huge improvement from my BR encoding change + per-storage reports. BTW, I had/have a working patch that replaced the triplets with sparse yet densely packed 2-dimensional primitive arrays. Everything is linked via indices to a greatly reduce the dirty cards to scan. Need to dig up the jira when my head is above water. > Improve performance and GC friendliness of startup and FBRs > --- > > Key: HDFS-9260 > URL: https://issues.apache.org/jira/browse/HDFS-9260 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, performance >Affects Versions: 2.7.1 >Reporter: Staffan Friberg >Assignee: Staffan Friberg > Attachments: FBR processing.png, HDFS Block and Replica Management > 20151013.pdf, HDFS-7435.001.patch, HDFS-7435.002.patch, HDFS-7435.003.patch, > HDFS-7435.004.patch, HDFS-7435.005.patch, HDFS-7435.006.patch, > HDFS-7435.007.patch, HDFS-9260.008.patch, HDFS-9260.009.patch > > > This patch changes the datastructures used for BlockInfos and Replicas to > keep them sorted. This allows faster and more GC friendly handling of full > block reports. > Would like to hear peoples feedback on this change and also some help > investigating/understanding a few outstanding issues if we are interested in > moving forward with this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988185#comment-14988185 ] Zhe Zhang commented on HDFS-9289: - Thanks Change. I just committed the patch to trunk. Do you mind creating a patch for branch-2? The main conflict is on the test. > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9369) Use ctest to run tests for hadoop-hdfs-native-client
[
https://issues.apache.org/jira/browse/HDFS-9369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988241#comment-14988241
]
Haohui Mai commented on HDFS-9369:
--
It's okay to redirect the output to a file.
[~aw], do you have concrete ideas on which file that ctest should store its
output to in order to be integrated with Yetus?
> Use ctest to run tests for hadoop-hdfs-native-client
>
>
> Key: HDFS-9369
> URL: https://issues.apache.org/jira/browse/HDFS-9369
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>Priority: Minor
> Attachments: HDFS-9369.000.patch
>
>
> Currently we write special rules in {{pom.xml}} to run tests in
> {{hadoop-hdfs-native-client}}. This jira proposes to run these tests using
> ctest to simplify {{pom.xml}} and improve portability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9369) Use ctest to run tests for hadoop-hdfs-native-client
[
https://issues.apache.org/jira/browse/HDFS-9369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988256#comment-14988256
]
Allen Wittenauer commented on HDFS-9369:
The output should be dumped into a directory in target. We haven't written a
ctest test format plug-in yet, but I suspect it will work like TAP does: give
the plug-in the directory where ctest output lives and it'll get processed as
appropriate.
> Use ctest to run tests for hadoop-hdfs-native-client
>
>
> Key: HDFS-9369
> URL: https://issues.apache.org/jira/browse/HDFS-9369
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>Priority: Minor
> Attachments: HDFS-9369.000.patch
>
>
> Currently we write special rules in {{pom.xml}} to run tests in
> {{hadoop-hdfs-native-client}}. This jira proposes to run these tests using
> ctest to simplify {{pom.xml}} and improve portability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9103) Retry reads on DN failure
[ https://issues.apache.org/jira/browse/HDFS-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988349#comment-14988349 ] James Clampffer commented on HDFS-9103: --- Thanks for the feedback Haohui! "The name_match function can be simplified. It looks like the comments are unnecessary." Simplified with a ternary expression or similar? I think I can get rid of that whole block of code if I change Entry to something that makes sense to have in a set. I disagree about removing comments; they aren't really making things cluttered and make it easier understand the code quickly. "This function is on the critical path thus the result should be cached" I considered this; I wanted to profile it with a handful of dead datanodes first before adding any complexity. I'll add caching now (ideally to something that doesn't need to grab the lock). "Looks like this is a test-only function. Let's name it "TEST_Clear()" for clarity." Right now it's test only. I figured eventually it wouldn't hurt to expose that in the C++ API but for now I'll rename to make things clear. "The timeout settings needs to be passed in through the Option object. timeout_duration_ can be marked as const." I totally missed this. I was waiting on Bob's configuration parser and forgot that this was already available in the Options object. "The clock needs to be a steady_clock for monotonicity" Nice catch, didn't think about that. "Are there any reasons of using a set instead of a vector / map? Using vector has much better cache locality results in this case." I was considering swapping out the std::pair for a struct that defined operator== and operator< based on node id so that the set semantics would make sense. I should have done that for this patch, I'll add that to the next. The set should be sitting in cache anyway if it's reasonably sized and accessed recently but if it turns cache misses are leading to pipeline stalls I'll certainly swap it out for a vector. "The above code can be simplified." Will do. "The function can be placed in the InputStream class." Good idea. That gets rid of that block of code that's duplicated in the gmock test as well. I'll get a patch up early tomorrow morning to address all of this. As far as node exclusion policy goes is it reasonable to assume that kResourceUnavailable will always refer to something local like too many open sockets? I wouldn't want to prevent exclusion if it's the DN sending that. > Retry reads on DN failure > - > > Key: HDFS-9103 > URL: https://issues.apache.org/jira/browse/HDFS-9103 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: James Clampffer > Fix For: HDFS-8707 > > Attachments: HDFS-9103.1.patch, HDFS-9103.2.patch, > HDFS-9103.HDFS-8707.006.patch, HDFS-9103.HDFS-8707.3.patch, > HDFS-9103.HDFS-8707.4.patch, HDFS-9103.HDFS-8707.5.patch > > > When AsyncPreadSome fails, add the failed DataNode to the excluded list and > try again. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9370) TestDataNodeUGIProvider fails intermittently due to non-deterministic cache expiry.
Chris Nauroth created HDFS-9370:
---
Summary: TestDataNodeUGIProvider fails intermittently due to
non-deterministic cache expiry.
Key: HDFS-9370
URL: https://issues.apache.org/jira/browse/HDFS-9370
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
{{TestDataNodeUGIProvider}} has hard-coded sleep times waiting for background
expiration of entries in a Guava cache. I have seen this test suite fail
intermittently, because expiration is not guaranteed to happen strictly on the
boundary of the period defined by the cache's expiration time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988093#comment-14988093 ] Chang Li commented on HDFS-9289: Thanks [~zhz] for further review! I have test those above failed unit tests above on my own machine with my patch on, and those test passed. They are not related to the changes of my patch. > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9103) Retry reads on DN failure
[
https://issues.apache.org/jira/browse/HDFS-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987971#comment-14987971
]
Haohui Mai commented on HDFS-9103:
--
Thanks for updating the patch.
{code}
+ auto name_match = [dn](const Entry& entry) {
+if (dn == entry.first) {
+ return true;
+}
+return false;
+ };
+ auto entry = std::find_if(datanodes_.begin(), datanodes_.end(), name_match);
+
+ if (entry != datanodes_.end()) {
+/* if entry already exists remove it so one with a new timestamp can be
+ * added */
+datanodes_.erase(entry);
+ }
+
+ /* insert a new entry with bad node and current time */
+ datanodes_.insert(Entry(std::string(dn), Clock::now()));
{code}
The {{name_match}} function can be simplified. It looks like the comments are
unnecessary.
{code}
+std::set BadDataNodeTracker::GetNodesToExclude() {
{code}
This function is on the critical path thus the result should be cached.
{code}
+void BadDataNodeTracker::Clear() {
+ std::lock_guard update_lock(datanodes_update_lock_);
+ datanodes_.clear();
+}
{code}
Looks like this is a test-only function. Let's name it "TEST_Clear()" for
clarity.
{code}
+ BadDataNodeTracker() : timeout_duration_(10 * 60) {}
+ unsigned int timeout_duration_; /* in seconds */
{code}
The timeout settings needs to be passed in through the {{Option}} object.
{{timeout_duration_}} can be marked as const.
{code}
+ typedef std::chrono::system_clock Clock;
+ typedef std::pair Entry;
+ std::set datanodes_;
{code}
The clock needs to be a {{steady_clock}} for monotonicity. Are there any
reasons of using a {{set}} instead of a {{vector}} / {{map}}? Using
{{vector}} has much better cache locality results in this case.
{code}
if (!s.ok()) {
+/* determine if DN gets marked bad */
+if (ShouldExclude(s)) {
+ bad_datanodes_->AddBadNode(contacted_datanode);
+}
+// resource might free up
+case Status::kResourceUnavailable:
+ return false;
{code}
The above code can be simplified.
{code}
+static bool ShouldExclude(const Status ) {
+ if (s.ok()) {
+return false;
+ }
+
+ switch (s.code()) {
+// resource might free up
+case Status::kResourceUnavailable:
+ return false;
+case Status::kInvalidArgument:
+case Status::kUnimplemented:
+case Status::kException:
+default:
+ return true;
+ }
+ return true;
+}
{code}
The function can be placed in the {{InputStream}} class.
> Retry reads on DN failure
> -
>
> Key: HDFS-9103
> URL: https://issues.apache.org/jira/browse/HDFS-9103
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Bob Hansen
>Assignee: James Clampffer
> Fix For: HDFS-8707
>
> Attachments: HDFS-9103.1.patch, HDFS-9103.2.patch,
> HDFS-9103.HDFS-8707.006.patch, HDFS-9103.HDFS-8707.3.patch,
> HDFS-9103.HDFS-8707.4.patch, HDFS-9103.HDFS-8707.5.patch
>
>
> When AsyncPreadSome fails, add the failed DataNode to the excluded list and
> try again.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7984) webhdfs:// needs to support provided delegation tokens
[
https://issues.apache.org/jira/browse/HDFS-7984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987995#comment-14987995
]
Hadoop QA commented on HDFS-7984:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 6s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 26s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 10s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 48s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk cannot run
convertXmlToText from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 17s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 10s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 15s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 15s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 15s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 1s
{color} | {color:red} Patch generated 1 new checkstyle issues in root (total
was 465, now 465). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 8s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 23s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_60.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 50m 19s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 50s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 58s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_79.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 50m 5s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 21s
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 165m 30s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK
[jira] [Commented] (HDFS-9362) TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings, fails on Windows.
[
https://issues.apache.org/jira/browse/HDFS-9362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988124#comment-14988124
]
Hudson commented on HDFS-9362:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8749 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8749/])
HDFS-9362. TestAuditLogger#testAuditLoggerWithCallContext assumes Unix
(cnauroth: rev 7e2829662b4c4bf33ebaf2fa09312d0bed3d6f92)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLogger.java
> TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings,
> fails on Windows.
> ---
>
> Key: HDFS-9362
> URL: https://issues.apache.org/jira/browse/HDFS-9362
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9362.001.patch
>
>
> {{TestAuditLogger#testAuditLoggerWithCallContext}} was added recently to
> exercise the new audit logging with caller context functionality. The tests
> assume Unix line endings by hard-coding "\n" in asserts. These tests fail on
> Windows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9319) Make DatanodeInfo thread safe
[
https://issues.apache.org/jira/browse/HDFS-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988148#comment-14988148
]
Jing Zhao commented on HDFS-9319:
-
Thanks for the review, Chris! Totally agree we should also correctly handle
data fields like {{volumeFailures}} and {{volumeFailureSummary}} in
DatanodeDescriptor. But in this jira my original plan is to clean up
DatanodeInfo, but handle DatanodeDescriptor in a separate jira which may still
need some more work. So can we leave the changes you suggested there?
> Make DatanodeInfo thread safe
> -
>
> Key: HDFS-9319
> URL: https://issues.apache.org/jira/browse/HDFS-9319
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-9319.000.patch
>
>
> This jira plans to make DatanodeInfo's internal states independent of
> external locks. Note because DatanodeInfo extends DatanodeID, we still need
> to change DatanodeID as a follow-on work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9369) Use ctest to run tests for hadoop-hdfs-native-client
[
https://issues.apache.org/jira/browse/HDFS-9369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988160#comment-14988160
]
Allen Wittenauer commented on HDFS-9369:
We need an option (or just force it) to send ctest output to a file for later
post-processing similar to what we do for bats+TAP.
> Use ctest to run tests for hadoop-hdfs-native-client
>
>
> Key: HDFS-9369
> URL: https://issues.apache.org/jira/browse/HDFS-9369
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>Priority: Minor
> Attachments: HDFS-9369.000.patch
>
>
> Currently we write special rules in {{pom.xml}} to run tests in
> {{hadoop-hdfs-native-client}}. This jira proposes to run these tests using
> ctest to simplify {{pom.xml}} and improve portability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9368) Implement reads with implicit offset state in libhdfs++
[ https://issues.apache.org/jira/browse/HDFS-9368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-9368: -- Attachment: HDFS-9368.HDFS-8707.000.patch Implemented just enough to unblock HDFS-9359 (test I want to use needs hdfsRead). Todo: -Add whence argument to FileHandle::Seek -Add bounds check to FileHandle::Seek, waiting to see what HDFS-9144 looks like before accessing members of InputStream. -Add hdfsTell > Implement reads with implicit offset state in libhdfs++ > --- > > Key: HDFS-9368 > URL: https://issues.apache.org/jira/browse/HDFS-9368 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-9368.HDFS-8707.000.patch > > > Currently only positional reads are supported. Implement a stateful read > that keeps track of offset into file. Also expose it in the c bindings as > hdfsRead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8849) fsck should report number of missing blocks with replication factor 1
[
https://issues.apache.org/jira/browse/HDFS-8849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HDFS-8849:
--
Target Version/s: 2.8.0 (was: 2.7.2)
Moving improvements out of 2.7 maintenance releases.
> fsck should report number of missing blocks with replication factor 1
> -
>
> Key: HDFS-8849
> URL: https://issues.apache.org/jira/browse/HDFS-8849
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>Priority: Minor
>
> HDFS-7165 supports reporting number of blocks with replication factor 1 in
> {{dfsadmin}} and NN metrics. But it didn't extend {{fsck}} with the same
> support, which is the aim of this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9368) Implement reads with implicit offset state in libhdfs++
James Clampffer created HDFS-9368: - Summary: Implement reads with implicit offset state in libhdfs++ Key: HDFS-9368 URL: https://issues.apache.org/jira/browse/HDFS-9368 Project: Hadoop HDFS Issue Type: Sub-task Reporter: James Clampffer Assignee: James Clampffer Currently only positional reads are supported. Implement a stateful read that keeps track of offset into file. Also expose it in the c bindings as hdfsRead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9354) Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows
[ https://issues.apache.org/jira/browse/HDFS-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988002#comment-14988002 ] Hudson commented on HDFS-9354: -- FAILURE: Integrated in Hadoop-trunk-Commit #8748 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8748/]) HDFS-9354. Fix TestBalancer#testBalancerWithZeroThreadsForMove on (cnauroth: rev 095ac834022df6136b42961c507ec745c6cf8f97) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows > -- > > Key: HDFS-9354 > URL: https://issues.apache.org/jira/browse/HDFS-9354 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Fix For: 2.8.0 > > Attachments: HDFS-9354.00.patch, HDFS-9354.01.patch > > > This negative test expect HadoopIllegalArgumentException on illegal > configuration. It uses JUnit (expected=HadoopIllegalArgumentException.class) > and passed fine on Linux. > On windows, this test passes as well. But it left open handles on NN metadata > directories used by MiniDFSCluster. As a result, quite a few of subsequent > TestBalancer unit tests can't start MiniDFSCluster. The open handles prevents > them from cleaning up NN metadata directories on Windows. > This JIRA is opened to explicitly catch the Exception and ensure the test > cluster is properly shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7984) webhdfs:// needs to support provided delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-7984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988102#comment-14988102 ] HeeSoo Kim commented on HDFS-7984: -- The test failures are unrelated to the change made for this jira. It has javadoc checkstyle error. However, I keep this for the code consistency. > webhdfs:// needs to support provided delegation tokens > -- > > Key: HDFS-7984 > URL: https://issues.apache.org/jira/browse/HDFS-7984 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: HeeSoo Kim >Priority: Blocker > Attachments: HDFS-7984.001.patch, HDFS-7984.002.patch, > HDFS-7984.003.patch, HDFS-7984.004.patch, HDFS-7984.005.patch, > HDFS-7984.006.patch, HDFS-7984.patch > > > When using the webhdfs:// filesystem (especially from distcp), we need the > ability to inject a delegation token rather than webhdfs initialize its own. > This would allow for cross-authentication-zone file system accesses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9369) Use ctest to run tests for hadoop-hdfs-native-client
[
https://issues.apache.org/jira/browse/HDFS-9369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-9369:
-
Status: Patch Available (was: Open)
> Use ctest to run tests for hadoop-hdfs-native-client
>
>
> Key: HDFS-9369
> URL: https://issues.apache.org/jira/browse/HDFS-9369
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>Priority: Minor
> Attachments: HDFS-9369.000.patch
>
>
> Currently we write special rules in {{pom.xml}} to run tests in
> {{hadoop-hdfs-native-client}}. This jira proposes to run these tests using
> ctest to simplify {{pom.xml}} and improve portability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9369) Use ctest to run tests for hadoop-hdfs-native-client
[
https://issues.apache.org/jira/browse/HDFS-9369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-9369:
-
Attachment: HDFS-9369.000.patch
> Use ctest to run tests for hadoop-hdfs-native-client
>
>
> Key: HDFS-9369
> URL: https://issues.apache.org/jira/browse/HDFS-9369
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>Priority: Minor
> Attachments: HDFS-9369.000.patch
>
>
> Currently we write special rules in {{pom.xml}} to run tests in
> {{hadoop-hdfs-native-client}}. This jira proposes to run these tests using
> ctest to simplify {{pom.xml}} and improve portability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9362) TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings, fails on Windows.
[
https://issues.apache.org/jira/browse/HDFS-9362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-9362:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
I have committed this to trunk and branch-2. Arpit, Mingliang and Xiaoyu,
thank you for the code reviews.
> TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings,
> fails on Windows.
> ---
>
> Key: HDFS-9362
> URL: https://issues.apache.org/jira/browse/HDFS-9362
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9362.001.patch
>
>
> {{TestAuditLogger#testAuditLoggerWithCallContext}} was added recently to
> exercise the new audit logging with caller context functionality. The tests
> assume Unix line endings by hard-coding "\n" in asserts. These tests fail on
> Windows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9354) Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows
[ https://issues.apache.org/jira/browse/HDFS-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988082#comment-14988082 ] Hudson commented on HDFS-9354: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2563 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2563/]) HDFS-9354. Fix TestBalancer#testBalancerWithZeroThreadsForMove on (cnauroth: rev 095ac834022df6136b42961c507ec745c6cf8f97) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows > -- > > Key: HDFS-9354 > URL: https://issues.apache.org/jira/browse/HDFS-9354 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Fix For: 2.8.0 > > Attachments: HDFS-9354.00.patch, HDFS-9354.01.patch > > > This negative test expect HadoopIllegalArgumentException on illegal > configuration. It uses JUnit (expected=HadoopIllegalArgumentException.class) > and passed fine on Linux. > On windows, this test passes as well. But it left open handles on NN metadata > directories used by MiniDFSCluster. As a result, quite a few of subsequent > TestBalancer unit tests can't start MiniDFSCluster. The open handles prevents > them from cleaning up NN metadata directories on Windows. > This JIRA is opened to explicitly catch the Exception and ensure the test > cluster is properly shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9362) TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings, fails on Windows.
[
https://issues.apache.org/jira/browse/HDFS-9362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988081#comment-14988081
]
Hudson commented on HDFS-9362:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2563 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2563/])
HDFS-9362. TestAuditLogger#testAuditLoggerWithCallContext assumes Unix
(cnauroth: rev 7e2829662b4c4bf33ebaf2fa09312d0bed3d6f92)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLogger.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings,
> fails on Windows.
> ---
>
> Key: HDFS-9362
> URL: https://issues.apache.org/jira/browse/HDFS-9362
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9362.001.patch
>
>
> {{TestAuditLogger#testAuditLoggerWithCallContext}} was added recently to
> exercise the new audit logging with caller context functionality. The tests
> assume Unix line endings by hard-coding "\n" in asserts. These tests fail on
> Windows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9369) Use ctest to run tests for hadoop-hdfs-native-client
Haohui Mai created HDFS-9369:
Summary: Use ctest to run tests for hadoop-hdfs-native-client
Key: HDFS-9369
URL: https://issues.apache.org/jira/browse/HDFS-9369
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Priority: Minor
Currently we write special rules in {{pom.xml}} to run tests in
{{hadoop-hdfs-native-client}}. This jira proposes to run these tests using
ctest to simplify {{pom.xml}} and improve portability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-9367) Get libhdfs++ gmock tests running with CI
[ https://issues.apache.org/jira/browse/HDFS-9367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai resolved HDFS-9367. -- Resolution: Duplicate This can be done in trunk. The issue should be fixed once HDFS-9369 lands. > Get libhdfs++ gmock tests running with CI > - > > Key: HDFS-9367 > URL: https://issues.apache.org/jira/browse/HDFS-9367 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Haohui Mai > > The gmock tests build with maven but there's no 'make test' target exposed to > maven/antrun. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9354) Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows
[ https://issues.apache.org/jira/browse/HDFS-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-9354: Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) I have committed this to trunk and branch-2, after a minor rebase due to the lack of the erasure coding changes in branch-2. Xiaoyu, thank you for contributing the patch. > Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows > -- > > Key: HDFS-9354 > URL: https://issues.apache.org/jira/browse/HDFS-9354 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Fix For: 2.8.0 > > Attachments: HDFS-9354.00.patch, HDFS-9354.01.patch > > > This negative test expect HadoopIllegalArgumentException on illegal > configuration. It uses JUnit (expected=HadoopIllegalArgumentException.class) > and passed fine on Linux. > On windows, this test passes as well. But it left open handles on NN metadata > directories used by MiniDFSCluster. As a result, quite a few of subsequent > TestBalancer unit tests can't start MiniDFSCluster. The open handles prevents > them from cleaning up NN metadata directories on Windows. > This JIRA is opened to explicitly catch the Exception and ensure the test > cluster is properly shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[
https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987991#comment-14987991
]
Hadoop QA commented on HDFS-9289:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 5s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 51s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk cannot run
convertXmlToText from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 12s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 19s
{color} | {color:red} Patch generated 1 new checkstyle issues in
hadoop-hdfs-project (total was 247, now 247). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 11s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 48s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 48s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 57s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 19s
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 155m 57s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_60 Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestNodeCount |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| JDK v1.7.0_79 Failed junit tests | hadoop.fs.shell.TestHdfsTextCommand |
| | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.1 Server=1.7.1
Image:test-patch-base-hadoop-date2015-11-03 |
| JIRA Patch URL |
[jira] [Updated] (HDFS-9007) Fix HDFS Balancer to honor upgrade domain policy
[ https://issues.apache.org/jira/browse/HDFS-9007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9007: -- Attachment: HDFS-9007-branch-2.patch Here is the patch for branch-2. > Fix HDFS Balancer to honor upgrade domain policy > > > Key: HDFS-9007 > URL: https://issues.apache.org/jira/browse/HDFS-9007 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ming Ma >Assignee: Ming Ma > Attachments: HDFS-9007-2.patch, HDFS-9007-branch-2.patch, > HDFS-9007.patch > > > In the current design of HDFS Balancer, it doesn't use BlockPlacementPolicy > used by namenode runtime. Instead, it has somewhat redundant code to make > sure block allocation conforms with the rack policy. > When namenode uses upgrade domain based policy, we need to make sure that > HDFS balancer doesn't move blocks in a way that could violate upgrade domain > block placement policy. > In the longer term, we should consider how to make Balancer independent of > the actual BlockPlacementPolicy as in HDFS-1431. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9357) NN UI is not showing which DN is "Decommissioned "and "Decommissioned & dead".
[ https://issues.apache.org/jira/browse/HDFS-9357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988111#comment-14988111 ] Haohui Mai commented on HDFS-9357: -- +1 pending jenkins. > NN UI is not showing which DN is "Decommissioned "and "Decommissioned & dead". > -- > > Key: HDFS-9357 > URL: https://issues.apache.org/jira/browse/HDFS-9357 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Archana T >Assignee: Surendra Singh Lilhore >Priority: Critical > Attachments: Decommissioned_Dead_Fixed.PNG, Decommissioned_Fixed.PNG, > HDFS-9357.001.patch, decommisioned_n_dead_.png, decommissioned_.png > > > NN UI is not showing which DN is "Decommissioned "and "Decommissioned & dead" > Root Cause -- > "Decommissioned" and "Decommissioned & dead" icon not reflected on NN UI > When DN is in Decommissioned status or in "Decommissioned & dead" status, > same status is not reflected on NN UI > DN status is as below -- > hdfs dfsadmin -report > Name: 10.xx.xx.xx1:50076 (host-xx1) > Hostname: host-xx > Decommission Status : Decommissioned > Configured Capacity: 230501634048 (214.67 GB) > DFS Used: 36864 (36 KB) > Dead datanodes (1): > Name: 10.xx.xx.xx2:50076 (host-xx2) > Hostname: host-xx > Decommission Status : Decommissioned > Same is not reflected on NN UI. > Attached NN UI snapshots for the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9370) TestDataNodeUGIProvider fails intermittently due to non-deterministic cache expiry.
[
https://issues.apache.org/jira/browse/HDFS-9370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-9370:
Status: Patch Available (was: Open)
> TestDataNodeUGIProvider fails intermittently due to non-deterministic cache
> expiry.
> ---
>
> Key: HDFS-9370
> URL: https://issues.apache.org/jira/browse/HDFS-9370
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-9370.001.patch
>
>
> {{TestDataNodeUGIProvider}} has hard-coded sleep times waiting for background
> expiration of entries in a Guava cache. I have seen this test suite fail
> intermittently, because expiration is not guaranteed to happen strictly on
> the boundary of the period defined by the cache's expiration time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9370) TestDataNodeUGIProvider fails intermittently due to non-deterministic cache expiry.
[
https://issues.apache.org/jira/browse/HDFS-9370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-9370:
Attachment: HDFS-9370.001.patch
With the attached patch, the test suite consistently passes in all of the
environments I could find. We need to be careful not to touch the entries in
the cache while we're waiting for expiration. If we did, then that would reset
the clock on expiration for those entries. Instead, I needed to use
{{VisibleForTesting}} to poke at some internals from within the test.
[~xyao], would you please code review?
> TestDataNodeUGIProvider fails intermittently due to non-deterministic cache
> expiry.
> ---
>
> Key: HDFS-9370
> URL: https://issues.apache.org/jira/browse/HDFS-9370
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-9370.001.patch
>
>
> {{TestDataNodeUGIProvider}} has hard-coded sleep times waiting for background
> expiration of entries in a Guava cache. I have seen this test suite fail
> intermittently, because expiration is not guaranteed to happen strictly on
> the boundary of the period defined by the cache's expiration time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9354) Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows
[ https://issues.apache.org/jira/browse/HDFS-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988456#comment-14988456 ] Hudson commented on HDFS-9354: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2504 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2504/]) HDFS-9354. Fix TestBalancer#testBalancerWithZeroThreadsForMove on (cnauroth: rev 095ac834022df6136b42961c507ec745c6cf8f97) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows > -- > > Key: HDFS-9354 > URL: https://issues.apache.org/jira/browse/HDFS-9354 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Fix For: 2.8.0 > > Attachments: HDFS-9354.00.patch, HDFS-9354.01.patch > > > This negative test expect HadoopIllegalArgumentException on illegal > configuration. It uses JUnit (expected=HadoopIllegalArgumentException.class) > and passed fine on Linux. > On windows, this test passes as well. But it left open handles on NN metadata > directories used by MiniDFSCluster. As a result, quite a few of subsequent > TestBalancer unit tests can't start MiniDFSCluster. The open handles prevents > them from cleaning up NN metadata directories on Windows. > This JIRA is opened to explicitly catch the Exception and ensure the test > cluster is properly shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9370) TestDataNodeUGIProvider fails intermittently due to non-deterministic cache expiry.
[
https://issues.apache.org/jira/browse/HDFS-9370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988511#comment-14988511
]
Xiaobing Zhou commented on HDFS-9370:
-
+1.
Thanks [~cnauroth] for fix.
Could you put your comment with awaitCacheEmptyDueToExpiration to make
follow-up persons aware of the potential pitfall?
{code}
We need to be careful not to touch the entries in the cache while we're waiting
for expiration. If we did, then that would reset the clock on expiration for
those entries.
{code}
> TestDataNodeUGIProvider fails intermittently due to non-deterministic cache
> expiry.
> ---
>
> Key: HDFS-9370
> URL: https://issues.apache.org/jira/browse/HDFS-9370
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-9370.001.patch
>
>
> {{TestDataNodeUGIProvider}} has hard-coded sleep times waiting for background
> expiration of entries in a Guava cache. I have seen this test suite fail
> intermittently, because expiration is not guaranteed to happen strictly on
> the boundary of the period defined by the cache's expiration time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9370) TestDataNodeUGIProvider fails intermittently due to non-deterministic cache expiry.
[
https://issues.apache.org/jira/browse/HDFS-9370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-9370:
Attachment: HDFS-9370.002.patch
[~xiaobingo], that's a good idea. Here is patch v002, just adding a comment on
{{awaitCacheEmptyDueToExpiration}}.
> TestDataNodeUGIProvider fails intermittently due to non-deterministic cache
> expiry.
> ---
>
> Key: HDFS-9370
> URL: https://issues.apache.org/jira/browse/HDFS-9370
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-9370.001.patch, HDFS-9370.002.patch
>
>
> {{TestDataNodeUGIProvider}} has hard-coded sleep times waiting for background
> expiration of entries in a Guava cache. I have seen this test suite fail
> intermittently, because expiration is not guaranteed to happen strictly on
> the boundary of the period defined by the cache's expiration time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9252) Change TestFileTruncate to use FsDatasetTestUtils to get block file size and genstamp.
[
https://issues.apache.org/jira/browse/HDFS-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-9252:
Attachment: HDFS-9252.03.patch
Rebase to {{trunk}}.
> Change TestFileTruncate to use FsDatasetTestUtils to get block file size and
> genstamp.
> --
>
> Key: HDFS-9252
> URL: https://issues.apache.org/jira/browse/HDFS-9252
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.7.1
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-9252.00.patch, HDFS-9252.01.patch,
> HDFS-9252.02.patch, HDFS-9252.03.patch
>
>
> {{TestFileTruncate}} verifies block size and genstamp by directly accessing
> the local filesystem, e.g.:
> {code}
> assertTrue(cluster.getBlockMetadataFile(dn0,
>newBlock.getBlock()).getName().endsWith(
>newBlock.getBlock().getGenerationStamp() + ".meta"));
> {code}
> Lets abstract the fsdataset-special logic behind FsDatasetTestUtils.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9351) checkNNStartup() need to be called when fsck calls FSNamesystem.getSnapshottableDirs()
[
https://issues.apache.org/jira/browse/HDFS-9351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HDFS-9351:
Summary: checkNNStartup() need to be called when fsck calls
FSNamesystem.getSnapshottableDirs() (was: No checkNNStartup() is called when
fsck calls FSNamesystem.getSnapshottableDirs())
> checkNNStartup() need to be called when fsck calls
> FSNamesystem.getSnapshottableDirs()
> --
>
> Key: HDFS-9351
> URL: https://issues.apache.org/jira/browse/HDFS-9351
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Reporter: Yongjun Zhang
>Assignee: Xiao Chen
> Attachments: HDFS-9351.patch
>
>
> I looked further at HDFS-9231 change I reviewed earlier, and realize I missed
> one thing.
> The patch changed
> {code}
> if (snapshottableDirs != null) {
> SnapshottableDirectoryStatus[] snapshotDirs = namenode.getRpcServer()
> .getSnapshottableDirListing();
> if (snapshotDirs != null) {
> for (SnapshottableDirectoryStatus dir : snapshotDirs) {
> snapshottableDirs.add(dir.getFullPath().toString());
> }
> }
> }
> {code}
> to
> {code}
> if (snapshottableDirs != null) {
> snapshottableDirs = namenode.getNamesystem().getSnapshottableDirs();
> }
> {code}
> In old code, namenode.getRpcServer().getSnapshottableDirListing() calls
> checkNNStartup(), however, this is not done with the new code. This is a hole
> of the earlier HDFS-9231 patch.
> Create this jira to fix the hole. Suggest to revert this portion of the
> change, and add code to do what we need at NamenodeFsck.
> Sorry for missing this in my earlier review of HDFS-9231.
> Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9354) Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows
[ https://issues.apache.org/jira/browse/HDFS-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988435#comment-14988435 ] Hudson commented on HDFS-9354: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #622 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/622/]) HDFS-9354. Fix TestBalancer#testBalancerWithZeroThreadsForMove on (cnauroth: rev 095ac834022df6136b42961c507ec745c6cf8f97) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows > -- > > Key: HDFS-9354 > URL: https://issues.apache.org/jira/browse/HDFS-9354 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Fix For: 2.8.0 > > Attachments: HDFS-9354.00.patch, HDFS-9354.01.patch > > > This negative test expect HadoopIllegalArgumentException on illegal > configuration. It uses JUnit (expected=HadoopIllegalArgumentException.class) > and passed fine on Linux. > On windows, this test passes as well. But it left open handles on NN metadata > directories used by MiniDFSCluster. As a result, quite a few of subsequent > TestBalancer unit tests can't start MiniDFSCluster. The open handles prevents > them from cleaning up NN metadata directories on Windows. > This JIRA is opened to explicitly catch the Exception and ensure the test > cluster is properly shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9362) TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings, fails on Windows.
[
https://issues.apache.org/jira/browse/HDFS-9362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988434#comment-14988434
]
Hudson commented on HDFS-9362:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #622 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/622/])
HDFS-9362. TestAuditLogger#testAuditLoggerWithCallContext assumes Unix
(cnauroth: rev 7e2829662b4c4bf33ebaf2fa09312d0bed3d6f92)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLogger.java
> TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings,
> fails on Windows.
> ---
>
> Key: HDFS-9362
> URL: https://issues.apache.org/jira/browse/HDFS-9362
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-9362.001.patch
>
>
> {{TestAuditLogger#testAuditLoggerWithCallContext}} was added recently to
> exercise the new audit logging with caller context functionality. The tests
> assume Unix line endings by hard-coding "\n" in asserts. These tests fail on
> Windows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9370) TestDataNodeUGIProvider fails intermittently due to non-deterministic cache expiry.
[
https://issues.apache.org/jira/browse/HDFS-9370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988470#comment-14988470
]
Xiaoyu Yao commented on HDFS-9370:
--
Thanks [~cnauroth] for fixing this! Patch LGTM, +1 pending Jenkins.
> TestDataNodeUGIProvider fails intermittently due to non-deterministic cache
> expiry.
> ---
>
> Key: HDFS-9370
> URL: https://issues.apache.org/jira/browse/HDFS-9370
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Attachments: HDFS-9370.001.patch
>
>
> {{TestDataNodeUGIProvider}} has hard-coded sleep times waiting for background
> expiration of entries in a Guava cache. I have seen this test suite fail
> intermittently, because expiration is not guaranteed to happen strictly on
> the boundary of the period defined by the cache's expiration time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9007) Fix HDFS Balancer to honor upgrade domain policy
[
https://issues.apache.org/jira/browse/HDFS-9007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988562#comment-14988562
]
Hadoop QA commented on HDFS-9007:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
26s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} branch-2 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} branch-2 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} branch-2 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 9s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in branch-2 cannot run
convertXmlToText from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} branch-2 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 55s
{color} | {color:green} branch-2 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s
{color} | {color:red} Patch generated 3 new checkstyle issues in
hadoop-hdfs-project/hadoop-hdfs (total was 109, now 104). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 59s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 53s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 36s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 22s
{color} | {color:red} Patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 157m 5s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.server.mover.TestStorageMover |
| | hadoop.hdfs.TestDistributedFileSystem |
| | hadoop.hdfs.TestRollingUpgradeRollback |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
| JDK v1.8.0_66 Timed out junit tests | org.apache.hadoop.hdfs.TestAclsEndToEnd
|
| JDK v1.7.0_79 Failed junit tests | hadoop.hdfs.TestDatanodeDeath |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestDistributedFileSystem |
| | hadoop.hdfs.server.datanode.TestBlockReplacement |
| | hadoop.hdfs.TestRollingUpgradeRollback |
| | hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot |
\\
\\
|| Subsystem ||
[jira] [Commented] (HDFS-6481) DatanodeManager#getDatanodeStorageInfos() should check the length of storageIDs
[
https://issues.apache.org/jira/browse/HDFS-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988636#comment-14988636
]
Tsz Wo Nicholas Sze commented on HDFS-6481:
---
Since the ArrayIndexOutOfBoundsException always happened with the index 0, I
strongly believe that the clients were running in an older version which did
not have storageIDs. Will post a patch to throw a better exception.
> DatanodeManager#getDatanodeStorageInfos() should check the length of
> storageIDs
> ---
>
> Key: HDFS-6481
> URL: https://issues.apache.org/jira/browse/HDFS-6481
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: BB2015-05-TBR
> Attachments: hdfs-6481-v1.txt
>
>
> Ian Brooks reported the following stack trace:
> {code}
> 2014-06-03 13:05:03,915 WARN [DataStreamer for file
> /user/hbase/WALs/,16020,1401716790638/%2C16020%2C1401716790638.1401796562200
> block BP-2121456822-10.143.38.149-1396953188241:blk_1074073683_332932]
> hdfs.DFSClient: DataStreamer Exception
> org.apache.hadoop.ipc.RemoteException(java.lang.ArrayIndexOutOfBoundsException):
> 0
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.getDatanodeStorageInfos(DatanodeManager.java:467)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:2779)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:594)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:430)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1962)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1958)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1956)
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy13.getAdditionalDatanode(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolTranslatorPB.java:352)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy14.getAdditionalDatanode(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:266)
> at com.sun.proxy.$Proxy15.getAdditionalDatanode(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:919)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:919)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1031)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:823)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:475)
> 2014-06-03 13:05:48,489 ERROR [RpcServer.handler=22,port=16020] wal.FSHLog:
> syncer encountered error,
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988639#comment-14988639 ] Hudson commented on HDFS-9289: -- SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #634 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/634/]) HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in (zhz: rev dac0463a4e20dfb3a802355919fc22b8e017a4e1) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaWithStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockWithInvalidGenStamp.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 2.8.0 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9371) Code cleanup for DatanodeManager
[
https://issues.apache.org/jira/browse/HDFS-9371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-9371:
Attachment: HDFS-9371.000.patch
> Code cleanup for DatanodeManager
>
>
> Key: HDFS-9371
> URL: https://issues.apache.org/jira/browse/HDFS-9371
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-9371.000.patch
>
>
> Some code cleanup for DatanodeManager. The main changes include:
> # make the synchronization of {{datanodeMap}} and
> {{datanodesSoftwareVersions}} consistent
> # remove unnecessary lock in {{handleHeartbeat}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6481) DatanodeManager#getDatanodeStorageInfos() should check the length of storageIDs
[
https://issues.apache.org/jira/browse/HDFS-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-6481:
--
Assignee: Tsz Wo Nicholas Sze (was: Ted Yu)
Priority: Minor (was: Major)
Component/s: namenode
Assigning this to myself. (Ted, I believe you won't mind ;)
> DatanodeManager#getDatanodeStorageInfos() should check the length of
> storageIDs
> ---
>
> Key: HDFS-6481
> URL: https://issues.apache.org/jira/browse/HDFS-6481
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.3.0
>Reporter: Ted Yu
>Assignee: Tsz Wo Nicholas Sze
>Priority: Minor
> Labels: BB2015-05-TBR
> Attachments: hdfs-6481-v1.txt
>
>
> Ian Brooks reported the following stack trace:
> {code}
> 2014-06-03 13:05:03,915 WARN [DataStreamer for file
> /user/hbase/WALs/,16020,1401716790638/%2C16020%2C1401716790638.1401796562200
> block BP-2121456822-10.143.38.149-1396953188241:blk_1074073683_332932]
> hdfs.DFSClient: DataStreamer Exception
> org.apache.hadoop.ipc.RemoteException(java.lang.ArrayIndexOutOfBoundsException):
> 0
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.getDatanodeStorageInfos(DatanodeManager.java:467)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:2779)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:594)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:430)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1962)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1958)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1956)
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy13.getAdditionalDatanode(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolTranslatorPB.java:352)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy14.getAdditionalDatanode(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:266)
> at com.sun.proxy.$Proxy15.getAdditionalDatanode(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:919)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:919)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1031)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:823)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:475)
> 2014-06-03 13:05:48,489 ERROR [RpcServer.handler=22,port=16020] wal.FSHLog:
> syncer encountered error, will retry.
[jira] [Created] (HDFS-9371) Code cleanup for DatanodeManager
Jing Zhao created HDFS-9371:
---
Summary: Code cleanup for DatanodeManager
Key: HDFS-9371
URL: https://issues.apache.org/jira/browse/HDFS-9371
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: Jing Zhao
Some code cleanup for DatanodeManager. The main changes include:
# make the synchronization of {{datanodeMap}} and {{datanodesSoftwareVersions}}
consistent
# remove unnecessary lock in {{handleHeartbeat}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8855) Webhdfs client leaks active NameNode connections
[
https://issues.apache.org/jira/browse/HDFS-8855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988449#comment-14988449
]
Chris Nauroth commented on HDFS-8855:
-
{{TestDataNodeUGIProvider}}, introduced in this patch, has been failing
intermittently. The root cause is reliance on hard-coded sleep times to
trigger cache expiration, which is non-deterministic. I've filed a patch on
HDFS-9370 to make the test work consistently.
> Webhdfs client leaks active NameNode connections
>
>
> Key: HDFS-8855
> URL: https://issues.apache.org/jira/browse/HDFS-8855
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: Bob Hansen
>Assignee: Xiaobing Zhou
> Fix For: 2.8.0
>
> Attachments: HDFS-8855.005.patch, HDFS-8855.006.patch,
> HDFS-8855.007.patch, HDFS-8855.1.patch, HDFS-8855.2.patch, HDFS-8855.3.patch,
> HDFS-8855.4.patch, HDFS_8855.prototype.patch
>
>
> The attached script simulates a process opening ~50 files via webhdfs and
> performing random reads. Note that there are at most 50 concurrent reads,
> and all webhdfs sessions are kept open. Each read is ~64k at a random
> position.
> The script periodically (once per second) shells into the NameNode and
> produces a summary of the socket states. For my test cluster with 5 nodes,
> it took ~30 seconds for the NameNode to have ~25000 active connections and
> fails.
> It appears that each request to the webhdfs client is opening a new
> connection to the NameNode and keeping it open after the request is complete.
> If the process continues to run, eventually (~30-60 seconds), all of the
> open connections are closed and the NameNode recovers.
> This smells like SoftReference reaping. Are we using SoftReferences in the
> webhdfs client to cache NameNode connections but never re-using them?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9236) Missing sanity check for block size during block recovery
[
https://issues.apache.org/jira/browse/HDFS-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tony Wu updated HDFS-9236:
--
Attachment: HDFS-9236.006.patch
In v6 patch:
* Address [~walter.k.su]'s comment by excluding RUR replicas from syncList.
{{syncBlock()}} now will work on a clean syncList containing only replicas that
will be used for recovery.
* Converted the check in previous patches for {{Long.MAX_VALUE}} to an assert.
* Reworked the test case.
* Add some comments.
> Missing sanity check for block size during block recovery
> -
>
> Key: HDFS-9236
> URL: https://issues.apache.org/jira/browse/HDFS-9236
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.7.1
>Reporter: Tony Wu
>Assignee: Tony Wu
> Attachments: HDFS-9236.001.patch, HDFS-9236.002.patch,
> HDFS-9236.003.patch, HDFS-9236.004.patch, HDFS-9236.005.patch,
> HDFS-9236.006.patch
>
>
> Ran into an issue while running test against faulty data-node code.
> Currently in DataNode.java:
> {code:java}
> /** Block synchronization */
> void syncBlock(RecoveringBlock rBlock,
> List syncList) throws IOException {
> …
> // Calculate the best available replica state.
> ReplicaState bestState = ReplicaState.RWR;
> …
> // Calculate list of nodes that will participate in the recovery
> // and the new block size
> List participatingList = new ArrayList();
> final ExtendedBlock newBlock = new ExtendedBlock(bpid, blockId,
> -1, recoveryId);
> switch(bestState) {
> …
> case RBW:
> case RWR:
> long minLength = Long.MAX_VALUE;
> for(BlockRecord r : syncList) {
> ReplicaState rState = r.rInfo.getOriginalReplicaState();
> if(rState == bestState) {
> minLength = Math.min(minLength, r.rInfo.getNumBytes());
> participatingList.add(r);
> }
> }
> newBlock.setNumBytes(minLength);
> break;
> …
> }
> …
> nn.commitBlockSynchronization(block,
> newBlock.getGenerationStamp(), newBlock.getNumBytes(), true, false,
> datanodes, storages);
> }
> {code}
> This code is called by the DN coordinating the block recovery. In the above
> case, it is possible for none of the rState (reported by DNs with copies of
> the replica being recovered) to match the bestState. This can either be
> caused by faulty DN code or stale/modified/corrupted files on DN. When this
> happens the DN will end up reporting the minLengh of Long.MAX_VALUE.
> Unfortunately there is no check on the NN for replica length. See
> FSNamesystem.java:
> {code:java}
> void commitBlockSynchronization(ExtendedBlock oldBlock,
> long newgenerationstamp, long newlength,
> boolean closeFile, boolean deleteblock, DatanodeID[] newtargets,
> String[] newtargetstorages) throws IOException {
> …
> if (deleteblock) {
> Block blockToDel = ExtendedBlock.getLocalBlock(oldBlock);
> boolean remove = iFile.removeLastBlock(blockToDel) != null;
> if (remove) {
> blockManager.removeBlock(storedBlock);
> }
> } else {
> // update last block
> if(!copyTruncate) {
> storedBlock.setGenerationStamp(newgenerationstamp);
>
> // XXX block length is updated without any check <<< storedBlock.setNumBytes(newlength);
> }
> …
> if (closeFile) {
> LOG.info("commitBlockSynchronization(oldBlock=" + oldBlock
> + ", file=" + src
> + (copyTruncate ? ", newBlock=" + truncatedBlock
> : ", newgenerationstamp=" + newgenerationstamp)
> + ", newlength=" + newlength
> + ", newtargets=" + Arrays.asList(newtargets) + ") successful");
> } else {
> LOG.info("commitBlockSynchronization(" + oldBlock + ") successful");
> }
> }
> {code}
> After this point the block length becomes Long.MAX_VALUE. Any subsequent
> block report (even with correct length) will cause the block to be marked as
> corrupted. Since this is block could be the last block of the file. If this
> happens and the client goes away, NN won’t be able to recover the lease and
> close the file because the last block is under-replicated.
> I believe we need to have a sanity check for block size on both DN and NN to
> prevent such case from happening.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988600#comment-14988600 ] Hudson commented on HDFS-9289: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2564 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2564/]) HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in (zhz: rev dac0463a4e20dfb3a802355919fc22b8e017a4e1) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaWithStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockWithInvalidGenStamp.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 2.8.0 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988619#comment-14988619 ] Hudson commented on HDFS-9289: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #1357 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1357/]) HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in (zhz: rev dac0463a4e20dfb3a802355919fc22b8e017a4e1) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockWithInvalidGenStamp.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaWithStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 2.8.0 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8976) Create HTML5 cluster webconsole for federated cluster
[
https://issues.apache.org/jira/browse/HDFS-8976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated HDFS-8976:
--
Target Version/s: 2.8.0 (was: 2.7.2)
Moving this new feature out from 2.7.2 and related maintenance lines and
putting it into 2.8.
> Create HTML5 cluster webconsole for federated cluster
> -
>
> Key: HDFS-8976
> URL: https://issues.apache.org/jira/browse/HDFS-8976
> Project: Hadoop HDFS
> Issue Type: New Feature
>Affects Versions: 2.7.0
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
> Attachments: HDFS-8976-01.patch, HDFS-8976-02.patch,
> HDFS-8976-03.patch, cluster-health.JPG
>
>
> Since the old jsp variant of cluster web console is no longer present from
> 2.7 onwards, there is a need for HTML 5 web console for overview of overall
> cluster.
> 2.7.1 docs says to check webconsole as below {noformat}Similar to the
> Namenode status web page, when using federation a Cluster Web Console is
> available to monitor the federated cluster at
> http:///dfsclusterhealth.jsp. Any Namenode in the cluster
> can be used to access this web page.{noformat}
> But this is no longer present,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
