[jira] [Work logged] (HDFS-13293) RBF: The RouterRPCServer should transfer CallerContext and client ip to NamenodeRpcServer
[
https://issues.apache.org/jira/browse/HDFS-13293?focusedWorklogId=496313=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-496313
]
ASF GitHub Bot logged work on HDFS-13293:
-
Author: ASF GitHub Bot
Created on: 07/Oct/20 05:55
Start Date: 07/Oct/20 05:55
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2363:
URL: https://github.com/apache/hadoop/pull/2363#issuecomment-704710737
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 14s | | trunk passed |
| +1 :green_heart: | compile | 0m 49s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 31s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 46s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 29s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 26s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 3s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 39s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 31s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 10m 21s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 90m 25s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2363/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2363 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 2d4e43bd84ef 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1cfe5916e20 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2363/4/testReport/ |
| Max. process+thread count | 3341 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2363/4/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus
[jira] [Updated] (HDFS-15611) Add list Snapshot command in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-15611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-15611: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Add list Snapshot command in WebHDFS > > > Key: HDFS-15611 > URL: https://issues.apache.org/jira/browse/HDFS-15611 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Idea here is to expose lsSnapshot cmd over WebHdfs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15611) Add list Snapshot command in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-15611?focusedWorklogId=496294=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-496294 ] ASF GitHub Bot logged work on HDFS-15611: - Author: ASF GitHub Bot Created on: 07/Oct/20 05:06 Start Date: 07/Oct/20 05:06 Worklog Time Spent: 10m Work Description: bshashikant commented on pull request #2355: URL: https://github.com/apache/hadoop/pull/2355#issuecomment-704694415 Thanks @szetszwo and @smengcl for the review. The 2 checkstyle issues reported are not addressed for better readability of the code. The unit test failures are not related. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 496294) Time Spent: 50m (was: 40m) > Add list Snapshot command in WebHDFS > > > Key: HDFS-15611 > URL: https://issues.apache.org/jira/browse/HDFS-15611 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Idea here is to expose lsSnapshot cmd over WebHdfs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15611) Add list Snapshot command in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-15611?focusedWorklogId=496293=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-496293 ] ASF GitHub Bot logged work on HDFS-15611: - Author: ASF GitHub Bot Created on: 07/Oct/20 05:04 Start Date: 07/Oct/20 05:04 Worklog Time Spent: 10m Work Description: bshashikant merged pull request #2355: URL: https://github.com/apache/hadoop/pull/2355 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 496293) Time Spent: 40m (was: 0.5h) > Add list Snapshot command in WebHDFS > > > Key: HDFS-15611 > URL: https://issues.apache.org/jira/browse/HDFS-15611 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Idea here is to expose lsSnapshot cmd over WebHdfs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15613) RBF: Router FSCK fails after HDFS-14442
[
https://issues.apache.org/jira/browse/HDFS-15613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka resolved HDFS-15613.
--
Fix Version/s: 3.4.0
3.3.1
Target Version/s: 3.3.1, 3.4.0 (was: 3.2.2, 3.3.1, 3.4.0)
Resolution: Fixed
Merged the PR into trunk and branch-3.3.
> RBF: Router FSCK fails after HDFS-14442
> ---
>
> Key: HDFS-15613
> URL: https://issues.apache.org/jira/browse/HDFS-15613
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Affects Versions: 3.3.0
> Environment: HA is enabled
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> After HDFS-14442 fsck uses getHAServiceState operation to detect Active
> NameNode, however, DFSRouter does not support the operation.
> {noformat}
> 20/10/05 16:41:30 DEBUG hdfs.HAUtil: Error while connecting to namenode
> org.apache.hadoop.ipc.RemoteException(java.lang.UnsupportedOperationException):
> Operation "getHAServiceState" is not supported
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.checkOperation(RouterRpcServer.java:488)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterClientProtocol.getHAServiceState(RouterClientProtocol.java:1773)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getHAServiceState(RouterRpcServer.java:1333)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getHAServiceState(ClientNamenodeProtocolServerSideTranslatorPB.java:2011)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:532)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1020)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:948)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2952)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562)
> at org.apache.hadoop.ipc.Client.call(Client.java:1508)
> at org.apache.hadoop.ipc.Client.call(Client.java:1405)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:234)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:119)
> at com.sun.proxy.$Proxy12.getHAServiceState(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getHAServiceState(ClientNamenodeProtocolTranslatorPB.java:2055)
> at org.apache.hadoop.hdfs.HAUtil.getAddressOfActive(HAUtil.java:281)
> at
> org.apache.hadoop.hdfs.tools.DFSck.getCurrentNamenodeAddress(DFSck.java:271)
> at org.apache.hadoop.hdfs.tools.DFSck.doWork(DFSck.java:339)
> at org.apache.hadoop.hdfs.tools.DFSck.access$000(DFSck.java:75)
> at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:164)
> at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:161)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
> at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:160)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:409)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15613) RBF: Router FSCK fails after HDFS-14442
[
https://issues.apache.org/jira/browse/HDFS-15613?focusedWorklogId=496287=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-496287
]
ASF GitHub Bot logged work on HDFS-15613:
-
Author: ASF GitHub Bot
Created on: 07/Oct/20 04:38
Start Date: 07/Oct/20 04:38
Worklog Time Spent: 10m
Work Description: aajisaka commented on pull request #2360:
URL: https://github.com/apache/hadoop/pull/2360#issuecomment-704686560
Merged. Thank you @ayushtkn and @goiri
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 496287)
Time Spent: 1h 50m (was: 1h 40m)
> RBF: Router FSCK fails after HDFS-14442
> ---
>
> Key: HDFS-15613
> URL: https://issues.apache.org/jira/browse/HDFS-15613
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Affects Versions: 3.3.0
> Environment: HA is enabled
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> After HDFS-14442 fsck uses getHAServiceState operation to detect Active
> NameNode, however, DFSRouter does not support the operation.
> {noformat}
> 20/10/05 16:41:30 DEBUG hdfs.HAUtil: Error while connecting to namenode
> org.apache.hadoop.ipc.RemoteException(java.lang.UnsupportedOperationException):
> Operation "getHAServiceState" is not supported
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.checkOperation(RouterRpcServer.java:488)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterClientProtocol.getHAServiceState(RouterClientProtocol.java:1773)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getHAServiceState(RouterRpcServer.java:1333)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getHAServiceState(ClientNamenodeProtocolServerSideTranslatorPB.java:2011)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:532)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1020)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:948)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2952)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562)
> at org.apache.hadoop.ipc.Client.call(Client.java:1508)
> at org.apache.hadoop.ipc.Client.call(Client.java:1405)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:234)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:119)
> at com.sun.proxy.$Proxy12.getHAServiceState(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getHAServiceState(ClientNamenodeProtocolTranslatorPB.java:2055)
> at org.apache.hadoop.hdfs.HAUtil.getAddressOfActive(HAUtil.java:281)
> at
> org.apache.hadoop.hdfs.tools.DFSck.getCurrentNamenodeAddress(DFSck.java:271)
> at org.apache.hadoop.hdfs.tools.DFSck.doWork(DFSck.java:339)
> at org.apache.hadoop.hdfs.tools.DFSck.access$000(DFSck.java:75)
> at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:164)
> at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:161)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
> at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:160)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:409)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
[jira] [Work logged] (HDFS-15613) RBF: Router FSCK fails after HDFS-14442
[
https://issues.apache.org/jira/browse/HDFS-15613?focusedWorklogId=496286=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-496286
]
ASF GitHub Bot logged work on HDFS-15613:
-
Author: ASF GitHub Bot
Created on: 07/Oct/20 04:37
Start Date: 07/Oct/20 04:37
Worklog Time Spent: 10m
Work Description: aajisaka merged pull request #2360:
URL: https://github.com/apache/hadoop/pull/2360
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 496286)
Time Spent: 1h 40m (was: 1.5h)
> RBF: Router FSCK fails after HDFS-14442
> ---
>
> Key: HDFS-15613
> URL: https://issues.apache.org/jira/browse/HDFS-15613
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Affects Versions: 3.3.0
> Environment: HA is enabled
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> After HDFS-14442 fsck uses getHAServiceState operation to detect Active
> NameNode, however, DFSRouter does not support the operation.
> {noformat}
> 20/10/05 16:41:30 DEBUG hdfs.HAUtil: Error while connecting to namenode
> org.apache.hadoop.ipc.RemoteException(java.lang.UnsupportedOperationException):
> Operation "getHAServiceState" is not supported
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.checkOperation(RouterRpcServer.java:488)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterClientProtocol.getHAServiceState(RouterClientProtocol.java:1773)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getHAServiceState(RouterRpcServer.java:1333)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getHAServiceState(ClientNamenodeProtocolServerSideTranslatorPB.java:2011)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:532)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1020)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:948)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2952)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562)
> at org.apache.hadoop.ipc.Client.call(Client.java:1508)
> at org.apache.hadoop.ipc.Client.call(Client.java:1405)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:234)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:119)
> at com.sun.proxy.$Proxy12.getHAServiceState(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getHAServiceState(ClientNamenodeProtocolTranslatorPB.java:2055)
> at org.apache.hadoop.hdfs.HAUtil.getAddressOfActive(HAUtil.java:281)
> at
> org.apache.hadoop.hdfs.tools.DFSck.getCurrentNamenodeAddress(DFSck.java:271)
> at org.apache.hadoop.hdfs.tools.DFSck.doWork(DFSck.java:339)
> at org.apache.hadoop.hdfs.tools.DFSck.access$000(DFSck.java:75)
> at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:164)
> at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:161)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
> at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:160)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:409)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For
[jira] [Commented] (HDFS-15543) RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount points with fault Tolerance enabled.
[ https://issues.apache.org/jira/browse/HDFS-15543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17209280#comment-17209280 ] Hemanth Boyina commented on HDFS-15543: --- thanks for the review [~elgoiri] committed to trunk > RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount > points with fault Tolerance enabled. > > > Key: HDFS-15543 > URL: https://issues.apache.org/jira/browse/HDFS-15543 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.1.1 >Reporter: Harshakiran Reddy >Assignee: Hemanth Boyina >Priority: Major > Attachments: HDFS-15543.001.patch, HDFS-15543.002.patch, > HDFS-15543.003.patch, HDFS-15543.004.patch, HDFS-15543.005.patch, > HDFS-15543.006.patch, HDFS-15543.007.patch, HDFS-15543_testrepro.patch > > > A RANDOM mount point should allow to creating new files if one subcluster is > down also with Fault Tolerance was enabled. but here it's failed. > MultiDestination_client]# hdfs dfsrouteradmin -ls /test_ec > *Mount Table Entries:* > Source Destinations Owner Group Mode Quota/Usage > /test_ec *hacluster->/tes_ec,hacluster1->/tes_ec* test ficommon rwxr-xr-x > [NsQuota: -/-, SsQuota: -/-] > *File Write throne the Exception:-* > 2020-08-26 19:13:21,839 WARN hdfs.DataStreamer: Abandoning blk_1073743375_2551 > 2020-08-26 19:13:21,877 WARN hdfs.DataStreamer: Excluding datanode > DatanodeInfoWithStorage[DISK] > 2020-08-26 19:13:21,878 WARN hdfs.DataStreamer: DataStreamer Exception > java.io.IOException: Unable to create new block. > at > org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1758) > at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:718) > 2020-08-26 19:13:21,879 WARN hdfs.DataStreamer: Could not get block > locations. Source file "/test_ec/f1._COPYING_" - Aborting...block==null > put: Could not get block locations. Source file "/test_ec/f1._COPYING_" - > Aborting...block==null -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15543) RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount points with fault Tolerance enabled.
[ https://issues.apache.org/jira/browse/HDFS-15543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Boyina updated HDFS-15543: -- Fix Version/s: 3.4.0 Resolution: Fixed Status: Resolved (was: Patch Available) > RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount > points with fault Tolerance enabled. > > > Key: HDFS-15543 > URL: https://issues.apache.org/jira/browse/HDFS-15543 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.1.1 >Reporter: Harshakiran Reddy >Assignee: Hemanth Boyina >Priority: Major > Fix For: 3.4.0 > > Attachments: HDFS-15543.001.patch, HDFS-15543.002.patch, > HDFS-15543.003.patch, HDFS-15543.004.patch, HDFS-15543.005.patch, > HDFS-15543.006.patch, HDFS-15543.007.patch, HDFS-15543_testrepro.patch > > > A RANDOM mount point should allow to creating new files if one subcluster is > down also with Fault Tolerance was enabled. but here it's failed. > MultiDestination_client]# hdfs dfsrouteradmin -ls /test_ec > *Mount Table Entries:* > Source Destinations Owner Group Mode Quota/Usage > /test_ec *hacluster->/tes_ec,hacluster1->/tes_ec* test ficommon rwxr-xr-x > [NsQuota: -/-, SsQuota: -/-] > *File Write throne the Exception:-* > 2020-08-26 19:13:21,839 WARN hdfs.DataStreamer: Abandoning blk_1073743375_2551 > 2020-08-26 19:13:21,877 WARN hdfs.DataStreamer: Excluding datanode > DatanodeInfoWithStorage[DISK] > 2020-08-26 19:13:21,878 WARN hdfs.DataStreamer: DataStreamer Exception > java.io.IOException: Unable to create new block. > at > org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1758) > at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:718) > 2020-08-26 19:13:21,879 WARN hdfs.DataStreamer: Could not get block > locations. Source file "/test_ec/f1._COPYING_" - Aborting...block==null > put: Could not get block locations. Source file "/test_ec/f1._COPYING_" - > Aborting...block==null -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13293) RBF: The RouterRPCServer should transfer CallerContext and client ip to NamenodeRpcServer
[
https://issues.apache.org/jira/browse/HDFS-13293?focusedWorklogId=496280=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-496280
]
ASF GitHub Bot logged work on HDFS-13293:
-
Author: ASF GitHub Bot
Created on: 07/Oct/20 03:55
Start Date: 07/Oct/20 03:55
Worklog Time Spent: 10m
Work Description: ferhui commented on a change in pull request #2363:
URL: https://github.com/apache/hadoop/pull/2363#discussion_r500722958
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java
##
@@ -1901,4 +1903,19 @@ private DFSClient getFileDFSClient(final String path) {
}
return null;
}
+
+ @Test
+ public void testCreateWithCallerContext() throws IOException {
+GenericTestUtils.LogCapturer auditlog =
+GenericTestUtils.LogCapturer.captureLogs(FSNamesystem.auditLog);
+
+// Create a directory via the router
+String dirPath = "/test_dir_with_callercontext";
+FsPermission permission = new FsPermission("705");
+routerProtocol.mkdirs(dirPath, permission, false);
+
+// The audit log should contains "callerContext=clientIp:"
+assertTrue(auditlog.getOutput().contains("callerContext=clientIp:"));
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 496280)
Time Spent: 1h 20m (was: 1h 10m)
> RBF: The RouterRPCServer should transfer CallerContext and client ip to
> NamenodeRpcServer
> -
>
> Key: HDFS-13293
> URL: https://issues.apache.org/jira/browse/HDFS-13293
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: maobaolong
>Assignee: Hui Fei
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-13293.001.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Otherwise, the namenode don't know the client's callerContext
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13293) RBF: The RouterRPCServer should transfer CallerContext and client ip to NamenodeRpcServer
[
https://issues.apache.org/jira/browse/HDFS-13293?focusedWorklogId=496279=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-496279
]
ASF GitHub Bot logged work on HDFS-13293:
-
Author: ASF GitHub Bot
Created on: 07/Oct/20 03:55
Start Date: 07/Oct/20 03:55
Worklog Time Spent: 10m
Work Description: ferhui commented on a change in pull request #2363:
URL: https://github.com/apache/hadoop/pull/2363#discussion_r500722838
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcClient.java
##
@@ -116,10 +118,13 @@
/** Optional perf monitor. */
private final RouterRpcMonitor rpcMonitor;
+ private final Configuration clientConf;
+
/** Pattern to parse a stack trace line. */
private static final Pattern STACK_TRACE_PATTERN =
Pattern.compile("\\tat (.*)\\.(.*)\\((.*):(\\d*)\\)");
+ private static final String CLIENT_IP_STR = "clientIp";
Review comment:
[HDFS-13248](https://issues.apache.org/jira/browse/HDFS-13248) focuses
on the locality, here we focus on audit log, does it make sense?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 496279)
Time Spent: 1h 10m (was: 1h)
> RBF: The RouterRPCServer should transfer CallerContext and client ip to
> NamenodeRpcServer
> -
>
> Key: HDFS-13293
> URL: https://issues.apache.org/jira/browse/HDFS-13293
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: maobaolong
>Assignee: Hui Fei
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-13293.001.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Otherwise, the namenode don't know the client's callerContext
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15253) Set default throttle value on dfs.image.transfer.bandwidthPerSec
[
https://issues.apache.org/jira/browse/HDFS-15253?focusedWorklogId=496275=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-496275
]
ASF GitHub Bot logged work on HDFS-15253:
-
Author: ASF GitHub Bot
Created on: 07/Oct/20 03:31
Start Date: 07/Oct/20 03:31
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2366:
URL: https://github.com/apache/hadoop/pull/2366#issuecomment-704670408
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 16s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 3s | | trunk passed |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 1m 9s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 5s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 49s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 22s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 10s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 8s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 8s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 1m 4s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 1m 4s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 48s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 8s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 16m 0s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 18s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 3m 17s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 110m 10s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2366/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 200m 21s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.TestFileChecksum |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2366/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2366 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 26ed9484402b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1cfe5916e20 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
[jira] [Work logged] (HDFS-15610) Reduce datanode upgrade/hardlink thread
[
https://issues.apache.org/jira/browse/HDFS-15610?focusedWorklogId=496271=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-496271
]
ASF GitHub Bot logged work on HDFS-15610:
-
Author: ASF GitHub Bot
Created on: 07/Oct/20 03:25
Start Date: 07/Oct/20 03:25
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2365:
URL: https://github.com/apache/hadoop/pull/2365#issuecomment-704669044
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 8s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 22s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 1m 8s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 50s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 17s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 19s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 17s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 4s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 1m 4s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 48s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 11s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 15m 36s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 3m 19s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 115m 28s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2365/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 207m 26s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2365/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2365 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux df8ebce07039 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Created] (HDFS-15616) [SBN] Disable trigger edit Log Roll for Observers
Janus Chow created HDFS-15616: - Summary: [SBN] Disable trigger edit Log Roll for Observers Key: HDFS-15616 URL: https://issues.apache.org/jira/browse/HDFS-15616 Project: Hadoop HDFS Issue Type: Improvement Reporter: Janus Chow Currently when Observer is transitioned from StandbyState, the editLogTailer will still send the request to roll editLog to ActiveNN, which should be disabled to keep the definition of " logRollPeriodMs" clear. One thing I'm not sure is for a cluster with multi standby Namenode, all the standby NN will trigger the roll. Should this feature be extended to all standby NNs or implementing on observers first? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15253) Set default throttle value on dfs.image.transfer.bandwidthPerSec
[ https://issues.apache.org/jira/browse/HDFS-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-15253: -- Labels: pull-request-available (was: ) > Set default throttle value on dfs.image.transfer.bandwidthPerSec > > > Key: HDFS-15253 > URL: https://issues.apache.org/jira/browse/HDFS-15253 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Karthik Palanisamy >Assignee: Karthik Palanisamy >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The default value dfs.image.transfer.bandwidthPerSec is set to 0 so it can > use maximum available bandwidth for fsimage transfers during checkpoint. I > think we should throttle this. Many users were experienced namenode failover > when transferring large image size along with fsimage replication on > dfs.namenode.name.dir. eg. >25Gb. > Thought to set, > dfs.image.transfer.bandwidthPerSec=52428800. (50 MB/s) > dfs.namenode.checkpoint.txns=200 (Default is 1M, good to avoid frequent > checkpoint. However, the default checkpoint runs every 6 hours once) > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15253) Set default throttle value on dfs.image.transfer.bandwidthPerSec
[ https://issues.apache.org/jira/browse/HDFS-15253?focusedWorklogId=496242=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-496242 ] ASF GitHub Bot logged work on HDFS-15253: - Author: ASF GitHub Bot Created on: 07/Oct/20 00:10 Start Date: 07/Oct/20 00:10 Worklog Time Spent: 10m Work Description: karthikhw opened a new pull request #2366: URL: https://github.com/apache/hadoop/pull/2366 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 496242) Remaining Estimate: 0h Time Spent: 10m > Set default throttle value on dfs.image.transfer.bandwidthPerSec > > > Key: HDFS-15253 > URL: https://issues.apache.org/jira/browse/HDFS-15253 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Karthik Palanisamy >Assignee: Karthik Palanisamy >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The default value dfs.image.transfer.bandwidthPerSec is set to 0 so it can > use maximum available bandwidth for fsimage transfers during checkpoint. I > think we should throttle this. Many users were experienced namenode failover > when transferring large image size along with fsimage replication on > dfs.namenode.name.dir. eg. >25Gb. > Thought to set, > dfs.image.transfer.bandwidthPerSec=52428800. (50 MB/s) > dfs.namenode.checkpoint.txns=200 (Default is 1M, good to avoid frequent > checkpoint. However, the default checkpoint runs every 6 hours once) > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15610) Reduce datanode upgrade/hardlink thread
[ https://issues.apache.org/jira/browse/HDFS-15610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-15610: -- Labels: pull-request-available (was: ) > Reduce datanode upgrade/hardlink thread > --- > > Key: HDFS-15610 > URL: https://issues.apache.org/jira/browse/HDFS-15610 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 3.1.4 >Reporter: Karthik Palanisamy >Assignee: Karthik Palanisamy >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > There is a kernel overhead on datanode upgrade. If datanode with millions of > blocks and 10+ disks then block-layout migration will be super expensive > during its hardlink operation. Slowness is observed when running with large > hardlink threads(dfs.datanode.block.id.layout.upgrade.threads, default is 12 > thread for each disk) and its runs for 2+ hours. > I.e 10*12=120 threads (for 10 disks) > Small test: > RHEL7, 32 cores, 20 GB RAM, 8 GB DN heap > ||dfs.datanode.block.id.layout.upgrade.threads||Blocks||Disks||Time taken|| > |12|3.3 Million|1|2 minutes and 59 seconds| > |6|3.3 Million|1|2 minutes and 35 seconds| > |3|3.3 Million|1|2 minutes and 51 seconds| > Tried same test twice and 95% is accurate (only a few sec difference on each > iteration). Using 6 thread is faster than 12 thread because of its overhead. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15610) Reduce datanode upgrade/hardlink thread
[ https://issues.apache.org/jira/browse/HDFS-15610?focusedWorklogId=496239=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-496239 ] ASF GitHub Bot logged work on HDFS-15610: - Author: ASF GitHub Bot Created on: 06/Oct/20 23:57 Start Date: 06/Oct/20 23:57 Worklog Time Spent: 10m Work Description: karthikhw opened a new pull request #2365: URL: https://github.com/apache/hadoop/pull/2365 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 496239) Remaining Estimate: 0h Time Spent: 10m > Reduce datanode upgrade/hardlink thread > --- > > Key: HDFS-15610 > URL: https://issues.apache.org/jira/browse/HDFS-15610 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 3.1.4 >Reporter: Karthik Palanisamy >Assignee: Karthik Palanisamy >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > There is a kernel overhead on datanode upgrade. If datanode with millions of > blocks and 10+ disks then block-layout migration will be super expensive > during its hardlink operation. Slowness is observed when running with large > hardlink threads(dfs.datanode.block.id.layout.upgrade.threads, default is 12 > thread for each disk) and its runs for 2+ hours. > I.e 10*12=120 threads (for 10 disks) > Small test: > RHEL7, 32 cores, 20 GB RAM, 8 GB DN heap > ||dfs.datanode.block.id.layout.upgrade.threads||Blocks||Disks||Time taken|| > |12|3.3 Million|1|2 minutes and 59 seconds| > |6|3.3 Million|1|2 minutes and 35 seconds| > |3|3.3 Million|1|2 minutes and 51 seconds| > Tried same test twice and 95% is accurate (only a few sec difference on each > iteration). Using 6 thread is faster than 12 thread because of its overhead. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15615) HDFS user permission does not work correctly
[
https://issues.apache.org/jira/browse/HDFS-15615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heba Aboulmagd resolved HDFS-15615.
---
Release Note: adding users to hdfs group, causes the permission to not work
correctly
Resolution: Resolved
> HDFS user permission does not work correctly
>
>
> Key: HDFS-15615
> URL: https://issues.apache.org/jira/browse/HDFS-15615
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Heba Aboulmagd
>Priority: Major
>
> HDFS users can create/delete folders for each other, while the permission is
> set to {{745}} on {{/user/username}}.
> I used the following commands to create users and set their permissions
> {{useradd -m -g hdfs username}}
> {{su hdfs}}
> {{hdfs dfs -mkdir -p /user/username}}
> {{hdfs dfs -chown username:hdfs /user/username}}
> {{hdfs dfs -chmod 745 /user/username}}
> when {{user1}} executes: {{hdfs dfs -mkdir /user/user2/from_user1}}
> {{The directory is created successfully}}
> The same happens when {{user1}} tries to delete files from {{/user/user2}}
> while the permission is set to read only for the users in the same group.g
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13293) RBF: The RouterRPCServer should transfer CallerContext and client ip to NamenodeRpcServer
[
https://issues.apache.org/jira/browse/HDFS-13293?focusedWorklogId=496128=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-496128
]
ASF GitHub Bot logged work on HDFS-13293:
-
Author: ASF GitHub Bot
Created on: 06/Oct/20 19:23
Start Date: 06/Oct/20 19:23
Worklog Time Spent: 10m
Work Description: goiri commented on a change in pull request #2363:
URL: https://github.com/apache/hadoop/pull/2363#discussion_r500540849
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcClient.java
##
@@ -116,10 +118,13 @@
/** Optional perf monitor. */
private final RouterRpcMonitor rpcMonitor;
+ private final Configuration clientConf;
+
/** Pattern to parse a stack trace line. */
private static final Pattern STACK_TRACE_PATTERN =
Pattern.compile("\\tat (.*)\\.(.*)\\((.*):(\\d*)\\)");
+ private static final String CLIENT_IP_STR = "clientIp";
Review comment:
One of the things that was brought up in the past was to send back in
the context which was the namespace that executed this.
Anyway, let's focus on the locality one for now.
Does it make sense to add a test to make sure we can do proper locality from
the Namenode side with this change? Or should we leave this for the future.
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java
##
@@ -1901,4 +1903,19 @@ private DFSClient getFileDFSClient(final String path) {
}
return null;
}
+
+ @Test
+ public void testCreateWithCallerContext() throws IOException {
+GenericTestUtils.LogCapturer auditlog =
+GenericTestUtils.LogCapturer.captureLogs(FSNamesystem.auditLog);
+
+// Create a directory via the router
+String dirPath = "/test_dir_with_callercontext";
+FsPermission permission = new FsPermission("705");
+routerProtocol.mkdirs(dirPath, permission, false);
+
+// The audit log should contains "callerContext=clientIp:"
+assertTrue(auditlog.getOutput().contains("callerContext=clientIp:"));
Review comment:
I was referring to having this test with a null context but also a test
with some already passed context.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 496128)
Time Spent: 1h (was: 50m)
> RBF: The RouterRPCServer should transfer CallerContext and client ip to
> NamenodeRpcServer
> -
>
> Key: HDFS-13293
> URL: https://issues.apache.org/jira/browse/HDFS-13293
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: maobaolong
>Assignee: Hui Fei
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-13293.001.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Otherwise, the namenode don't know the client's callerContext
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13293) RBF: The RouterRPCServer should transfer CallerContext and client ip to NamenodeRpcServer
[
https://issues.apache.org/jira/browse/HDFS-13293?focusedWorklogId=496107=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-496107
]
ASF GitHub Bot logged work on HDFS-13293:
-
Author: ASF GitHub Bot
Created on: 06/Oct/20 18:52
Start Date: 06/Oct/20 18:52
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2363:
URL: https://github.com/apache/hadoop/pull/2363#issuecomment-704482782
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 25s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 3s | | trunk passed |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 33s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 41s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 39s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 44s | | the patch passed |
| +1 :green_heart: | compile | 0m 47s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 47s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| -0 :warning: | checkstyle | 0m 21s |
[/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2363/3/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 41s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 17m 28s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 6s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 51s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 13m 59s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2363/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 106m 3s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterRpc |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2363/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2363 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6d1d7495d180 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c9ea344f984 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
[jira] [Updated] (HDFS-15611) Add list Snapshot command in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-15611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-15611: -- Status: Patch Available (was: Open) > Add list Snapshot command in WebHDFS > > > Key: HDFS-15611 > URL: https://issues.apache.org/jira/browse/HDFS-15611 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Idea here is to expose lsSnapshot cmd over WebHdfs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13293) RBF: The RouterRPCServer should transfer CallerContext and client ip to NamenodeRpcServer
[
https://issues.apache.org/jira/browse/HDFS-13293?focusedWorklogId=496040=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-496040
]
ASF GitHub Bot logged work on HDFS-13293:
-
Author: ASF GitHub Bot
Created on: 06/Oct/20 17:17
Start Date: 06/Oct/20 17:17
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2363:
URL: https://github.com/apache/hadoop/pull/2363#issuecomment-704425407
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 6s | | trunk passed |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 23s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 46s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 29s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 25s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| -0 :warning: | checkstyle | 0m 20s |
[/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2363/2/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 56s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 41s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 34s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 10m 11s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 91m 4s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2363/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2363 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6944466a4ff6 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4347a5c9556 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2363/2/testReport/ |
| Max. process+thread count | 3068 (vs. ulimit of 5500) |
[jira] [Commented] (HDFS-15543) RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount points with fault Tolerance enabled.
[ https://issues.apache.org/jira/browse/HDFS-15543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17208865#comment-17208865 ] Íñigo Goiri commented on HDFS-15543: Thanks [~hemanthboyina] for [^HDFS-15543.007.patch]. +1 on [^HDFS-15543.007.patch]. > RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount > points with fault Tolerance enabled. > > > Key: HDFS-15543 > URL: https://issues.apache.org/jira/browse/HDFS-15543 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.1.1 >Reporter: Harshakiran Reddy >Assignee: Hemanth Boyina >Priority: Major > Attachments: HDFS-15543.001.patch, HDFS-15543.002.patch, > HDFS-15543.003.patch, HDFS-15543.004.patch, HDFS-15543.005.patch, > HDFS-15543.006.patch, HDFS-15543.007.patch, HDFS-15543_testrepro.patch > > > A RANDOM mount point should allow to creating new files if one subcluster is > down also with Fault Tolerance was enabled. but here it's failed. > MultiDestination_client]# hdfs dfsrouteradmin -ls /test_ec > *Mount Table Entries:* > Source Destinations Owner Group Mode Quota/Usage > /test_ec *hacluster->/tes_ec,hacluster1->/tes_ec* test ficommon rwxr-xr-x > [NsQuota: -/-, SsQuota: -/-] > *File Write throne the Exception:-* > 2020-08-26 19:13:21,839 WARN hdfs.DataStreamer: Abandoning blk_1073743375_2551 > 2020-08-26 19:13:21,877 WARN hdfs.DataStreamer: Excluding datanode > DatanodeInfoWithStorage[DISK] > 2020-08-26 19:13:21,878 WARN hdfs.DataStreamer: DataStreamer Exception > java.io.IOException: Unable to create new block. > at > org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1758) > at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:718) > 2020-08-26 19:13:21,879 WARN hdfs.DataStreamer: Could not get block > locations. Source file "/test_ec/f1._COPYING_" - Aborting...block==null > put: Could not get block locations. Source file "/test_ec/f1._COPYING_" - > Aborting...block==null -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13293) RBF: The RouterRPCServer should transfer CallerContext and client ip to NamenodeRpcServer
[ https://issues.apache.org/jira/browse/HDFS-13293?focusedWorklogId=495971=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-495971 ] ASF GitHub Bot logged work on HDFS-13293: - Author: ASF GitHub Bot Created on: 06/Oct/20 15:51 Start Date: 06/Oct/20 15:51 Worklog Time Spent: 10m Work Description: ferhui commented on pull request #2363: URL: https://github.com/apache/hadoop/pull/2363#issuecomment-704372005 Fix checkstyle, failed tests pass locally This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 495971) Time Spent: 0.5h (was: 20m) > RBF: The RouterRPCServer should transfer CallerContext and client ip to > NamenodeRpcServer > - > > Key: HDFS-13293 > URL: https://issues.apache.org/jira/browse/HDFS-13293 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: maobaolong >Assignee: Hui Fei >Priority: Major > Labels: pull-request-available > Attachments: HDFS-13293.001.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > Otherwise, the namenode don't know the client's callerContext -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13293) RBF: The RouterRPCServer should transfer CallerContext and client ip to NamenodeRpcServer
[
https://issues.apache.org/jira/browse/HDFS-13293?focusedWorklogId=495947=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-495947
]
ASF GitHub Bot logged work on HDFS-13293:
-
Author: ASF GitHub Bot
Created on: 06/Oct/20 15:10
Start Date: 06/Oct/20 15:10
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2363:
URL: https://github.com/apache/hadoop/pull/2363#issuecomment-704337731
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 45s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 41s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 59s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 23s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 21s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| -0 :warning: | checkstyle | 0m 19s |
[/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2363/1/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 2 new + 0
unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 30s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 39s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 52s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 24s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 9m 39s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2363/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 86m 8s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.federation.security.token.TestZKDelegationTokenSecretManagerImpl
|
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2363/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2363 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 7785ca08b9cb 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4347a5c9556 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions
[jira] [Commented] (HDFS-15543) RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount points with fault Tolerance enabled.
[
https://issues.apache.org/jira/browse/HDFS-15543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17208802#comment-17208802
]
Hadoop QA commented on HDFS-15543:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
30s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 1 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
40s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 23s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
12s{color} | | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 34s{color} | | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
27s{color} | | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense
[jira] [Commented] (HDFS-15543) RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount points with fault Tolerance enabled.
[ https://issues.apache.org/jira/browse/HDFS-15543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17208732#comment-17208732 ] Hemanth Boyina commented on HDFS-15543: --- thanks for the review [~elgoiri] updated the patch , please review > RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount > points with fault Tolerance enabled. > > > Key: HDFS-15543 > URL: https://issues.apache.org/jira/browse/HDFS-15543 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.1.1 >Reporter: Harshakiran Reddy >Assignee: Hemanth Boyina >Priority: Major > Attachments: HDFS-15543.001.patch, HDFS-15543.002.patch, > HDFS-15543.003.patch, HDFS-15543.004.patch, HDFS-15543.005.patch, > HDFS-15543.006.patch, HDFS-15543.007.patch, HDFS-15543_testrepro.patch > > > A RANDOM mount point should allow to creating new files if one subcluster is > down also with Fault Tolerance was enabled. but here it's failed. > MultiDestination_client]# hdfs dfsrouteradmin -ls /test_ec > *Mount Table Entries:* > Source Destinations Owner Group Mode Quota/Usage > /test_ec *hacluster->/tes_ec,hacluster1->/tes_ec* test ficommon rwxr-xr-x > [NsQuota: -/-, SsQuota: -/-] > *File Write throne the Exception:-* > 2020-08-26 19:13:21,839 WARN hdfs.DataStreamer: Abandoning blk_1073743375_2551 > 2020-08-26 19:13:21,877 WARN hdfs.DataStreamer: Excluding datanode > DatanodeInfoWithStorage[DISK] > 2020-08-26 19:13:21,878 WARN hdfs.DataStreamer: DataStreamer Exception > java.io.IOException: Unable to create new block. > at > org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1758) > at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:718) > 2020-08-26 19:13:21,879 WARN hdfs.DataStreamer: Could not get block > locations. Source file "/test_ec/f1._COPYING_" - Aborting...block==null > put: Could not get block locations. Source file "/test_ec/f1._COPYING_" - > Aborting...block==null -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13293) RBF: The RouterRPCServer should transfer CallerContext and client ip to NamenodeRpcServer
[ https://issues.apache.org/jira/browse/HDFS-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17208760#comment-17208760 ] Hui Fei commented on HDFS-13293: After HADOOP-17276, router could add client ip to the callercontext. [~aajisaka][~elgoiri], Could you please review the github PR? Thanks. > RBF: The RouterRPCServer should transfer CallerContext and client ip to > NamenodeRpcServer > - > > Key: HDFS-13293 > URL: https://issues.apache.org/jira/browse/HDFS-13293 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: maobaolong >Assignee: Hui Fei >Priority: Major > Labels: pull-request-available > Attachments: HDFS-13293.001.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Otherwise, the namenode don't know the client's callerContext -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15543) RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount points with fault Tolerance enabled.
[ https://issues.apache.org/jira/browse/HDFS-15543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Boyina updated HDFS-15543: -- Attachment: HDFS-15543.007.patch > RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount > points with fault Tolerance enabled. > > > Key: HDFS-15543 > URL: https://issues.apache.org/jira/browse/HDFS-15543 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.1.1 >Reporter: Harshakiran Reddy >Assignee: Hemanth Boyina >Priority: Major > Attachments: HDFS-15543.001.patch, HDFS-15543.002.patch, > HDFS-15543.003.patch, HDFS-15543.004.patch, HDFS-15543.005.patch, > HDFS-15543.006.patch, HDFS-15543.007.patch, HDFS-15543_testrepro.patch > > > A RANDOM mount point should allow to creating new files if one subcluster is > down also with Fault Tolerance was enabled. but here it's failed. > MultiDestination_client]# hdfs dfsrouteradmin -ls /test_ec > *Mount Table Entries:* > Source Destinations Owner Group Mode Quota/Usage > /test_ec *hacluster->/tes_ec,hacluster1->/tes_ec* test ficommon rwxr-xr-x > [NsQuota: -/-, SsQuota: -/-] > *File Write throne the Exception:-* > 2020-08-26 19:13:21,839 WARN hdfs.DataStreamer: Abandoning blk_1073743375_2551 > 2020-08-26 19:13:21,877 WARN hdfs.DataStreamer: Excluding datanode > DatanodeInfoWithStorage[DISK] > 2020-08-26 19:13:21,878 WARN hdfs.DataStreamer: DataStreamer Exception > java.io.IOException: Unable to create new block. > at > org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1758) > at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:718) > 2020-08-26 19:13:21,879 WARN hdfs.DataStreamer: Could not get block > locations. Source file "/test_ec/f1._COPYING_" - Aborting...block==null > put: Could not get block locations. Source file "/test_ec/f1._COPYING_" - > Aborting...block==null -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13293) RBF: The RouterRPCServer should transfer CallerContext and client ip to NamenodeRpcServer
[ https://issues.apache.org/jira/browse/HDFS-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-13293: -- Labels: pull-request-available (was: ) > RBF: The RouterRPCServer should transfer CallerContext and client ip to > NamenodeRpcServer > - > > Key: HDFS-13293 > URL: https://issues.apache.org/jira/browse/HDFS-13293 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: maobaolong >Assignee: Hui Fei >Priority: Major > Labels: pull-request-available > Attachments: HDFS-13293.001.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Otherwise, the namenode don't know the client's callerContext -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13293) RBF: The RouterRPCServer should transfer CallerContext and client ip to NamenodeRpcServer
[ https://issues.apache.org/jira/browse/HDFS-13293?focusedWorklogId=495911=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-495911 ] ASF GitHub Bot logged work on HDFS-13293: - Author: ASF GitHub Bot Created on: 06/Oct/20 13:42 Start Date: 06/Oct/20 13:42 Worklog Time Spent: 10m Work Description: ferhui opened a new pull request #2363: URL: https://github.com/apache/hadoop/pull/2363 …d client ip to NamenodeRpcServer ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 495911) Remaining Estimate: 0h Time Spent: 10m > RBF: The RouterRPCServer should transfer CallerContext and client ip to > NamenodeRpcServer > - > > Key: HDFS-13293 > URL: https://issues.apache.org/jira/browse/HDFS-13293 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: maobaolong >Assignee: Hui Fei >Priority: Major > Labels: pull-request-available > Attachments: HDFS-13293.001.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Otherwise, the namenode don't know the client's callerContext -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15613) RBF: Router FSCK fails after HDFS-14442
[
https://issues.apache.org/jira/browse/HDFS-15613?focusedWorklogId=495772=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-495772
]
ASF GitHub Bot logged work on HDFS-15613:
-
Author: ASF GitHub Bot
Created on: 06/Oct/20 07:53
Start Date: 06/Oct/20 07:53
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2360:
URL: https://github.com/apache/hadoop/pull/2360#issuecomment-704097550
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 24s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 42s | | trunk passed |
| +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 34s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 49s | | trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 44s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 42s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 38s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 41s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 17m 15s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 50s | | the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK
Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 52s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 13m 18s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 105m 7s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2360/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2360 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 183a5e9dbcd5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4347a5c9556 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2360/5/testReport/ |
| Max. process+thread count | 3068 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2360/5/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus
[jira] [Work logged] (HDFS-15613) RBF: Router FSCK fails after HDFS-14442
[
https://issues.apache.org/jira/browse/HDFS-15613?focusedWorklogId=495738=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-495738
]
ASF GitHub Bot logged work on HDFS-15613:
-
Author: ASF GitHub Bot
Created on: 06/Oct/20 06:32
Start Date: 06/Oct/20 06:32
Worklog Time Spent: 10m
Work Description: aajisaka commented on pull request #2360:
URL: https://github.com/apache/hadoop/pull/2360#issuecomment-704061579
> hadoop.hdfs.server.federation.router.TestRouterClientRejectOverload
The test failure is not related and the test passed in my local environment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 495738)
Time Spent: 1h 20m (was: 1h 10m)
> RBF: Router FSCK fails after HDFS-14442
> ---
>
> Key: HDFS-15613
> URL: https://issues.apache.org/jira/browse/HDFS-15613
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Affects Versions: 3.3.0
> Environment: HA is enabled
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> After HDFS-14442 fsck uses getHAServiceState operation to detect Active
> NameNode, however, DFSRouter does not support the operation.
> {noformat}
> 20/10/05 16:41:30 DEBUG hdfs.HAUtil: Error while connecting to namenode
> org.apache.hadoop.ipc.RemoteException(java.lang.UnsupportedOperationException):
> Operation "getHAServiceState" is not supported
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.checkOperation(RouterRpcServer.java:488)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterClientProtocol.getHAServiceState(RouterClientProtocol.java:1773)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getHAServiceState(RouterRpcServer.java:1333)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getHAServiceState(ClientNamenodeProtocolServerSideTranslatorPB.java:2011)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:532)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1020)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:948)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2952)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562)
> at org.apache.hadoop.ipc.Client.call(Client.java:1508)
> at org.apache.hadoop.ipc.Client.call(Client.java:1405)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:234)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:119)
> at com.sun.proxy.$Proxy12.getHAServiceState(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getHAServiceState(ClientNamenodeProtocolTranslatorPB.java:2055)
> at org.apache.hadoop.hdfs.HAUtil.getAddressOfActive(HAUtil.java:281)
> at
> org.apache.hadoop.hdfs.tools.DFSck.getCurrentNamenodeAddress(DFSck.java:271)
> at org.apache.hadoop.hdfs.tools.DFSck.doWork(DFSck.java:339)
> at org.apache.hadoop.hdfs.tools.DFSck.access$000(DFSck.java:75)
> at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:164)
> at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:161)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
> at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:160)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:409)
> {noformat}
--
This message
