[jira] [Commented] (HDFS-15839) RBF: Cannot get method setBalancerBandwidth on Router Client
[
https://issues.apache.org/jira/browse/HDFS-15839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286309#comment-17286309
]
Hadoop QA commented on HDFS-15839:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} yetus {color} | {color:red} 0m 7s{color}
| {color:red}{color} | {color:red} Unprocessed flag(s):
--findbugs-strict-precheck {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/479/artifact/out/Dockerfile
|
| JIRA Issue | HDFS-15839 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13020611/HDFS-15839.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/479/console |
| versions | git=2.25.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> RBF: Cannot get method setBalancerBandwidth on Router Client
>
>
> Key: HDFS-15839
> URL: https://issues.apache.org/jira/browse/HDFS-15839
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: Yang Yun
>Assignee: Yang Yun
>Priority: Major
> Attachments: HDFS-15839.patch
>
>
> When call setBalancerBandwidth, throw exeption,
> {code:java}
> 02-18 14:39:59,186 [IPC Server handler 0 on default port 43545] ERROR
> router.RemoteMethod (RemoteMethod.java:getMethod(146)) - Cannot get method
> setBalancerBandwidth with types [class java.lang.Long] from
> ClientProtocoljava.lang.NoSuchMethodException:
> org.apache.hadoop.hdfs.protocol.ClientProtocol.setBalancerBandwidth(java.lang.Long)
> at java.lang.Class.getDeclaredMethod(Class.java:2130) at
> org.apache.hadoop.hdfs.server.federation.router.RemoteMethod.getMethod(RemoteMethod.java:140)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient.invokeConcurrent(RouterRpcClient.java:1312)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient.invokeConcurrent(RouterRpcClient.java:1250)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient.invokeConcurrent(RouterRpcClient.java:1221)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient.invokeConcurrent(RouterRpcClient.java:1194)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterClientProtocol.setBalancerBandwidth(RouterClientProtocol.java:1188)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.setBalancerBandwidth(RouterRpcServer.java:1211)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setBalancerBandwidth(ClientNamenodeProtocolServerSideTranslatorPB.java:1254)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:537)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1086) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1037) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:965) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2972){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15839) RBF: Cannot get method setBalancerBandwidth on Router Client
[
https://issues.apache.org/jira/browse/HDFS-15839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Yun updated HDFS-15839:
Attachment: HDFS-15839.patch
Status: Patch Available (was: Open)
> RBF: Cannot get method setBalancerBandwidth on Router Client
>
>
> Key: HDFS-15839
> URL: https://issues.apache.org/jira/browse/HDFS-15839
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: Yang Yun
>Assignee: Yang Yun
>Priority: Major
> Attachments: HDFS-15839.patch
>
>
> When call setBalancerBandwidth, throw exeption,
> {code:java}
> 02-18 14:39:59,186 [IPC Server handler 0 on default port 43545] ERROR
> router.RemoteMethod (RemoteMethod.java:getMethod(146)) - Cannot get method
> setBalancerBandwidth with types [class java.lang.Long] from
> ClientProtocoljava.lang.NoSuchMethodException:
> org.apache.hadoop.hdfs.protocol.ClientProtocol.setBalancerBandwidth(java.lang.Long)
> at java.lang.Class.getDeclaredMethod(Class.java:2130) at
> org.apache.hadoop.hdfs.server.federation.router.RemoteMethod.getMethod(RemoteMethod.java:140)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient.invokeConcurrent(RouterRpcClient.java:1312)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient.invokeConcurrent(RouterRpcClient.java:1250)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient.invokeConcurrent(RouterRpcClient.java:1221)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient.invokeConcurrent(RouterRpcClient.java:1194)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterClientProtocol.setBalancerBandwidth(RouterClientProtocol.java:1188)
> at
> org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.setBalancerBandwidth(RouterRpcServer.java:1211)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setBalancerBandwidth(ClientNamenodeProtocolServerSideTranslatorPB.java:1254)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:537)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1086) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1037) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:965) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2972){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15839) RBF: Cannot get method setBalancerBandwidth on Router Client
Yang Yun created HDFS-15839:
---
Summary: RBF: Cannot get method setBalancerBandwidth on Router
Client
Key: HDFS-15839
URL: https://issues.apache.org/jira/browse/HDFS-15839
Project: Hadoop HDFS
Issue Type: Bug
Components: rbf
Reporter: Yang Yun
Assignee: Yang Yun
When call setBalancerBandwidth, throw exeption,
{code:java}
02-18 14:39:59,186 [IPC Server handler 0 on default port 43545] ERROR
router.RemoteMethod (RemoteMethod.java:getMethod(146)) - Cannot get method
setBalancerBandwidth with types [class java.lang.Long] from
ClientProtocoljava.lang.NoSuchMethodException:
org.apache.hadoop.hdfs.protocol.ClientProtocol.setBalancerBandwidth(java.lang.Long)
at java.lang.Class.getDeclaredMethod(Class.java:2130) at
org.apache.hadoop.hdfs.server.federation.router.RemoteMethod.getMethod(RemoteMethod.java:140)
at
org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient.invokeConcurrent(RouterRpcClient.java:1312)
at
org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient.invokeConcurrent(RouterRpcClient.java:1250)
at
org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient.invokeConcurrent(RouterRpcClient.java:1221)
at
org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient.invokeConcurrent(RouterRpcClient.java:1194)
at
org.apache.hadoop.hdfs.server.federation.router.RouterClientProtocol.setBalancerBandwidth(RouterClientProtocol.java:1188)
at
org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.setBalancerBandwidth(RouterRpcServer.java:1211)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setBalancerBandwidth(ClientNamenodeProtocolServerSideTranslatorPB.java:1254)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:537)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1086) at
org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1037) at
org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:965) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2972){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15785) Datanode to support using DNS to resolve nameservices to IP addresses to get list of namenodes
[
https://issues.apache.org/jira/browse/HDFS-15785?focusedWorklogId=554083=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554083
]
ASF GitHub Bot logged work on HDFS-15785:
-
Author: ASF GitHub Bot
Created on: 18/Feb/21 06:15
Start Date: 18/Feb/21 06:15
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2639:
URL: https://github.com/apache/hadoop/pull/2639#issuecomment-781085217
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 24m 4s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 55s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 1s | | trunk passed |
| +1 :green_heart: | compile | 22m 30s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 19m 4s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 2s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 36s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 45s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 44s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +0 :ok: | spotbugs | 39m 21s | | Both FindBugs and SpotBugs are
enabled, using SpotBugs. |
| +1 :green_heart: | spotbugs | 8m 19s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 54s | | the patch passed |
| +1 :green_heart: | compile | 21m 53s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 21m 53s | |
root-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 0 new + 1957 unchanged - 2
fixed = 1957 total (was 1959) |
| +1 :green_heart: | compile | 19m 7s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 19m 7s | |
root-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 0 new + 1853 unchanged -
2 fixed = 1853 total (was 1855) |
| +1 :green_heart: | checkstyle | 4m 5s | | root: The patch generated
0 new + 558 unchanged - 2 fixed = 558 total (was 560) |
| +1 :green_heart: | mvnsite | 3m 58s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 15m 30s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 40s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 3m 50s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| -1 :x: | spotbugs | 2m 18s |
[/patch-spotbugs-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2639/6/artifact/out/patch-spotbugs-hadoop-common-project_hadoop-common.txt)
| hadoop-common-project/hadoop-common cannot run computeBugHistory from
spotbugs |
| -1 :x: | spotbugs | 2m 41s |
[/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2639/6/artifact/out/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt)
| hadoop-hdfs-project/hadoop-hdfs-client cannot run computeBugHistory from
spotbugs |
| -1 :x: | spotbugs | 3m 21s |
[/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2639/6/artifact/out/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs cannot run computeBugHistory from spotbugs
|
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 9s | | hadoop-common in the patch
passed. |
| +1 :green_heart:
[jira] [Commented] (HDFS-15793) Add command to DFSAdmin for Balancer max concurrent threads
[ https://issues.apache.org/jira/browse/HDFS-15793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286295#comment-17286295 ] Yang Yun commented on HDFS-15793: - Thanks [~ayushtkn] for your review. Sorry for the last patch lost some functions, updated to HDFS-15793.002.patch with following changes, * Add logical to process comand 'DNA_BALANCERBANDWIDTHUPDATE' in BPOfferService and call updateBalancerMaxConcurrentMovers of DataXceiverServer to make change thread bother smoothly. * Add more test to check all datanode or call '-getBalancerMaxThreads' to make sure that actaully the threads increased/decreased. * Fix the bug in router code and add a test for Router. > Add command to DFSAdmin for Balancer max concurrent threads > > > Key: HDFS-15793 > URL: https://issues.apache.org/jira/browse/HDFS-15793 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer mover >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15793.001.patch, HDFS-15793.002.patch > > > We have DFSAdmin command '-setBalancerBandwidth' to dynamically change the > max number of bytes per second of network bandwidth to be used by a datanode > during balancing. Also add '-setBalancerMaxThreads' to dynamically change > the balancer maxThread number which may be used concurrently for moving > blocks. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15793) Add command to DFSAdmin for Balancer max concurrent threads
[
https://issues.apache.org/jira/browse/HDFS-15793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286294#comment-17286294
]
Hadoop QA commented on HDFS-15793:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
40s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} yetus {color} | {color:red} 0m 9s{color}
| {color:red}{color} | {color:red} Unprocessed flag(s):
--findbugs-strict-precheck {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/478/artifact/out/Dockerfile
|
| JIRA Issue | HDFS-15793 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13020605/HDFS-15793.002.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/478/console |
| versions | git=2.25.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Add command to DFSAdmin for Balancer max concurrent threads
>
>
> Key: HDFS-15793
> URL: https://issues.apache.org/jira/browse/HDFS-15793
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer mover
>Reporter: Yang Yun
>Assignee: Yang Yun
>Priority: Minor
> Attachments: HDFS-15793.001.patch, HDFS-15793.002.patch
>
>
> We have DFSAdmin command '-setBalancerBandwidth' to dynamically change the
> max number of bytes per second of network bandwidth to be used by a datanode
> during balancing. Also add '-setBalancerMaxThreads' to dynamically change
> the balancer maxThread number which may be used concurrently for moving
> blocks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15793) Add command to DFSAdmin for Balancer max concurrent threads
[ https://issues.apache.org/jira/browse/HDFS-15793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15793: Attachment: HDFS-15793.002.patch Status: Patch Available (was: Open) > Add command to DFSAdmin for Balancer max concurrent threads > > > Key: HDFS-15793 > URL: https://issues.apache.org/jira/browse/HDFS-15793 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer mover >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15793.001.patch, HDFS-15793.002.patch > > > We have DFSAdmin command '-setBalancerBandwidth' to dynamically change the > max number of bytes per second of network bandwidth to be used by a datanode > during balancing. Also add '-setBalancerMaxThreads' to dynamically change > the balancer maxThread number which may be used concurrently for moving > blocks. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15793) Add command to DFSAdmin for Balancer max concurrent threads
[ https://issues.apache.org/jira/browse/HDFS-15793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15793: Status: Open (was: Patch Available) > Add command to DFSAdmin for Balancer max concurrent threads > > > Key: HDFS-15793 > URL: https://issues.apache.org/jira/browse/HDFS-15793 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer mover >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15793.001.patch > > > We have DFSAdmin command '-setBalancerBandwidth' to dynamically change the > max number of bytes per second of network bandwidth to be used by a datanode > during balancing. Also add '-setBalancerMaxThreads' to dynamically change > the balancer maxThread number which may be used concurrently for moving > blocks. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15785) Datanode to support using DNS to resolve nameservices to IP addresses to get list of namenodes
[
https://issues.apache.org/jira/browse/HDFS-15785?focusedWorklogId=554070=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554070
]
ASF GitHub Bot logged work on HDFS-15785:
-
Author: ASF GitHub Bot
Created on: 18/Feb/21 05:14
Start Date: 18/Feb/21 05:14
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2639:
URL: https://github.com/apache/hadoop/pull/2639#issuecomment-781060345
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 56s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 14s | | trunk passed |
| +1 :green_heart: | compile | 22m 44s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 19m 39s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 3m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 13s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 1s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 3m 12s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 11s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +0 :ok: | spotbugs | 37m 32s | | Both FindBugs and SpotBugs are
enabled, using SpotBugs. |
| +1 :green_heart: | spotbugs | 8m 9s | | trunk passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 51s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 50s | | the patch passed |
| +1 :green_heart: | compile | 19m 58s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 19m 58s | |
root-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 0 new + 1957 unchanged - 2
fixed = 1957 total (was 1959) |
| +1 :green_heart: | compile | 17m 56s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 17m 56s | |
root-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 generated 0 new + 1852 unchanged -
2 fixed = 1852 total (was 1854) |
| +1 :green_heart: | checkstyle | 3m 56s | | root: The patch generated
0 new + 557 unchanged - 2 fixed = 557 total (was 559) |
| +1 :green_heart: | mvnsite | 4m 11s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | shadedclient | 13m 19s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 3m 1s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 5s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| -1 :x: | spotbugs | 2m 21s |
[/patch-spotbugs-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2639/5/artifact/out/patch-spotbugs-hadoop-common-project_hadoop-common.txt)
| hadoop-common-project/hadoop-common cannot run computeBugHistory from
spotbugs |
| -1 :x: | spotbugs | 2m 37s |
[/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2639/5/artifact/out/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt)
| hadoop-hdfs-project/hadoop-hdfs-client cannot run computeBugHistory from
spotbugs |
| -1 :x: | spotbugs | 3m 17s |
[/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2639/5/artifact/out/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs cannot run computeBugHistory from spotbugs
|
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 13s | | hadoop-common in the patch
passed. |
| +1 :green_heart:
[jira] [Work logged] (HDFS-15808) Add metrics for FSNamesystem read/write lock hold long time
[ https://issues.apache.org/jira/browse/HDFS-15808?focusedWorklogId=554034=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554034 ] ASF GitHub Bot logged work on HDFS-15808: - Author: ASF GitHub Bot Created on: 18/Feb/21 02:20 Start Date: 18/Feb/21 02:20 Worklog Time Spent: 10m Work Description: tomscut edited a comment on pull request #2668: URL: https://github.com/apache/hadoop/pull/2668#issuecomment-780991248 I found this JIRA in the log. https://issues.apache.org/jira/browse/HADOOP-16870 Hi @aajisaka , Could you please help to take a look at this problem? And how I should deal with it? Thank you. `[2021-02-17T01:59:03.362Z] HEAD is now at 4cf35315838 HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454) [2021-02-17T01:59:03.887Z] Already on 'trunk' [2021-02-17T01:59:03.887Z] Your branch is up to date with 'origin/trunk'. [2021-02-17T01:59:04.412Z] Already up to date. [2021-02-17T01:59:04.412Z] Current branch trunk is up to date. [2021-02-17T01:59:04.412Z] Already on 'trunk' [2021-02-17T01:59:04.412Z] Your branch is up to date with 'origin/trunk'. [2021-02-17T01:59:05.444Z] HEAD is now at 4cf35315838 HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454) [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] Testing https://github.com/apache/hadoop/pull/2668 diff on trunk. [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] Re-exec mode detected. Continuing. [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] ERROR: Unprocessed flag(s): --findbugs-strict-precheck` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 554034) Time Spent: 4h (was: 3h 50m) > Add metrics for FSNamesystem read/write lock hold long time > --- > > Key: HDFS-15808 > URL: https://issues.apache.org/jira/browse/HDFS-15808 > Project: Hadoop HDFS > Issue Type: Wish > Components: hdfs >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: hdfs, lock, metrics, pull-request-available > Time Spent: 4h > Remaining Estimate: 0h > > To monitor how often read/write locks exceed thresholds, we can add two > metrics(ReadLockWarning/WriteLockWarning), which are exposed in JMX. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15808) Add metrics for FSNamesystem read/write lock hold long time
[ https://issues.apache.org/jira/browse/HDFS-15808?focusedWorklogId=554033=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554033 ] ASF GitHub Bot logged work on HDFS-15808: - Author: ASF GitHub Bot Created on: 18/Feb/21 02:20 Start Date: 18/Feb/21 02:20 Worklog Time Spent: 10m Work Description: tomscut edited a comment on pull request #2668: URL: https://github.com/apache/hadoop/pull/2668#issuecomment-780991248 I found this JIRA in the log. https://issues.apache.org/jira/browse/HADOOP-16870 Hi @aajisaka , Could you please help to take a look at this problem? And how I should deal with this problem? Thank you. `[2021-02-17T01:59:03.362Z] HEAD is now at 4cf35315838 HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454) [2021-02-17T01:59:03.887Z] Already on 'trunk' [2021-02-17T01:59:03.887Z] Your branch is up to date with 'origin/trunk'. [2021-02-17T01:59:04.412Z] Already up to date. [2021-02-17T01:59:04.412Z] Current branch trunk is up to date. [2021-02-17T01:59:04.412Z] Already on 'trunk' [2021-02-17T01:59:04.412Z] Your branch is up to date with 'origin/trunk'. [2021-02-17T01:59:05.444Z] HEAD is now at 4cf35315838 HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454) [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] Testing https://github.com/apache/hadoop/pull/2668 diff on trunk. [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] Re-exec mode detected. Continuing. [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] ERROR: Unprocessed flag(s): --findbugs-strict-precheck` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 554033) Time Spent: 3h 50m (was: 3h 40m) > Add metrics for FSNamesystem read/write lock hold long time > --- > > Key: HDFS-15808 > URL: https://issues.apache.org/jira/browse/HDFS-15808 > Project: Hadoop HDFS > Issue Type: Wish > Components: hdfs >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: hdfs, lock, metrics, pull-request-available > Time Spent: 3h 50m > Remaining Estimate: 0h > > To monitor how often read/write locks exceed thresholds, we can add two > metrics(ReadLockWarning/WriteLockWarning), which are exposed in JMX. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15808) Add metrics for FSNamesystem read/write lock hold long time
[ https://issues.apache.org/jira/browse/HDFS-15808?focusedWorklogId=554032=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554032 ] ASF GitHub Bot logged work on HDFS-15808: - Author: ASF GitHub Bot Created on: 18/Feb/21 02:18 Start Date: 18/Feb/21 02:18 Worklog Time Spent: 10m Work Description: tomscut edited a comment on pull request #2668: URL: https://github.com/apache/hadoop/pull/2668#issuecomment-780991248 I found this JIRA in the log. https://issues.apache.org/jira/browse/HADOOP-16870 Hi @aajisaka , Could you please help to take a look at this problem? And how I should deal with this problem? Thank you. `[2021-02-17T01:59:03.362Z] HEAD is now at 4cf35315838 HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454) [2021-02-17T01:59:03.887Z] Already on 'trunk' [2021-02-17T01:59:03.887Z] Your branch is up to date with 'origin/trunk'. [2021-02-17T01:59:04.412Z] Already up to date. [2021-02-17T01:59:04.412Z] Current branch trunk is up to date. [2021-02-17T01:59:04.412Z] Already on 'trunk' [2021-02-17T01:59:04.412Z] Your branch is up to date with 'origin/trunk'. [2021-02-17T01:59:05.444Z] HEAD is now at 4cf35315838 HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454) [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] Testing https://github.com/apache/hadoop/pull/2668 diff on trunk. [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] Re-exec mode detected. Continuing. [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] ERROR: Unprocessed flag(s): --findbugs-strict-precheck` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 554032) Time Spent: 3h 40m (was: 3.5h) > Add metrics for FSNamesystem read/write lock hold long time > --- > > Key: HDFS-15808 > URL: https://issues.apache.org/jira/browse/HDFS-15808 > Project: Hadoop HDFS > Issue Type: Wish > Components: hdfs >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: hdfs, lock, metrics, pull-request-available > Time Spent: 3h 40m > Remaining Estimate: 0h > > To monitor how often read/write locks exceed thresholds, we can add two > metrics(ReadLockWarning/WriteLockWarning), which are exposed in JMX. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15808) Add metrics for FSNamesystem read/write lock hold long time
[ https://issues.apache.org/jira/browse/HDFS-15808?focusedWorklogId=554031=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554031 ] ASF GitHub Bot logged work on HDFS-15808: - Author: ASF GitHub Bot Created on: 18/Feb/21 02:13 Start Date: 18/Feb/21 02:13 Worklog Time Spent: 10m Work Description: tomscut edited a comment on pull request #2668: URL: https://github.com/apache/hadoop/pull/2668#issuecomment-780991248 I found this JIRA in the log. https://issues.apache.org/jira/browse/HADOOP-16870 Hi @aajisaka , Could you please help to take a look at this problem? And how I should deal with this problem? Thank you. `[2021-02-17T01:59:03.362Z] HEAD is now at 4cf35315838 HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454) [2021-02-17T01:59:03.887Z] Already on 'trunk' [2021-02-17T01:59:03.887Z] Your branch is up to date with 'origin/trunk'. [2021-02-17T01:59:04.412Z] Already up to date. [2021-02-17T01:59:04.412Z] Current branch trunk is up to date. [2021-02-17T01:59:04.412Z] Already on 'trunk' [2021-02-17T01:59:04.412Z] Your branch is up to date with 'origin/trunk'. [2021-02-17T01:59:05.444Z] HEAD is now at 4cf35315838 HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454) [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] Testing https://github.com/apache/hadoop/pull/2668 diff on trunk. [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] Re-exec mode detected. Continuing. [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] ERROR: Unprocessed flag(s): --findbugs-strict-precheck` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 554031) Time Spent: 3.5h (was: 3h 20m) > Add metrics for FSNamesystem read/write lock hold long time > --- > > Key: HDFS-15808 > URL: https://issues.apache.org/jira/browse/HDFS-15808 > Project: Hadoop HDFS > Issue Type: Wish > Components: hdfs >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: hdfs, lock, metrics, pull-request-available > Time Spent: 3.5h > Remaining Estimate: 0h > > To monitor how often read/write locks exceed thresholds, we can add two > metrics(ReadLockWarning/WriteLockWarning), which are exposed in JMX. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15808) Add metrics for FSNamesystem read/write lock hold long time
[ https://issues.apache.org/jira/browse/HDFS-15808?focusedWorklogId=554030=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-554030 ] ASF GitHub Bot logged work on HDFS-15808: - Author: ASF GitHub Bot Created on: 18/Feb/21 02:12 Start Date: 18/Feb/21 02:12 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #2668: URL: https://github.com/apache/hadoop/pull/2668#issuecomment-780991248 I found this JIRA in the log. https://issues.apache.org/jira/browse/HADOOP-16870 Hi @aajisaka , Could you please help to take a look at this problem? And how I should deal with this problem? Thank you. `[2021-02-17T01:59:03.362Z] HEAD is now at 4cf35315838 HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454) [2021-02-17T01:59:03.887Z] Already on 'trunk' [2021-02-17T01:59:03.887Z] Your branch is up to date with 'origin/trunk'. [2021-02-17T01:59:04.412Z] Already up to date. [2021-02-17T01:59:04.412Z] Current branch trunk is up to date. [2021-02-17T01:59:04.412Z] Already on 'trunk' [2021-02-17T01:59:04.412Z] Your branch is up to date with 'origin/trunk'. [2021-02-17T01:59:05.444Z] HEAD is now at 4cf35315838 HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454) [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] Testing https://github.com/apache/hadoop/pull/2668 diff on trunk. [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] Re-exec mode detected. Continuing. [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] [2021-02-17T01:59:05.444Z] ERROR: Unprocessed flag(s): --findbugs-strict-precheck` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 554030) Time Spent: 3h 20m (was: 3h 10m) > Add metrics for FSNamesystem read/write lock hold long time > --- > > Key: HDFS-15808 > URL: https://issues.apache.org/jira/browse/HDFS-15808 > Project: Hadoop HDFS > Issue Type: Wish > Components: hdfs >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: hdfs, lock, metrics, pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > To monitor how often read/write locks exceed thresholds, we can add two > metrics(ReadLockWarning/WriteLockWarning), which are exposed in JMX. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15816) If NO stale node in last choosing, the chooseTarget don't need to retry with stale nodes.
[
https://issues.apache.org/jira/browse/HDFS-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286197#comment-17286197
]
Hadoop QA commented on HDFS-15816:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} yetus {color} | {color:red} 0m 8s{color}
| {color:red}{color} | {color:red} Unprocessed flag(s):
--findbugs-strict-precheck {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/477/artifact/out/Dockerfile
|
| JIRA Issue | HDFS-15816 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13020592/HDFS-15816.002.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/477/console |
| versions | git=2.25.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> If NO stale node in last choosing, the chooseTarget don't need to retry with
> stale nodes.
> -
>
> Key: HDFS-15816
> URL: https://issues.apache.org/jira/browse/HDFS-15816
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: block placement
>Reporter: Yang Yun
>Assignee: Yang Yun
>Priority: Minor
> Attachments: HDFS-15816.001.patch, HDFS-15816.002.patch
>
>
> If NO stale node in last choosing, the chooseTarget don't need to retry with
> stale nodes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15816) If NO stale node in last choosing, the chooseTarget don't need to retry with stale nodes.
[ https://issues.apache.org/jira/browse/HDFS-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15816: Attachment: (was: HDFS-15816.002.patch) > If NO stale node in last choosing, the chooseTarget don't need to retry with > stale nodes. > - > > Key: HDFS-15816 > URL: https://issues.apache.org/jira/browse/HDFS-15816 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15816.001.patch, HDFS-15816.002.patch > > > If NO stale node in last choosing, the chooseTarget don't need to retry with > stale nodes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15816) If NO stale node in last choosing, the chooseTarget don't need to retry with stale nodes.
[ https://issues.apache.org/jira/browse/HDFS-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15816: Attachment: HDFS-15816.002.patch Status: Patch Available (was: Open) > If NO stale node in last choosing, the chooseTarget don't need to retry with > stale nodes. > - > > Key: HDFS-15816 > URL: https://issues.apache.org/jira/browse/HDFS-15816 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15816.001.patch, HDFS-15816.002.patch > > > If NO stale node in last choosing, the chooseTarget don't need to retry with > stale nodes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15816) If NO stale node in last choosing, the chooseTarget don't need to retry with stale nodes.
[ https://issues.apache.org/jira/browse/HDFS-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15816: Status: Open (was: Patch Available) > If NO stale node in last choosing, the chooseTarget don't need to retry with > stale nodes. > - > > Key: HDFS-15816 > URL: https://issues.apache.org/jira/browse/HDFS-15816 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15816.001.patch, HDFS-15816.002.patch > > > If NO stale node in last choosing, the chooseTarget don't need to retry with > stale nodes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15808) Add metrics for FSNamesystem read/write lock hold long time
[
https://issues.apache.org/jira/browse/HDFS-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286133#comment-17286133
]
Konstantin Shvachko commented on HDFS-15808:
Hey [~tomscut]. The patch looks fine, but I doubt the metric will be useful in
its current form. Monotonically increasing counter doesn't tell you much when
plotted. Over time it just becomes an incredibly large number, hard to see its
fluctuations. And you cannot set alerts if the threshold is exceeded often.
See e.g. {{ExpiredHeartbeats}} or {{LastWrittenTransactionId}} - not useful.
I assume you need something like a rate.
> Add metrics for FSNamesystem read/write lock hold long time
> ---
>
> Key: HDFS-15808
> URL: https://issues.apache.org/jira/browse/HDFS-15808
> Project: Hadoop HDFS
> Issue Type: Wish
> Components: hdfs
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
> Labels: hdfs, lock, metrics, pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> To monitor how often read/write locks exceed thresholds, we can add two
> metrics(ReadLockWarning/WriteLockWarning), which are exposed in JMX.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15815) if required storageType are unavailable, log the failed reason during choosing Datanode
[ https://issues.apache.org/jira/browse/HDFS-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-15815: Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) > if required storageType are unavailable, log the failed reason during > choosing Datanode > > > Key: HDFS-15815 > URL: https://issues.apache.org/jira/browse/HDFS-15815 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Fix For: 3.4.0 > > Attachments: HDFS-15815.001.patch, HDFS-15815.002.patch, > HDFS-15815.003.patch > > > For better debug, if required storageType are unavailable, log the failed > reason "NO_REQUIRED_STORAGE_TYPE" when choosing Datanode. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15815) if required storageType are unavailable, log the failed reason during choosing Datanode
[ https://issues.apache.org/jira/browse/HDFS-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286130#comment-17286130 ] Ayush Saxena commented on HDFS-15815: - Committed to trunk. Thanx [~hadoop_yangyun] for the contribution!!! > if required storageType are unavailable, log the failed reason during > choosing Datanode > > > Key: HDFS-15815 > URL: https://issues.apache.org/jira/browse/HDFS-15815 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15815.001.patch, HDFS-15815.002.patch, > HDFS-15815.003.patch > > > For better debug, if required storageType are unavailable, log the failed > reason "NO_REQUIRED_STORAGE_TYPE" when choosing Datanode. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15837) Incorrect bytes causing block corruption after update pipeline and recovery failure
[
https://issues.apache.org/jira/browse/HDFS-15837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286111#comment-17286111
]
Kihwal Lee commented on HDFS-15837:
---
The successful pipeline update with 3072 bytes was at 03:42:40. The
replication of a 3544 byte block and corruption detection was 4 minutes later.
The client must have wrote more after the recovery and closed the file.
Replication happens only after closing the file. It looks like the file was
closed at around 03:46:40.
I don't think the size difference observed on datanodes are anomaly. Clients
usually write more data after a pipeline recovery. Is the file still existing?
What's the size you see when you do "hadoop fs -ls /pat_to_the_file"? Is the
block appearing as missing on the UI?
If there was a corruption issue, I don't think it was because of the size. The
fact that the replication happened means the NN was agreeing with the size
3544. If the datanode had unexpected size of the replica, the transfer wouldn't
have happened. The corruption report came from the destination node for the
replication, 172.21.234.181. The replication source does not perform any
checks and there is no feedback mechanism for reporting error back to the
source node. So even if a corruption was detected during the replication, only
the target (172.21.234.181) would notice it. Check the log of 172.21.234.181
at around 03:46:43. It must have detected a corruption and logged it.
The replica on the source (172.21.226.26) must be the only copy and if it is
corrupt, it will show up as missing. If the file still exists, you can get to
the node and run "hdfs debug verifyMeta" command against the block file and the
meta file to manually check the integrity.
If it is really corrupt, it could be due to 172.21.226.26's local issue.
Extreme load can cause write failures and pipeline updates, but we haven't seen
such condition causing data corruption.
> Incorrect bytes causing block corruption after update pipeline and recovery
> failure
> ---
>
> Key: HDFS-15837
> URL: https://issues.apache.org/jira/browse/HDFS-15837
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, hdfs
>Affects Versions: 2.8.5
>Reporter: Udit Mehrotra
>Priority: Major
>
> We are seeing cases on HDFS blocks being marked as *bad* after the initial
> block receive fails during *update pipeline* followed by *HDFS* *recovery*
> for the block failing as well. Here is the life cycle of a block
> *{{blk_1342440165_272630578}}* that was ultimately marked as corrupt:
> 1. The block creation starts at name node as part of *update pipeline*
> process:
> {noformat}
> 2021-01-25 03:41:17,335 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem (IPC Server handler 61 on
> 8020): updatePipeline(blk_1342440165_272500939 => blk_1342440165_272630578)
> success{noformat}
> 2. The block receiver on the data node fails with a socket timeout exception,
> and so do the retries:
> {noformat}
> 2021-01-25 03:42:22,525 INFO org.apache.hadoop.hdfs.server.datanode.DataNode
> (PacketResponder:
> BP-908477295-172.21.224.178-1606768078949:blk_1342440165_272630578,
> type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[172.21.246.239:50010]):
> PacketResponder:
> BP-908477295-172.21.224.178-1606768078949:blk_1342440165_272630578,
> type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[172.21.246.239:50010]
> java.net.SocketTimeoutException: 65000 millis timeout while waiting for
> channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
> local=/172.21.226.26:56294 remote=/172.21.246.239:50010]
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:400)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1305)
> at java.lang.Thread.run(Thread.java:748)
> 2021-01-25 03:42:22,526 WARN org.apache.hadoop.hdfs.server.datanode.DataNode
> (PacketResponder:
> BP-908477295-172.21.224.178-1606768078949:blk_1342440165_272630578,
> type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=1:[172.21.246.239:50010]):
> IOException in BlockReceiver.run():
> java.io.IOException: Connection
[jira] [Work logged] (HDFS-15808) Add metrics for FSNamesystem read/write lock hold long time
[ https://issues.apache.org/jira/browse/HDFS-15808?focusedWorklogId=553632=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-553632 ] ASF GitHub Bot logged work on HDFS-15808: - Author: ASF GitHub Bot Created on: 17/Feb/21 14:31 Start Date: 17/Feb/21 14:31 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #2668: URL: https://github.com/apache/hadoop/pull/2668#issuecomment-780595016 > This message was automatically generated. Hi @anuengineer, the failed check looks unrelated to the change. May I ask how I should deal with it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 553632) Time Spent: 3h 10m (was: 3h) > Add metrics for FSNamesystem read/write lock hold long time > --- > > Key: HDFS-15808 > URL: https://issues.apache.org/jira/browse/HDFS-15808 > Project: Hadoop HDFS > Issue Type: Wish > Components: hdfs >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: hdfs, lock, metrics, pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > To monitor how often read/write locks exceed thresholds, we can add two > metrics(ReadLockWarning/WriteLockWarning), which are exposed in JMX. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15838) Yarn Job execution get failed when LZ4 Compression Codec is used
[
https://issues.apache.org/jira/browse/HDFS-15838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bhavik Patel updated HDFS-15838:
Attachment: LZ4.png
> Yarn Job execution get failed when LZ4 Compression Codec is used
>
>
> Key: HDFS-15838
> URL: https://issues.apache.org/jira/browse/HDFS-15838
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Bhavik Patel
>Priority: Major
> Attachments: HDFS-15838.001.patch, LZ4.png
>
>
> When we try to compress a file using the LZ4 codec compression type then the
> yarn job gets failed with the error message :
> {code:java}
> net.jpountz.lz4.LZ4Compressorcompres(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)V
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15838) Yarn Job execution get failed when LZ4 Compression Codec is used
[
https://issues.apache.org/jira/browse/HDFS-15838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bhavik Patel updated HDFS-15838:
Attachment: HDFS-15838.001.patch
> Yarn Job execution get failed when LZ4 Compression Codec is used
>
>
> Key: HDFS-15838
> URL: https://issues.apache.org/jira/browse/HDFS-15838
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Bhavik Patel
>Priority: Major
> Attachments: HDFS-15838.001.patch
>
>
> When we try to compress a file using the LZ4 codec compression type then the
> yarn job gets failed with the error message :
> {code:java}
> net.jpountz.lz4.LZ4Compressorcompres(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)V
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15838) Yarn Job execution get failed when LZ4 Compression Codec is used
Bhavik Patel created HDFS-15838:
---
Summary: Yarn Job execution get failed when LZ4 Compression Codec
is used
Key: HDFS-15838
URL: https://issues.apache.org/jira/browse/HDFS-15838
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs
Reporter: Bhavik Patel
When we try to compress a file using the LZ4 codec compression type then the
yarn job gets failed with the error message :
{code:java}
net.jpountz.lz4.LZ4Compressorcompres(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)V
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15836) RBF: Fix contract tests after HADOOP-13327
[
https://issues.apache.org/jira/browse/HDFS-15836?focusedWorklogId=553556=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-553556
]
ASF GitHub Bot logged work on HDFS-15836:
-
Author: ASF GitHub Bot
Created on: 17/Feb/21 11:19
Start Date: 17/Feb/21 11:19
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #2702:
URL: https://github.com/apache/hadoop/pull/2702#issuecomment-780487969
I'm sorry. I thought I'd tested everything, at least through the IDE, but
was wrong. The new test really tries to determine what supports hflush/hsync
and what just pretends to by not raising an exception in hflush() but which
doesn't actually update the output. Apparently the HBase team care about
durable storage of their WAL. I've given them the probes, the validation *and
the ability to use a RAID-5 local FS as the WAL destination*
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 553556)
Time Spent: 1h 20m (was: 1h 10m)
> RBF: Fix contract tests after HADOOP-13327
> --
>
> Key: HDFS-15836
> URL: https://issues.apache.org/jira/browse/HDFS-15836
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> {noformat}
> [ERROR] Tests run: 16, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 19.094 s <<< FAILURE! - in
> org.apache.hadoop.fs.contract.router.TestRouterHDFSContractCreate
> [ERROR]
> testSyncable(org.apache.hadoop.fs.contract.router.TestRouterHDFSContractCreate)
> Time elapsed: 0.102 s <<< FAILURE!
> java.lang.AssertionError: Should not have capability: hflush in
> FSDataOutputStream{wrappedStream=DFSOutputStream:block==null}
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at
> org.apache.hadoop.fs.contract.ContractTestUtils.assertCapabilities(ContractTestUtils.java:1553)
> at
> org.apache.hadoop.fs.contract.AbstractContractCreateTest.validateSyncableSemantics(AbstractContractCreateTest.java:497)
> at
> org.apache.hadoop.fs.contract.AbstractContractCreateTest.testSyncable(AbstractContractCreateTest.java:459)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2696/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15816) If NO stale node in last choosing, the chooseTarget don't need to retry with stale nodes.
[
https://issues.apache.org/jira/browse/HDFS-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285780#comment-17285780
]
Hadoop QA commented on HDFS-15816:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} yetus {color} | {color:red} 0m 7s{color}
| {color:red}{color} | {color:red} Unprocessed flag(s):
--findbugs-strict-precheck {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/476/artifact/out/Dockerfile
|
| JIRA Issue | HDFS-15816 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13020557/HDFS-15816.002.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/476/console |
| versions | git=2.25.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> If NO stale node in last choosing, the chooseTarget don't need to retry with
> stale nodes.
> -
>
> Key: HDFS-15816
> URL: https://issues.apache.org/jira/browse/HDFS-15816
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: block placement
>Reporter: Yang Yun
>Assignee: Yang Yun
>Priority: Minor
> Attachments: HDFS-15816.001.patch, HDFS-15816.002.patch
>
>
> If NO stale node in last choosing, the chooseTarget don't need to retry with
> stale nodes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15816) If NO stale node in last choosing, the chooseTarget don't need to retry with stale nodes.
[ https://issues.apache.org/jira/browse/HDFS-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15816: Attachment: HDFS-15816.002.patch Status: Patch Available (was: Open) > If NO stale node in last choosing, the chooseTarget don't need to retry with > stale nodes. > - > > Key: HDFS-15816 > URL: https://issues.apache.org/jira/browse/HDFS-15816 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15816.001.patch, HDFS-15816.002.patch > > > If NO stale node in last choosing, the chooseTarget don't need to retry with > stale nodes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15816) If NO stale node in last choosing, the chooseTarget don't need to retry with stale nodes.
[ https://issues.apache.org/jira/browse/HDFS-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15816: Status: Open (was: Patch Available) > If NO stale node in last choosing, the chooseTarget don't need to retry with > stale nodes. > - > > Key: HDFS-15816 > URL: https://issues.apache.org/jira/browse/HDFS-15816 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15816.001.patch, HDFS-15816.002.patch > > > If NO stale node in last choosing, the chooseTarget don't need to retry with > stale nodes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15816) If NO stale node in last choosing, the chooseTarget don't need to retry with stale nodes.
[ https://issues.apache.org/jira/browse/HDFS-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15816: Attachment: (was: HDFS-15816.002.patch) > If NO stale node in last choosing, the chooseTarget don't need to retry with > stale nodes. > - > > Key: HDFS-15816 > URL: https://issues.apache.org/jira/browse/HDFS-15816 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15816.001.patch, HDFS-15816.002.patch > > > If NO stale node in last choosing, the chooseTarget don't need to retry with > stale nodes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15816) If NO stale node in last choosing, the chooseTarget don't need to retry with stale nodes.
[
https://issues.apache.org/jira/browse/HDFS-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285760#comment-17285760
]
Hadoop QA commented on HDFS-15816:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
47s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} yetus {color} | {color:red} 0m 7s{color}
| {color:red}{color} | {color:red} Unprocessed flag(s):
--findbugs-strict-precheck {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/475/artifact/out/Dockerfile
|
| JIRA Issue | HDFS-15816 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13020552/HDFS-15816.002.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/475/console |
| versions | git=2.25.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> If NO stale node in last choosing, the chooseTarget don't need to retry with
> stale nodes.
> -
>
> Key: HDFS-15816
> URL: https://issues.apache.org/jira/browse/HDFS-15816
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: block placement
>Reporter: Yang Yun
>Assignee: Yang Yun
>Priority: Minor
> Attachments: HDFS-15816.001.patch, HDFS-15816.002.patch
>
>
> If NO stale node in last choosing, the chooseTarget don't need to retry with
> stale nodes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15816) If NO stale node in last choosing, the chooseTarget don't need to retry with stale nodes.
[ https://issues.apache.org/jira/browse/HDFS-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285754#comment-17285754 ] Yang Yun commented on HDFS-15816: - Thanks [~ayushtkn] for your review. Add a new variable 'ThreadLocal hasStaleNode' to trace if it met node in new patch HDFS-15816.002.patch. > If NO stale node in last choosing, the chooseTarget don't need to retry with > stale nodes. > - > > Key: HDFS-15816 > URL: https://issues.apache.org/jira/browse/HDFS-15816 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15816.001.patch, HDFS-15816.002.patch > > > If NO stale node in last choosing, the chooseTarget don't need to retry with > stale nodes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15816) If NO stale node in last choosing, the chooseTarget don't need to retry with stale nodes.
[ https://issues.apache.org/jira/browse/HDFS-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15816: Attachment: HDFS-15816.002.patch Status: Patch Available (was: Open) > If NO stale node in last choosing, the chooseTarget don't need to retry with > stale nodes. > - > > Key: HDFS-15816 > URL: https://issues.apache.org/jira/browse/HDFS-15816 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15816.001.patch, HDFS-15816.002.patch > > > If NO stale node in last choosing, the chooseTarget don't need to retry with > stale nodes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15816) If NO stale node in last choosing, the chooseTarget don't need to retry with stale nodes.
[ https://issues.apache.org/jira/browse/HDFS-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15816: Status: Open (was: Patch Available) > If NO stale node in last choosing, the chooseTarget don't need to retry with > stale nodes. > - > > Key: HDFS-15816 > URL: https://issues.apache.org/jira/browse/HDFS-15816 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15816.001.patch > > > If NO stale node in last choosing, the chooseTarget don't need to retry with > stale nodes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9536) OOM errors during parallel upgrade to Block-ID based layout
[ https://issues.apache.org/jira/browse/HDFS-9536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285707#comment-17285707 ] Luca Toscano commented on HDFS-9536: Hi! I have recently upgraded an Hadoop cluster from 2.6.0 (CDH 5.16.2, so not a vanilla 2.6) to 2.10.1 (Apache Bigtop 1.5) and I have experienced a lot of troubles while upgrading DNs. Before the upgrade we used to run the DNs with -Xmx 4G without any issue, but during the upgrade we had to bump the value to 16G on some nodes to avoid OOMs and allow the cluster to complete the upgrade. The nodes causing the most problems were the one with more blocks (order of some millions of blocks). Is this an issue that happens only when upgrading from 2.6.0 (due to https://issues.apache.org/jira/browse/HDFS-6482) or can it happen also on later versions? If so, how should the heap sizes of DNs be tuned? And would reducing `dfs.datanode.block.id.layout.upgrade.threads` help? Thanks in advance :) > OOM errors during parallel upgrade to Block-ID based layout > --- > > Key: HDFS-9536 > URL: https://issues.apache.org/jira/browse/HDFS-9536 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Major > > This is a follow-up jira for the OOM errors observed during parallel upgrade > to Block-ID based datanode layout using HDFS-8578 fix. > more clue > [here|https://issues.apache.org/jira/browse/HDFS-8578?focusedCommentId=15042012=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15042012] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
