[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=744590&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744590
]
ASF GitHub Bot logged work on HDFS-13248:
-
Author: ASF GitHub Bot
Created on: 19/Mar/22 05:54
Start Date: 19/Mar/22 05:54
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4081:
URL: https://github.com/apache/hadoop/pull/4081#issuecomment-1072948703
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 18m 8s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 52s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 9s | | trunk passed |
| +1 :green_heart: | compile | 24m 38s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 21m 6s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 8s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 8s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 4m 23s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 8m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 20s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 6s | | the patch passed |
| +1 :green_heart: | compile | 28m 51s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 28m 51s | | the patch passed |
| +1 :green_heart: | compile | 23m 41s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 23m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 54s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 17s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 3m 20s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 4m 36s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 8m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 34s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 45s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 383m 30s | | hadoop-hdfs in the patch
passed. |
| -1 :x: | unit | 42m 56s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4081/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 37s | | The patch does not
generate ASF License warnings. |
| | | 709m 22s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4081/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4081 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint xml |
| uname | Linux 677bea8344a7 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29
06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
|
[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=744580&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744580
]
ASF GitHub Bot logged work on HDFS-13248:
-
Author: ASF GitHub Bot
Created on: 19/Mar/22 04:35
Start Date: 19/Mar/22 04:35
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4081:
URL: https://github.com/apache/hadoop/pull/4081#issuecomment-1072938611
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 37s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 51s | | trunk passed |
| +1 :green_heart: | compile | 22m 35s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 53s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 3m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 25s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 4m 44s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 29s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 49s | | the patch passed |
| +1 :green_heart: | compile | 22m 1s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 22m 1s | | the patch passed |
| +1 :green_heart: | compile | 19m 49s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 19m 48s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 35s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4081/7/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 415 unchanged - 0 fixed = 416 total (was
415) |
| +1 :green_heart: | mvnsite | 4m 22s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 3m 24s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 4m 48s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 43s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 44s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 227m 31s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | unit | 21m 21s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 10s | | The patch does not
generate ASF License warnings. |
| | | 497m 33s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4081/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4081 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint xml |
| uname | Linux 0fa07506a736 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3
10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 901d5420f0e25c4368057ce3e391085c0fbaf771 |
| Default Java | Private Build-1.8
[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[ https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=744387&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744387 ] ASF GitHub Bot logged work on HDFS-13248: - Author: ASF GitHub Bot Created on: 18/Mar/22 20:06 Start Date: 18/Mar/22 20:06 Worklog Time Spent: 10m Work Description: omalley commented on a change in pull request #4081: URL: https://github.com/apache/hadoop/pull/4081#discussion_r830318314 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRpcServer.java ## @@ -1,4 +1,4 @@ -/** +/* Review comment: Those license headers aren't javadoc and shouldn't start with the two stars. If you load it in intellij, they show up as a warning. Since I was fixing the imports, I figured I might as well fix that too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 744387) Time Spent: 3h 40m (was: 3.5h) > RBF: Namenode need to choose block location for the client > -- > > Key: HDFS-13248 > URL: https://issues.apache.org/jira/browse/HDFS-13248 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wu Weiwei >Assignee: Íñigo Goiri >Priority: Major > Labels: pull-request-available > Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch, > HDFS-13248.002.patch, HDFS-13248.003.patch, HDFS-13248.004.patch, > HDFS-13248.005.patch, HDFS-Router-Data-Locality.odt, RBF Data Locality > Design.pdf, clientMachine-call-path.jpeg, debug-info-1.jpeg, debug-info-2.jpeg > > Time Spent: 3h 40m > Remaining Estimate: 0h > > When execute a put operation via router, the NameNode will choose block > location for the router, not for the real client. This will affect the file's > locality. > I think on both NameNode and Router, we should add a new addBlock method, or > add a parameter for the current addBlock method, to pass the real client > information. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[ https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=744379&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744379 ] ASF GitHub Bot logged work on HDFS-13248: - Author: ASF GitHub Bot Created on: 18/Mar/22 19:53 Start Date: 18/Mar/22 19:53 Worklog Time Spent: 10m Work Description: goiri commented on a change in pull request #4081: URL: https://github.com/apache/hadoop/pull/4081#discussion_r830310862 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRpcServer.java ## @@ -1,4 +1,4 @@ -/** +/* Review comment: ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 744379) Time Spent: 3.5h (was: 3h 20m) > RBF: Namenode need to choose block location for the client > -- > > Key: HDFS-13248 > URL: https://issues.apache.org/jira/browse/HDFS-13248 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wu Weiwei >Assignee: Íñigo Goiri >Priority: Major > Labels: pull-request-available > Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch, > HDFS-13248.002.patch, HDFS-13248.003.patch, HDFS-13248.004.patch, > HDFS-13248.005.patch, HDFS-Router-Data-Locality.odt, RBF Data Locality > Design.pdf, clientMachine-call-path.jpeg, debug-info-1.jpeg, debug-info-2.jpeg > > Time Spent: 3.5h > Remaining Estimate: 0h > > When execute a put operation via router, the NameNode will choose block > location for the router, not for the real client. This will affect the file's > locality. > I think on both NameNode and Router, we should add a new addBlock method, or > add a parameter for the current addBlock method, to pass the real client > information. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=744377&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744377
]
ASF GitHub Bot logged work on HDFS-13248:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 19:52
Start Date: 18/Mar/22 19:52
Worklog Time Spent: 10m
Work Description: omalley commented on a change in pull request #4081:
URL: https://github.com/apache/hadoop/pull/4081#discussion_r830310255
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
##
@@ -1899,7 +1907,27 @@ private void verifySoftwareVersion(DatanodeRegistration
dnReg)
}
}
- private static String getClientMachine() {
+ private String getClientMachine() {
+if (ipProxyUsers != null) {
+ // Get the real user (or effective if it isn't a proxy user)
+ UserGroupInformation user = Server.getRemoteUser().getRealUserOrSelf();
Review comment:
Although doing the user operations on the NN without RPC would be
unexpected. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744377)
Time Spent: 3h 20m (was: 3h 10m)
> RBF: Namenode need to choose block location for the client
> --
>
> Key: HDFS-13248
> URL: https://issues.apache.org/jira/browse/HDFS-13248
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wu Weiwei
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch,
> HDFS-13248.002.patch, HDFS-13248.003.patch, HDFS-13248.004.patch,
> HDFS-13248.005.patch, HDFS-Router-Data-Locality.odt, RBF Data Locality
> Design.pdf, clientMachine-call-path.jpeg, debug-info-1.jpeg, debug-info-2.jpeg
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> When execute a put operation via router, the NameNode will choose block
> location for the router, not for the real client. This will affect the file's
> locality.
> I think on both NameNode and Router, we should add a new addBlock method, or
> add a parameter for the current addBlock method, to pass the real client
> information.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=744375&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744375
]
ASF GitHub Bot logged work on HDFS-13248:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 19:50
Start Date: 18/Mar/22 19:50
Worklog Time Spent: 10m
Work Description: omalley commented on a change in pull request #4081:
URL: https://github.com/apache/hadoop/pull/4081#discussion_r830309182
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
##
@@ -1899,7 +1907,27 @@ private void verifySoftwareVersion(DatanodeRegistration
dnReg)
}
}
- private static String getClientMachine() {
+ private String getClientMachine() {
+if (ipProxyUsers != null) {
+ // Get the real user (or effective if it isn't a proxy user)
+ UserGroupInformation user = Server.getRemoteUser().getRealUserOrSelf();
Review comment:
Nice catch, thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744375)
Time Spent: 3h 10m (was: 3h)
> RBF: Namenode need to choose block location for the client
> --
>
> Key: HDFS-13248
> URL: https://issues.apache.org/jira/browse/HDFS-13248
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wu Weiwei
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch,
> HDFS-13248.002.patch, HDFS-13248.003.patch, HDFS-13248.004.patch,
> HDFS-13248.005.patch, HDFS-Router-Data-Locality.odt, RBF Data Locality
> Design.pdf, clientMachine-call-path.jpeg, debug-info-1.jpeg, debug-info-2.jpeg
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> When execute a put operation via router, the NameNode will choose block
> location for the router, not for the real client. This will affect the file's
> locality.
> I think on both NameNode and Router, we should add a new addBlock method, or
> add a parameter for the current addBlock method, to pass the real client
> information.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=744335&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744335
]
ASF GitHub Bot logged work on HDFS-13248:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 18:55
Start Date: 18/Mar/22 18:55
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #4081:
URL: https://github.com/apache/hadoop/pull/4081#discussion_r830271672
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
##
@@ -1899,7 +1907,27 @@ private void verifySoftwareVersion(DatanodeRegistration
dnReg)
}
}
- private static String getClientMachine() {
+ private String getClientMachine() {
+if (ipProxyUsers != null) {
+ // Get the real user (or effective if it isn't a proxy user)
+ UserGroupInformation user = Server.getRemoteUser().getRealUserOrSelf();
Review comment:
One more here:
Can this shoot a NPE?
There is a null check below for ``Server.getRemoteUser()``
```
String clientMachine = Server.getRemoteAddress();
if (clientMachine == null) { //not a RPC client
```
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRpcServer.java
##
@@ -59,5 +74,67 @@ public void testNamenodeRpcBindAny() throws IOException {
conf.unset(DFS_NAMENODE_RPC_BIND_HOST_KEY);
}
}
+
+ /**
+ * A test to make sure that if an authorized user adds "clientIp:" to their
+ * caller context, it will be used to make locality decisions on the NN.
+ */
+ @Test
+ public void testNamenodeRpcClientIpProxy()
+ throws InterruptedException, IOException {
+Configuration conf = new HdfsConfiguration();
+
+conf.set(DFS_NAMENODE_IP_PROXY_USERS, "fake_joe");
Review comment:
makes sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744335)
Time Spent: 3h (was: 2h 50m)
> RBF: Namenode need to choose block location for the client
> --
>
> Key: HDFS-13248
> URL: https://issues.apache.org/jira/browse/HDFS-13248
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wu Weiwei
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch,
> HDFS-13248.002.patch, HDFS-13248.003.patch, HDFS-13248.004.patch,
> HDFS-13248.005.patch, HDFS-Router-Data-Locality.odt, RBF Data Locality
> Design.pdf, clientMachine-call-path.jpeg, debug-info-1.jpeg, debug-info-2.jpeg
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> When execute a put operation via router, the NameNode will choose block
> location for the router, not for the real client. This will affect the file's
> locality.
> I think on both NameNode and Router, we should add a new addBlock method, or
> add a parameter for the current addBlock method, to pass the real client
> information.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=744313&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744313
]
ASF GitHub Bot logged work on HDFS-13248:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 18:37
Start Date: 18/Mar/22 18:37
Worklog Time Spent: 10m
Work Description: omalley commented on a change in pull request #4081:
URL: https://github.com/apache/hadoop/pull/4081#discussion_r830260468
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRpcServer.java
##
@@ -59,5 +69,67 @@ public void testNamenodeRpcBindAny() throws IOException {
conf.unset(DFS_NAMENODE_RPC_BIND_HOST_KEY);
}
}
+
+ /**
+ * A test to make sure that if an authorized user adds "clientIp:" to their
+ * caller context, it will be used to make locality decisions on the NN.
+ */
+ @Test
+ public void testNamenodeRpcClientIpProxy()
+ throws InterruptedException, IOException {
+Configuration conf = new HdfsConfiguration();
+
+conf.set(DFS_NAMENODE_IP_PROXY_USERS, "fake_joe");
+// Make 3 nodes & racks so that we have a decent shot of detecting when
+// our change overrides the random choice of datanode.
+final String[] racks = new String[]{"/rack1", "/rack2", "/rack3"};
+final String[] hosts = new String[]{"node1", "node2", "node3"};
+MiniDFSCluster cluster = null;
+final CallerContext original = CallerContext.getCurrent();
+
+try {
+ cluster = new MiniDFSCluster.Builder(conf)
+ .racks(racks).hosts(hosts).numDataNodes(hosts.length)
+ .build();
+ cluster.waitActive();
+ DistributedFileSystem fs = cluster.getFileSystem();
+ // Write a sample file
+ final Path fooName = fs.makeQualified(new Path("/foo"));
+ FSDataOutputStream stream = fs.create(fooName);
+ stream.write("Hello world!\n".getBytes(StandardCharsets.UTF_8));
+ stream.close();
+ // Set the caller context to set the ip address
+ CallerContext.setCurrent(
+ new CallerContext.Builder("test", conf)
+ .append(CallerContext.CLIENT_IP_STR, hosts[0])
+ .build());
+ // Run as fake joe to authorize the test
+ UserGroupInformation
+ .createUserForTesting("fake_joe", new String[]{"fake_group"})
+ .doAs(new PrivilegedExceptionAction() {
+@Override
+public Object run() throws Exception {
+ // Create a new file system as the joe user
+ DistributedFileSystem joeFs =
+ (DistributedFileSystem) fooName.getFileSystem(conf);
Review comment:
Ah, interesting. I'm still happier using a path to get the file system,
but you're right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744313)
Time Spent: 2h 50m (was: 2h 40m)
> RBF: Namenode need to choose block location for the client
> --
>
> Key: HDFS-13248
> URL: https://issues.apache.org/jira/browse/HDFS-13248
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wu Weiwei
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch,
> HDFS-13248.002.patch, HDFS-13248.003.patch, HDFS-13248.004.patch,
> HDFS-13248.005.patch, HDFS-Router-Data-Locality.odt, RBF Data Locality
> Design.pdf, clientMachine-call-path.jpeg, debug-info-1.jpeg, debug-info-2.jpeg
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> When execute a put operation via router, the NameNode will choose block
> location for the router, not for the real client. This will affect the file's
> locality.
> I think on both NameNode and Router, we should add a new addBlock method, or
> add a parameter for the current addBlock method, to pass the real client
> information.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=744311&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744311
]
ASF GitHub Bot logged work on HDFS-13248:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 18:33
Start Date: 18/Mar/22 18:33
Worklog Time Spent: 10m
Work Description: omalley commented on a change in pull request #4081:
URL: https://github.com/apache/hadoop/pull/4081#discussion_r830257443
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRpcServer.java
##
@@ -59,5 +74,67 @@ public void testNamenodeRpcBindAny() throws IOException {
conf.unset(DFS_NAMENODE_RPC_BIND_HOST_KEY);
}
}
+
+ /**
+ * A test to make sure that if an authorized user adds "clientIp:" to their
+ * caller context, it will be used to make locality decisions on the NN.
+ */
+ @Test
+ public void testNamenodeRpcClientIpProxy()
+ throws InterruptedException, IOException {
+Configuration conf = new HdfsConfiguration();
+
+conf.set(DFS_NAMENODE_IP_PROXY_USERS, "fake_joe");
+// Make 3 nodes & racks so that we have a decent shot of detecting when
+// our change overrides the random choice of datanode.
+final String[] racks = new String[]{"/rack1", "/rack2", "/rack3"};
+final String[] hosts = new String[]{"node1", "node2", "node3"};
+MiniDFSCluster cluster = null;
+final CallerContext original = CallerContext.getCurrent();
+
+try {
+ cluster = new MiniDFSCluster.Builder(conf)
+ .racks(racks).hosts(hosts).numDataNodes(hosts.length)
+ .build();
+ cluster.waitActive();
+ DistributedFileSystem fs = cluster.getFileSystem();
+ // Write a sample file
+ final Path fooName = fs.makeQualified(new Path("/foo"));
Review comment:
Because I wanted to make sure that we always get the same file system,
but with joe as the user, when we make joeFs down below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744311)
Time Spent: 2h 40m (was: 2.5h)
> RBF: Namenode need to choose block location for the client
> --
>
> Key: HDFS-13248
> URL: https://issues.apache.org/jira/browse/HDFS-13248
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wu Weiwei
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch,
> HDFS-13248.002.patch, HDFS-13248.003.patch, HDFS-13248.004.patch,
> HDFS-13248.005.patch, HDFS-Router-Data-Locality.odt, RBF Data Locality
> Design.pdf, clientMachine-call-path.jpeg, debug-info-1.jpeg, debug-info-2.jpeg
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> When execute a put operation via router, the NameNode will choose block
> location for the router, not for the real client. This will affect the file's
> locality.
> I think on both NameNode and Router, we should add a new addBlock method, or
> add a parameter for the current addBlock method, to pass the real client
> information.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=744310&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744310
]
ASF GitHub Bot logged work on HDFS-13248:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 18:31
Start Date: 18/Mar/22 18:31
Worklog Time Spent: 10m
Work Description: omalley commented on a change in pull request #4081:
URL: https://github.com/apache/hadoop/pull/4081#discussion_r830256594
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
##
@@ -1899,7 +1907,27 @@ private void verifySoftwareVersion(DatanodeRegistration
dnReg)
}
}
- private static String getClientMachine() {
+ private String getClientMachine() {
+if (ipProxyUsers != null) {
+ // Get the real user (or effective if it isn't a proxy user)
+ UserGroupInformation user = Server.getRemoteUser().getRealUserOrSelf();
+ if (ArrayUtils.contains(ipProxyUsers, user.getShortUserName())) {
+CallerContext context = CallerContext.getCurrent();
+if (context != null && context.isContextValid()) {
+ String cc = context.getContext();
+ // if the rpc has a caller context of "clientIp:1.2.3.4,CLI",
+ // return "1.2.3.4" as the client machine.
+ String key = CallerContext.CLIENT_IP_STR +
+ CallerContext.Builder.KEY_VALUE_SEPARATOR;
+ int posn = cc.indexOf(key);
+ if (posn != -1) {
+posn += key.length();
+int end = cc.indexOf(",", posn);
+return end == -1 ? cc.substring(posn) : cc.substring(posn, end);
Review comment:
The Routers always puts the "clientIp:" at the front of the caller
context. This code doesn't assume that, but it means that it is much less
sensitive that that kind of problem.
We can't pre-compute using the caller context, because it is sent along with
each rpc request as part of the RpcRequestHeader. One cost of that is that we
have to compute it as part of the call handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744310)
Time Spent: 2.5h (was: 2h 20m)
> RBF: Namenode need to choose block location for the client
> --
>
> Key: HDFS-13248
> URL: https://issues.apache.org/jira/browse/HDFS-13248
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wu Weiwei
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch,
> HDFS-13248.002.patch, HDFS-13248.003.patch, HDFS-13248.004.patch,
> HDFS-13248.005.patch, HDFS-Router-Data-Locality.odt, RBF Data Locality
> Design.pdf, clientMachine-call-path.jpeg, debug-info-1.jpeg, debug-info-2.jpeg
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> When execute a put operation via router, the NameNode will choose block
> location for the router, not for the real client. This will affect the file's
> locality.
> I think on both NameNode and Router, we should add a new addBlock method, or
> add a parameter for the current addBlock method, to pass the real client
> information.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=744304&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744304
]
ASF GitHub Bot logged work on HDFS-13248:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 18:26
Start Date: 18/Mar/22 18:26
Worklog Time Spent: 10m
Work Description: omalley commented on a change in pull request #4081:
URL: https://github.com/apache/hadoop/pull/4081#discussion_r830253165
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRpcServer.java
##
@@ -59,5 +74,67 @@ public void testNamenodeRpcBindAny() throws IOException {
conf.unset(DFS_NAMENODE_RPC_BIND_HOST_KEY);
}
}
+
+ /**
+ * A test to make sure that if an authorized user adds "clientIp:" to their
+ * caller context, it will be used to make locality decisions on the NN.
+ */
+ @Test
+ public void testNamenodeRpcClientIpProxy()
+ throws InterruptedException, IOException {
+Configuration conf = new HdfsConfiguration();
+
+conf.set(DFS_NAMENODE_IP_PROXY_USERS, "fake_joe");
Review comment:
I could add multiple trials. Doing the math 1/3 ^ 10 = 1.7e-5 and 1/3 ^
20 = 2.9e-10, so 20 trials will bring the failure rate down to 2.9e-10. That
seems doable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744304)
Time Spent: 2h 20m (was: 2h 10m)
> RBF: Namenode need to choose block location for the client
> --
>
> Key: HDFS-13248
> URL: https://issues.apache.org/jira/browse/HDFS-13248
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wu Weiwei
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch,
> HDFS-13248.002.patch, HDFS-13248.003.patch, HDFS-13248.004.patch,
> HDFS-13248.005.patch, HDFS-Router-Data-Locality.odt, RBF Data Locality
> Design.pdf, clientMachine-call-path.jpeg, debug-info-1.jpeg, debug-info-2.jpeg
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> When execute a put operation via router, the NameNode will choose block
> location for the router, not for the real client. This will affect the file's
> locality.
> I think on both NameNode and Router, we should add a new addBlock method, or
> add a parameter for the current addBlock method, to pass the real client
> information.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16457) Make fs.getspaceused.classname reconfigurable
[
https://issues.apache.org/jira/browse/HDFS-16457?focusedWorklogId=744302&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744302
]
ASF GitHub Bot logged work on HDFS-16457:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 18:23
Start Date: 18/Mar/22 18:23
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4069:
URL: https://github.com/apache/hadoop/pull/4069#issuecomment-1072677183
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 11s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 37s | | trunk passed |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 3s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 48s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 23m 9s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 21s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 21s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 13s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 372m 13s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4069/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 489m 35s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.mover.TestMover |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4069/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4069 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 7f5a469c023e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 49afa2485c6a2d018a790e74ba83f622a200a72b |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https:/
[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=744300&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744300
]
ASF GitHub Bot logged work on HDFS-13248:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 18:21
Start Date: 18/Mar/22 18:21
Worklog Time Spent: 10m
Work Description: omalley commented on a change in pull request #4081:
URL: https://github.com/apache/hadoop/pull/4081#discussion_r830249689
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRpcServer.java
##
@@ -59,5 +74,67 @@ public void testNamenodeRpcBindAny() throws IOException {
conf.unset(DFS_NAMENODE_RPC_BIND_HOST_KEY);
}
}
+
+ /**
+ * A test to make sure that if an authorized user adds "clientIp:" to their
+ * caller context, it will be used to make locality decisions on the NN.
+ */
+ @Test
+ public void testNamenodeRpcClientIpProxy()
+ throws InterruptedException, IOException {
+Configuration conf = new HdfsConfiguration();
+
+conf.set(DFS_NAMENODE_IP_PROXY_USERS, "fake_joe");
Review comment:
That isn't an easy test to write, because in the mini dfs cluster, all
of the datanodes are on the same ip. So without the clientIp being forced, the
namenode will pick a datanode randomly. That is why I made three datanodes, so
the test will only accidentally pass 1/3 of the time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744300)
Time Spent: 2h 10m (was: 2h)
> RBF: Namenode need to choose block location for the client
> --
>
> Key: HDFS-13248
> URL: https://issues.apache.org/jira/browse/HDFS-13248
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wu Weiwei
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch,
> HDFS-13248.002.patch, HDFS-13248.003.patch, HDFS-13248.004.patch,
> HDFS-13248.005.patch, HDFS-Router-Data-Locality.odt, RBF Data Locality
> Design.pdf, clientMachine-call-path.jpeg, debug-info-1.jpeg, debug-info-2.jpeg
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> When execute a put operation via router, the NameNode will choose block
> location for the router, not for the real client. This will affect the file's
> locality.
> I think on both NameNode and Router, we should add a new addBlock method, or
> add a parameter for the current addBlock method, to pass the real client
> information.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=744294&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744294
]
ASF GitHub Bot logged work on HDFS-13248:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 18:12
Start Date: 18/Mar/22 18:12
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #4081:
URL: https://github.com/apache/hadoop/pull/4081#discussion_r830233015
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
##
@@ -1899,7 +1907,27 @@ private void verifySoftwareVersion(DatanodeRegistration
dnReg)
}
}
- private static String getClientMachine() {
+ private String getClientMachine() {
+if (ipProxyUsers != null) {
+ // Get the real user (or effective if it isn't a proxy user)
+ UserGroupInformation user = Server.getRemoteUser().getRealUserOrSelf();
+ if (ArrayUtils.contains(ipProxyUsers, user.getShortUserName())) {
+CallerContext context = CallerContext.getCurrent();
+if (context != null && context.isContextValid()) {
+ String cc = context.getContext();
+ // if the rpc has a caller context of "clientIp:1.2.3.4,CLI",
+ // return "1.2.3.4" as the client machine.
+ String key = CallerContext.CLIENT_IP_STR +
+ CallerContext.Builder.KEY_VALUE_SEPARATOR;
+ int posn = cc.indexOf(key);
+ if (posn != -1) {
+posn += key.length();
+int end = cc.indexOf(",", posn);
+return end == -1 ? cc.substring(posn) : cc.substring(posn, end);
Review comment:
Someone passes something like ``ABclientIp:1.2.3.4,CLI``. Guess this
logic will still catch that entry right?
Moreover, why don't we store the key & its length as well above when we get
``ipProxyUsers`` rather than doing this concatenation & length computation for
each call?
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRpcServer.java
##
@@ -59,5 +74,67 @@ public void testNamenodeRpcBindAny() throws IOException {
conf.unset(DFS_NAMENODE_RPC_BIND_HOST_KEY);
}
}
+
+ /**
+ * A test to make sure that if an authorized user adds "clientIp:" to their
+ * caller context, it will be used to make locality decisions on the NN.
+ */
+ @Test
+ public void testNamenodeRpcClientIpProxy()
+ throws InterruptedException, IOException {
+Configuration conf = new HdfsConfiguration();
+
+conf.set(DFS_NAMENODE_IP_PROXY_USERS, "fake_joe");
Review comment:
We should even extend a test where this values isn't set, but someone
passes the client details as part of the CallerContext, so in that case it
shouldn't be honoured.
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRpcServer.java
##
@@ -59,5 +74,67 @@ public void testNamenodeRpcBindAny() throws IOException {
conf.unset(DFS_NAMENODE_RPC_BIND_HOST_KEY);
}
}
+
+ /**
+ * A test to make sure that if an authorized user adds "clientIp:" to their
+ * caller context, it will be used to make locality decisions on the NN.
+ */
+ @Test
+ public void testNamenodeRpcClientIpProxy()
+ throws InterruptedException, IOException {
+Configuration conf = new HdfsConfiguration();
+
+conf.set(DFS_NAMENODE_IP_PROXY_USERS, "fake_joe");
+// Make 3 nodes & racks so that we have a decent shot of detecting when
+// our change overrides the random choice of datanode.
+final String[] racks = new String[]{"/rack1", "/rack2", "/rack3"};
+final String[] hosts = new String[]{"node1", "node2", "node3"};
+MiniDFSCluster cluster = null;
+final CallerContext original = CallerContext.getCurrent();
+
+try {
+ cluster = new MiniDFSCluster.Builder(conf)
+ .racks(racks).hosts(hosts).numDataNodes(hosts.length)
+ .build();
+ cluster.waitActive();
+ DistributedFileSystem fs = cluster.getFileSystem();
+ // Write a sample file
+ final Path fooName = fs.makeQualified(new Path("/foo"));
Review comment:
Can you help Why are you making it qualified here? The test passes
without that also, and we have only one FS only that is the default I suppose
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRpcServer.java
##
@@ -59,5 +69,67 @@ public void testNamenodeRpcBindAny() throws IOException {
conf.unset(DFS_NAMENODE_RPC_BIND_HOST_KEY);
}
}
+
+ /**
+ * A test to make sure that if an authorized user adds "clientIp:" to their
+ * caller context, it will be used to make locality decisions on the NN.
+ */
+ @Test
+ public void testNamenodeRpcClien
[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=744246&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744246
]
ASF GitHub Bot logged work on HDFS-13248:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 17:05
Start Date: 18/Mar/22 17:05
Worklog Time Spent: 10m
Work Description: omalley commented on a change in pull request #4081:
URL: https://github.com/apache/hadoop/pull/4081#discussion_r830193659
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRpcServer.java
##
@@ -59,5 +74,67 @@ public void testNamenodeRpcBindAny() throws IOException {
conf.unset(DFS_NAMENODE_RPC_BIND_HOST_KEY);
}
}
+
+ /**
+ * A test to make sure that if an authorized user adds "clientIp:" to their
+ * caller context, it will be used to make locality decisions on the NN.
+ */
+ @Test
+ public void testNamenodeRpcClientIpProxy()
+ throws InterruptedException, IOException {
+Configuration conf = new HdfsConfiguration();
+
+conf.set(DFS_NAMENODE_IP_PROXY_USERS, "fake_joe");
+// Make 3 nodes & racks so that we have a decent shot of detecting when
+// our change overrides the random choice of datanode.
+final String[] racks = new String[]{"/rack1", "/rack2", "/rack3"};
+final String[] hosts = new String[]{"node1", "node2", "node3"};
+MiniDFSCluster cluster = null;
+final CallerContext original = CallerContext.getCurrent();
+
+try {
+ cluster = new MiniDFSCluster.Builder(conf)
+ .racks(racks).hosts(hosts).numDataNodes(hosts.length)
+ .build();
+ cluster.waitActive();
+ DistributedFileSystem fs = cluster.getFileSystem();
+ // Write a sample file
+ final Path fooName = fs.makeQualified(new Path("/foo"));
+ FSDataOutputStream stream = fs.create(fooName);
+ stream.write("Hello world!\n".getBytes(StandardCharsets.UTF_8));
+ stream.close();
+ // Set the caller context to set the ip address
+ CallerContext.setCurrent(
+ new CallerContext.Builder("test", conf)
+ .append(CallerContext.CLIENT_IP_STR, hosts[0])
+ .build());
+ // Run as fake joe to authorize the test
+ UserGroupInformation
+ .createUserForTesting("fake_joe", new String[]{"fake_group"})
+ .doAs(new PrivilegedExceptionAction() {
+@Override
+public Object run() throws Exception {
+ // Create a new file system as the joe user
+ DistributedFileSystem joe_fs =
Review comment:
Oops. Learning Rust in my spare time is leaking through. ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744246)
Time Spent: 1h 50m (was: 1h 40m)
> RBF: Namenode need to choose block location for the client
> --
>
> Key: HDFS-13248
> URL: https://issues.apache.org/jira/browse/HDFS-13248
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wu Weiwei
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch,
> HDFS-13248.002.patch, HDFS-13248.003.patch, HDFS-13248.004.patch,
> HDFS-13248.005.patch, HDFS-Router-Data-Locality.odt, RBF Data Locality
> Design.pdf, clientMachine-call-path.jpeg, debug-info-1.jpeg, debug-info-2.jpeg
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> When execute a put operation via router, the NameNode will choose block
> location for the router, not for the real client. This will affect the file's
> locality.
> I think on both NameNode and Router, we should add a new addBlock method, or
> add a parameter for the current addBlock method, to pass the real client
> information.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[ https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=744245&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744245 ] ASF GitHub Bot logged work on HDFS-13248: - Author: ASF GitHub Bot Created on: 18/Mar/22 17:02 Start Date: 18/Mar/22 17:02 Worklog Time Spent: 10m Work Description: omalley commented on a change in pull request #4081: URL: https://github.com/apache/hadoop/pull/4081#discussion_r830191758 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRpcServer.java ## @@ -24,14 +24,29 @@ */ package org.apache.hadoop.hdfs.server.namenode; +import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_IP_PROXY_USERS; import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_RPC_BIND_HOST_KEY; import static org.junit.Assert.assertEquals; import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.charset.StandardCharsets; +import java.security.PrivilegedExceptionAction; + import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FSDataInputStream; Review comment: Sorry about that. I missed the fact that a bunch of imports were left over from my iteration on writing the test case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 744245) Time Spent: 1h 40m (was: 1.5h) > RBF: Namenode need to choose block location for the client > -- > > Key: HDFS-13248 > URL: https://issues.apache.org/jira/browse/HDFS-13248 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wu Weiwei >Assignee: Íñigo Goiri >Priority: Major > Labels: pull-request-available > Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch, > HDFS-13248.002.patch, HDFS-13248.003.patch, HDFS-13248.004.patch, > HDFS-13248.005.patch, HDFS-Router-Data-Locality.odt, RBF Data Locality > Design.pdf, clientMachine-call-path.jpeg, debug-info-1.jpeg, debug-info-2.jpeg > > Time Spent: 1h 40m > Remaining Estimate: 0h > > When execute a put operation via router, the NameNode will choose block > location for the router, not for the real client. This will affect the file's > locality. > I think on both NameNode and Router, we should add a new addBlock method, or > add a parameter for the current addBlock method, to pass the real client > information. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=744227&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744227
]
ASF GitHub Bot logged work on HDFS-13248:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 16:40
Start Date: 18/Mar/22 16:40
Worklog Time Spent: 10m
Work Description: goiri commented on a change in pull request #4081:
URL: https://github.com/apache/hadoop/pull/4081#discussion_r830172952
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRpcServer.java
##
@@ -59,5 +74,67 @@ public void testNamenodeRpcBindAny() throws IOException {
conf.unset(DFS_NAMENODE_RPC_BIND_HOST_KEY);
}
}
+
+ /**
+ * A test to make sure that if an authorized user adds "clientIp:" to their
+ * caller context, it will be used to make locality decisions on the NN.
+ */
+ @Test
+ public void testNamenodeRpcClientIpProxy()
+ throws InterruptedException, IOException {
+Configuration conf = new HdfsConfiguration();
+
+conf.set(DFS_NAMENODE_IP_PROXY_USERS, "fake_joe");
+// Make 3 nodes & racks so that we have a decent shot of detecting when
+// our change overrides the random choice of datanode.
+final String[] racks = new String[]{"/rack1", "/rack2", "/rack3"};
+final String[] hosts = new String[]{"node1", "node2", "node3"};
+MiniDFSCluster cluster = null;
+final CallerContext original = CallerContext.getCurrent();
+
+try {
+ cluster = new MiniDFSCluster.Builder(conf)
+ .racks(racks).hosts(hosts).numDataNodes(hosts.length)
+ .build();
+ cluster.waitActive();
+ DistributedFileSystem fs = cluster.getFileSystem();
+ // Write a sample file
+ final Path fooName = fs.makeQualified(new Path("/foo"));
+ FSDataOutputStream stream = fs.create(fooName);
+ stream.write("Hello world!\n".getBytes(StandardCharsets.UTF_8));
+ stream.close();
+ // Set the caller context to set the ip address
+ CallerContext.setCurrent(
+ new CallerContext.Builder("test", conf)
+ .append(CallerContext.CLIENT_IP_STR, hosts[0])
+ .build());
+ // Run as fake joe to authorize the test
+ UserGroupInformation
+ .createUserForTesting("fake_joe", new String[]{"fake_group"})
+ .doAs(new PrivilegedExceptionAction() {
+@Override
+public Object run() throws Exception {
+ // Create a new file system as the joe user
+ DistributedFileSystem joe_fs =
Review comment:
joeFs
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
##
@@ -987,6 +987,8 @@
"dfs.namenode.lifeline.handler.count";
public static final String DFS_NAMENODE_SERVICE_HANDLER_COUNT_KEY =
"dfs.namenode.service.handler.count";
public static final int DFS_NAMENODE_SERVICE_HANDLER_COUNT_DEFAULT = 10;
+ // List of users that can override their client ip
+ public static final String DFS_NAMENODE_IP_PROXY_USERS =
"dfs.namenode.ip-proxy-users";
Review comment:
TestHdfsConfigFields is not happy.
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRpcServer.java
##
@@ -24,14 +24,29 @@
*/
package org.apache.hadoop.hdfs.server.namenode;
+import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_IP_PROXY_USERS;
import static
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_RPC_BIND_HOST_KEY;
import static org.junit.Assert.assertEquals;
import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.security.PrivilegedExceptionAction;
+
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
Review comment:
A bunch of these seem unused
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744227)
Time Spent: 1.5h (was: 1h 20m)
> RBF: Namenode need to choose block location for the client
> --
>
> Key: HDFS-13248
> URL: https://issues.apache.org/jira/browse/HDFS-13248
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wu Weiwei
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch,
> HDFS
[jira] [Work logged] (HDFS-16504) add `dfs.namenode.get-blocks.check.operation` to enable or disable checkOperation when NNs process getBlocks
[
https://issues.apache.org/jira/browse/HDFS-16504?focusedWorklogId=744217&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744217
]
ASF GitHub Bot logged work on HDFS-16504:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 16:21
Start Date: 18/Mar/22 16:21
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4068:
URL: https://github.com/apache/hadoop/pull/4068#issuecomment-1072572722
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 30s | | trunk passed |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 3s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 35s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 39s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 54s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 19s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 229m 52s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not
generate ASF License warnings. |
| | | 341m 57s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4068/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4068 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 5fb053b5c1b0 4.15.0-161-generic #169-Ubuntu SMP Fri Oct 15
13:41:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 54b4ff8d3c7a9edc7245745bce652f0281537197 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4068/5/testReport/ |
| Max. process+thread count | 3263 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4068/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.
[jira] [Updated] (HDFS-16470) Make HDFS find tool cross platform
[ https://issues.apache.org/jira/browse/HDFS-16470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gautham Banasandra updated HDFS-16470: -- Description: The source files for *hdfs_find* uses *getopt* for parsing the command line arguments. getopt is available only on Linux and thus, isn't cross platform. We need to replace getopt with *boost::program_options* to make hdfs_find cross platform. (was: The source files for *hdfs_find* uses *getopt* for parsing the command line arguments. getopt is available only on Linux and thus, isn't cross platform. We need to replace getopt with *boost::program_options* to make these tools cross platform.) > Make HDFS find tool cross platform > -- > > Key: HDFS-16470 > URL: https://issues.apache.org/jira/browse/HDFS-16470 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, libhdfs++, tools >Affects Versions: 3.4.0 > Environment: Centos 7, Centos 8, Debian 10, Ubuntu Focal >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: libhdfscpp, pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > The source files for *hdfs_find* uses *getopt* for parsing the command line > arguments. getopt is available only on Linux and thus, isn't cross platform. > We need to replace getopt with *boost::program_options* to make hdfs_find > cross platform. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16470) Make HDFS find tool cross platform
[ https://issues.apache.org/jira/browse/HDFS-16470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gautham Banasandra resolved HDFS-16470. --- Fix Version/s: 3.4.0 Resolution: Fixed Merged PR https://github.com/apache/hadoop/pull/4076 to trunk. > Make HDFS find tool cross platform > -- > > Key: HDFS-16470 > URL: https://issues.apache.org/jira/browse/HDFS-16470 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, libhdfs++, tools >Affects Versions: 3.4.0 > Environment: Centos 7, Centos 8, Debian 10, Ubuntu Focal >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: libhdfscpp, pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > The source files for *hdfs_find* uses *getopt* for parsing the command line > arguments. getopt is available only on Linux and thus, isn't cross platform. > We need to replace getopt with *boost::program_options* to make these tools > cross platform. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16470) Make HDFS find tool cross platform
[ https://issues.apache.org/jira/browse/HDFS-16470?focusedWorklogId=744210&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744210 ] ASF GitHub Bot logged work on HDFS-16470: - Author: ASF GitHub Bot Created on: 18/Mar/22 16:11 Start Date: 18/Mar/22 16:11 Worklog Time Spent: 10m Work Description: GauthamBanasandra merged pull request #4076: URL: https://github.com/apache/hadoop/pull/4076 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 744210) Time Spent: 1h (was: 50m) > Make HDFS find tool cross platform > -- > > Key: HDFS-16470 > URL: https://issues.apache.org/jira/browse/HDFS-16470 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, libhdfs++, tools >Affects Versions: 3.4.0 > Environment: Centos 7, Centos 8, Debian 10, Ubuntu Focal >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: libhdfscpp, pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > The source files for *hdfs_find* uses *getopt* for parsing the command line > arguments. getopt is available only on Linux and thus, isn't cross platform. > We need to replace getopt with *boost::program_options* to make these tools > cross platform. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16507) Avoid purging edit log which is in progress
[
https://issues.apache.org/jira/browse/HDFS-16507?focusedWorklogId=744204&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744204
]
ASF GitHub Bot logged work on HDFS-16507:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 16:02
Start Date: 18/Mar/22 16:02
Worklog Time Spent: 10m
Work Description: xkrogen commented on a change in pull request #4082:
URL: https://github.com/apache/hadoop/pull/4082#discussion_r830141292
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
##
@@ -1509,13 +1509,18 @@ synchronized void abortCurrentLogSegment() {
* effect.
*/
@Override
- public synchronized void purgeLogsOlderThan(final long minTxIdToKeep) {
+ public synchronized void purgeLogsOlderThan(long minTxIdToKeep) {
// Should not purge logs unless they are open for write.
// This prevents the SBN from purging logs on shared storage, for example.
if (!isOpenForWrite()) {
return;
}
-
+
+// Reset purgeLogsFrom to avoid purging edit log which is in progress.
+if (isSegmentOpen()) {
+ minTxIdToKeep = minTxIdToKeep > curSegmentTxId ? curSegmentTxId :
minTxIdToKeep;
Review comment:
Just below this we have:
```java
assert curSegmentTxId == HdfsServerConstants.INVALID_TXID || // on
format this is no-op
minTxIdToKeep <= curSegmentTxId :
"cannot purge logs older than txid " + minTxIdToKeep +
" when current segment starts at " + curSegmentTxId;
```
We assert that `minTxIdToKeep <= curSegmentTxId`. So in the situation you
described where `minTxIdToKeep > curSegmentTxId`, shouldn't the assert fail?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744204)
Time Spent: 1.5h (was: 1h 20m)
> Avoid purging edit log which is in progress
> ---
>
> Key: HDFS-16507
> URL: https://issues.apache.org/jira/browse/HDFS-16507
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: tomscut
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> We introduced [Standby Read] feature in branch-3.1.0, but found a FATAL
> exception. It looks like it's purging edit logs which is in process.
> According to the analysis, I suspect that the editlog which is in progress to
> be purged(after SNN checkpoint) does not finalize(See HDFS-14317) before ANN
> rolls edit its self.
> The stack:
> {code:java}
> java.lang.Thread.getStackTrace(Thread.java:1552)
> org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1032)
>
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager.purgeLogsOlderThan(FileJournalManager.java:185)
>
> org.apache.hadoop.hdfs.server.namenode.JournalSet$5.apply(JournalSet.java:623)
>
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:388)
>
> org.apache.hadoop.hdfs.server.namenode.JournalSet.purgeLogsOlderThan(JournalSet.java:620)
>
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.purgeLogsOlderThan(FSEditLog.java:1512)
> org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldStorage(NNStorageRetentionManager.java:177)
>
> org.apache.hadoop.hdfs.server.namenode.FSImage.purgeOldStorage(FSImage.java:1249)
>
> org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:617)
>
> org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:516)
> java.security.AccessController.doPrivileged(Native Method)
> javax.security.auth.Subject.doAs(Subject.java:422)
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
>
> org.apache.hadoop.hdfs.server.namenode.ImageServlet.doPut(ImageServlet.java:515)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
>
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1604)
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
>
> org.ecli
[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[ https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=744076&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744076 ] ASF GitHub Bot logged work on HDFS-13248: - Author: ASF GitHub Bot Created on: 18/Mar/22 15:03 Start Date: 18/Mar/22 15:03 Worklog Time Spent: 10m Work Description: omalley opened a new pull request #4081: URL: https://github.com/apache/hadoop/pull/4081 The NN makes decisions based on the client machine that control the locality of data access. Currently that is done by finding the ip address using the rpc connection, however in the RBF configuration, that will always be one of the router's ip address. We'd added the client's ip to the caller context in the router, so now the NN has the information. This patch makes the NN use the caller context information. From a security point of view, this patch adds a new configuration knob (dfs.namenode.ip-proxy-users) on the NN that defines the list of users that can set their client ip address. Sites should add "hdfs" (or the account that runs the routers) to "dfs.namenode.ip-proxy-users" on the NN to enable this feature. Note that the audit log does NOT currently use this information, so the client ip in the audit log will be the RBF proxy. Sites should turn on caller context logging so that the client ip addresses are captured. ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 744076) Time Spent: 1h 20m (was: 1h 10m) > RBF: Namenode need to choose block location for the client > -- > > Key: HDFS-13248 > URL: https://issues.apache.org/jira/browse/HDFS-13248 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wu Weiwei >Assignee: Íñigo Goiri >Priority: Major > Labels: pull-request-available > Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch, > HDFS-13248.002.patch, HDFS-13248.003.patch, HDFS-13248.004.patch, > HDFS-13248.005.patch, HDFS-Router-Data-Locality.odt, RBF Data Locality > Design.pdf, clientMachine-call-path.jpeg, debug-info-1.jpeg, debug-info-2.jpeg > > Time Spent: 1h 20m > Remaining Estimate: 0h > > When execute a put operation via router, the NameNode will choose block > location for the router, not for the real client. This will affect the file's > locality. > I think on both NameNode and Router, we should add a new addBlock method, or > add a parameter for the current addBlock method, to pass the real client > information. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16510) Fix EC decommission when rack is not enough
[ https://issues.apache.org/jira/browse/HDFS-16510?focusedWorklogId=743919&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-743919 ] ASF GitHub Bot logged work on HDFS-16510: - Author: ASF GitHub Bot Created on: 18/Mar/22 14:48 Start Date: 18/Mar/22 14:48 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #4078: URL: https://github.com/apache/hadoop/pull/4078#issuecomment-1071052723 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 743919) Time Spent: 50m (was: 40m) > Fix EC decommission when rack is not enough > --- > > Key: HDFS-16510 > URL: https://issues.apache.org/jira/browse/HDFS-16510 > Project: Hadoop HDFS > Issue Type: Bug > Components: block placement, ec >Affects Versions: 3.3.1, 3.3.2 >Reporter: daimin >Assignee: daimin >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > The decommission always fail when we start decommission multiple nodes on a > cluster whose racks is not enough, a cluster with 6 racks to deploy RS-6-3, > for example. > We find that those decommission nodes cover at least a rack, it's actulaly > like we are decommission one or more racks. And rack decommission is not well > supported currently, especially for cluster whose racks is not enough already. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16507) Avoid purging edit log which is in progress
[
https://issues.apache.org/jira/browse/HDFS-16507?focusedWorklogId=744041&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744041
]
ASF GitHub Bot logged work on HDFS-16507:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 14:59
Start Date: 18/Mar/22 14:59
Worklog Time Spent: 10m
Work Description: tomscut opened a new pull request #4079:
URL: https://github.com/apache/hadoop/pull/4079
JIRA: [HDFS-16507](https://issues.apache.org/jira/browse/HDFS-16507).
Avoid purging edit log which is in progress. This is described in detail in
JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744041)
Time Spent: 1h 20m (was: 1h 10m)
> Avoid purging edit log which is in progress
> ---
>
> Key: HDFS-16507
> URL: https://issues.apache.org/jira/browse/HDFS-16507
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: tomscut
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> We introduced [Standby Read] feature in branch-3.1.0, but found a FATAL
> exception. It looks like it's purging edit logs which is in process.
> According to the analysis, I suspect that the editlog which is in progress to
> be purged(after SNN checkpoint) does not finalize(See HDFS-14317) before ANN
> rolls edit its self.
> The stack:
> {code:java}
> java.lang.Thread.getStackTrace(Thread.java:1552)
> org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1032)
>
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager.purgeLogsOlderThan(FileJournalManager.java:185)
>
> org.apache.hadoop.hdfs.server.namenode.JournalSet$5.apply(JournalSet.java:623)
>
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:388)
>
> org.apache.hadoop.hdfs.server.namenode.JournalSet.purgeLogsOlderThan(JournalSet.java:620)
>
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.purgeLogsOlderThan(FSEditLog.java:1512)
> org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldStorage(NNStorageRetentionManager.java:177)
>
> org.apache.hadoop.hdfs.server.namenode.FSImage.purgeOldStorage(FSImage.java:1249)
>
> org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:617)
>
> org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:516)
> java.security.AccessController.doPrivileged(Native Method)
> javax.security.auth.Subject.doAs(Subject.java:422)
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
>
> org.apache.hadoop.hdfs.server.namenode.ImageServlet.doPut(ImageServlet.java:515)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
>
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1604)
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> org.eclipse.jetty.server.Server.handle(Server.java:539)
> org.eclipse.jetty
[jira] [Work logged] (HDFS-15987) Improve oiv tool to parse fsimage file in parallel with delimited format
[
https://issues.apache.org/jira/browse/HDFS-15987?focusedWorklogId=744062&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744062
]
ASF GitHub Bot logged work on HDFS-15987:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 15:02
Start Date: 18/Mar/22 15:02
Worklog Time Spent: 10m
Work Description: Hexiaoqiao commented on pull request #2918:
URL: https://github.com/apache/hadoop/pull/2918#issuecomment-1070851373
Thanks @whbing for your updates. It almost look good to me. Please fix
checkstyle issue also.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744062)
Time Spent: 6h 10m (was: 6h)
> Improve oiv tool to parse fsimage file in parallel with delimited format
>
>
> Key: HDFS-15987
> URL: https://issues.apache.org/jira/browse/HDFS-15987
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Hongbing Wang
>Assignee: Hongbing Wang
>Priority: Major
> Labels: pull-request-available
> Attachments: Improve_oiv_tool_001.pdf
>
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> The purpose of this Jira is to improve oiv tool to parse fsimage file with
> sub-sections (see -HDFS-14617-) in parallel with delmited format.
> 1.Serial parsing is time-consuming
> The time to serially parse a large fsimage with delimited format (e.g. `hdfs
> oiv -p Delimited -t ...`) is as follows:
> {code:java}
> 1) Loading string table: -> Not time consuming.
> 2) Loading inode references: -> Not time consuming
> 3) Loading directories in INode section: -> Slightly time consuming (3%)
> 4) Loading INode directory section: -> A bit time consuming (11%)
> 5) Output: -> Very time consuming (86%){code}
> Therefore, output is the most parallelized stage.
> 2.How to output in parallel
> The sub-sections are grouped in order, and each thread processes a group and
> outputs it to the file corresponding to each thread, and finally merges the
> output files.
> 3. The result of a test
> {code:java}
> input fsimage file info:
> 3.4G, 12 sub-sections, 55976500 INodes
> -
> Threads TotalTime OutputTime MergeTime
> 1 18m37s 16m18s –
> 48m7s 4m49s 41s{code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15987) Improve oiv tool to parse fsimage file in parallel with delimited format
[
https://issues.apache.org/jira/browse/HDFS-15987?focusedWorklogId=743986&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-743986
]
ASF GitHub Bot logged work on HDFS-15987:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 14:54
Start Date: 18/Mar/22 14:54
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2918:
URL: https://github.com/apache/hadoop/pull/2918#issuecomment-1072132368
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 30s | | trunk passed |
| +1 :green_heart: | compile | 1m 33s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 29s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 49s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 24s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 43s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 359m 32s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 469m 28s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2918/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2918 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 3635b55300a7 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8
19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 362762d88d75c8d11cab324c3f9ce79d638e7be3 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2918/4/testReport/ |
| Max. process+thread count | 2219 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2918/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This
[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=743980&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-743980
]
ASF GitHub Bot logged work on HDFS-13248:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 14:54
Start Date: 18/Mar/22 14:54
Worklog Time Spent: 10m
Work Description: goiri commented on a change in pull request #4081:
URL: https://github.com/apache/hadoop/pull/4081#discussion_r829554238
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
##
@@ -1899,7 +1907,29 @@ private void verifySoftwareVersion(DatanodeRegistration
dnReg)
}
}
- private static String getClientMachine() {
+ private String getClientMachine() {
+if (ipProxyUsers != null) {
+ // Get the real user (or effective if it isn't a proxy user)
Review comment:
To avoid having a long getClientMachine(), it may be better to have a
separate private to get the real user.
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
##
@@ -987,6 +987,8 @@
"dfs.namenode.lifeline.handler.count";
public static final String DFS_NAMENODE_SERVICE_HANDLER_COUNT_KEY =
"dfs.namenode.service.handler.count";
public static final int DFS_NAMENODE_SERVICE_HANDLER_COUNT_DEFAULT = 10;
+ // List of users that can override their client ip
+ public static final String DFS_NAMENODE_IP_PROXY_USERS =
"dfs.namenode.ip-proxy-users";
Review comment:
Space to align with the rest.
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
##
@@ -1899,7 +1907,29 @@ private void verifySoftwareVersion(DatanodeRegistration
dnReg)
}
}
- private static String getClientMachine() {
+ private String getClientMachine() {
+if (ipProxyUsers != null) {
+ // Get the real user (or effective if it isn't a proxy user)
Review comment:
It would also help to have a comment with the example of what the key
you are passing looks like.
Potentially could have its own unit test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 743980)
Time Spent: 1h 10m (was: 1h)
> RBF: Namenode need to choose block location for the client
> --
>
> Key: HDFS-13248
> URL: https://issues.apache.org/jira/browse/HDFS-13248
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wu Weiwei
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch,
> HDFS-13248.002.patch, HDFS-13248.003.patch, HDFS-13248.004.patch,
> HDFS-13248.005.patch, HDFS-Router-Data-Locality.odt, RBF Data Locality
> Design.pdf, clientMachine-call-path.jpeg, debug-info-1.jpeg, debug-info-2.jpeg
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> When execute a put operation via router, the NameNode will choose block
> location for the router, not for the real client. This will affect the file's
> locality.
> I think on both NameNode and Router, we should add a new addBlock method, or
> add a parameter for the current addBlock method, to pass the real client
> information.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16507) Avoid purging edit log which is in progress
[
https://issues.apache.org/jira/browse/HDFS-16507?focusedWorklogId=743922&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-743922
]
ASF GitHub Bot logged work on HDFS-16507:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 14:48
Start Date: 18/Mar/22 14:48
Worklog Time Spent: 10m
Work Description: tomscut closed pull request #4079:
URL: https://github.com/apache/hadoop/pull/4079
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 743922)
Time Spent: 1h 10m (was: 1h)
> Avoid purging edit log which is in progress
> ---
>
> Key: HDFS-16507
> URL: https://issues.apache.org/jira/browse/HDFS-16507
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: tomscut
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> We introduced [Standby Read] feature in branch-3.1.0, but found a FATAL
> exception. It looks like it's purging edit logs which is in process.
> According to the analysis, I suspect that the editlog which is in progress to
> be purged(after SNN checkpoint) does not finalize(See HDFS-14317) before ANN
> rolls edit its self.
> The stack:
> {code:java}
> java.lang.Thread.getStackTrace(Thread.java:1552)
> org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1032)
>
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager.purgeLogsOlderThan(FileJournalManager.java:185)
>
> org.apache.hadoop.hdfs.server.namenode.JournalSet$5.apply(JournalSet.java:623)
>
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:388)
>
> org.apache.hadoop.hdfs.server.namenode.JournalSet.purgeLogsOlderThan(JournalSet.java:620)
>
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.purgeLogsOlderThan(FSEditLog.java:1512)
> org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldStorage(NNStorageRetentionManager.java:177)
>
> org.apache.hadoop.hdfs.server.namenode.FSImage.purgeOldStorage(FSImage.java:1249)
>
> org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:617)
>
> org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:516)
> java.security.AccessController.doPrivileged(Native Method)
> javax.security.auth.Subject.doAs(Subject.java:422)
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
>
> org.apache.hadoop.hdfs.server.namenode.ImageServlet.doPut(ImageServlet.java:515)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
>
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1604)
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> org.eclipse.jetty.server.Server.handle(Server.java:539)
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
>
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
>
> org.eclipse.jetty.io.Abstr
[jira] [Work logged] (HDFS-16499) [SPS]: Should not start indefinitely while another SPS process is running
[ https://issues.apache.org/jira/browse/HDFS-16499?focusedWorklogId=743908&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-743908 ] ASF GitHub Bot logged work on HDFS-16499: - Author: ASF GitHub Bot Created on: 18/Mar/22 14:47 Start Date: 18/Mar/22 14:47 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #4058: URL: https://github.com/apache/hadoop/pull/4058#issuecomment-1071901635 Thanks @umamaheswararao and @ferhui . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 743908) Time Spent: 2h 20m (was: 2h 10m) > [SPS]: Should not start indefinitely while another SPS process is running > - > > Key: HDFS-16499 > URL: https://issues.apache.org/jira/browse/HDFS-16499 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Normally, we can only start one SPS process at a time. When one process is > running, start another process and retry indefinitely. I think, in this case, > we should exit immediately. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16507) Avoid purging edit log which is in progress
[
https://issues.apache.org/jira/browse/HDFS-16507?focusedWorklogId=743884&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-743884
]
ASF GitHub Bot logged work on HDFS-16507:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 14:44
Start Date: 18/Mar/22 14:44
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4082:
URL: https://github.com/apache/hadoop/pull/4082#issuecomment-1072003345
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 26s | | trunk passed |
| +1 :green_heart: | compile | 1m 27s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 18s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 11s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 11s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 51s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 16s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 52s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 21s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 110m 34s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4082/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +0 :ok: | asflicense | 0m 33s | | ASF License check generated no
output? |
| | | 210m 38s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestReplication |
| | hadoop.hdfs.TestWriteReadStripedFile |
| | hadoop.hdfs.TestDFSStartupVersions |
| | hadoop.hdfs.TestRestartDFS |
| | hadoop.hdfs.TestReadStripedFileWithDecodingDeletedData |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestDatanodeReport |
| | hadoop.hdfs.TestFileAppend2 |
| | hadoop.hdfs.TestReadStripedFileWithDecodingCorruptData |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4082/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4082 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux c4f147c716ed 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | mave
[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[ https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=743792&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-743792 ] ASF GitHub Bot logged work on HDFS-13248: - Author: ASF GitHub Bot Created on: 18/Mar/22 14:36 Start Date: 18/Mar/22 14:36 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #4081: URL: https://github.com/apache/hadoop/pull/4081#issuecomment-1072237010 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 743792) Time Spent: 1h (was: 50m) > RBF: Namenode need to choose block location for the client > -- > > Key: HDFS-13248 > URL: https://issues.apache.org/jira/browse/HDFS-13248 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Wu Weiwei >Assignee: Íñigo Goiri >Priority: Major > Labels: pull-request-available > Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch, > HDFS-13248.002.patch, HDFS-13248.003.patch, HDFS-13248.004.patch, > HDFS-13248.005.patch, HDFS-Router-Data-Locality.odt, RBF Data Locality > Design.pdf, clientMachine-call-path.jpeg, debug-info-1.jpeg, debug-info-2.jpeg > > Time Spent: 1h > Remaining Estimate: 0h > > When execute a put operation via router, the NameNode will choose block > location for the router, not for the real client. This will affect the file's > locality. > I think on both NameNode and Router, we should add a new addBlock method, or > add a parameter for the current addBlock method, to pass the real client > information. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16507) Avoid purging edit log which is in progress
[ https://issues.apache.org/jira/browse/HDFS-16507?focusedWorklogId=743799&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-743799 ] ASF GitHub Bot logged work on HDFS-16507: - Author: ASF GitHub Bot Created on: 18/Mar/22 14:36 Start Date: 18/Mar/22 14:36 Worklog Time Spent: 10m Work Description: tomscut opened a new pull request #4082: URL: https://github.com/apache/hadoop/pull/4082 JIRA: [HDFS-16507](https://issues.apache.org/jira/browse/HDFS-16507). Avoid purging edit log which is in progress. This is described in detail in JIRA. The stack: ``` java.lang.Thread.getStackTrace(Thread.java:1552) org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1032) org.apache.hadoop.hdfs.server.namenode.FileJournalManager.purgeLogsOlderThan(FileJournalManager.java:185) org.apache.hadoop.hdfs.server.namenode.JournalSet$5.apply(JournalSet.java:623) org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:388) org.apache.hadoop.hdfs.server.namenode.JournalSet.purgeLogsOlderThan(JournalSet.java:620) org.apache.hadoop.hdfs.server.namenode.FSEditLog.purgeLogsOlderThan(FSEditLog.java:1512) org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldStorage(NNStorageRetentionManager.java:177) org.apache.hadoop.hdfs.server.namenode.FSImage.purgeOldStorage(FSImage.java:1249) org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:617) org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:516) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:422) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) org.apache.hadoop.hdfs.server.namenode.ImageServlet.doPut(ImageServlet.java:515) javax.servlet.http.HttpServlet.service(HttpServlet.java:710) javax.servlet.http.HttpServlet.service(HttpServlet.java:790) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772) org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1604) org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) org.eclipse.jetty.server.Server.handle(Server.java:539) org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) java.lang.Thread.run(Thread.java:745) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, pleas
[jira] [Work logged] (HDFS-16504) add `dfs.namenode.get-blocks.check.operation` to enable or disable checkOperation when NNs process getBlocks
[ https://issues.apache.org/jira/browse/HDFS-16504?focusedWorklogId=743787&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-743787 ] ASF GitHub Bot logged work on HDFS-16504: - Author: ASF GitHub Bot Created on: 18/Mar/22 14:35 Start Date: 18/Mar/22 14:35 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #4068: URL: https://github.com/apache/hadoop/pull/4068#issuecomment-1070973085 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 743787) Time Spent: 1.5h (was: 1h 20m) > add `dfs.namenode.get-blocks.check.operation` to enable or disable > checkOperation when NNs process getBlocks > > > Key: HDFS-16504 > URL: https://issues.apache.org/jira/browse/HDFS-16504 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover, namanode >Affects Versions: 3.4.0 >Reporter: Max Xie >Assignee: Max Xie >Priority: Minor > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > HDFS-13183 add a nice feature that Standby NameNode can process getBlocks > request to reduce Active load. Namenode must set `dfs.ha.allow.stale.reads > = true` to enable this feature. However, if we set `dfs.ha.allow.stale.reads > = true`, Standby Namenode will be able to process all read requests, which > may lead to yarn jobs fail because Standby Namenode is stale . > Maybe we should add a config `dfs.namenode.get-blocks.check.operation=false` > for namenode to disable check operation when namenode process getBlocks > request. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16457) Make fs.getspaceused.classname reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16457?focusedWorklogId=743765&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-743765 ] ASF GitHub Bot logged work on HDFS-16457: - Author: ASF GitHub Bot Created on: 18/Mar/22 14:33 Start Date: 18/Mar/22 14:33 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #4069: URL: https://github.com/apache/hadoop/pull/4069#issuecomment-1071181384 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 743765) Time Spent: 1h 20m (was: 1h 10m) > Make fs.getspaceused.classname reconfigurable > - > > Key: HDFS-16457 > URL: https://issues.apache.org/jira/browse/HDFS-16457 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: yanbin.zhang >Assignee: yanbin.zhang >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Now if we want to switch fs.getspaceused.classname we need to restart the > NameNode. It would be convenient if we can switch it at runtime. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=743669&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-743669
]
ASF GitHub Bot logged work on HDFS-13248:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 10:09
Start Date: 18/Mar/22 10:09
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4081:
URL: https://github.com/apache/hadoop/pull/4081#issuecomment-1072261340
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 29s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 32s | | trunk passed |
| +1 :green_heart: | compile | 28m 35s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 23m 4s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 7s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 9s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 29s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 8m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 58s | | the patch passed |
| +1 :green_heart: | compile | 27m 29s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 27m 29s | | the patch passed |
| +1 :green_heart: | compile | 23m 21s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 23m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 46s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4081/2/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 6 new + 415 unchanged - 0 fixed = 421 total (was
415) |
| +1 :green_heart: | mvnsite | 4m 3s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 2s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 4m 32s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 8m 41s | | the patch passed |
| +1 :green_heart: | shadedclient | 27m 12s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 21m 33s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 375m 31s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4081/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 41m 3s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4081/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 19s | | The patch does not
generate ASF License warnings. |
| | | 693m 40s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4081/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4081 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkst
[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=743663&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-743663
]
ASF GitHub Bot logged work on HDFS-13248:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 09:53
Start Date: 18/Mar/22 09:53
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4081:
URL: https://github.com/apache/hadoop/pull/4081#issuecomment-1072249735
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 18s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 32m 39s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4081/3/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 24m 49s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 21m 13s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 49s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 8s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 4m 27s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 38s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 55s | | the patch passed |
| +1 :green_heart: | compile | 24m 1s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 24m 1s | | the patch passed |
| +1 :green_heart: | compile | 21m 10s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 21m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 43s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4081/3/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 6 new + 415 unchanged - 0 fixed = 421 total (was
415) |
| +1 :green_heart: | mvnsite | 3m 59s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 1s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 4m 27s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 8m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 34s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 323m 2s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4081/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 37m 47s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4081/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 1s | | The patch does not
generate ASF License warnings. |
| | | 619m 1s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/
[jira] [Commented] (HDFS-14630) Configuration.getTimeDurationHelper() should not log time unit warning in info log.
[
https://issues.apache.org/jira/browse/HDFS-14630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17508676#comment-17508676
]
Masatake Iwasaki commented on HDFS-14630:
-
cherry-picked this to branch-3.2.
https://lists.apache.org/thread/2pn7go8wx6w8tftwf3gotjh7rvzndv6z
> Configuration.getTimeDurationHelper() should not log time unit warning in
> info log.
> ---
>
> Key: HDFS-14630
> URL: https://issues.apache.org/jira/browse/HDFS-14630
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Hemanth Boyina
>Priority: Minor
> Fix For: 3.3.0, 3.2.4
>
> Attachments: HDFS-14630.001.patch, HDFS-14630.patch
>
>
> To solve [HDFS-12920|https://issues.apache.org/jira/browse/HDFS-12920] issue
> we configured "dfs.client.datanode-restart.timeout" without time unit. No log
> file is full of
> {noformat}
> 2019-06-22 20:13:14,605 | INFO | pool-12-thread-1 | No unit for
> dfs.client.datanode-restart.timeout(30) assuming SECONDS
> org.apache.hadoop.conf.Configuration.logDeprecation(Configuration.java:1409){noformat}
> No need to log this, just give the behavior in property description.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14630) Configuration.getTimeDurationHelper() should not log time unit warning in info log.
[
https://issues.apache.org/jira/browse/HDFS-14630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated HDFS-14630:
Fix Version/s: 3.2.4
> Configuration.getTimeDurationHelper() should not log time unit warning in
> info log.
> ---
>
> Key: HDFS-14630
> URL: https://issues.apache.org/jira/browse/HDFS-14630
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Hemanth Boyina
>Priority: Minor
> Fix For: 3.3.0, 3.2.4
>
> Attachments: HDFS-14630.001.patch, HDFS-14630.patch
>
>
> To solve [HDFS-12920|https://issues.apache.org/jira/browse/HDFS-12920] issue
> we configured "dfs.client.datanode-restart.timeout" without time unit. No log
> file is full of
> {noformat}
> 2019-06-22 20:13:14,605 | INFO | pool-12-thread-1 | No unit for
> dfs.client.datanode-restart.timeout(30) assuming SECONDS
> org.apache.hadoop.conf.Configuration.logDeprecation(Configuration.java:1409){noformat}
> No need to log this, just give the behavior in property description.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13248) RBF: Namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?focusedWorklogId=743653&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-743653
]
ASF GitHub Bot logged work on HDFS-13248:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 09:36
Start Date: 18/Mar/22 09:36
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4081:
URL: https://github.com/apache/hadoop/pull/4081#issuecomment-1072237010
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 26s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 6s | | trunk passed |
| +1 :green_heart: | compile | 27m 13s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 24m 40s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 49s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 20s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 5m 16s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 8m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 45s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 9s | | the patch passed |
| +1 :green_heart: | compile | 25m 28s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 25m 28s | | the patch passed |
| +1 :green_heart: | compile | 25m 4s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 25m 4s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 0s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4081/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 6 new + 350 unchanged - 0 fixed = 356 total (was
350) |
| +1 :green_heart: | mvnsite | 4m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 19s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 5m 13s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 8m 54s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 22s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 19s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 376m 56s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4081/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 42m 38s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4081/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 32s | | The patch does not
generate ASF License warnings. |
| | | 690m 29s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
| | hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate |
| | hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4081/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4081 |
| Optional Tests | dupname asflicense com
[jira] [Work logged] (HDFS-16510) Fix EC decommission when rack is not enough
[
https://issues.apache.org/jira/browse/HDFS-16510?focusedWorklogId=743648&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-743648
]
ASF GitHub Bot logged work on HDFS-16510:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 09:11
Start Date: 18/Mar/22 09:11
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4078:
URL: https://github.com/apache/hadoop/pull/4078#issuecomment-1072218060
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 0s | | trunk passed |
| +1 :green_heart: | compile | 1m 37s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 39s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 29s | | the patch passed |
| +1 :green_heart: | compile | 1m 43s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 43s | | the patch passed |
| +1 :green_heart: | compile | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 59s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 26s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 28m 42s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 268m 12s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4078/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 383m 28s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.tools.TestECAdmin |
| | hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr |
| | hadoop.hdfs.TestErasureCodingExerciseAPIs |
| | hadoop.hdfs.tools.TestViewFileSystemOverloadSchemeWithDFSAdmin |
| | hadoop.hdfs.tools.TestDFSAdmin |
| | hadoop.hdfs.server.namenode.TestFsck |
| |
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks |
| | hadoop.cli.TestHDFSCLI |
| | hadoop.hdfs.TestDFSStripedOutputStream |
| | hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.tools.TestDFSAdminWithHA |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4078/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4078 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux c0ddad626b35 4.15
[jira] [Work logged] (HDFS-16504) add `dfs.namenode.get-blocks.check.operation` to enable or disable checkOperation when NNs process getBlocks
[
https://issues.apache.org/jira/browse/HDFS-16504?focusedWorklogId=743637&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-743637
]
ASF GitHub Bot logged work on HDFS-16504:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 08:56
Start Date: 18/Mar/22 08:56
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4068:
URL: https://github.com/apache/hadoop/pull/4068#issuecomment-1072206848
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 34s | | trunk passed |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 21s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 21s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 19s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 25s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 21s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 228m 41s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4068/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 329m 8s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4068/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4068 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 046f135c3b52 4.15.0-161-generic #169-Ubuntu SMP Fri Oct 15
13:41:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 92b0436f89d0fd4f8dd147e0b9699e57e7664164 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4068/4/testReport/
[jira] [Work logged] (HDFS-15987) Improve oiv tool to parse fsimage file in parallel with delimited format
[
https://issues.apache.org/jira/browse/HDFS-15987?focusedWorklogId=743630&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-743630
]
ASF GitHub Bot logged work on HDFS-15987:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 08:17
Start Date: 18/Mar/22 08:17
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2918:
URL: https://github.com/apache/hadoop/pull/2918#issuecomment-1072132368
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 30s | | trunk passed |
| +1 :green_heart: | compile | 1m 33s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 29s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 49s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 24s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 43s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 359m 32s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 469m 28s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2918/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2918 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 3635b55300a7 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8
19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 362762d88d75c8d11cab324c3f9ce79d638e7be3 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2918/4/testReport/ |
| Max. process+thread count | 2219 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2918/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This
[jira] [Work logged] (HDFS-16457) Make fs.getspaceused.classname reconfigurable
[
https://issues.apache.org/jira/browse/HDFS-16457?focusedWorklogId=743622&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-743622
]
ASF GitHub Bot logged work on HDFS-16457:
-
Author: ASF GitHub Bot
Created on: 18/Mar/22 08:05
Start Date: 18/Mar/22 08:05
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4069:
URL: https://github.com/apache/hadoop/pull/4069#issuecomment-1072121348
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 30s | | trunk passed |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 35s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 36s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 22m 56s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 52s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 16s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 24s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 20s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 227m 54s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4069/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not
generate ASF License warnings. |
| | | 328m 2s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeReconfiguration |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4069/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4069 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux c7e150c9e875 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3
10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d0f7902b8546b76020460e0fa84ce477d1a70dfd |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| T
