[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[ https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17724052#comment-17724052 ] Ayush Saxena commented on HDFS-16979: - Committed to trunk. Thanx [~liuguanghua] for the contribution, [~elgoiri] and [~simbadzina] for the reviews!!! Note: Added [~liuguanghua] as HDFS contributor to assign the ticket. Welcome to Hadoop!!! > RBF: Add dfsrouter port in hdfsauditlog > --- > > Key: HDFS-16979 > URL: https://issues.apache.org/jira/browse/HDFS-16979 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: liuguanghua >Assignee: liuguanghua >Priority: Major > Labels: pull-request-available > > > when client is using proxyuser via realuser, the hdfs aduilg log is lack of > dfsrouter port infomation. > client (using proxyuser)-> dfsrouter -> namenode > clientport dfsrouterport > hdfsauditlog should record dfsrouterport -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722103#comment-17722103
]
ASF GitHub Bot commented on HDFS-16979:
---
hadoop-yetus commented on PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#issuecomment-1545509896
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 57s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 53s | | trunk passed |
| +1 :green_heart: | compile | 17m 30s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 15m 47s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 4m 3s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 58s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 1s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 7m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 32s | | the patch passed |
| +1 :green_heart: | compile | 16m 33s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 16m 33s | | the patch passed |
| +1 :green_heart: | compile | 15m 36s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 15m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 51s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 40s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 51s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 2s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 7m 41s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 19s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 15s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 225m 0s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/14/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 21m 29s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 7s | | The patch does not
generate ASF License warnings. |
| | | 468m 22s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/14/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3ed84b55e9bf 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3
19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / aa90ae5efbf8082da46544d828ecf076c857b669 |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions |
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[ https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721982#comment-17721982 ] ASF GitHub Bot commented on HDFS-16979: --- LiuGuH commented on PR #5552: URL: https://github.com/apache/hadoop/pull/5552#issuecomment-1545015145 > The new code looks good to me. > > @LiuGuH for future changes, please keep the commit history so people can see the changes between reviews. > The new code looks good to me. > > @LiuGuH for future changes, please keep the commit history so people can see the changes between reviews. OK，I'll pay attention to that. Thank you. > RBF: Add dfsrouter port in hdfsauditlog > --- > > Key: HDFS-16979 > URL: https://issues.apache.org/jira/browse/HDFS-16979 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: liuguanghua >Priority: Major > Labels: pull-request-available > > > when client is using proxyuser via realuser, the hdfs aduilg log is lack of > dfsrouter port infomation. > client (using proxyuser)-> dfsrouter -> namenode > clientport dfsrouterport > hdfsauditlog should record dfsrouterport -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721507#comment-17721507
]
ASF GitHub Bot commented on HDFS-16979:
---
hadoop-yetus commented on PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#issuecomment-1542630798
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 37s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 55s | | trunk passed |
| +1 :green_heart: | compile | 17m 24s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 15m 41s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 4m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 4s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 29s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 7m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 4s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 32s | | the patch passed |
| +1 :green_heart: | compile | 16m 39s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 16m 39s | | the patch passed |
| +1 :green_heart: | compile | 15m 38s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 15m 38s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 52s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/13/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 113 unchanged - 0 fixed = 114 total (was
113) |
| +1 :green_heart: | mvnsite | 3m 43s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 59s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 25s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 7m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 12s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 226m 14s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/13/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 21m 58s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/13/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 2s | | The patch does not
generate ASF License warnings. |
| | | 470m 49s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| |
hadoop.hdfs.server.federation.router.TestRouterRPCMultipleDestinationMountTableResolver
|
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/13/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721482#comment-17721482
]
ASF GitHub Bot commented on HDFS-16979:
---
ayushtkn commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1190216959
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java:
##
@@ -2116,6 +2121,8 @@ public void testRealUserPropagationInCallerContext()
// Real user is added to the caller context.
assertTrue("The audit log should contain the real user.",
logOutput.contains(String.format("realUser:%s",
realUser.getUserName(;
+assertTrue("The audit log should contain the proxyuser port.",
+logOutput.contains(PROXY_USER_PORT));
Review Comment:
hmm, I think then we can live without it :)
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721315#comment-17721315
]
ASF GitHub Bot commented on HDFS-16979:
---
LiuGuH commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1189796280
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java:
##
@@ -2116,6 +2121,8 @@ public void testRealUserPropagationInCallerContext()
// Real user is added to the caller context.
assertTrue("The audit log should contain the real user.",
logOutput.contains(String.format("realUser:%s",
realUser.getUserName(;
+assertTrue("The audit log should contain the proxyuser port.",
+logOutput.contains(PROXY_USER_PORT));
Review Comment:
I saw it. It needs some time for me to achieve it. The correct router port
should get from correct Namenode Server.getRemotePort via the right rpc call.
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721289#comment-17721289
]
ASF GitHub Bot commented on HDFS-16979:
---
ayushtkn commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1189719329
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java:
##
@@ -2116,6 +2121,8 @@ public void testRealUserPropagationInCallerContext()
// Real user is added to the caller context.
assertTrue("The audit log should contain the real user.",
logOutput.contains(String.format("realUser:%s",
realUser.getUserName(;
+assertTrue("The audit log should contain the proxyuser port.",
+logOutput.contains(PROXY_USER_PORT));
Review Comment:
@LiuGuH did you miss this? checking for the actual port in the assert, if it
is the correct router port or not
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721288#comment-17721288
]
ASF GitHub Bot commented on HDFS-16979:
---
LiuGuH commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1189716104
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -462,13 +462,24 @@ private void appendClientPortToCallerContextIfAbsent() {
.setSignature(origSignature)
.build());
}
+ctx = CallerContext.getCurrent();
+if (isFormProxyUser(ctx)) {
+ CallerContext.setCurrent(
+ new CallerContext.Builder(ctx.getContext(), contextFieldSeparator)
+ .append(CallerContext.PROXY_USER_PORT,
String.valueOf(Server.getRemotePort()))
+ .setSignature(ctx.getSignature())
+ .build());
+}
}
private boolean isClientPortInfoAbsent(CallerContext ctx){
return ctx == null || ctx.getContext() == null
|| !ctx.getContext().contains(CallerContext.CLIENT_PORT_STR);
}
+ private boolean isFormProxyUser(CallerContext ctx) {
+return ctx != null &&
ctx.getContext().contains(CallerContext.REAL_USER_STR);
+ }
Review Comment:
Fixed. Thanks
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721287#comment-17721287
]
ASF GitHub Bot commented on HDFS-16979:
---
LiuGuH commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1189714857
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java:
##
@@ -2116,6 +2121,8 @@ public void testRealUserPropagationInCallerContext()
// Real user is added to the caller context.
assertTrue("The audit log should contain the real user.",
logOutput.contains(String.format("realUser:%s",
realUser.getUserName(;
+assertTrue("The audit log should contain the proxyuser port.",
+logOutput.contains(PROXY_USER_PORT));
Review Comment:
Fixed. Thanks
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721235#comment-17721235
]
ASF GitHub Bot commented on HDFS-16979:
---
ayushtkn commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1189473819
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -462,13 +462,24 @@ private void appendClientPortToCallerContextIfAbsent() {
.setSignature(origSignature)
.build());
}
+ctx = CallerContext.getCurrent();
+if (isFormProxyUser(ctx)) {
+ CallerContext.setCurrent(
+ new CallerContext.Builder(ctx.getContext(), contextFieldSeparator)
+ .append(CallerContext.PROXY_USER_PORT,
String.valueOf(Server.getRemotePort()))
+ .setSignature(ctx.getSignature())
+ .build());
+}
}
private boolean isClientPortInfoAbsent(CallerContext ctx){
return ctx == null || ctx.getContext() == null
|| !ctx.getContext().contains(CallerContext.CLIENT_PORT_STR);
}
+ private boolean isFormProxyUser(CallerContext ctx) {
+return ctx != null &&
ctx.getContext().contains(CallerContext.REAL_USER_STR);
+ }
Review Comment:
typo: form -> from
Add a null check for getContext as well.
```
private boolean isFromProxyUser(CallerContext ctx) {
return ctx != null && ctx.getContext() != null && ctx.getContext()
.contains(CallerContext.REAL_USER_STR);
}
```
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java:
##
@@ -2116,6 +2121,8 @@ public void testRealUserPropagationInCallerContext()
// Real user is added to the caller context.
assertTrue("The audit log should contain the real user.",
logOutput.contains(String.format("realUser:%s",
realUser.getUserName(;
+assertTrue("The audit log should contain the proxyuser port.",
+logOutput.contains(PROXY_USER_PORT));
Review Comment:
can you assert the router port as well? There are 2 routers and it should be
either of them?
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720197#comment-17720197
]
ASF GitHub Bot commented on HDFS-16979:
---
hadoop-yetus commented on PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#issuecomment-1537110397
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 21s | | trunk passed |
| +1 :green_heart: | compile | 17m 15s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 15m 41s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 4m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 4s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 28s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 7m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 30s | | the patch passed |
| +1 :green_heart: | compile | 16m 27s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 16m 27s | | the patch passed |
| +1 :green_heart: | compile | 15m 45s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 15m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 51s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/12/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 113 unchanged - 0 fixed = 114 total (was
113) |
| +1 :green_heart: | mvnsite | 3m 41s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 56s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 26s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 7m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 4s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 14s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 223m 55s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/12/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 21m 37s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 2s | | The patch does not
generate ASF License warnings. |
| | | 468m 46s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/12/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 65942375dada 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3
19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk /
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17719381#comment-17719381
]
ASF GitHub Bot commented on HDFS-16979:
---
hadoop-yetus commented on PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#issuecomment-1534947516
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 0s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 2s | | trunk passed |
| +1 :green_heart: | compile | 17m 26s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 15m 34s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 4m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 2s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 26s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 7m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 31s | | the patch passed |
| +1 :green_heart: | compile | 16m 33s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 16m 33s | | the patch passed |
| +1 :green_heart: | compile | 15m 37s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 15m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 46s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 41s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 56s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 27s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 7m 41s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 56s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 29s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 229m 50s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/11/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 21m 56s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/11/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 3s | | The patch does not
generate ASF License warnings. |
| | | 475m 41s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| | hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/11/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux def343e76b56 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3
19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
|
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17719315#comment-17719315
]
ASF GitHub Bot commented on HDFS-16979:
---
ayushtkn commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1184989929
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestNamenodeAuditlogWithDFSRouterPort.java:
##
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.server.federation.MiniRouterDFSCluster;
+import org.apache.hadoop.hdfs.server.federation.RouterConfigBuilder;
+import org.apache.hadoop.hdfs.server.namenode.FSNamesystem;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.regex.Pattern;
+
+import static
org.apache.hadoop.fs.CommonConfigurationKeysPublic.HADOOP_CALLER_CONTEXT_ENABLED_KEY;
+import static
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_AUDIT_LOG_WITH_REMOTE_PORT_KEY;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Test namenode auditlog log the dfsrouterPort info when the request is form
dfsrouter.
+ */
+public class TestNamenodeAuditlogWithDFSRouterPort {
+ private static final Logger LOG = LoggerFactory.getLogger(
+ TestNamenodeAuditlogWithDFSRouterPort.class);
+ private static final Pattern AUDIT_WITH_PORT_PATTERN = Pattern.compile(
Review Comment:
TestRouterRpc is in hadoop-rbf only, why you need to add hdfs-rbf jar
hadoop-hdfs?
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17719304#comment-17719304
]
ASF GitHub Bot commented on HDFS-16979:
---
LiuGuH commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1184952734
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestNamenodeAuditlogWithDFSRouterPort.java:
##
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.server.federation.MiniRouterDFSCluster;
+import org.apache.hadoop.hdfs.server.federation.RouterConfigBuilder;
+import org.apache.hadoop.hdfs.server.namenode.FSNamesystem;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.regex.Pattern;
+
+import static
org.apache.hadoop.fs.CommonConfigurationKeysPublic.HADOOP_CALLER_CONTEXT_ENABLED_KEY;
+import static
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_AUDIT_LOG_WITH_REMOTE_PORT_KEY;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Test namenode auditlog log the dfsrouterPort info when the request is form
dfsrouter.
+ */
+public class TestNamenodeAuditlogWithDFSRouterPort {
+ private static final Logger LOG = LoggerFactory.getLogger(
+ TestNamenodeAuditlogWithDFSRouterPort.class);
+ private static final Pattern AUDIT_WITH_PORT_PATTERN = Pattern.compile(
Review Comment:
In my environment, there is a maven conflict if I add dependence
hadoop-hdfs-rbf test jar in hadoop-hdfs . The error detail is { java: Modules
hadoop-yarn-server-tests and hadoop-yarn-server-resourcemanager must have the
same 'additional command line parameters' specified because of cyclic
dependencies between them } . I confused
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17719266#comment-17719266
]
ASF GitHub Bot commented on HDFS-16979:
---
hadoop-yetus commented on PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#issuecomment-1534552305
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 3s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 49s | | trunk passed |
| +1 :green_heart: | compile | 17m 17s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 15m 40s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 4m 4s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 4s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 23s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 7m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 26s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 33s | | the patch passed |
| +1 :green_heart: | compile | 16m 33s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 16m 33s | | the patch passed |
| +1 :green_heart: | compile | 15m 40s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 15m 40s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 54s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 41s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 56s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 26s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 7m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 25s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 20s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 227m 52s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/10/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 22m 24s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/10/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 1m 2s | | The patch does not
generate ASF License warnings. |
| | | 473m 21s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestRollingUpgrade |
| |
hadoop.hdfs.server.federation.router.TestRouterRPCMultipleDestinationMountTableResolver
|
| | hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 48816cbfe49a 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17719255#comment-17719255
]
ASF GitHub Bot commented on HDFS-16979:
---
ayushtkn commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1184829824
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallerContext.java:
##
@@ -50,7 +50,8 @@ public final class CallerContext {
public static final String CLIENT_ID_STR = "clientId";
public static final String CLIENT_CALL_ID_STR = "clientCallId";
public static final String REAL_USER_STR = "realUser";
-
+ public static final String IS_DFSROUTER = "isDfsRouter";
+ public static final String DFSROUTER_PORT_STR = "dfsRouterPort";
Review Comment:
There is no point having dfsRouterPort or is isDfsRouter, the code is very
generic and adding Router specific code should be the last thing to do in the
Namenode, Namenode doesn't care whether router is proxying or any xyz client
or service is proxying.
Change it to be generic Proxy Client port nothing exactly router specific in
the Namenode
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17719192#comment-17719192
]
ASF GitHub Bot commented on HDFS-16979:
---
LiuGuH commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1184664573
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -453,15 +453,15 @@ private void logAuditEvent(boolean succeeded,
private void appendClientPortToCallerContextIfAbsent() {
final CallerContext ctx = CallerContext.getCurrent();
-if (isClientPortInfoAbsent(ctx)) {
- String origContext = ctx == null ? null : ctx.getContext();
- byte[] origSignature = ctx == null ? null : ctx.getSignature();
- CallerContext.setCurrent(
- new CallerContext.Builder(origContext, contextFieldSeparator)
- .append(CallerContext.CLIENT_PORT_STR,
String.valueOf(Server.getRemotePort()))
- .setSignature(origSignature)
- .build());
-}
+String origContext = ctx == null ? null : ctx.getContext();
+byte[] origSignature = ctx == null ? null : ctx.getSignature();
+String clientPort = isClientPortInfoAbsent(ctx) ?
CallerContext.CLIENT_PORT_STR :
Review Comment:
I add via isdfsrouter now.
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715114#comment-17715114
]
ASF GitHub Bot commented on HDFS-16979:
---
simbadzina commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1174022112
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestNamenodeAuditlogWithDFSRouterPort.java:
##
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.server.federation.MiniRouterDFSCluster;
+import org.apache.hadoop.hdfs.server.federation.RouterConfigBuilder;
+import org.apache.hadoop.hdfs.server.namenode.FSNamesystem;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.regex.Pattern;
+
+import static
org.apache.hadoop.fs.CommonConfigurationKeysPublic.HADOOP_CALLER_CONTEXT_ENABLED_KEY;
+import static
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_AUDIT_LOG_WITH_REMOTE_PORT_KEY;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Test namenode auditlog log the dfsrouterPort info when the request is form
dfsrouter.
+ */
+public class TestNamenodeAuditlogWithDFSRouterPort {
+ private static final Logger LOG = LoggerFactory.getLogger(
Review Comment:
Logger is not used.
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714314#comment-17714314
]
ASF GitHub Bot commented on HDFS-16979:
---
hadoop-yetus commented on PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#issuecomment-1515408175
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 18m 30s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 10s | | trunk passed |
| +1 :green_heart: | compile | 25m 18s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 21m 51s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 4m 3s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 11s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 2s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 29s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 7m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 8s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 54s | | the patch passed |
| +1 :green_heart: | compile | 24m 29s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 24m 29s | | the patch passed |
| +1 :green_heart: | compile | 21m 44s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 21m 44s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 51s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/9/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 113 unchanged - 0 fixed = 114 total (was
113) |
| +1 :green_heart: | mvnsite | 4m 7s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 57s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 22s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 8m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 21s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 223m 35s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/9/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 21m 57s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 3s | | The patch does not
generate ASF License warnings. |
| | | 506m 54s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 1c7bde7ff63f 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3
19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b33cc54c1ee1a0fe8270440386260887bf88d31a |
|
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714313#comment-17714313
]
ASF GitHub Bot commented on HDFS-16979:
---
virajjasani commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1171870853
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallerContext.java:
##
@@ -47,6 +47,7 @@ public final class CallerContext {
// field names
public static final String CLIENT_IP_STR = "clientIp";
public static final String CLIENT_PORT_STR = "clientPort";
+ public static final String DFSROUTER_PORT_STR = "dfsrouterPort";
Review Comment:
@LiuGuH if this comment is addressed, then also things look good IMO.
The point is both `dfsrouterPort` and using `isClientPortInfoAbsent()` to
set it in the audit logs, together seems fragile because anyone in future
passes clientPort in the context and will get dfsrouterPort printed, which is
not correct for non-router usecases.
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714311#comment-17714311
]
ASF GitHub Bot commented on HDFS-16979:
---
ayushtkn commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1171843571
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallerContext.java:
##
@@ -47,6 +47,7 @@ public final class CallerContext {
// field names
public static final String CLIENT_IP_STR = "clientIp";
public static final String CLIENT_PORT_STR = "clientPort";
+ public static final String DFSROUTER_PORT_STR = "dfsrouterPort";
Review Comment:
As of now it is only router that proxies request and in "Hadoop" code,
anyone can pass the CLIENT_IP_PORT, lets keep the name generic, something
like``PROXY_USER_PORT``, maybe? or something better
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestNamenodeAuditlogWithDFSRouterPort.java:
##
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.server.federation.MiniRouterDFSCluster;
+import org.apache.hadoop.hdfs.server.federation.RouterConfigBuilder;
+import org.apache.hadoop.hdfs.server.namenode.FSNamesystem;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.regex.Pattern;
+
+import static
org.apache.hadoop.fs.CommonConfigurationKeysPublic.HADOOP_CALLER_CONTEXT_ENABLED_KEY;
+import static
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_AUDIT_LOG_WITH_REMOTE_PORT_KEY;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Test namenode auditlog log the dfsrouterPort info when the request is form
dfsrouter.
+ */
+public class TestNamenodeAuditlogWithDFSRouterPort {
+ private static final Logger LOG = LoggerFactory.getLogger(
+ TestNamenodeAuditlogWithDFSRouterPort.class);
+ private static final Pattern AUDIT_WITH_PORT_PATTERN = Pattern.compile(
Review Comment:
Do you need a new class for this test? No, right?
Explore adding the ``TestRouterRpc``, can derive pointers from
``testRealUserPropagationInCallerContext``
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -453,15 +453,16 @@ private void logAuditEvent(boolean succeeded,
private void appendClientPortToCallerContextIfAbsent() {
final CallerContext ctx = CallerContext.getCurrent();
-if (isClientPortInfoAbsent(ctx)) {
- String origContext = ctx == null ? null : ctx.getContext();
- byte[] origSignature = ctx == null ? null : ctx.getSignature();
- CallerContext.setCurrent(
- new CallerContext.Builder(origContext, contextFieldSeparator)
- .append(CallerContext.CLIENT_PORT_STR,
String.valueOf(Server.getRemotePort()))
- .setSignature(origSignature)
- .build());
-}
+String origContext = ctx == null ? null : ctx.getContext();
+byte[] origSignature = ctx == null ? null : ctx.getSignature();
+//When the request comes from the dfsrouter, origContext must contains
CLIENT_PORT_STR. See RouterRpcClient.addClientInfoToCallerContext
+String clientPort = isClientPortInfoAbsent(ctx) ?
CallerContext.CLIENT_PORT_STR :
Review Comment:
I don't find this a problem, Even if someone else sends Client port, we are
just logging his actual port in the audit log,
Though I don't like the name of the config mentioning dfsrouter, let it be
any client who sends you the client port, if someone sends you a port via
client port, we just added the actual client port.
At worst if someone finds serious objections on having this in, We can
consider doing this only if the guy sending the request is configured under
```dfs.namenode.ip-proxy-users```
> RBF: Add dfsrouter port in
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714275#comment-17714275
]
ASF GitHub Bot commented on HDFS-16979:
---
virajjasani commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1171753252
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -453,15 +453,15 @@ private void logAuditEvent(boolean succeeded,
private void appendClientPortToCallerContextIfAbsent() {
final CallerContext ctx = CallerContext.getCurrent();
-if (isClientPortInfoAbsent(ctx)) {
- String origContext = ctx == null ? null : ctx.getContext();
- byte[] origSignature = ctx == null ? null : ctx.getSignature();
- CallerContext.setCurrent(
- new CallerContext.Builder(origContext, contextFieldSeparator)
- .append(CallerContext.CLIENT_PORT_STR,
String.valueOf(Server.getRemotePort()))
- .setSignature(origSignature)
- .build());
-}
+String origContext = ctx == null ? null : ctx.getContext();
+byte[] origSignature = ctx == null ? null : ctx.getSignature();
+String clientPort = isClientPortInfoAbsent(ctx) ?
CallerContext.CLIENT_PORT_STR :
Review Comment:
> Are we even sure that if it is not a client, it is a router?
This is exactly why I was suggesting to provide better protection for this
very condition
https://github.com/apache/hadoop/pull/5552#issuecomment-1507395844
Today, we know that if it is not a client, it is a router. But this is so
_fragile_ and can be easily broken if someone other than router also starts
using `clientPort` in the context.
The better way is to have a check that only router would pass, something
similar to the comment.
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714065#comment-17714065
]
ASF GitHub Bot commented on HDFS-16979:
---
hadoop-yetus commented on PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#issuecomment-1514599015
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 28s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 3s | | trunk passed |
| +1 :green_heart: | compile | 25m 6s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 21m 37s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 3m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 6s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 2s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 27s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 7m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 54s | | the patch passed |
| +1 :green_heart: | compile | 24m 28s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 24m 28s | | the patch passed |
| +1 :green_heart: | compile | 22m 10s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 22m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 53s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/8/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 113 unchanged - 0 fixed = 114 total (was
113) |
| +1 :green_heart: | mvnsite | 4m 2s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 55s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 29s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 8m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 34s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 232m 41s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/8/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 21m 44s | | hadoop-hdfs-rbf in the patch
passed. |
| -1 :x: | asflicense | 1m 2s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/8/artifact/out/results-asflicense.txt)
| The patch generated 1 ASF License warnings. |
| | | 513m 10s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 917fafb546d8 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3
19:10:13 UTC 2023
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714045#comment-17714045
]
ASF GitHub Bot commented on HDFS-16979:
---
hadoop-yetus commented on PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#issuecomment-1514511170
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 17m 4s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 28s | | trunk passed |
| +1 :green_heart: | compile | 25m 14s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 21m 46s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 3m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 8s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 2s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 27s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 7m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 58s | | the patch passed |
| +1 :green_heart: | compile | 24m 28s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 24m 28s | | the patch passed |
| +1 :green_heart: | compile | 21m 43s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 21m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 50s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 9s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 54s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 24s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 8m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 30s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 227m 20s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 22m 2s | | hadoop-hdfs-rbf in the patch
passed. |
| -1 :x: | asflicense | 1m 3s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/7/artifact/out/results-asflicense.txt)
| The patch generated 1 ASF License warnings. |
| | | 509m 31s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 28222c61ea26 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3
19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 01e62b7aba24a875fe0377fdf82086d705263dfb |
| Default Java | Private
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713862#comment-17713862
]
ASF GitHub Bot commented on HDFS-16979:
---
LiuGuH commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1170763729
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -453,15 +453,15 @@ private void logAuditEvent(boolean succeeded,
private void appendClientPortToCallerContextIfAbsent() {
final CallerContext ctx = CallerContext.getCurrent();
-if (isClientPortInfoAbsent(ctx)) {
- String origContext = ctx == null ? null : ctx.getContext();
- byte[] origSignature = ctx == null ? null : ctx.getSignature();
- CallerContext.setCurrent(
- new CallerContext.Builder(origContext, contextFieldSeparator)
- .append(CallerContext.CLIENT_PORT_STR,
String.valueOf(Server.getRemotePort()))
- .setSignature(origSignature)
- .build());
-}
+String origContext = ctx == null ? null : ctx.getContext();
+byte[] origSignature = ctx == null ? null : ctx.getSignature();
+String clientPort = isClientPortInfoAbsent(ctx) ?
CallerContext.CLIENT_PORT_STR :
Review Comment:
Yes，it is. Because when is from dfsrouter ,the callcontext will add
clientPort . The code is in RouterRpcClient.addClientInfoToCallerContext().
In a word， when is comes form dfsrouter , the callcontext has clientPort, or it
is not contains clientPort，so we can use isClientPortInfoAbsent() to
distinguish sources
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713852#comment-17713852
]
ASF GitHub Bot commented on HDFS-16979:
---
goiri commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1170739964
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -453,15 +453,15 @@ private void logAuditEvent(boolean succeeded,
private void appendClientPortToCallerContextIfAbsent() {
final CallerContext ctx = CallerContext.getCurrent();
-if (isClientPortInfoAbsent(ctx)) {
- String origContext = ctx == null ? null : ctx.getContext();
- byte[] origSignature = ctx == null ? null : ctx.getSignature();
- CallerContext.setCurrent(
- new CallerContext.Builder(origContext, contextFieldSeparator)
- .append(CallerContext.CLIENT_PORT_STR,
String.valueOf(Server.getRemotePort()))
- .setSignature(origSignature)
- .build());
-}
+String origContext = ctx == null ? null : ctx.getContext();
+byte[] origSignature = ctx == null ? null : ctx.getSignature();
+String clientPort = isClientPortInfoAbsent(ctx) ?
CallerContext.CLIENT_PORT_STR :
Review Comment:
Are we even sure that if it is not a client, it is a router?
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713850#comment-17713850
]
ASF GitHub Bot commented on HDFS-16979:
---
goiri commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1170739316
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -453,15 +453,15 @@ private void logAuditEvent(boolean succeeded,
private void appendClientPortToCallerContextIfAbsent() {
final CallerContext ctx = CallerContext.getCurrent();
-if (isClientPortInfoAbsent(ctx)) {
- String origContext = ctx == null ? null : ctx.getContext();
- byte[] origSignature = ctx == null ? null : ctx.getSignature();
- CallerContext.setCurrent(
- new CallerContext.Builder(origContext, contextFieldSeparator)
- .append(CallerContext.CLIENT_PORT_STR,
String.valueOf(Server.getRemotePort()))
- .setSignature(origSignature)
- .build());
-}
+String origContext = ctx == null ? null : ctx.getContext();
+byte[] origSignature = ctx == null ? null : ctx.getSignature();
+String clientPort = isClientPortInfoAbsent(ctx) ?
CallerContext.CLIENT_PORT_STR :
Review Comment:
It would be nice to have a comment explaining the logic for when this is a
Router request, etc.
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17712412#comment-17712412
]
ASF GitHub Bot commented on HDFS-16979:
---
hadoop-yetus commented on PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#issuecomment-1508634372
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 10s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 37s | | trunk passed |
| +1 :green_heart: | compile | 26m 12s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 24m 13s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 4m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 2s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 25s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 2m 27s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 6m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 43s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 28s | | the patch passed |
| +1 :green_heart: | compile | 25m 37s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 25m 37s | | the patch passed |
| +1 :green_heart: | compile | 22m 36s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 22m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 56s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 9s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 2m 29s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 6m 45s | | the patch passed |
| +1 :green_heart: | shadedclient | 27m 5s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 48s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 225m 48s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 14s | | The patch does not
generate ASF License warnings. |
| | | 485m 55s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux dd8fe2db34f5 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3
19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a21ff6c64466ae1b77c9bb46e97ad23ecc2a3d60 |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions |
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17712405#comment-17712405
]
ASF GitHub Bot commented on HDFS-16979:
---
hadoop-yetus commented on PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#issuecomment-1508605229
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 19s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 17s | | trunk passed |
| +1 :green_heart: | compile | 27m 24s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 23m 19s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 3m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 28s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 2m 30s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 6m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 2s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 25s | | the patch passed |
| +1 :green_heart: | compile | 25m 29s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 25m 29s | | the patch passed |
| +1 :green_heart: | compile | 23m 1s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 23m 1s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 39s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 18s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 2m 34s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 6m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 50s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 211m 31s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 14s | | The patch does not
generate ASF License warnings. |
| | | 460m 38s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux db6e308b6852 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3
19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3ceae918edf1756058f45f4d273552251588f4f9 |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions |
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17712401#comment-17712401
]
ASF GitHub Bot commented on HDFS-16979:
---
hadoop-yetus commented on PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#issuecomment-1508592775
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 8s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 13s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 5s | | trunk passed |
| +1 :green_heart: | compile | 27m 15s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 23m 29s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 3m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 26s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 2m 42s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 6m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 59s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 23s | | the patch passed |
| +1 :green_heart: | compile | 25m 32s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 25m 32s | | the patch passed |
| +1 :green_heart: | compile | 23m 14s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 23m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 43s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 21s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 2m 39s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 6m 48s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 11s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 40s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 209m 15s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 7s | | The patch does not
generate ASF License warnings. |
| | | 459m 1s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 690210f3c80b 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3
19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3ceae918edf1756058f45f4d273552251588f4f9 |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/5/testReport/ |
| Max. process+thread count | 3708 (vs. ulimit of 5500) |
| modules | C:
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17712310#comment-17712310
]
ASF GitHub Bot commented on HDFS-16979:
---
hadoop-yetus commented on PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#issuecomment-1508279582
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 10s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 0s | | trunk passed |
| +1 :green_heart: | compile | 25m 12s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 21m 45s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 4m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 18s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 14s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 2m 22s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 6m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 34s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 21s | | the patch passed |
| +1 :green_heart: | compile | 24m 35s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 24m 35s | | the patch passed |
| +1 :green_heart: | compile | 21m 36s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 21m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 49s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 6s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 2m 22s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 6m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 33s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 20s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 222m 48s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 1s | | The patch does not
generate ASF License warnings. |
| | | 472m 12s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 1424aa61836a 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3
19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 08b3decfacd14bba48305c6a1a32c4d6f5a2dd4a |
| Default Java | Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| Multi-JDK versions |
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17712230#comment-17712230
]
ASF GitHub Bot commented on HDFS-16979:
---
hadoop-yetus commented on PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#issuecomment-1508053103
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 8s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 29s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 1m 1s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/4/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 0m 23s |
[/branch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/4/artifact/out/branch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt)
| root in trunk failed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | compile | 0m 49s |
[/branch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/4/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09.txt)
| root in trunk failed with JDK Private
Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09. |
| +1 :green_heart: | checkstyle | 5m 40s | | trunk passed |
| -1 :x: | mvnsite | 0m 28s |
[/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/4/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in trunk failed. |
| -1 :x: | mvnsite | 0m 24s |
[/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/4/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in trunk failed. |
| +1 :green_heart: | javadoc | 2m 30s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 2m 30s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| -1 :x: | spotbugs | 0m 13s |
[/branch-spotbugs-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/4/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common.txt)
| hadoop-common in trunk failed. |
| -1 :x: | spotbugs | 0m 12s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/4/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in trunk failed. |
| -1 :x: | shadedclient | 24m 2s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 23s |
[/patch-mvninstall-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/4/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| -1 :x: | mvninstall | 0m 22s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/4/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | compile | 0m 23s |
[/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/4/artifact/out/patch-compile-root-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt)
| root in the patch failed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | javac | 0m 23s |
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17712204#comment-17712204
]
ASF GitHub Bot commented on HDFS-16979:
---
lgh-cn commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1166325561
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -461,6 +462,14 @@ private void appendClientPortToCallerContextIfAbsent() {
.append(CallerContext.CLIENT_PORT_STR,
String.valueOf(Server.getRemotePort()))
.setSignature(origSignature)
.build());
+} else {
+ String origContext = ctx == null ? null : ctx.getContext();
+ byte[] origSignature = ctx == null ? null : ctx.getSignature();
Review Comment:
Yes, It can. I moved them to a public location
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17712181#comment-17712181
]
ASF GitHub Bot commented on HDFS-16979:
---
virajjasani commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1166242347
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -461,6 +462,14 @@ private void appendClientPortToCallerContextIfAbsent() {
.append(CallerContext.CLIENT_PORT_STR,
String.valueOf(Server.getRemotePort()))
.setSignature(origSignature)
.build());
+} else {
+ String origContext = ctx == null ? null : ctx.getContext();
+ byte[] origSignature = ctx == null ? null : ctx.getSignature();
Review Comment:
Since it is redundant check, we can remove `ctx == null ? null : ` from
both L466-L467 correct?
```
String origContext = ctx.getContext();
byte[] origSignature = ctx.getSignature();
```
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17712180#comment-17712180
]
ASF GitHub Bot commented on HDFS-16979:
---
virajjasani commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1166242347
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -461,6 +462,14 @@ private void appendClientPortToCallerContextIfAbsent() {
.append(CallerContext.CLIENT_PORT_STR,
String.valueOf(Server.getRemotePort()))
.setSignature(origSignature)
.build());
+} else {
+ String origContext = ctx == null ? null : ctx.getContext();
+ byte[] origSignature = ctx == null ? null : ctx.getSignature();
Review Comment:
Since it is redundant check, we can remove `ctx == null ? null : ` from
both L466-L467 correct?
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[ https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17712172#comment-17712172 ] ASF GitHub Bot commented on HDFS-16979: --- lgh-cn commented on code in PR #5552: URL: https://github.com/apache/hadoop/pull/5552#discussion_r1166235559 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java: ## @@ -202,6 +202,7 @@ import org.apache.hadoop.fs.InvalidPathException; import org.apache.hadoop.fs.Options; import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.QuotaUsage; Review Comment: It is unused . Removed. > RBF: Add dfsrouter port in hdfsauditlog > --- > > Key: HDFS-16979 > URL: https://issues.apache.org/jira/browse/HDFS-16979 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: liuguanghua >Priority: Major > Labels: pull-request-available > > When remote client request through dfsrouter to namenode, the hdfsauditlog > record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port. > This patch is done for this scene. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17712171#comment-17712171
]
ASF GitHub Bot commented on HDFS-16979:
---
lgh-cn commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1166234652
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -461,6 +462,14 @@ private void appendClientPortToCallerContextIfAbsent() {
.append(CallerContext.CLIENT_PORT_STR,
String.valueOf(Server.getRemotePort()))
.setSignature(origSignature)
.build());
+} else {
+ String origContext = ctx == null ? null : ctx.getContext();
+ byte[] origSignature = ctx == null ? null : ctx.getSignature();
Review Comment:
if ctx was null , it should not add dfsrouterPort because the rpc call is
directory connect namenode.
When a rpc call is form dfsrouter, it will call
RouterRpcClient.addClientInfoToCallerContext method to add clientIP and
clientPort.
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17712152#comment-17712152
]
ASF GitHub Bot commented on HDFS-16979:
---
lgh-cn commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1166211712
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -461,6 +462,14 @@ private void appendClientPortToCallerContextIfAbsent() {
.append(CallerContext.CLIENT_PORT_STR,
String.valueOf(Server.getRemotePort()))
.setSignature(origSignature)
.build());
+} else {
+ String origContext = ctx == null ? null : ctx.getContext();
+ byte[] origSignature = ctx == null ? null : ctx.getSignature();
+ CallerContext.setCurrent(
+ new CallerContext.Builder(origContext, contextFieldSeparator)
+ .append(CallerContext.DFSROUTER_PORT_STR,
String.valueOf(Server.getRemotePort()))
Review Comment:
I will add. It need some time
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17712066#comment-17712066
]
ASF GitHub Bot commented on HDFS-16979:
---
hadoop-yetus commented on PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#issuecomment-1507583774
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 17s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 51s | | trunk passed |
| +1 :green_heart: | compile | 25m 10s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 21m 40s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | checkstyle | 3m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 18s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 13s | | trunk passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 2m 24s | | trunk passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 6m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 2s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 22s | | the patch passed |
| +1 :green_heart: | compile | 24m 26s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 24m 26s | | the patch passed |
| +1 :green_heart: | compile | 21m 41s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | javac | 21m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 53s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 113 unchanged - 0 fixed = 114 total (was
113) |
| +1 :green_heart: | mvnsite | 3m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 5s | | the patch passed with JDK
Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 2m 20s | | the patch passed with JDK
Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09 |
| +1 :green_heart: | spotbugs | 6m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 37s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 11s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 226m 24s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 2s | | The patch does not
generate ASF License warnings. |
| | | 474m 13s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5552/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5552 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 28422c5e748e 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3
19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17712004#comment-17712004
]
ASF GitHub Bot commented on HDFS-16979:
---
virajjasani commented on PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#issuecomment-1507395844
IMHO we could do something like this: add a new key/value pair in the
CallerContext at RouterRpcClient level, e.g. `append("isRouter", "true")` and
then at audit log level, instead of only relying on `isClientPortInfoAbsent()`,
we can also check if the context has `isRouter` key present with value `true`.
If so, then add `dfsrouterPort` with remote port (same logic as `else` part in
this PR).
Thoughts?
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17712001#comment-17712001
]
ASF GitHub Bot commented on HDFS-16979:
---
virajjasani commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1165847902
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -461,6 +462,14 @@ private void appendClientPortToCallerContextIfAbsent() {
.append(CallerContext.CLIENT_PORT_STR,
String.valueOf(Server.getRemotePort()))
.setSignature(origSignature)
.build());
+} else {
+ String origContext = ctx == null ? null : ctx.getContext();
+ byte[] origSignature = ctx == null ? null : ctx.getSignature();
Review Comment:
If `ctx` was null, we wouldn't be here because `isClientPortInfoAbsent(ctx)`
would have returned true already.
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711964#comment-17711964
]
ASF GitHub Bot commented on HDFS-16979:
---
goiri commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1165751071
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -461,6 +462,14 @@ private void appendClientPortToCallerContextIfAbsent() {
.append(CallerContext.CLIENT_PORT_STR,
String.valueOf(Server.getRemotePort()))
.setSignature(origSignature)
.build());
+} else {
+ String origContext = ctx == null ? null : ctx.getContext();
+ byte[] origSignature = ctx == null ? null : ctx.getSignature();
+ CallerContext.setCurrent(
+ new CallerContext.Builder(origContext, contextFieldSeparator)
Review Comment:
Should we be adding this to the fsnamesystem?
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
> Labels: pull-request-available
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16979) RBF: Add dfsrouter port in hdfsauditlog
[
https://issues.apache.org/jira/browse/HDFS-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711962#comment-17711962
]
ASF GitHub Bot commented on HDFS-16979:
---
goiri commented on code in PR #5552:
URL: https://github.com/apache/hadoop/pull/5552#discussion_r1165749819
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -202,6 +202,7 @@
import org.apache.hadoop.fs.InvalidPathException;
import org.apache.hadoop.fs.Options;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.QuotaUsage;
Review Comment:
Why?
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -461,6 +462,14 @@ private void appendClientPortToCallerContextIfAbsent() {
.append(CallerContext.CLIENT_PORT_STR,
String.valueOf(Server.getRemotePort()))
.setSignature(origSignature)
.build());
+} else {
+ String origContext = ctx == null ? null : ctx.getContext();
+ byte[] origSignature = ctx == null ? null : ctx.getSignature();
+ CallerContext.setCurrent(
+ new CallerContext.Builder(origContext, contextFieldSeparator)
+ .append(CallerContext.DFSROUTER_PORT_STR,
String.valueOf(Server.getRemotePort()))
Review Comment:
Can we add a test?
> RBF: Add dfsrouter port in hdfsauditlog
> ---
>
> Key: HDFS-16979
> URL: https://issues.apache.org/jira/browse/HDFS-16979
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: liuguanghua
>Priority: Major
>
> When remote client request through dfsrouter to namenode, the hdfsauditlog
> record the remote client ip and port ，dfsrouter IP，but lack of dfsrouter port.
> This patch is done for this scene.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
