[
https://issues.apache.org/jira/browse/HADOOP-19249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17929441#comment-17929441
]
ASF GitHub Bot commented on HADOOP-19249:
-----------------------------------------
hadoop-yetus commented on PR #7428:
URL: https://github.com/apache/hadoop/pull/7428#issuecomment-2676718536
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 6m 37s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 31s | | trunk passed |
| +1 :green_heart: | compile | 15m 38s | | trunk passed with JDK
Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 13m 56s | | trunk passed with JDK
Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 4m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 20m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 9m 26s | | trunk passed with JDK
Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 32s | | trunk passed with JDK
Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +0 :ok: | spotbugs | 0m 21s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 63m 51s | | branch has no errors
when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 36s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 32m 15s | | the patch passed |
| +1 :green_heart: | compile | 15m 18s | | the patch passed with JDK
Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 15m 18s | | the patch passed |
| +1 :green_heart: | compile | 13m 45s | | the patch passed with JDK
Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 13m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 46s | | the patch passed |
| +1 :green_heart: | mvnsite | 18m 9s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 9m 24s | | the patch passed with JDK
Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 40s | | the patch passed with JDK
Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +0 :ok: | spotbugs | 0m 20s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 64m 10s | | patch has no errors
when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 394m 15s | | root in the patch passed. |
| +1 :green_heart: | asflicense | 1m 19s | | The patch does not
generate ASF License warnings. |
| | | 750m 25s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7428/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/7428 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
shellcheck shelldocs |
| uname | Linux 11d2e22bb251 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18
17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e1abb77241a083e103178a82d93562d3eadc0ff9 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7428/1/testReport/ |
| Max. process+thread count | 3559 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-hdfs-project/hadoop-hdfs-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common . U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7428/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Getting NullPointerException when the unauthorised user tries to perform the
> key operation
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-19249
> URL: https://issues.apache.org/jira/browse/HADOOP-19249
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, security
> Reporter: Dhaval Shah
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0, 3.4.1
>
>
> While validating the tomcat 9.x in apache Ranger when user doesn't have
> appropriate permission in Ranger policies we faced the NPE for key operation
> using hadoop cmd.
> *Problem :*
> _Functionally -_ We are facing the NPE while performing key operations from
> hadoop cmd with the user not having permission in policy on cluster with
> tomcat v9.x. However with curl to Ranger KSM Server is working as expected.
> _Technically -_ Getting response message as null on client side in
> hadoop-common at
> [KMSClientProvider.java|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java#L565]
> *E.G.*
> _with Ranger KMS tomcat v9.x_
> {code:java}
> hadoop key list
> The list subcommand displays the keynames contained within
> a particular provider as configured in core-site.xml or
> specified with the -provider argument. -metadata displays
> the metadata. If -strict is supplied, fail immediately if
> the provider requires a password and none is given.
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.hadoop.crypto.key.KeyShell.prettifyException(KeyShell.java:541)
> at
> org.apache.hadoop.crypto.key.KeyShell.printException(KeyShell.java:536)
> at org.apache.hadoop.tools.CommandShell.run(CommandShell.java:79)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
> at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:553) {code}
> _on_ _Ranger KMS_ _tomcat v8.5.x_
> {code:java}
> hadoop key list
> The list subcommand displays the keynames contained within
> a particular provider as configured in core-site.xml or
> specified with the -provider argument. -metadata displays
> the metadata. If -strict is supplied, fail immediately ifthe provider
> requires a password and none is given.
> Executing command failed with the following exception:
> AuthorizationException: User:xyzuser not allowed to do 'GET_KEYS'{code}
> *Debug logs on Ranger KMS Server side*
> 1.) Added logs in
> [KMSExceptionsProvider.java|https://github.com/apache/ranger/blob/master/kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java]
> in method _createResponse()_ and _toResponse()_ where we are generating
> response to send it to client i.e. _hadoop-common_
> Logs are exactly same on both the tomcat scenario. Refer below the added
> logs, detailed logs will be available in ranger kms log file on cluster.
> {code:java}
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ==== Entered into toResponse =========
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ==== exception
> =========org.apache.hadoop.security.authorize.AuthorizationException:
> User:systest not allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ==== exception.getClass() =========class
> org.apache.hadoop.security.authorize.AuthorizationException
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ==== AuthorizationException =========
> 2024-07-25 11:35:51,452 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> [https-jsse-nio-9494-exec-2]: User [email protected] (auth:KERBEROS)
> request GET
> https://ccycloud-1.ss-tomcat-test1.root.comops.site:9494/kms/v1/keys/names
> caused exception.
> org.apache.hadoop.security.authorize.AuthorizationException: User:systest not
> allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ===== Entered into createResponse ======
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ==== status ======= Forbidden
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ======= ex =======
> org.apache.hadoop.security.authorize.AuthorizationException: User:systest not
> allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ======= ex.getStackTrace() =======
> [Ljava.lang.StackTraceElement;@3e75ae9d
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ======= ex.getMessage() ======= User:systest
> not allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ======= ex.toString() =======
> org.apache.hadoop.security.authorize.AuthorizationException: User:systest not
> allowed to do 'GET_KEYS' {code}
> 2.) Also added logs in
> [KMSExceptionsProvider.java|https://github.com/apache/ranger/blob/master/kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java]
> Adding logs in code base
> {code:java}
> public void setStatus(int sc, String sm) {
> LOG.info("========= setStatus with message============ ");
> statusCode = sc;
> msg = sm;
> LOG.info("========= sc ============ " +sc);
> LOG.info("========= msg ============ " +msg);
> if(sc == 403) {
> LOG.info("===== its 403 ====");
> super.setStatus(sc, sm);
> } else{
> super.setStatus(sc, sm);
> }
> } {code}
> LOGS:
> {code:java}
> 2024-07-25 11:35:51,460 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter:
> [https-jsse-nio-9494-exec-2]: ========= setStatus with message============
> 2024-07-25 11:35:51,460 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter:
> [https-jsse-nio-9494-exec-2]: ========= sc ============ 403
> 2024-07-25 11:35:51,460 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter:
> [https-jsse-nio-9494-exec-2]: ========= msg ============ Forbidden
> 2024-07-25 11:35:51,460 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter:
> [https-jsse-nio-9494-exec-2]: ===== its 403 ==== {code}
> This explains that the KMS server is sending the code and message
> appropriately.
> *Debug logs on Hadoop Common Client side*
> 1.) Added logs in
> [HttpExceptionUtils.java|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java]
> to make sure whether appropriate response is received.
> Logs will be available in ranger kms log file.
> {code:java}
> 2024-07-25 11:35:51,453 INFO org.apache.hadoop.util.HttpExceptionUtils:
> [https-jsse-nio-9494-exec-2]: ====== Entered into
> createJerseyExceptionResponse ====
> 2024-07-25 11:35:51,453 INFO org.apache.hadoop.util.HttpExceptionUtils:
> [https-jsse-nio-9494-exec-2]: ========== ex ========
> org.apache.hadoop.security.authorize.AuthorizationException: User:systest not
> allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils:
> [https-jsse-nio-9494-exec-2]: ========== ex.getMessage ========
> User:systest not allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils:
> [https-jsse-nio-9494-exec-2]: ========== status ======== Forbidden
> 2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils:
> [https-jsse-nio-9494-exec-2]: ========== status.getStatusCode ======== 403
> 2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils:
> [https-jsse-nio-9494-exec-2]: ========== status.getReasonPhrase ========
> Forbidden
> 2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils:
> [https-jsse-nio-9494-exec-2]: ======= response ========
> com.sun.jersey.core.spi.factory.ResponseImpl@5bd8a59b {code}
> 2.) Added logs exactly before NPE occurs i.e.
> [KMSClientProvider.java|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java#L564]
> Adding logs in code base
> LOG.info(" =========== conn ======== " + conn);
> Map<String, List<String>> map = conn.getHeaderFields();
> LOG.info("======= map ======== " + map);for (Map.Entry<String, List<String>>
> entry : map.entrySet()) {
> LOG.info("=============== " + "Key : " + entry.getKey() + " ,Value
> : " + entry.getValue());
> }
> LOG.info(" =========== conn.getResponseMessage ======== " +
> conn.getResponseMessage());
> LOG.info(" =========== conn.getResponseCode ======== " +
> conn.getResponseCode());if ((conn.getResponseCode() ==
> HttpURLConnection.HTTP_FORBIDDEN
> && (conn.getResponseMessage().equals(ANONYMOUS_REQUESTS_DISALLOWED) ||
> conn.getResponseMessage().contains(INVALID_SIGNATURE)))
> || conn.getResponseCode() == HttpURLConnection.HTTP_UNAUTHORIZED) {
> LOGS: This logs gets printed on terminal where we execute hadoop cmd .
> _with Ranger KMS tomcat v9.x_
> {code:java}
> hadoop key list
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: ======== Entered into call
> ========
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =========== conn ========
> sun.net.www.protocol.https.DelegateHttpsURLConnection:https://ccycloud-1.ss-tomcat-test1.root.comops.site:9494/kms/v1/keys/names
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: ======= map ========
> {Keep-Alive=[timeout=60], null=[HTTP/1.1 403],
> Strict-Transport-Security=[max-age=31536000; includeSubDomains; preload],
> Server=[Apache Ranger], Connection=[keep-alive], Content-Length=[220],
> Date=[Thu, 25 Jul 2024 11:38:15 GMT], Content-Type=[application/json]}
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key :
> Keep-Alive ,Value : [timeout=60]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : null
> ,Value : [HTTP/1.1 403]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key :
> Strict-Transport-Security ,Value : [max-age=31536000; includeSubDomains;
> preload]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Server
> ,Value : [Apache Ranger]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key :
> Connection ,Value : [keep-alive]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key :
> Content-Length ,Value : [220]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Date
> ,Value : [Thu, 25 Jul 2024 11:38:15 GMT]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key :
> Content-Type ,Value : [application/json]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: ===========
> conn.getResponseMessage ======== null
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: ===========
> conn.getResponseCode ======== 403
> list [-provider <provider>] [-strict] [-metadata] [-help]:
> The list subcommand displays the keynames contained within
> a particular provider as configured in core-site.xml or
> specified with the -provider argument. -metadata displays
> the metadata. If -strict is supplied, fail immediately if
> the provider requires a password and none is given.
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.hadoop.crypto.key.KeyShell.prettifyException(KeyShell.java:541)
> at
> org.apache.hadoop.crypto.key.KeyShell.printException(KeyShell.java:536)
> at org.apache.hadoop.tools.CommandShell.run(CommandShell.java:79)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
> at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:553) {code}
> _with Ranger KMS tomcat v8.5.x_
> hadoop key list
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: ======== Entered into call
> ========
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =========== conn ========
> sun.net.www.protocol.https.DelegateHttpsURLConnection:https://ccycloud-1.ds-tomcat-test1.root.comops.site:9494/kms/v1/keys/names24/07/25
> 11:02:25 INFO kms.KMSClientProvider: ======= map ========
> {Keep-Alive=[timeout=60], null=[HTTP/1.1 403 Forbidden],
> Strict-Transport-Security=[max-age=31536000; includeSubDomains; preload],
> Server=[Apache Ranger], Connection=[keep-alive], Content-Length=[220],
> Date=[Thu, 25 Jul 2024 11:02:25 GMT], Content-Type=[application/json]}
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key :
> Keep-Alive ,Value : [timeout=60]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : null
> ,Value : [HTTP/1.1 403 Forbidden]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key :
> Strict-Transport-Security ,Value : [max-age=31536000; includeSubDomains;
> preload]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Server
> ,Value : [Apache Ranger]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key :
> Connection ,Value : [keep-alive]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key :
> Content-Length ,Value : [220]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Date
> ,Value : [Thu, 25 Jul 2024 11:02:25 GMT]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key :
> Content-Type ,Value : [application/json]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: ===========
> conn.getResponseMessage ======== Forbidden
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: ===========
> conn.getResponseCode ======== 403
> Cannot list keys for KeyProvider:
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@209da20d
> list [-provider <provider>] [-strict] [-metadata] [-help]:
> The list subcommand displays the keynames contained within
> a particular provider as configured in core-site.xml or
> specified with the -provider argument. -metadata displays
> the metadata. If -strict is supplied, fail immediately ifthe provider
> requires a password and none is given.
> Executing command failed with the following exception:
> AuthorizationException: User:xyzuser not allowed to do 'GET_KEYS'
> Please notice
> _with tomcat v9.x : *Key : null ,Value : [HTTP/1.1 403]*_
> _with. tomcat v8.5.x : *Key : null ,Value : [HTTP/1.1 403 Forbidden]*_
> **
> Message "Forbidden" is not present with tomcat v9.x.
> It seems that tomcat v9.x is not setting the message and hadoop-common is
> trying to get where we are facing NPE.
> Also checked for _*org.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER*_ but
> its not available in tomcat 9.x
> Ref:
> Tomcat Doc for 8.5.x
> [https://tomcat.apache.org/tomcat-8.5-doc/api/org/apache/coyote/Constants.html#USE_CUSTOM_STATUS_MSG_IN_HEADER]
> Tomcat Doc for 9.x
> [https://tomcat.apache.org/tomcat-9.0-doc/api/org/apache/coyote/Constants.html]
> Thanks
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
