[
https://issues.apache.org/jira/browse/HADOOP-19249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-19249.
-------------------------------------
Fix Version/s: 3.5.0
3.4.1
Resolution: Fixed
> Getting NullPointerException when the unauthorised user tries to perform the
> key operation
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-19249
> URL: https://issues.apache.org/jira/browse/HADOOP-19249
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, security
> Reporter: Dhaval Shah
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0, 3.4.1
>
>
> While validating the tomcat 9.x in apache Ranger when user doesn't have
> appropriate permission in Ranger policies we faced the NPE for key operation
> using hadoop cmd.
> *Problem :*
> _Functionally -_ We are facing the NPE while performing key operations from
> hadoop cmd with the user not having permission in policy on cluster with
> tomcat v9.x. However with curl to Ranger KSM Server is working as expected.
> _Technically -_ Getting response message as null on client side in
> hadoop-common at
> [KMSClientProvider.java|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java#L565]
> *E.G.*
> _with Ranger KMS tomcat v9.x_
> {code:java}
> hadoop key list
> The list subcommand displays the keynames contained within
> a particular provider as configured in core-site.xml or
> specified with the -provider argument. -metadata displays
> the metadata. If -strict is supplied, fail immediately if
> the provider requires a password and none is given.
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.hadoop.crypto.key.KeyShell.prettifyException(KeyShell.java:541)
> at
> org.apache.hadoop.crypto.key.KeyShell.printException(KeyShell.java:536)
> at org.apache.hadoop.tools.CommandShell.run(CommandShell.java:79)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
> at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:553) {code}
> _on_ _Ranger KMS_ _tomcat v8.5.x_
> {code:java}
> hadoop key list
> The list subcommand displays the keynames contained within
> a particular provider as configured in core-site.xml or
> specified with the -provider argument. -metadata displays
> the metadata. If -strict is supplied, fail immediately ifthe provider
> requires a password and none is given.
> Executing command failed with the following exception:
> AuthorizationException: User:xyzuser not allowed to do 'GET_KEYS'{code}
> *Debug logs on Ranger KMS Server side*
> 1.) Added logs in
> [KMSExceptionsProvider.java|https://github.com/apache/ranger/blob/master/kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java]
> in method _createResponse()_ and _toResponse()_ where we are generating
> response to send it to client i.e. _hadoop-common_
> Logs are exactly same on both the tomcat scenario. Refer below the added
> logs, detailed logs will be available in ranger kms log file on cluster.
> {code:java}
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ==== Entered into toResponse =========
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ==== exception
> =========org.apache.hadoop.security.authorize.AuthorizationException:
> User:systest not allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ==== exception.getClass() =========class
> org.apache.hadoop.security.authorize.AuthorizationException
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ==== AuthorizationException =========
> 2024-07-25 11:35:51,452 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
> [https-jsse-nio-9494-exec-2]: User syst...@root.comops.site (auth:KERBEROS)
> request GET
> https://ccycloud-1.ss-tomcat-test1.root.comops.site:9494/kms/v1/keys/names
> caused exception.
> org.apache.hadoop.security.authorize.AuthorizationException: User:systest not
> allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ===== Entered into createResponse ======
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ==== status ======= Forbidden
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ======= ex =======
> org.apache.hadoop.security.authorize.AuthorizationException: User:systest not
> allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ======= ex.getStackTrace() =======
> [Ljava.lang.StackTraceElement;@3e75ae9d
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ======= ex.getMessage() ======= User:systest
> not allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,452 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
> [https-jsse-nio-9494-exec-2]: ======= ex.toString() =======
> org.apache.hadoop.security.authorize.AuthorizationException: User:systest not
> allowed to do 'GET_KEYS' {code}
> 2.) Also added logs in
> [KMSExceptionsProvider.java|https://github.com/apache/ranger/blob/master/kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java]
> Adding logs in code base
> {code:java}
> public void setStatus(int sc, String sm) {
> LOG.info("========= setStatus with message============ ");
> statusCode = sc;
> msg = sm;
> LOG.info("========= sc ============ " +sc);
> LOG.info("========= msg ============ " +msg);
> if(sc == 403) {
> LOG.info("===== its 403 ====");
> super.setStatus(sc, sm);
> } else{
> super.setStatus(sc, sm);
> }
> } {code}
> LOGS:
> {code:java}
> 2024-07-25 11:35:51,460 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter:
> [https-jsse-nio-9494-exec-2]: ========= setStatus with message============
> 2024-07-25 11:35:51,460 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter:
> [https-jsse-nio-9494-exec-2]: ========= sc ============ 403
> 2024-07-25 11:35:51,460 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter:
> [https-jsse-nio-9494-exec-2]: ========= msg ============ Forbidden
> 2024-07-25 11:35:51,460 INFO
> org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter:
> [https-jsse-nio-9494-exec-2]: ===== its 403 ==== {code}
> This explains that the KMS server is sending the code and message
> appropriately.
> *Debug logs on Hadoop Common Client side*
> 1.) Added logs in
> [HttpExceptionUtils.java|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java]
> to make sure whether appropriate response is received.
> Logs will be available in ranger kms log file.
> {code:java}
> 2024-07-25 11:35:51,453 INFO org.apache.hadoop.util.HttpExceptionUtils:
> [https-jsse-nio-9494-exec-2]: ====== Entered into
> createJerseyExceptionResponse ====
> 2024-07-25 11:35:51,453 INFO org.apache.hadoop.util.HttpExceptionUtils:
> [https-jsse-nio-9494-exec-2]: ========== ex ========
> org.apache.hadoop.security.authorize.AuthorizationException: User:systest not
> allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils:
> [https-jsse-nio-9494-exec-2]: ========== ex.getMessage ========
> User:systest not allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils:
> [https-jsse-nio-9494-exec-2]: ========== status ======== Forbidden
> 2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils:
> [https-jsse-nio-9494-exec-2]: ========== status.getStatusCode ======== 403
> 2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils:
> [https-jsse-nio-9494-exec-2]: ========== status.getReasonPhrase ========
> Forbidden
> 2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils:
> [https-jsse-nio-9494-exec-2]: ======= response ========
> com.sun.jersey.core.spi.factory.ResponseImpl@5bd8a59b {code}
> 2.) Added logs exactly before NPE occurs i.e.
> [KMSClientProvider.java|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java#L564]
> Adding logs in code base
> LOG.info(" =========== conn ======== " + conn);
> Map<String, List<String>> map = conn.getHeaderFields();
> LOG.info("======= map ======== " + map);for (Map.Entry<String, List<String>>
> entry : map.entrySet()) {
> LOG.info("=============== " + "Key : " + entry.getKey() + " ,Value
> : " + entry.getValue());
> }
> LOG.info(" =========== conn.getResponseMessage ======== " +
> conn.getResponseMessage());
> LOG.info(" =========== conn.getResponseCode ======== " +
> conn.getResponseCode());if ((conn.getResponseCode() ==
> HttpURLConnection.HTTP_FORBIDDEN
> && (conn.getResponseMessage().equals(ANONYMOUS_REQUESTS_DISALLOWED) ||
> conn.getResponseMessage().contains(INVALID_SIGNATURE)))
> || conn.getResponseCode() == HttpURLConnection.HTTP_UNAUTHORIZED) {
> LOGS: This logs gets printed on terminal where we execute hadoop cmd .
> _with Ranger KMS tomcat v9.x_
> {code:java}
> hadoop key list
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: ======== Entered into call
> ========
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =========== conn ========
> sun.net.www.protocol.https.DelegateHttpsURLConnection:https://ccycloud-1.ss-tomcat-test1.root.comops.site:9494/kms/v1/keys/names
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: ======= map ========
> {Keep-Alive=[timeout=60], null=[HTTP/1.1 403],
> Strict-Transport-Security=[max-age=31536000; includeSubDomains; preload],
> Server=[Apache Ranger], Connection=[keep-alive], Content-Length=[220],
> Date=[Thu, 25 Jul 2024 11:38:15 GMT], Content-Type=[application/json]}
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key :
> Keep-Alive ,Value : [timeout=60]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : null
> ,Value : [HTTP/1.1 403]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key :
> Strict-Transport-Security ,Value : [max-age=31536000; includeSubDomains;
> preload]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Server
> ,Value : [Apache Ranger]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key :
> Connection ,Value : [keep-alive]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key :
> Content-Length ,Value : [220]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Date
> ,Value : [Thu, 25 Jul 2024 11:38:15 GMT]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key :
> Content-Type ,Value : [application/json]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: ===========
> conn.getResponseMessage ======== null
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: ===========
> conn.getResponseCode ======== 403
> list [-provider <provider>] [-strict] [-metadata] [-help]:
> The list subcommand displays the keynames contained within
> a particular provider as configured in core-site.xml or
> specified with the -provider argument. -metadata displays
> the metadata. If -strict is supplied, fail immediately if
> the provider requires a password and none is given.
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.hadoop.crypto.key.KeyShell.prettifyException(KeyShell.java:541)
> at
> org.apache.hadoop.crypto.key.KeyShell.printException(KeyShell.java:536)
> at org.apache.hadoop.tools.CommandShell.run(CommandShell.java:79)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
> at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:553) {code}
> _with Ranger KMS tomcat v8.5.x_
> hadoop key list
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: ======== Entered into call
> ========
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =========== conn ========
> sun.net.www.protocol.https.DelegateHttpsURLConnection:https://ccycloud-1.ds-tomcat-test1.root.comops.site:9494/kms/v1/keys/names24/07/25
> 11:02:25 INFO kms.KMSClientProvider: ======= map ========
> {Keep-Alive=[timeout=60], null=[HTTP/1.1 403 Forbidden],
> Strict-Transport-Security=[max-age=31536000; includeSubDomains; preload],
> Server=[Apache Ranger], Connection=[keep-alive], Content-Length=[220],
> Date=[Thu, 25 Jul 2024 11:02:25 GMT], Content-Type=[application/json]}
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key :
> Keep-Alive ,Value : [timeout=60]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : null
> ,Value : [HTTP/1.1 403 Forbidden]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key :
> Strict-Transport-Security ,Value : [max-age=31536000; includeSubDomains;
> preload]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Server
> ,Value : [Apache Ranger]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key :
> Connection ,Value : [keep-alive]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key :
> Content-Length ,Value : [220]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Date
> ,Value : [Thu, 25 Jul 2024 11:02:25 GMT]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key :
> Content-Type ,Value : [application/json]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: ===========
> conn.getResponseMessage ======== Forbidden
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: ===========
> conn.getResponseCode ======== 403
> Cannot list keys for KeyProvider:
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@209da20d
> list [-provider <provider>] [-strict] [-metadata] [-help]:
> The list subcommand displays the keynames contained within
> a particular provider as configured in core-site.xml or
> specified with the -provider argument. -metadata displays
> the metadata. If -strict is supplied, fail immediately ifthe provider
> requires a password and none is given.
> Executing command failed with the following exception:
> AuthorizationException: User:xyzuser not allowed to do 'GET_KEYS'
> Please notice
> _with tomcat v9.x : *Key : null ,Value : [HTTP/1.1 403]*_
> _with. tomcat v8.5.x : *Key : null ,Value : [HTTP/1.1 403 Forbidden]*_
> **
> Message "Forbidden" is not present with tomcat v9.x.
> It seems that tomcat v9.x is not setting the message and hadoop-common is
> trying to get where we are facing NPE.
> Also checked for _*org.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER*_ but
> its not available in tomcat 9.x
> Ref:
> Tomcat Doc for 8.5.x
> [https://tomcat.apache.org/tomcat-8.5-doc/api/org/apache/coyote/Constants.html#USE_CUSTOM_STATUS_MSG_IN_HEADER]
> Tomcat Doc for 9.x
> [https://tomcat.apache.org/tomcat-9.0-doc/api/org/apache/coyote/Constants.html]
> Thanks
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
