Dhaval Shah created HADOOP-19249:
------------------------------------
Summary: Getting NullPointerException when the unauthorised user
tries to perform the key operation
Key: HADOOP-19249
URL: https://issues.apache.org/jira/browse/HADOOP-19249
Project: Hadoop Common
Issue Type: Improvement
Components: common
Reporter: Dhaval Shah
While validating the tomcat 9.x in apache Ranger when user doesn't have
appropriate permission in Ranger policies we faced the NPE for key operation
using hadoop cmd.
*Problem :*
_Functionally -_ We are facing the NPE while performing key operations from
hadoop cmd with the user not having permission in policy on cluster with tomcat
v9.x. However with curl to Ranger KSM Server is working as expected.
_Technically -_ Getting response message as null on client side in
hadoop-common at
[KMSClientProvider.java|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java#L565]
*E.G.*
_with Ranger KMS tomcat v9.x_
{code:java}
hadoop key list
The list subcommand displays the keynames contained within
a particular provider as configured in core-site.xml or
specified with the -provider argument. -metadata displays
the metadata. If -strict is supplied, fail immediately if
the provider requires a password and none is given.
Exception in thread "main" java.lang.NullPointerException
at
org.apache.hadoop.crypto.key.KeyShell.prettifyException(KeyShell.java:541)
at
org.apache.hadoop.crypto.key.KeyShell.printException(KeyShell.java:536)
at org.apache.hadoop.tools.CommandShell.run(CommandShell.java:79)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:553) {code}
_on_ _Ranger KMS_ _tomcat v8.5.x_
{code:java}
hadoop key list
The list subcommand displays the keynames contained within
a particular provider as configured in core-site.xml or
specified with the -provider argument. -metadata displays
the metadata. If -strict is supplied, fail immediately ifthe provider requires
a password and none is given.
Executing command failed with the following exception: AuthorizationException:
User:xyzuser not allowed to do 'GET_KEYS'{code}
*Debug logs on Ranger KMS Server side*
1.) Added logs in
[KMSExceptionsProvider.java|https://github.com/apache/ranger/blob/master/kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java]
in method _createResponse()_ and _toResponse()_ where we are generating
response to send it to client i.e. _hadoop-common_
Logs are exactly same on both the tomcat scenario. Refer below the added logs,
detailed logs will be available in ranger kms log file on cluster.
{code:java}
2024-07-25 11:35:51,452 INFO
org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
[https-jsse-nio-9494-exec-2]: ==== Entered into toResponse =========
2024-07-25 11:35:51,452 INFO
org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
[https-jsse-nio-9494-exec-2]: ==== exception
=========org.apache.hadoop.security.authorize.AuthorizationException:
User:systest not allowed to do 'GET_KEYS'
2024-07-25 11:35:51,452 INFO
org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
[https-jsse-nio-9494-exec-2]: ==== exception.getClass() =========class
org.apache.hadoop.security.authorize.AuthorizationException
2024-07-25 11:35:51,452 INFO
org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
[https-jsse-nio-9494-exec-2]: ==== AuthorizationException =========
2024-07-25 11:35:51,452 WARN org.apache.hadoop.crypto.key.kms.server.KMS:
[https-jsse-nio-9494-exec-2]: User [email protected] (auth:KERBEROS)
request GET
https://ccycloud-1.ss-tomcat-test1.root.comops.site:9494/kms/v1/keys/names
caused exception.
org.apache.hadoop.security.authorize.AuthorizationException: User:systest not
allowed to do 'GET_KEYS'
2024-07-25 11:35:51,452 INFO
org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
[https-jsse-nio-9494-exec-2]: ===== Entered into createResponse ======
2024-07-25 11:35:51,452 INFO
org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
[https-jsse-nio-9494-exec-2]: ==== status ======= Forbidden
2024-07-25 11:35:51,452 INFO
org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
[https-jsse-nio-9494-exec-2]: ======= ex =======
org.apache.hadoop.security.authorize.AuthorizationException: User:systest not
allowed to do 'GET_KEYS'
2024-07-25 11:35:51,452 INFO
org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
[https-jsse-nio-9494-exec-2]: ======= ex.getStackTrace() =======
[Ljava.lang.StackTraceElement;@3e75ae9d
2024-07-25 11:35:51,452 INFO
org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
[https-jsse-nio-9494-exec-2]: ======= ex.getMessage() ======= User:systest not
allowed to do 'GET_KEYS'
2024-07-25 11:35:51,452 INFO
org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider:
[https-jsse-nio-9494-exec-2]: ======= ex.toString() =======
org.apache.hadoop.security.authorize.AuthorizationException: User:systest not
allowed to do 'GET_KEYS' {code}
2.) Also added logs in
[KMSExceptionsProvider.java|https://github.com/apache/ranger/blob/master/kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java]
Adding logs in code base
{code:java}
public void setStatus(int sc, String sm) {
LOG.info("========= setStatus with message============ ");
statusCode = sc;
msg = sm;
LOG.info("========= sc ============ " +sc);
LOG.info("========= msg ============ " +msg);
if(sc == 403) {
LOG.info("===== its 403 ====");
super.setStatus(sc, sm);
} else{
super.setStatus(sc, sm);
}
} {code}
LOGS:
{code:java}
2024-07-25 11:35:51,460 INFO
org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter:
[https-jsse-nio-9494-exec-2]: ========= setStatus with message============
2024-07-25 11:35:51,460 INFO
org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter:
[https-jsse-nio-9494-exec-2]: ========= sc ============ 403
2024-07-25 11:35:51,460 INFO
org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter:
[https-jsse-nio-9494-exec-2]: ========= msg ============ Forbidden
2024-07-25 11:35:51,460 INFO
org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter:
[https-jsse-nio-9494-exec-2]: ===== its 403 ==== {code}
This explains that the KMS server is sending the code and message appropriately.
*Debug logs on Hadoop Common Client side*
1.) Added logs in
[HttpExceptionUtils.java|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java]
to make sure whether appropriate response is received.
Logs will be available in ranger kms log file.
{code:java}
2024-07-25 11:35:51,453 INFO org.apache.hadoop.util.HttpExceptionUtils:
[https-jsse-nio-9494-exec-2]: ====== Entered into createJerseyExceptionResponse
====
2024-07-25 11:35:51,453 INFO org.apache.hadoop.util.HttpExceptionUtils:
[https-jsse-nio-9494-exec-2]: ========== ex ========
org.apache.hadoop.security.authorize.AuthorizationException: User:systest not
allowed to do 'GET_KEYS'
2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils:
[https-jsse-nio-9494-exec-2]: ========== ex.getMessage ======== User:systest
not allowed to do 'GET_KEYS'
2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils:
[https-jsse-nio-9494-exec-2]: ========== status ======== Forbidden
2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils:
[https-jsse-nio-9494-exec-2]: ========== status.getStatusCode ======== 403
2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils:
[https-jsse-nio-9494-exec-2]: ========== status.getReasonPhrase ========
Forbidden
2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils:
[https-jsse-nio-9494-exec-2]: ======= response ========
com.sun.jersey.core.spi.factory.ResponseImpl@5bd8a59b {code}
2.) Added logs exactly before NPE occurs i.e.
[KMSClientProvider.java|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java#L564]
Adding logs in code base
LOG.info(" =========== conn ======== " + conn);
Map<String, List<String>> map = conn.getHeaderFields();
LOG.info("======= map ======== " + map);for (Map.Entry<String, List<String>>
entry : map.entrySet()) {
LOG.info("=============== " + "Key : " + entry.getKey() + " ,Value :
" + entry.getValue());
}
LOG.info(" =========== conn.getResponseMessage ======== " +
conn.getResponseMessage());
LOG.info(" =========== conn.getResponseCode ======== " +
conn.getResponseCode());if ((conn.getResponseCode() ==
HttpURLConnection.HTTP_FORBIDDEN
&& (conn.getResponseMessage().equals(ANONYMOUS_REQUESTS_DISALLOWED) ||
conn.getResponseMessage().contains(INVALID_SIGNATURE)))
|| conn.getResponseCode() == HttpURLConnection.HTTP_UNAUTHORIZED) {
LOGS: This logs gets printed on terminal where we execute hadoop cmd .
_with Ranger KMS tomcat v9.x_
{code:java}
hadoop key list
24/07/25 11:38:15 INFO kms.KMSClientProvider: ======== Entered into call
========
24/07/25 11:38:15 INFO kms.KMSClientProvider: =========== conn ========
sun.net.www.protocol.https.DelegateHttpsURLConnection:https://ccycloud-1.ss-tomcat-test1.root.comops.site:9494/kms/v1/keys/names
24/07/25 11:38:15 INFO kms.KMSClientProvider: ======= map ========
{Keep-Alive=[timeout=60], null=[HTTP/1.1 403],
Strict-Transport-Security=[max-age=31536000; includeSubDomains; preload],
Server=[Apache Ranger], Connection=[keep-alive], Content-Length=[220],
Date=[Thu, 25 Jul 2024 11:38:15 GMT], Content-Type=[application/json]}
24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Keep-Alive
,Value : [timeout=60]
24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : null ,Value
: [HTTP/1.1 403]
24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key :
Strict-Transport-Security ,Value : [max-age=31536000; includeSubDomains;
preload]
24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Server
,Value : [Apache Ranger]
24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Connection
,Value : [keep-alive]
24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key :
Content-Length ,Value : [220]
24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Date ,Value
: [Thu, 25 Jul 2024 11:38:15 GMT]
24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key :
Content-Type ,Value : [application/json]
24/07/25 11:38:15 INFO kms.KMSClientProvider: ===========
conn.getResponseMessage ======== null
24/07/25 11:38:15 INFO kms.KMSClientProvider: =========== conn.getResponseCode
======== 403
list [-provider <provider>] [-strict] [-metadata] [-help]:
The list subcommand displays the keynames contained within
a particular provider as configured in core-site.xml or
specified with the -provider argument. -metadata displays
the metadata. If -strict is supplied, fail immediately if
the provider requires a password and none is given.
Exception in thread "main" java.lang.NullPointerException
at
org.apache.hadoop.crypto.key.KeyShell.prettifyException(KeyShell.java:541)
at
org.apache.hadoop.crypto.key.KeyShell.printException(KeyShell.java:536)
at org.apache.hadoop.tools.CommandShell.run(CommandShell.java:79)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:553) {code}
_with Ranger KMS tomcat v8.5.x_
hadoop key list
24/07/25 11:02:25 INFO kms.KMSClientProvider: ======== Entered into call
========
24/07/25 11:02:25 INFO kms.KMSClientProvider: =========== conn ========
sun.net.www.protocol.https.DelegateHttpsURLConnection:https://ccycloud-1.ds-tomcat-test1.root.comops.site:9494/kms/v1/keys/names24/07/25
11:02:25 INFO kms.KMSClientProvider: ======= map ========
{Keep-Alive=[timeout=60], null=[HTTP/1.1 403 Forbidden],
Strict-Transport-Security=[max-age=31536000; includeSubDomains; preload],
Server=[Apache Ranger], Connection=[keep-alive], Content-Length=[220],
Date=[Thu, 25 Jul 2024 11:02:25 GMT], Content-Type=[application/json]}
24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Keep-Alive
,Value : [timeout=60]
24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : null ,Value
: [HTTP/1.1 403 Forbidden]
24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key :
Strict-Transport-Security ,Value : [max-age=31536000; includeSubDomains;
preload]
24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Server
,Value : [Apache Ranger]
24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Connection
,Value : [keep-alive]
24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key :
Content-Length ,Value : [220]
24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Date ,Value
: [Thu, 25 Jul 2024 11:02:25 GMT]
24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key :
Content-Type ,Value : [application/json]
24/07/25 11:02:25 INFO kms.KMSClientProvider: ===========
conn.getResponseMessage ======== Forbidden
24/07/25 11:02:25 INFO kms.KMSClientProvider: =========== conn.getResponseCode
======== 403
Cannot list keys for KeyProvider:
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@209da20d
list [-provider <provider>] [-strict] [-metadata] [-help]:
The list subcommand displays the keynames contained within
a particular provider as configured in core-site.xml or
specified with the -provider argument. -metadata displays
the metadata. If -strict is supplied, fail immediately ifthe provider requires
a password and none is given.
Executing command failed with the following exception: AuthorizationException:
User:xyzuser not allowed to do 'GET_KEYS'
Please notice
_with tomcat v9.x : *Key : null ,Value : [HTTP/1.1 403]*_
_with. tomcat v8.5.x : *Key : null ,Value : [HTTP/1.1 403 Forbidden]*_
**
Message "Forbidden" is not present with tomcat v9.x.
It seems that tomcat v9.x is not setting the message and hadoop-common is
trying to get where we are facing NPE.
Also checked for _*org.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER*_ but its
not available in tomcat 9.x
Ref:
Tomcat Doc for 8.5.x
[https://tomcat.apache.org/tomcat-8.5-doc/api/org/apache/coyote/Constants.html#USE_CUSTOM_STATUS_MSG_IN_HEADER]
Tomcat Doc for 9.x
[https://tomcat.apache.org/tomcat-9.0-doc/api/org/apache/coyote/Constants.html]
Thanks
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
