[
https://issues.apache.org/jira/browse/HDDS-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17340569#comment-17340569
]
Pakapoj Tulsuk edited comment on HDDS-5193 at 5/7/21, 5:21 AM:
---------------------------------------------------------------
{{PERMISSION_DENIED}} issue throwing this error only when we auth to ozone
using {{auth:token}} - {{auth:token}} was used on spark executors so it
suppose happen on both *client* and *cluster* mode (however we found it when we
deploy on *cluster* mode)
{code:java}
2021-05-06 12:12:11,945 [OM StateMachine ApplyTransaction Thread - 0] DEBUG
org.apache.hadoop.ozone.om.KeyManagerImpl: user:[email protected]
(auth:SIMPLE) has access rights for key:mykey1/106186534408552448 :false
2021-05-06 12:12:11,945 [OM StateMachine ApplyTransaction Thread - 0] DEBUG
org.apache.hadoop.ozone.om.lock.OzoneManagerLock: Release read BUCKET_LOCK,
lock on resource /vol1/bucket1
2021-05-06 12:12:11,945 [OM StateMachine ApplyTransaction Thread - 0] WARN
org.apache.hadoop.ozone.om.OzoneManager: User [email protected] doesn't have
WRITE permission to access key /vol1/bucket1/mykey1/106186534408552448
2021-05-06 12:12:11,947 [OMDoubleBufferFlushThread] DEBUG
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer: Sync Iteration 3
flushed transactions in this iteration1
2021-05-06 12:12:11,947 [OM StateMachine ApplyTransaction Thread - 0] ERROR
org.apache.hadoop.ozone.om.request.key.OMKeyCommitRequest: Key commit failed.
Volume:vol1, Bucket:bucket1, Key:mykey1.
PERMISSION_DENIED org.apache.hadoop.ozone.om.exceptions.OMException: User
[email protected] doesn't have WRITE permission to access key vol1 bucket1
mykey1/106186534408552448
at
org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1803)
at
org.apache.hadoop.ozone.om.request.OMClientRequest.checkAcls(OMClientRequest.java:207)
at
org.apache.hadoop.ozone.om.request.OMClientRequest.checkAcls(OMClientRequest.java:185)
at
org.apache.hadoop.ozone.om.request.key.OMKeyRequest.checkKeyAcls(OMKeyRequest.java:437)
at
org.apache.hadoop.ozone.om.request.key.OMKeyRequest.checkKeyAclsInOpenKeyTable(OMKeyRequest.java:485)
at
org.apache.hadoop.ozone.om.request.key.OMKeyCommitRequest.validateAndUpdateCache(OMKeyCommitRequest.java:139)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:415)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:240)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-05-06 12:12:11,947 [OMDoubleBufferFlushThread] DEBUG
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine:
ComputeAndUpdateLastAppliedIndex due to SM is (t:82, i:3665)
{code}
last week i have confirm the reproduction by using {{auth:token}} and also
check on the client side, RPC call, which seem to be normal. where the error
will happed when {{commitKey}} is requested to OM. from that point I understand
that it’s the server side who response the *denial*.
so I went to OM log, see above, during the failure it says it failed on
{{[org.apache.hadoop.ozone.om|http://org.apache.hadoop.ozone.om/]}}{{.OzoneManager.checkAcls(OzoneManager.java:1803)}}
I went there and It looks fine then I move on to
{{org.apache.hadoop.ozone.om.KeyManagerImpl}} on {{checkAcls}} since it the one
who pass the value to the function and print out some log… and I got
{code:java}
[TOKEN]
2021-05-06 12:12:11,945 [OM StateMachine ApplyTransaction Thread - 0] DEBUG
org.apache.hadoop.ozone.om.helpers.OzoneAclUtil: Type: user, A Name:
pakapoj_tul, username: [email protected], aclToCheck: WRITE, rights: {7}
[KERBEROS]
2021-05-06 12:13:35,305 [OM StateMachine ApplyTransaction Thread - 0] DEBUG
org.apache.hadoop.ozone.om.helpers.OzoneAclUtil: Type: user, A Name:
[email protected], username: [email protected], aclToCheck: WRITE, rights:
{7}{code}
It looks like when we auth with TOKEN the username resolve into {{pakapoj_tul}}
but KERBEROS is {{[email protected]}}
In my understanding *pakapoj_tul* is username and *[email protected]* is
principle which both is the same identity on DFS but the code
{{OzoneAclUtil.java#L105}} says
{code:java}
if (a.getName().equals(username))
return checkIfAclBitIsSet(aclToCheck, rights);{code}
so it’s not equal, so I workaround the issue by add function
{{normalizedUsername}} to remove realm and apply it before the comparison
was (Author: pakapoj):
*Explanation* {{PERMISSION_DENIED}} issue throwing this error only when we auth
to ozone using {{auth:token}} - {{auth:token}} was used on spark executors so
it suppose happen on both *client* and *cluster* mode (however we found it when
we deploy on *cluster* mode)
2021-05-06 12:12:11,945 [OM StateMachine ApplyTransaction Thread - 0] DEBUG
org.apache.hadoop.ozone.om.KeyManagerImpl: [user:[email protected]]
(auth:SIMPLE) has access rights for key:mykey1/106186534408552448 :false
2021-05-06 12:12:11,945 [OM StateMachine ApplyTransaction Thread - 0] DEBUG
org.apache.hadoop.ozone.om.lock.OzoneManagerLock: Release read BUCKET_LOCK,
lock on resource /vol1/bucket1
2021-05-06 12:12:11,945 [OM StateMachine ApplyTransaction Thread - 0] WARN
org.apache.hadoop.ozone.om.OzoneManager: User [email protected] doesn't have
WRITE permission to access key /vol1/bucket1/mykey1/106186534408552448
2021-05-06 12:12:11,947 [OMDoubleBufferFlushThread] DEBUG
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer: Sync Iteration 3
flushed transactions in this iteration1
2021-05-06 12:12:11,947 [OM StateMachine ApplyTransaction Thread - 0] ERROR
org.apache.hadoop.ozone.om.request.key.OMKeyCommitRequest: Key commit failed.
Volume:vol1, Bucket:bucket1, Key:mykey1.
PERMISSION_DENIED org.apache.hadoop.ozone.om.exceptions.OMException: User
[email protected] doesn't have WRITE permission to access key vol1 bucket1
mykey1/106186534408552448
at
org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1803)
at
org.apache.hadoop.ozone.om.request.OMClientRequest.checkAcls(OMClientRequest.java:207)
at
org.apache.hadoop.ozone.om.request.OMClientRequest.checkAcls(OMClientRequest.java:185)
at
org.apache.hadoop.ozone.om.request.key.OMKeyRequest.checkKeyAcls(OMKeyRequest.java:437)
at
org.apache.hadoop.ozone.om.request.key.OMKeyRequest.checkKeyAclsInOpenKeyTable(OMKeyRequest.java:485)
at
org.apache.hadoop.ozone.om.request.key.OMKeyCommitRequest.validateAndUpdateCache(OMKeyCommitRequest.java:139)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:415)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:240)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-05-06 12:12:11,947 [OMDoubleBufferFlushThread] DEBUG
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine:
ComputeAndUpdateLastAppliedIndex due to SM is (t:82, i:3665)
last week it have confirm the reproduction by using {{auth:token}} and also
check on the client side, RPC call, which seem to be normal. where the error
will happed when {{commitKey}} is requested to OM. from that point I understand
that it’s the server side who response the *denial*.
so I went to OM log, see above, during the failure it says it failed on
{{[org.apache.hadoop.ozone.om|http://org.apache.hadoop.ozone.om/]}}{{.OzoneManager.checkAcls(OzoneManager.java:1803)}}
I went there and It looks fine then I move on to
{{org.apache.hadoop.ozone.om.KeyManagerImpl}} on {{checkAcls}} since it the
one who pass the value to the function and print out some log… and I got
[TOKEN]
2021-05-06 12:12:11,945 [OM StateMachine ApplyTransaction Thread - 0] DEBUG
org.apache.hadoop.ozone.om.helpers.OzoneAclUtil: Type: user, A Name:
pakapoj_tul, username: [email protected], aclToCheck: WRITE, rights:
\{7}[KERBEROS]
2021-05-06 12:13:35,305 [OM StateMachine ApplyTransaction Thread - 0] DEBUG
org.apache.hadoop.ozone.om.helpers.OzoneAclUtil: Type: user, A Name:
[email protected], username: [email protected], aclToCheck: WRITE, rights:
\{7}
It looks like when we auth with TOKEN the username resolve into {{pakapoj_tul}}
but KERBEROS is {{[email protected]}}
In my understanding *pakapoj_tul* is username and *[email protected]* is
principle which both is the same identity on DFS but the code
{{OzoneAclUtil.java#L105}} says
if (a.getName().equals(username))
return checkIfAclBitIsSet(aclToCheck, rights);
so it’s not equal, so I workaround the issue by add function
{{normalizedUsername}} to remove realm and apply it before the comparison
> Permission Deny when using auth:TOKEN
> -------------------------------------
>
> Key: HDDS-5193
> URL: https://issues.apache.org/jira/browse/HDDS-5193
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Affects Versions: 1.1.0
> Environment: A Ozone (version 1.1 build from source) cluster with 3
> master 3 datanode deploy on baremetal(VMs) running CentOS 7
>
> Reporter: Pakapoj Tulsuk
> Priority: Major
> Attachments: ozone-client-kerberos.log, ozone-client-token.log,
> ozone-java-client.java
>
>
> Hi I’m got stuck on the permission issue where I gonna write the data, a text
> file to a ozone path {{/vol1/bucket1/mykey}} * with {{auth:KERBEROS}} It be
> able to complete the task
> {code:java}
> 2021-04-29 11:49:01,145 Socket Reader #1 for port 9862 INFO
> SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
> [email protected] (auth:KERBEROS) from ip.ip.ip.ip:40294 * with auth:TOKEN
> It got stuck on this error despite the given permission to /vol1 /bucket1
> see below
> 2021-04-29 11:49:08,327 Socket Reader #1 for port 9862 INFO
> SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
> [email protected] (auth:TOKEN) from ip.ip.ip.ip:40412
> 2021-04-29 11:49:12,228 Socket Reader #1 for port 9862 INFO
> SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
> [email protected] (auth:TOKEN) from ip.ip.ip.ip:35266
> 2021-04-29 11:49:14,671 [OM StateMachine ApplyTransaction Thread - 0] WARN
> org.apache.hadoop.ozone.om.OzoneManager: User [email protected] doesn't
> have WRITE permission to access key
> /vol1/bucket1/mykey/_temporary/0/_temporary/attempt_202104290449105826106778232640855_0000_m_000000_0/part-00000-9f9c4fcc-5e96-43ee-b53e-913a06729109-c000.txt/106146807974133768
> 2021-04-29 11:49:14,672 [OM StateMachine ApplyTransaction Thread - 0] ERROR
> org.apache.hadoop.ozone.om.request.key.OMKeyCommitRequest: Key commit failed.
> Volume:vol1, Bucket:bucket1,
> Key:mykey/_temporary/0/_temporary/attempt_202104290449105826106778232640855_0000_m_000000_0/part-00000-9f9c4fcc-5e96-43ee-b53e-913a06729109-c000.txt.
> PERMISSION_DENIED org.apache.hadoop.ozone.om.exceptions.OMException: User
> [email protected] doesn't have WRITE permission to access key vol1 bucket1
> mykey/_temporary/0/_temporary/attempt_202104290449105826106778232640855_0000_m_000000_0/part-00000-9f9c4fcc-5e96-43ee-b53e-913a06729109-c000.txt/106146807974133768
> at org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1803)
> at
> org.apache.hadoop.ozone.om.request.OMClientRequest.checkAcls(OMClientRequest.java:207)
> at
> org.apache.hadoop.ozone.om.request.OMClientRequest.checkAcls(OMClientRequest.java:185)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyRequest.checkKeyAcls(OMKeyRequest.java:437)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyRequest.checkKeyAclsInOpenKeyTable(OMKeyRequest.java:485)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyCommitRequest.validateAndUpdateCache(OMKeyCommitRequest.java:139)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:415)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:240)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> *Given Permission*
> {code:java}
> $ ozone sh vol getacl /vol1/
> [ {
> "type" : "USER",
> "name" : "pakapoj_tul",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> }, {
> "type" : "USER",
> "name" : "[email protected]",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> }, {
> "type" : "USER",
> "name" : "[email protected]",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-admin",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-users",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> } ]
> $ ozone sh bucket getacl /vol1/bucket1/
> [ {
> "type" : "USER",
> "name" : "[email protected]",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-admin",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-users",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "USER",
> "name" : "[email protected]",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> }, {
> "type" : "USER",
> "name" : "pakapoj_tul",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> } ]
> $ ozone sh key getacl /vol1/bucket1/mykey/
> [ {
> "type" : "USER",
> "name" : "[email protected]",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-admin",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-users",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "USER",
> "name" : "[email protected]",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> }, {
> "type" : "USER",
> "name" : "pakapoj_tul",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> } ]{code}
>
> The spark code was deployed in Kubernetes in spark cluster mode. Then, the
> error would happed on spark executors side when the do {{commitKey}} with
> {{auth:TOKEN}} , BTW the spark driver was using {{auth:KERBEROS}} .
>
> so I reproduce using ozone java client writing to ozone with {{OzoneClient}}
> using # Token by {{export HADOOP_TOKEN_FILE_LOCATION=credential/ozone.token}}
> before running the program
> # Keytab by running {{/usr/bin/kinit -kt credential/pakapoj_tul.keytab
> [email protected]}} before running the program
> the code, output for #1 and #2 (DEBUG) is in attach
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
