[
https://issues.apache.org/jira/browse/HDDS-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17364111#comment-17364111
]
Pakapoj Tulsuk commented on HDDS-5193:
--------------------------------------
Hi update
I think I found root causes of the event. As I mentioned in the description
that I'm running spark on cluster mode against kerberized ozone with
authorization enable. The application intents to read an orc file from ozone
and write it to ozone.
code.
{code:java}
val data = spark.read.format("orc").load(src)
data.write.format("orc").save(des)
{code}
given
{code:java}
src = "ofs://dev-ozone/vol1/bucket1/sample_seq_data.snappy.orc"
des = "ofs://dev-ozone/vol1/bucket1/mykey"{code}
the ERROR happened on {{save}} function where it invoke {{createFile}} from
{{OzoneBucket}} in this sequence
# From *spark driver* using {{auth:KERBEROS}} so it have
{{[email protected]}} identity and create temp file and dir such as
{{/vol1/bucket1/mykey/_temporary/0/_temporary/attempt_202106151221294802544738954213121_0000_m_000000_0/}}
with acl {{USER:[email protected]:ALL}} and related groups from local user
{{pakapoj_tul}} from client local machine
# From *spark executor* using {{auth:TOKEN}} so it have {{pakapoj_tul}}
identity and create data files in *the temp dir* since the dir have acl
{{USER:[email protected]:ALL}} so {{pakapoj_tul}} is not allowed. the ERROR
is here
# In the end, after all executors finish writing files to temp dir the driver
will move those temp dir to user specified destination, in the case is
{{"ofs://dev-ozone/vol1/bucket1/mykey"}}
IMHO, in kerberized cluster identity {{[email protected] }}and
{{pakapoj_tul}} are equal so I propose to use getShortUsername() instead of
getShortUsername() from UGI interface when assign and/or compare any ACLs. To
leverage {{auth_to_local}} property to translate any identity from
{{auth:KERBEROS}} to plain username which should be consistent with identity
from {{auth:TOKEN}}
> Permission Deny when using auth:TOKEN
> -------------------------------------
>
> Key: HDDS-5193
> URL: https://issues.apache.org/jira/browse/HDDS-5193
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Affects Versions: 1.1.0
> Environment: A Ozone (version 1.1 build from source) cluster with 3
> master 3 datanode deploy on baremetal(VMs) running CentOS 7
>
> Reporter: Pakapoj Tulsuk
> Priority: Critical
> Attachments: ozone-client-kerberos.log, ozone-client-token.log,
> ozone-java-client.java
>
>
> Hi I’m got stuck on the permission issue where I gonna write the data, a text
> file to a ozone path {{/vol1/bucket1/mykey}} * with {{auth:KERBEROS}} It be
> able to complete the task
> {code:java}
> 2021-04-29 11:49:01,145 Socket Reader #1 for port 9862 INFO
> SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
> [email protected] (auth:KERBEROS) from ip.ip.ip.ip:40294 * with auth:TOKEN
> It got stuck on this error despite the given permission to /vol1 /bucket1
> see below
> 2021-04-29 11:49:08,327 Socket Reader #1 for port 9862 INFO
> SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
> [email protected] (auth:TOKEN) from ip.ip.ip.ip:40412
> 2021-04-29 11:49:12,228 Socket Reader #1 for port 9862 INFO
> SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
> [email protected] (auth:TOKEN) from ip.ip.ip.ip:35266
> 2021-04-29 11:49:14,671 [OM StateMachine ApplyTransaction Thread - 0] WARN
> org.apache.hadoop.ozone.om.OzoneManager: User [email protected] doesn't
> have WRITE permission to access key
> /vol1/bucket1/mykey/_temporary/0/_temporary/attempt_202104290449105826106778232640855_0000_m_000000_0/part-00000-9f9c4fcc-5e96-43ee-b53e-913a06729109-c000.txt/106146807974133768
> 2021-04-29 11:49:14,672 [OM StateMachine ApplyTransaction Thread - 0] ERROR
> org.apache.hadoop.ozone.om.request.key.OMKeyCommitRequest: Key commit failed.
> Volume:vol1, Bucket:bucket1,
> Key:mykey/_temporary/0/_temporary/attempt_202104290449105826106778232640855_0000_m_000000_0/part-00000-9f9c4fcc-5e96-43ee-b53e-913a06729109-c000.txt.
> PERMISSION_DENIED org.apache.hadoop.ozone.om.exceptions.OMException: User
> [email protected] doesn't have WRITE permission to access key vol1 bucket1
> mykey/_temporary/0/_temporary/attempt_202104290449105826106778232640855_0000_m_000000_0/part-00000-9f9c4fcc-5e96-43ee-b53e-913a06729109-c000.txt/106146807974133768
> at org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1803)
> at
> org.apache.hadoop.ozone.om.request.OMClientRequest.checkAcls(OMClientRequest.java:207)
> at
> org.apache.hadoop.ozone.om.request.OMClientRequest.checkAcls(OMClientRequest.java:185)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyRequest.checkKeyAcls(OMKeyRequest.java:437)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyRequest.checkKeyAclsInOpenKeyTable(OMKeyRequest.java:485)
> at
> org.apache.hadoop.ozone.om.request.key.OMKeyCommitRequest.validateAndUpdateCache(OMKeyCommitRequest.java:139)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:415)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:240)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> *Given Permission*
> {code:java}
> $ ozone sh vol getacl /vol1/
> [ {
> "type" : "USER",
> "name" : "pakapoj_tul",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> }, {
> "type" : "USER",
> "name" : "[email protected]",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> }, {
> "type" : "USER",
> "name" : "[email protected]",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-admin",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-users",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> } ]
> $ ozone sh bucket getacl /vol1/bucket1/
> [ {
> "type" : "USER",
> "name" : "[email protected]",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-admin",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-users",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "USER",
> "name" : "[email protected]",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> }, {
> "type" : "USER",
> "name" : "pakapoj_tul",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> } ]
> $ ozone sh key getacl /vol1/bucket1/mykey/
> [ {
> "type" : "USER",
> "name" : "[email protected]",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-admin",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-users",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "USER",
> "name" : "[email protected]",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> }, {
> "type" : "USER",
> "name" : "pakapoj_tul",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> } ]{code}
>
> The spark code was deployed in Kubernetes in spark cluster mode. Then, the
> error would happed on spark executors side when the do {{commitKey}} with
> {{auth:TOKEN}} , BTW the spark driver was using {{auth:KERBEROS}} .
>
> so I reproduce using ozone java client writing to ozone with {{OzoneClient}}
> using # Token by {{export HADOOP_TOKEN_FILE_LOCATION=credential/ozone.token}}
> before running the program
> # Keytab by running {{/usr/bin/kinit -kt credential/pakapoj_tul.keytab
> [email protected]}} before running the program
> the code, output for #1 and #2 (DEBUG) is in attach
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
