[
https://issues.apache.org/jira/browse/HDDS-9162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17754539#comment-17754539
]
Maksim Buslovaev edited comment on HDDS-9162 at 8/15/23 10:22 AM:
------------------------------------------------------------------
Hi [~umamaheswararao] thank you for the reply.
Unfortunaletly, according to this
[https://spark.apache.org/docs/latest/cluster-overview.html#cluster-manager-types]
there is no Spark on bare metal at all.
Spark as a cluster supports delegation tokens in YARN and Kubernetes (see
[https://spark.apache.org/docs/3.2.1/security.html#kerberos] and
[https://spark.apache.org/docs/3.2.1/security.html#secure-interaction-with-kubernetes]).
Thus, Spark relies on delegation tokens to authenticate non-local processes
like Spark workers.
In case of HDFS, Spark on Kubernetes uses Hadoop delegation tokens and it
works. Here you can find logs [https://github.com/apache/ozone/discussions/4996]
But in case of Ozone, Spark doesn't receive delegation tokens and it doesn't
work.
Guys, could you please help me out with the Ozone bug?
was (Author: JIRAUSER301511):
Hi [~umamaheswararao] thank you for the reply.
Unfortunaletly, according to this
[https://spark.apache.org/docs/latest/cluster-overview.html#cluster-manager-types]
there is no Spark on bare metal.
Spark as a cluster supports delegation tokens in YARN and Kubernetes (see
[https://spark.apache.org/docs/3.2.1/security.html#kerberos] and
[https://spark.apache.org/docs/3.2.1/security.html#secure-interaction-with-kubernetes]).
Thus, Spark relies on delegation tokens to authenticate non-local processes
like Spark workers.
In case of HDFS, Spark on Kubernetes uses Hadoop delegation tokens and it
works. Here you can find logs [https://github.com/apache/ozone/discussions/4996]
But in case of Ozone, Spark doesn't receive delegation tokens and it doesn't
work.
Guys, could you please help me out with the Ozone bug?
> Ozone Delegation tokens don't work with Spark
> ---------------------------------------------
>
> Key: HDDS-9162
> URL: https://issues.apache.org/jira/browse/HDDS-9162
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Affects Versions: 1.3.0
> Environment: Ozone 1.3.0
> Spark 3.2.1
> Reporter: Maksim Buslovaev
> Priority: Major
>
> Working on production with Ozone, I have been for Ozone project as a real
> contributor for several months.
> I have developed sort of a test for reproduce the bug about Ozone delegation
> tokens.
> You can find all the details in
> [https://github.com/MerelyOneMax1986/ozone-test.git]
>
> Also, I have discovered that the closed issue
> https://issues.apache.org/jira/browse/HDDS-6926 is not resolved in Ozone
> 1.3.0 distro properly.
> I have found that the fix is missing in the distro but it seems to be in
> source code.
> Linked it in this ticket.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
