Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19103
>Oozie client gets the necessary tokens the application needs before
launching. It passes those tokens along to the oozie launcher job (MR job)
which will then actually call the Spark client to launch the spark app and pass
the tokens along.
>The oozie launcher job cannot get anymore tokens because all it has is
tokens ( you can't get tokens with tokens, you need tgt or keytab).
>The error here is because the launcher job runs the Spark Client to submit
the spark job but the spark client doesn't see that it already has the hdfs
tokens so it tries to get more, which ends with the exception.
So the problem is that Oozie will get tokens for Spark instead of letting
Spark do itself, and in Oozie launcher we should not let Spark `Yarn#client` to
get tokens itself since there might not have tgt available in Oozie launcher.
From my understanding of your issue, this seems like a more general issue
regarding Oozie launcher and Spark token manage stuff. With the patch, looks
like it only address the HDFS issue, how do we handle hive/hbase, looks like
still have issues.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
