Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17335
Thanks @vanzin , I agree with you. The scenario what @subrotosanyal
mentioned is a little bit customized, so this problem might be better to handle
out of Spark
Sure, I will update it.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17335
@jerryshao the PR description seems to be out of sync with the current
code, can you update it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17335
@subrotosanyal
I was able to write some code that should work for your use case even
without the fix for SPARK-15754. I reverted that change and ran the following
code a few times in the
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17335
So I had to dig up some e-mails to refresh my brain about SPARK-15754. It
is not related to YARN-2964 (that one is for things like Oozie, where the same
token is using by multiple YARN apps). It's
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17335
I'm not sure if I understand your scenario correctly. In your case Spark
application is embeded into your own application, your application is still
worked after Spark is stopped. And because
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17335
@subrotosanyal would you please elaborate more about this:
> Resource Manager expires the tokens of an application after a certain
period of time lead to expiration of the token which is
Github user subrotosanyal commented on the issue:
https://github.com/apache/spark/pull/17335
hi @yaooqinn ,
This is a scenario where Spark is embed in client application (spark-client
mode).
In the method `Client#createContainerLaunchContext ()`, the
credentials(delegation
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17335
@subrotosanyal would you please help to describe
https://github.com/apache/spark/pull/13499 in detailï¼Thanks
---
If your project is set up for it, you can reply to this email and have your
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17335
I've no idea about that issue, the description is so vague ("Resource
Manager cancels the Delegation Token after 10 minutes of shutting down the
spark context."). Not pretty sure the scenario
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17335
https://issues.apache.org/jira/browse/SPARK-15754 will this patch cause
this problem?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17335
Broaden this issue a bit. Currently in driver side (client mode), issued
delegation tokens are not added into current ugi, this makes follow-up
hdfs/metastore/hbase communication still use tgt
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17335
Ping @vanzin , mind reviewing again? Thanks a lot.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17335
I have tested this with my kerberized hdfs and it works for me. LGTM,
thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17335
@yaooqinn , You only need one principal (for example principal
"f...@example.com") to get authentication from different services, the
configurations for hive and NN mentioned above is only for
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17335
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74954/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17335
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17335
**[Test build #74954 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74954/testReport)**
for PR 17335 at commit
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17335
with this creds provided by HiveCredentialProvider and configured by
`hive.metastore.kerberos.principal`, do we need to re-login with
`spark.yarn.principal` aiming to connect metastore?
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17335
**[Test build #74954 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74954/testReport)**
for PR 17335 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17335
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74946/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17335
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17335
**[Test build #74946 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74946/testReport)**
for PR 17335 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17335
**[Test build #74946 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74946/testReport)**
for PR 17335 at commit
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17335
@yaooqinn , pushed another way to fix this issue, I think hdfs folder owner
should be the right user (proxy user).
---
If your project is set up for it, you can reply to this email and have your
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17335
Thanks @yaooqinn , that's really an issue here. That was my concern when I
had this fix, since we wrap the whole `SessionState.start` with real user, it
means all the operations inside this
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/17335
The dbs and tbls may be created on hdfs via the real userï¼so that the
proxy user may have no rights to things such as:
```
Error: java.lang.RuntimeException: Cannot create staging
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/17335
Thank you, @jerryshao . I'll test on this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17335
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17335
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74740/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17335
**[Test build #74740 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74740/testReport)**
for PR 17335 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17335
**[Test build #74740 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74740/testReport)**
for PR 17335 at commit
31 matches
Mail list logo
