[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22624 There's stuff that I need to fix for the recent changes in the kubernetes code; also I'm going to do the work I meant to do for SPARK-25693 here, since it requires as much testing and isn't that much more code. So hang on a bit. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22624 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22624 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97173/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22624 **[Test build #97173 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97173/testReport)** for PR 22624 at commit [`638bb07`](https://github.com/apache/spark/commit/638bb0700d3faa6753a857f70035fce8f95b6a8b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22624 **[Test build #97173 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97173/testReport)** for PR 22624 at commit [`638bb07`](https://github.com/apache/spark/commit/638bb0700d3faa6753a857f70035fce8f95b6a8b). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22624 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22624 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3834/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22624 Another small change to rename AbstractCredentialRenewer -> AbstractCredentialManager. I prefer that name better since I'm adding more than just renewal functionality to that class, but it sort of clashes with the existing classes. I'll file a separate bug to rename those classes instead. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22624 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97164/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22624 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22624 **[Test build #97164 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97164/testReport)** for PR 22624 at commit [`90e878d`](https://github.com/apache/spark/commit/90e878ddfdbe1048f01983bcfa8cecfb526a81dc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22624 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97163/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22624 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22624 **[Test build #97163 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97163/testReport)** for PR 22624 at commit [`ef8628a`](https://github.com/apache/spark/commit/ef8628ac1a3059bb4871ebd8f1daa6deacf5d7f3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22624 **[Test build #97164 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97164/testReport)** for PR 22624 at commit [`90e878d`](https://github.com/apache/spark/commit/90e878ddfdbe1048f01983bcfa8cecfb526a81dc). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22624 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22624 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3827/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22624 **[Test build #97163 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97163/testReport)** for PR 22624 at commit [`ef8628a`](https://github.com/apache/spark/commit/ef8628ac1a3059bb4871ebd8f1daa6deacf5d7f3). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22624 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3826/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22624 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22624 I merged start() with createAndUpdateTokens() in the last commit; the API is still awkward because of YARN's usage, but it simplifies the Mesos code a little bit. It can be easily reverted in any case. Tested on Mesos, testing on YARN at the moment. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22624 Given the lack of reviews let me ping some people: @skonto @jerryshao --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22624 > ...calls fetchDelegationTokens() twice since the tokenRenewalInterval... That is true - the second fetch is only needed for YARN because reasons. But that's also completely unrelated to this change, so please file a separate bug for it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user ifilonenko commented on the issue:
https://github.com/apache/spark/pull/22624
If we are talking about the token renewal functionality, could we possibly
refactor the `HadoopFSDelegationTokenProvider` as well. I found that within the
function `obtainDelegationTokens()`:
This code-block:
```
val fetchCreds = fetchDelegationTokens(getTokenRenewer(hadoopConf),
fsToGetTokens, creds)
// Get the token renewal interval if it is not set. It will only be
called once.
if (tokenRenewalInterval == null) {
tokenRenewalInterval = getTokenRenewalInterval(hadoopConf, sparkConf,
fsToGetTokens)
}
```
calls `fetchDelegationTokens()` twice since the `tokenRenewalInterval` will
always be null upon creation of the `TokenManager` which I think is unnecessary
in the case of Kubernetes (as you are creating 2 DTs when only one is needed.)
Idk if use-case is different in Mesos / Yarn, but could this possibly be
refactored to only call `fetchDelegationTokens()` once upon startup or to have
a param to specify `tokenRenewalInterval`? I could send a follow-up PR if
desired, but idk if this fits better within the scope of this PR.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22624 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96915/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22624 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22624 **[Test build #96915 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96915/testReport)** for PR 22624 at commit [`5b7604e`](https://github.com/apache/spark/commit/5b7604e77122b4c44048b14e8be47d17eeffd866). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22624 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22624 **[Test build #96913 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96913/testReport)** for PR 22624 at commit [`01e3946`](https://github.com/apache/spark/commit/01e39466d622665b63576e90b8fd0da7cae7e43f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22624 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96913/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22624 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3665/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22624 **[Test build #96915 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96915/testReport)** for PR 22624 at commit [`5b7604e`](https://github.com/apache/spark/commit/5b7604e77122b4c44048b14e8be47d17eeffd866). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22624 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22624 I'm trying to get things to work on Mesos in my hacked cluster, might need some small adjustments to the code. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22624 **[Test build #96913 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96913/testReport)** for PR 22624 at commit [`01e3946`](https://github.com/apache/spark/commit/01e39466d622665b63576e90b8fd0da7cae7e43f). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22624 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3664/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22624: [SPARK-23781][CORE] Add base class for token renewal fun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22624 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
