[jira] [Commented] (SPARK-27997) kubernetes client token expired
[ https://issues.apache.org/jira/browse/SPARK-27997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17395308#comment-17395308 ] Apache Spark commented on SPARK-27997: -- User 'haodemon' has created a pull request for this issue: https://github.com/apache/spark/pull/33675 > kubernetes client token expired > > > Key: SPARK-27997 > URL: https://issues.apache.org/jira/browse/SPARK-27997 > Project: Spark > Issue Type: Improvement > Components: Kubernetes, Spark Core >Affects Versions: 3.1.0 >Reporter: Henry Yu >Priority: Major > > Hi , > when I try to submit spark to k8s in cluster mode, I need an authtoken to > talk with k8s. > unfortunately, many cloud provider provide token and expired with 10-15 mins. > so we need to fresh this token. > client mode is event worse, because scheduler is created on submit process. > Should I also make a pr on this ? I fix it by adding > RotatingOAuthTokenProvider and some configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27997) kubernetes client token expired
[ https://issues.apache.org/jira/browse/SPARK-27997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17395307#comment-17395307 ] Apache Spark commented on SPARK-27997: -- User 'haodemon' has created a pull request for this issue: https://github.com/apache/spark/pull/33675 > kubernetes client token expired > > > Key: SPARK-27997 > URL: https://issues.apache.org/jira/browse/SPARK-27997 > Project: Spark > Issue Type: Improvement > Components: Kubernetes, Spark Core >Affects Versions: 3.1.0 >Reporter: Henry Yu >Priority: Major > > Hi , > when I try to submit spark to k8s in cluster mode, I need an authtoken to > talk with k8s. > unfortunately, many cloud provider provide token and expired with 10-15 mins. > so we need to fresh this token. > client mode is event worse, because scheduler is created on submit process. > Should I also make a pr on this ? I fix it by adding > RotatingOAuthTokenProvider and some configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27997) kubernetes client token expired
[ https://issues.apache.org/jira/browse/SPARK-27997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295825#comment-17295825 ] rameshkrishnan muthusamy commented on SPARK-27997: -- The PR to upgrade the underlying library to handle OIDC based auth is merged. Initial test looks good, I will be able to update the thread once the IT is complete. > kubernetes client token expired > > > Key: SPARK-27997 > URL: https://issues.apache.org/jira/browse/SPARK-27997 > Project: Spark > Issue Type: Improvement > Components: Kubernetes, Spark Core >Affects Versions: 3.1.0 >Reporter: Henry Yu >Priority: Major > > Hi , > when I try to submit spark to k8s in cluster mode, I need an authtoken to > talk with k8s. > unfortunately, many cloud provider provide token and expired with 10-15 mins. > so we need to fresh this token. > client mode is event worse, because scheduler is created on submit process. > Should I also make a pr on this ? I fix it by adding > RotatingOAuthTokenProvider and some configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27997) kubernetes client token expired
[ https://issues.apache.org/jira/browse/SPARK-27997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175605#comment-17175605 ] ramesh krishnan m commented on SPARK-27997: --- I can share the PR branch with a small set of UT for access before end of next week. On Tue, 11 Aug 2020 at 7:49 PM, Janek Bevendorff (Jira) > kubernetes client token expired > > > Key: SPARK-27997 > URL: https://issues.apache.org/jira/browse/SPARK-27997 > Project: Spark > Issue Type: Improvement > Components: Kubernetes, Spark Core >Affects Versions: 3.1.0 >Reporter: Henry Yu >Priority: Major > > Hi , > when I try to submit spark to k8s in cluster mode, I need an authtoken to > talk with k8s. > unfortunately, many cloud provider provide token and expired with 10-15 mins. > so we need to fresh this token. > client mode is event worse, because scheduler is created on submit process. > Should I also make a pr on this ? I fix it by adding > RotatingOAuthTokenProvider and some configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27997) kubernetes client token expired
[ https://issues.apache.org/jira/browse/SPARK-27997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175604#comment-17175604 ] Janek Bevendorff commented on SPARK-27997: -- Cool, thanks. How long will it take approximately until it is available in a pre-release or snapshot build of any sort? > kubernetes client token expired > > > Key: SPARK-27997 > URL: https://issues.apache.org/jira/browse/SPARK-27997 > Project: Spark > Issue Type: Improvement > Components: Kubernetes, Spark Core >Affects Versions: 3.1.0 >Reporter: Henry Yu >Priority: Major > > Hi , > when I try to submit spark to k8s in cluster mode, I need an authtoken to > talk with k8s. > unfortunately, many cloud provider provide token and expired with 10-15 mins. > so we need to fresh this token. > client mode is event worse, because scheduler is created on submit process. > Should I also make a pr on this ? I fix it by adding > RotatingOAuthTokenProvider and some configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27997) kubernetes client token expired
[ https://issues.apache.org/jira/browse/SPARK-27997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175601#comment-17175601 ] rameshkrishnan muthusamy commented on SPARK-27997: -- [~phoerious] I have completed the OIDC integration required, working on the UT for the same. You should have the PR in the coming week. > kubernetes client token expired > > > Key: SPARK-27997 > URL: https://issues.apache.org/jira/browse/SPARK-27997 > Project: Spark > Issue Type: Improvement > Components: Kubernetes, Spark Core >Affects Versions: 3.1.0 >Reporter: Henry Yu >Priority: Major > > Hi , > when I try to submit spark to k8s in cluster mode, I need an authtoken to > talk with k8s. > unfortunately, many cloud provider provide token and expired with 10-15 mins. > so we need to fresh this token. > client mode is event worse, because scheduler is created on submit process. > Should I also make a pr on this ? I fix it by adding > RotatingOAuthTokenProvider and some configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27997) kubernetes client token expired
[ https://issues.apache.org/jira/browse/SPARK-27997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175598#comment-17175598 ] Janek Bevendorff commented on SPARK-27997: -- Is there any progress on this issue? My driver starts throwing token expiration stacktraces a few minutes after submitting a job. The job itself keeps running, but I don't know what will happen when new containers need to be scheduled. The job will probably fail or get stuck at that point. > kubernetes client token expired > > > Key: SPARK-27997 > URL: https://issues.apache.org/jira/browse/SPARK-27997 > Project: Spark > Issue Type: Improvement > Components: Kubernetes, Spark Core >Affects Versions: 3.1.0 >Reporter: Henry Yu >Priority: Major > > Hi , > when I try to submit spark to k8s in cluster mode, I need an authtoken to > talk with k8s. > unfortunately, many cloud provider provide token and expired with 10-15 mins. > so we need to fresh this token. > client mode is event worse, because scheduler is created on submit process. > Should I also make a pr on this ? I fix it by adding > RotatingOAuthTokenProvider and some configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27997) kubernetes client token expired
[ https://issues.apache.org/jira/browse/SPARK-27997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116925#comment-17116925 ] rameshkrishnan muthusamy commented on SPARK-27997: -- I am currently working on this request. Will be sharing the details of the PR and design link soon. > kubernetes client token expired > > > Key: SPARK-27997 > URL: https://issues.apache.org/jira/browse/SPARK-27997 > Project: Spark > Issue Type: Improvement > Components: Kubernetes, Spark Core >Affects Versions: 3.1.0 >Reporter: Henry Yu >Priority: Major > > Hi , > when I try to submit spark to k8s in cluster mode, I need an authtoken to > talk with k8s. > unfortunately, many cloud provider provide token and expired with 10-15 mins. > so we need to fresh this token. > client mode is event worse, because scheduler is created on submit process. > Should I also make a pr on this ? I fix it by adding > RotatingOAuthTokenProvider and some configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27997) kubernetes client token expired
[ https://issues.apache.org/jira/browse/SPARK-27997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883714#comment-16883714 ] Stavros Kontopoulos commented on SPARK-27997: - This is interesting. With K8s 1.16 sa tokens change as well: [https://github.com/kubernetes/community/blob/master/contributors/design-proposals/auth/bound-service-account-tokens.md] [https://thenewstack.io/no-more-forever-tokens-changes-in-identity-management-for-kubernetes|https://thenewstack.io/no-more-forever-tokens-changes-in-identity-management-for-kubernetes/] Probably k8s client libs will handle this case, but we should keep an eye on this, in case we need to provide support eg. via a config (renew or not to renew). Another question is what do you do with key rotations and long running jobs: [https://github.com/kubernetes/kubernetes/issues/20165] Back to the original ticket. [https://github.com/fabric8io/kubernetes-client/pull/1339] RotatingOAuthTokenProvider was added recently to fabric8io and its quite simple to implement a custom one. The problem is that this dependents on the service that will implement that interface. I am not aware of what we should test against. We could add support via reflection and fail if we find no class implements it but the user enabled it. [~erikerlandson] thoughts? > kubernetes client token expired > > > Key: SPARK-27997 > URL: https://issues.apache.org/jira/browse/SPARK-27997 > Project: Spark > Issue Type: Improvement > Components: Kubernetes >Affects Versions: 3.0.0 >Reporter: Henry Yu >Priority: Major > > Hi , > when I try to submit spark to k8s in cluster mode, I need an authtoken to > talk with k8s. > unfortunately, many cloud provider provide token and expired with 10-15 mins. > so we need to fresh this token. > client mode is event worse, because scheduler is created on submit process. > Should I also make a pr on this ? I fix it by adding > RotatingOAuthTokenProvider and some configuration. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
