haodemon opened a new pull request #33675: URL: https://github.com/apache/spark/pull/33675
### What changes were proposed in this pull request? This change allows a spark user to provide a class which implements fabric's OAuthTokenProvider to refresh tokens throughout the life of the spark app. ``` spark.kubernetes.client.oauth.token.provider.class=<className> ``` https://javadoc.io/doc/io.fabric8/kubernetes-client/4.12.0/io/fabric8/kubernetes/client/OAuthTokenProvider.html ### Why are the changes needed? Currently, while running spark on kubernetes, one should specify oauth tokens via config before starting an application. ``` spark.kubernetes.authenticate.submission.oauthToken=<token> spark.kubernetes.authenticate.oauthToken=<token> spark.kubernetes.authenticate.oauthToken=<token> ``` The token has an expiration time (usually an hour, for GKE) and there is no way to update the token in the runtime. The spark app starts to throw exceptions. ``` io.fabric8.kubernetes.client.KubernetesClientException: Unauthorized at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:202) at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) ``` ### Does this PR introduce _any_ user-facing change? Yes, a configuration option `spark.kubernetes.client.oauth.token.provider.class` is added. ### How was this patch tested? A class which implements OAuthTokenProvider interface[0] was added into the classpath on driver node with no other spark options for tokens specified It was also tested with expired tokens specified, and the token was updated via the user-provided class. ``` --conf spark.kubernetes.authenticate.submission.oauthToken=<expired> --conf spark.kubernetes.authenticate.driver.oauthToken=<expired> --conf spark.kubernetes.authenticate.oauthToken=<expired> ``` There is no need to use any other token-related configuration options if this class is provided. An example of the user-provided class for GKE [0] https://gist.github.com/haodemon/5490fefdb258275c1f805d584319090b -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
