ecapoccia commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName URL: https://github.com/apache/spark/pull/27034#issuecomment-583735410 I managed to get the full picture on the above. TL;RD Long story short, I confirm that in 2.4.4 the executors do *not* take the driver's service account if set. They are on "default" regardless. Long version. The change that introduces the new behaviour is this one https://issues.apache.org/jira/browse/SPARK-27872 and is *not* back ported to 2.4.4. So in essence, @ayudovin 's commit does not change the behaviour compared to the codebase he had when he started the work, but Stravos' previous commit (27872) *did* change the behaviour compared to v2.4.4. As I see it, this is a breaking change for applications when upgrading from 2.4.4 to 3.0.0. All of a sudden if you upgrade Spark, executors that used to be on default will start to pick up a different service account. Which on paper is formally ok, given the major version change; however as I already pointed out 1) it is a bit surprising 2) not sure is the correct thing to do, as the permissions for drivers and executors are different. However, I think this can be accounted for in the cost of the migration from 2.4.4 to 3.0.0. A final note on general software crafting practice, @dongjoon-hyun @liyinan926. I think that exposing the configuration is yet another area where the DRY principles apply. Now you're exposing two completely different ways of setting the executors' service account, one via the pod template, the other via the direct Spark configuration parameters. They are competing with one another, if there are inconsistencies (I set A on the pod template and B via the conf param) I think the pod template prevails (in the current implementation), but then you'll have users trying to change the service account via the config parameter in vain. I'd rather go for killing the ability to set the executors service accounts via individual properties, given 3.0.0 is not yet released and given the general initiative to expose a template rather than individual parameters. And deprecating the existing method of setting the driver service account directly with a plan to eliminate it in a future release. Hope the above makes sense, although I am not sure if it's worth raising a JIRA for such a big change. Happy to hear your views on the subject, at this stage I'm pretty happy with the back porting that I have done to 2.4.4.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
