Spark executor on Mesos - how to set effective user id?
Hi everyone! While we are trying to utilize Spark On Mesos cluster, we are facing an issue related to effective linux user id being used to start executors on Mesos slaves: all executors are trying to use driver's linux user id to start on Mesos slaves. Let me explain in detail: spark driver program (which is going to spawn Spark on Mesos in coarse mode) is started as unprivileged linux user, for example 'user1'. We have Spark distribution unpacked and ready-to-use on every mesos slave (placed at /opt/spark, 'spark.mesos.executor.home' is pointing to this folder). And after attempt to run every executor fails to start with error log telling user 'user1' is not available. And it is really true - there is no 'user1' present on Mesos slaves. So my question is: how can I control effective user id which will be used for start executors on Mesos? Actually I was trying to setup SPARK_USER=nobody on every slave but it wasn't useful. Thanks for advice if any. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-executor-on-Mesos-how-to-set-effective-user-id-tp25118.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Spark executor on Mesos - how to set effective user id?
Can you try setting SPARK_USER at the driver? It is used to impersonate users at the executor. So if you have user setup for launching spark jobs on the executor machines, simply set it to that user name for SPARK_USER. There is another configuration that will prevents jobs being launched with a different user except the one that is configured. I don't remember the name of it but it is in the documentation. Sent from my iPhone > On 19 Oct, 2015, at 8:14 am, Eugene Chepurniy <eug...@zoomdata.com> wrote: > > Hi everyone! > While we are trying to utilize Spark On Mesos cluster, we are facing an > issue related to effective linux user id being used to start executors on > Mesos slaves: all executors are trying to use driver's linux user id to > start on Mesos slaves. > Let me explain in detail: spark driver program (which is going to spawn > Spark on Mesos in coarse mode) is started as unprivileged linux user, for > example 'user1'. We have Spark distribution unpacked and ready-to-use on > every mesos slave (placed at /opt/spark, 'spark.mesos.executor.home' is > pointing to this folder). And after attempt to run every executor fails to > start with error log telling user 'user1' is not available. And it is really > true - there is no 'user1' present on Mesos slaves. > So my question is: how can I control effective user id which will be used > for start executors on Mesos? > Actually I was trying to setup SPARK_USER=nobody on every slave but it > wasn't useful. > Thanks for advice if any. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-executor-on-Mesos-how-to-set-effective-user-id-tp25118.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Spark executor on Mesos - how to set effective user id?
Hi Jerry, I think you are referring to --no-switch_user. =) chiling...@gmail.com>于2015年10月19日 周一21:05写道： > Can you try setting SPARK_USER at the driver? It is used to impersonate > users at the executor. So if you have user setup for launching spark jobs > on the executor machines, simply set it to that user name for SPARK_USER. > There is another configuration that will prevents jobs being launched with > a different user except the one that is configured. I don't remember the > name of it but it is in the documentation. > > > Sent from my iPhone > > > On 19 Oct, 2015, at 8:14 am, Eugene Chepurniy <eug...@zoomdata.com> > wrote: > > > > Hi everyone! > > While we are trying to utilize Spark On Mesos cluster, we are facing an > > issue related to effective linux user id being used to start executors on > > Mesos slaves: all executors are trying to use driver's linux user id to > > start on Mesos slaves. > > Let me explain in detail: spark driver program (which is going to spawn > > Spark on Mesos in coarse mode) is started as unprivileged linux user, for > > example 'user1'. We have Spark distribution unpacked and ready-to-use on > > every mesos slave (placed at /opt/spark, 'spark.mesos.executor.home' is > > pointing to this folder). And after attempt to run every executor fails > to > > start with error log telling user 'user1' is not available. And it is > really > > true - there is no 'user1' present on Mesos slaves. > > So my question is: how can I control effective user id which will be used > > for start executors on Mesos? > > Actually I was trying to setup SPARK_USER=nobody on every slave but it > > wasn't useful. > > Thanks for advice if any. > > > > > > > > -- > > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-executor-on-Mesos-how-to-set-effective-user-id-tp25118.html > > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > > > - > > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > > For additional commands, e-mail: user-h...@spark.apache.org > > > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
