Re: User Impersonation Configuration
On 11 May 2017 at 08:39, Prabhjyot Singh wrote: > Also, can you quickly check if you are able to connect to spark with user > impersonation? typo, can you quickly check if you are able to connect to spark without user impersonation? -- Warm Regards, Prabhjyot Singh
Re: User Impersonation Configuration
>From the exception, it looks like you are on 0.7.1. Which has the above-mentioned patch (https://github.com/apache/zeppelin/pull/1840). Without ZEPPELIN_IMPERSONATE_CMD and ZEPPELIN_IMPERSONATE_SPARK_PROXY_USER set it should work just fine. However, if you want to setup password-less login, you can use this doc, its was written for Zeppelin-0.6.0 but it works fine https://community.hortonworks.com/content/kbentry/81069/how-to-enable-user-impersonation-for-sh-interprete.html . Also, can you quickly check if you are able to connect to spark with user impersonation? On 10 May 2017 at 23:05, Yeshwanth Jagini wrote: > Hi prabhjyot, > thanks for your reply. > > i am using zeppelin 0.7.0 version. > when i do not specify impersonation config in zeppelin-env.sh and only in > interpreter setting, > it is throwing following exception > > ERROR [2017-05-10 17:26:30,551] ({pool-2-thread-3} Job.java[run]:188) - > Job failed > org.apache.zeppelin.interpreter.InterpreterException: Host key > verification failed. > > at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProces > s.start(RemoteInterpreterManagedProcess.java:143) > at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess. > reference(RemoteInterpreterProcess.java:73) > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open( > RemoteInterpreter.java:258) > at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType( > RemoteInterpreter.java:423) > at org.apache.zeppelin.interpreter.LazyOpenInterpreter.getFormType( > LazyOpenInterpreter.java:106) > at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:387) > at org.apache.zeppelin.scheduler.Job.run(Job.java:175) > at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run( > RemoteScheduler.java:329) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > > > i am running zeppelin as root user, root user doesn't had a password less > ssh setup where as the end web user *user1* has. > > how should i proceed now? > > Thanks, > Yeshwanth Jagini > > > > > > > > > > > On Wed, May 10, 2017 at 1:45 AM, Prabhjyot Singh < > prabhjyotsi...@apache.org> wrote: > >> Hi Yeshwant, >> >> Which version of Zeppelin are you on? >> >> If you are on latest then you don't need to do any of >> ZEPPELIN_IMPERSONATE_CMD >> or ZEPPELIN_IMPERSONATE_SPARK_PROXY_USER. Just by enabling User >> Impersonation check-box should be sufficient. >> >> Can you confirm by `ps aux | grep spark`. This is what I see on my >> machine; >> >> prabhjyotsingh@MACHINE:~/ps-zeppelin/logs$ ps aux | grep spark >> prabhjyotsingh2496 0.2 3.9 5179540 657660 s000 S12:08PM >> 0:30.68 >> /Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home/bin/java >> -cp /Users/prabhjyotsingh/ps-zeppelin/interpreter/spark/*:/Users >> /prabhjyotsingh/ps-zeppelin/zeppelin-interpreter/target/ >> lib/*:/Users/prabhjyotsingh/ps-zeppelin/zeppelin- >> interpreter/target/classes/:/Users/prabhjyotsingh/ps- >> zeppelin/zeppelin-interpreter/target/test-classes/:/Users/pr >> abhjyotsingh/ps-zeppelin/zeppelin-zengine/target/test-classe >> s/:/Users/prabhjyotsingh/ps-zeppelin/interpreter/spark/ >> zeppelin-spark_2.10-0.8.0-SNAPSHOT.jar:/Users/prabhjyotsingh/spark-2.0.0- >> bin-hadoop2.7/conf/:/Users/prabhjyotsingh/spark-2.0.0-bin-hadoop2.7/jars/* >> -Xmx1g -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:/// >> Users/prabhjyotsingh/ps-zeppelin/conf/log4j.properties >> -Dzeppelin.log.file=/Users/prabhjyotsingh/ps-zeppelin/logs/ >> zeppelin-interpreter-spark-user1-spark-prabhjyotsingh-HW11610.local.log >> org.apache.spark.deploy.SparkSubmit --conf spark.driver.extraClassPath=:/ >> Users/prabhjyotsingh/ps-zeppelin/interpreter/spark/*:/Users/ >> prabhjyotsingh/ps-zeppelin/zeppelin-interpreter/target/ >> lib/*::/Users/prabhjyotsingh/ps-zeppelin/zeppelin- >> interpreter/target/classes:/Users/prabhjyotsingh/ps- >> zeppelin/zeppelin-interpreter/target/test-classes:/Users/ >> prabhjyotsingh/ps-zeppelin/zeppelin-zengine/target/test- >> classes:/Users/prabhjyotsingh/ps-zeppelin/interpreter/spark/ >> zeppelin-spark_2.10-0.8.0-SNAPSHOT.jar --conf >> spark.driver.extraJavaOptions= -Dfile.encoding=UTF-8 >> -Dlog4j.configuration=file:///Users/prabhjyotsingh/ps-zeppelin/conf/log4j.properties >> -Dzeppelin.log.file=/Users/prabhjyotsingh/ps-zeppelin/logs/ >> zeppelin-interpreter-spark-user1-spark-prabhjyotsingh-HW11610.local.log >> --class org.apache.zeppelin.interpreter.remote.Remo
Re: User Impersonation Configuration
Hi prabhjyot, thanks for your reply. i am using zeppelin 0.7.0 version. when i do not specify impersonation config in zeppelin-env.sh and only in interpreter setting, it is throwing following exception ERROR [2017-05-10 17:26:30,551] ({pool-2-thread-3} Job.java[run]:188) - Job failed org.apache.zeppelin.interpreter.InterpreterException: Host key verification failed. at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:143) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.reference(RemoteInterpreterProcess.java:73) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:258) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:423) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.getFormType(LazyOpenInterpreter.java:106) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:387) at org.apache.zeppelin.scheduler.Job.run(Job.java:175) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:329) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) i am running zeppelin as root user, root user doesn't had a password less ssh setup where as the end web user *user1* has. how should i proceed now? Thanks, Yeshwanth Jagini On Wed, May 10, 2017 at 1:45 AM, Prabhjyot Singh wrote: > Hi Yeshwant, > > Which version of Zeppelin are you on? > > If you are on latest then you don't need to do any of ZEPPELIN_IMPERSONATE_CMD > or ZEPPELIN_IMPERSONATE_SPARK_PROXY_USER. Just by enabling User > Impersonation check-box should be sufficient. > > Can you confirm by `ps aux | grep spark`. This is what I see on my > machine; > > prabhjyotsingh@MACHINE:~/ps-zeppelin/logs$ ps aux | grep spark > prabhjyotsingh2496 0.2 3.9 5179540 657660 s000 S12:08PM > 0:30.68 > /Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home/bin/java > -cp /Users/prabhjyotsingh/ps-zeppelin/interpreter/spark/*:/ > Users/prabhjyotsingh/ps-zeppelin/zeppelin-interpreter/target/lib/*:/Users/ > prabhjyotsingh/ps-zeppelin/zeppelin-interpreter/target/classes/:/Users/ > prabhjyotsingh/ps-zeppelin/zeppelin-interpreter/target/ > test-classes/:/Users/prabhjyotsingh/ps-zeppelin/ > zeppelin-zengine/target/test-classes/:/Users/prabhjyotsingh/ps-zeppelin/ > interpreter/spark/zeppelin-spark_2.10-0.8.0-SNAPSHOT.jar: > /Users/prabhjyotsingh/spark-2.0.0-bin-hadoop2.7/conf/:/ > Users/prabhjyotsingh/spark-2.0.0-bin-hadoop2.7/jars/* -Xmx1g > -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:/// > Users/prabhjyotsingh/ps-zeppelin/conf/log4j.properties > -Dzeppelin.log.file=/Users/prabhjyotsingh/ps-zeppelin/ > logs/zeppelin-interpreter-spark-user1-spark-prabhjyotsingh-HW11610.local.log > org.apache.spark.deploy.SparkSubmit --conf spark.driver.extraClassPath=:/ > Users/prabhjyotsingh/ps-zeppelin/interpreter/spark/*:/ > Users/prabhjyotsingh/ps-zeppelin/zeppelin-interpreter/ > target/lib/*::/Users/prabhjyotsingh/ps-zeppelin/ > zeppelin-interpreter/target/classes:/Users/prabhjyotsingh/ > ps-zeppelin/zeppelin-interpreter/target/test- > classes:/Users/prabhjyotsingh/ps-zeppelin/zeppelin-zengine/ > target/test-classes:/Users/prabhjyotsingh/ps-zeppelin/ > interpreter/spark/zeppelin-spark_2.10-0.8.0-SNAPSHOT.jar --conf > spark.driver.extraJavaOptions= -Dfile.encoding=UTF-8 > -Dlog4j.configuration=file:///Users/prabhjyotsingh/ps-zeppelin/conf/log4j.properties > -Dzeppelin.log.file=/Users/prabhjyotsingh/ps-zeppelin/ > logs/zeppelin-interpreter-spark-user1-spark-prabhjyotsingh-HW11610.local.log > --class org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer > *--proxy-user > user1* /Users/prabhjyotsingh/ps-zeppelin/interpreter/spark/ > zeppelin-spark_2.10-0.8.0-SNAPSHOT.jar 50911 > prabhjyotsingh2508 0.0 0.0 2445100860 s000 S+ 12:08PM > 0:00.00 grep spark > prabhjyotsingh2495 0.0 0.0 2465144764 s000 S12:08PM > 0:00.00 /bin/bash /Users/prabhjyotsingh/ps-zeppelin/bin/interpreter.sh -d > /Users/prabhjyotsingh/ps-zeppelin/interpreter/spark -p 50911 -u user1 -l > /Users/prabhjyotsingh/ps-zeppelin/local-repo/2CEZC4JXN -g spark > prabhjyotsingh2484 0.0 0.0 2465144 1368 s000 S12:08PM > 0:00.01 /bin/bash /Users/prabhjyotsingh/ps-zeppelin/bin/interpreter.sh -d > /Users/prabhjyotsingh/ps-zeppelin/interpreter/spark -p 50911 -u user1 -l > /Users/prabhjyotsingh/ps-zeppelin/local-repo/2CEZC4JXN -g spark
Re: User Impersonation Configuration
Hi Yeshwant, Which version of Zeppelin are you on? If you are on latest then you don't need to do any of ZEPPELIN_IMPERSONATE_CMD or ZEPPELIN_IMPERSONATE_SPARK_PROXY_USER. Just by enabling User Impersonation check-box should be sufficient. Can you confirm by `ps aux | grep spark`. This is what I see on my machine; prabhjyotsingh@MACHINE:~/ps-zeppelin/logs$ ps aux | grep spark prabhjyotsingh2496 0.2 3.9 5179540 657660 s000 S12:08PM 0:30.68 /Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home/bin/java -cp /Users/prabhjyotsingh/ps-zeppelin/interpreter/spark/*:/Users/prabhjyotsingh/ps-zeppelin/zeppelin-interpreter/target/lib/*:/Users/prabhjyotsingh/ps-zeppelin/zeppelin-interpreter/target/classes/:/Users/prabhjyotsingh/ps-zeppelin/zeppelin-interpreter/target/test-classes/:/Users/prabhjyotsingh/ps-zeppelin/zeppelin-zengine/target/test-classes/:/Users/prabhjyotsingh/ps-zeppelin/interpreter/spark/zeppelin-spark_2.10-0.8.0-SNAPSHOT.jar:/Users/prabhjyotsingh/spark-2.0.0-bin-hadoop2.7/conf/:/Users/prabhjyotsingh/spark-2.0.0-bin-hadoop2.7/jars/* -Xmx1g -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///Users/prabhjyotsingh/ps-zeppelin/conf/log4j.properties -Dzeppelin.log.file=/Users/prabhjyotsingh/ps-zeppelin/logs/zeppelin-interpreter-spark-user1-spark-prabhjyotsingh-HW11610.local.log org.apache.spark.deploy.SparkSubmit --conf spark.driver.extraClassPath=:/Users/prabhjyotsingh/ps-zeppelin/interpreter/spark/*:/Users/prabhjyotsingh/ps-zeppelin/zeppelin-interpreter/target/lib/*::/Users/prabhjyotsingh/ps-zeppelin/zeppelin-interpreter/target/classes:/Users/prabhjyotsingh/ps-zeppelin/zeppelin-interpreter/target/test-classes:/Users/prabhjyotsingh/ps-zeppelin/zeppelin-zengine/target/test-classes:/Users/prabhjyotsingh/ps-zeppelin/interpreter/spark/zeppelin-spark_2.10-0.8.0-SNAPSHOT.jar --conf spark.driver.extraJavaOptions= -Dfile.encoding=UTF-8 -Dlog4j.configuration=file:///Users/prabhjyotsingh/ps-zeppelin/conf/log4j.properties -Dzeppelin.log.file=/Users/prabhjyotsingh/ps-zeppelin/logs/zeppelin-interpreter-spark-user1-spark-prabhjyotsingh-HW11610.local.log --class org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer *--proxy-user user1* /Users/prabhjyotsingh/ps-zeppelin/interpreter/spark/zeppelin-spark_2.10-0.8.0-SNAPSHOT.jar 50911 prabhjyotsingh2508 0.0 0.0 2445100860 s000 S+ 12:08PM 0:00.00 grep spark prabhjyotsingh2495 0.0 0.0 2465144764 s000 S12:08PM 0:00.00 /bin/bash /Users/prabhjyotsingh/ps-zeppelin/bin/interpreter.sh -d /Users/prabhjyotsingh/ps-zeppelin/interpreter/spark -p 50911 -u user1 -l /Users/prabhjyotsingh/ps-zeppelin/local-repo/2CEZC4JXN -g spark prabhjyotsingh2484 0.0 0.0 2465144 1368 s000 S12:08PM 0:00.01 /bin/bash /Users/prabhjyotsingh/ps-zeppelin/bin/interpreter.sh -d /Users/prabhjyotsingh/ps-zeppelin/interpreter/spark -p 50911 -u user1 -l /Users/prabhjyotsingh/ps-zeppelin/local-repo/2CEZC4JXN -g spark Ref: https://github.com/apache/zeppelin/pull/1840 On 10 May 2017 at 06:10, Yeshwanth Jagini wrote: > Hi Users, > > I am trying to setup Zeppelin for multiple users. > and i found there are multiple configurations in different places. started > tinkering with them and i didn't had any luck . > here's my setup and configuration. > > Zeppelin server is running as root. > > i edited zeppelin-env.sh and uncommented > > export ZEPPELIN_IMPERSONATE_CMD='sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} > bash -c' # Optional, when user want to run interpreter as end web > user. eg) 'sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c ' > export ZEPPELIN_IMPERSONATE_SPARK_PROXY_USER=true #Optional, by default > is true; can be set to false if you don't want to use --proxy-user option > with Spark interpreter when impersonation enabled > > and the interpreter settings are as follows > > [image: Inline image 1] > > > for different combination of configurations i am getting different types > of errors > > if i do not specify impersonation configuration in zeppelin-env.sh and > specify impersonation in interpreter setting i am getting a > org.apache.zeppelin.interpreter.InterpreterException: Host key > verification failed. > if i specify both zeppelin impersonation configuration and interpreter > impersonation config, it's throwing error as user1 cannot impersonate user1 > > if i do not specify any impersonation configuration at all , interpreter > is launching spark-submit as root. that's expected. > > could some one please explain me how to set impersonation config and > which configuration i am messing up here > > > Thanks, > Yeshwanth Jagini > -- Warm Regards, Prabhjyot Singh
User Impersonation Configuration
Hi Users, I am trying to setup Zeppelin for multiple users. and i found there are multiple configurations in different places. started tinkering with them and i didn't had any luck . here's my setup and configuration. Zeppelin server is running as root. i edited zeppelin-env.sh and uncommented export ZEPPELIN_IMPERSONATE_CMD='sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c' # Optional, when user want to run interpreter as end web user. eg) 'sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c ' export ZEPPELIN_IMPERSONATE_SPARK_PROXY_USER=true #Optional, by default is true; can be set to false if you don't want to use --proxy-user option with Spark interpreter when impersonation enabled and the interpreter settings are as follows [image: Inline image 1] for different combination of configurations i am getting different types of errors if i do not specify impersonation configuration in zeppelin-env.sh and specify impersonation in interpreter setting i am getting a org.apache.zeppelin.interpreter.InterpreterException: Host key verification failed. if i specify both zeppelin impersonation configuration and interpreter impersonation config, it's throwing error as user1 cannot impersonate user1 if i do not specify any impersonation configuration at all , interpreter is launching spark-submit as root. that's expected. could some one please explain me how to set impersonation config and which configuration i am messing up here Thanks, Yeshwanth Jagini