Re: Runtime Exception while trying to connect to remote spark interpreters, (In yarn-cluster mode in Zeppelin-0.8.0)

2018-11-20 Thread Xun Liu
hi,Sarthak Sharma You can be at the zeppelin server, Run ./bin/spark-submit --class org.apache.spark.examples.SparkPi, Test it to see if there is a problem with the spark runtime environment on the zeppelin server. > 在 2018年11月20日,下午5:39,Sarthak Sharma 写道: > > Is it similar to an existing bug

Re: Runtime Exception while trying to connect to remote spark interpreters, (In yarn-cluster mode in Zeppelin-0.8.0)

2018-11-20 Thread Sarthak Sharma
Is it similar to an existing bug related to the interpreter processes getting stuck ? (wherein the workaround is to kill the application on yarn, restart the interpreter from the interface and then try resubmitting the query again). The problem in this case is that it is intermittently happening

Re: Runtime Exception while trying to connect to remote spark interpreters, (In yarn-cluster mode in Zeppelin-0.8.0)

2018-11-20 Thread Jeff Zhang
If *zeppelin.interpreter.connect.timeout *is reached, but the yarn app is still in ACCEPTED state, then this should be a bug. The yarn app should be killed it it can not be created in the timeout threashold Sarthak Sharma 于2018年11月20日周二 下午4:47写道: > Hey, > > Like you mentioned, I'm already using

Re: Runtime Exception while trying to connect to remote spark interpreters, (In yarn-cluster mode in Zeppelin-0.8.0)

2018-11-20 Thread Sarthak Sharma
Hey, Like you mentioned, I'm already using the *spark.yarn.queue* parameter, hence I know which yarn queue it is getting scheduled in and this queue has resources available for applications since other apps are also getting scheduled there. However, assuming the queue does NOT have resources for

Re: Runtime Exception while trying to connect to remote spark interpreters, (In yarn-cluster mode in Zeppelin-0.8.0)

2018-11-19 Thread Xun Liu
HI,Sarthak Sharma The log shows that the task submitted by spark-submmit has been waiting for execution in the queue of YARN. Is there no resource for the queue of YARN? You can specify a queue with resources in the spark interpreter via the spark.yarn.queue parameter. > 在

Runtime Exception while trying to connect to remote spark interpreters, (In yarn-cluster mode in Zeppelin-0.8.0)

2018-11-19 Thread Sarthak Sharma
Hi, We already have a zeppelin-0.7.3 setup which runs fine and is in use currently but we are looking into the yarn cluster mode support for spark interpreter in zeppelin-0.8. I've built it from source from *branch-0.8 (As of Nov-15) *and am facing the following issues intermittently in some of