I tried to run Tutorial against mesos cluster by:
- define MASTER to my mesos zk
- define SPARK_EXECUTOR_URI to point to my Spark distribution (http address)
- define ZEPPELIN_JAVA_OPTS to contain -Dspark.executor.uri= and point to
the same content as SPARK_EXECUTOR_URI
I got of : Executor Spark home `spark.mesos.executor.home` is not set!
More details below. Appreciate advice in advance.
Yang.
INFO [2015-08-28 18:26:17,106] ({pool-1-thread-4}
SchedulerFactory.java[jobStarted]:132) - Job
paragraph_1423500782552_-1439281894 started by scheduler
remoteinterpreter_1299369783
INFO [2015-08-28 18:26:17,107] ({pool-1-thread-4}
Paragraph.java[jobRun]:189) - run paragraph 20150210-015302_1492795503
using sql org.apache.zeppelin.interpreter.LazyOpenInterpreter@4a1f826d
INFO [2015-08-28 18:26:17,108] ({pool-1-thread-4}
Paragraph.java[jobRun]:206) - RUN :
select age, count(1) value
from bank
where age < 30
group by age
order by age
INFO [2015-08-28 18:26:17,208] ({Thread-44}
NotebookServer.java[broadcast]:253) - SEND >> NOTE
INFO [2015-08-28 18:26:17,625] ({Thread-45}
NotebookServer.java[broadcast]:253) - SEND >> PROGRESS
INFO [2015-08-28 18:26:18,141] ({Thread-45}
NotebookServer.java[broadcast]:253) - SEND >> PROGRESS
INFO [2015-08-28 18:26:18,672] ({Thread-45}
NotebookServer.java[broadcast]:253) - SEND >> PROGRESS
INFO [2015-08-28 18:26:19,197] ({Thread-45}
NotebookServer.java[broadcast]:253) - SEND >> PROGRESS
INFO [2015-08-28 18:26:19,718] ({Thread-45}
NotebookServer.java[broadcast]:253) - SEND >> PROGRESS
INFO [2015-08-28 18:26:20,235] ({Thread-45}
NotebookServer.java[broadcast]:253) - SEND >> PROGRESS
INFO [2015-08-28 18:26:20,747] ({pool-1-thread-4}
NotebookServer.java[afterStatusChange]:673) - Job
20150210-015302_1492795503 is finished
INFO [2015-08-28 18:26:20,752] ({pool-1-thread-4}
NotebookServer.java[broadcast]:253) - SEND >> NOTE
INFO [2015-08-28 18:26:20,758] ({pool-1-thread-4}
SchedulerFactory.java[jobFinished]:138) - Job
paragraph_1423500782552_-1439281894 finished by scheduler
remoteinterpreter_1299369783
ERROR [2015-08-28 18:26:20,794] ({Thread-45}
JobProgressPoller.java[run]:57) - Can not get or update progress
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getProgress(RemoteInterpreter.java:294)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:110)
at org.apache.zeppelin.notebook.Paragraph.progress(Paragraph.java:174)
at
org.apache.zeppelin.scheduler.JobProgressPoller.run(JobProgressPoller.java:54)
Caused by: org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_getProgress(RemoteInterpreterService.java:265)
at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.getProgress(RemoteInterpreterService.java:251)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getProgress(RemoteInterpreter.java:292)
... 3 more
==> ./logs/zeppelin--mesos-s1.out <==
Exception in thread "Thread-90" org.apache.spark.SparkException: Executor
Spark home `spark.mesos.executor.home` is not set!
at
org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend$$anonfun$2.apply(MesosSchedulerBackend.scala:74)
at
org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend$$anonfun$2.apply(MesosSchedulerBackend.scala:74)
at scala.Option.getOrElse(Option.scala:120)
at
org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend.createExecutorInfo(MesosSchedulerBackend.scala:73)
at
org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend.createMesosTask(MesosSchedulerBackend.scala:270)
at
org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend$$anonfun$resourceOffers$1$$anonfun$apply$mcV$sp$2$$anonfun$apply$1.apply(MesosSchedulerBackend.scala:232)
at
org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend$$anonfun$resourceOffers$1$$anonfun$apply$mcV$sp$2$$anonfun$apply$1.apply(MesosSchedulerBackend.scala:226)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend$$anonfun$resourceOffers$1$$anonfun$apply$mcV$sp$2.apply(MesosSchedulerBackend.scala:226)
at
org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend$$anonfun$resourceOffers$1$$anonfun$apply$mcV$sp$2.apply(MesosSchedulerBackend.scala:225)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend$$anonfun$resourceOffers$1.apply$mcV$sp(MesosSchedulerBackend.scala:225)
at
org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend.inClassLoader(MesosSchedulerBackend.scala:172)
at
org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend.resourceOffers(MesosSchedulerBackend.scala:188)
I0828 18:26:20.574872 189 sched.cpp:1625] Asked to abort the driver
I0828 18:26:20.575503 189 sched.cpp:861] Aborting framework
'20150826-124922-1935808454-5050-6926-0004'
I0828 18:26:20.713506 229 sched.cpp:1591] Asked to stop the driver
I0828 18:26:20.713806 188 sched.cpp:835] Stopping framework
'20150826-124922-1935808454-5050-6926-0004'
