Well, for me, When I need to supply external libraries, I'm using export ADD_JARS="~~~.jar" export ZEPPELIN_CLASSPATH="~~~.jar" in zeppelin-env.sh
and using ADD_JARS="~~~.jar" in spark-env.sh for spark clusters. (the library jar is deployed across all clusters) I want to note that the config I'm using is quite old and deprecated. So I'm testing #308 for replace this. Of course a contribution is always welcomed, It would be cool supplying it via PR if the code is simple, or the code is large, it would be good to discuss it before writing codes. Regards, Kevin On Thu Jan 29 2015 at 2:21:34 PM Jongyoul Lee <[email protected]> wrote: > I'll resend email 'cause my attachment's size if larger than 1000000 bytes > > > ---------- Forwarded message ---------- > From: Jongyoul Lee <[email protected]> > Date: Thu, Jan 29, 2015 at 2:14 PM > Subject: Re: Zeppelin with external cluster > To: [email protected] > > > Hi Kevin, > > I also change master to spark://dicc-m002:7077. Actually, I think > interpreter.json affect what cluster is used on running codes. Anyway, my > interpreter screenshot is below, and my error is like this. > > org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 > in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage > 0.0 (TID 6, DICc-r1n029): java.lang.UnsatisfiedLinkError: no snappyjava > in java.library.path at java.lang.ClassLoader. > loadLibrary(ClassLoader.java:1886) at > java.lang.Runtime.loadLibrary0(Runtime.java:849) > at java.lang.System.loadLibrary(System.java:1088) at org.xerial.snappy. > SnappyLoader.loadNativeLibrary(SnappyLoader.java:170) at > org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:145) at > org.xerial.snappy.Snappy.<clinit>(Snappy.java:47) at org.xerial.snappy. > SnappyInputStream.hasNextChunk(SnappyInputStream.java:358) at > org.xerial.snappy.SnappyInputStream.rawRead(SnappyInputStream.java:167) > at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:150) > at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2310) > at > java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2323) > at java.io.ObjectInputStream$BlockDataInputStream. > readShort(ObjectInputStream.java:2794) at java.io.ObjectInputStream. > readStreamHeader(ObjectInputStream.java:801) at > java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at > org.apache.spark.serializer.JavaDeserializationStream$$ > anon$1.<init>(JavaSerializer.scala:57) at org.apache.spark.serializer. > JavaDeserializationStream.<init>(JavaSerializer.scala:57) at > org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:95) > at > org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:215) > at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$ > readBroadcastBlock$1.apply(TorrentBroadcast.scala:177) at > org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1000) at > org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:164) > at org.apache.spark.broadcast.TorrentBroadcast._value$ > lzycompute(TorrentBroadcast.scala:64) at org.apache.spark.broadcast. > TorrentBroadcast._value(TorrentBroadcast.scala:64) at > org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:87) > at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) at > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58) at > org.apache.spark.scheduler.Task.run(Task.scala:56) at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) Driver stacktrace: at > org.apache.spark.scheduler.DAGScheduler.org$apache$spark$ > scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1214) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1203) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1202) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at > org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1202) > at org.apache.spark.scheduler.DAGScheduler$$anonfun$ > handleTaskSetFailed$1.apply(DAGScheduler.scala:696) at > org.apache.spark.scheduler.DAGScheduler$$anonfun$ > handleTaskSetFailed$1.apply(DAGScheduler.scala:696) at > scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler. > DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:696) at > org.apache.spark.scheduler.DAGSchedulerEventProcessActor$ > $anonfun$receive$2.applyOrElse(DAGScheduler.scala:1420) at > akka.actor.Actor$class.aroundReceive(Actor.scala:465) at > org.apache.spark.scheduler.DAGSchedulerEventProcessActor. > aroundReceive(DAGScheduler.scala:1375) at akka.actor.ActorCell. > receiveMessage(ActorCell.scala:516) at > akka.actor.ActorCell.invoke(ActorCell.scala:487) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) at > akka.dispatch.Mailbox.run(Mailbox.scala:220) at akka.dispatch. > ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue. > runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin. > ForkJoinPool.runWorker(ForkJoinPool.java:1979) at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run( > ForkJoinWorkerThread.java:107) > > I think that this error is about class path. I'm running zeppelin under > /home/1001079/apache-zeppelin. Which means all classes are located under > this directory. Because zeppelin adds classes to SPARK_CLASSPATH, if slave > doesn't have that libraries on the same path, It might be no class error > occurs. > > I want to contribute by fixing this issue. Could you please tell me > regular steps for dealing with an issue? Or Is it ok to make a PR without > JIRA issue? > > Regards, > JL > > On Thu, Jan 29, 2015 at 1:55 PM, Kevin (Sangwoo) Kim <[email protected]> > wrote: > >> Hi Jongyoul, >> I'm using Zeppelin with external cluster. >> (standalone mode) >> >> All I needed to do is, writing master setting like >> export MASTER="spark://IP-ADDRESS:7077" >> in $ZEPPELIN/conf/zeppelin-env.sh >> >> If your error persists, plz post the error message in reply! >> I'm gonna looking at it. >> >> Regards, >> Kevin >> >> >> On Thu Jan 29 2015 at 12:58:41 PM Jongyoul Lee <[email protected]> >> wrote: >> >> > Hi dev, >> > >> > I've succeeded zeppelin with spark 1.2. Thanks, Moon. Now, I'm trying to >> > use zeppelin with external cluster. I've tested yesterday with >> standalone, >> > mesos, but the results are not good. In case of standalone, No >> snappyjava >> > error occurs, and in case of mesos, Nothing's happened. Do you have any >> > reference to run zeppelin with external cluster? If you don't have >> anyone, >> > I can write references for running with external cluster. >> > >> > Regards, >> > JL >> > >> > -- >> > 이종열, Jongyoul Lee, 李宗烈 >> > http://madeng.net >> > >> > > > > -- > 이종열, Jongyoul Lee, 李宗烈 > http://madeng.net > > > > -- > 이종열, Jongyoul Lee, 李宗烈 > http://madeng.net >
