I tried all approaches mentioned here:
https://zeppelin.apache.org/docs/latest/interpreter/spark.html#2-loading-spark-properties
1. conf
2. SPARK_SUBMIT_OPTIONS
3. add as artifacts using intepreter config
4. add using spark.dep
All lead to NPE....
What can I try next?
2017-06-26 22:37 GMT+02:00 Serega Sheypak <serega.shey...@gmail.com>:
> Ok, seems like something wrong when you try to use deps. I was able run
> simple spark job w/o third party dependecies.
> Zeppelin always throw NPE when you try to use local files using %spark.dep
> or spark interpreter conf (there is an option to set local file).
> Did anyone make it work?
>
> 2017-06-26 21:31 GMT+02:00 Serega Sheypak <serega.shey...@gmail.com>:
>
>> Hi, I'm getting strange NPE w/o any obvious reason.
>>
>> My notebook contains two paragraphs:
>>
>>
>> res0: org.apache.zeppelin.dep.Dependency = org.apache.zeppelin.dep.Depend
>> ency@6ce5acd
>>
>> %spark.dep z.load("some-local-jar.jar")
>> and
>>
>> import com.SuperClass
>> // bla-bla
>>
>> val features = sc.sequenceFile[NullWritable, BytesWritable](path)
>> // bla-bla
>> }
>>
>> println(features)
>>
>> Can you give me any input, how can I debug it?
>>
>> exception in UI is:
>> java.lang.NullPointerException at org.apache.zeppelin.spark.Spar
>> kInterpreter.open(SparkInterpreter.java:887) at
>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServ
>> er$InterpretJob.jobRun(RemoteInterpreterServer.java:489) at
>> org.apache.zeppelin.scheduler.Job.run(Job.java:181) at
>> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:47
>>
>> exception in interpreter log is:
>>
>> ERROR [2017-06-26 08:48:15,846] ({pool-2-thread-2}
>> Utils.java[invokeMethod]:41) -
>>
>> java.lang.reflect.InvocationTargetException
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>> ssorImpl.java:57)
>>
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>> thodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:606)
>>
>> at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:39)
>>
>> at org.apache.zeppelin.spark.SparkInterpreter.interpret(SparkIn
>> terpreter.java:1008)
>>
>> at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpr
>> eter.java:880)
>>
>> at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(Laz
>> yOpenInterpreter.java:70)
>>
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServ
>> er$InterpretJob.jobRun(RemoteInterpreterServer.java:489)
>>
>> at org.apache.zeppelin.scheduler.Job.run(Job.java:181)
>>
>> at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOSchedu
>> ler.java:139)
>>
>> at java.util.concurrent.Executors$RunnableAdapter.call(
>> Executors.java:471)
>>
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.run(ScheduledThreadPoolExecutor.java:292)
>>
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1145)
>>
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:615)
>>
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Caused by: java.lang.NullPointerException
>>
>> at scala.reflect.internal.SymbolTable.exitingPhase(SymbolTable.scala:256)
>>
>> at scala.tools.nsc.interpreter.IMain$Request.x$20$lzycompute(IM
>> ain.scala:896)
>>
>> at scala.tools.nsc.interpreter.IMain$Request.x$20(IMain.scala:895)
>>
>> at scala.tools.nsc.interpreter.IMain$Request.headerPreamble$lzy
>> compute(IMain.scala:895)
>>
>> at scala.tools.nsc.interpreter.IMain$Request.headerPreamble(IMa
>> in.scala:895)
>>
>> at scala.tools.nsc.interpreter.IMain$Request$Wrapper.preamble(
>> IMain.scala:918)
>>
>> at scala.tools.nsc.interpreter.IMain$CodeAssembler$$anonfun$app
>> ly$23.apply(IMain.scala:1337)
>>
>> at scala.tools.nsc.interpreter.IMain$CodeAssembler$$anonfun$app
>> ly$23.apply(IMain.scala:1336)
>>
>> at scala.tools.nsc.util.package$.stringFromWriter(package.scala:64)
>>
>> at scala.tools.nsc.interpreter.IMain$CodeAssembler$class.apply(
>> IMain.scala:1336)
>>
>> at scala.tools.nsc.interpreter.IMain$Request$Wrapper.apply(IMai
>> n.scala:908)
>>
>> at scala.tools.nsc.interpreter.IMain$Request.compile$lzycompute
>> (IMain.scala:1002)
>>
>> at scala.tools.nsc.interpreter.IMain$Request.compile(IMain.scala:997)
>>
>> at scala.tools.nsc.interpreter.IMain.compile(IMain.scala:579)
>>
>> at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:567)
>>
>> at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
>>
>> ... 18 more
>>
>> ERROR [2017-06-26 08:48:15,847] ({pool-2-thread-2} Job.java[run]:186) -
>> Job failed
>>
>> java.lang.NullPointerException
>>
>> at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpr
>> eter.java:887)
>>
>> at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(Laz
>> yOpenInterpreter.java:70)
>>
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServ
>> er$InterpretJob.jobRun(RemoteInterpreterServer.java:489)
>>
>> at org.apache.zeppelin.scheduler.Job.run(Job.java:181)
>>
>> at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOSchedu
>> ler.java:139)
>>
>> at java.util.concurrent.Executors$RunnableAdapter.call(
>> Executors.java:471)
>>
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.run(ScheduledThreadPoolExecutor.java:292)
>>
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1145)
>>
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:615)
>>
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>> Server log:
>>
>> INFO [2017-06-26 08:47:53,248] ({pool-2-thread-12}
>> RemoteInterpreterManagedProcess.java[start]:132) - Run interpreter
>> process [/home/ssheypak/zeppelin-0.8.0-SNAPSHOT/bin/interpreter.sh, -d,
>> /home/ssheypak/zeppelin-0.8.0-SNAPSHOT/interpreter/spark, -p, 45601, -l,
>> /home/ssheypak/zeppelin-0.8.0-SNAPSHOT/local-repo/2CNTZMQPW, -g, spark]
>>
>> INFO [2017-06-26 08:47:54,255] ({pool-2-thread-12}
>> RemoteInterpreter.java[init]:222) - Create remote interpreter
>> org.apache.zeppelin.spark.SparkInterpreter
>>
>> INFO [2017-06-26 08:47:54,387] ({pool-2-thread-12}
>> RemoteInterpreter.java[pushAngularObjectRegistryToRemote]:552) - Push
>> local angular object registry from ZeppelinServer to remote interpreter
>> group 2CNTZMQPW:shared_process
>>
>> INFO [2017-06-26 08:47:54,405] ({pool-2-thread-12}
>> RemoteInterpreter.java[init]:222) - Create remote interpreter
>> org.apache.zeppelin.spark.SparkSqlInterpreter
>>
>> INFO [2017-06-26 08:47:54,409] ({pool-2-thread-12}
>> RemoteInterpreter.java[init]:222) - Create remote interpreter
>> org.apache.zeppelin.spark.DepInterpreter
>>
>> INFO [2017-06-26 08:47:54,411] ({pool-2-thread-12}
>> RemoteInterpreter.java[init]:222) - Create remote interpreter
>> org.apache.zeppelin.spark.PySparkInterpreter
>>
>> ERROR [2017-06-26 08:47:54,424] ({qtp1537491187-239}
>> NotebookServer.java[onMessage]:380) - Can't handle message
>>
>> java.lang.NullPointerException
>>
>> at org.apache.zeppelin.socket.NotebookServer.addNewParagraphIfL
>> astParagraphIsExecuted(NotebookServer.java:1713)
>>
>> at org.apache.zeppelin.socket.NotebookServer.persistAndExecuteS
>> ingleParagraph(NotebookServer.java:1741)
>>
>> at org.apache.zeppelin.socket.NotebookServer.runAllParagraphs(N
>> otebookServer.java:1641)
>>
>> at org.apache.zeppelin.socket.NotebookServer.onMessage(Notebook
>> Server.java:291)
>>
>> at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(No
>> tebookSocket.java:59)
>>
>> at org.eclipse.jetty.websocket.common.events.JettyListenerEvent
>> Driver.onTextMessage(JettyListenerEventDriver.java:128)
>>
>> at org.eclipse.jetty.websocket.common.message.SimpleTextMessage
>> .messageComplete(SimpleTextMessage.java:69)
>>
>> at org.eclipse.jetty.websocket.common.events.AbstractEventDrive
>> r.appendMessage(AbstractEventDriver.java:65)
>>
>> at org.eclipse.jetty.websocket.common.events.JettyListenerEvent
>> Driver.onTextFrame(JettyListenerEventDriver.java:122)
>>
>> at org.eclipse.jetty.websocket.common.events.AbstractEventDrive
>> r.incomingFrame(AbstractEventDriver.java:161)
>>
>> at org.eclipse.jetty.websocket.common.WebSocketSession.incoming
>> Frame(WebSocketSession.java:309)
>>
>> at org.eclipse.jetty.websocket.common.extensions.ExtensionStack
>> .incomingFrame(ExtensionStack.java:214)
>>
>> at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
>>
>> at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
>>
>> at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConne
>> ction.readParse(AbstractWebSocketConnection.java:632)
>>
>> at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConne
>> ction.onFillable(AbstractWebSocketConnection.java:480)
>>
>> at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnec
>> tion.java:544)
>>
>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued
>> ThreadPool.java:635)
>>
>> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedT
>> hreadPool.java:555)
>>
>> at java.lang.Thread.run(Thread.java:745)
>>
>> INFO [2017-06-26 08:47:54,924] ({pool-2-thread-10}
>> SchedulerFactory.java[jobStarted]:131) - Job
>> paragraph_1498490257732_1130121833 started by scheduler
>> org.apache.zeppelin.interpreter.remote.RemoteInterpretershar
>> ed_session426301004
>>
>> INFO [2017-06-26 08:47:54,925] ({pool-2-thread-10}
>> Paragraph.java[jobRun]:363) - run paragraph 20170626-081737_1275486301
>> using null org.apache.zeppelin.interpreter.LazyOpenInterpreter@55b139c6
>>
>> INFO [2017-06-26 08:48:00,542] ({pool-2-thread-12}
>> NotebookServer.java[afterStatusChange]:2160) - Job
>> 20170626-081602_215470268 is finished successfully, status: FINISHED
>>
>> INFO [2017-06-26 08:48:00,567] ({pool-2-thread-12}
>> SchedulerFactory.java[jobFinished]:137) - Job paragraph_1498490162079_
>> 2075880602 <(207)%20588-0602> finished by scheduler
>> org.apache.zeppelin.interpreter.remote.RemoteInterpretershar
>> ed_session426301004
>>
>> WARN [2017-06-26 08:48:15,876] ({pool-2-thread-10}
>> NotebookServer.java[afterStatusChange]:2162) - Job
>> 20170626-081737_1275486301 is finished, status: ERROR, exception: null,
>> result: %text java.lang.NullPointerException
>>
>> at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpr
>> eter.java:887)
>>
>> at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(Laz
>> yOpenInterpreter.java:70)
>>
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServ
>> er$InterpretJob.jobRun(RemoteInterpreterServer.java:489)
>>
>> at org.apache.zeppelin.scheduler.Job.run(Job.java:181)
>>
>> at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOSchedu
>> ler.java:139)
>>
>> at java.util.concurrent.Executors$RunnableAdapter.call(
>> Executors.java:471)
>>
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.run(ScheduledThreadPoolExecutor.java:292)
>>
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1145)
>>
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:615)
>>
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>>
>>
>>
>
