Re: Setting Zeppelin to work with multiple Hadoop clusters when running Spark.
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html You don't have to rely on single NN. You can specify a kind of "NN HA alias" and underlying HDFS client would connect to NN which is active right now. Thanks for pointing HADOOP_CONF_DIR, seems like it's the thing I need. 2017-03-26 14:31 GMT+02:00 Jianfeng (Jeff) Zhang <jzh...@hortonworks.com>: > > What do you mean non-reliable ? If you want to read/write 2 hadoop cluster > in one program, I am afraid this is the only way. It is impossible to > specify multiple HADOOP_CONF_DIR under one jvm classpath. Only one > default configuration will be used. > > > Best Regard, > Jeff Zhang > > > From: Serega Sheypak <serega.shey...@gmail.com> > Reply-To: "users@zeppelin.apache.org" <users@zeppelin.apache.org> > Date: Sunday, March 26, 2017 at 7:47 PM > To: "users@zeppelin.apache.org" <users@zeppelin.apache.org> > Subject: Re: Setting Zeppelin to work with multiple Hadoop clusters when > running Spark. > > I know it, thanks, but it's non reliable solution. > > 2017-03-26 5:23 GMT+02:00 Jianfeng (Jeff) Zhang <jzh...@hortonworks.com>: > >> >> You can try to specify the namenode address for hdfs file. e.g >> >> spark.read.csv(“hdfs://localhost:9009/file”) >> >> Best Regard, >> Jeff Zhang >> >> >> From: Serega Sheypak <serega.shey...@gmail.com> >> Reply-To: "users@zeppelin.apache.org" <users@zeppelin.apache.org> >> Date: Sunday, March 26, 2017 at 2:47 AM >> To: "users@zeppelin.apache.org" <users@zeppelin.apache.org> >> Subject: Setting Zeppelin to work with multiple Hadoop clusters when >> running Spark. >> >> Hi, I have three hadoop clusters. Each cluster has it's own NN HA >> configured and YARN. >> I want to allow user to read from ant cluster and write to any cluster. >> Also user should be able to choose where to run is spark job. >> What is the right way to configure it in Zeppelin? >> >> >
Re: Setting Zeppelin to work with multiple Hadoop clusters when running Spark.
I know it, thanks, but it's non reliable solution. 2017-03-26 5:23 GMT+02:00 Jianfeng (Jeff) Zhang <jzh...@hortonworks.com>: > > You can try to specify the namenode address for hdfs file. e.g > > spark.read.csv(“hdfs://localhost:9009/file”) > > Best Regard, > Jeff Zhang > > > From: Serega Sheypak <serega.shey...@gmail.com> > Reply-To: "users@zeppelin.apache.org" <users@zeppelin.apache.org> > Date: Sunday, March 26, 2017 at 2:47 AM > To: "users@zeppelin.apache.org" <users@zeppelin.apache.org> > Subject: Setting Zeppelin to work with multiple Hadoop clusters when > running Spark. > > Hi, I have three hadoop clusters. Each cluster has it's own NN HA > configured and YARN. > I want to allow user to read from ant cluster and write to any cluster. > Also user should be able to choose where to run is spark job. > What is the right way to configure it in Zeppelin? > >
Why does zeppelin try to do during web application startup?
Hi, I'm trying run Zeppelin 0.8.0-SNAPSHOT in Docker. Startup takes forever. It starts in seconds when launched on host, not in Docker container. I suspect Docker container has poorly configured network and some part of zeppelin tries to reach remote resource. SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Minutes here... Mar 26, 2017 11:44:49 AM com.sun.jersey.api.core.PackagesResourceConfig init Then WARNING: The following warnings have been detected with resource and/or provider classes: WARNING: A HTTP GET method, public javax.ws.rs.core.Response org.apache.zeppelin.rest.CredentialRestApi.getCredentials(java.lang.String) throws java.io.IOException,java.lang.IllegalArgumentException, should not consume any entity. WARNING: A HTTP GET method, public javax.ws.rs.core.Response org.apache.zeppelin.rest.InterpreterRestApi.listInterpreter(java.lang.String), should not consume any entity. WARNING: A sub-resource method, public javax.ws.rs.core.Response org.apache.zeppelin.rest.NotebookRestApi.createNote(java.lang.String) throws java.io.IOException, with URI template, "/", is treated as a resource method WARNING: A sub-resource method, public javax.ws.rs.core.Response org.apache.zeppelin.rest.NotebookRestApi.getNoteList() throws java.io.IOException, with URI template, "/", is treated as a resource method And nothing happens after.
Setting Zeppelin to work with multiple Hadoop clusters when running Spark.
Hi, I have three hadoop clusters. Each cluster has it's own NN HA configured and YARN. I want to allow user to read from ant cluster and write to any cluster. Also user should be able to choose where to run is spark job. What is the right way to configure it in Zeppelin?
Preconfigure Spark interpreter
Hi, I need to pre-configure spark interpreter with my own artifacts and internal repositories. How can I do it?
Re: Preconfigure Spark interpreter
Aha, thanks. I'm building Zeppelin from source, so I can put my custom settings directly? BTW, why does interpreter-list file don't contain spark interpreter? 2017-04-22 13:33 GMT+02:00 Fabian Böhnlein <fabian.boehnl...@gmail.com>: > Do it via the Ui once and you'll see how interpreter.json of the Zeppelin > installation will be changed. > > On Sat, Apr 22, 2017, 11:35 Serega Sheypak <serega.shey...@gmail.com> > wrote: > >> Hi, I need to pre-configure spark interpreter with my own artifacts and >> internal repositories. How can I do it? >> >
Custom spark for zeppelin and interpreter-list
Hi, I have few concerns I can't resolve right now. I definitely can go though the source code and find the solution, but I would like to understand the idea behind. I'm building Zeppelin from sources using 0.8.0-SNAPSHOT. I do build it with custom cloudera CDH spark 2.0-something. I can't understand if built and started zeppelin uses my custom zeppelin-spark interpreter or not? interpreter-list has maven coordinates. What is it for? Will zeppelin try to grab interpreters from remote maven repo? interpreter-list doesn't have spark in it. How does zeppelin figure out what and how spark interpreter use?
Re: Is there any possibility to get link to Spark YARN application master from notebook
Nevermind, I forgot that it's in intepreter settings https://cloud.githubusercontent.com/assets/5082742/20110797/c6852202-a60b-11e6-8264-93437a58f752.gif 2017-07-10 10:46 GMT+02:00 Serega Sheypak <serega.shey...@gmail.com>: > Super stupid question, sorry. > I can't find button / link to Spark Master in notebook IU. >
Re: Interpreter %sq not found Zeppelin swallows last "l" for some reason...?
I'm using Mac. I don't know what was the problem, I just switched to
Safari. Not sure if it wasn't caused by some other issue.
I was trying to run spark compiled for scala 2.11 with 3rd party 2.10
libraries at that time.
It was causing super-non-obvious exceptions. I stopped to do it and it
solved all my problems.
2017-07-10 20:37 GMT+02:00 Jongyoul Lee <jongy...@gmail.com>:
> Thanks for telling me that. I'll also test it with chrome. Might you use
> it in Windows? I never heard about it so I'm just asking something to find
> a clue.
>
> On Mon, 10 Jul 2017 at 17:10 Serega Sheypak <serega.shey...@gmail.com>
> wrote:
>
>> It was Chrome, probably Version 59.0.3071.115 (Official Build) (64-bit)
>> Can't reproduce same issue in Safari. Safari works fine.
>>
>> 2017-07-10 5:58 GMT+02:00 Jongyoul Lee <jongy...@gmail.com>:
>>
>>> Which browser do you use it?
>>>
>>> On Mon, Jun 26, 2017 at 11:49 PM, Serega Sheypak <
>>> serega.shey...@gmail.com> wrote:
>>>
>>>> Hi, I get super weird exception:
>>>>
>>>> ERROR [2017-06-26 07:44:17,523] ({qtp2016336095-99} NotebookServer.java[
>>>> persistAndExecuteSingleParagraph]:1749) - Exception from run
>>>>
>>>> org.apache.zeppelin.interpreter.InterpreterException:
>>>> paragraph_1498480084440_1578830546's Interpreter %sq not found
>>>>
>>>> I have three paragraphs in my notebook
>>>>
>>>>
>>>>
>>>> %spark.dep
>>>>
>>>> z.load("my.local.jar.jar")
>>>>
>>>>
>>>> %spark
>>>>
>>>> import com.myorg.SuperClass
>>>>
>>>> // bla-bla
>>>>
>>>> features.toDF().registerTempTable("features")
>>>>
>>>>
>>>> %sql
>>>>
>>>> select f1, f2, count(*) as cnt from features;
>>>>
>>>>
>>>> The last one gets this weird exception. Where did "l" go?
>>>>
>>>
>>>
>>>
>>> --
>>> 이종열, Jongyoul Lee, 李宗烈
>>> http://madeng.net
>>>
>>
>> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>
Re: Zeppelin without internet, speedup startup
Thanks! Will these buttons disappear if I disable Helium? [image: Встроенное изображение 1] I don't quite well understand what Helium is https://cwiki.apache.org/confluence/display/ZEPPELIN/Helium+proposal So what I propose is a framework, code-named *Helium* that turns Zeppelin into a data analytics application platform by: - Leveraging computing resources provided by Interpreters - Generalizing dependency loader - Providing SDK on top of Angular Display system - adding a package repository I don't understand is it something internal to Zeppelin devs or it's some public feature used by end-users like data analysts. 2017-06-29 21:11 GMT+02:00 Иван Шаповалов <shapovalov.iva...@gmail.com>: > if you use helium - it will be installing npm at start time. See > HeliumVisualizationFactory.java > > 2017-06-29 17:09 GMT+03:00 Serega Sheypak <serega.shey...@gmail.com>: > >> Hi, I'm starting zeppelin w/o internet. Looks like it tries to access >> some external resources. Is it true? >> Can I stop it somehow? It takes 2 minutes to start. I failed to find it >> in source code. >> >> Thanks! >> > > > > -- > Ivan Shapovalov > Kharkov, Ukraine > >
Re: java.lang.ClassNotFoundException: org.apache.zeppelin.spark.SparkInterpreter with 0.7.2 binary
Yeah, I was my misunderstanding of Intepreter REST API behaviour. Update
endpoint actually overwrites everything.
I tried to set specific spark properties only:
custom_properties = {
'spark.app.name': 'Zeppelin',
'master': 'yarn-client',
'spark.dynamicAllocation.enabled': 'false',
'spark.executor.instances': '10',
'spark.executor.memory': '16g',
'spark.cores.max': '4'}
and it erased everything in Interpreter except these properties.
Now I get spark interpreter config, "patch" part of it and send the whole
conf back using Update endpoint.
Issue is solved. I misunderstood how update works.
2017-06-29 21:14 GMT+02:00 Иван Шаповалов <shapovalov.iva...@gmail.com>:
> looks like you create an interpreter setting via rest api and it is
> configured well enough
>
> 2017-06-29 18:32 GMT+03:00 Serega Sheypak <serega.shey...@gmail.com>:
>
>> hi, I don't quite well understand what do I do wrong.\
>>
>> can you help me please?
>>
>> ERROR [2017-06-29 08:21:53,695] ({qtp1521110008-116}
>> InterpreterFactory.java[createRepl]:246) - exception checking server
>> classloader driver
>>
>> java.lang.ClassNotFoundException: org.apache.zeppelin.spark.Spar
>> kInterpreter
>>
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>
>> at java.security.AccessController.doPrivileged(Native Method)
>>
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>
>> at java.lang.Class.forName0(Native Method)
>>
>> at java.lang.Class.forName(Class.java:191)
>>
>> at org.apache.zeppelin.interpreter.InterpreterFactory.
>> createRepl(InterpreterFactory.java:241)
>>
>> at org.apache.zeppelin.interpreter.InterpreterFactory.createInt
>> erpretersForNote(InterpreterFactory.java:206)
>>
>> at org.apache.zeppelin.interpreter.InterpreterFactory.createOrG
>> etInterpreterList(InterpreterFactory.java:332)
>>
>> at org.apache.zeppelin.interpreter.InterpreterFactory.getInterp
>> reter(InterpreterFactory.java:395)
>>
>> at org.apache.zeppelin.notebook.Note.run(Note.java:600)
>>
>> at org.apache.zeppelin.socket.NotebookServer.persistAndExecuteS
>> ingleParagraph(NotebookServer.java:1641)
>>
>> at org.apache.zeppelin.socket.NotebookServer.runAllParagraphs(N
>> otebookServer.java:1588)
>>
>> at org.apache.zeppelin.socket.NotebookServer.onMessage(Notebook
>> Server.java:268)
>>
>> at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(No
>> tebookSocket.java:59)
>>
>> at org.eclipse.jetty.websocket.common.events.JettyListenerEvent
>> Driver.onTextMessage(JettyListenerEventDriver.java:128)
>>
>> at org.eclipse.jetty.websocket.common.message.SimpleTextMessage
>> .messageComplete(SimpleTextMessage.java:69)
>>
>> at org.eclipse.jetty.websocket.common.events.AbstractEventDrive
>> r.appendMessage(AbstractEventDriver.java:65)
>>
>> at org.eclipse.jetty.websocket.common.events.JettyListenerEvent
>> Driver.onTextFrame(JettyListenerEventDriver.java:122)
>>
>> at org.eclipse.jetty.websocket.common.events.AbstractEventDrive
>> r.incomingFrame(AbstractEventDriver.java:161)
>>
>> at org.eclipse.jetty.websocket.common.WebSocketSession.incoming
>> Frame(WebSocketSession.java:309)
>>
>> at org.eclipse.jetty.websocket.common.extensions.ExtensionStack
>> .incomingFrame(ExtensionStack.java:214)
>>
>> at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
>>
>> at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
>>
>> at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConne
>> ction.readParse(AbstractWebSocketConnection.java:632)
>>
>> at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConne
>> ction.onFillable(AbstractWebSocketConnection.java:480)
>>
>> at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnec
>> tion.java:544)
>>
>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued
>> ThreadPool.java:635)
>>
>> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedT
>> hreadPool.java:555)
>>
>> at java.lang.Thread.run(Thread.java:745)
>>
>
>
>
> --
> Ivan Shapovalov
> Kharkov, Ukraine
>
>
Re: Hitting strange NPE
Hi, resolved. root cause:
I've recompiled zeppelin with spark 2.11, used spark 2.0 complied for scala
2.11 but external artifacts were complied for scala 2.10
I did provide correct external artifacts and Zeppelin started to work.
2017-06-26 22:49 GMT+02:00 Serega Sheypak <serega.shey...@gmail.com>:
> I tried all approaches mentioned here:
> https://zeppelin.apache.org/docs/latest/interpreter/spark.
> html#2-loading-spark-properties
> 1. conf
> 2. SPARK_SUBMIT_OPTIONS
> 3. add as artifacts using intepreter config
> 4. add using spark.dep
>
> All lead to NPE
> What can I try next?
>
> 2017-06-26 22:37 GMT+02:00 Serega Sheypak <serega.shey...@gmail.com>:
>
>> Ok, seems like something wrong when you try to use deps. I was able run
>> simple spark job w/o third party dependecies.
>> Zeppelin always throw NPE when you try to use local files using %spark.dep
>> or spark interpreter conf (there is an option to set local file).
>> Did anyone make it work?
>>
>> 2017-06-26 21:31 GMT+02:00 Serega Sheypak <serega.shey...@gmail.com>:
>>
>>> Hi, I'm getting strange NPE w/o any obvious reason.
>>>
>>> My notebook contains two paragraphs:
>>>
>>>
>>> res0: org.apache.zeppelin.dep.Dependency =
>>> org.apache.zeppelin.dep.Dependency@6ce5acd
>>>
>>> %spark.dep z.load("some-local-jar.jar")
>>> and
>>>
>>> import com.SuperClass
>>> // bla-bla
>>>
>>> val features = sc.sequenceFile[NullWritable, BytesWritable](path)
>>> // bla-bla
>>> }
>>>
>>> println(features)
>>>
>>> Can you give me any input, how can I debug it?
>>>
>>> exception in UI is:
>>> java.lang.NullPointerException at org.apache.zeppelin.spark.Spar
>>> kInterpreter.open(SparkInterpreter.java:887) at
>>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServ
>>> er$InterpretJob.jobRun(RemoteInterpreterServer.java:489) at
>>> org.apache.zeppelin.scheduler.Job.run(Job.java:181) at
>>> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:47
>>>
>>> exception in interpreter log is:
>>>
>>> ERROR [2017-06-26 08:48:15,846] ({pool-2-thread-2}
>>> Utils.java[invokeMethod]:41) -
>>>
>>> java.lang.reflect.InvocationTargetException
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>> ssorImpl.java:57)
>>>
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>> thodAccessorImpl.java:43)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>
>>> at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:39)
>>>
>>> at org.apache.zeppelin.spark.SparkInterpreter.interpret(SparkIn
>>> terpreter.java:1008)
>>>
>>> at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpr
>>> eter.java:880)
>>>
>>> at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(Laz
>>> yOpenInterpreter.java:70)
>>>
>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServ
>>> er$InterpretJob.jobRun(RemoteInterpreterServer.java:489)
>>>
>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:181)
>>>
>>> at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOSchedu
>>> ler.java:139)
>>>
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executor
>>> s.java:471)
>>>
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>> tureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>> tureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>> Executor.java:1145)
>>>
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>> lExecutor.java:615)
>>>
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> Caused by: java.lang.NullPointerException
>>>
>>> at scala.reflect.internal.SymbolTable.exitingPhase(SymbolTable.
>>
Re: Configuring Zeppelin spark interpreter to work with different hadoop clusters
hi, thanks for your reply. How should I set this variable? I'm looking at Spark interpreter config UI. It doesn't allow me to set env variable. https://zeppelin.apache.org/docs/latest/interpreter/spark.html#1-export-spark_home tells that HADOOP_CONF_DIR should be set once per whole Zeppelin instance. What do I miss? Thanks! 2017-06-30 16:43 GMT+02:00 Jeff Zhang <zjf...@gmail.com>: > > Right, create three spark interpreters for your 3 yarn cluster. > > > > Serega Sheypak <serega.shey...@gmail.com>于2017年6月30日周五 下午10:33写道： > >> Hi, thanks for your reply! >> What do you mean by that? >> I can have only one env variable HADOOP_CONF_DIR... >> And how can user pick which env to run? >> >> Or you mean I have to create three Spark interpreters and each of them >> would have it's own HADOOP_CONF_DIR pointed to single cluster config? >> >> 2017-06-30 16:21 GMT+02:00 Jeff Zhang <zjf...@gmail.com>: >> >>> >>> Try set HADOOP_CONF_DIR for each yarn conf in interpreter setting. >>> >>> Serega Sheypak <serega.shey...@gmail.com>于2017年6月30日周五 下午10:11写道： >>> >>>> Hi I have several different hadoop clusters, each of them has it's own >>>> YARN. >>>> Is it possible to configure single Zeppelin instance to work with >>>> different clusters? >>>> I want to run spark on cluster A if data is there. Right now my >>>> Zeppelin runs on single cluster and it sucks data from remote clusters >>>> which is inefficient. Zeppelin can easily access any HDFS cluster, but what >>>> about YARN? >>>> >>>> What are the correct approaches to solve the problem? >>>> >>> >>
Re: Configuring Zeppelin spark interpreter to work with different hadoop clusters
Hi, thanks for your reply! What do you mean by that? I can have only one env variable HADOOP_CONF_DIR... And how can user pick which env to run? Or you mean I have to create three Spark interpreters and each of them would have it's own HADOOP_CONF_DIR pointed to single cluster config? 2017-06-30 16:21 GMT+02:00 Jeff Zhang <zjf...@gmail.com>: > > Try set HADOOP_CONF_DIR for each yarn conf in interpreter setting. > > Serega Sheypak <serega.shey...@gmail.com>于2017年6月30日周五 下午10:11写道： > >> Hi I have several different hadoop clusters, each of them has it's own >> YARN. >> Is it possible to configure single Zeppelin instance to work with >> different clusters? >> I want to run spark on cluster A if data is there. Right now my Zeppelin >> runs on single cluster and it sucks data from remote clusters which is >> inefficient. Zeppelin can easily access any HDFS cluster, but what about >> YARN? >> >> What are the correct approaches to solve the problem? >> >
Re: Zeppelin REST api for interpreters
Ah, it's there, thanks! 2017-06-28 12:44 GMT+02:00 Иван Шаповалов <shapovalov.iva...@gmail.com>: > for 3.2 https://zeppelin.apache.org/docs/0.7.2/rest-api/rest- > interpreter.html should work > > 2017-06-28 12:14 GMT+03:00 Serega Sheypak <serega.shey...@gmail.com>: > >> Hi, I'm reading >> https://zeppelin.apache.org/docs/0.7.2/rest-api/rest-notebook.html >> >> I has great REST API for notebooks and paragraphs. >> I'm looking for interpreter configuration. I want to automate Zeppelin >> deployment and I need: >> 1. put zeppelin war on node (done) >> 2. start war and connect to monitoring, logging, alerting (done) >> == >> 3. Configure spark interpreter somehow >> 3.1. spark conf location >> 3.2. spark settings >> 3.3. add 3rd party shared libs stored on the same node. >> >> What is the right way to do 3.1 .. 3.3? Right now I do it manually and >> it's pretty annoying. Wanna automate it. >> >> Thanks! >> > > > > -- > Ivan Shapovalov > Kharkov, Ukraine > >
Zeppelin REST api for interpreters
Hi, I'm reading https://zeppelin.apache.org/docs/0.7.2/rest-api/rest-notebook.html I has great REST API for notebooks and paragraphs. I'm looking for interpreter configuration. I want to automate Zeppelin deployment and I need: 1. put zeppelin war on node (done) 2. start war and connect to monitoring, logging, alerting (done) == 3. Configure spark interpreter somehow 3.1. spark conf location 3.2. spark settings 3.3. add 3rd party shared libs stored on the same node. What is the right way to do 3.1 .. 3.3? Right now I do it manually and it's pretty annoying. Wanna automate it. Thanks!
Re: Is Zeppelin spark-version agnostic?
Hi Jeff! Am I right that I don't have to recomplie Zeppelin for scala 2.11 to make it work with spark 2.0 complied for scala 2.11? Zeppelin doesn't really care about spark scala version and spark version overall (1.6 ... 2.0) Thanks! 2017-06-27 18:08 GMT+02:00 Serega Sheypak <serega.shey...@gmail.com>: > Awesome, thanks! > > вт, 27 июня 2017 г. в 13:47, Jeff Zhang <zjf...@gmail.com>: > >> >> Zeppelin 0.7.2 support all these versions. You can create spark >> interpreter for each version in interpreter setting page. And you need to >> set the SPARK_HOME to the path where your spark is installed. >> >> >> >> Serega Sheypak <serega.shey...@gmail.com>于2017年6月27日周二 下午6:14写道： >> >>> Hi, can zeppelin spark interpreter support spark 1.6 / 2.0 / 2.1 >>> I didn't find which spark versions are supported... >>> >>
Zeppelin without internet, speedup startup
Hi, I'm starting zeppelin w/o internet. Looks like it tries to access some external resources. Is it true? Can I stop it somehow? It takes 2 minutes to start. I failed to find it in source code. Thanks!
Customizing sparkconfig before starting spark app
Hi, I have few questions about spark application customization 1. Is it possible to set spark app name from notebook, not from zeppelin conf? 2. Is is possible to register custom kryo serializers? 3. Is it possible to configure user name? Right now I'm running zeppelin as root and all jobs are submitted as root. I want to use logged user name instead.
How to debug spark.dep job?
Hi, seems like I was able to start Zeppelin. I have inhouse artifactory and I want zeppelin to download my artifacts from artifactory and use classes in spark job afterwards. Notebook submission hangs %spark.dep and never finishes. Zeppelin outputs to log that DepInterpreter job has been staerted. What is the rgiht way to figure out what it tires t do?
Re: Custom spark for zeppelin and interpreter-list
I'm building Zeppelin from sources. I suppose it means that "default spark interpreter" which has my custom spark deps is included into built zeppelin dist. It solves my problem! Thanks for explanation. 2017-04-23 5:08 GMT+02:00 moon soo Lee <m...@apache.org>: > Hi, > > 'conf/interpreter-list' is just catalogue file that > `/bin/install-interpreter.sh' uses. > The information is not being used any other place. > > '/bin/install-interpreter.sh' use 'conf/interpreter-list' to 1) print list > of interpreter that Zeppelin community provides 2) convert short name to > group:artifact:version, so user doesn't have to provide -t option. > > Spark interpreter is being included in both zeppelin-bin-all and > zeppelin-bin-netinst package, that's why `conf/interpreter-list' doesn't > have it. > > So, if you're trying to install your custom interpreter using > 'bin/install-interpreter.sh', you can still do without modifying > 'conf/interpreter-list' by providing '-t' option. If you're installing your > custom interpreter without using 'bin/install-interpreter.sh', then > 'conf/interpreter-list' is not related at all. > > Hope this helps. > > Best, > moon > > > On Sat, Apr 22, 2017 at 1:04 PM Serega Sheypak <serega.shey...@gmail.com> > wrote: > >> Hi, I have few concerns I can't resolve right now. I definitely can go >> though the source code and find the solution, but I would like to >> understand the idea behind. >> I'm building Zeppelin from sources using 0.8.0-SNAPSHOT. I do build it >> with custom cloudera CDH spark 2.0-something. >> I can't understand if built and started zeppelin uses my custom >> zeppelin-spark interpreter or not? >> >> interpreter-list has maven coordinates. What is it for? Will zeppelin try >> to grab interpreters from remote maven repo? >> >> interpreter-list doesn't have spark in it. How does zeppelin figure out >> what and how spark interpreter use? >> >
Interpreter %sq not found Zeppelin swallows last "l" for some reason...?
Hi, I get super weird exception:
ERROR [2017-06-26 07:44:17,523] ({qtp2016336095-99}
NotebookServer.java[persistAndExecuteSingleParagraph]:1749) - Exception
from run
org.apache.zeppelin.interpreter.InterpreterException:
paragraph_1498480084440_1578830546's Interpreter %sq not found
I have three paragraphs in my notebook
%spark.dep
z.load("my.local.jar.jar")
%spark
import com.myorg.SuperClass
// bla-bla
features.toDF().registerTempTable("features")
%sql
select f1, f2, count(*) as cnt from features;
The last one gets this weird exception. Where did "l" go?
Hitting strange NPE
Hi, I'm getting strange NPE w/o any obvious reason.
My notebook contains two paragraphs:
res0: org.apache.zeppelin.dep.Dependency =
org.apache.zeppelin.dep.Dependency@6ce5acd
%spark.dep z.load("some-local-jar.jar")
and
import com.SuperClass
// bla-bla
val features = sc.sequenceFile[NullWritable, BytesWritable](path)
// bla-bla
}
println(features)
Can you give me any input, how can I debug it?
exception in UI is:
java.lang.NullPointerException at
org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:887)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:489)
at org.apache.zeppelin.scheduler.Job.run(Job.java:181) at
org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:47
exception in interpreter log is:
ERROR [2017-06-26 08:48:15,846] ({pool-2-thread-2}
Utils.java[invokeMethod]:41) -
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:39)
at
org.apache.zeppelin.spark.SparkInterpreter.interpret(SparkInterpreter.java:1008)
at
org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:880)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:489)
at org.apache.zeppelin.scheduler.Job.run(Job.java:181)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at scala.reflect.internal.SymbolTable.exitingPhase(SymbolTable.scala:256)
at
scala.tools.nsc.interpreter.IMain$Request.x$20$lzycompute(IMain.scala:896)
at scala.tools.nsc.interpreter.IMain$Request.x$20(IMain.scala:895)
at
scala.tools.nsc.interpreter.IMain$Request.headerPreamble$lzycompute(IMain.scala:895)
at scala.tools.nsc.interpreter.IMain$Request.headerPreamble(IMain.scala:895)
at
scala.tools.nsc.interpreter.IMain$Request$Wrapper.preamble(IMain.scala:918)
at
scala.tools.nsc.interpreter.IMain$CodeAssembler$$anonfun$apply$23.apply(IMain.scala:1337)
at
scala.tools.nsc.interpreter.IMain$CodeAssembler$$anonfun$apply$23.apply(IMain.scala:1336)
at scala.tools.nsc.util.package$.stringFromWriter(package.scala:64)
at
scala.tools.nsc.interpreter.IMain$CodeAssembler$class.apply(IMain.scala:1336)
at scala.tools.nsc.interpreter.IMain$Request$Wrapper.apply(IMain.scala:908)
at
scala.tools.nsc.interpreter.IMain$Request.compile$lzycompute(IMain.scala:1002)
at scala.tools.nsc.interpreter.IMain$Request.compile(IMain.scala:997)
at scala.tools.nsc.interpreter.IMain.compile(IMain.scala:579)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:567)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
... 18 more
ERROR [2017-06-26 08:48:15,847] ({pool-2-thread-2} Job.java[run]:186) - Job
failed
java.lang.NullPointerException
at
org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:887)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:489)
at org.apache.zeppelin.scheduler.Job.run(Job.java:181)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Server log:
INFO
Re: NPE in SparkInterpreter.java
Hi, I have more or less the same symptom
if (Utils.isScala2_10()) {
binder = (Map) getValue("_binder");
} else {
binder = (Map) getLastObject();
}
binder.put("sc", sc); // EXCEPTION HERE
java.lang.NullPointerException at
org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:887)
at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:489)
at org.apache.zeppelin.scheduler.Job.run(Job.java:181) at
org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
I'm going to check this one stuff:
https://stackoverflow.com/questions/43289067/getting-nullpointerexception-when-running-spark-code-in-zeppelin-0-7-1
2017-06-26 20:41 GMT+02:00 Richard Xin :
> Anyone?
>
>
>
> On Thursday, June 22, 2017, 3:00:01 PM PDT, Richard Xin <
> richardxin...@yahoo.com> wrote:
>
>
> apache/zeppelin
>
>
> apache/zeppelin
>
> zeppelin - Mirror of Apache Zeppelin
>
>
>
> in the code I see // Some case,
> scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call
> throws an NPE
>
> We had issues that Zeppelin note errored out immediately without any hint
> messages, and we are seeing this NPE in the spark_interpreter log (Zeppelin
> 0.7.1 on AWS EMR cluster). can someone explain what's the likely cause for
> this error and how to prevent it from happening again?
>
> Thanks,
>
>
Re: Hitting strange NPE
I tried all approaches mentioned here:
https://zeppelin.apache.org/docs/latest/interpreter/spark.html#2-loading-spark-properties
1. conf
2. SPARK_SUBMIT_OPTIONS
3. add as artifacts using intepreter config
4. add using spark.dep
All lead to NPE
What can I try next?
2017-06-26 22:37 GMT+02:00 Serega Sheypak <serega.shey...@gmail.com>:
> Ok, seems like something wrong when you try to use deps. I was able run
> simple spark job w/o third party dependecies.
> Zeppelin always throw NPE when you try to use local files using %spark.dep
> or spark interpreter conf (there is an option to set local file).
> Did anyone make it work?
>
> 2017-06-26 21:31 GMT+02:00 Serega Sheypak <serega.shey...@gmail.com>:
>
>> Hi, I'm getting strange NPE w/o any obvious reason.
>>
>> My notebook contains two paragraphs:
>>
>>
>> res0: org.apache.zeppelin.dep.Dependency = org.apache.zeppelin.dep.Depend
>> ency@6ce5acd
>>
>> %spark.dep z.load("some-local-jar.jar")
>> and
>>
>> import com.SuperClass
>> // bla-bla
>>
>> val features = sc.sequenceFile[NullWritable, BytesWritable](path)
>> // bla-bla
>> }
>>
>> println(features)
>>
>> Can you give me any input, how can I debug it?
>>
>> exception in UI is:
>> java.lang.NullPointerException at org.apache.zeppelin.spark.Spar
>> kInterpreter.open(SparkInterpreter.java:887) at
>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServ
>> er$InterpretJob.jobRun(RemoteInterpreterServer.java:489) at
>> org.apache.zeppelin.scheduler.Job.run(Job.java:181) at
>> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:47
>>
>> exception in interpreter log is:
>>
>> ERROR [2017-06-26 08:48:15,846] ({pool-2-thread-2}
>> Utils.java[invokeMethod]:41) -
>>
>> java.lang.reflect.InvocationTargetException
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>> ssorImpl.java:57)
>>
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>> thodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:606)
>>
>> at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:39)
>>
>> at org.apache.zeppelin.spark.SparkInterpreter.interpret(SparkIn
>> terpreter.java:1008)
>>
>> at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpr
>> eter.java:880)
>>
>> at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(Laz
>> yOpenInterpreter.java:70)
>>
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServ
>> er$InterpretJob.jobRun(RemoteInterpreterServer.java:489)
>>
>> at org.apache.zeppelin.scheduler.Job.run(Job.java:181)
>>
>> at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOSchedu
>> ler.java:139)
>>
>> at java.util.concurrent.Executors$RunnableAdapter.call(
>> Executors.java:471)
>>
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.run(ScheduledThreadPoolExecutor.java:292)
>>
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1145)
>>
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:615)
>>
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Caused by: java.lang.NullPointerException
>>
>> at scala.reflect.internal.SymbolTable.exitingPhase(SymbolTable.scala:256)
>>
>> at scala.tools.nsc.interpreter.IMain$Request.x$20$lzycompute(IM
>> ain.scala:896)
>>
>> at scala.tools.nsc.interpreter.IMain$Request.x$20(IMain.scala:895)
>>
>> at scala.tools.nsc.interpreter.IMain$Request.headerPreamble$lzy
>> compute(IMain.scala:895)
>>
>> at scala.tools.nsc.interpreter.IMain$Request.headerPreamble(IMa
>> in.scala:895)
>>
>> at scala.tools.nsc.interpreter.IMain$Request$Wrapper.preamble(
>> IMain.scala:918)
>>
>> at scala.tools.nsc.interpreter.IMain$CodeAssembler$$anonfun$app
>> ly$23.apply(IMain.scala:1337)
>>
>> at scala.tools.nsc.interpreter.IMain$CodeAssembler$$anonfun$app
>> ly$23.apply(IMain.scala:1336)
>>
Re: Hitting strange NPE
Ok, seems like something wrong when you try to use deps. I was able run
simple spark job w/o third party dependecies.
Zeppelin always throw NPE when you try to use local files using %spark.dep
or spark interpreter conf (there is an option to set local file).
Did anyone make it work?
2017-06-26 21:31 GMT+02:00 Serega Sheypak <serega.shey...@gmail.com>:
> Hi, I'm getting strange NPE w/o any obvious reason.
>
> My notebook contains two paragraphs:
>
>
> res0: org.apache.zeppelin.dep.Dependency = org.apache.zeppelin.dep.
> Dependency@6ce5acd
>
> %spark.dep z.load("some-local-jar.jar")
> and
>
> import com.SuperClass
> // bla-bla
>
> val features = sc.sequenceFile[NullWritable, BytesWritable](path)
> // bla-bla
> }
>
> println(features)
>
> Can you give me any input, how can I debug it?
>
> exception in UI is:
> java.lang.NullPointerException at org.apache.zeppelin.spark.
> SparkInterpreter.open(SparkInterpreter.java:887) at org.apache.zeppelin.
> interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70) at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$
> InterpretJob.jobRun(RemoteInterpreterServer.java:489) at
> org.apache.zeppelin.scheduler.Job.run(Job.java:181) at
> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:47
>
> exception in interpreter log is:
>
> ERROR [2017-06-26 08:48:15,846] ({pool-2-thread-2}
> Utils.java[invokeMethod]:41) -
>
> java.lang.reflect.InvocationTargetException
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:57)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:39)
>
> at org.apache.zeppelin.spark.SparkInterpreter.interpret(
> SparkInterpreter.java:1008)
>
> at org.apache.zeppelin.spark.SparkInterpreter.open(
> SparkInterpreter.java:880)
>
> at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(
> LazyOpenInterpreter.java:70)
>
> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$
> InterpretJob.jobRun(RemoteInterpreterServer.java:489)
>
> at org.apache.zeppelin.scheduler.Job.run(Job.java:181)
>
> at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(
> FIFOScheduler.java:139)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: java.lang.NullPointerException
>
> at scala.reflect.internal.SymbolTable.exitingPhase(SymbolTable.scala:256)
>
> at scala.tools.nsc.interpreter.IMain$Request.x$20$lzycompute(
> IMain.scala:896)
>
> at scala.tools.nsc.interpreter.IMain$Request.x$20(IMain.scala:895)
>
> at scala.tools.nsc.interpreter.IMain$Request.headerPreamble$
> lzycompute(IMain.scala:895)
>
> at scala.tools.nsc.interpreter.IMain$Request.headerPreamble(
> IMain.scala:895)
>
> at scala.tools.nsc.interpreter.IMain$Request$Wrapper.
> preamble(IMain.scala:918)
>
> at scala.tools.nsc.interpreter.IMain$CodeAssembler$$anonfun$
> apply$23.apply(IMain.scala:1337)
>
> at scala.tools.nsc.interpreter.IMain$CodeAssembler$$anonfun$
> apply$23.apply(IMain.scala:1336)
>
> at scala.tools.nsc.util.package$.stringFromWriter(package.scala:64)
>
> at scala.tools.nsc.interpreter.IMain$CodeAssembler$class.
> apply(IMain.scala:1336)
>
> at scala.tools.nsc.interpreter.IMain$Request$Wrapper.apply(
> IMain.scala:908)
>
> at scala.tools.nsc.interpreter.IMain$Request.compile$
> lzycompute(IMain.scala:1002)
>
> at scala.tools.nsc.interpreter.IMain$Request.compile(IMain.scala:997)
>
> at scala.tools.nsc.interpreter.IMain.compile(IMain.scala:579)
>
> at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:567)
>
> at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
>
> ... 18 more
>
> ERROR [2017-06-26 08:48:15,847] ({pool-2-thread-2} Job.java[run]:186) -
> Job failed
>
> java.lang.NullPointerException
&g
Re: Is Zeppelin spark-version agnostic?
Awesome, thanks! вт, 27 июня 2017 г. в 13:47, Jeff Zhang <zjf...@gmail.com>: > > Zeppelin 0.7.2 support all these versions. You can create spark > interpreter for each version in interpreter setting page. And you need to > set the SPARK_HOME to the path where your spark is installed. > > > > Serega Sheypak <serega.shey...@gmail.com>于2017年6月27日周二 下午6:14写道： > >> Hi, can zeppelin spark interpreter support spark 1.6 / 2.0 / 2.1 >> I didn't find which spark versions are supported... >> >
Re: NPE in SparkInterpreter.java
It was my fault, I'm so sorry, I've recompiled zeppelin for scala 2.11 to
make it run with cloudera spark 2.0 and used scala 2.10 third party libs. I
replaced them with 2.11 versions and it started to work
вт, 27 июня 2017 г. в 9:52, Serega Sheypak <serega.shey...@gmail.com>:
> Hi, I'm stuck.
> I think I've localized the problem. I get non-obvious NPE when I try to
> add 3rd party jars to spark interpreter using any approach metioned in
> documentation.
> Zeppelin works fine for me when I just read text (no external dependencies
> needed.)
>
> 2017-06-26 23:29 GMT+02:00 Richard Xin <richardxin...@yahoo.com>:
>
>> just try to understand the comment in the code:
>> // Some case, scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call
>> throws an NPE
>>
>> what are the "Some case", it sounds like Zeppelin developers are not sure
>> how it happened and how to prevent it?
>>
>> I am trying to make a strong case for my company to switch from other
>> notebook application to Zeppelin, Zeppelin looks good and only this issue
>> concerns me.
>>
>> I'm looking forward for any insights, thanks.
>> On Monday, June 26, 2017, 11:56:45 AM PDT, Serega Sheypak <
>> serega.shey...@gmail.com> wrote:
>>
>>
>> Hi, I have more or less the same symptom
>>
>>
>> if (Utils.isScala2_10()) {
>> binder = (Map<String, Object>) getValue("_binder");
>> } else {
>> binder = (Map<String, Object>) getLastObject();
>> }
>> binder.put("sc", sc); // EXCEPTION HERE
>>
>>
>> java.lang.NullPointerException at
>> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:887)
>> at
>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:489)
>> at org.apache.zeppelin.scheduler.Job.run(Job.java:181) at
>> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> I'm going to check this one stuff:
>>
>> https://stackoverflow.com/questions/43289067/getting-nullpointerexception-when-running-spark-code-in-zeppelin-0-7-1
>>
>>
>> 2017-06-26 20:41 GMT+02:00 Richard Xin <richardxin...@yahoo.com>:
>>
>> Anyone?
>>
>>
>>
>> On Thursday, June 22, 2017, 3:00:01 PM PDT, Richard Xin <
>> richardxin...@yahoo.com> wrote:
>>
>>
>> apache/zeppelin
>> <https://github.com/apache/zeppelin/blob/master/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java#L1294>
>>
>> apache/zeppelin
>>
>> zeppelin - Mirror of Apache Zeppelin
>>
>> <https://github.com/apache/zeppelin/blob/master/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java#L1294>
>>
>> in the code I see // Some case, scala.tools.nsc.interpreter.
>> IMain$ReadEvalPrint.call throws an NPE
>>
>> We had issues that Zeppelin note errored out immediately without any hint
>> messages, and we are seeing this NPE in the spark_interpreter log (Zeppelin
>> 0.7.1 on AWS EMR cluster). can someone explain what's the likely cause for
>> this error and how to prevent it from happening again?
>>
>> Thanks,
>>
>>
>>
>
Re: NPE in SparkInterpreter.java
Hi, I'm stuck.
I think I've localized the problem. I get non-obvious NPE when I try to add
3rd party jars to spark interpreter using any approach metioned in
documentation.
Zeppelin works fine for me when I just read text (no external dependencies
needed.)
2017-06-26 23:29 GMT+02:00 Richard Xin <richardxin...@yahoo.com>:
> just try to understand the comment in the code:
> // Some case, scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call throws
> an NPE
>
> what are the "Some case", it sounds like Zeppelin developers are not sure
> how it happened and how to prevent it?
>
> I am trying to make a strong case for my company to switch from other
> notebook application to Zeppelin, Zeppelin looks good and only this issue
> concerns me.
>
> I'm looking forward for any insights, thanks.
> On Monday, June 26, 2017, 11:56:45 AM PDT, Serega Sheypak <
> serega.shey...@gmail.com> wrote:
>
>
> Hi, I have more or less the same symptom
>
>
> if (Utils.isScala2_10()) {
> binder = (Map<String, Object>) getValue("_binder");
> } else {
> binder = (Map<String, Object>) getLastObject();
> }
> binder.put("sc", sc); // EXCEPTION HERE
>
>
> java.lang.NullPointerException at org.apache.zeppelin.spark.
> SparkInterpreter.open(SparkInterpreter.java:887) at org.apache.zeppelin.
> interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70) at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$
> InterpretJob.jobRun(RemoteInterpreterServer.java:489) at
> org.apache.zeppelin.scheduler.Job.run(Job.java:181) at
> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262) at
> java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(
> ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.
> ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> I'm going to check this one stuff:
> https://stackoverflow.com/questions/43289067/getting-
> nullpointerexception-when-running-spark-code-in-zeppelin-0-7-1
>
>
> 2017-06-26 20:41 GMT+02:00 Richard Xin <richardxin...@yahoo.com>:
>
> Anyone?
>
>
>
> On Thursday, June 22, 2017, 3:00:01 PM PDT, Richard Xin <
> richardxin...@yahoo.com> wrote:
>
>
> apache/zeppelin
> <https://github.com/apache/zeppelin/blob/master/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java#L1294>
>
> apache/zeppelin
>
> zeppelin - Mirror of Apache Zeppelin
>
> <https://github.com/apache/zeppelin/blob/master/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java#L1294>
>
> in the code I see // Some case, scala.tools.nsc.interpreter.
> IMain$ReadEvalPrint.call throws an NPE
>
> We had issues that Zeppelin note errored out immediately without any hint
> messages, and we are seeing this NPE in the spark_interpreter log (Zeppelin
> 0.7.1 on AWS EMR cluster). can someone explain what's the likely cause for
> this error and how to prevent it from happening again?
>
> Thanks,
>
>
>
Can't run simple example with scala and spark SQL. Some non obvious syntax error in SQL
Here is my sample notebook:
%spark
val linesText = sc.textFile("hdfs://cluster/user/me/lines.txt")
case class Line(id:Long, firstField:String, secondField:String)
val lines = linesText.map{ line =>
val splitted = line.split(" ")
println("splitted => " + splitted)
Line(splitted(0).toLong, splitted(1), splitted(2))
}
lines.toDF().registerTempTable("lines")
%sql select firstField, secondField, count(1) from lines group by
firstField, secondField order by firstField, secondField
1. I can see that spark job was started on my YARN cluster.
2. It failed
UI shows exception. Can't understand what do I do wrong:
%sql select firstField, secondField, count(1) from lines group by
firstField, secondField order by firstField, secondField ^
3. There is suspicious output in zeppelin log:
INFO [2017-05-02 11:50:02,846] ({pool-2-thread-8}
SchedulerFactory.java[jobFinished]:137) - Job
paragraph_1493724118696_868476558 finished by scheduler
org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session1712472970
ERROR [2017-05-02 11:50:18,809] ({qtp1286783232-166}
NotebookServer.java[onMessage]:380) - Can't handle message
java.lang.NullPointerException
at
org.apache.zeppelin.socket.NotebookServer.addNewParagraphIfLastParagraphIsExecuted(NotebookServer.java:1713)
at
org.apache.zeppelin.socket.NotebookServer.persistAndExecuteSingleParagraph(NotebookServer.java:1741)
at
org.apache.zeppelin.socket.NotebookServer.runAllParagraphs(NotebookServer.java:1641)
at
org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:291)
at
org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:59)
at
org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
at
org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at
org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
at
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
at
org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
at
org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
INFO [2017-05-02 11:50:18,811] ({pool-2-thread-14}
SchedulerFactory.java[jobStarted]:131) - Job
paragraph_1493724118696_868476558 started by scheduler
org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session1712472970
INFO [2017-05-02 11:50:18,812] ({pool-2-thread-14}
Paragraph.java[jobRun]:363) - run paragraph 20170502-112158_458502255 using
spark org.apache.zeppelin.interpreter.LazyOpenInterpreter@24ca4045
WARN [2017-05-02 11:50:19,810] ({pool-2-thread-14}
NotebookServer.java[afterStatusChange]:2162) - Job
20170502-112158_458502255 is finished, status: ERROR, exception: null,
result: %text linesText: org.apache.spark.rdd.RDD[String] =
hdfs://path/to/my/file.txt MapPartitionsRDD[21] at textFile at :27
defined class Line
lines: org.apache.spark.rdd.RDD[Line] = MapPartitionsRDD[22] at map at
:31
warning: there was one deprecation warning; re-run with -deprecation for
details
:1: error: ';' expected but ',' found.
%sql select firstField, secondField, count(1) from lines group by
firstField, secondField order by firstField, secondField
4. Interpreter log is also confusing:
INFO [2017-05-02 11:41:52,706] ({pool-2-thread-2}
Logging.scala[logInfo]:54) - Warehouse location for Hive client (version
1.1.0) is file:/spark-warehouse
INFO [2017-05-02 11:41:52,983] ({pool-2-thread-2}
PerfLogger.java[PerfLogBegin]:122) -
INFO [2017-05-02 11:41:52,984] ({pool-2-thread-2}
HiveMetaStore.java[logInfo]:795) - 0: create_database:
Database(name:default, description:default database,
locationUri:file:/spark-warehouse, parameters:{})
INFO [2017-05-02 11:41:52,984] ({pool-2-thread-2}
HiveMetaStore.java[logAuditEvent]:388) - ugi=zblenessy
ip=unknown-ip-addr cmd=create_database:
Database(name:default, description:default database,
locationUri:file:/spark-warehouse, parameters:{})
ERROR [2017-05-02 11:41:52,992] ({pool-2-thread-2}
RetryingHMSHandler.java[invokeInternal]:189) -
Re: Can't run simple example with scala and spark SQL. Some non obvious syntax error in SQL
Just spark code works fine:
val linesText = sc.textFile("hdfs://my/file/onhdfs.txt")
case class Line(id:Long, firstField:String, secondField:String)
val lines = linesText.map{ line =>
val splitted = line.split(" ")
println("splitted => " + splitted)
Line(splitted(0).toLong, splitted(1), splitted(2))
}
lines.collect().foreach(println)
prints file contexts to UI. I have some trouble with sql...
2017-05-02 13:57 GMT+02:00 Serega Sheypak <serega.shey...@gmail.com>:
> Here is my sample notebook:
> %spark
> val linesText = sc.textFile("hdfs://cluster/user/me/lines.txt")
>
> case class Line(id:Long, firstField:String, secondField:String)
>
> val lines = linesText.map{ line =>
> val splitted = line.split(" ")
> println("splitted => " + splitted)
> Line(splitted(0).toLong, splitted(1), splitted(2))
> }
>
> lines.toDF().registerTempTable("lines")
>
> %sql select firstField, secondField, count(1) from lines group by
> firstField, secondField order by firstField, secondField
>
> 1. I can see that spark job was started on my YARN cluster.
> 2. It failed
> UI shows exception. Can't understand what do I do wrong:
> %sql select firstField, secondField, count(1) from lines group by
> firstField, secondField order by firstField, secondField ^
> 3. There is suspicious output in zeppelin log:
>
> INFO [2017-05-02 11:50:02,846] ({pool-2-thread-8}
> SchedulerFactory.java[jobFinished]:137)
> - Job paragraph_1493724118696_868476558 finished by scheduler
> org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_
> session1712472970
> ERROR [2017-05-02 11:50:18,809] ({qtp1286783232-166}
> NotebookServer.java[onMessage]:380) - Can't handle message
> java.lang.NullPointerException
> at org.apache.zeppelin.socket.NotebookServer.
> addNewParagraphIfLastParagraphIsExecuted(NotebookServer.java:1713)
> at org.apache.zeppelin.socket.NotebookServer.
> persistAndExecuteSingleParagraph(NotebookServer.java:1741)
> at org.apache.zeppelin.socket.NotebookServer.runAllParagraphs(
> NotebookServer.java:1641)
> at org.apache.zeppelin.socket.NotebookServer.onMessage(
> NotebookServer.java:291)
> at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(
> NotebookSocket.java:59)
> at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.
> onTextMessage(JettyListenerEventDriver.java:128)
> at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.
> messageComplete(SimpleTextMessage.java:69)
> at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.
> appendMessage(AbstractEventDriver.java:65)
> at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.
> onTextFrame(JettyListenerEventDriver.java:122)
> at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.
> incomingFrame(AbstractEventDriver.java:161)
> at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(
> WebSocketSession.java:309)
> at org.eclipse.jetty.websocket.common.extensions.
> ExtensionStack.incomingFrame(ExtensionStack.java:214)
> at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
> at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
> at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.
> readParse(AbstractWebSocketConnection.java:632)
> at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.
> onFillable(AbstractWebSocketConnection.java:480)
> at org.eclipse.jetty.io.AbstractConnection$2.run(
> AbstractConnection.java:544)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:635)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> INFO [2017-05-02 11:50:18,811] ({pool-2-thread-14}
> SchedulerFactory.java[jobStarted]:131)
> - Job paragraph_1493724118696_868476558 started by scheduler
> org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_
> session1712472970
> INFO [2017-05-02 11:50:18,812] ({pool-2-thread-14}
> Paragraph.java[jobRun]:363) - run paragraph 20170502-112158_458502255 using
> spark org.apache.zeppelin.interpreter.LazyOpenInterpreter@24ca4045
> WARN [2017-05-02 11:50:19,810] ({pool-2-thread-14}
> NotebookServer.java[afterStatusChange]:2162)
> - Job 20170502-112158_458502255 is finished, status: ERROR, exception:
> null, result: %text linesText: org.apache.spark.rdd.RDD[String] =
> hdfs://path/to/my/file.txt MapPartitionsRDD[21] at textFile at :27
> defined class Line
> lines: org.apache.spark.rdd.RDD[Line] = MapPartitionsRDD[22] at map at
> :31
> warning: there was one deprecation warning; re-run with -deprecation
Is Zeppelin spark-version agnostic?
Hi, can zeppelin spark interpreter support spark 1.6 / 2.0 / 2.1 I didn't find which spark versions are supported...
sql paragraph doesn't see my 3rd party jars
Hi, I'm trying to use spark and sql paragraphs with 3rd party jars added to
spark interpreter configuration.
My spark code works fine.
My sql paragraph fails with class not found exception
%sql
create external table MY_TABLE row format serde 'com.my.MyAvroSerde'
with serdeproperties ('serialization.class'='com.my.ContainerProto') stored
as inputformat 'com.my.ProtoAvroFileFormat'
LOCATION 'hdfs://my/data'
Exception:
MetaException(message:org.apache.hadoop.hive.serde2.SerDeException
java.lang.ClassNotFoundException: Class com.my.ContainerProto not found)
It's confusing since spark paragraph works well with
code
import com.my.ContainerProto
// bla-bla
rdd.map{bytes => ContainerProto.fromBytes(bytes)}
Code executed and result produced. Why sql paragraph doesn't see my 3rd
party jars?
Can't use newly create intepreter, Zeppelin 0.7.2 paragraph_XXX's Interpreter spark_my not found
Hi, I've create spark interpreter with name spark_my I was able to restart it and zeppelin shows "green" marker near it. spark_my is copy-paste of default spark intepreter with few changes. I try to use spark_my in notebook: %spark_my import java.nio.ByteBuffer more code Zeppelin shows exception when I try to run paragraph. What do I do wrong? org.apache.zeppelin.interpreter.InterpreterException: paragraph_1508587957017_510563027's Interpreter spark_my not found at org.apache.zeppelin.notebook.Note.run(Note.java:605) at org.apache.zeppelin.socket.NotebookServer.persistAndExecuteSingleParagraph(NotebookServer.java:1641) at org.apache.zeppelin.socket.NotebookServer.runParagraph(NotebookServer.java:1615) at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:265) at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:59) at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128) at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69) at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65) at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122) at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161) at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309) at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214) at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220) at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:748)
How does user user jar conflict resolved in spark interpreter?
Hi zeppelin users!
I have the question about dependencies users are using while running
notebooks using spark interpreter.
Imagine I have configured spark intepreter.
Two users write their spark notebooks.
the first user does
z.load("com:best-it-company:0.1")
the second one user adds to his notebook:
z.load("com:best-it-company:0.2")
Then they start to execute two notebooks concurrently.
What will happen to dependencies?
They have same classes... will spark isolate 0.1 version from 0.2 version
somehow?
Re: Can't use newly create intepreter, Zeppelin 0.7.2 paragraph_XXX's Interpreter spark_my not found
Ah.. thanks :) That wasn't easy :) 2017-10-23 13:13 GMT+02:00 Jeff Zhang <zjf...@gmail.com>: > > But the error message here is not so user friendly, created ticket for > this https://issues.apache.org/jira/browse/ZEPPELIN-3005 > > > > Jeff Zhang <zjf...@gmail.com>于2017年10月23日周一 下午7:08写道： > >> >> Please bind this interpreter to your note first. >> >> Serega Sheypak <serega.shey...@gmail.com>于2017年10月23日周一 下午6:14写道： >> >>> Hi, I've create spark interpreter with name spark_my >>> I was able to restart it and zeppelin shows "green" marker near it. >>> spark_my is copy-paste of default spark intepreter with few changes. >>> >>> I try to use spark_my in notebook: >>> >>> %spark_my >>> import java.nio.ByteBuffer >>> more code >>> >>> Zeppelin shows exception when I try to run paragraph. >>> What do I do wrong? >>> >>> org.apache.zeppelin.interpreter.InterpreterException: >>> paragraph_1508587957017_510563027's Interpreter spark_my not found at >>> org.apache.zeppelin.notebook.Note.run(Note.java:605) at >>> org.apache.zeppelin.socket.NotebookServer.persistAndExecuteSingleParagraph(NotebookServer.java:1641) >>> at >>> org.apache.zeppelin.socket.NotebookServer.runParagraph(NotebookServer.java:1615) >>> at >>> org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:265) >>> at >>> org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:59) >>> at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver. >>> onTextMessage(JettyListenerEventDriver.java:128) at >>> org.eclipse.jetty.websocket.common.message.SimpleTextMessage. >>> messageComplete(SimpleTextMessage.java:69) at >>> org.eclipse.jetty.websocket.common.events.AbstractEventDriver. >>> appendMessage(AbstractEventDriver.java:65) at >>> org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver. >>> onTextFrame(JettyListenerEventDriver.java:122) at >>> org.eclipse.jetty.websocket.common.events.AbstractEventDriver. >>> incomingFrame(AbstractEventDriver.java:161) at >>> org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309) >>> at org.eclipse.jetty.websocket.common.extensions. >>> ExtensionStack.incomingFrame(ExtensionStack.java:214) at >>> org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220) >>> at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258) at >>> org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection. >>> readParse(AbstractWebSocketConnection.java:632) at >>> org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection. >>> onFillable(AbstractWebSocketConnection.java:480) at org.eclipse.jetty.io >>> .AbstractConnection$2.run(AbstractConnection.java:544) at >>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) >>> at >>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) >>> at java.lang.Thread.run(Thread.java:748) >>> >>
