Re: Zeppelin unable to respond after some time

[小野圭二]

I am facing on the same issue now.

2017-02-17 18:25 GMT+09:00 RUSHIKESH RAUT :

> Hi all,
>
> I am facing a issue while using Zeppelin. I am trying to load some
> data(not that big data) into Zeppelin and then build some visualization on
> it. The problem is that when I try to run the code first time it's working
> but after some time the same code doesn't work. It remains in running state
> on gui, but no logs are generated in Zeppelin logs. Also all further tasks
> are hanging in pending state.
> As soon as I restart  Zeppelin it works. So I am guessing it's some memory
> issue. I have read that Zeppelin stores the data in memory so it is
> possible that it runs out of memory after some time.
> How do I debug this issue? How much is the default memory that Zeppelin
> takes at start? Also is there any way that I can run Zeppelin with
> specified memory so that I can start the process with more memory. Because
> it doesn't make sense to restart Zeppelin after every half hour
>
> Thanks,
> Rushikesh Raut
>

visualize sql generator

[小野圭二]

Hi,

I am beginner in Zeppelin, but i am getting impressed by this marvelous
front end software.
And to introduce it into us, i am searching a sort of plugin that createsql
code by visual tool, like making ER figure in DB.

I appreciate If you have some candidates one, whichever plugin type or hard
coded.

Thx

Keiji

Are there any way a notebook to be black box?

[小野圭二]

Hi,

I wonder if this mail title would be realize.
I mean, when I do not want to be changed my notebook programing code, but
just give a way to change the query of programing by dynamic forms.

1. are there way to hide all code in the notebook?
2. are there rock system to prevent changing the code in notebook?

If No.1 would be available, it makes easy to provide commercial codes
and/or secret codes to widely.
In fact, from the view of code writers, especially in the case of
commercial products, they do not want to provide any source code to their
customers definitely. Another hand, it is costly to force R or Python
programer to write Java or Scala program to change their naked codes to
binary code.

I have a image Script programers write and test their
R/Python/Spark-Shell/SQL codes on notebooks and be it to
BLACKBOX-NOTEBOOK, then upload it to github/svn where endusers who want
to use those notebooks could access by themselves. The endusers down load
those BLACKBOX-NOTEBOOK and import into their Zeppelin and it works fine.
Sound like happy the both.

No.2 seems like easier than No.1, because the issue is depend on
authorization. But it is incomplete, because the codes reveal to the
endusers.

I saw zepl.com as well.
Seems like Zeppelin and ZELP are aimed to help collaboration between
analysts. That is good. And if adding the above NO.1 feature into it, they
will be the best, i think.

Do not You think so?

-Ono

Re: Are there any way a notebook to be black box?

[小野圭二]

Hi there,
I created this issue on JIRA as ZEPPELIN-2721.

-Ono

<https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail>
ウイルス
フリー｡ www.avast.com
<https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

2017-07-05 16:10 GMT+09:00 Khalid Huseynov <khalid...@gmail.com>:

> Hello Ono,
>
> Thanks for reporting your use case scenario and it's indeed valid that
> users would like to share their notes across Zeppelin instances with their
> original permissions. It definitely deserves an issue and should be handled
> toward maturity of Zeppelin (e.g. in v. 0.9~1.0). Possible solutions could
> be including permission info inside of note, or some hash identifying the
> original creator. Also as you mentioned, some of scenarios possibly could
> be handled in zepl.com.
>
> On Wed, Jul 5, 2017 at 1:36 PM, Hyung Sung Shim <hss...@nflabs.com> wrote:
>
>> Thank you very much to describe in detail.
>> I totally understand your question now.
>> There is no way to solve your question for now but it's really
>> interesting issue.
>> so Could you make a Jira issue for this with your experience?
>> Thanks!
>>
>> 2017-07-05 10:16 GMT+09:00 小野圭二 <onoke...@gmail.com>:
>>
>>> Thx for your reply, Hyung.
>>>
>>> Thank you for introducing some docs, but i knew them already.And my
>>> point was each notebook execute but hide/invisible the code.
>>> The zeppelin auth system manage every notebooks at once. That feature
>>> effect in the case of all user share same zeppelin system, like cloud
>>> computing.
>>>
>>> Against it, how about in the case of just sharing notebook?
>>> For example i set strict roles  into my notebook, and share it with
>>> someone who has their zeppelin. After they download and import my strict
>>> auth notebook into their zeppelin, they can arrange my roles setting by
>>> themselves. In this case, they can see my hided code on their zeppelin.
>>>
>>> I do not misunderstand about the zeppelin auth system, do i ?
>>>
>>> To prevent this issue, may need another authenticate/management feature
>>> on Zeppelin, this is my point. :^)
>>>
>>> What for it, imagine such a case.
>>> In the case of selling my big data analyze application through zeppelin
>>> eco-system, i write and test my code on my zeppelin. Then export it and
>>> upload to sharing site.
>>> May someone see my sharing notebook on the site, then they download and
>>> import it into their zeppelin, and execute it,
>>>
>>> Ok that is good, fantastic, viva Zeppelin, if i am a volunteer
>>> programmer.
>>>
>>> But i would like to get some revenue from my code if someone go down my
>>> analyzing, hopefully.
>>> In this case, i just want to provide the result of analyzing, not my
>>> algorithm.
>>> Even my client share my notebook on his zeppelin, my code is protected
>>> by independent authentication, encryption and so on.
>>>
>>> This is my point. :-)
>>> Sorry for getting longer.
>>>
>>> -Ono
>>>
>>>
>>> 2017-07-04 23:35 GMT+09:00 Hyung Sung Shim <hss...@nflabs.com>:
>>>
>>>> Hello onokeiji.
>>>>
>>>> You can refer http://zeppelin.apache.org/docs/0.7.2/quickstart/explo
>>>> rezeppelinui.html#note-toolbar if you want to hide code in the
>>>> notebook.
>>>> and http://zeppelin.apache.org/docs/0.7.2/security/notebook_
>>>> authorization.html#authorization-setting is relate to lock code your
>>>> note.
>>>>
>>>> Hope this helps.
>>>>
>>>>
>>>> 2017-07-04 22:05 GMT+09:00 小野圭二 <onoke...@gmail.com>:
>>>>
>>>>> Hey, how do you think about my idea of BLACKBOX-NOTEBOOK?
>>>>>
>>>>> -Ono.
>>>>>
>>>>>
>>>>> <https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail>
>>>>>  ウイルス
>>>>> フリー｡ www.avast.com
>>>>> <https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail>
>>>>> <#m_-2740405082895278623_m_7229949389285741499_m_-8962903669315512691_m_3961825290808015369_m_-7798815867934407536_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>>>
>>>>> 2017-06-30 10:38 GMT+09:00 小野圭二 <onoke...@gmail.com

Re: Are there any way a notebook to be black box?

[小野圭二]

Thx for your reply, Hyung.

Thank you for introducing some docs, but i knew them already.And my point
was each notebook execute but hide/invisible the code.
The zeppelin auth system manage every notebooks at once. That feature
effect in the case of all user share same zeppelin system, like cloud
computing.

Against it, how about in the case of just sharing notebook?
For example i set strict roles  into my notebook, and share it with someone
who has their zeppelin. After they download and import my strict auth
notebook into their zeppelin, they can arrange my roles setting by
themselves. In this case, they can see my hided code on their zeppelin.

I do not misunderstand about the zeppelin auth system, do i ?

To prevent this issue, may need another authenticate/management feature on
Zeppelin, this is my point. :^)

What for it, imagine such a case.
In the case of selling my big data analyze application through zeppelin
eco-system, i write and test my code on my zeppelin. Then export it and
upload to sharing site.
May someone see my sharing notebook on the site, then they download and
import it into their zeppelin, and execute it,

Ok that is good, fantastic, viva Zeppelin, if i am a volunteer programmer.

But i would like to get some revenue from my code if someone go down my
analyzing, hopefully.
In this case, i just want to provide the result of analyzing, not my
algorithm.
Even my client share my notebook on his zeppelin, my code is protected by
independent authentication, encryption and so on.

This is my point. :-)
Sorry for getting longer.

-Ono


2017-07-04 23:35 GMT+09:00 Hyung Sung Shim <hss...@nflabs.com>:

> Hello onokeiji.
>
> You can refer http://zeppelin.apache.org/docs/0.7.2/quickstart/
> explorezeppelinui.html#note-toolbar if you want to hide code in the
> notebook.
> and http://zeppelin.apache.org/docs/0.7.2/security/
> notebook_authorization.html#authorization-setting is relate to lock code
> your note.
>
> Hope this helps.
>
>
> 2017-07-04 22:05 GMT+09:00 小野圭二 <onoke...@gmail.com>:
>
>> Hey, how do you think about my idea of BLACKBOX-NOTEBOOK?
>>
>> -Ono.
>>
>>
>> <https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail>
>>  ウイルス
>> フリー｡ www.avast.com
>> <https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail>
>> <#m_3961825290808015369_m_-7798815867934407536_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>
>> 2017-06-30 10:38 GMT+09:00 小野圭二 <onoke...@gmail.com>:
>>
>>> Hi,
>>>
>>> I wonder if this mail title would be realize.
>>> I mean, when I do not want to be changed my notebook programing code,
>>> but just give a way to change the query of programing by dynamic forms.
>>>
>>> 1. are there way to hide all code in the notebook?
>>> 2. are there rock system to prevent changing the code in notebook?
>>>
>>> If No.1 would be available, it makes easy to provide commercial codes
>>> and/or secret codes to widely.
>>> In fact, from the view of code writers, especially in the case of
>>> commercial products, they do not want to provide any source code to their
>>> customers definitely. Another hand, it is costly to force R or Python
>>> programer to write Java or Scala program to change their naked codes to
>>> binary code.
>>>
>>> I have a image Script programers write and test their
>>> R/Python/Spark-Shell/SQL codes on notebooks and be it to
>>> BLACKBOX-NOTEBOOK, then upload it to github/svn where endusers who want
>>> to use those notebooks could access by themselves. The endusers down load
>>> those BLACKBOX-NOTEBOOK and import into their Zeppelin and it works fine.
>>> Sound like happy the both.
>>>
>>> No.2 seems like easier than No.1, because the issue is depend on
>>> authorization. But it is incomplete, because the codes reveal to the
>>> endusers.
>>>
>>> I saw zepl.com as well.
>>> Seems like Zeppelin and ZELP are aimed to help collaboration between
>>> analysts. That is good. And if adding the above NO.1 feature into it, they
>>> will be the best, i think.
>>>
>>> Do not You think so?
>>>
>>> -Ono
>>>
>>
>>
>

How to write a style of printout in a application?

[小野圭二]

Hi,

I am super beginner on here, but this is an amazing project, thx.
And not sure all of them, so am sorry if this is a little bit silly
question.

I saw how to write some code on Zeppelin, at the same time, the same code
can run on it as a application program like this.

[in notebook]
% hogehoge.scala

But in the case of this type, how should i write the code in the
application to use variety figures and so on?
I could not find out the relation between a application / code and display
procedure in the docs. :-b

Any info and advices thank.

-Keiji

Re: Zeppelin can let users write Java code?

[小野圭二]

I am not sure but could not Zeppelin run a binary program on notebook?
You know Spark support Java program and it could run on it. I wonder
Zeppelin simulate Spark console environment or not.

-Keiji

2017-06-25 5:11 GMT+09:00 Alexander Bezzubov :

> It's not very well known, but Apache Zeppelin already has one Java
> interpreter.
> If you look at Apache Beam one [1]  - underneath [2] it has quite generic
> facilities to interpreter java code, so you can use it for anything, not
> just Java API of Beam.
>
> But Felix's suggestion on using JShell in Java9 as soon as it's released
> definitely sounds as a more mature solution.
>
>
>  1. http://zeppelin.apache.org/docs/0.7.2/interpreter/beam.html
>  2. https://github.com/apache/zeppelin/tree/master/beam
>
> --
> Alex
>
> On Sat, Jun 24, 2017 at 9:26 PM, Felix Cheung 
> wrote:
>
>> There has been discussions on leveraging a Java REPL project to create
>> such Java interpreter.
>>
>> Also Java 9 has JShell which seems to target a release in the next few
>> months.
>>
>> --
>> *From:* Yi Lyu 
>> *Sent:* Saturday, June 24, 2017 9:17:32 AM
>> *To:* users
>> *Subject:* Zeppelin can let users write Java code?
>>
>> hi Team
>> In our company, there are a lot of Java engineers, they are not good
>> at Scala, may I ask, Zeppelin can let users write Java code, and then run
>> them? Currently, I have tried Scala, Python, and can run it.
>> and, Is there a plan to develop Java Interpreter ?
>> --
>>  Yi Lyu
>>
>>
>

Re: How to write a style of printout in a application?

[小野圭二]

Thank you for your rapid and kindly reply, Park. :-)

To make clear a little bit.
Let's look in conf/interpreter.json on ver.0.7.0, focusing on Spark setting
to make simple now, they have 'spark,sql, pyspark,r, scala' in it.
I guess they are for executing those kinds programs as script on notebook.
And i wonder a binary program could run like script on notebook?

[for example]
>%spark main<- this is main class that is described in hogehoge.scala
or
>%spark hogehoge.jar <-this jar is java binary program

Any advance great.

-Keiji


2017-06-26 12:55 GMT+09:00 Park Hoon <1am...@gmail.com>:

> Hi,
>
> `%something` is called *interpreter* in Zeppelin.
>
> Here are few docs you can refer to write new interpreters.
>
> - http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/development/
> writingzeppelininterpreter.html
>
> You can write interpreters work on the browser using javascript.
>
> It's Spell which has few limitations but also provides good features. (e.g
> can be display system as well)
>
> - http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/development/
> writingzeppelinspell.html
>
>
>
> Thanks.
>
> On Fri, Jun 23, 2017 at 8:27 AM, 小野圭二 <onoke...@gmail.com> wrote:
>
>> Hi,
>>
>> I am super beginner on here, but this is an amazing project, thx.
>> And not sure all of them, so am sorry if this is a little bit silly
>> question.
>>
>> I saw how to write some code on Zeppelin, at the same time, the same code
>> can run on it as a application program like this.
>>
>> [in notebook]
>> % hogehoge.scala
>>
>> But in the case of this type, how should i write the code in the
>> application to use variety figures and so on?
>> I could not find out the relation between a application / code and
>> display procedure in the docs. :-b
>>
>> Any info and advices thank.
>>
>> -Keiji
>>
>>
>>
>

How to execute spark-submit on Note

[小野圭二]

Hi all,

I searched this topic on the archive of ml, but still could not find out
the solution clearly.
So i have tried to post this again(maybe).

I am using ver 0.8.0, and have installed spark 2.2 on the other path, just
for checking my test program.
Then i wrote a quite simple sample python code to check the how to.

1. the code works fine on a note in Zeppelin
2. the same code but added the initialize code for SparkContext in it works
fine on the Spark by using 'spark-submit'.
3. tried to execute "2" from a note in Zeppelin with the following script.
yes, "spark" interpreter has been implemented in the note.
then on the note,
%spark-submit 
  -> interpreter not found error
4.I have arranged 'SPARK_SUBMIT_OPTIONS' in zeppelin-env.sh order by the doc
ex. export SPARK_SUBMIT_OPTIONS='--packages
com.databricks:spark-csv_2.10:1.2.0'
5. then running
 %spark-submit 
  -> interpreter not found error  (as same as "3")

How can i use spark-submit from a note?
Any advice thanks.

-Keiji

Re: How to execute spark-submit on Note

[小野圭二]

Hi Dave,

Thank You for your suggestion.
It worked fine order by my expectation so far.
I did not know "%sh" could use like that.

Anyhow, i would like to explain why i would like to execute "spark-submit"
in a note, to be clear your wondering.
Yes, i know the basic instruction of Zeppelin as You explained to me in
your reply, Dave.
So, now, i tried to find the prospect of the environment of execution in
Zeppelin.
That mean, we were considering how to deliver our programs to users widely
after we made a program with collaboration on Zeppelin. In this case, we
might do not want to disclose our source code to them, but want to keep the
execution environment for rejecting any unnecessary issues.
Now i succeeded with a script code. Next will try to run a binary one.

That was the reason why, i posted this question into ML.
And i asked similar but another solution into JIRA,(#2721)

Once again, thank You Dave.

-Keiji


2017-10-03 19:12 GMT+09:00 David Howell <david.how...@zipmoney.com.au>:

> Hi Keiji,
>
>
>
> In the paragraph you would write:
>
> %sh
>
> spark-submit myapp.jar ...
>
>
>
> The %sh interpreter is a shell, and runs as the zeppelin service user with
> whatever permissions it has. You can run any shell commands in it.
>
>
>
> Although, this is a fairly strange way to run zeppelin so I’m not really
> sure that is what you want.
>
>
>
> You can just use the %spark.pyspark interpreter and write your python
> spark code in there. The spark interpreters in Zeppelin already create the
> Spark Context for you, as well as sqlContext and spark session. These are
> available as sc, sqlContext and spark. If you have a program that is ready
> for spark submit, I would use some other tool to schedule and run it, like
> cron, oozie, NiFi, Luigi, Airflow etc. Or if you want to run manually just
> use spark submit from the shell directly or ssh.
>
>
>
>
>
> Dave
>
>
>
> *From: *小野圭二 <onoke...@gmail.com>
> *Sent: *Tuesday, 3 October 2017 8:43 PM
> *To: *users@zeppelin.apache.org
> *Subject: *Re: How to execute spark-submit on Note
>
>
> Thank you for your quick reply again, Jeff.
>
> Yes i know the difference of the execution environment between "%sh" and
> ">spark-submit".
> And my question was "how to execute spark-submit as shell interpreter".
> That mean, i am searching how to execute a binary program from a note of
> zeppelin.
> This time it has been limited on Spark.
>
> Seems like Zeppelin have several procedure to execute Spark shell, like
> spark.pyspark, spark.sql
> So how to do "spark-submit" was my wondering.
>
> I am sorry for bothering Your time, but at the same time, i am appreciated
> if You get my wondering clearly, and show me some tips.
>
> -Keiji
>
>
> 2017-10-03 18:30 GMT+09:00 Jeff Zhang <zjf...@gmail.com>:
>
>> %sh is shell interpreter, you can run spark-submit just as you run it in
>> shell terminal.
>>
>> 小野圭二 <onoke...@gmail.com>于2017年10月3日周二 下午4:58写道：
>>
>>> Thank you for your reply, Jeff
>>>
>>> "%sh" ?
>>> "sh" seems like request something execution code.
>>> I tried "%sh", then
>>>
>>> %sh 
>>>   %sh bash: : no permission
>>>
>>> I made binary file from .py to .pyc, but the answer was as same.
>>> I am sorry seems like doubting you, but Is "%sh" the resolution?
>>>
>>> -Keiji
>>>
>>> 2017-10-03 17:35 GMT+09:00 Jianfeng (Jeff) Zhang <jzh...@hortonworks.com
>>> >:
>>>
>>>>
>>>> I am surprised why would you use %spark-submit, there’s no document
>>>> about %spark-submit.   If you want to use spark-submit in zeppelin, then
>>>> you could use %sh
>>>>
>>>>
>>>> Best Regard,
>>>> Jeff Zhang
>>>>
>>>>
>>>> From: 小野圭二 <onoke...@gmail.com>
>>>> Reply-To: "users@zeppelin.apache.org" <users@zeppelin.apache.org>
>>>> Date: Tuesday, October 3, 2017 at 12:49 PM
>>>> To: "users@zeppelin.apache.org" <users@zeppelin.apache.org>
>>>> Subject: How to execute spark-submit on Note
>>>>
>>>> Hi all,
>>>>
>>>> I searched this topic on the archive of ml, but still could not find
>>>> out the solution clearly.
>>>> So i have tried to post this again(maybe).
>>>>
>>>> I am using ver 0.8.0, and have installed spark 2.2 on the other path,
>>>> just for checking my test program.
&g

Re: How to execute spark-submit on Note

[小野圭二]

Thank You for your information, Lucas.
Sound interesting and good tip for how to deliver it.
I think this tip should be up to Zeppelin wiki if there is it. :-)
And i should check the Databricks notebook anyhow.

-Keiji

2017-10-04 17:20 GMT+09:00 Partridge, Lucas (GE Aviation) <
lucas.partri...@ge.com>:

> “we were considering how to deliver our programs to users widely after we
> made a program with collaboration on Zeppelin”
>
>
>
> - This is a common question/use case in my experience with Zeppelin: “How
> do we roll out code to everyone once it’s been prototyped in Zeppelin?”.
> Our approach is to package it up in jars or Python packages and make them
> available on the environment. Then users can just import it like any other
> code in their own Zeppelin %spark or %pyspark paragraphs.  No %sh is
> required.  Other notebook-based environments like Databricks make this
> packaging and importing of libraries part of their UI.
>
>
>
> Thanks, Lucas.
>
>
>
> *From:* 小野圭二 [mailto:onoke...@gmail.com]
> *Sent:* 04 October 2017 02:24
> *To:* users@zeppelin.apache.org
> *Subject:* EXT: Re: How to execute spark-submit on Note
>
>
>
> Hi Dave,
>
>
>
> Thank You for your suggestion.
>
> It worked fine order by my expectation so far.
>
> I did not know "%sh" could use like that.
>
>
>
> Anyhow, i would like to explain why i would like to execute "spark-submit"
> in a note, to be clear your wondering.
>
> Yes, i know the basic instruction of Zeppelin as You explained to me in
> your reply, Dave.
>
> So, now, i tried to find the prospect of the environment of execution in
> Zeppelin.
>
> That mean, we were considering how to deliver our programs to users widely
> after we made a program with collaboration on Zeppelin. In this case, we
> might do not want to disclose our source code to them, but want to keep the
> execution environment for rejecting any unnecessary issues.
>
> Now i succeeded with a script code. Next will try to run a binary one.
>
>
>
> That was the reason why, i posted this question into ML.
>
> And i asked similar but another solution into JIRA,(#2721)
>
>
>
> Once again, thank You Dave.
>
>
>
> -Keiji
>
>
>
>
>
> 2017-10-03 19:12 GMT+09:00 David Howell <david.how...@zipmoney.com.au>:
>
> Hi Keiji,
>
>
>
> In the paragraph you would write:
>
> %sh
>
> spark-submit myapp.jar ...
>
>
>
> The %sh interpreter is a shell, and runs as the zeppelin service user with
> whatever permissions it has. You can run any shell commands in it.
>
>
>
> Although, this is a fairly strange way to run zeppelin so I’m not really
> sure that is what you want.
>
>
>
> You can just use the %spark.pyspark interpreter and write your python
> spark code in there. The spark interpreters in Zeppelin already create the
> Spark Context for you, as well as sqlContext and spark session. These are
> available as sc, sqlContext and spark. If you have a program that is ready
> for spark submit, I would use some other tool to schedule and run it, like
> cron, oozie, NiFi, Luigi, Airflow etc. Or if you want to run manually just
> use spark submit from the shell directly or ssh.
>
>
>
>
>
> Dave
>
>
>
> *From: *小野圭二 <onoke...@gmail.com>
> *Sent: *Tuesday, 3 October 2017 8:43 PM
> *To: *users@zeppelin.apache.org
> *Subject: *Re: How to execute spark-submit on Note
>
>
>
> Thank you for your quick reply again, Jeff.
>
>
>
> Yes i know the difference of the execution environment between "%sh" and
> ">spark-submit".
>
> And my question was "how to execute spark-submit as shell interpreter".
>
> That mean, i am searching how to execute a binary program from a note of
> zeppelin.
>
> This time it has been limited on Spark.
>
>
>
> Seems like Zeppelin have several procedure to execute Spark shell, like
> spark.pyspark, spark.sql
>
> So how to do "spark-submit" was my wondering.
>
>
>
> I am sorry for bothering Your time, but at the same time, i am appreciated
> if You get my wondering clearly, and show me some tips.
>
>
>
> -Keiji
>
>
>
>
>
> 2017-10-03 18:30 GMT+09:00 Jeff Zhang <zjf...@gmail.com>:
>
> %sh is shell interpreter, you can run spark-submit just as you run it in
> shell terminal.
>
>
>
> 小野圭二 <onoke...@gmail.com>于2017年10月3日周二 下午4:58写道：
>
> Thank you for your reply, Jeff
>
>
>
> "%sh" ?
>
> "sh" seems like request something execution code.
>
> I tried "%sh", then
>
>
>
> %sh 
>
>   

How to initialize ZeppelinContext in a stand alone program

[小野圭二]

Hi,

I am sure Zeppelin manage printouts of a program to a note.
Then i would like to know how to initialize ZeppelinContext in python/scala
code like Spark?

In the case of SparkContext, it is clear like that,
[python]
from pyspark import SparkConf, SparkContext
conf = SparkConf().setAppName("...").setMaster("...")
sc = SparkContext(conf=conf)

So i wonder ZeppelinContext could be called like above?

Any advice is welcome

-Keiji

Re: How to execute spark-submit on Note

[小野圭二]

Thank you for your reply, Jeff

"%sh" ?
"sh" seems like request something execution code.
I tried "%sh", then

%sh 
  %sh bash: : no permission

I made binary file from .py to .pyc, but the answer was as same.
I am sorry seems like doubting you, but Is "%sh" the resolution?

-Keiji

2017-10-03 17:35 GMT+09:00 Jianfeng (Jeff) Zhang <jzh...@hortonworks.com>:

>
> I am surprised why would you use %spark-submit, there’s no document about
> %spark-submit.   If you want to use spark-submit in zeppelin, then you
> could use %sh
>
>
> Best Regard,
> Jeff Zhang
>
>
> From: 小野圭二 <onoke...@gmail.com>
> Reply-To: "users@zeppelin.apache.org" <users@zeppelin.apache.org>
> Date: Tuesday, October 3, 2017 at 12:49 PM
> To: "users@zeppelin.apache.org" <users@zeppelin.apache.org>
> Subject: How to execute spark-submit on Note
>
> Hi all,
>
> I searched this topic on the archive of ml, but still could not find out
> the solution clearly.
> So i have tried to post this again(maybe).
>
> I am using ver 0.8.0, and have installed spark 2.2 on the other path, just
> for checking my test program.
> Then i wrote a quite simple sample python code to check the how to.
>
> 1. the code works fine on a note in Zeppelin
> 2. the same code but added the initialize code for SparkContext in it
> works fine on the Spark by using 'spark-submit'.
> 3. tried to execute "2" from a note in Zeppelin with the following script.
> yes, "spark" interpreter has been implemented in the note.
> then on the note,
> %spark-submit 
>   -> interpreter not found error
> 4.I have arranged 'SPARK_SUBMIT_OPTIONS' in zeppelin-env.sh order by the
> doc
> ex. export SPARK_SUBMIT_OPTIONS='--packages
> com.databricks:spark-csv_2.10:1.2.0'
> 5. then running
>  %spark-submit 
>   -> interpreter not found error  (as same as "3")
>
> How can i use spark-submit from a note?
> Any advice thanks.
>
> -Keiji
>

Re: How to execute spark-submit on Note

[小野圭二]

Thank you for your quick reply again, Jeff.

Yes i know the difference of the execution environment between "%sh" and
">spark-submit".
And my question was "how to execute spark-submit as shell interpreter".
That mean, i am searching how to execute a binary program from a note of
zeppelin.
This time it has been limited on Spark.

Seems like Zeppelin have several procedure to execute Spark shell, like
spark.pyspark, spark.sql
So how to do "spark-submit" was my wondering.

I am sorry for bothering Your time, but at the same time, i am appreciated
if You get my wondering clearly, and show me some tips.

-Keiji


2017-10-03 18:30 GMT+09:00 Jeff Zhang <zjf...@gmail.com>:

> %sh is shell interpreter, you can run spark-submit just as you run it in
> shell terminal.
>
> 小野圭二 <onoke...@gmail.com>于2017年10月3日周二 下午4:58写道：
>
>> Thank you for your reply, Jeff
>>
>> "%sh" ?
>> "sh" seems like request something execution code.
>> I tried "%sh", then
>>
>> %sh 
>>   %sh bash: : no permission
>>
>> I made binary file from .py to .pyc, but the answer was as same.
>> I am sorry seems like doubting you, but Is "%sh" the resolution?
>>
>> -Keiji
>>
>> 2017-10-03 17:35 GMT+09:00 Jianfeng (Jeff) Zhang <jzh...@hortonworks.com>
>> :
>>
>>>
>>> I am surprised why would you use %spark-submit, there’s no document
>>> about %spark-submit.   If you want to use spark-submit in zeppelin, then
>>> you could use %sh
>>>
>>>
>>> Best Regard,
>>> Jeff Zhang
>>>
>>>
>>> From: 小野圭二 <onoke...@gmail.com>
>>> Reply-To: "users@zeppelin.apache.org" <users@zeppelin.apache.org>
>>> Date: Tuesday, October 3, 2017 at 12:49 PM
>>> To: "users@zeppelin.apache.org" <users@zeppelin.apache.org>
>>> Subject: How to execute spark-submit on Note
>>>
>>> Hi all,
>>>
>>> I searched this topic on the archive of ml, but still could not find out
>>> the solution clearly.
>>> So i have tried to post this again(maybe).
>>>
>>> I am using ver 0.8.0, and have installed spark 2.2 on the other path,
>>> just for checking my test program.
>>> Then i wrote a quite simple sample python code to check the how to.
>>>
>>> 1. the code works fine on a note in Zeppelin
>>> 2. the same code but added the initialize code for SparkContext in it
>>> works fine on the Spark by using 'spark-submit'.
>>> 3. tried to execute "2" from a note in Zeppelin with the following
>>> script.
>>> yes, "spark" interpreter has been implemented in the note.
>>> then on the note,
>>> %spark-submit 
>>>   -> interpreter not found error
>>> 4.I have arranged 'SPARK_SUBMIT_OPTIONS' in zeppelin-env.sh order by the
>>> doc
>>> ex. export SPARK_SUBMIT_OPTIONS='--packages
>>> com.databricks:spark-csv_2.10:1.2.0'
>>> 5. then running
>>>  %spark-submit 
>>>   -> interpreter not found error  (as same as "3")
>>>
>>> How can i use spark-submit from a note?
>>> Any advice thanks.
>>>
>>> -Keiji
>>>
>>
>>

python is not responding on ver.0.7.2

[小野圭二]

Hi,

This might be an old and popular issue, but i could not find the solution
on Jira, mail archive and so on.
The python demo does not work.

[Env]:  CentOS 6.2  Python 2.6.6
 Zeppelin   0.7.2 netinst
 beneath 'interpreter': jdbc,lib,md,python,shell,spark
[notebook]: from demo
%python
import matplotlib.pyplot as plt
plt.plot([1,2,3])

Traceback (most recent call last):
File "/tmp/zeppelin_python-2865335277309763273.py", line 20, in 
from py4j.java_gateway import java_import, JavaGateway, GatewayClient
ImportError: No module named py4j.java_gateway
Traceback (most recent call last):
File "/tmp/zeppelin_python-2865335277309763273.py", line 20, in 
from py4j.java_gateway import java_import, JavaGateway, GatewayClient
ImportError: No module named py4j.java_gateway

python is not responding


Any advice would be appreciated.

-Keiji

Re: How to apache zeppelin front ui?

[小野圭二]

Hi Steve,

I am not sure the front ui can replace with the bootstrap theme, but I have
been customizing my Zep by changing the files below
'zeppelin-web/src/app/home' and/or 'zeppelin-web/src/components'.
You know after changing those files You need recompile to reflect it to
yours.

Hey guys, am I correct?

-keiji

2017-11-21 15:24 GMT+09:00 Steve Fox :

> Hi zeppelin users!
>
> I am trying to change the front ui of apache zeppelin.
>
> Can I change to the bootstrap theme when I change front ui?
>
> ex https://bootswatch.com/
>
> I can not seem to find it, but where is the document to refer to?
>
> Thanks.
>

How to get the data url

[小野圭二]

Hi,

I am searching a useful API for getting a data URL that is accessed in a
notebook.
For example, when this URL is in a application

   new URL("https://s3.amazonaws.com/apache-zeppelin/tutorial/bank/
bank.csv")

How to get this url from using Zeppelin API?
I guess it is usually and easy to execute by a interpreter API like Spark,
but that way would be forced to implement the logic in all interpreters, I
do not want to do it.
It is convenient If Zeppelin provide this function.

Any advice are welcome.

-Keiji

Re: Importing notes

[小野圭二]

interest question, i also want to know how to do it. :-)

2018-01-31 9:25 GMT+09:00 Ricardo Martinelli de Oliveira <
rmart...@redhat.com>:

> Hello,
>
> Is there a way to import notes from a command line? I am trying to write a
> script that imports all my notes in a zeppelin docker container, but the
> only ways to import is from the web UI or REST. In the script, zeppelin is
> not started (there is no reason to be) so I'm trying to find a way to
> import the note in a disconnected way.
>
> Any ideas to do this?
>

Re: Thrift server for call back will start. Port:

[小野圭二]

)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at
org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at
org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection
refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
 WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2}
NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973
is finished, status: ERROR, exception:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused), result:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused)
 INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2}
VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
 INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2}
SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973
finished by scheduler
org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session

2018-03-02 15:07 GMT+09:00 小野圭二 <onoke...@gmail.com>:

> Hi Jeff,
>
> No one graved 63039 so far on Mac.
> Yes, this issue has been reproduced as changing the port NO randomly.
>
> Ono
>
>
> 2018-03-02 14:49 GMT+09:00 Jeff Zhang <zjf...@gmail.com>:
>
>>
>> It is possible that 63039 is used by other processes, as Mac can not
>> detect whether this port is used or not from my experience.
>>
>> And can you reproduce this issue consistently ? Because each time
>> zeppelin will chose a random port as long as you don't specify the
>> portRange, so if you reproduce it consistently, it might due to other
>> issues.
>>
>>
>>
>>
>> 小野圭二 <onoke...@gmail.com>于2018年3月2日周五 下午1:32写道：
>>
>>> Hi there,
>>>
>>> I am so confusing now, let me know why if You know this, please.
>>>
>>> The env:
>>>Mac:  High Sierra ver.10.13.3
>>>Zep: 0.7.3
>>>Spark: 2.2.0
>>>Java: 1.8
>>>notebook script:  %pyspark   print 'aaa'  (just it for
>>> test)
>>>
>>> In this env,  i got the err

Re: Thrift server for call back will start. Port:

[小野圭二]

Actually, there was not created the interpreter log file as said 'Cannot
run interpreter' in the log. And it was run on local machine as MAC, i mean
stand alone. i stopped every firewall to make it easy. And again,this
happened just on Mac. On CentOS, it works fine.
That makes me confusion. :-b

2018-03-05 14:06 GMT+09:00 Jeff Zhang <zjf...@gmail.com>:

>
> Could you check the interpreter log ? The issue of not be able to
> connecting to thrift port could happen due to network issue, like firewall
> or you use VPN, etc.
>
>
>
> 小野圭二 <onoke...@gmail.com>于2018年3月5日周一 下午12:32写道：
>
>> Hi,
>>
>> I show the all log on this error.There are 2 type error. The 1 is at the
>> first run. The 2 is at the second run. Seems like could not established to
>> Thrift callback port 51348 that was set at the first run.
>> First of all, does it need to establish the callback port of Thrift
>> server?
>> The notebook script was just
>> %pyspark
>> print 'aaa'
>> Any advice appreciate.
>>
>> 1.
>> INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13}
>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05
>> 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109)
>> - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.
>> interpreter.remote.RemoteInterpreter-shared_session INFO [2018-03-05
>> 13:11:13,936] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run
>> paragraph 20180302-135047_99102973 using pyspark
>> RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
>> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
>> ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) - Create
>> InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process WARN
>> [2018-03-05 13:11:14,010] ({pool-2-thread-2} 
>> InterpreterSetting.java[setupPropertiesForSparkR]:796)
>> - sparkr.zip is not found, SparkR may not work. INFO [2018-03-05
>> 13:11:14,017] ({pool-2-thread-2} 
>> RemoteInterpreterManagedProcess.java[start]:123)
>> - Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05
>> 13:11:14,023] ({pool-2-thread-2} 
>> RemoteInterpreterManagedProcess.java[start]:130)
>> - Thrift server for callback will start. Port: 51348 INFO [2018-03-05
>> 13:11:14,049] ({pool-2-thread-2} 
>> RemoteInterpreterManagedProcess.java[start]:205)
>> - Run interpreter process 
>> [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh,
>> -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c,
>> 172.16.20.10, -p, 51348, -l, 
>> /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX,
>> -g, spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor}
>> RemoteInterpreterManagedProcess.java[onProcessComplete]:258) -
>> Interpreter process exited 0 ERROR [2018-03-05 13:12:14,058]
>> ({pool-2-thread-2} Job.java[run]:184) - Job failed org.apache.zeppelin.
>> interpreter.InterpreterException: Cannot run interpreter at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProces
>> s.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.
>> interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101)
>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_
>> create(RemoteInterpreter.java:153) at org.apache.zeppelin.
>> interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
>> at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
>> org.apache.zeppelin.scheduler.Job.run(Job.java:182) at
>> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
>> java.util.concurrent.ScheduledThreadPoolExecutor$
>> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(
>> ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.
>> ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05 13:12:14,075]
>> ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error
>> org.apache.zeppelin.interpreter.InterpreterException: Cannot run
>> interpreter at org.apache.ze

Re: Thrift server for call back will start. Port:

[小野圭二]

Hi Jeff,

No one graved 63039 so far on Mac.
Yes, this issue has been reproduced as changing the port NO randomly.

Ono


2018-03-02 14:49 GMT+09:00 Jeff Zhang <zjf...@gmail.com>:

>
> It is possible that 63039 is used by other processes, as Mac can not
> detect whether this port is used or not from my experience.
>
> And can you reproduce this issue consistently ? Because each time zeppelin
> will chose a random port as long as you don't specify the portRange, so if
> you reproduce it consistently, it might due to other issues.
>
>
>
>
> 小野圭二 <onoke...@gmail.com>于2018年3月2日周五 下午1:32写道：
>
>> Hi there,
>>
>> I am so confusing now, let me know why if You know this, please.
>>
>> The env:
>>Mac:  High Sierra ver.10.13.3
>>Zep: 0.7.3
>>Spark: 2.2.0
>>Java: 1.8
>>notebook script:  %pyspark   print 'aaa'  (just it for
>> test)
>>
>> In this env,  i got the error in log as
>>
>> ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed
>>
>> org.apache.zeppelin.interpreter.InterpreterException:
>> org.apache.zeppelin.interpreter.InterpreterException:
>> org.apache.thrift.transport.TTransportException:
>> java.net.ConnectException: Connection refused (Connection refused)
>>
>>
>> Indeed, before running notebook, RemoteInterpreterManagedProcess.java
>> prepares a Port for callback to Thrift.
>>
>>
>> INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130)
>> - Thrift server for callback will start. Port: 63039
>>
>>
>> I guess, the reason why the error was Zeppelin could not attache to the
>> Thrift PORT.
>>
>>
>> And my confusion is, when i run the same module on CentOS, it worked fine.
>> CentOS:  7
>> Zep: same
>> Spark: same
>> Java: same
>>  notebook script: same
>>
>> And there was no the Thift port description in the log file. From this
>> consequence, the fine working was cause of did not use Thrift port on
>> CentOS.
>>
>>
>> Do You know actually there is different behavior between Mac and CentOS?
>> And i am so appreciated if You know the workaround on Mac.
>>
>>
>> Best,
>> Ono
>>
>

Thrift server for call back will start. Port:

[小野圭二]

Hi there,

I am so confusing now, let me know why if You know this, please.

The env:
   Mac:  High Sierra ver.10.13.3
   Zep: 0.7.3
   Spark: 2.2.0
   Java: 1.8
   notebook script:  %pyspark   print 'aaa'  (just it for test)

In this env,  i got the error in log as

ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed

org.apache.zeppelin.interpreter.InterpreterException:
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.thrift.transport.TTransportException: java.net.ConnectException:
Connection refused (Connection refused)


Indeed, before running notebook, RemoteInterpreterManagedProcess.java
prepares a Port for callback to Thrift.


INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130) -
Thrift server for callback will start. Port: 63039


I guess, the reason why the error was Zeppelin could not attache to the
Thrift PORT.


And my confusion is, when i run the same module on CentOS, it worked fine.
CentOS:  7
Zep: same
Spark: same
Java: same
 notebook script: same

And there was no the Thift port description in the log file. From this
consequence, the fine working was cause of did not use Thrift port on
CentOS.


Do You know actually there is different behavior between Mac and CentOS?
And i am so appreciated if You know the workaround on Mac.


Best,
Ono

Re: Thrift server for call back will start. Port:

[小野圭二]

Nothing changed. I guess this property does not effect cause of failing to
call the interpreter.
i challenged it as log4j...zeppelin=DEBUG, but no way.

2018-03-05 18:24 GMT+09:00 Jeff Zhang <zjf...@gmail.com>:

>
> Try to add the following to log4j.properties for more log
>
> log4j.logger.org.apache.zeppelin.interpreter=DEBUG
>
>
>
> 小野圭二 <onoke...@gmail.com>于2018年3月5日周一 下午5:13写道：
>
>> Actually, there was not created the interpreter log file as said 'Cannot
>> run interpreter' in the log. And it was run on local machine as MAC, i mean
>> stand alone. i stopped every firewall to make it easy. And again,this
>> happened just on Mac. On CentOS, it works fine.
>> That makes me confusion. :-b
>>
>> 2018-03-05 14:06 GMT+09:00 Jeff Zhang <zjf...@gmail.com>:
>>
>>>
>>> Could you check the interpreter log ? The issue of not be able to
>>> connecting to thrift port could happen due to network issue, like firewall
>>> or you use VPN, etc.
>>>
>>>
>>>
>>> 小野圭二 <onoke...@gmail.com>于2018年3月5日周一 下午12:32写道：
>>>
>>>> Hi,
>>>>
>>>> I show the all log on this error.There are 2 type error. The 1 is at
>>>> the first run. The 2 is at the second run. Seems like could not established
>>>> to Thrift callback port 51348 that was set at the first run.
>>>> First of all, does it need to establish the callback port of Thrift
>>>> server?
>>>> The notebook script was just
>>>> %pyspark
>>>> print 'aaa'
>>>> Any advice appreciate.
>>>>
>>>> 1.
>>>> INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13}
>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO
>>>> [2018-03-05 13:11:13,934] ({pool-2-thread-2} 
>>>> SchedulerFactory.java[jobStarted]:109)
>>>> - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.
>>>> interpreter.remote.RemoteInterpreter-shared_session INFO [2018-03-05
>>>> 13:11:13,936] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run
>>>> paragraph 20180302-135047_99102973 using pyspark
>>>> RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
>>>> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
>>>> ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) -
>>>> Create InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process
>>>> WARN [2018-03-05 13:11:14,010] ({pool-2-thread-2} 
>>>> InterpreterSetting.java[setupPropertiesForSparkR]:796)
>>>> - sparkr.zip is not found, SparkR may not work. INFO [2018-03-05
>>>> 13:11:14,017] ({pool-2-thread-2} 
>>>> RemoteInterpreterManagedProcess.java[start]:123)
>>>> - Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05
>>>> 13:11:14,023] ({pool-2-thread-2} 
>>>> RemoteInterpreterManagedProcess.java[start]:130)
>>>> - Thrift server for callback will start. Port: 51348 INFO [2018-03-05
>>>> 13:11:14,049] ({pool-2-thread-2} 
>>>> RemoteInterpreterManagedProcess.java[start]:205)
>>>> - Run interpreter process 
>>>> [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh,
>>>> -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c,
>>>> 172.16.20.10, -p, 51348, -l, 
>>>> /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX,
>>>> -g, spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor}
>>>> RemoteInterpreterManagedProcess.java[onProcessComplete]:258) -
>>>> Interpreter process exited 0 ERROR [2018-03-05 13:12:14,058]
>>>> ({pool-2-thread-2} Job.java[run]:184) - Job failed org.apache.zeppelin.
>>>> interpreter.InterpreterException: Cannot run interpreter at
>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProces
>>>> s.start(RemoteInterpreterManagedProcess.java:220) at
>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.
>>>> getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at
>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_
>>>> create(RemoteInterpreter.java:153) at org.apache.zeppelin.
>>>> interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
>>>> at org.apache.zeppelin.interpreter.remote.
>>>> RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at
>>>> org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at
>>>> org.apache.zeppel

Re: Thrift server for call back will start. Port:

[小野圭二]

I wonder why, but it seems work fine, and would like to share how to fix it.

First of all, this issue happened just on MAC.
[THE ENV WAS]
  macOS High Sierra v 10.13.3
  cpu core i5
  mem  8GB
  Zeppelin 0.8.0 snapshot
  Spark 2.2.0
[ISSUE]
   see this previous mails
[SOLUTION]
   ref.
http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Zeppelin-0-6-2-Error-Interpreter-process-failed-td4743.html
   According to the above ref, i removed 'SPARK_HOME',
'SPARK_SUBMIT_OPTIONS' from my environment parameters and
conf/zeppelin-env.sh(set it as same like CentOS). Then restart zeppelin,
that's all.
[TO DO]
   Seems like it is not good chemistry between Spark and Zeppelin on Mac.
>From the reason that the consequence of removing the environment
parameters, Zeppelin headed to own spark lib.
   Maybe should write this in the document as caution.


Cheers,

Ono Keiji




2018-03-06 9:15 GMT+09:00 小野圭二 <onoke...@gmail.com>:

> Nothing changed. I guess this property does not effect cause of failing to
> call the interpreter.
> i challenged it as log4j...zeppelin=DEBUG, but no way.
>
> 2018-03-05 18:24 GMT+09:00 Jeff Zhang <zjf...@gmail.com>:
>
>>
>> Try to add the following to log4j.properties for more log
>>
>> log4j.logger.org.apache.zeppelin.interpreter=DEBUG
>>
>>
>>
>> 小野圭二 <onoke...@gmail.com>于2018年3月5日周一 下午5:13写道：
>>
>>> Actually, there was not created the interpreter log file as said 'Cannot
>>> run interpreter' in the log. And it was run on local machine as MAC, i mean
>>> stand alone. i stopped every firewall to make it easy. And again,this
>>> happened just on Mac. On CentOS, it works fine.
>>> That makes me confusion. :-b
>>>
>>> 2018-03-05 14:06 GMT+09:00 Jeff Zhang <zjf...@gmail.com>:
>>>
>>>>
>>>> Could you check the interpreter log ? The issue of not be able to
>>>> connecting to thrift port could happen due to network issue, like firewall
>>>> or you use VPN, etc.
>>>>
>>>>
>>>>
>>>> 小野圭二 <onoke...@gmail.com>于2018年3月5日周一 下午12:32写道：
>>>>
>>>>> Hi,
>>>>>
>>>>> I show the all log on this error.There are 2 type error. The 1 is at
>>>>> the first run. The 2 is at the second run. Seems like could not 
>>>>> established
>>>>> to Thrift callback port 51348 that was set at the first run.
>>>>> First of all, does it need to establish the callback port of Thrift
>>>>> server?
>>>>> The notebook script was just
>>>>> %pyspark
>>>>> print 'aaa'
>>>>> Any advice appreciate.
>>>>>
>>>>> 1.
>>>>> INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13}
>>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO
>>>>> [2018-03-05 13:11:13,934] ({pool-2-thread-2} 
>>>>> SchedulerFactory.java[jobStarted]:109)
>>>>> - Job 20180302-135047_99102973 started by scheduler
>>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
>>>>> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
>>>>> Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using
>>>>> pyspark 
>>>>> RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
>>>>> INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2}
>>>>> ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) -
>>>>> Create InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process
>>>>> WARN [2018-03-05 13:11:14,010] ({pool-2-thread-2}
>>>>> InterpreterSetting.java[setupPropertiesForSparkR]:796) - sparkr.zip
>>>>> is not found, SparkR may not work. INFO [2018-03-05 13:11:14,017]
>>>>> ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123) -
>>>>> Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05
>>>>> 13:11:14,023] ({pool-2-thread-2} 
>>>>> RemoteInterpreterManagedProcess.java[start]:130)
>>>>> - Thrift server for callback will start. Port: 51348 INFO [2018-03-05
>>>>> 13:11:14,049] ({pool-2-thread-2} 
>>>>> RemoteInterpreterManagedProcess.java[start]:205)
>>>>> - Run interpreter process 
>>>>> [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh,
>>>>> -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c,
>>>>> 172.16.20.10, -p, 51348, -l,

Re: Thrift server for call back will start. Port:

[小野圭二]

Hi Jeff,

I did set SPARK_HOME to use external spark at first. And that was fine work
on CentOS, but not on Mac.
Do you mean try it again?

2018-03-06 11:12 GMT+09:00 Jeff Zhang <zjf...@gmail.com>:

> >>> Seems like it is not good chemistry between Spark and Zeppelin on
> Mac. From the reason that the consequence of removing the environment
> parameters, Zeppelin headed to own spark lib.
>
> Could you specify SPARK_HOME in your spark interpreter to use external
> spark instead of zeppelin's own spark lib ?
>
>
>
>
> 小野圭二 <onoke...@gmail.com>于2018年3月6日周二 上午10:06写道：
>
>> I wonder why, but it seems work fine, and would like to share how to fix
>> it.
>>
>> First of all, this issue happened just on MAC.
>> [THE ENV WAS]
>>   macOS High Sierra v 10.13.3
>>   cpu core i5
>>   mem  8GB
>>   Zeppelin 0.8.0 snapshot
>>   Spark 2.2.0
>> [ISSUE]
>>see this previous mails
>> [SOLUTION]
>>ref. http://apache-zeppelin-users-incubating-mailing-list.
>> 75479.x6.nabble.com/Zeppelin-0-6-2-Error-Interpreter-
>> process-failed-td4743.html
>>According to the above ref, i removed 'SPARK_HOME',
>> 'SPARK_SUBMIT_OPTIONS' from my environment parameters and
>> conf/zeppelin-env.sh(set it as same like CentOS). Then restart zeppelin,
>> that's all.
>> [TO DO]
>>Seems like it is not good chemistry between Spark and Zeppelin on Mac.
>> From the reason that the consequence of removing the environment
>> parameters, Zeppelin headed to own spark lib.
>>Maybe should write this in the document as caution.
>>
>>
>> Cheers,
>>
>> Ono Keiji
>>
>>
>>
>>
>> 2018-03-06 9:15 GMT+09:00 小野圭二 <onoke...@gmail.com>:
>>
>>> Nothing changed. I guess this property does not effect cause of failing
>>> to call the interpreter.
>>> i challenged it as log4j...zeppelin=DEBUG, but no way.
>>>
>>> 2018-03-05 18:24 GMT+09:00 Jeff Zhang <zjf...@gmail.com>:
>>>
>>>>
>>>> Try to add the following to log4j.properties for more log
>>>>
>>>> log4j.logger.org.apache.zeppelin.interpreter=DEBUG
>>>>
>>>>
>>>>
>>>> 小野圭二 <onoke...@gmail.com>于2018年3月5日周一 下午5:13写道：
>>>>
>>>>> Actually, there was not created the interpreter log file as said
>>>>> 'Cannot run interpreter' in the log. And it was run on local machine as
>>>>> MAC, i mean stand alone. i stopped every firewall to make it easy. And
>>>>> again,this happened just on Mac. On CentOS, it works fine.
>>>>> That makes me confusion. :-b
>>>>>
>>>>> 2018-03-05 14:06 GMT+09:00 Jeff Zhang <zjf...@gmail.com>:
>>>>>
>>>>>>
>>>>>> Could you check the interpreter log ? The issue of not be able to
>>>>>> connecting to thrift port could happen due to network issue, like 
>>>>>> firewall
>>>>>> or you use VPN, etc.
>>>>>>
>>>>>>
>>>>>>
>>>>>> 小野圭二 <onoke...@gmail.com>于2018年3月5日周一 下午12:32写道：
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I show the all log on this error.There are 2 type error. The 1 is at
>>>>>>> the first run. The 2 is at the second run. Seems like could not 
>>>>>>> established
>>>>>>> to Thrift callback port 51348 that was set at the first run.
>>>>>>> First of all, does it need to establish the callback port of Thrift
>>>>>>> server?
>>>>>>> The notebook script was just
>>>>>>> %pyspark
>>>>>>> print 'aaa'
>>>>>>> Any advice appreciate.
>>>>>>>
>>>>>>> 1.
>>>>>>> INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13}
>>>>>>> VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO
>>>>>>> [2018-03-05 13:11:13,934] ({pool-2-thread-2} 
>>>>>>> SchedulerFactory.java[jobStarted]:109)
>>>>>>> - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.
>>>>>>> interpreter.remote.RemoteInterpreter-shared_session INFO
>>>>>>> [2018-03-05 13:11:13,936] ({pool-2-thread-2} 
>>>>>>> Paragraph.java[jobRun]:397) -
>>>>>>> run paragraph 20180302-135047_9910