Re: [VOTE] Release Apache Zeppelin 0.10.1 RC2

[moon soo Lee]

+1

On Thu, Feb 24, 2022 at 9:31 AM Prabhjyot Singh 
wrote:

> +1
>
> On Thu, Feb 24, 2022 at 1:58 PM Jeff Zhang  wrote:
>
>> Hi folks,
>>
>> I propose the following RC to be released for the Apache Zeppelin
>> 0.10.1 release.
>>
>>
>> The commit id is 1654ce3847f4ebce7b228f54671340569d25a32e :
>>
>> https://git-wip-us.apache.org/repos/asf?p=zeppelin.git;a=commit;h=1654ce3847f4ebce7b228f54671340569d25a32e
>>
>>
>>
>> This corresponds to the tag: v0.10.1-rc2 :
>>
>> https://git-wip-us.apache.org/repos/asf?p=zeppelin.git;a=shortlog;h=refs/tags/v0.10.1-rc2
>>
>> The release archives (tgz), signature, and checksums are here
>> https://dist.apache.org/repos/dist/dev/zeppelin/zeppelin-0.10.1-rc2/
>>
>> The release candidate consists of the following source distribution
>> archive
>> zeppelin-0.10.1.tgz
>>
>> In addition, the following supplementary binary distributions are provided
>> for user convenience at the same location
>> zeppelin-0.10.1-bin-all.tgz
>>
>>
>> The maven artifacts are here
>>
>> https://repository.apache.org/content/repositories/orgapachezeppelin-1323/org/apache/zeppelin/
>>
>> You can find the KEYS file here:
>> https://dist.apache.org/repos/dist/release/zeppelin/KEYS
>>
>> Release notes available at
>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316221=12350543
>>
>>
>> Vote will be open for next 72 hours (close at 12PM 26/Feb PDT).
>>
>> [ ] +1 approve
>> [ ] 0 no opinion
>>
>> [ ] -1 disapprove (and reason why)
>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>
>
> --
> Regards,
> Prabhjyot Singh
>

Re: [VOTE] Release Apache Zeppelin 0.10.0 RC2

[moon soo Lee]

+1

Thank Jeff for preparing the RC.


On Thu, Aug 19, 2021 at 7:24 PM Jeff Zhang  wrote:

> +1，  Verified most of the tutorial notes
>
> Jongyoul Lee  于2021年8月19日周四 下午12:20写道：
>
>> +1
>>
>> I've checked the basic example about this RC
>>
>> 2021년 8월 18일 (수) 오후 3:28, 님이 작성:
>>
>>> +1
>>>
>>>
>>>
>>> *From:* Philipp Dallig 
>>> *Sent:* Wednesday, August 18, 2021 3:24 PM
>>> *To:* users@zeppelin.apache.org
>>> *Subject:* Re: [VOTE] Release Apache Zeppelin 0.10.0 RC2
>>>
>>>
>>>
>>> +1
>>>
>>> Am 18.08.21 um 08:01 schrieb Francois Reynald:
>>>
>>> +1
>>>
>>>
>>>
>>> On 18 Aug 2021, at 05:52, Jeff Zhang  wrote:
>>>
>>>
>>>
>>>
>>> Hi folks,
>>>
>>>
>>>
>>> I propose the following RC to be released for the Apache Zeppelin 0.10.0 
>>> release.
>>>
>>>
>>>
>>>
>>>
>>> The commit id is 8f85476c40df3536718c5aa2c07b519b6bb6607e :
>>>
>>> https://git-wip-us.apache.org/repos/asf?p=zeppelin.git;a=commit;h=8f85476c40df3536718c5aa2c07b519b6bb6607e
>>>
>>>
>>>
>>> This corresponds to the tag: v0.10.0-rc2 :
>>>
>>> https://git-wip-us.apache.org/repos/asf?p=zeppelin.git;a=shortlog;h=refs/tags/v0.10.0-rc2
>>>
>>> The release archives (tgz), signature, and checksums are here
>>>
>>> https://dist.apache.org/repos/dist/dev/zeppelin/zeppelin-0.10.0-rc2/
>>>
>>>
>>>
>>> The release candidate consists of the following source distribution archive
>>>
>>> zeppelin-0.10.0.tgz
>>>
>>>
>>>
>>> In addition, the following supplementary binary distributions are provided
>>>
>>> for user convenience at the same location
>>>
>>> zeppelin-0.10.0-bin-all.tgz
>>>
>>>
>>>
>>>
>>>
>>> The maven artifacts are here
>>>
>>> https://repository.apache.org/content/repositories/orgapachezeppelin-1302/org/apache/zeppelin/
>>>
>>>
>>>
>>> You can find the KEYS file here:
>>>
>>> https://dist.apache.org/repos/dist/release/zeppelin/KEYS
>>>
>>>
>>>
>>> Release notes available at 
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12349536==12316221
>>>
>>>
>>>
>>> Vote will be open for next 72 hours (close at 10PM 20/Aug PDT).
>>>
>>>
>>>
>>> [ ] +1 approve
>>>
>>> [ ] 0 no opinion
>>>
>>> [ ] -1 disapprove (and reason why)
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>>
>>>
>>> *François Reynald | Educational Services*
>>> t. +33 (0) 624 200 125
>>> cloudera.com 
>>>
>>>
>>>
>>>
>>
>> --
>> 이종열, Jongyoul Lee, 李宗烈
>> http://madeng.net
>>
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: [DISCUSS] Releasing 0.9.1

[moon soo Lee]

+1

On Tue, Jul 6, 2021 at 03:20 Eric Pugh 
wrote:

> I’d love to see another release as well.
>
>
> On Jul 5, 2021, at 11:14 PM, Jeff Zhang  wrote:
>
> Hi Bill,
>
> I haven't calculated that, you can refer this list for all the tickets
> that are merged in 0.9.1
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20ZEPPELIN%20and%20status%20%3D%20RESOLVED%20AND%20fixVersion%20%3D%200.9.1
>
>
> Bill Brooks  于2021年7月6日周二 上午10:38写道：
>
>> Jeff,
>>
>> Apologies if you already covered this on the slack channel, but I was
>> wondering, for the proposed release:
>> - how many accepted commits that have never reached any Apache zeppelin
>> official release are outstanding?
>>
>> Thanks,
>>
>> Bill
>>
>> On Mon, Jul 5, 2021 at 7:46 AM Jeff Zhang  wrote:
>>
>>> Hi Folks,
>>>
>>> It has been about half year since the last release of 0.9.0, after that
>>> we have many fixes and improvements, what do you think of the release of
>>> 0.9.1 ?
>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>
>
> --
> Best Regards
>
> Jeff Zhang
>
>
> ___
> *Eric Pugh **| *Founder & CEO | OpenSource Connections, LLC | 434.466.1467
> | http://www.opensourceconnections.com | My Free/Busy
> 
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> 
> This e-mail and all contents, including attachments, is considered to be
> Company Confidential unless explicitly stated otherwise, regardless
> of whether attachments are marked as such.
>
>

Re: Zeppelin 0.9 / Kubernetes / Spark interpreter

[moon soo Lee]

Hi,

Thanks for sharing the issue.

I tried zeppelin 0.9+ on k8s with per note scoped, scala 2.12, spark 3.0+.
And I could reproduce the problem. But isolated mode works without problem.
Does isolated mode work for your use case?

Best,
moon



On Tue, Apr 27, 2021 at 12:39 PM Sylvain Gibier 
wrote:

> Any idea?
>
> Actually anyone using zeppelin 0.9+ on k8s, with spark interpreter scope
> per note ?
>
>
> On 2021/04/24 10:46:06, Sylvain Gibier  wrote:
> > Hi,
> >
> > we have an issue with our current deployment of zeppelin on k8s, and more
> > precisely with spark interpreter.
> >
> > For reference - the spark context is: scala 2.12.10 / spark 2.4.7
> >
> > We have a weird behaviour, running the spark interpreter in per note,
> scoped
> >
> > To reproduce currently - we restart the spark interpreter in scoped per
> > note, and create two notebooks (A & B) with the same following code:
> >
> > %spark
> > > import spark.implicits._
> > >
> > > List(1, 2, 3).toDS.map(_ + 1).show
> > >
> >
> > 1- we run notebook A successfully
> > 2 - we run notebook B  - it fails with class cast exception
> >
> > org.apache.spark.SparkException: Job aborted due to stage failure: Task 0
> > > in stage 24.0 failed 4 times, most recent failure: Lost task 0.3 in
> stage
> > > 24.0 (TID 161, 10.11.18.133, executor 2): java.lang.ClassCastException:
> > > cannot assign instance of java.lang.invoke.SerializedLambda to field
> > > org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in
> instance
> > > of org.apache.spark.rdd.MapPartitionsRDD at
> > >
> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287)
> > > at
> java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417)
> > > at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2293)
> > >
> >
> > Anyone having a working zeppelin deployment with k8s / spark 2.4 - scala
> > 2.12 ?
> >
> > or let anyone interested to make some $$$ to help us fix the issue?
> >
> > cheers
> >
>

Re: Notebook security not working properly 0.9.0

[moon soo Lee]

Hi,

It looks like an issue. I think expected behavior supposed to allow other
users to read the notebook when the reader's permission is the wildcard. I
think you can file an issue.
There was a discussion about the read-only mode in the last community
meeting, as far as I know.
Someone, who was in the community meeting can add some context if this is
related.

Thanks,
moon


On Thu, Apr 15, 2021 at 8:59 AM Carlos Diogo  wrote:

> Dear all
> Any idea?
>
> On Fri, Apr 9, 2021 at 11:42 AM Carlos Diogo  wrote:
>
>> Dear team
>> I cannot seem to be able to set the security of the note right.
>>
>> Target security model
>> - userA should be able to write
>> - all other user should be able to see the code but not able to change
>> the note
>>
>> If i set UserA and Owner and Writer and leave blank Executor and Reader ,
>> the users cannot see the code only the titles of the note
>>
>> If i set UserA and Owner and Writer and Executor and Reader i set (*) ,
>> the users can see the code momentarily but if they refresh they get an
>> permission error . Furthermore they no longer can see the notebook on the
>> notebook tree
>>
>> Is this a known issue?
>> Thanks in advance
>>
>> --
>> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
>> Carlos Diogo
>>
>
>
> --
> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
> Carlos Diogo
>

Re: Spark interpreter Repl injection

[moon soo Lee]

Pyspark interpreter have 'intp' variable exposed in its repl environment
(for internal use). And we can resolve reference to Spark interpreter from
the 'intp' variable. However, scala repl environment in Spark Interpreter
doesn't expose any variables that is useful for finding Spark Interpreter
itself. So had to find a way from pyspark interpreter.

z.interpret() doesn't look like it can bring some problem, in my opinion.

Thanks,
moon




On Tue, Mar 9, 2021 at 8:54 AM Carlos Diogo  wrote:

> Looks good Moon
> Is there a specific reason why you needed the pyspark interpreter  to
> access the spark interpreter? Could not the spark interpreter
> programmatically access itself (and the same for the pyspark interpreter)
>
> Would the issue be to expose the z.interpret() method?
>
> Best regards
> Carlos
>
> On Tue, Mar 9, 2021 at 5:10 PM moon soo Lee  wrote:
>
>> I see. If you want to specify a file, precode might not the best option.
>> I found a hacky way to do it. Accessing SparkInterpreter instance object
>> from PysparkInterpreter.
>>
>> %pyspark
>> sparkIntpField = intp.getClass().getDeclaredField("sparkInterpreter")
>> sparkIntpField.setAccessible(True)
>> sparkIntp = sparkIntpField.get(intp)
>> # run my scala code
>> sparkIntp.interpret("val a=10", z.getInterpreterContext())
>>
>>
>> See attached screenshot.
>>
>> [image: image.png]
>>
>> This is accessing internal variables outside the official API. So it may
>> break at any time.
>>
>> I think it's better to expose interpret() method through
>> 'ZeppelinContext'. So inside Note,
>>
>> z.interpret(any_string)
>>
>> can work without accessing this method in a hacky way.
>> Please feel free to file an issue.
>>
>> Thanks,
>> moon
>>
>>
>>
>>
>> On Mon, Mar 8, 2021 at 10:23 PM Carlos Diogo  wrote:
>>
>>> Are you able to specify a file on the precode?
>>> For now my work around is from within the note and with the rest api ,
>>> to add a paragraph with the code I want to inject ( which can come from a
>>> file )
>>> It works ok , but with run all or schedule the code gets updated in the
>>> note , but the old Code still executes . Only on the next run it will take
>>> effect
>>>
>>> On Mon 8. Mar 2021 at 22:48, moon soo Lee  wrote:
>>>
>>>> Hi,
>>>>
>>>> How about precode
>>>> <http://zeppelin.apache.org/docs/0.9.0/usage/interpreter/overview.html#precode>?
>>>>  "zeppelin.SparkInterpreter.precode"
>>>> can run scala code.
>>>>
>>>> Thanks,
>>>> moon
>>>>
>>>>
>>>> On Sat, Mar 6, 2021 at 4:51 AM Carlos Diogo  wrote:
>>>>
>>>>> That does not work if you want to have Scala code in a file ( common
>>>>> functions) which you want to invoke in the note
>>>>> The alternative is to compile the code and then add the jar which
>>>>> would be normal for an application.
>>>>> But zeppelin is about scripting so this is a request I get very often
>>>>> from the users.
>>>>> Specially because the z.run does not work properly most of the times
>>>>> Carlos
>>>>>
>>>>> On Sat 6. Mar 2021 at 11:36, Jeff Zhang  wrote:
>>>>>
>>>>>> Why not copying scala code in zeppelin and run the notebook directly ?
>>>>>>
>>>>>> Carlos Diogo  于2021年3月6日周六 下午3:51写道：
>>>>>>
>>>>>>> Dear all
>>>>>>> I have been  trying  to find a was to inject scala Code ( from
>>>>>>> String) into the spark interpreter
>>>>>>> In pyspark is easy with the exec function
>>>>>>> It should not be very difficult  to access from the Note scala repl
>>>>>>> interpreter but i could not find a way . I was even able to create a new
>>>>>>> repl session but then I could not bind the objects
>>>>>>> Any tips ?
>>>>>>> Thanks
>>>>>>> --
>>>>>>> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
>>>>>>> Carlos Diogo
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards
>>>>>>
>>>>>> Jeff Zhang
>>>>>>
>>>>> --
>>>>> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
>>>>> Carlos Diogo
>>>>>
>>>> --
>>> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
>>> Carlos Diogo
>>>
>>
>
> --
> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
> Carlos Diogo
>

Re: Spark interpreter Repl injection

[moon soo Lee]

I see. If you want to specify a file, precode might not the best option.
I found a hacky way to do it. Accessing SparkInterpreter instance object
from PysparkInterpreter.

%pyspark
sparkIntpField = intp.getClass().getDeclaredField("sparkInterpreter")
sparkIntpField.setAccessible(True)
sparkIntp = sparkIntpField.get(intp)
# run my scala code
sparkIntp.interpret("val a=10", z.getInterpreterContext())


See attached screenshot.

[image: image.png]

This is accessing internal variables outside the official API. So it may
break at any time.

I think it's better to expose interpret() method through 'ZeppelinContext'.
So inside Note,

z.interpret(any_string)

can work without accessing this method in a hacky way.
Please feel free to file an issue.

Thanks,
moon




On Mon, Mar 8, 2021 at 10:23 PM Carlos Diogo  wrote:

> Are you able to specify a file on the precode?
> For now my work around is from within the note and with the rest api , to
> add a paragraph with the code I want to inject ( which can come from a file
> )
> It works ok , but with run all or schedule the code gets updated in the
> note , but the old Code still executes . Only on the next run it will take
> effect
>
> On Mon 8. Mar 2021 at 22:48, moon soo Lee  wrote:
>
>> Hi,
>>
>> How about precode
>> <http://zeppelin.apache.org/docs/0.9.0/usage/interpreter/overview.html#precode>?
>>  "zeppelin.SparkInterpreter.precode"
>> can run scala code.
>>
>> Thanks,
>> moon
>>
>>
>> On Sat, Mar 6, 2021 at 4:51 AM Carlos Diogo  wrote:
>>
>>> That does not work if you want to have Scala code in a file ( common
>>> functions) which you want to invoke in the note
>>> The alternative is to compile the code and then add the jar which would
>>> be normal for an application.
>>> But zeppelin is about scripting so this is a request I get very often
>>> from the users.
>>> Specially because the z.run does not work properly most of the times
>>> Carlos
>>>
>>> On Sat 6. Mar 2021 at 11:36, Jeff Zhang  wrote:
>>>
>>>> Why not copying scala code in zeppelin and run the notebook directly ?
>>>>
>>>> Carlos Diogo  于2021年3月6日周六 下午3:51写道：
>>>>
>>>>> Dear all
>>>>> I have been  trying  to find a was to inject scala Code ( from String)
>>>>> into the spark interpreter
>>>>> In pyspark is easy with the exec function
>>>>> It should not be very difficult  to access from the Note scala repl
>>>>> interpreter but i could not find a way . I was even able to create a new
>>>>> repl session but then I could not bind the objects
>>>>> Any tips ?
>>>>> Thanks
>>>>> --
>>>>> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
>>>>> Carlos Diogo
>>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards
>>>>
>>>> Jeff Zhang
>>>>
>>> --
>>> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
>>> Carlos Diogo
>>>
>> --
> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
> Carlos Diogo
>

Re: Spark interpreter Repl injection

[moon soo Lee]

Hi,

How about precode
?
"zeppelin.SparkInterpreter.precode"
can run scala code.

Thanks,
moon


On Sat, Mar 6, 2021 at 4:51 AM Carlos Diogo  wrote:

> That does not work if you want to have Scala code in a file ( common
> functions) which you want to invoke in the note
> The alternative is to compile the code and then add the jar which would be
> normal for an application.
> But zeppelin is about scripting so this is a request I get very often from
> the users.
> Specially because the z.run does not work properly most of the times
> Carlos
>
> On Sat 6. Mar 2021 at 11:36, Jeff Zhang  wrote:
>
>> Why not copying scala code in zeppelin and run the notebook directly ?
>>
>> Carlos Diogo  于2021年3月6日周六 下午3:51写道：
>>
>>> Dear all
>>> I have been  trying  to find a was to inject scala Code ( from String)
>>> into the spark interpreter
>>> In pyspark is easy with the exec function
>>> It should not be very difficult  to access from the Note scala repl
>>> interpreter but i could not find a way . I was even able to create a new
>>> repl session but then I could not bind the objects
>>> Any tips ?
>>> Thanks
>>> --
>>> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
>>> Carlos Diogo
>>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
> --
> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
> Carlos Diogo
>

Re: Zeppelin behind authenticating proxy

[moon soo Lee]

Glad to hear it helped! Thanks for sharing!

On Fri, Feb 12, 2021 at 12:12 PM Vladimir Prus 
wrote:

> I have tried some permutations, and one of them ended up working fine, so
> shiro-remote-user appears to be perfectly OK, thanks.
>
> (Still no idea what is wrong in my original setup, but it involved two
> proxies, and a load balancer, and one of them must
> be messing up some part of protocol)
>
> On Fri, Feb 12, 2021 at 10:46 PM Vladimir Prus 
> wrote:
>
>> Hi,
>>
>> that seems exactly what I was looking for. I gave it a try, and got
>> half-way through:
>>
>> - Zeppelin shows the username I set in header, and websocket is
>> connected, and I can use menu with no issues
>> - The main content is however empty - I see no list of notebooks at all.
>> Looks at websocket messages, I see LIST_NOTES that returns
>>   empty list of notes. I have verified that if I revert shiro.ini to my
>> previous version (which uses ldap), the list of notebooks is present.
>>
>> Does this point to some obvious misconfiguration on my side?
>>
>> On Fri, Feb 12, 2021 at 3:12 AM moon soo Lee  wrote:
>>
>>> Hi,
>>>
>>> I haven't tried it personally, but this repository might help
>>> https://github.com/leighklotz/shiro-remote-user
>>>
>>> Thanks,
>>> moon
>>>
>>>
>>>
>>> On Tue, Feb 9, 2021 at 3:25 AM Vladimir Prus 
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I would like to run Zeppelin behind authenticating proxy, so that:
>>>>
>>>> - The proxy handles all authentication, including setting a cookie to
>>>> remember the user
>>>> - It passes a username header to Zeppelin
>>>> - Zeppelin takes that username header and trusts it - it should show
>>>> the user as
>>>> authorized and use that username when starting interpreter or
>>>> evaluating notebook
>>>> permissions
>>>>
>>>> While the documentation mentions how to setup nginx as proxy, I can't
>>>> find any information about the second part - passing username to
>>>> Zeppelin, and actually using it.
>>>> Shiro documentation is likewise not helpful.
>>>>
>>>> How can I accomplish what I want?
>>>>
>>>> --
>>>> Vladimir Prus
>>>> http://vladimirprus.com
>>>>
>>>
>>
>> --
>> Vladimir Prus
>> http://vladimirprus.com
>>
>
>
> --
> Vladimir Prus
> http://vladimirprus.com
>

Re: Zeppelin behind authenticating proxy

[moon soo Lee]

Hi,

I haven't tried it personally, but this repository might help
https://github.com/leighklotz/shiro-remote-user

Thanks,
moon



On Tue, Feb 9, 2021 at 3:25 AM Vladimir Prus 
wrote:

> Hi,
>
> I would like to run Zeppelin behind authenticating proxy, so that:
>
> - The proxy handles all authentication, including setting a cookie to
> remember the user
> - It passes a username header to Zeppelin
> - Zeppelin takes that username header and trusts it - it should show the
> user as
> authorized and use that username when starting interpreter or evaluating
> notebook
> permissions
>
> While the documentation mentions how to setup nginx as proxy, I can't
> find any information about the second part - passing username to Zeppelin,
> and actually using it.
> Shiro documentation is likewise not helpful.
>
> How can I accomplish what I want?
>
> --
> Vladimir Prus
> http://vladimirprus.com
>

Re: [DISCUSS] Zeppelin community sync up meeting

[moon soo Lee]

+1  

Looking forward to it!

On Mon, Jan 25, 2021 at 6:20 PM Jeff Zhang  wrote:

> Hi Bill,
>
> That's right, it has been more than one year we haven't do community sync
> up. I'd like to continue that to bring the community together for more wide
> discussion around zeppelin.
>
> Bill Brooks  于2021年1月26日周二 上午10:11写道：
>
>> This is a great idea, but I thought that Mei Long was already putting
>> these events together and prior to the pandemic, they were happening.
>>
>> Bill
>>
>> On Mon, Jan 25, 2021 at 3:09 PM Jeff Zhang  wrote:
>> >
>> > Hi folks (users & developers),
>> >
>> > I'd like to raise this discussion about regular community sync up
>> meeting to discuss various things of Zeppelin. You may know me that I have
>> been active in the Zeppelin community for the last 4 years. I believe
>> Zeppelin is a very promising and important project in the data science and
>> big data area. But it needs more community involvement to improve it ,no
>> matter whether you are zeppelin users or zeppelin developers. I hope this
>> kind of sync up meeting can gather people who are interested in zeppelin
>> together, and let's discuss what we can do to make zeppelin standout in the
>> data science and big data area.
>> >
>> > I hope it will be a regular sync up meeting, maybe one time per month.
>> Let us know whether you are interested in such kind of regular sync up
>> meeting and what kind of things do you like to discuss.
>> >
>> > --
>> > Best Regards
>> >
>> > Jeff Zhang
>>
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: Vega on zeppelin Helium

[moon soo Lee]

Hi,

Thanks for sharing your problem here. Let me try to help.
Do you see the same error when you run

"npm -i" or "yarn install" in your visualizatioin package
dir? (/home/romain/visu-vegas/Test01_HelloWorld)

Thanks,
moon


On Wed, Sep 30, 2020 at 2:34 AM  wrote:

> Hello,
>
>
>
> I wish to integrate VEGA visualizations on zeppelin (version : *0.8.2*) .
> So I created 3 files to import these visualizations on Helium :
>
>
>
> -HelloWorld.js :
> /home/romain/visu-vegas/Test01_HelloWorld/HelloWorld.js
>
> -HelloWorld.json :
> /home/romain/visu-vegas/Test01_HelloWorld/HelloWorld.json
>
> -package.json :
> /home/romain/visu-vegas/Test01_HelloWorld/package.json
>
>
>
> This is my package.json :
>
> {
>
> "name": "HelloWorld",
>
> "description": "affichage d'une phrase",
>
> "version": "1.0.0",
>
> "main": "HelloWorld",
>
> "author": "",
>
> "license": "Apache-2.0",
>
> "dependencies": {
>
> "zeppelin-tabledata": "*",
>
> "zeppelin-vis": "*",
>
> "vega": "^5.16.1",
>
> "vega-embed": "^6.12.2",
>
> "vega-lite": "^4.16.7"
>
> }
>
> }
>
>
>
> When I click on "enable" for import “HelloWorld” visualizations, it
> returns me an error whereas when I do the manipulation in command line
> (putty) there is no error. Below is a copy of the error.
>
>
>
> To see the logs, I run a command line: *"tail -f
> zeppelin-romain-vmlinux.log*" from this directory: cd
> zeppelin-0.8.2-bin-all/logs.
>
> So, here are the corresponding error logs when I click on "enable":
>
>
>
> But this command goes to CLI when I'm in the bundle directory.
>
>
>
> Do you have any idea why it doesn't work?
>
>
>
> Thank you.
>
> Sincerely.
>
>
>
> Romain D.
>
>
>
> *De :* DUBOIS Romain OBS/CSO
> *Envoyé :* mercredi 30 septembre 2020 10:28
> *À :* 'users@zeppelin.apache.org' 
> *Objet :* Vega on zeppelin Helium
>
>
>
> Hello,
>
>
>
> I wish to integrate VEGA visualizations on zeppelin. So I created 3 files
> to import these visualizations on Helium.
>
> When I click on "enable" in Helium, it returns me an error whereas when I
> do the manipulation in command line (putty) there is no error. Below is a
> copy of the error. Do you have an idea?
>
> Thank you.
>
> Sincerely.
>
>
>
> Romain D.
>
>
>
>
>
> *De :* DUBOIS Romain OBS/CSO
> *Envoyé :* mercredi 30 septembre 2020 10:28
> *À :* 'users@zeppelin.apache.org' 
> *Objet :* Vega on zeppelin Helium
>
>
>
> Hello,
>
>
>
> I wish to integrate VEGA visualizations on zeppelin. So I created 3 files
> to import these visualizations on Helium.
>
> When I click on "enable" in Helium, it returns me an error whereas when I
> do the manipulation in command line (putty) there is no error. Below is a
> copy of the error. Do you have an idea?
>
> Thank you.
>
> Sincerely.
>
>
>
> Romain D.
>
>
>
> _
>
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu 
> ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
> electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou 
> falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged 
> information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete 
> this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been 
> modified, changed or falsified.
> Thank you.
>
>

Re: [DISCUSS] Official release of 0.9.0

[moon soo Lee]

+1. I also think 0.9 is ready!


On Fri, Sep 18, 2020 at 9:27 AM cronoik  wrote:

> Hi Jeff,
>
> I think 0.9 is ready to be released. We are already using preview 2 and
> haven't faced any major issues.
>
> Thank you everyone for your contributions!
>
> Best regards
> cronoik
>
> On 9/18/20 6:02 PM, Jeff Zhang wrote:
>
>
> Hi Folks,
>
> We released 0.9.0 preview2 around 2 months ago, we have made lots of
> improvements after that, I am thinking that it might be good to have
> official release of 0.9. Let me know your thoughts. Thanks.
>
>
> --
> Best Regards
>
> Jeff Zhang
>
>

Re: Results of testing preview2

[moon soo Lee]

Thanks for the valuable feedback!


Importing notes doesn't work in chrome.  It displays the file rather than
> uploading it. It works in firefox and Safari, though Safari is otherwise
> unusable.


I tried import notes from chrome (on MacOS) and it works for me. Is your
chrome runs on other OS?
Could you try clean chrome browser cache and see if it helps?




On Thu, Jul 30, 2020 at 8:46 AM clhedrick  wrote:

> In case anyone is interested, I'm running Zeppelin in a podman container,
> using sssd for authentication (from the host). Our home directories are on
> a
> kerberized NFS mount, so I need to make sure the user has a kerberos
> ticket,
> and it gets renewed as long as the process is active. This requires a fair
> amount of special setup. The container uses macvlan networking so that
> Zeppelin runs on a dedicated IP address.
>
>
>
>
>
> --
> Sent from:
> http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/
>

Re: [VOTE] Release Apache Zeppelin 0.9.0-preview2 (RC2)

[moon soo Lee]

+1

I tested

 - build from source
 - src package license file
 - bin package license file
 - new ui access

On Fri, Jul 24, 2020 at 12:36 AM Alex Ott  wrote:

> +1 from me. What's done:
>
> - checked the checksum
> - run spark samples
> - tested cassandra interpreter
> - installed some plugins from helium registry
>
>
> On Thu, Jul 23, 2020 at 5:04 PM Jeff Zhang  wrote:
>
>>
>> Hi folks,
>>
>> I propose the following RC to be released for the Apache Zeppelin 
>> 0.9.0-preview2 release.
>>
>>
>> The commit id is 31b9ed51f946fed934885d8fbb29e9c183552e70 : 
>> https://gitbox.apache.org/repos/asf?p=zeppelin.git;a=commit;h=31b9ed51f946fed934885d8fbb29e9c183552e70
>>
>> This corresponds to the tag: v0.9.0-preview2-rc2 : 
>> https://gitbox.apache.org/repos/asf?p=zeppelin.git;a=shortlog;h=refs/tags/v0.9.0-preview2-rc2
>>
>> The release archives (tgz), signature, and checksums are here 
>> https://dist.apache.org/repos/dist/dev/zeppelin/zeppelin-0.9.0-preview2-rc2/
>>
>> The release candidate consists of the following source distribution archive
>> zeppelin-v0.9.0-preview2.tgz
>>
>> In addition, the following supplementary binary distributions are provided
>> for user convenience at the same location
>> zeppelin-0.9.0-preview2-bin-all.tgz
>>
>>
>> The maven artifacts are here 
>> https://repository.apache.org/content/repositories/orgapachezeppelin-1283/org/apache/zeppelin/
>>
>> You can find the KEYS file here: 
>> https://dist.apache.org/repos/dist/release/zeppelin/KEYS
>>
>> Release notes available at 
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342692==12316221
>>
>> Vote will be open for next 72 hours (close at 8am 26/July PDT).
>>
>> [ ] +1 approve
>> [ ] 0 no opinion
>> [ ] -1 disapprove (and reason why)
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>
>
> --
> With best wishes,Alex Ott
> http://alexott.net/
> Twitter: alexott_en (English), alexott (Russian)
>

Re: Blog of jdbc interpreter on zeppelin

[moon soo Lee]

Thank Jeff for sharing the article!


On Mon, Jul 6, 2020 at 5:38 PM Wesley  wrote:

>
>
> Jeff Zhang wrote:
> > I write a article about jdbc interpreter on zeppelin, hope it is helpful
> > for you.
> >
> >
> https://medium.com/@zjffdu/jdbc-interpreter-on-zeppelin-tutorial-8a7958b8c94e
> >
>
> Nice article. Thanks for share.
>
> regards.
>

Re: [Announce] New Zeppelin Committer: Philipp Dallig

[moon soo Lee]

It's exciting news because I was always impressed by high-quality code
contributions from Philipp Dalling.
Congratulations! Welcome aboard!

On Thu, Jun 25, 2020 at 11:23 PM Jeff Zhang  wrote:

>
> The Project Management Committee (PMC) for Apache Zeppelin
> has invited Philipp Dallig to become a committer and we are very pleased
> to announce that he has accepted.
>
> We greatly appreciate all of Philipp Dallig's hard work and generous
> contributions to the project. We look forward to continued involvement in
> the project.
>
> Congratulations & Welcome aboard Philipp Dallig !
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: intercepter does not work after pod restart

[moon soo Lee]

Regarding (1),

If an Interpreter is stateless (for example jdbc interpreter), they can be
silently restarted by Deployment.

However, if an Interpreter is stateful (like spark, python) and restarted
silently, user may experience unexpected behavior or result. It can be
quite confusing. So when an Interpreter is restarted itself by Deployment,
it should let the user know that it is restarted and all states may be
lost. But currently, the user does not have visibility through Zeppelin UI
to know whether the interpreter is restarted or not.

One reason why the interpreter is running as a Pod at the moment is, when
it is removed, Zeppelin UI will show some error to the user, so the user
can know the state is lost.

It would be great if it is restarted automatically and UI lets users know
about it.


Regarding (2),

When the interpreter is started and the connection is made, there are some
initialization processes  (invoke open()) initiated by Zeppelin Server.
When the Interpreter container is restarted and Zeppelin Server does not
aware of it, so the initialization process is not started on the new
container, there could be some error.  Not 100% sure if this is the case.

Thanks,
moon

On Sun, May 31, 2020 at 8:37 AM James Yu  wrote:

> I deploy zeppelin in Kubernetes environment and found zeppelin does not
> behave as expected in the following cases:
> (1) when I delete the interceptor pod with "kubectl delete", the pod does
> not restart itself
> (2) when I delete the interceptor container with "docker rm -f", the pod
> restarts itself, but zeppelin reports "Internal error processing interpret"
> error
>
> I tried these cases to see if zeppelin is able to heal itself in
> production environment.
> For case (1), the pod will definitely restart when deployed as deployment
> or at least managed by replication set controller.
> For case (2),  since the restarted pod has the same, I have not idea why
> zeppelin reports error. Perhaps zeppelin can access interpreter through
> service instead of using pod straightaway?
>
>
> This is a UTF-8 formatted mail
> ---
> James C.-C.Yu
> +886988713275
> +8615618429976
>

Re: [ANNOUNCE] Welcome Yadong Xie, Hsuan Lee as New Committer

[moon soo Lee]

Excited to see great work with new UI. Welcome Yadong Xie, Hsuan Lee.

On Wed, Apr 29, 2020 at 7:03 AM Jeff Zhang  wrote:

> The Project Management Committee (PMC) for Apache Zeppelin
> has invited Yadong Xie and Hsuan Lee  to become a committer and we are very
> pleased
> to announce that they have accepted.
>
> We greatly appreciate all of Yadong Xie and Hsuan Lee's hard work and
> generous contributions to the project, especially on the new angular UI
> implementation. We look forward to continued involvement in the project.
>
> Congratulations & Welcome aboard Yadong Xie and Hsuan Lee !
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: [VOTE] Release Apache Zeppelin 0.9.0-preview1 (RC1)

[moon soo Lee]

+1

Tested
 - The binary package can run tutorial notebook out of the box
 - Can build docker image
- scripts/docker/zeppelin/bin/Dockerfile give me some error on R
dependency installation. But I think it's not a blocker. We can address it
on next RC.
 - Zeppelin on Kubernetes
- Everything but Spark-2.4.5 does not run out of the box. I think it is
not a blocker. We can address it on next RC.




On Wed, Mar 25, 2020 at 11:21 AM Prabhjyot Singh 
wrote:

> +1
>
> On Wed, 25 Mar 2020 at 21:12, Eric Pugh 
> wrote:
>
> > Out of curiosity, is there a Docker version of 0.9.0 preview1 available?
> > I tend to run Zeppelin as a Docker image, and I know I had some
> challenges
> > with the 0.8.x line of Docker images from
> > https://hub.docker.com/r/apache/zeppelin.
> >
> > I did modify the ./scripts/docker/zeppelin/Dockerfile to load the
> > v0.0.0-preview1-rc1.bin.tgz distribution.
> >
> > Then I followed the steps in docs/setup/deployment/docker.md to try out
> > Zeppelin and it all appears to work as a Docker image!
> >
> > One thing to note was that it didn’t auto upgrade my old .json files into
> > the new .zpln files.  I had to manually upload them.  Is there any auto
> > upgrade feature for existing notebook files?
> >
> > So non binding +1 ;-)
> >
> > Eric
> >
> >
> >
> > On Mar 25, 2020, at 1:54 AM, Jongyoul Lee  wrote:
> >
> > +1
> >
> > 2020년 3월 24일 (화) 오전 12:15, Jeff Zhang 님이 작성:
> >
> >> Hi folks,
> >>
> >> I propose the following RC to be released for the Apache Zeppelin
> >> 0.9.0-preview1 release.
> >>
> >>
> >> The commit id is a74365c0813b451db1bc78def7d1ad1279429224 :
> >>
> >>
> https://gitbox.apache.org/repos/asf?p=zeppelin.git;a=commit;h=a74365c0813b451db1bc78def7d1ad1279429224
> >>
> >> This corresponds to the tag: v0.9.0-preview1-rc1 :
> >>
> >>
> https://gitbox.apache.org/repos/asf?p=zeppelin.git;a=shortlog;h=refs/tags/v0.9.0-preview1-rc1
> >>
> >> The release archives (tgz), signature, and checksums are here
> >>
> >>
> https://dist.apache.org/repos/dist/dev/zeppelin/zeppelin-0.9.0-preview1-rc1/
> >>
> >> The release candidate consists of the following source distribution
> >> archive
> >> zeppelin-v0.9.0-preview1.tgz
> >>
> >> In addition, the following supplementary binary distributions are
> provided
> >> for user convenience at the same location
> >> zeppelin-0.9.0-preview1-bin-all.tgz
> >>
> >>
> >> The maven artifacts are here
> >>
> >>
> https://repository.apache.org/content/repositories/orgapachezeppelin-1276/org/apache/zeppelin/
> >>
> >> You can find the KEYS file here:
> >> https://dist.apache.org/repos/dist/release/zeppelin/KEYS
> >>
> >> Release notes available at
> >>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342692==12316221
> >>
> >> Vote will be open for next 72 hours (close at 7am 26/March PDT).
> >>
> >> [ ] +1 approve
> >> [ ] 0 no opinion
> >> [ ] -1 disapprove (and reason why)
> >>
> >>
> >> --
> >> Best Regards
> >>
> >> Jeff Zhang
> >>
> >
> >
> > --
> > 이종열, Jongyoul Lee, 李宗烈
> > http://madeng.net
> >
> >
> > ___
> > *Eric Pugh **| *Founder & CEO | OpenSource Connections, LLC |
> 434.466.1467
> > | http://www.opensourceconnections.com | My Free/Busy
> > 
> > Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> > <
> https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw
> >
> > This e-mail and all contents, including attachments, is considered to be
> > Company Confidential unless explicitly stated otherwise, regardless
> > of whether attachments are marked as such.
> >
> >
>
> --
>
> Warm Regards,
>
> Prabhjyot Singh
>

Re: [DISCUSS] Release for 0.9.0-preview-1

[moon soo Lee]

Definitely +1

On Mon, Mar 16, 2020 at 8:02 AM Prabhjyot Singh 
wrote:

> Yes, +1, we have put in a lot of features. We should do it soon.
>
> On Mon, Mar 16, 2020, 8:28 PM Yadong Xie  wrote:
>
>> Big +1
>>
>> Eric Pugh  于2020年3月16日周一 下午10:54写道：
>>
>> > Big +1 from me.
>> >
>> >
>> > > On Mar 16, 2020, at 10:45 AM, Jeff Zhang  wrote:
>> > >
>> > > Hi Folks,
>> > >
>> > > I'd like to start the discussion of releasing of 0.9.0-preview-1. As
>> we
>> > discussed before releasing a preview version first is suitable for a
>> major
>> > release based on our previous experience[1]. The last major release
>> 0.8.0
>> > is in 2018, I do believe we should release 0.9 to allow more people try
>> > that and gather feedback. What do you think ?
>> > >
>> > >
>> > > [1]
>> >
>> https://lists.apache.org/thread.html/3812fd7dab53e1fed4140c826fc69fa5b718636371b6c7c677b4ae88%40%3Cdev.zeppelin.apache.org%3E
>> > <
>> >
>> https://lists.apache.org/thread.html/3812fd7dab53e1fed4140c826fc69fa5b718636371b6c7c677b4ae88%40%3Cdev.zeppelin.apache.org%3E
>> > >
>> > >
>> > > --
>> > > Best Regards
>> > >
>> > > Jeff Zhang
>> >
>> > ___
>> > Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 |
>> > http://www.opensourceconnections.com <
>> > http://www.opensourceconnections.com/> | My Free/Busy <
>> > http://tinyurl.com/eric-cal>
>> > Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <
>> >
>> https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw
>> >
>> >
>> > This e-mail and all contents, including attachments, is considered to be
>> > Company Confidential unless explicitly stated otherwise, regardless of
>> > whether attachments are marked as such.
>> >
>> >
>>
>

Re: Intellij announce Big Data Tools – Integration with Zeppelin

[moon soo Lee]

I really like this!

On Wed, Oct 16, 2019 at 8:03 AM Felix Cheung 
wrote:

> Cool!
>
> --
> *From:* Jeff Zhang 
> *Sent:* Wednesday, October 16, 2019 6:52:02 AM
> *To:* users 
> *Subject:* Intellij announce Big Data Tools – Integration with Zeppelin
>
>
> Here's the related details
>
>
> https://blog.jetbrains.com/scala/2019/10/16/meet-big-data-tools-spark-integration-and-zeppelin-notebooks-in-intellij-idea/
>
>
> https://plugins.jetbrains.com/plugin/12494-big-data-tools?_ga=2.41180706.1434705875.1571232785-884153734.1558949232
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: [ANNOUNCE] Apache Zeppelin 0.8.2 released

[moon soo Lee]

Great work!

On Sun, Sep 29, 2019 at 9:46 PM Prabhjyot Singh 
wrote:

> Awesome!
>
> Thank you so much Jeff for taking care of this.
>
> Congratulations community.
>
> On Sun, 29 Sep 2019, 2:15 pm Jeff Zhang,  wrote:
>
>> The Apache Zeppelin community is pleased to announce the availability of
>> the 0.8.2 release.
>>
>> Zeppelin is a collaborative data analytics and visualization tool for
>> distributed, general-purpose data processing system such as Apache Spark,
>> Apache Flink, etc.
>>
>> This is another minor release after the last minor release 0.8.1
>> The community put significant effort into improving Apache Zeppelin since 
>> the last release. 26 patches for improvements and bug fixes.
>>
>> We encourage you to download the latest release 
>> fromhttp://zeppelin.apache.org/download.html
>> Release note is available 
>> athttp://zeppelin.apache.org/releases/zeppelin-release-0.8.2.html
>>
>> We welcome your help and feedback. For more information on the project and
>> how to get involved, visit our website at http://zeppelin.apache.org/
>>
>> Thank you all users and contributors who have helped to improve Apache
>> Zeppelin.
>>
>> Regards,
>> The Apache Zeppelin community
>>
>>

Re: [DISCUSS] Release 0.8.2

[moon soo Lee]

+1

On Mon, Sep 23, 2019 at 8:16 PM Prabhjyot Singh 
wrote:

> +1
>
> I see a lot of issues are fixed in the past few months. This sounds a
> good intermediate step before 0.9 release.
>
> On Tue, 24 Sep 2019 at 00:23, Eric Pugh 
> wrote:
>
>> The iPython related fixes would address some issues I’ve run into!
>>
>> On Sep 22, 2019, at 11:31 PM, Jeff Zhang  wrote:
>>
>> Hi Folks,
>>
>> 0.8.1 is released half year ago (Jan 2019), I'd to start the discussion
>> about releasing of 0.8.2 which fixed several critical issues. Let me know
>> your thoughts, thanks.
>>
>>
>> https://jira.apache.org/jira/issues/?jql=project%20%3D%20ZEPPELIN%20AND%20fixVersion%20%3D%200.8.2
>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>>
>> ___
>> *Eric Pugh **| *Founder & CEO | OpenSource Connections, LLC | 434.466.1467
>> | http://www.opensourceconnections.com | My Free/Busy
>> 
>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>> 
>> This e-mail and all contents, including attachments, is considered to be
>> Company Confidential unless explicitly stated otherwise, regardless
>> of whether attachments are marked as such.
>>
>>
>
> --
> Regards,
> Prabhjyot Singh
>

Re: zeppelin-daemon doesn't work in docker container

[moon soo Lee]

Hao Ren,

It looks like other interpreters are working even if Zeppelin is started
inside the container using "bin/zeppelin-daemon.sh start" command, except
for Spark interpreter. So I guess it is somehow related to Spark.

One question, do you have any particular reason try to use
bin/zeppelin-daemon.sh script instead of bin/zeppelin.sh inside the
container?

Thanks,
moon

On Thu, Apr 4, 2019 at 11:38 PM Hao Ren  wrote:

> Here is the ps tree for the two cases.
> And I have no idea why Zeppelin fork some sub processes for STTY when
> Zeppelin is hanging
>
> * Working
> USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
> root 1  0.0  0.0   1144 4 ?Ss   08:24   0:00 /dev/init
> -- /usr/local/bin/bootstrap.sh
> root 7  0.0  0.0  48696  3132 ?S08:24   0:00 su -l
> h.ren
> h.ren   25  0.0  0.0  20144  3816 ?S08:25   0:00  \_ -su
> h.ren  279  0.0  0.0  36144  3272 ?R+   08:27   0:00  \_
> ps auxf -G1
> h.ren   79  6.0  1.5 6305268 520520 ?  Sl   08:25   0:10
> /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dfile.encoding=UTF-8 -Xms1024m
> -Xmx1024m -XX:MaxPermSize=512m -Dlog4j
> h.ren  133  0.0  0.0  11492  3124 ?S08:27   0:00  \_
> /bin/bash /app/zeppelin/bin/interpreter.sh -d
> /app/zeppelin/interpreter/spark -c 172.17.0.2 -p 45475 -r : -l
> h.ren  145  0.0  0.0  11492  2112 ?S08:27   0:00  \_
> /bin/bash /app/zeppelin/bin/interpreter.sh -d
> /app/zeppelin/interpreter/spark -c 172.17.0.2 -p 45475 -r :
> h.ren  146  110  2.0 14354788 670188 ? Sl   08:27   0:18
> \_ /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
> /app/zeppelin/interpreter/spark/*:/app/zeppelin/lib/in
>
> * Hanging
> USER   PID %CPU %MEMVSZ   RSS TTY  STAT START   TIME COMMAND
> root 1  0.1  0.0   1144 4 ?Ss   12:56   0:00 /dev/init
> -- /usr/local/bin/bootstrap.sh
> root 7  0.0  0.0  48696  3128 ?S12:56   0:00 su -l
> h.ren
> h.ren   25  0.0  0.0  20152  3832 ?S12:56   0:00  \_ -su
> h.ren  208  0.0  0.0  36144  3172 ?R+   12:57   0:00  \_
> ps auxf -g1
> h.ren   79 40.2  1.5 6238704 516152 ?  Tl   12:57   0:09
> /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dfile.encoding=UTF-8 -Xms1024m
> -Xmx1024m -XX:MaxPermSize=512m -Dlog4j
> h.ren  131  0.0  0.0  11492  3088 ?T12:57   0:00  \_
> /bin/bash /app/zeppelin/bin/interpreter.sh -d
> /app/zeppelin/interpreter/spark -c 172.17.0.2 -p 37333 -r : -l
> h.ren  143  0.0  0.0  11492  1980 ?T12:57   0:00  \_
> /bin/bash /app/zeppelin/bin/interpreter.sh -d
> /app/zeppelin/interpreter/spark -c 172.17.0.2 -p 37333 -r :
> h.ren  144 24.3  0.8 12053996 287476 ? Tl   12:57   0:03
> \_ /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp
> /app/zeppelin/interpreter/spark/*:/app/zeppelin/lib/in
> h.ren  205  0.0  0.0   4524   720 ?T12:57   0:00
> \_ sh -c stty -icanon min 1 -icrnl -inlcr -ixon < /dev/tty
> h.ren  206  0.0  0.0   6136   708 ?T12:57   0:00
> \_ stty -icanon min 1 -icrnl -inlcr -ixon
>
> On Thu, Apr 4, 2019 at 12:45 AM Hao Ren  wrote:
>
>> Hi,
>> I am testing zeppelin in docker container by using the official image:
>> https://hub.docker.com/r/apache/zeppelin/dockerfile
>>
>> I notice that the Dockfile above is using "bin/zeppelin.sh" to start
>> zeppelin, instead of "bin/zeppelin-daemon start".
>>
>> I am not clear on the difference between the two scripts. So I did some
>> tests.
>>
>> What I did:
>> * remove ENTRYPOINT and CMD from Dockfile
>> * build and run the container
>> * Inside the container terminal, use "bin/zeppelin-daemon.sh start" to
>> start Zeppelin
>>
>> The problem I met:
>> * Zeppelin can be launched
>> * However, when executing "spark.version" in notebook, Zeppelin hangs
>> * http://localhost:8080/#/interpreter does not show anything
>>
>> If I stop the daemon and use "bin/zeppelin.sh" to start Zeppelin,
>> everything works as expected.
>>
>> Further investigation shows that the two scripts seem to run the same
>> command, except "bin/zeppelin-daemon start" using "nohup nice" to run the
>> command in the background. So I tried to run "nohup nice bin/zeppelin.sh",
>> unsurprisingly, the problem mentioned above appears again.
>>
>> Once I bring the process to the foreground, the frozen page works again!
>>
>> Could anyone explain what has happened? I  am so confused about this. It
>> might be related to the way background process works in the docker
>> container. I am new to docker and Zeppelin.
>> Any help is highly appreciated.
>>
>> Thank you.
>>
>> --
>> Hao Ren
>>
>> Software Engineer in Machine Learning @ Criteo
>>
>> Paris, France
>>
>
>
> --
> Hao Ren
>
> Software Engineer in Machine Learning @ Criteo
>
> Paris, France
>

Re: [DISCUSS] Deprecate support for Spark 2.2.x and earlier version

[moon soo Lee]

+1

On Sat, Feb 16, 2019 at 06:52 Xun Liu  wrote:

> +1
>
>
> 在 2019年2月16日，下午2:44，Felix Cheung  写道：
>
> +1
>
>
> --
> *From:* Jeff Zhang 
> *Sent:* Thursday, February 14, 2019 10:28 PM
> *To:* users
> *Subject:* [DISCUSS] Deprecate support for Spark 2.2.x and earlier
> version
>
> Hi Folks,
>
> Spark 2.2.x will be EOL[1] from January of 2019. So I am considering to
> deprecate support for spark 2.2.x and earlier version in Zeppelin 0.9.0.
> Deprecation means that from Zeppelin 0.9 user is still able to run spark
> 2.2.x and earlier version, but will see a warning message in frontend about
> this deprecation. And in the next major version(maybe 0.10, or 1.0), we
> would totally remove support for spark 2.2.x and earlier version. The
> impact for users is the deprecation message in frontend. It may cause
> issues for users if they use rest api of zeppelin to run paragraph, then
> fetch and parse the result.
>
> Let me know your concern about this. Thanks
>
> [1] https://spark.apache.org/versioning-policy.html
>
>
> --
> Best Regards
>
> Jeff Zhang
>
>
>

Re: [VOTE] Release Apache Zeppelin 0.8.1 (RC1)

[moon soo Lee]

+1

Thanks Jeff

On Tue, Jan 22, 2019 at 01:07 Terry Wang  wrote:

> +1, thanks for your effort
>
> > 在 2019年1月20日，上午1:33，antonkul...@gmail.com 写道：
> >
> > +1
> > cause, I desperately need Spark 2.4.0 support
> >
> > On 2019/01/17 05:40:17, Jeff Zhang  wrote:
> >> I will start with my +1
> >>
> >> Jeff Zhang  于2019年1月17日周四 上午11:28写道：
> >>
> >>> Hi folks,
> >>>
> >>> I propose the following RC to be released for the Apache Zeppelin
> >>> 0.8.1 release.
> >>>
> >>>
> >>> The commit id is c46f55f1d81df944fd1b69a7ccb68d0647294543 :
> >>>
> >>>
> https://gitbox.apache.org/repos/asf?p=zeppelin.git;a=commit;h=c46f55f1d81df944fd1b69a7ccb68d0647294543
> >>>
> >>>
> >>> This corresponds to the tag: v0.8.1-rc1 : *
> https://gitbox.apache.org/repos/asf?p=zeppelin.git;a=shortlog;h=refs/tags/v0.8.1-rc1
> <
> https://gitbox.apache.org/repos/asf?p=zeppelin.git;a=shortlog;h=refs/tags/v0.8.1-rc1
> >*
> >>>
> >>>
> >>> The release archives (tgz), signature, and checksums are here *
> https://dist.apache.org/repos/dist/dev/zeppelin/zeppelin-0.8.1-rc1/ <
> https://dist.apache.org/repos/dist/dev/zeppelin/zeppelin-0.8.1-rc1/>*
> >>>
> >>> The release candidate consists of the following source distribution
> archive
> >>> zeppelin-0.8.1.tgz
> >>>
> >>> In addition, the following supplementary binary distributions are
> provided
> >>> for user convenience at the same location
> >>> zeppelin-0.8.1-bin-all.tgz
> >>>
> >>>
> >>> The maven artifacts are here
> https://repository.apache.org/content/repositories/orgapachezeppelin-1269/org/apache/zeppelin/
> >>> You can find the KEYS file here:
> >>>
> >>> https://dist.apache.org/repos/dist/release/zeppelin/KEYS
> >>>
> >>> Release notes available at
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12343240==12316221
> >>> Vote will be open for next 72 hours (close at 7pm 19/Jan PDT).
> >>>
> >>> [ ] +1 approve
> >>> [ ] 0 no opinion
> >>> [ ] -1 disapprove (and reason why)
> >>>
> >>>
> >>>
> >>
> >> --
> >> Best Regards
> >>
> >> Jeff Zhang
> >>
>
>

Re: [Discuss] 0.8.1 Release

[moon soo Lee]

+1

On Tue, Jan 15, 2019 at 18:28 Bicky Ealias  wrote:

> Keen to see
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/ZEPPELIN-3800
>
> Hope it's coming in the new version.
>
> On Wed., 16 Jan. 2019, 1:13 pm Felix Cheung  wrote:
>
>> +1 thanks!
>>
>>
>> --
>> *From:* Jeff Zhang 
>> *Sent:* Tuesday, January 15, 2019 5:39 PM
>> *To:* users
>> *Subject:* Re: [Discuss] 0.8.1 Release
>>
>> Hi Folks,
>>
>> I will start the 0.8.1 release since there's no concerns on this.
>>
>> Jeff Zhang  于2019年1月14日周一 下午8:50写道：
>>
>>> Hi Everyone,
>>>
>>> Sorry for replying this mail so late, I am busy with other stuff. I plan
>>> to start the 0.8.1 release this week, does anyone have any concerns ?
>>>
>>>
>>>
>>> Jeff Zhang  于2018年10月10日周三 上午8:31写道：
>>>
 Community is working on that.

 Paul Brenner 于2018年10月10日周三 上午12:35写道：

> I would second this if it doesn’t hold up the release too much. We
> would love to see this implemented.
>
>  
>  *Paul Brenner* 
>  
>  
> 
> 
> SR. DATA SCIENTIST
> *(217) 390-3033 <(217)%20390-3033>*
>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> [image:
> PlaceIQ:CES 2018]
> 
> On Oct 9, 2018, 12:33 PM -0400, Павел Мяснов ,
> wrote:
>
> I know it is marked as improvement, not a bug; but is it possible to
> include ticket https://jira.apache.org/jira/browse/ZEPPELIN-3307 to
> this minor release? I hit this problem quite often and it would be really
> nice to see this problem solved soon.
>
> Pavel Myasnov
>
> On 2018/09/27 07:08:31, Jeff Zhang  wrote:
> > Hi folks,>
> >
> > It has been a while for 0.8.0 release, and we got many feedback for
> that,>
> > so I think it is time for us to make 0.8.1 release for fix the bugs
> of>
> > 0.8.0. Here's the umbrella tickets for 0.8.1 release>
> > https://jira.apache.org/jira/browse/ZEPPELIN-3629>
> >
> > If you find any ticket that is necessary for 0.8.1 but not under
> this>
> > umbrella ticket, feel free to link that. I will start the 0.8.1
> release at>
> > the beginning of Oct.>
> >
>
>
>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>

Re: [DISCUSS] Is interpreter binding necessary ?

[moon soo Lee]

+1 for remove binding. But keep ability to set default interpreter and
restart interpreter (in note level)

On Sat, Jul 7, 2018 at 01:21 Ruslan Dautkhanov  wrote:

> That interpreting binding UI was also used to restart interpreters that
> our users were using form time to time.
> If we remove interpreter binding, there should still be a way to restart
> them quickly without going deep into menus.
>
> And also there should be a way to show interpreter status, like for
> example proposed in https://github.com/apache/zeppelin/pull/2974
>
> --
> Ruslan Dautkhanov
>
>
> On Fri, Jul 6, 2018 at 9:56 AM Ruslan Dautkhanov 
> wrote:
>
>> +1 to remove it
>>
>> Setting default interpreter is not very useful anyway (for example, we
>> can't make %pyspark default without manually editing xml files in Zeppelin
>> distro). https://issues.apache.org/jira/browse/ZEPPELIN-3282
>>
>> --
>> Ruslan Dautkhanov
>>
>>
>> On Fri, Jul 6, 2018 at 7:27 AM Paul Brenner  wrote:
>>
>>> I agree with Partridge. We have different interpreters defined with
>>> different queues and settings. So we need a way to quickly change the
>>> default interpreter and can’t rely on typing the desired interpreter at the
>>> start of each paragraph.
>>>
>>>
>>> 
>>> 
>>> 
>>>  *Paul
>>> Brenner*
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> SR. DATA SCIENTIST
>>> *(217) 390-3033 *
>>>
>>> 
>>> 
>>> 
>>> 

Re: Spark SQL Tutorial in Apache Zeppelin Notebook

[moon soo Lee]

Cool! Thanks Jeff for sharing!

Thanks,
moon

On Sun, Jun 3, 2018 at 6:34 PM Jeff Zhang  wrote:

>
> Hi Folks,
>
> I wrote 2 tutorials for Spark SQL in Apache Zeppelin (Scala & PySpark),
> hope it is useful for you.
>
>
> https://medium.com/@zjffdu/spark-sql-tutorial-in-apache-zeppelin-notebook-2a210d2f4ff1
>
> BTW, I would also encourage people in the community to share their
> experience of using zeppelin in whatever kind of format. Don't be shy or
> worry if you think your usage is simple, sharing is another kind of
> learning which is more efficient sometimes.
>
>
>
>
>

Re: Highlight Zeppelin 0.8 New Features

[moon soo Lee]

Looks great. I think online registry (helium) for visualization and spell
is another important feature.

Thanks,
moon

On Tue, Mar 13, 2018 at 12:41 AM Jeff Zhang  wrote:

>
> I planned to publish this article after 0.8 release, but I think it would
> be helpful for users to experience and verify these features before 0.8
> release. So I sent it out before 0.8 release. I would be very appreciated
> that users can try these features via branch-0.8, and this is not a full
> list of 0.8 new features, feel free to add if I miss any important features.
>
> https://medium.com/@zjffdu/zeppelin-0-8-0-new-features-ea53e8810235
>
>
>

Re: File upload

[moon soo Lee]

I saw google colab let user upload/download file directly from notebook
(browser) to filesystem of an execution environment. Not sure what Jens
mentioning is the same or not, but this direct file upload/download
capability from browser is very useful i think.

Thanks,
moon

On Wed, Mar 7, 2018 at 5:36 PM Jeff Zhang  wrote:

>
> Could you be more specific on this ? What kind of interpreter do you use ?
> What kind of file you want to upload and what would like to do on this file
> after you upload ?
>
> On Wed, Mar 7, 2018 at 11:40 PM, Jens Grassel  wrote:
>
>> Hi,
>>
>> where would I need to look regarding documentation on how to implement
>> upload functionality in a zeppelin notebook.
>>
>> The basic idea is to pick a file and pipe this either onto a server
>> directory or directly into an interpreter if possible.
>>
>> Regards,
>>
>> Jens
>>
>> --
>> CTO, Wegtam GmbH, 07. Lenzing 2018, 16:38
>> Homepage : https://www.wegtam.com
>>
>> Secrecy is the beginning of tyranny.
>>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: [DISCUSS] Roadmap 0.9 and future

[moon soo Lee]

I think there were couple of discussions about 0.8.0 release in the
community. We have umbrella issue that tracks 0.8.0 release [1]. And i see
not many blockers are left. Jeff volunteered the release manager for 0.8.0
and I think he can give better idea of 0.8.0 release estimate.

Thanks,
moon

[1] https://issues.apache.org/jira/browse/ZEPPELIN-2385


On Wed, Mar 7, 2018 at 10:49 AM Paul Brenner <pbren...@placeiq.com> wrote:

> Thanks for sharing the results of the meeting!
>
> Regarding the point "Need more frequent release”, was there any discussion
> around when 0.8.0 will be officially released? I remember a message a few
> months ago suggesting that it might be soon.
>
> <http://www.placeiq.com/> <http://www.placeiq.com/>
> <http://www.placeiq.com/> *Paul Brenner* <https://twitter.com/placeiq>
> <https://twitter.com/placeiq> <https://twitter.com/placeiq>
> <https://www.facebook.com/PlaceIQ> <https://www.facebook.com/PlaceIQ>
> <https://www.linkedin.com/company/placeiq>
> <https://www.linkedin.com/company/placeiq>
> DATA SCIENTIST
> *(217) 390-3033 <(217)%20390-3033> *
>
> <http://www.placeiq.com/2015/05/26/placeiq-named-winner-of-prestigious-2015-oracle-data-cloud-activate-award/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2016/03/08/measuring-addressable-tv-campaigns-is-now-possible/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://pages.placeiq.com/Location-Data-Accuracy-Whitepaper-Download.html?utm_source=Signature_medium=Email_campaign=AccuracyWP>
> <http://placeiq.com/2016/08/03/placeiq-bolsters-location-intelligence-platform-with-mastercard-insights/>
> <http://placeiq.com/2016/10/26/the-making-of-a-location-data-industry-milestone/>
> <http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/>
> <http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/>
> <https://www.placeiq.com/2017/05/placeiqs-landmark-powers-location-based-insight-innovation-for-ansible-gstv-havas-media-the-media-kitchen-and-more/>
> <http://pages.placeiq.com/CES2018_MeetingRequest.html>[image: PlaceIQ:CES
> 2018]
> <http://pages.placeiq.com/2017-Integrated-Marketing-Whitepaper_LP_Download.html>
>
> On Wed, Mar 07, 2018 at 1:21 PM moon soo Lee  <moon+soo+lee+%3cm...@apache.org%3E>> wrote:
>
>> Hi forks,
>>
>> There were an offline meeting yesterday at PaloAlto with contributors and
>> users. We've shared idea about current state of project and future project
>> roadmap and wishlists (meeting note [1]). It was really inspiring and
>> exciting time. Let me try summarize, move this discussion to online.
>>
>> There were many ideas related to Interpreter. Especially, there were
>> consensus that Spark support is one of biggest strength of Zeppelin and
>> need to make further improvement to keep the strengths.
>>
>>- Spark
>>- Immediate support of new spark release
>>   - Ramp up support of current Spark feature (e.g. Display job
>>   progress correctly)
>>   - Spark streaming support
>>   - Handling Livy timeout
>>   - Other interpreters
>>- Better Hive support (e.g. configuration)
>>   - Latest version PrestoDB support (pass property correctly)
>>   - Run interpreter in containerized environment
>>- Let individual user upload custom library from user's machine
>>directly
>>- Interpreter documentation is not detail enough
>>
>> And people in the meeting excited about ConInterpreter ZEPPELIN-3085 [2]
>> in upcoming release, regarding dynamic/inlin

[DISCUSS] Roadmap 0.9 and future

[moon soo Lee]

Hi forks,

There were an offline meeting yesterday at PaloAlto with contributors and
users. We've shared idea about current state of project and future project
roadmap and wishlists (meeting note [1]). It was really inspiring and
exciting time. Let me try summarize, move this discussion to online.

There were many ideas related to Interpreter. Especially, there were
consensus that Spark support is one of biggest strength of Zeppelin and
need to make further improvement to keep the strengths.

   - Spark
   - Immediate support of new spark release
  - Ramp up support of current Spark feature (e.g. Display job progress
  correctly)
  - Spark streaming support
  - Handling Livy timeout
  - Other interpreters
   - Better Hive support (e.g. configuration)
  - Latest version PrestoDB support (pass property correctly)
  - Run interpreter in containerized environment
   - Let individual user upload custom library from user's machine directly
   - Interpreter documentation is not detail enough

And people in the meeting excited about ConInterpreter ZEPPELIN-3085 [2] in
upcoming release, regarding dynamic/inline configuration of interpreter.

And there were ideas on other areas, too. like

   - Separate Admin role and user role
   - Sidebar with plugin widget
   - Better integration with emerging framework like Tensorflow/MXNet/Ray
   - Sharing data
   - Schedule notebook from external scheduler

Regarding scheduling notebook, Luciano shared his project NotebookTools[3]
and it made people really excited.

Also, there were inspiring discussions about the community/project. Current
status and how can we make community/project more healthy. And here's some
ideas around the topic

   - Need more frequent release
   - More attention to code review to speed up
   - Publishing roadmap beforehand to help contribution
   - 'Newbie', 'low hanging fruit' tag helps contribution
   - Enterprise friendly is another biggest strength of Zeppelin (in
   addition to Spark support) need to keep improve.


I probably missed many idea shared yesterday. Please feel free to
add/correct the summary. Hope more people in the mailinglist join and
develop the idea together. And I think this discussion can leads community
shape 0.9 and future version of Zeppelin, and update and publish future
roadmap[4].

Best,
moon

Special thanks to ZEPL  for the swag and dinner.

[1]
https://docs.google.com/document/d/18Wc3pEFx3qm9XoME_V_B9k_LlAd1PLyKQQEveR1on1o/edit?usp=sharing
[2] https://issues.apache.org/jira/browse/ZEPPELIN-3085
[3]
https://github.com/SparkTC/notebook-exporter/tree/master/notebook-exporter
[4] https://cwiki.apache.org/confluence/display/ZEPPELIN/Zeppelin+Roadmap

Re: How to pass a note to Zeppelin on invocation

[moon soo Lee]

Hi,

Currently we don't have a way I think. But it will be really nice to have.
Especially with https://issues.apache.org/jira/browse/ZEPPELIN-2619, it'll
be even better i think.

Thanks,
moon

On Tue, Feb 6, 2018 at 8:10 AM Leon Katsnelson  wrote:

> When we use Jupyter and JupyterLab we are able to pass the notebook as a
> parameter so that when the Jupyter starts the notebook is already loaded.
> We would like to do the same with Zeppelin 0.7.3.
>
> Is it possible and if so, how?
>

Re: Importing notes

[moon soo Lee]

I'm not sure this helps but there's a rest api endpoint that reload the
notebooks from notebook storage.
https://github.com/apache/zeppelin/blob/master/zeppelin-server/src/main/java/org/apache/zeppelin/rest/NotebookRepoRestApi.java#L82

Thanks,
moon

On Wed, Jan 31, 2018 at 2:44 AM Ricardo Martinelli de Oliveira <
rmart...@redhat.com> wrote:

> I did a quick research in the configuration and the code and the store
> type is what defines the way the notes will be stored. The default is Git,
> which means it will be stored in the notebook directory and can be
> versioned using git.
>
> However, I see that VFS implementation is supported and by VFS I was
> wondering if there is a mechanism that scans the notebook directory and
> look for new notebooks.
>
> On Tue, Jan 30, 2018 at 10:43 PM, Michael Segel  > wrote:
>
>> I don’t think you can…
>>
>> If you look in the ../notebook directory, the notes are all identified by
>> an unique id.
>>
>> My guess? That the references are stored within a database. (Derby? )
>>
>>
>> > On Jan 30, 2018, at 6:25 PM, Ricardo Martinelli de Oliveira <
>> rmart...@redhat.com> wrote:
>> >
>> > Hello,
>> >
>> > Is there a way to import notes from a command line? I am trying to
>> write a script that imports all my notes in a zeppelin docker container,
>> but the only ways to import is from the web UI or REST. In the script,
>> zeppelin is not started (there is no reason to be) so I'm trying to find a
>> way to import the note in a disconnected way.
>> >
>> > Any ideas to do this?
>>
>>
>
>
> --
> Ricardo Martinelli de Oliveira
> Senior Software Engineer
> T: +55 11 3524-6125 <+55%2011%203524-6126> | M: +55 11 9 7069-6531
> Av. Brigadeiro Faria Lima 3900, 8° Andar. São Paulo, Brasil
> 
> .
> 
> TRIED. TESTED. TRUSTED. 
>
>  Red Hat é reconhecida entre as melhores empresas para trabalhar no Brasil
> pelo *Great Place to Work*.
>

Re: Demo: Zeppelin Table of Contents Spell

[moon soo Lee]

Wow!
I didn't expected that Spell can be used in this way! I think it opens new
possibility.
Thanks for sharing!

Regards,
moon

On Mon, Oct 16, 2017 at 11:21 AM Cyril Scetbon 
wrote:

> Nice !
>
> On Oct 16, 2017, at 1:56 PM, Ryan  wrote:
>
> Check it out!
>
> This feature was the biggest thing stopping me from switching from Jupyter
> to Zeppelin, it makes it very easy to navigate large notebooks.
>
> Don’t look at the code, I wrote it from within the chrome developer
> console lol.
>
> https://github.com/Submersible/zeppelin-toc-spell
>
>
>

Re: sql paragraph doesn't see my 3rd party jars

[moon soo Lee]

Could you share little more about how did you add 3rd party jars?

ie. through %spark.dep, Interpreter setting menu on GUI, environment
variable, and so on.

Thanks,
moon

On Sat, Oct 7, 2017 at 3:47 AM Serega Sheypak 
wrote:

> Hi, I'm trying to use spark and sql paragraphs with 3rd party jars added
> to spark interpreter configuration.
>
> My spark code works fine.
>
>
> My sql paragraph fails with class not found exception
> %sql
> create external table MY_TABLE row format serde 'com.my.MyAvroSerde'
> with serdeproperties ('serialization.class'='com.my.ContainerProto')
> stored as inputformat 'com.my.ProtoAvroFileFormat'
> LOCATION 'hdfs://my/data'
>
> Exception:
> MetaException(message:org.apache.hadoop.hive.serde2.SerDeException
> java.lang.ClassNotFoundException: Class com.my.ContainerProto not found)
>
>
> It's confusing since spark paragraph works well with
> code
>
> import com.my.ContainerProto
> // bla-bla
> rdd.map{bytes => ContainerProto.fromBytes(bytes)}
>
> Code executed and result produced. Why sql paragraph doesn't see my 3rd
> party jars?
>
>
>
>

Re: question about how to update zeppelin interpreters.

[moon soo Lee]

Hi,

I think id value of interpreter is just randomly generated ID [1]. You can
find ID generation routine here [2].

Thanks,
moon

[1]
https://github.com/apache/zeppelin/blob/v0.7.3/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java#L106
[2]
https://github.com/apache/zeppelin/blob/v0.7.3/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/utility/IdHashes.java#L74

On Tue, Oct 3, 2017 at 9:46 PM Jeffrey Rodriguez 
wrote:

> Hi folks,
>I would like to upgrade the zeppelin interpreter properties
> programmatically. I found two ways.
> 1. Update the conf/interpreters.json
> 2. User the interpreter REST API.
>
> My question is, should users/developers update the interpreter directly?
>
> id values for interpreters are like "2CVTZCCU4'", which seems to me not
> very ramdom and more like a signature or class hash.
>
> Is the prefer way to maintain consistency, the REST API??
>
> Regards,
>Jeff Rodriguez
>

Re: Implementing run all paragraphs sequentially

[moon soo Lee]

d in note and the running order should not depend on how we
>>> launch it (button / scheduler / API). In this way, our objectives are to
>>> implement “dependency definition engine” and to use it in “run engine”.
>>> What are the options?
>>> 1)  Explicit dependency definition.
>>> We could take for a rule that each paragraph should wait for the end of
>>> execution of ALL previous paragraphs. Then we add paragraph option “Wait
>>> for …” where we can choose paragraph for which we are waiting for to start
>>> execution. In case where the option is set, we start execution immediately
>>> after the end of execution of selected paragraph. This pattern allows us to
>>> implement full-parallel DAG running order. What are the disadvantages? All
>>> of them are about the same – not easy understanding of the dependency
>>> management process from the perspective of users (and probably redundancy
>>> of the functionality – my personal view). At first, we should use strange
>>> format of paragraph IDs, which in addition is hidden. We could come up with
>>> visible and handsome paragraph ID aliases, but then it appears necessity of
>>> duplication control. The second thing is in some kind of scenarios where we
>>> should change existing dependencies (e.g. you need to add new paragraph
>>> between one and dependent group – you have to change option “Wait for …”
>>> for each paragraph in group).
>>> 2)  Implicit dependency definition.
>>>
>>> We could take for a rule that each paragraph should wait for the end of
>>> execution of ALL previous paragraphs. Then we add paragraph option “Run in
>>> parallel with previous” which allows us to create paragraph groups to run
>>> in parallel. It turns out that we have the way of sequential running of
>>> paragraph groups – group by group in which paragraphs run in parallel. This
>>> approach is much more understandable for the users, but the obvious defect
>>> in comparison with “Explicit definition” is the fact that dependency graph
>>> and level of parallelism are not so cool.
>>> I am not sure which option (1) or (2) is correct to implement at the
>>> moment. I hope to hear from product visionaries which way to choose and to
>>> get approval for the start of implementation.
>>> Thank you!
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *Valeriy Polyakov *
>>>
>>>
>>> *From:* Michael Segel [mailto:msegel_had...@hotmail.com
>>> <msegel_had...@hotmail.com>]
>>> *Sent:* Saturday, September 30, 2017 4:22 PM
>>> *To:* users@zeppelin.apache.org
>>> *Subject:* Re: Implementing run all paragraphs sequentially
>>>
>>>
>>> Sorry to jump in…
>>>
>>>
>>> If you want to run paragraphs in parallel, you are going to want to have
>>> some sort of dependency graph.  Think of a common set up where you need to
>>> set up common functions and imports. (setup of %spark.dep)
>>>
>>>
>>> A good example is if your notebook is a bunch of unit tests and you need
>>> to build the common tear down / set up methods to be used by the other
>>> paragraphs.
>>>
>>>
>>> If you’re going to do that, you’ll need to build out a metadata
>>> structure where you can set up your dependencies  as well as add things
>>> like labels beyond the ids (which only need to be unique to the given
>>> notebook. )
>>>
>>>
>>> Just my $0.02
>>>
>>>
>>>
>>> On Sep 29, 2017, at 1:30 PM, moon soo Lee <m...@apache.org> wrote:
>>>
>>>
>>> Current behavior is as parallel as possible.
>>> Run notebook button currently submits all paragraphs in a notebook into
>>> each interpreter's own scheduler (FIFO, Parallel) at once. And each
>>> individual scheduler of interpreter runs the paragraphs.
>>>
>>>
>>> I think we can provide "sequential" run button for easier use, which
>>> submits paragraph one and waits for finish before submit next paragraphs.
>>>
>>>
>>> And I think sequential run button doesn't stop having more complex /
>>> flexible DAG in the future?
>>>
>>>
>>> Thanks,
>>> moon
>>>
>>>
>>> On Fri, Sep 29, 2017 at 10:08 AM Mohit Jaggi <mohitja...@gmail.com>
>>> wrote:
>>>
>>> What is the current behavior?
>>>
>>>
>

Re: Implementing run all paragraphs sequentially

[moon soo Lee]

Current behavior is as parallel as possible.
Run notebook button currently submits all paragraphs in a notebook into
each interpreter's own scheduler (FIFO, Parallel) at once. And each
individual scheduler of interpreter runs the paragraphs.

I think we can provide "sequential" run button for easier use, which
submits paragraph one and waits for finish before submit next paragraphs.

And I think sequential run button doesn't stop having more complex /
flexible DAG in the future?

Thanks,
moon

On Fri, Sep 29, 2017 at 10:08 AM Mohit Jaggi <mohitja...@gmail.com> wrote:

> What is the current behavior?
>
> On Fri, Sep 29, 2017 at 6:56 AM, Herval Freire <hfre...@twitter.com>
> wrote:
>
>> At least in our case, the notebooks that we need to run sequentially are
>> expected to *always* run sequentially - thus it makes more sense to be a
>> note option than a per-run mode
>>
>> H
>>
>> _
>> From: moon soo Lee <m...@apache.org>
>> Sent: Thursday, September 28, 2017 9:03 PM
>> Subject: Re: Implementing run all paragraphs sequentially
>> To: <users@zeppelin.apache.org>
>>
>>
>> This is going to be really useful!
>>
>> Curios why do you prefer 'note option' instead of 'run option'?
>> Could you compare their pros and cons?
>>
>> Thanks,
>> moon
>>
>> On Thu, Sep 28, 2017 at 8:32 AM Herval Freire <hfre...@twitter.com>
>> wrote:
>>
>>> +1, our internal users at Twitter also often request this
>>>
>>> --
>>> *From:* Belousov Maksim Eduardovich <m.belou...@tinkoff.ru>
>>> *Sent:* Thursday, September 28, 2017 8:28:58 AM
>>> *To:* users@zeppelin.apache.org
>>> *Subject:* Implementing run all paragraphs sequentially
>>>
>>>
>>> Hello, users!
>>>
>>> At the moment our analysts often use mixes of interpreters in their
>>> notes.
>>>
>>> For example, they prepare data using %jdbc and then use it in %pyspark.
>>> Besides, they often use scheduling to make some regular reporting. And they
>>> should do something like `time.sleep()` to wait for the data from %jdbc. It
>>> doesn`t guarantee the result and doesn`t look cool.
>>>
>>>
>>>
>>> You can find early attempts to implement sequential running of all
>>> paragraphs in [1].
>>>
>>> We are really interested in implementation of the issue [2] and are
>>> ready to solve it.
>>>
>>> It seems a good idea to discuss any requirements.
>>>
>>> My idea is to introduce note setting that defines the type of running to
>>> use (parallel or sequential) and leave "Run all" to be the only button
>>> running all the cells in the note. This will make sequential or parallel
>>> running the `note option` but not `run option`.
>>>
>>> Option will be controlled by nearby button as shown
>>>
>>> [image: image002.jpg]
>>>
>>>
>>>
>>>
>>>
>>> For new notes the default state would be "Run sequential all", for old -
>>> "Run parallel for interpreters"
>>>
>>> We are glad to hear any thoughts.
>>>
>>> Thank you.
>>>
>>>
>>>
>>> [1] https://issues.apache.org/jira/browse/ZEPPELIN-1165
>>>
>>> [2] https://issues.apache.org/jira/browse/ZEPPELIN-2368
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *Maksim Belousov *
>>>
>>>
>>>
>>
>>
>>
>

Re: Implementing run all paragraphs sequentially

[moon soo Lee]

This is going to be really useful!

Curios why do you prefer 'note option' instead of 'run option'?
Could you compare their pros and cons?

Thanks,
moon

On Thu, Sep 28, 2017 at 8:32 AM Herval Freire  wrote:

> +1, our internal users at Twitter also often request this
>
> --
> *From:* Belousov Maksim Eduardovich 
> *Sent:* Thursday, September 28, 2017 8:28:58 AM
> *To:* users@zeppelin.apache.org
> *Subject:* Implementing run all paragraphs sequentially
>
>
> Hello, users!
>
> At the moment our analysts often use mixes of interpreters in their notes.
>
> For example, they prepare data using %jdbc and then use it in %pyspark.
> Besides, they often use scheduling to make some regular reporting. And they
> should do something like `time.sleep()` to wait for the data from %jdbc. It
> doesn`t guarantee the result and doesn`t look cool.
>
>
>
> You can find early attempts to implement sequential running of all
> paragraphs in [1].
>
> We are really interested in implementation of the issue [2] and are ready
> to solve it.
>
> It seems a good idea to discuss any requirements.
>
> My idea is to introduce note setting that defines the type of running to
> use (parallel or sequential) and leave "Run all" to be the only button
> running all the cells in the note. This will make sequential or parallel
> running the `note option` but not `run option`.
>
> Option will be controlled by nearby button as shown
>
> [image: image002.jpg]
>
>
>
>
>
> For new notes the default state would be "Run sequential all", for old -
> "Run parallel for interpreters"
>
> We are glad to hear any thoughts.
>
> Thank you.
>
>
>
> [1] https://issues.apache.org/jira/browse/ZEPPELIN-1165
>
> [2] https://issues.apache.org/jira/browse/ZEPPELIN-2368
>
>
>
>
>
>
>
>
> *Maksim Belousov *
>
>
>

Re: Managing credentials question

[moon soo Lee]

Sounds like a plan!

On Mon, Sep 25, 2017 at 11:33 AM Herval Freire <hfre...@twitter.com> wrote:

> I started something on that direction here, for internal use:
> https://github.com/herval/zeppelin/tree/encrypt-credentials
>
> If that's the kind of thing that may interest everyone else, I can get a
> PR going
>
> h
>
> On Mon, Sep 25, 2017 at 7:07 AM, Adam Iezzi <adam.ie...@gmail.com> wrote:
>
>> Yes, encrypting and storing the credentials would be ideal. Essentially,
>> I'm looking for some sort of secrets store which can be accessed via the
>> Zeppelin paragraphs.
>>
>> Adam
>>
>> On Sun, Sep 24, 2017 at 6:30 AM, moon soo Lee <m...@apache.org> wrote:
>>
>>> Hi,
>>>
>>> "Credential" menu provides closest feature I think.
>>>
>>> Through "Credential" menu, each user can pass user-specific credential
>>> informations to Interpreters. And interpreter can retrieve those
>>> informations and use it internally. Also interpreter exposes API to user,
>>> so user can access those informations in Python, Scala, etc.
>>>
>>> Current limitation is, credential menu store it's information in memory
>>> only or in file without encryption.
>>>
>>> If "Credential" menu store credential in a file with encryption, does
>>> this solve your problem?
>>>
>>> Thanks,
>>> moon
>>>
>>> On Wed, Sep 20, 2017 at 4:06 PM Adam Iezzi <adam.ie...@gmail.com> wrote:
>>>
>>>> I'm trying to figure out the best way (and most secure) to use
>>>> user-specific credentials for various data stores. For example, I have a
>>>> few python paragraphs setup to query an external MySQL DB using
>>>> python's mysql.connector package. In order to establish the connection, I
>>>> have to add the DB username/password as arguments in my paragraph, which is
>>>> probably not the most secure approach.
>>>>
>>>> I'm wondering if there is a way to store these credentials somewhere
>>>> else (not in clear text in my notebook), so they can be referenced via the
>>>> notebook paragraphs in a more secure way? Or better yet, is there another
>>>> way to solve this issue that I may be missing?
>>>>
>>>> Thank you for all of the help.
>>>>
>>>> Adam
>>>>
>>>
>>
>

Re: Managing credentials question

[moon soo Lee]

Hi,

"Credential" menu provides closest feature I think.

Through "Credential" menu, each user can pass user-specific credential
informations to Interpreters. And interpreter can retrieve those
informations and use it internally. Also interpreter exposes API to user,
so user can access those informations in Python, Scala, etc.

Current limitation is, credential menu store it's information in memory
only or in file without encryption.

If "Credential" menu store credential in a file with encryption, does this
solve your problem?

Thanks,
moon

On Wed, Sep 20, 2017 at 4:06 PM Adam Iezzi  wrote:

> I'm trying to figure out the best way (and most secure) to use
> user-specific credentials for various data stores. For example, I have a
> few python paragraphs setup to query an external MySQL DB using
> python's mysql.connector package. In order to establish the connection, I
> have to add the DB username/password as arguments in my paragraph, which is
> probably not the most secure approach.
>
> I'm wondering if there is a way to store these credentials somewhere else
> (not in clear text in my notebook), so they can be referenced via the
> notebook paragraphs in a more secure way? Or better yet, is there another
> way to solve this issue that I may be missing?
>
> Thank you for all of the help.
>
> Adam
>

Re: Encript Users in shiro.ini

[moon soo Lee]

Hi,

You can encrypt password in shiro.ini. Check
http://shiro.apache.org/configuration.html#Configuration-EncryptingPasswords
http://shiro.apache.org/command-line-hasher.html

Hope this helps.

Thanks,
moon


On Wed, Sep 20, 2017 at 4:38 PM Carlos Andres Zambrano Barrera <
cza...@gmail.com> wrote:

> Hi Everyone,
>
> I am working with EMR with Zeppelin Version 0.7.0, and I want to know if
> I could encript the users in shiro.ini or encript the complete file
> shiro.ini.
>
> I do not have LDAP.
>
> --
> Ing. Carlos Andrés Zambrano Barrera
> Cel: 3123825834 <(312)%20382-5834>
>
>
>
>
>
>
>  Sent with Mailtrack
> 
> <#m_7557984433985723577_>
>

Re: Downloading Zeppelin 0.7.1

[moon soo Lee]

Thanks for reporting the problem.

Just fixed the broken link https://github.com/apache/zeppelin/pull/2570
and deployed the fix. Please refresh browser and try again.

Best,
moon

On Wed, Sep 6, 2017 at 2:26 AM Luis Angel Vicente Sanchez <
zeppelin-us...@bigcente.ch> wrote:

> Hi,
>
> I'm trying to recreate a docker container that is based on Zeppelin
> 0.7.1 but it seems that the tar file doesn't exist anymore in any of the
> mirrors. Are all older versions gone?
>
> Kind regards,
>
> Luis Angel Vicente Sanchez
> l...@bigcente.ch
>

Re: How to build zeppelin-*-bin-netinst.tgz package?

[moon soo Lee]

Hi,

dev/create_release.sh builds release artifacts [1]. Please take a look.

[1]
https://github.com/apache/zeppelin/blob/master/dev/create_release.sh#L110


On Sun, Aug 27, 2017 at 9:35 PM Tao Wang  wrote:

> Hi,
>
> I followed the instructions at
> http://zeppelin.apache.org/docs/snapshot/install/build.html
>
> And by using command:
>
> mvn clean package -Pbuild-distr -DskipTests -Pspark-2.1 -Phadoop-2.7
> -Pr -Pscala-2.11 -Dspark.version=2.1.1
>
> I built a zeppelin-0.7.2.tar.gz package, however, by look at the size,
> it's 700MB, so it's the package contains all the interpreters.
> However, I want to recreate the package
> 'zeppelin-0.7.2-bin-netinst.tga', may I ask how to do that?
>
> The reason I'm rebuild the package is that I'm creating a set of
> Docker images of Zeppelin for different Spark/Hadoop versions.
>
> Regards,
> Tao Wang
>

Re: Notebook permissions lost after restart

[moon soo Lee]

Notebook permissions are stored in conf/notebook-authorization.json.
You'll need to save/restore this file, too.

Thanks,
moon

On Thu, Aug 31, 2017 at 9:53 AM Luis Angel Vicente Sanchez <
zeppelin-us...@bigcente.ch> wrote:

> Hi,
>
> I'm running a Zeppelin instance on Kubernetes. The notebooks are stored
> on S3 and I have set the environment variable ZEPPELIN_NOTEBOOK_PUBLIC
> to false.
>
> When I redeploy Zeppelin, the notebook permissions (Owner, Readers,
> Writers) are lost? Is this not being kept on S3?
>
> Kind regards,
>
> Luis Angel Vicente Sanchez
> zeppelin-us...@bigcente.ch
>

Re: Metadata browser for JDBC

[moon soo Lee]

Thanks for sharing the idea.
Sounds very interesting and useful.

I wish this feature be general enough so we can use it with not only JDBC
but also with SparkSQL, Pandas dataframe, etc.

Then i guess internal data representation will be important topic.
Zeppelin community had several attempt and related discussions.

Following proposal is one of proposal that Zeppelin community made.
https://cwiki.apache.org/confluence/display/ZEPPELIN/Proposal%3A+Extending+TableData+API
This proposal talk more about foundation of internal data representation.
But i believe it opens many possibilities like metadata browser, cross
interpreter join, etc.

What do you think?

Thanks,
moon

On Tue, Aug 29, 2017 at 2:38 AM Polyakov Valeriy 
wrote:

> Hello, Team!
>
>
>
> What do you think about implementation of high-grade metadata browser for
> JDBC interpreter? As it is done in any popular SQL development tools, for
> example, DataGrip. I'm talking about new detailed panel which will contain
> metadata info of all included in note interpreters. I've found only one
> task [1] with a similar problem, but in my opinion its objectives are too
> limited. We could work on something more ambitious and convenient. So, I
> would like to hear your opinion on this matter.
>
>
>
> Thank you!
>
>
>
> [1] https://issues.apache.org/jira/browse/ZEPPELIN-1824
>
>
>
>
>
>
>
> Kind regards,
> *Valeriy Polyakov*
>
>
>

Re: Execute permission only

[moon soo Lee]

Hi Bhavin,

You can check https://github.com/apache/zeppelin/pull/2526.

Thanks,
moon

On Tue, Aug 22, 2017 at 4:58 AM Bhavin Tandel 
wrote:

> Hello team zeppelin,
>
> Is there a feature, such that reader user cannot edit the notebook but is
> able to execute the notebook.
>
> Best regards,
> Bhavin
>
>
> *Disclaimer: The information in this email is confidential and may be
> legally privileged. Access to this email by anyone other than the intended
> addressee is unauthorized. If you are not the intended recipient of this
> message, any review, disclosure, copying, distribution, retention, or any
> action taken or omitted to be taken in reliance on it is prohibited and may
> be unlawful.*

Re: Dynamic Dependencies

[moon soo Lee]

Thanks for sharing your problem.
For now, only way is clean local-repo to download the artifact again.
Do you mind file a jira issue track this problem?

Thanks,
moon

On Tue, Jul 11, 2017 at 4:04 AM Edgardo Vega  wrote:

> I successfully added a maven snapshot repository and was able to resolve
> the dependencies. Unfortunately I have published new versions to the
> repository and restarted the interpreter yet the new artifact is not being
> pulled in.
>
> I set it up using the following template
>
> z.addRepo("RepoName").url("RepoURL").snapshot()
>
>
>  Is there a way to force the artifact to be downloaded on any updates?
>
>
> --
> Cheers,
>
> Edgardo
>

Re: Use Bokeh in Apache Zeppelin

[moon soo Lee]

Cool. Thanks for sharing!

On Sun, 2 Jul 2017 at 8:00 AM Jeff Zhang  wrote:

>
> I write a tutorial on using bokeh in apache zeppelin. If you are
> interested in data visualization in zeppelin notebook, bokeh would be a
> very good library for you. And you can take a look at the tutorial here.
>
>
> https://community.hortonworks.com/articles/109837/use-bokeh-in-apache-zeppelin.html
>
>
>

Re: Query about the high availability of Zeppelin

[moon soo Lee]

Basically active-(hot)standby model would work.
Two or more Zeppelin instance can be started and pointing the same notebook
storage, if only one Zeppelin instance (active) change notebook at any
given time.

In case of the active instance fails, one of rest instance can take over
the role by refreshing notebook list and start make change.

But all these fail over is not provided by Zeppelin and need to depends on
external script or HA software (like Heartbeat).

Like Jeff mentioned, community does not have concrete plan for having HA
built-in at this moment.

Hope this helps,

Thanks,
moon

On Fri, Jun 30, 2017 at 1:01 PM Jeff Zhang  wrote:

>
> No concrete plan for that. There're other higher priority things need to
> be done. At least it would not be available in 0.8, maybe after 1.0
>
>
> wenxing zheng 于2017年6月30日周五 上午11:47写道：
>
>> Thanks to Jianfeng.
>>
>> Do you  know any plan on this?
>>
>>
>> On Fri, Jun 30, 2017 at 11:32 AM, Jianfeng (Jeff) Zhang <
>> jzh...@hortonworks.com> wrote:
>>
>>> HA is not supported, there’s still  lots of configuration files stored
>>> in local file system.
>>>
>>>
>>>
>>> Best Regard,
>>> Jeff Zhang
>>>
>>>
>>> From: wenxing zheng 
>>> Reply-To: "users@zeppelin.apache.org" 
>>> Date: Friday, June 30, 2017 at 9:40 AM
>>> To: "users@zeppelin.apache.org" 
>>> Subject: Query about the high availability of Zeppelin
>>>
>>> Hello all,
>>>
>>> I still didn't find any docs on this topic? Appreciated if anyone can
>>> shed some lights on how to get the Zeppelin into a cluster with
>>> shared/centralized storage
>>>
>>> Regards, Wenxing
>>>
>>
>>

Re: Notebook Storage and Git

[moon soo Lee]

Hi,

There's a related issue https://issues.apache.org/jira/browse/ZEPPELIN-2702 for
naming scheme.

I think Version Control System friendly notebook file format is interesting
subject to discuss. related issue is
https://issues.apache.org/jira/browse/ZEPPELIN-451

Thanks,
moon

On Thu, Jun 29, 2017 at 5:51 PM Tw UxTLi51Nus 
wrote:

> Hi,
>
> not sure if I should write this on dev@, but I thought I'll give it a
> try here first ...
>
> I am using Zeppelin with Git controlled notebook storage. However, I
> find the "git client" integrated in Zeppelin quite rudimentary. So I do
> most of the VCS stuff via the CLI.
>
> Two things are bothering me:
>
> 1) the naming scheme
> On the file system, the notebooks are named with some random names
> (well, the folders, the notebooks itself are all note.json). Wouldn't it
> be better to reflect the structure of the notebooks in Zeppelin also on
> the file system, e.g. a notebook named "nbfolder1/nbfolder2/nb1" is on
> the file system on "NOTEBOOK-STORAGE/nbfolder1/nbfolder2/nb1.json" ?
> Was this or something similar discussed / discarded at some point? If
> discarded, why?
>
> 2) The notebooks containing the results
> ... this leads to a change in the note.json files when the notebook is
> run again, even when the "code" itself has not changed, which makes
> comparing diffs really difficult. Why not use a second file (e.g.
> notebook_results.json) to store the results and thus have a "clean"
> notebook file to put into VC?
>
> Thanks,
>
> --
> Tw UxTLi51Nus
> Email: twuxtli51...@posteo.co
>

Re: Centos 7 Compatibility

[moon soo Lee]

Website and doc is part of opensource project and anyone can update them by
making pull request.

https://github.com/apache/zeppelin/blob/master/docs/install/install.md#installation

Please anyone feel free to maintain OS / environment compatibility table.

Thanks,
moon

On Wed, Jun 21, 2017 at 3:20 PM Jim Lola  wrote:

> Excellent.  I am not sure who or how the OS compatibility matrix is
> maintained.  But it is good to know that Zeppelin works on CentOS 7.2.
> Thanks for sharing.
>
>
> On Wed, Jun 21, 2017 at 2:07 PM, Ben Vogan  wrote:
>
>> I've been running Zeppelin 0.7.1 and no I didn't have to make any
>> non-standard configuration changes that I recall.  I was very pleased with
>> how easy it was to get up and running.
>>
>> --Ben
>>
>> On Wed, Jun 21, 2017 at 1:43 PM, Jim Lola  wrote:
>>
>>> Which version of Zeppelin do you have working on CentOS 7.2?  Did you
>>> make any different/non-standard configuration changes to get it to work
>>> properly?  If so, could you please share them.
>>>
>>> On Wed, Jun 21, 2017 at 12:30 PM, Ben Vogan  wrote:
>>>
 I have been running Zeppelin on CentOS 7.2 for the last couple of
 months without issue.

 --Ben

 On Wed, Jun 21, 2017 at 12:37 PM, Jim Lola  wrote:

> The beauty of Open Source, like Apache Zeppelin, is that you can try
> SW on new OS's.
>
> Per the Apache Zepplin documentation, CentOS 6 is supported.  CentOS 7
> is NOT mentioned.
>
> There is actually a very large difference is Linux OS kernels between
> CentOS 6 and CentOS 7.   CentOS 6 is based on the Linux kernel version
> 2.6.32-71 while CentOS 7 is based on Linux kernel version 3.10.0-123.  The
> default file system is different as are the run levels.  The init system 
> in
> CentOS 7 is now using systemd and so init is being replaced/updated.  
> There
> are a lot more changes between CentOS 6 to CentOS 7.
>
> It sounds like a good opportunity to get involved w/ future
> development of Apache Zeppelin.
>
>
>
> On Wed, Jun 21, 2017 at 11:10 AM, Benjamin Kim 
> wrote:
>
>> All,
>>
>> I’m curious to know if Zeppelin will work with CentOS 7. I don’t see
>> it in the list of OS’s supported.
>>
>> Thanks,
>> Ben
>
>
>


 --
 *BENJAMIN VOGAN* | Data Platform Team Lead

 
 
 
  
 

>>>
>>>
>>
>>
>> --
>> *BENJAMIN VOGAN* | Data Platform Team Lead
>>
>> 
>>  
>>  
>> 
>>
>
>

Re: can Zeppelin runParagraph from different notebooks?

[moon soo Lee]

In spark interpreter, you can try

%spark
run(NOTE_ID, PARAGRAPH_ID)

Hope this helps.

Thanks,
moon

On Mon, Jun 12, 2017 at 9:52 AM Richard Xin  wrote:

> Angular (frontend API)
> 
>
> Angular (frontend API)
>
> The Apache Software Foundation
>
> Angular (frontend API)
>
> 
>
>
> This method doesn't seem to support running paragraphs from different
> notebooks, did I miss anything?
>

Re: unable to restart zeppelin on AWS EMR

[moon soo Lee]

Have you tried restart with 'sudo -u zeppelin'?

sudo -u zeppelin /usr/lib/zeppelin/bin/zeppelin-daemon.sh restart

worked for me.


Thanks,
moon

On Sat, Jun 3, 2017 at 11:51 PM Jonathan Kelly 
wrote:

> Shyla,
>
> What release of EMR are you using? There was a problem with earlier
> releases of EMR where upstart was not properly tracking the Zeppelin PID,
> so "sudo stop zeppelin" wouldn't work. This has been fixed for a while, but
> I don't remember what version contained the fix. If you are not using the
> latest release, you might want to try that. Otherwise, if you're already
> using the latest version, there might be something else going on.
>
> ~ Jonathan
> On Sat, Jun 3, 2017 at 8:28 PM shyla deshpande 
> wrote:
>
>> I changed some configuration and want to restart  zeppelin on AWS EMR,
>> but unable to do. My local Zeppelin works fine.
>>
>> I have tried
>> 1.  zeppelin-daemon.sh restart   outputs[  OK  ]   but
>> has no effect.
>> 2.  sudo stop zeppelin outputsstop: Unknown
>> instance:
>> 3.  sudo start zeppelin outputs .  start: Job failed
>> to start
>>
>> Zeppelin is running and able to login as anonymous user, unable to
>> restart . Appreciate your input.
>>
>> Thanks
>>
>

Re: [DISCUSS] Reducing default interpreters while building and releasing Zeppelin

[moon soo Lee]

Following is last discussion related to release package size.

https://lists.apache.org/thread.html/69f606409790d7ba11422e8c6df941a75c5dfae0aca63eccf2f840bf@%3Cusers.zeppelin.apache.org%3E

at this time, we have discussed about having bin-all (every interpreters),
bin-min (selected interpreters), bin-netinst (no interpreters) package but
didn't conclude the criteria and how we make a decision.

Jongyoul, do you have any idea about criteria?

Thanks,
moon

On Sun, Jun 4, 2017 at 10:47 AM Felix Cheung 
wrote:

> Sure - I think it will be important to discuss what criteria to use to
> decide what is included vs what will be released separately.
>
> _
> From: Jongyoul Lee 
> Sent: Sunday, June 4, 2017 9:47 AM
> Subject: Re: [DISCUSS] Reducing default interpreters while building and
> releasing Zeppelin
> To: dev 
> Cc: 
>
>
>
> It means we release with some interpreters and deploy all interpreters into
> maven separately. We already had a install-interpreter script inside it. If
> someone wants to install specific interpreter not included in default
> release package, they can use that script to install specific one.
>
> On Sun, Jun 4, 2017 at 9:11 AM, Felix Cheung 
> wrote:
>
> > Are we proposing some interpreters to be built and released separately?
> >
> > Is this going to be separate packaging? Or separate release pipeline?
> >
> >
> > _
> > From: Jongyoul Lee  >>
> > Sent: Friday, June 2, 2017 11:04 PM
> > Subject: [DISCUSS] Reducing default interpreters while building and
> > releasing Zeppelin
> > To: dev  >>, <
> > users@zeppelin.apache.org >>
> >
> >
> > Hi dev and users,
> >
> > Recently, zeppelin.apache.org is being
> > changed for increasing user experiences and convenience. I like this kind
> > of changes. I, however, saw some arguments that which interpreters we
> will
> > locate in the first page. I'd like to expand its argument to the package
> we
> > release.
> >
> > Current zeppelin packages exceed 700MB with default option because
> > Zeppelin tried to include all interpreters by default. It was good at the
> > early age but, nowadays, Zeppelin community suffer from the size because
> > ASF infra allows the package size under 500MB. So I'd like to reduce the
> > package size by reducing default packages.
> >
> > In case of rebuilding homepage, community proposed some criteria
> including
> > mailing list and # of question in stackoverflow. I think we can adapt
> same
> > criteria into release version of Zeppelin.
> >
> > To handle this kind of issue, I think consensus of community is the most
> > important factor. If someone wants to have an idea to deal with it,
> please
> > feel free to talk about it.
> >
> > Thanks,
> > Jongyoul Lee
> >
> > --
> > 이종열, Jongyoul Lee, 李宗烈
> > http://madeng.net
> >
> >
> >
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>
>
>

Re: How to change the python installation location for Python interpreter?

[moon soo Lee]

Hi,

You can set your python executable path in 'zeppelin.python' property,
Python interpreter setting in the interpreter menu.

If your system has conda installed, then you can manage environment using
%python.conda. Please take a look
http://zeppelin.apache.org/docs/latest/interpreter/python.html#conda

Thanks,
moon

On Sat, May 27, 2017 at 1:59 PM BigData Consultant <
bigdata.consultant2...@gmail.com> wrote:

> Hi Team,
>
> Where is the python installation location specified for Python interpreter?
>
> How to change the python installation location for Python interpreter with
> new python installation directory?
>
> Help would be much appreciated.
>
> Thanks.
>

Re: Running a notebook in a standalone cluster mode issues

[moon soo Lee]

Any workaround except for using client mode, it's difficult to think ...

Thanks,
moon

On Wed, 3 May 2017 at 3:49 PM Sofiane Cherchalli <sofian...@gmail.com>
wrote:

> Hi Moon,
>
> Great, I am keen to see Zeppelin-2040 resolved soon. But meanwhile is
> there any workaround?
>
> Thanks.
>
> Sofiane
>
>
> El El mié, 3 may 2017 a las 20:40, moon soo Lee <m...@apache.org>
> escribió:
>
>> Zeppelin don't need to be installed in every workers.
>> You can think the way SparkInterpreter in Zeppelin work is very similar
>> to spark-shell (which works in client mode), until ZEPPELIN-2040 is
>> resolved.
>>
>> Therefore, if spark-shell works in a machine with your standalone
>> cluster, Zeppelin will work in the same machine with the standalone cluster.
>>
>> Thanks,
>> moon
>>
>> On Wed, May 3, 2017 at 2:28 PM Sofiane Cherchalli <sofian...@gmail.com>
>> wrote:
>>
>>> Hi Moon,
>>>
>>> So in my case, if II have standalone or yarn cluster, the workaround
>>> would be to install zeppelin along every worker, proxy them,  and run each
>>> zeppelin in client mode ?
>>>
>>> Thanks,
>>> Sofiane
>>>
>>> El El mié, 3 may 2017 a las 19:12, moon soo Lee <m...@apache.org>
>>> escribió:
>>>
>>>> Hi,
>>>>
>>>> Zeppelin does not support cluster mode deploy at the moment.
>>>> Fortunately, there will be a support for cluster mode, soon!
>>>> Please keep an eye on
>>>> https://issues.apache.org/jira/browse/ZEPPELIN-2040.
>>>>
>>>> Thanks,
>>>> moon
>>>>
>>>> On Wed, May 3, 2017 at 11:00 AM Sofiane Cherchalli <sofian...@gmail.com>
>>>> wrote:
>>>>
>>>>> Shall I configure a remote interpreter to my notebook to run on the
>>>>> worker?
>>>>>
>>>>> Mayday!
>>>>>
>>>>> On Wed, May 3, 2017 at 4:18 PM, Sofiane Cherchalli <
>>>>> sofian...@gmail.com> wrote:
>>>>>
>>>>>> What port does the remote interpreter use?
>>>>>>
>>>>>> On Wed, May 3, 2017 at 2:14 PM, Sofiane Cherchalli <
>>>>>> sofian...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Moon and al,
>>>>>>>
>>>>>>> I have a standalone cluster with one master, one worker. I submit
>>>>>>> jobs through zeppelin. master, worker, and zeppelin run in a separate
>>>>>>> container.
>>>>>>>
>>>>>>> My zeppelin-env.sh:
>>>>>>>
>>>>>>> # spark home
>>>>>>> export SPARK_HOME=/usr/local/spark
>>>>>>>
>>>>>>> # set hadoop conf dir
>>>>>>> export HADOOP_CONF_DIR=/opt/hadoop-2.7.3/etc/hadoop
>>>>>>>
>>>>>>> # set options to pass spark-submit command
>>>>>>> export SPARK_SUBMIT_OPTIONS="--packages
>>>>>>> com.databricks:spark-csv_2.11:1.5.0 --deploy-mode cluster"
>>>>>>>
>>>>>>> # worker memory
>>>>>>> export ZEPPELIN_JAVA_OPTS="-Dspark.driver.memory=7g
>>>>>>> -Dspark.submit.deployMode=cluster"
>>>>>>>
>>>>>>> # master
>>>>>>> export MASTER="spark://:7077"
>>>>>>>
>>>>>>> My notebook code is very simple. It read csv and write it again in
>>>>>>> directory /data previously created:
>>>>>>> %spark.pyspark
>>>>>>> def read_input(fin):
>>>>>>> '''
>>>>>>> Read input file from filesystem and return dataframe
>>>>>>> '''
>>>>>>> df = sqlContext.read.load(fin,
>>>>>>> format='com.databricks.spark.csv', mode='PERMISSIVE', header='false',
>>>>>>> inferSchema='true')
>>>>>>> return df
>>>>>>>
>>>>>>> def write_output(df, fout):
>>>>>>> '''
>>>>>>> Write dataframe to filesystem
>>>>>>> '''
>>>>>>>
>>>>>>> df.write.mode('overwrite').format('com.databricks.spark.csv').options(delimiter=',',
>>>>>>> header='true').save(fout)
&g

Re: Running a notebook in a standalone cluster mode issues

[moon soo Lee]

Zeppelin don't need to be installed in every workers.
You can think the way SparkInterpreter in Zeppelin work is very similar to
spark-shell (which works in client mode), until ZEPPELIN-2040 is resolved.

Therefore, if spark-shell works in a machine with your standalone cluster,
Zeppelin will work in the same machine with the standalone cluster.

Thanks,
moon

On Wed, May 3, 2017 at 2:28 PM Sofiane Cherchalli <sofian...@gmail.com>
wrote:

> Hi Moon,
>
> So in my case, if II have standalone or yarn cluster, the workaround would
> be to install zeppelin along every worker, proxy them,  and run each
> zeppelin in client mode ?
>
> Thanks,
> Sofiane
>
> El El mié, 3 may 2017 a las 19:12, moon soo Lee <m...@apache.org>
> escribió:
>
>> Hi,
>>
>> Zeppelin does not support cluster mode deploy at the moment. Fortunately,
>> there will be a support for cluster mode, soon!
>> Please keep an eye on https://issues.apache.org/jira/browse/ZEPPELIN-2040
>> .
>>
>> Thanks,
>> moon
>>
>> On Wed, May 3, 2017 at 11:00 AM Sofiane Cherchalli <sofian...@gmail.com>
>> wrote:
>>
>>> Shall I configure a remote interpreter to my notebook to run on the
>>> worker?
>>>
>>> Mayday!
>>>
>>> On Wed, May 3, 2017 at 4:18 PM, Sofiane Cherchalli <sofian...@gmail.com>
>>> wrote:
>>>
>>>> What port does the remote interpreter use?
>>>>
>>>> On Wed, May 3, 2017 at 2:14 PM, Sofiane Cherchalli <sofian...@gmail.com
>>>> > wrote:
>>>>
>>>>> Hi Moon and al,
>>>>>
>>>>> I have a standalone cluster with one master, one worker. I submit jobs
>>>>> through zeppelin. master, worker, and zeppelin run in a separate 
>>>>> container.
>>>>>
>>>>> My zeppelin-env.sh:
>>>>>
>>>>> # spark home
>>>>> export SPARK_HOME=/usr/local/spark
>>>>>
>>>>> # set hadoop conf dir
>>>>> export HADOOP_CONF_DIR=/opt/hadoop-2.7.3/etc/hadoop
>>>>>
>>>>> # set options to pass spark-submit command
>>>>> export SPARK_SUBMIT_OPTIONS="--packages
>>>>> com.databricks:spark-csv_2.11:1.5.0 --deploy-mode cluster"
>>>>>
>>>>> # worker memory
>>>>> export ZEPPELIN_JAVA_OPTS="-Dspark.driver.memory=7g
>>>>> -Dspark.submit.deployMode=cluster"
>>>>>
>>>>> # master
>>>>> export MASTER="spark://:7077"
>>>>>
>>>>> My notebook code is very simple. It read csv and write it again in
>>>>> directory /data previously created:
>>>>> %spark.pyspark
>>>>> def read_input(fin):
>>>>> '''
>>>>> Read input file from filesystem and return dataframe
>>>>> '''
>>>>> df = sqlContext.read.load(fin, format='com.databricks.spark.csv',
>>>>> mode='PERMISSIVE', header='false', inferSchema='true')
>>>>> return df
>>>>>
>>>>> def write_output(df, fout):
>>>>> '''
>>>>> Write dataframe to filesystem
>>>>> '''
>>>>>
>>>>> df.write.mode('overwrite').format('com.databricks.spark.csv').options(delimiter=',',
>>>>> header='true').save(fout)
>>>>>
>>>>> data_in = '/data/01.csv'
>>>>> data_out = '/data/02.csv'
>>>>> df = read_input(data_in)
>>>>> newdf = del_columns(df)
>>>>> write_output(newdf, data_out)
>>>>>
>>>>>
>>>>> I used --deploy-mode to *cluster* so that the driver is run in the
>>>>> worker in order to read the CSV in the /data directory and not in 
>>>>> zeppelin.
>>>>> When running the notebook it complains that
>>>>> /opt/zeppelin-0.7.1/interpreter/spark/zeppelin-spark_2.11-0.7.1.jar is
>>>>> missing:
>>>>> org.apache.zeppelin.interpreter.InterpreterException: Ivy Default
>>>>> Cache set to: /root/.ivy2/cache The jars for the packages stored in:
>>>>> /root/.ivy2/jars :: loading settings :: url =
>>>>> jar:file:/opt/spark-2.1.0/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
>>>>> com.databricks#spark-csv_2.11 added as a dependency :: resolving
>>>>> dependencies :: org.apache.spark#spark-submit-parent;1.0 confs: [default]
>>&g

Re: Running a notebook in a standalone cluster mode issues

[moon soo Lee]

Hi,

Zeppelin does not support cluster mode deploy at the moment. Fortunately,
there will be a support for cluster mode, soon!
Please keep an eye on https://issues.apache.org/jira/browse/ZEPPELIN-2040.

Thanks,
moon

On Wed, May 3, 2017 at 11:00 AM Sofiane Cherchalli 
wrote:

> Shall I configure a remote interpreter to my notebook to run on the worker?
>
> Mayday!
>
> On Wed, May 3, 2017 at 4:18 PM, Sofiane Cherchalli 
> wrote:
>
>> What port does the remote interpreter use?
>>
>> On Wed, May 3, 2017 at 2:14 PM, Sofiane Cherchalli 
>> wrote:
>>
>>> Hi Moon and al,
>>>
>>> I have a standalone cluster with one master, one worker. I submit jobs
>>> through zeppelin. master, worker, and zeppelin run in a separate container.
>>>
>>> My zeppelin-env.sh:
>>>
>>> # spark home
>>> export SPARK_HOME=/usr/local/spark
>>>
>>> # set hadoop conf dir
>>> export HADOOP_CONF_DIR=/opt/hadoop-2.7.3/etc/hadoop
>>>
>>> # set options to pass spark-submit command
>>> export SPARK_SUBMIT_OPTIONS="--packages
>>> com.databricks:spark-csv_2.11:1.5.0 --deploy-mode cluster"
>>>
>>> # worker memory
>>> export ZEPPELIN_JAVA_OPTS="-Dspark.driver.memory=7g
>>> -Dspark.submit.deployMode=cluster"
>>>
>>> # master
>>> export MASTER="spark://:7077"
>>>
>>> My notebook code is very simple. It read csv and write it again in
>>> directory /data previously created:
>>> %spark.pyspark
>>> def read_input(fin):
>>> '''
>>> Read input file from filesystem and return dataframe
>>> '''
>>> df = sqlContext.read.load(fin, format='com.databricks.spark.csv',
>>> mode='PERMISSIVE', header='false', inferSchema='true')
>>> return df
>>>
>>> def write_output(df, fout):
>>> '''
>>> Write dataframe to filesystem
>>> '''
>>>
>>> df.write.mode('overwrite').format('com.databricks.spark.csv').options(delimiter=',',
>>> header='true').save(fout)
>>>
>>> data_in = '/data/01.csv'
>>> data_out = '/data/02.csv'
>>> df = read_input(data_in)
>>> newdf = del_columns(df)
>>> write_output(newdf, data_out)
>>>
>>>
>>> I used --deploy-mode to *cluster* so that the driver is run in the
>>> worker in order to read the CSV in the /data directory and not in zeppelin.
>>> When running the notebook it complains that
>>> /opt/zeppelin-0.7.1/interpreter/spark/zeppelin-spark_2.11-0.7.1.jar is
>>> missing:
>>> org.apache.zeppelin.interpreter.InterpreterException: Ivy Default Cache
>>> set to: /root/.ivy2/cache The jars for the packages stored in:
>>> /root/.ivy2/jars :: loading settings :: url =
>>> jar:file:/opt/spark-2.1.0/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
>>> com.databricks#spark-csv_2.11 added as a dependency :: resolving
>>> dependencies :: org.apache.spark#spark-submit-parent;1.0 confs: [default]
>>> found com.databricks#spark-csv_2.11;1.5.0 in central found
>>> org.apache.commons#commons-csv;1.1 in central found
>>> com.univocity#univocity-parsers;1.5.1 in central :: resolution report ::
>>> resolve 310ms :: artifacts dl 6ms :: modules in use:
>>> com.databricks#spark-csv_2.11;1.5.0 from central in [default]
>>> com.univocity#univocity-parsers;1.5.1 from central in [default]
>>> org.apache.commons#commons-csv;1.1 from central in [default]
>>> - | |
>>> modules || artifacts | | conf | number| search|dwnlded|evicted||
>>> number|dwnlded|
>>> - |
>>> default | 3 | 0 | 0 | 0 || 3 | 0 |
>>> - ::
>>> retrieving :: org.apache.spark#spark-submit-parent confs: [default] 0
>>> artifacts copied, 3 already retrieved (0kB/8ms) Running Spark using the
>>> REST application submission protocol. SLF4J: Class path contains multiple
>>> SLF4J bindings. SLF4J: Found binding in
>>> [jar:file:/opt/zeppelin-0.7.1/interpreter/spark/zeppelin-spark_2.11-0.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>> SLF4J: Found binding in
>>> [jar:file:/opt/zeppelin-0.7.1/lib/interpreter/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>> SLF4J: Found binding in
>>> [jar:file:/opt/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>>> explanation. SLF4J: Actual binding is of type
>>> [org.slf4j.impl.Log4jLoggerFactory] Warning: Master endpoint
>>> spark://spark-drone-master-sofiane.autoetl.svc.cluster.local:7077 was not a
>>> REST server. Falling back to legacy submission gateway instead. Ivy Default
>>> Cache set to: /root/.ivy2/cache The jars for the packages stored in:
>>> /root/.ivy2/jars com.databricks#spark-csv_2.11 added as a dependency ::
>>> resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 confs:
>>> [default] found com.databricks#spark-csv_2.11;1.5.0 in central found
>>> org.apache.commons#commons-csv;1.1 in central 

Re: Can't delete an empty paragraph?

[moon soo Lee]

Checked last 3 version 0.6.2, 0.7.0, 0.7.1, and all they can remove empty
paragraph.

Thanks,
moon

On Mon, Apr 24, 2017 at 1:44 AM Partridge, Lucas (GE Aviation) <
lucas.partri...@ge.com> wrote:

> Thanks moon.  Unfortunately I’m not an admin for the system I’m using and
> don’t control when it gets updated.
>
>
>
> Do you happen to know which version of Zeppelin this issue was fixed in?
> Is it only 0.7.2?
>
>
>
> Thanks, Lucas.
>
>
>
> *From:* moon soo Lee [mailto:m...@apache.org]
> *Sent:* 22 April 2017 06:32
> *To:* users@zeppelin.apache.org
> *Subject:* EXT: Re: Can't delete an empty paragraph?
>
>
>
> Thanks for reporting the problem.
>
> If 'About Zeppelin' dialog missing version number, i guess it's 0.5.x.
>
>
>
> We'll release 0.7.2 [1] in next few weeks.
>
> Please consider use recent version if possible.
>
>
>
> But of course, please feel free share any issues on your version.
>
>
>
> Thanks,
>
> moon
>
>
>
> [1] https://issues.apache.org/jira/browse/ZEPPELIN-2276
>
>
>
> On Fri, Apr 21, 2017 at 6:22 AM Partridge, Lucas (GE Aviation) <
> lucas.partri...@ge.com> wrote:
>
> I can’t delete an empty paragraph in a notebook.  I’m talking about an
> empty paragraph that’s not the last paragraph. I can click on Remove under
> the paragraph’s settings icon and be prompted to delete it, but when I
> click OK the paragraph doesn’t get deleted!
>
>
>
> To delete it I have to put something in the paragraph (random text will
> do) and then execute it. Only then will the paragraph be deleted.
>
>
>
> This might have been fixed in a later version of Zeppelin. I know I’m not
> using the latest but unfortunately the ‘About Zeppelin’ dialog is missing
> the version number!
>
>
>
> Thanks, Lucas.
>
>
>
>

Re: paragraph log is shown always

[moon soo Lee]

You can clear output in %spark

z.getInterpreterContext.out.clear

in %pyspark

z.getInterpreterContext().out().clear()

It may help hide unwanted output displayed.

Thanks,
moon

On Tue, Apr 25, 2017 at 11:09 AM Jan Rasehorn 
wrote:

> Hello,
>
> I'm currently running version 0.7.0 and 0.7.1 .
>
> When I execute a paragraph it will display the console log no matter if I
> selected the simple or report mode for the notebook.
>
> I wonder if there is a trick to hide the paragraph console output but
> still display the actual output like selection fields or tables/charts.
>
> Thanks for any suggestions and BR,
> Jan
>

Re: Data Source Authorization - JDBC Credential

[moon soo Lee]

q.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://pages.placeiq.com/Location-Data-Accuracy-Whitepaper-Download.html?utm_source=Signature_medium=Email_campaign=AccuracyWP>
> <http://placeiq.com/2016/08/03/placeiq-bolsters-location-intelligence-platform-with-mastercard-insights/>
> <http://placeiq.com/2016/10/26/the-making-of-a-location-data-industry-milestone/>[image:
> PlaceIQ:Location Data Accuracy]
> <http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/>
>
>
> On Sat, Apr 22, 2017 at 1:21 AM moon soo Lee  <moon+soo+lee+%3cm...@apache.org%3E>> wrote:
>
>> Hi,
>>
>> Which version of Zeppelin are you using? Have you tried remove
>> 'default.user' and 'default.password' property from interpreter menu and
>> create entity in credential menu?
>>
>> Thanks,
>> moon
>>
>> On Fri, Apr 21, 2017 at 11:55 AM Paul Brenner <pbren...@placeiq.com>
>> wrote:
>>
>>> Are credentials confirmed working with the JDBC interpreter? I’m trying
>>> to get jdbc working with snowflake. If I hardcore my username and password
>>> into the default url everything works great… but this of course won’t work
>>> in a multi user environment.
>>>
>>> However, if I try to use default username, default password, or zeppelin
>>> credentials I get stuck at "java.sql.SQLException: Missing user name.”
>>>
>>> Sounds like a different issue than the OP.
>>>
>>> <http://www.placeiq.com/> <http://www.placeiq.com/>
>>> <http://www.placeiq.com/> Paul Brenner <https://twitter.com/placeiq>
>>> <https://twitter.com/placeiq> <https://twitter.com/placeiq>
>>> <https://www.facebook.com/PlaceIQ> <https://www.facebook.com/PlaceIQ>
>>> <https://www.linkedin.com/company/placeiq>
>>> <https://www.linkedin.com/company/placeiq>
>>> DATA SCIENTIST
>>> *(217) 390-3033 <(217)%20390-3033> *
>>>
>>> <http://www.placeiq.com/2015/05/26/placeiq-named-winner-of-prestigious-2015-oracle-data-cloud-activate-award/>
>>> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
>>> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
>>> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
>>> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
>>> <http://placeiq.com/2016/03/08/measuring-addressable-tv-campaigns-is-now-possible/>
>>> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
>>> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
>>> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
>>> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
>>> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
>>> <http://pages.placeiq.com/Location-Data-Accuracy-Whitepaper-Download.html?utm_source=Signature_medium=Email_campaign=AccuracyWP>
>>> <http://placeiq.com/2016/08/03/placeiq-bolsters-location-intelligence-platform-with-mastercard-insights/>
>>> <http://placeiq.com/2016/10/26/the-making-of-a-location-data-industry-milestone/>[image:
>>> PlaceIQ:Location Data Accuracy]
>>> <http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/>
>>>
>>> On Thu, Apr 20, 2017 at 8:49 PM moon soo Lee >> <moon+soo+lee+%3cm...@apache.org%3E>> wrote:
>>>
>>>> Hi,
>>>>
>>>> If you remove
>>>>
>>>> 'default.user'
>>>> 'default.password'
>>>>
>>>> properties from jdbc interpreter setting, then Zeppelin will use
>>>> database username and password from credential database for each user.
>>>>
>>>> I also created a patch [1] to use credential database when
>>>> 'default'.user' and 'default.password' is empty string.
>>>>
>>>> Hope this helps.
>>>>
>>>> Best,
>>>> moon
>>>>
>>>> [1] https://github.com/apache/zeppelin/pull/2269
>>>>
>>>> On Mon, Apr 10, 2017 at 3:37 AM Arpad Beregszaszi <
>>>> arpad.beregsza...@gmx.de> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>>
>>>>>
>>>>> I’m pretty new with Zeppeln and I need help with one problem regarding
>>>>> data source authorization as its described here:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> https://zeppelin.apache.org/docs/0.7.0/security/datasource_authorization.html
>>>>>
>>>>>
>>>>>
>>>>> I can successfully connect Zeppelin to my MySQL Server. Now I want to
>>>>> give different users access to their databases.
>>>>>
>>>>> For that I want the JDBC interpreter to use different database
>>>>> credentials, depending on the Zeppelin user as defined in shiro.ini.
>>>>>
>>>>> But when I create credential information, the JDBC interpreter doesn’t
>>>>> use them for the db connection, but the default username and password,
>>>>> which is empty.
>>>>>
>>>>> My Interpreter Is instantiated per user.
>>>>>
>>>>>
>>>>>
>>>>> Java.sql.SQLException: Access denied for user ‘ ‘ @ ‘IP‘ (using
>>>>> password: NO)
>>>>>
>>>>>
>>>>>
>>>>> Anyone an idea of whats wrong?
>>>>>
>>>>> Thanks, Arpad
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>
>

Re: Custom spark for zeppelin and interpreter-list

[moon soo Lee]

Hi,

'conf/interpreter-list' is just catalogue file that
`/bin/install-interpreter.sh' uses.
The information is not being used any other place.

'/bin/install-interpreter.sh' use 'conf/interpreter-list' to 1) print list
of interpreter that Zeppelin community provides 2) convert short name to
group:artifact:version, so user doesn't have to provide -t option.

Spark interpreter is being included in both zeppelin-bin-all and
zeppelin-bin-netinst package, that's why `conf/interpreter-list' doesn't
have it.

So, if you're trying to install your custom interpreter using
'bin/install-interpreter.sh', you can still do without modifying
'conf/interpreter-list' by providing '-t' option. If you're installing your
custom interpreter without using 'bin/install-interpreter.sh', then
'conf/interpreter-list' is not related at all.

Hope this helps.

Best,
moon

On Sat, Apr 22, 2017 at 1:04 PM Serega Sheypak 
wrote:

> Hi, I have few concerns I can't resolve right now. I definitely can go
> though the source code and find the solution, but I would like to
> understand the idea behind.
> I'm building Zeppelin from sources using 0.8.0-SNAPSHOT. I do build it
> with custom cloudera CDH spark 2.0-something.
> I can't understand if built and started zeppelin uses my custom
> zeppelin-spark interpreter or not?
>
> interpreter-list has maven coordinates. What is it for? Will zeppelin try
> to grab interpreters from remote maven repo?
>
> interpreter-list doesn't have spark in it. How does zeppelin figure out
> what and how spark interpreter use?
>

Re: struggling with LDAP

[moon soo Lee]

Hi Paul, Knapp,

Please don't mind update LDAP documentation if you would like to.
That would save many people!

Documentation (published in zeppelin website) is also part of opensource
and you can update them by making pull request. I think related file is
https://github.com/apache/zeppelin/blob/master/docs/security/shiroauthentication.md#ldap
.
Let me know if you need help on making pull request.

Thanks,
moon

On Thu, Apr 20, 2017 at 3:18 PM Knapp, Michael 
wrote:

> I finally got LDAP to work.  This was one of the most difficult tasks I
> have ever had.  I spent about three weeks trying to make this work!
>
>
>
> One very hard lesson learned:  LDAP/JNDI code will not use the truststore
> that people pass into zeppelin-site.xml.  It will only use the JRE’s
> cacerts file.  This cost me so much time, it should definitely be mentioned
> in the Zeppelin documentation.
>
>
>
> I also think the documentation should offer more help on how to determine
> what values you need in the shiro.ini file.  I eventually figured out there
> was a principalSuffix I needed to use, but the value was not my first guess
> at all.  Some guidance on how to use ldapsearch would save people weeks of
> work here.
>
>
>
> Also the shiro logging is TERRIBLE!  It offers almost no help when it
> comes time to troubleshoot things and discover where it went wrong.  This
> is true even when it is set to trace.
>
>
>
>
>
> *From: *"Knapp, Michael" 
> *Reply-To: *"users@zeppelin.apache.org" 
> *Date: *Wednesday, April 19, 2017 at 1:20 PM
> *To: *"users@zeppelin.apache.org" 
>
>
> *Cc: *"Krishna, Krish" 
> *Subject: *Re: struggling with LDAP
>
>
>
> My mac is configured to forbid installing software by unidentified
> developers.  I cannot install jxplorer.  Is there an alternative?
>
>
>
> The error is coming up when I try to login.  I tried using the
> principalSuffix, it did not change things.
>
>
>
> I discovered a co-worker had LDAP working for a different LDAP server
> under different conditions.  He told me that he is logging in as the system
> account from the UI, which I had never tried or thought of before.  I was
> always using my personal username and password, and figured that the system
> account should just be used on the backend to interact with LDAP.
>
>
>
> Is that the expected way for things to work?  Like the user should enter
> the system username and password on the front end instead of their own?
> Because I don’t think that will be an acceptable long term solution in my
> case.
>
>
>
> I also noticed that if I add “admin = *” to my roles section, that alone
> breaks the application, and I have no idea why.  I’m having trouble finding
> documentation on what is expected in the roles section of the shiro file.
>
>
>
> When I did get it to work:
>
> · I was logging in as the system user on the front end.  Any
> other user fails.
>
> · I did NOT have the principalSuffix defined, adding it seems to
> break things
>
> · I was able to use ldap or ldaps.
>
>
>
>
>
>
>
>
>
>
>
> *From: *Paul Brenner 
> *Reply-To: *"users@zeppelin.apache.org" 
> *Date: *Wednesday, April 19, 2017 at 11:21 AM
> *To: *"Knapp, Michael" , "
> users@zeppelin.apache.org" 
> *Cc: *"Krishna, Krish" 
> *Subject: *Re: struggling with LDAP
>
>
>
> [image:
> ttps://share.polymail.io/v2/z/a/NThmNzdlMmY1M2Q4/4ULIk0PWssT9m_JkSH0DABBQXnzuCgzhdMyhnPBuhOSlesqCbWa29gO]
>
> Have you tried downloading jxplorer (http://jxplorer.org/
> )
> and confirming that you can connect to the ldaps server with your
> credentials?
>
>
>
> Also, when is this error coming up, at start up or when you try to login
> through zeppelin? When I switched to ldap instead of logging in as pbrenner
> for my user I had to use pbren...@corp.placeiq.net. Had to add
> “activeDirectoryRealm.principalSuffix“ to shiro.ini to get around that.
>
> *[image:
> ttps://ci3.googleusercontent.com/proxy/tFn1I-GEOnccUtv8DHHEc49-6g3x3CbuQKzbfl2Z1BObEy0Qz6QebJimpP96TK3Za]*
> 
>
> *Paul Brenner*
>
> *[image:
> ttps://ci4.googleusercontent.com/proxy/490PXYv9O6OiIp_DL4vuabJqVn53fMon5xNYZdftCVea9ySR2LcFDHe6Cdntb2G68]*
> 
>
> *[image:
> ttps://ci3.googleusercontent.com/proxy/fztHf1lRKLQYcAxebqfp2PYXCwVap3GobHVIbyp0j3NcuJOY16bUAZBibVOFf-fd1]*
> 
>
> *[image:
> ttps://ci5.googleusercontent.com/proxy/H26ThD7R6DOqxoLTgzi6k5SMrHoF2Tj44xI_7XlD9KfOIiGwe1WIMc5iQBxUBA9Eu]*
> 

Re: Returning more than the default 1000 rows

[moon soo Lee]

Thanks for reporting problem and share workaround.
It looks like PythonInterpreter uses hardcoded value [1] instead of reading
max_result from interpreter property.
Definitely it looks like a bug. Do you mind file an issue to project JIRA
[2]?

Thanks,
moon

[1]
https://github.com/apache/zeppelin/blob/v0.7.1/python/src/main/resources/python/zeppelin_python.py#L58
[2] https://issues.apache.org/jira/browse/ZEPPELIN

On Fri, Apr 21, 2017 at 5:10 AM Paul-Armand Verhaegen <
paularmand.verhae...@gmail.com> wrote:

>
> running z.__dict in a zeppelin %python paragraph shows that the max_result
> is not set (the configuration setting is not applied):
>
> $ z.__dict__
>
> {'javaList': ,
> 'paramOption': , 'z':
> JavaObject id=t, 'max_result': 2000, '_displayhook':  at 0x7fe18291baa0>}
>
> as a workaround, I now issue z.max_result = 2000 to increase the size of
> the returned csv and that works fine.
>
> Thanks,
> Paul
>
> On 21 Apr 2017, at 13:48, Paul-Armand Verhaegen <
> paularmand.verhae...@gmail.com> wrote:
>
>
> Thanks for your reply. Based on your suggestions I've
> edited conf/zeppelin-env.sh adding:
> export ZEPPELIN_SPARK_MAXRESULT=1  # Max number of Spark SQL
> result to display. 1000 by default.
> export ZEPPELIN_WEBSOCKET_MAX_TEXT_MESSAGE_SIZE=1024   # Size in
> characters of the maximum text message to be received by
> websocket. Defaults to 1024000
>
> I've restarted zeppelin, but it still does not show any rows above 1000
> (neither visualisation nor csv download).
> I also doubled checked the settings by adding "env" in the common.sh to
> ensure that the settings are properly sourced into the shell, and they are.
>
> Paul
>
> On 20 Apr 2017, at 23:28, So good <33146...@qq.com> wrote:
>
> The zeppelin configuration file has settings for the maximum number of
> rows and the maximum size of the file.
>
> -- 原始邮件 --
> *发件人:* "Paul-Armand Verhaegen" ;
> *发送时间:* 2017年4月21日(星期五) 3:46
> *收件人:* "users" ;
> *主题:* Returning more than the default 1000 rows
>
>
> Hi,
>
> I have problems making zeppelin 0.7.1 (in %python or %spark.pyspark) to
> return more than the default 1000 rows (from a pandas dataframe) in a
> visualisation or csv download.
> I tried to increase the values of all maxResults settings in
> interpreter.json, but to no avail (and restarted zeppelin after config
> change).
>
> Can someone point me in the right direction?
>
> Thanks,
> Paul
>
>
>
>

Re: Can't delete an empty paragraph?

[moon soo Lee]

Thanks for reporting the problem.
If 'About Zeppelin' dialog missing version number, i guess it's 0.5.x.

We'll release 0.7.2 [1] in next few weeks.
Please consider use recent version if possible.

But of course, please feel free share any issues on your version.

Thanks,
moon

[1] https://issues.apache.org/jira/browse/ZEPPELIN-2276

On Fri, Apr 21, 2017 at 6:22 AM Partridge, Lucas (GE Aviation) <
lucas.partri...@ge.com> wrote:

> I can’t delete an empty paragraph in a notebook.  I’m talking about an
> empty paragraph that’s not the last paragraph. I can click on Remove under
> the paragraph’s settings icon and be prompted to delete it, but when I
> click OK the paragraph doesn’t get deleted!
>
>
>
> To delete it I have to put something in the paragraph (random text will
> do) and then execute it. Only then will the paragraph be deleted.
>
>
>
> This might have been fixed in a later version of Zeppelin. I know I’m not
> using the latest but unfortunately the ‘About Zeppelin’ dialog is missing
> the version number!
>
>
>
> Thanks, Lucas.
>
>
>

Re: Data Source Authorization - JDBC Credential

[moon soo Lee]

Hi,

Which version of Zeppelin are you using? Have you tried remove
'default.user' and 'default.password' property from interpreter menu and
create entity in credential menu?

Thanks,
moon

On Fri, Apr 21, 2017 at 11:55 AM Paul Brenner <pbren...@placeiq.com> wrote:

> Are credentials confirmed working with the JDBC interpreter? I’m trying to
> get jdbc working with snowflake. If I hardcore my username and password
> into the default url everything works great… but this of course won’t work
> in a multi user environment.
>
> However, if I try to use default username, default password, or zeppelin
> credentials I get stuck at "java.sql.SQLException: Missing user name.”
>
> Sounds like a different issue than the OP.
>
> <http://www.placeiq.com/> <http://www.placeiq.com/>
> <http://www.placeiq.com/> Paul Brenner <https://twitter.com/placeiq>
> <https://twitter.com/placeiq> <https://twitter.com/placeiq>
> <https://www.facebook.com/PlaceIQ> <https://www.facebook.com/PlaceIQ>
> <https://www.linkedin.com/company/placeiq>
> <https://www.linkedin.com/company/placeiq>
> DATA SCIENTIST
> *(217) 390-3033 <(217)%20390-3033> *
>
> <http://www.placeiq.com/2015/05/26/placeiq-named-winner-of-prestigious-2015-oracle-data-cloud-activate-award/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2016/03/08/measuring-addressable-tv-campaigns-is-now-possible/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://pages.placeiq.com/Location-Data-Accuracy-Whitepaper-Download.html?utm_source=Signature_medium=Email_campaign=AccuracyWP>
> <http://placeiq.com/2016/08/03/placeiq-bolsters-location-intelligence-platform-with-mastercard-insights/>
> <http://placeiq.com/2016/10/26/the-making-of-a-location-data-industry-milestone/>[image:
> PlaceIQ:Location Data Accuracy]
> <http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/>
>
> On Thu, Apr 20, 2017 at 8:49 PM moon soo Lee  <moon+soo+lee+%3cm...@apache.org%3E>> wrote:
>
>> Hi,
>>
>> If you remove
>>
>> 'default.user'
>> 'default.password'
>>
>> properties from jdbc interpreter setting, then Zeppelin will use database
>> username and password from credential database for each user.
>>
>> I also created a patch [1] to use credential database when
>> 'default'.user' and 'default.password' is empty string.
>>
>> Hope this helps.
>>
>> Best,
>> moon
>>
>> [1] https://github.com/apache/zeppelin/pull/2269
>>
>> On Mon, Apr 10, 2017 at 3:37 AM Arpad Beregszaszi <
>> arpad.beregsza...@gmx.de> wrote:
>>
>>> Hi all,
>>>
>>>
>>>
>>> I’m pretty new with Zeppeln and I need help with one problem regarding
>>> data source authorization as its described here:
>>>
>>>
>>>
>>>
>>> https://zeppelin.apache.org/docs/0.7.0/security/datasource_authorization.html
>>>
>>>
>>>
>>> I can successfully connect Zeppelin to my MySQL Server. Now I want to
>>> give different users access to their databases.
>>>
>>> For that I want the JDBC interpreter to use different database
>>> credentials, depending on the Zeppelin user as defined in shiro.ini.
>>>
>>> But when I create credential information, the JDBC interpreter doesn’t
>>> use them for the db connection, but the default username and password,
>>> which is empty.
>>>
>>> My Interpreter Is instantiated per user.
>>>
>>>
>>>
>>> Java.sql.SQLException: Access denied for user ‘ ‘ @ ‘IP‘ (using
>>> password: NO)
>>>
>>>
>>>
>>> Anyone an idea of whats wrong?
>>>
>>> Thanks, Arpad
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>

Re: Data Source Authorization - JDBC Credential

[moon soo Lee]

Hi,

If you remove

'default.user'
'default.password'

properties from jdbc interpreter setting, then Zeppelin will use database
username and password from credential database for each user.

I also created a patch [1] to use credential database when 'default'.user'
and 'default.password' is empty string.

Hope this helps.

Best,
moon

[1] https://github.com/apache/zeppelin/pull/2269

On Mon, Apr 10, 2017 at 3:37 AM Arpad Beregszaszi 
wrote:

> Hi all,
>
>
>
> I’m pretty new with Zeppeln and I need help with one problem regarding
> data source authorization as its described here:
>
>
>
>
> https://zeppelin.apache.org/docs/0.7.0/security/datasource_authorization.html
>
>
>
> I can successfully connect Zeppelin to my MySQL Server. Now I want to give
> different users access to their databases.
>
> For that I want the JDBC interpreter to use different database
> credentials, depending on the Zeppelin user as defined in shiro.ini.
>
> But when I create credential information, the JDBC interpreter doesn’t use
> them for the db connection, but the default username and password, which is
> empty.
>
> My Interpreter Is instantiated per user.
>
>
>
> Java.sql.SQLException: Access denied for user ‘ ‘ @ ‘IP‘ (using password:
> NO)
>
>
>
> Anyone an idea of whats wrong?
>
> Thanks, Arpad
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

Re: UnicodeDecodeError in zeppelin 0.7.1

[moon soo Lee]

Hi,

0.7.1 didn't changed any encoding type as far as i know.
One difference is 0.7.1 official artifact has been built with JDK8 while
0.7.0 built with JDK7 (we'll use JDK7 to build upcoming 0.7.2 binary). But
i'm not sure that can make pyspark and spark encoding type changes.

Do you have exactly the same interpreter setting in 0.7.1 and 0.7.0?

Thanks,
moon

On Wed, Apr 19, 2017 at 5:30 AM Meethu Mathew 
wrote:

> Hi,
>
> I just migrated from zeppelin 0.7.0 to zeppelin 0.7.1 and I am facing this
> error while creating an RDD(in pyspark).
>
> UnicodeDecodeError: 'utf8' codec can't decode byte 0x80 in position 0:
>> invalid start byte
>
>
> I was able to create the RDD without any error after adding
> use_unicode=False as follows
>
>> sc.textFile("file.csv",use_unicode=False)
>
>
> ​But it fails when I try to stem the text. I am getting similar error when
> trying to apply stemming to the text using python interpreter.
>
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 4:
>> ordinal not in range(128)
>
> All these code is working in 0.7.0 version. There is no change in the
> dataset and code. ​Is there any change in the encoding type in the new
> version of zeppelin?
>
> Regards,
>
>
> Meethu Mathew
>
>

Re: Windows Build Support

[moon soo Lee]

Hi,

Thanks for staring the thread.

I've seen almost every month people report build broken on windows. Windows
CI env will definitely help solving the problem.

I'm +1 on setting up appveyor for the project.

Due to very high usage on ASF's travis account, Zeppelin project guide each
contributor setup travis account and check success/fail using Jenkins.

I don't know how busy ASF's appveyor account. But I think we can first try
just turn on appveyor with ASF's account and see how busy it is.

Thanks,
moon

On Wed, Apr 19, 2017 at 5:15 AM Park Hoon <1am...@gmail.com> wrote:

> Hi all,
>
> Some people are struggling with build Zeppelin in windows.
>
> -
> http://stackoverflow.com/questions/40498638/install-zeppelin-on-windows/42560442#42560442
>
> IMO, it's enough to verify that zeppelin works well on windows.
>
> But it's not that hard to setup CI for windows and it's possible to
> execute integration tests on windows, it's useful. As you know, we have
> some issues related with windows env
>
> -
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20ZEPPELIN%20and%20text%20~%20windows
>
> I found that some apache projects already use appveyor to run CI on
> windows. (sadly, travis doesn't support)
>
> - (thrift) https://github.com/apache/thrift/blob/master/appveyor.yml
> - (arrow) https://issues.apache.org/jira/browse/INFRA-12967
>
>
>
> Thanks.
>
>
>

Re: How to launch a Helium Application in local deploy mode

[moon soo Lee]

Hi,

Helium application launch will appear when ResourcePool has required
resource [1] for an application.

Spark interpreter put last evaluated object in the ResourcePool.
For example, if you run "new java.util.Date()" in a paragraph, a date
object will be created in the ResourcePool. And there will be a button
displayed in the paragraph and list all applications that consumes
java.util.Date type object.

if you build Zeppelin with -Pexamples flag, then you can run "new
java.util.Date()" in spark interpreter and run example application.

Let me know if it helps.

Thanks,
moon

[1]
http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/development/writingzeppelinapplication.html#resources



On Mon, Apr 17, 2017 at 12:55 PM fish fish <fishfish...@gmail.com> wrote:

> Hi Lee,
>
> Thank you for your reply. Actually I already tried dev mode and succeeded.
> My question is how to launch Helium application not in development mode but
> in deploy mode like 'SPELL' described in
> https://zeppelin.apache.org/docs/snapshot/development/writingzeppelinspell.html#1-enabling.
> Could you please kindly guide more? Thanks!
>
> Best,
> Chen
>
> 2017-04-15 17:13 GMT+08:00 moon soo Lee <m...@apache.org>:
>
>> Hi,
>>
>> Dev mode in actual implementation became '%dev run' instead of '%helium
>> run' in proposal.
>>
>> Please check
>> http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/development/writingzeppelinapplication.html#development-mode
>> .
>>
>> To use '%dev run', you need build Zeppelin with -Phelium-dev flag.
>>
>> Hope this helps!
>>
>> Best,
>> moon
>>
>> On Fri, Apr 14, 2017 at 8:16 PM fish fish <fishfish...@gmail.com> wrote:
>>
>>> Hi Group,
>>>
>>> I have written a Helium application and deploy in local mode. Also I
>>> enable it in 'Helium' page. However, I don't know how to launch the enabled
>>> application in notebook. I tried type "%helium run" as showed in this
>>> video:
>>> https://cwiki.apache.org/confluence/display/ZEPPELIN/Helium+proposal
>>> , but error occurs which indicate no helium interpreter. Could anyone
>>> kindly tell me how to launch Helium application in right way? Thanks!
>>>
>>> Zeppelin version: 0.8.0-snapshot
>>>
>>> application.json:
>>>
>>> {
>>>   "type" : "APPLICATION",
>>>   "name" : "test_app",
>>>   "description" : "test Helium App",
>>>   "license" : "Apache-2.0",
>>>   "artifact" : "./examples/zeppelin-mytest-0.0.1-SNAPSHOT.jar",
>>>   "className" : "com.test.Test",
>>>   "resources" : [[":org.apache.spark.sql.SparkSession", 
>>> ":org.apache.spark.SparkContext", ":org.apache.spark.sql.SQLContext"]],
>>>   "icon" : ""
>>> }
>>>
>>>
>

Re: Dependency management

[moon soo Lee]

Hi,

Thanks for reporting the problem.

Downloaded dependency will be stored under 'local-repo' directory (by
default). For example after i add com.databricks:spark-xml_2.11:0.4.1 in
spark interpreter setting,

moon$ ls local-repo/2CD5YP3GK/
scala-library-2.11.7.jar spark-xml_2.11-0.4.1.jar

I see two files downloaded under ZEPPELIN_HOME/local-repo/[INTERPRETER_ID]
directory.

Hope this helps
Thanks,
moon

On Thu, Apr 13, 2017 at 10:42 AM David Howell 
wrote:

> Hi users,
>
> I hope this is a simple one and you can help me 
>
> I am having trouble adding dependency to Zeppelin Notebook (0.7.0) on AWS
> EMR (emr-5.4.0). I notice that the %dep interpreter is not available on AWS
> EMR so I can’t use that option.
>
>
>
> I follow these instructions to add the dependency:
> https://zeppelin.apache.org/docs/latest/manual/dependencymanagement.html
>
>
>
> I want to add the databricks spark-xml package for importing xml files to
> dataframes:  https://github.com/databricks/spark-xml
>
>
>
> This is the groupId:artifactId:version:
>
> com.databricks:spark-xml_2.11:0.4.1
>
>
>
> In Zeppelin, when I go to edit spark interpreter,
>
> *I enter  com.databricks:spark-xml_2.11:0.4.1 to the artifact field
>
> *click save
>
> *and then when I click OK to this dialog “Do you want to update this
> interpreter and restart with new settings – cancel | OK” click OK does
> nothing, the dialog stays on screen.
>
>
>
> I assume this is writing dependency to spark group in the
> interpreter.json, is that correct? I tried altering write permissions for
> that file but didn’t help.
>
>
>
> I confirm this is correct for my Spark/Scala version by running
> spark-shell, and since this works I assume I don’t need to add any
> additional maven repo.
>
> Maybe I do need new repo?
>
> Maybe I need to put the jar in my local repo? Interpreter.json says my
> local repo is /var/lib/zeppelin/.m2/repository but this directory does not
> exist.
>
>
>
>
>
> I can use this package from spark shell successfully:
>
>
>
> $spark-shell --packages com.databricks:spark-xml_2.11:0.4.1
>
> import org.apache.spark.sql.SQLContext
>
> val sqlContext = new SQLContext(sc)
>
> val df = sqlContext.read
>
> .format("com.databricks.spark.xml")
>
> …
>
>
>
>
>
>
>
> [image: image002.png]
>
>
>
>
>
> *David Howell*
>
> *Data Engineering*
>
>
> +61 477 150 379 <+61%20477%20150%20379>
>
>
>
> 
> [image: image004.png]
>
> 
> [image: image006.png]
>
> 
> [image: image008.png]
>
> 
> [image: image010.png]
>
>
>
>
>

Re: Restricting interpreters to users in certain groups

[moon soo Lee]

Thanks for sharing the problem.

Currently, interpreter permission does not support group/role but only
users, not like notebook permission support all group/role/user.

Could you help creating an issue for it?

Regards,
moon

On Sat, Apr 15, 2017 at 5:07 AM Paul Brenner  wrote:

> I’m trying to limit one group of users to only be able to use a single
> interpreter on 0.7.1. Is this possible right now? I managed to get AD
> authentication working and I can setup roles and was able to limit the
> group so that users can’t access the interpreter page. But I can’t figure
> out how to set the interpreter permissions based on groups. In the
> interpreter permissions I only get auto suggestion of user names. So, a few
> questions:
>
>
>1. Has https://github.com/apache/zeppelin/pull/1236
>
> 
> / https://issues.apache.org/jira/browse/ZEPPELIN-1237
>
> 
>  actually
>been implemented? It says it is for notebook permissions but when I try to
>set notebook permissions I don’t see roles there either
>2. Is pull 1236 only for notebook permissions? Is it possible that
>roles based permissions just aren’t implemented for interpreters yet?
>3. What is the autosuggest even based on? If I try to start typing my
>first name “Paul” it finds no hits. No hits with my user name “pbrenner”
>either. But if I type my last name “Brenner” then autosuggest finds me.
>4. What should be in the [roles] section in shiro.ini? I currently
>have group name = group name. Should it be group name =*? What does that
>even mean? Maybe group name = something else? See below to see what I have
>now which may be wrong.
>
>
> Here is my shiro.ini which I did a lot of guessing to get working. Maybe
> there is an issue in there? Can anyone point me to anything at all that
> might be helpful?
>
> [main]
> ### A sample for configuring Active Directory Realm
> #activeDirectoryRealm =
> org.apache.zeppelin.server.ActiveDirectoryGroupRealm
> activeDirectoryRealm = org.apache.zeppelin.realm.ActiveDirectoryGroupRealm
> activeDirectoryRealm.systemUsername = ***
> activeDirectoryRealm.systemPassword = 
> #activeDirectoryRealm.searchBase =
> CN=Users,OU=Departments,OU=PlaceIQ,DC=corp,DC=placeiq,DC=net
> activeDirectoryRealm.searchBase =
> OU=Departments,OU=PlaceIQ,DC=corp,DC=placeiq,DC=net
> #activeDirectoryRealm.url = ldaps://corp.placeiq.net:636
> #activeDirectoryRealm.url = ldaps://piq-corp-100.corp.placeiq.net:636
> activeDirectoryRealm.url = ldap://piq-corp-100.corp.placeiq.net
> activeDirectoryRealm.groupRolesMap = "CN=Security Data Science
> Department,OU=Department Security Groups,OU=Security
> Groups,OU=PlaceIQ,DC=corp,DC=placeiq,DC=net":"data_science", "CN=Security
> Development Department,OU=Department Security Groups,OU=Security
> Groups,OU=PlaceIQ,DC=corp,DC=placeiq,DC=net":"engineering", "CN=Security
> Infrastructure Department,OU=Department Security Groups,OU=Security
> Groups,OU=PlaceIQ,DC=corp,DC=placeiq,DC=net":"infra", "CN=Security Research
> & Development Department,OU=Department Security Groups,OU=Security
> Groups,OU=PlaceIQ,DC=corp,DC=placeiq,DC=net":"tech_heads", "CN=Security
> Reporting & Analytics Department,OU=Department Security Groups,OU=Security
> Groups,OU=PlaceIQ,DC=corp,DC=placeiq,DC=net":"reporting", "CN=Security
> Product Department,OU=Department Security Groups,OU=Security
> Groups,OU=PlaceIQ,DC=corp,DC=placeiq,DC=net":"product", "CN=Security Data
> Operations Department,OU=Department Security Groups,OU=Security
> Groups,OU=PlaceIQ,DC=corp,DC=placeiq,DC=net":"data_ops"
> activeDirectoryRealm.authorizationCachingEnabled = true
>
>
> ### A sample for configuring LDAP Directory Realm
> #ldapRealm = org.apache.zeppelin.server.LdapGroupRealm
> ## search base for ldap groups (only relevant for LdapGroupRealm):
> #ldapRealm.contextFactory.environment[ldap.searchBase] = dc=COMPANY,dc=COM
> #ldapRealm.contextFactory.url = ldap://ldap.test.com:389
> #ldapRealm.userDnTemplate = uid={0},ou=Users,dc=COMPANY,dc=COM
> #ldapRealm.contextFactory.authenticationMechanism = SIMPLE
>
> #ldapRealm = org.apache.zeppelin.realm.LdapGroupRealm
> #ldapRealm = org.apache.zeppelin.server.LdapGroupRealm
> # search base for ldap groups (only relevant for LdapGroupRealm):
> #ldapRealm.contextFactory.environment[ldap.searchBase] =
> cn=users,cn=accounts,dc=placeiq,dc=net
> #ldapRealm.contextFactory.url = ldap://ldap.placeiq.net:389

Re: How to launch a Helium Application in local deploy mode

[moon soo Lee]

Hi,

Dev mode in actual implementation became '%dev run' instead of '%helium
run' in proposal.

Please check
http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/development/writingzeppelinapplication.html#development-mode
.

To use '%dev run', you need build Zeppelin with -Phelium-dev flag.

Hope this helps!

Best,
moon

On Fri, Apr 14, 2017 at 8:16 PM fish fish  wrote:

> Hi Group,
>
> I have written a Helium application and deploy in local mode. Also I
> enable it in 'Helium' page. However, I don't know how to launch the enabled
> application in notebook. I tried type "%helium run" as showed in this
> video:
> https://cwiki.apache.org/confluence/display/ZEPPELIN/Helium+proposal
> , but error occurs which indicate no helium interpreter. Could anyone
> kindly tell me how to launch Helium application in right way? Thanks!
>
> Zeppelin version: 0.8.0-snapshot
>
> application.json:
>
> {
>   "type" : "APPLICATION",
>   "name" : "test_app",
>   "description" : "test Helium App",
>   "license" : "Apache-2.0",
>   "artifact" : "./examples/zeppelin-mytest-0.0.1-SNAPSHOT.jar",
>   "className" : "com.test.Test",
>   "resources" : [[":org.apache.spark.sql.SparkSession", 
> ":org.apache.spark.SparkContext", ":org.apache.spark.sql.SQLContext"]],
>   "icon" : ""
> }
>
>

Re: Dynamic Forms

[moon soo Lee]

Hi Ekta,

We don't have it yet. But i think it's good to have one.
You can always create JIRA[1] issue for new feature.

Alternative is using Angular Display System [2] to create custom UI element.

Thanks,
moon

[1] https://issues.apache.org/jira/browse/ZEPPELIN
[2]
http://zeppelin.apache.org/docs/latest/displaysystem/back-end-angular.html


On Sat, Apr 15, 2017 at 12:03 AM Ekta Nandwani 
wrote:

> Hello,
>
> I see there are dropdown, checkboxes and text inputs available in dynamic
> forms.
> https://zeppelin.apache.org/docs/0.7.1/manual/dynamicform.html
>
> I wanted to ask if there is anyway to take "date" type inputs in these
> dynamic forms?
>
> --
> Regards,
> Ekta
>

Re: Changing zeppelin theme/appearance

[moon soo Lee]

Hi,

Looks like file location in the doc is bit out dated. Please take a look
Paragraph.controller.js instead
https://github.com/apache/zeppelin/blob/master/zeppelin-web/src/app/notebook/paragraph/paragraph.controller.js
.

And thanks for feedback on mobile env. I didn't see any related issue in
JIRA. Please feel free to create one.

Thanks,
moon

On Sat, Apr 1, 2017 at 6:22 AM Paul Brenner  wrote:

> I’m trying to figure out how to make changes to zeppelin’s appearance when
> editing code. For example, when I have code after an open quote the code is
> colored blue text with red background which is illegible. I was hoping to
> just change the Ace theme following this
> https://github.com/apache/zeppelin/blob/master/STYLE.md
> 
> But I don’t know where app/scripts/controllers/paragraph.js or any other
> related file is???
>
> Bonus points if anyone knows how to hide the toolbar when accessing
> notebooks on mobile that would be amazing… that thing is gigantic and
> blocks basically everything.
>
>  
>  Paul Brenner 
>  
>  
> 
> 
> DATA SCIENTIST
> *(217) 390-3033 *
>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> [image:
> PlaceIQ:Location Data Accuracy]
> 
>

Re: Auto completion for defined variable names

[moon soo Lee]

Hi,

Thanks for reporting the problem.
Although some interpreter implements auto-completions, overall experience
on auto-completion is something that Zeppelin need to improve.

If you can file some JIRA issues, that will encourage code contributions.

Thanks,
moon

On Mon, Mar 20, 2017 at 4:51 AM Meethu Mathew 
wrote:

> Hi,
>
> Is there any way to get auto-completion or suggestions for the defined
> variable names? In Jupyter notebooks, once defined variables will show
> under suggestions.
> Ctrl+. is giving awkward suggestions for related functions also. For a
> spark data frame, it wont show the relevant functions.
>
> Please improve the suggestion functionality.
>
> Regards,
>
>
> Meethu Mathew
>
>

Re: How to let Zeppelin have map visulization function

[moon soo Lee]

Pluggable module list(Helium) on website list user package (3rd party) in
npm registry. Each plugin can have it's own license which may/may not
compatible to Apache 2 License, while zeppelin is not including them in the
release.

So, accepting the plugin license is up to individual user when enabling the
plugin.

Map is not on the list yet, hope anyone create a map visualization plugin
and publish it in the npm, so everyone can use it through Helium menu.

License issue of leaflet used in
https://github.com/apache/zeppelin/pull/765/files is similar. Leaflet
library itself is BSD2, but it also matters with license that data provider
you use. I think easiest way dealing with is make map visualization as
Helium plugin package and let individual user accept license when enabling
the package.

Thanks,
moon

On Tue, Mar 21, 2017 at 9:05 AM mingda li  wrote:

> Dear moon,
> Thanks for your explain. But I don't find map visualization in the
> pluggable list.
> What is the map visualization without license issue? Is it this one
> https://github.com/apache/zeppelin/pull/765/files?
> Or you can provide me your link.
> If I pull from that one to my source tree, do I need to use mvn build
> again to let it run ?
>
> Thanks so much. We really need the map visualization function.
>
>
> Hi Mingda,
>
> Zeppelin had map visualization once in it's source tree, but unfortunately
> it's removed because of license issue[1].
>
> Of course we can try find another map visualization without license issue
> to embed it in the source tree.
>
> Or there's another way.
> Visualization became pluggable since 0.7.0. Zeppelin 0.8.0 (snapshot) even
> supports online catalogue for all available zeppelin visualization in npm
> registry, so user can just click a button and use.
>
> You can see there're already some plugins for visualizations[2], and it'll
> not be very difficult to create one and publish to npm [3].
>
> Thanks,
> moon
>
> [1] https://github.com/apache/zeppelin/pull/1501
> [2] http://zeppelin.apache.org/helium_packages.html
> [3]
> http://zeppelin.apache.org/docs/snapshot/development/writingzeppelinvisualization.html
>
> On Mon, Mar 20, 2017 at 11:06 PM mingda li  wrote:
>
> Dear all,
>
> Now, I wan to let Zeppelin 0.7.0 and 0.6.0 support the map visualization.
> Is that function support by the official version? Or we need to do
> something like here
> https://github.com/apache/zeppelin/pull/152
> https://github.com/apache/zeppelin/pull/765
>
> Thanks,
> Mingda
>
>

Re: Re: Roadmap for 0.8.0

[moon soo Lee]

Thanks for the opinion. Yes we can think about proper label. These are
labels mentioned in this threads so far.

'Unstable', 'Stable', 'Old'
'Latest', 'Stable', 'Old'
'Beta', 'Stable', 'Old'

Intention is not that we want to release 'unstable' version, i think.
The intention is give user proper expectation that latest release may(and
may not) include bug which we couldn't discovered in verification process
like what happened in our previous release 0.7.0 and 0.6.0.

These are how other apache projects describe their releases.

Kafka - x.x.z is the latest release. current stable version is x.y.z.
Flink - x.y.z is our latest stable release
Cassandra - even-numbered contains new features, odd-numbered contains bug
fixes only
Spark - available 2.1.0, 2.0.2, 2.0.1, 2.0.0  1.4.0 as a 'stable'
release, others are available as 'archived' releases.
Mesos - most recent stable release: x.y.z
Hadoop - 'x.y.z-alpha' or 'x.y.z' or 'x.y.z (stable)'
Hbase - 1.2.x series is current stable release. (while 1.3.x series does
not have a label)

As you can see, it's difficult to find common rule what 'latest' should
mean in Apache projects.

Considering the intention that we're not intentionally releasing 'unstable'
version, i prefer 'latest / stable' tiny bit more than 'beta / stable'.

I'd like hear more opinions.

Thanks,
moon

On Tue, Mar 21, 2017 at 9:16 AM Jan Rasehorn <j.raseh...@gmx.de> wrote:

> Hi moon,
>
> I think assuming the latest release would be unstable is confusing and not
> in line with other Apache projects. If you want to have a instable
> prerelease version, I would suggest to call it a beta version and once the
> major bugs are removed, a new stable release could be provided.
>
> BR, Jan
> --
> Diese Nachricht wurde von meinem Android Mobiltelefon mit GMX Mail
> gesendet.
> Am 21.03.17, 16:41, moon soo Lee <m...@apache.org> schrieb:
>
> And if i suggest simplest way for us to set quality expectation to user,
> which will be labeling release in download page.
>
> Currently releases are divided into 2 categories in download page. 'Latest
> release' and 'Old releases'. I think we can treat 'Latest' as unstable and
> add one more category 'Stable release'.
>
> For example, once 0.8.0 is released,
>
> Latest release : 0.8.0
> Stable release : 0.7.1
> Old release : 0.6.2, 0.6.1 
>
> Once we feel confident about the stability of latest release, we can just
> change label from latest to stable in the download page. (and previous
> stable goes to old releases)
> We can even include formal vote for moving release from 'latest' to
> 'stable' in our release process, if it is necessary.
>
> Thanks,
> moon
>
> On Tue, Mar 21, 2017 at 6:59 AM moon soo Lee <m...@apache.org> wrote:
>
> Yes, having longer RC period will help.
>
> But if i recall 0.7.0 release, although 21 people participated verifying
> through 4 RC for 15days, it wasn't enough to catch all critical problems
> during the release process. After the release, we've got much more number
> of bug reports, in next few days.
>
> Basically, verifying RC is limited to people who subscribe mailing list +
> willing to contribute time to verify RC, which is much smaller number of
> people who download release from download page. So having longer RC period
> will definitely help and i think we should do, but I think it's still not
> enough to make sure the quality, considering past history.
>
> AFAIK, releasing 0.8.0-preview, calling it unstable is up to the project.
> ASF release process defines how to release source code, but it does not
> really restrict what kind of 'version' the project should have releases.
> For example, spark released spark-2.0.0-preview[1] before spark-2.0.0.
>
> Thanks,
> moon
>
> [1] http://spark.apache.org/news/spark-2.0.0-preview.html
>
>
> On Mon, Mar 20, 2017 at 11:31 PM Jongyoul Lee <jongy...@gmail.com> wrote:
>
> I agree that it will help prolong RC period and use it actually. And also
> we need code freeze for the new features and spend time to stabilize RC.
>
> On Tue, Mar 21, 2017 at 1:25 PM, Felix Cheung <felixcheun...@hotmail.com>
> wrote:
>
> +1 on quality and stabilization.
>
> I'm not sure if releasing as preview or calling it unstable fits with the
> ASF release process though.
>
> Other projects have code freeze, RC (and longer RC iteration time) etc. -
> do we think those will help improve quality when the release is finally cut?
>
>
> _
> From: Jianfeng (Jeff) Zhang <jzh...@hortonworks.com>
> Sent: Monday, March 20, 2017 6:13 PM
> Subject: Re: Roadmap for 0.8.0
> To: <users@zeppelin.apache.org>, dev <d...@zeppelin.apache.org>
>
>
>
> Strongly +1 for adding system test for diffe

Re: Roadmap for 0.8.0

[moon soo Lee]

And if i suggest simplest way for us to set quality expectation to user,
which will be labeling release in download page.

Currently releases are divided into 2 categories in download page. 'Latest
release' and 'Old releases'. I think we can treat 'Latest' as unstable and
add one more category 'Stable release'.

For example, once 0.8.0 is released,

Latest release : 0.8.0
Stable release : 0.7.1
Old release : 0.6.2, 0.6.1 

Once we feel confident about the stability of latest release, we can just
change label from latest to stable in the download page. (and previous
stable goes to old releases)
We can even include formal vote for moving release from 'latest' to
'stable' in our release process, if it is necessary.

Thanks,
moon

On Tue, Mar 21, 2017 at 6:59 AM moon soo Lee <m...@apache.org> wrote:

> Yes, having longer RC period will help.
>
> But if i recall 0.7.0 release, although 21 people participated verifying
> through 4 RC for 15days, it wasn't enough to catch all critical problems
> during the release process. After the release, we've got much more number
> of bug reports, in next few days.
>
> Basically, verifying RC is limited to people who subscribe mailing list +
> willing to contribute time to verify RC, which is much smaller number of
> people who download release from download page. So having longer RC period
> will definitely help and i think we should do, but I think it's still not
> enough to make sure the quality, considering past history.
>
> AFAIK, releasing 0.8.0-preview, calling it unstable is up to the project.
> ASF release process defines how to release source code, but it does not
> really restrict what kind of 'version' the project should have releases.
> For example, spark released spark-2.0.0-preview[1] before spark-2.0.0.
>
> Thanks,
> moon
>
> [1] http://spark.apache.org/news/spark-2.0.0-preview.html
>
>
> On Mon, Mar 20, 2017 at 11:31 PM Jongyoul Lee <jongy...@gmail.com> wrote:
>
> I agree that it will help prolong RC period and use it actually. And also
> we need code freeze for the new features and spend time to stabilize RC.
>
> On Tue, Mar 21, 2017 at 1:25 PM, Felix Cheung <felixcheun...@hotmail.com>
> wrote:
>
> +1 on quality and stabilization.
>
> I'm not sure if releasing as preview or calling it unstable fits with the
> ASF release process though.
>
> Other projects have code freeze, RC (and longer RC iteration time) etc. -
> do we think those will help improve quality when the release is finally cut?
>
>
> _
> From: Jianfeng (Jeff) Zhang <jzh...@hortonworks.com>
> Sent: Monday, March 20, 2017 6:13 PM
> Subject: Re: Roadmap for 0.8.0
> To: <users@zeppelin.apache.org>, dev <d...@zeppelin.apache.org>
>
>
>
> Strongly +1 for adding system test for different interpreter modes and
> focus on bug fixing than new features. I do heard from some users complain
> about the bugs of zeppelin major release. A stabilized release is very
> necessary for community.
>
>
>
>
> Best Regard,
> Jeff Zhang
>
>
> From: moon soo Lee <m...@apache.org<mailto:m...@apache.org
> <m...@apache.org>>>
> Reply-To: "users@zeppelin.apache.org<mailto:users@zeppelin.apache.org
> <users@zeppelin.apache.org>>" <users@zeppelin.apache.org<
> mailto:users@zeppelin.apache.org <users@zeppelin.apache.org>>>
> Date: Tuesday, March 21, 2017 at 4:10 AM
> To: "users@zeppelin.apache.org<mailto:users@zeppelin.apache.org
> <users@zeppelin.apache.org>>" <users@zeppelin.apache.org<
> mailto:users@zeppelin.apache.org <users@zeppelin.apache.org>>>, dev <
> d...@zeppelin.apache.org<mailto:d...@zeppelin.apache.org
> <d...@zeppelin.apache.org>>>
>
> Subject: Re: Roadmap for 0.8.0
>
> Great to see discussion for 0.8.0.
> List of features for 0.8.0 looks really good.
>
> Interpreter factory refactoring
> Interpreter layer supports various behavior depends on combination of
> PerNote,PerUser / Shared,Scoped,Isolated. We'll need strong test cases for
> each combination as a first step.
> Otherwise, any pullrequest will silently break one of behavior at any time
> no matter we refactor or not. And fixing and testing this behavior is so
> hard.
> Once we have complete test cases, not only guarantee the behavior but also
> make refactoring much easier.
>
>
> 0.8.0 release
> I'd like to suggest improvements on how we release a new version.
>
> In the past, 0.6.0 and 0.7.0 release with some critical problems. (took 3
> months to stabilize 0.6 and we're working on stabilizing 0.7.0 for 2 months)
>
> I think the same thing will happen again with 0.8.0, while we're going to

Re: How to let Zeppelin have map visulization function

[moon soo Lee]

Hi Mingda,

Zeppelin had map visualization once in it's source tree, but unfortunately
it's removed because of license issue[1].

Of course we can try find another map visualization without license issue
to embed it in the source tree.

Or there's another way.
Visualization became pluggable since 0.7.0. Zeppelin 0.8.0 (snapshot) even
supports online catalogue for all available zeppelin visualization in npm
registry, so user can just click a button and use.

You can see there're already some plugins for visualizations[2], and it'll
not be very difficult to create one and publish to npm [3].

Thanks,
moon

[1] https://github.com/apache/zeppelin/pull/1501
[2] http://zeppelin.apache.org/helium_packages.html
[3]
http://zeppelin.apache.org/docs/snapshot/development/writingzeppelinvisualization.html

On Mon, Mar 20, 2017 at 11:06 PM mingda li  wrote:

> Dear all,
>
> Now, I wan to let Zeppelin 0.7.0 and 0.6.0 support the map visualization.
> Is that function support by the official version? Or we need to do
> something like here
> https://github.com/apache/zeppelin/pull/152
> https://github.com/apache/zeppelin/pull/765
>
> Thanks,
> Mingda
>

Re: Roadmap for 0.8.0

[moon soo Lee]

Great to see discussion for 0.8.0.
List of features for 0.8.0 looks really good.

*Interpreter factory refactoring*
Interpreter layer supports various behavior depends on combination of
PerNote,PerUser / Shared,Scoped,Isolated. We'll need strong test cases for
each combination as a first step.
Otherwise, any pullrequest will silently break one of behavior at any time
no matter we refactor or not. And fixing and testing this behavior is so
hard.
Once we have complete test cases, not only guarantee the behavior but also
make refactoring much easier.


*0.8.0 release*
I'd like to suggest improvements on how we release a new version.

In the past, 0.6.0 and 0.7.0 release with some critical problems. (took 3
months to stabilize 0.6 and we're working on stabilizing 0.7.0 for 2 months)

I think the same thing will happen again with 0.8.0, while we're going to
make lots of changes and add many new features.
After we released 0.8.0, while 'Stabilizing' the new release, user who
tried the new release may get wrong impression of the quality. Which is
very bad and we already repeated the mistake in 0.6.0 and 0.7.0.

So from 0.8.0 release, I'd suggest we improve way we release new version to
give user proper expectation. I think there're several ways of doing it.

1. Release 0.8.0-preview officially and then release 0.8.0.
2. Release 0.8.0 with 'beta' or 'unstable' label. And keep 0.7.x as a
'stable' release in the download page. Once 0.8.x release becomes stable
enough make 0.8.x release as a 'stable' and move 0.7.x to 'old' releases.


*After 0.8.0, *
Since Zeppelin projects starts, project went through some major milestone,
like

- project gets first users and first contributor
- project went into Apache Incubator
- project became TLP.

And I think it's time to think about hitting another major milestone.

Considering features we already have, features we're planning on 0.8, wide
adoption of Zeppelin in the industry, I think it's time to focus on make
project more mature and make a 1.0 release. Which i think big milestone for
the project.

After 0.8.0 release, I suggest we more focus on bug fixes, stability
improvement, optimizing user experience than adding new features. And with
subsequent minor release, 0.8.1, 0.8.2 ... moment we feel confident about
the quality, release it as a 1.0.0 instead of 0.8.x.

Once we have 1.0.0 released, then I think we can make larger, experimental
changes on 2.0.0 branch aggressively, while we keep maintaining 1.0.x
branch.


Thanks,
moon

On Mon, Mar 20, 2017 at 8:55 AM Felix Cheung 
wrote:

> There are several pending visualization improvements/PRs that would be
> very good to get them in as well.
>
>
> --
> *From:* Jongyoul Lee 
> *Sent:* Sunday, March 19, 2017 9:03:24 PM
> *To:* dev; users@zeppelin.apache.org
> *Subject:* Roadmap for 0.8.0
>
> Hi dev & users,
>
> Recently, community submits very new features for Apache Zeppelin. I think
> it's very positive signals to improve Apache Zeppelin and its community.
> But in another aspect, we should focus on what the next release includes. I
> think we need to summarize and prioritize them. Here is what I know:
>
> * Cluster management
> * Admin feature
> * Replace some context to separate users
> * Helium online
>
> Feel free to talk if you want to add more things. I think we need to
> choose which features will be included in 0.8.0, too.
>
> Regards,
> Jongyoul Lee
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>

Re: Should zeppelin.pyspark.python be used on the worker nodes ?

[moon soo Lee]

When property key in interpreter configuration screen matches certain
condition [1], it'll be treated as a environment variable.

You can remove PYSPARK_PYTHON from conf/zeppelin-env.sh and place it in
interpreter configuration.

Thanks,
moon

[1]
https://github.com/apache/zeppelin/blob/master/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreter.java#L152


On Mon, Mar 20, 2017 at 12:21 PM William Markito Oliveira <
william.mark...@gmail.com> wrote:

> Thanks for the quick response Ruslan.
>
> But given that it's an environment variable, I can't quickly change that
> value and point to a different python environment without restarting the
> Zeppelin process, can I ? I mean is there a way to set the value for
> PYSPARK_PYTHON from the Interpreter configuration screen ?
>
> Thanks,
>
>
> On Mon, Mar 20, 2017 at 2:15 PM, Ruslan Dautkhanov 
> wrote:
>
> You can set PYSPARK_PYTHON environment variable for that.
>
> Not sure about zeppelin.pyspark.python. I think it does not work
> See comments in https://issues.apache.org/jira/browse/ZEPPELIN-1265
>
> Eventually, i think we can remove zeppelin.pyspark.python and use only
> PYSPARK_PYTHON instead to avoid confusion.
>
>
> --
> Ruslan Dautkhanov
>
> On Mon, Mar 20, 2017 at 12:59 PM, William Markito Oliveira <
> mark...@apache.org> wrote:
>
> I'm trying to use zeppelin.pyspark.python as the variable to set the
> python that Spark worker nodes should use for my job, but it doesn't seem
> to be working.
>
> Am I missing something or this variable does not do that ?
>
> My goal is to change that variable to point to different conda
> environments.  These environments are available in all worker nodes since
> it's on a shared location and ideally all nodes then would have access to
> the same libraries and dependencies.
>
> Thanks,
>
> ~/William
>
>
>
>
>
> --
> ~/William
>

Re: How to bind angular object with backend when write Helium Application

[moon soo Lee]

Hi Hishfish,

If you take a look Clock example [1], you'll see how it creates angular
objects and update every seconds from backend, so front-end can be updated
accordingly.

After you add your object into AngularObjectRegistry, you can get
AngularObject and add watcher [2]. Then any changes of the angular scope
variable from front end will trigger your watcher where you can place your
backend logics.

Hope this helps,

Thanks,
moon

[1]
https://github.com/apache/zeppelin/blob/master/zeppelin-examples/zeppelin-example-clock/src/main/java/org/apache/zeppelin/example/app/clock/Clock.java#L72
[2]
https://github.com/apache/zeppelin/blob/master/zeppelin-interpreter/src/main/java/org/apache/zeppelin/display/AngularObject.java#L219


On Mon, Mar 20, 2017 at 8:47 AM fish fish  wrote:

> Hi Group,
>
> Recently we are exploring building data analysis application based on
> Zeppelin. We checked Helium document and think it could be an appropriate
> way to customize both frontend and backend in Zeppelin. However, we did not
> find a way to bind angular object with backend data when extends
> Application. More detail, we want to catch the frontend angular object
> value / change event in the backend and then process in some way to
> feedback to frontend. Could someone kindly show us the right way to do
> this? We are now using 0.8.0 snapshot code. Thank you very much in advance!
>
> Best,
>
> Fishfish
>

Re: How do I configure R interpreter in Zeppelin?

[moon soo Lee]

Easiest way to figure out what your environment needs is,

1. run SPARK_HOME/bin/sparkR in your shell and make sure it works in the
same host where Zeppelin going to run.
2. try use %spark.r in Zeppelin with SPARK_HOME configured. Normally it
should work when 1) works without problem, otherwise take a look error
message and error log to get more informations.

Thanks,
moon

On Sat, Mar 18, 2017 at 8:47 PM Shanmukha Sreenivas Potti <
shanmu...@utexas.edu> wrote:

> I'm not 100% sure as I haven't set it up but it looks like I'm using
> Zeppelin preconfigured with Spark and I've also taken a snapshot of the
> Spark Interpreter configuration that I have access to/using in Zeppelin.
> This interpreter comes with SQL and Python integration and I'm figuring out
> how do I get to use R.
>
> On Sat, Mar 18, 2017 at 8:06 PM, moon soo Lee <m...@apache.org> wrote:
>
> AFAIK, Amazon EMR service has an option that launches Zeppelin
> (preconfigured) with Spark. Do you use Zeppelin provided by EMR or are you
> setting up Zeppelin separately?
>
> Thanks,
> moon
>
> On Sat, Mar 18, 2017 at 4:13 PM Shanmukha Sreenivas Potti <
> shanmu...@utexas.edu> wrote:
>
> ​​
> Hi Moon,
>
> Thanks for responding. Exporting Spark_home is exactly where I have a
> problem. I'm using Zeppelin notebook with Spark on EMR clusters from an AWS
> account on cloud. I'm not the master account holder for that AWS account
> but I'm guessing I'm a client account with limited access probably. Can I
> still do it?
>
> If yes, can you explain where and how should I do that shell scripting to
> export the variable? Can I do this in the notebook itself by starting the
> paragraph with sh% or do I need to do something else?
> If you can share any video that would be great. I would like to let you
> know that I'm a novice user just getting to explore Big Data.
>
> Sharing more info for better context.
>
> Here's my AWS account detail type:
> assumed-role/ConduitAccessClientRole-DO-NOT-DELETE/shan
>
> Spark Interpreter config in Zeppelin:
> [image: image.png]
>
> Thanks for your help.
>
> Shan
>
> On Sat, Mar 18, 2017 at 8:39 AM, moon soo Lee <m...@apache.org> wrote:
>
> If you don't have spark cluster, then you don't need to do 2).
> After 1) %spark.r interpreter should work.
>
> If you do have spark cluster, export SPARK_HOME env variable in
> conf/zeppelin-env.sh, that should be enough make it work.
>
> Hope this helps.
>
> Thanks,
> moon
>
> On Fri, Mar 17, 2017 at 2:41 PM Shanmukha Sreenivas Potti <
> shanmu...@utexas.edu> wrote:
>
> Hello Group!
>
> I'm trying to leverage various R functions in Zeppelin but am having
> challenges in figuring out how to configure the Spark interpreter/
> SPARK_HOME variable.
>
> I'm going by this
> <https://zeppelin.apache.org/docs/0.6.0/interpreter/r.html> documentation
> for now, and specifically have issues with the following steps:
>
>1.
>
>To run R code and visualize plots in Apache Zeppelin, you will need R
>on your master node (or your dev laptop).
>
>For Centos: yum install R R-devel libcurl-devel openssl-devel For
>Ubuntu: apt-get install r-base
>
> How do I figure out the master node and install the R interpreter? Novice
> user here.
>
>
> 2. To run Zeppelin with the R Interpreter, the SPARK_HOME environment
> variable must be set. The best way to do this is by editing
> conf/zeppelin-env.sh. If it is not set, the R Interpreter will not be able
> to interface with Spark. You should also copy
> conf/zeppelin-site.xml.template to conf/zeppelin-site.xml. That will ensure
> that Zeppelin sees the R Interpreter the first time it starts up.
>
> No idea as to how to do step 2 either.
>
> Appreciate your help. If there is a video that you can point me to that
> talks about these steps, that would be fantabulous.
>
> Thanks! Shan
>
> --
> Shan S. Potti,
>
>
>
>
> --
> Shan S. Potti,
> 737-333-1952 <(737)%20333-1952>
> https://www.linkedin.com/in/shanmukhasreenivas
>
> On Sat, Mar 18, 2017 at 8:39 AM, moon soo Lee <m...@apache.org> wrote:
>
> If you don't have spark cluster, then you don't need to do 2).
> After 1) %spark.r interpreter should work.
>
> If you do have spark cluster, export SPARK_HOME env variable in
> conf/zeppelin-env.sh, that should be enough make it work.
>
> Hope this helps.
>
> Thanks,
> moon
>
> On Fri, Mar 17, 2017 at 2:41 PM Shanmukha Sreenivas Potti <
> shanmu...@utexas.edu> wrote:
>
> Hello Group!
>
> I'm trying to leverage various R functions in Zeppelin but am having
> challenges in figuring out how to configure the Spark interpreter/
>

Re: How do I configure R interpreter in Zeppelin?

[moon soo Lee]

AFAIK, Amazon EMR service has an option that launches Zeppelin
(preconfigured) with Spark. Do you use Zeppelin provided by EMR or are you
setting up Zeppelin separately?

Thanks,
moon

On Sat, Mar 18, 2017 at 4:13 PM Shanmukha Sreenivas Potti <
shanmu...@utexas.edu> wrote:

> ​​
> Hi Moon,
>
> Thanks for responding. Exporting Spark_home is exactly where I have a
> problem. I'm using Zeppelin notebook with Spark on EMR clusters from an AWS
> account on cloud. I'm not the master account holder for that AWS account
> but I'm guessing I'm a client account with limited access probably. Can I
> still do it?
>
> If yes, can you explain where and how should I do that shell scripting to
> export the variable? Can I do this in the notebook itself by starting the
> paragraph with sh% or do I need to do something else?
> If you can share any video that would be great. I would like to let you
> know that I'm a novice user just getting to explore Big Data.
>
> Sharing more info for better context.
>
> Here's my AWS account detail type:
> assumed-role/ConduitAccessClientRole-DO-NOT-DELETE/shan
>
> Spark Interpreter config in Zeppelin:
> [image: image.png]
>
> Thanks for your help.
>
> Shan
>
> On Sat, Mar 18, 2017 at 8:39 AM, moon soo Lee <m...@apache.org> wrote:
>
> If you don't have spark cluster, then you don't need to do 2).
> After 1) %spark.r interpreter should work.
>
> If you do have spark cluster, export SPARK_HOME env variable in
> conf/zeppelin-env.sh, that should be enough make it work.
>
> Hope this helps.
>
> Thanks,
> moon
>
> On Fri, Mar 17, 2017 at 2:41 PM Shanmukha Sreenivas Potti <
> shanmu...@utexas.edu> wrote:
>
> Hello Group!
>
> I'm trying to leverage various R functions in Zeppelin but am having
> challenges in figuring out how to configure the Spark interpreter/
> SPARK_HOME variable.
>
> I'm going by this
> <https://zeppelin.apache.org/docs/0.6.0/interpreter/r.html> documentation
> for now, and specifically have issues with the following steps:
>
>1.
>
>To run R code and visualize plots in Apache Zeppelin, you will need R
>on your master node (or your dev laptop).
>
>For Centos: yum install R R-devel libcurl-devel openssl-devel For
>Ubuntu: apt-get install r-base
>
> How do I figure out the master node and install the R interpreter? Novice
> user here.
>
>
> 2. To run Zeppelin with the R Interpreter, the SPARK_HOME environment
> variable must be set. The best way to do this is by editing
> conf/zeppelin-env.sh. If it is not set, the R Interpreter will not be able
> to interface with Spark. You should also copy
> conf/zeppelin-site.xml.template to conf/zeppelin-site.xml. That will ensure
> that Zeppelin sees the R Interpreter the first time it starts up.
>
> No idea as to how to do step 2 either.
>
> Appreciate your help. If there is a video that you can point me to that
> talks about these steps, that would be fantabulous.
>
> Thanks! Shan
>
> --
> Shan S. Potti,
>
>
>
>
> --
> Shan S. Potti,
> 737-333-1952 <(737)%20333-1952>
> https://www.linkedin.com/in/shanmukhasreenivas
>
> On Sat, Mar 18, 2017 at 8:39 AM, moon soo Lee <m...@apache.org> wrote:
>
> If you don't have spark cluster, then you don't need to do 2).
> After 1) %spark.r interpreter should work.
>
> If you do have spark cluster, export SPARK_HOME env variable in
> conf/zeppelin-env.sh, that should be enough make it work.
>
> Hope this helps.
>
> Thanks,
> moon
>
> On Fri, Mar 17, 2017 at 2:41 PM Shanmukha Sreenivas Potti <
> shanmu...@utexas.edu> wrote:
>
> Hello Group!
>
> I'm trying to leverage various R functions in Zeppelin but am having
> challenges in figuring out how to configure the Spark interpreter/
> SPARK_HOME variable.
>
> I'm going by this
> <https://zeppelin.apache.org/docs/0.6.0/interpreter/r.html> documentation
> for now, and specifically have issues with the following steps:
>
>1.
>
>To run R code and visualize plots in Apache Zeppelin, you will need R
>on your master node (or your dev laptop).
>
>For Centos: yum install R R-devel libcurl-devel openssl-devel For
>Ubuntu: apt-get install r-base
>
> How do I figure out the master node and install the R interpreter? Novice
> user here.
>
>
> 2. To run Zeppelin with the R Interpreter, the SPARK_HOME environment
> variable must be set. The best way to do this is by editing
> conf/zeppelin-env.sh. If it is not set, the R Interpreter will not be able
> to interface with Spark. You should also copy
> conf/zeppelin-site.xml.template to conf/zeppelin-site.xml. That will ensure
> that Zeppelin sees the R Interpreter the first time it starts up.
>
> No idea as to how to do step 2 either.
>
> Appreciate your help. If there is a video that you can point me to that
> talks about these steps, that would be fantabulous.
>
> Thanks! Shan
>
> --
> Shan S. Potti,
>
>
>
>
> --
> Shan S. Potti,
> 737-333-1952 <(737)%20333-1952>
> https://www.linkedin.com/in/shanmukhasreenivas
>

Re: How do I configure R interpreter in Zeppelin?

[moon soo Lee]

If you don't have spark cluster, then you don't need to do 2).
After 1) %spark.r interpreter should work.

If you do have spark cluster, export SPARK_HOME env variable in
conf/zeppelin-env.sh, that should be enough make it work.

Hope this helps.

Thanks,
moon

On Fri, Mar 17, 2017 at 2:41 PM Shanmukha Sreenivas Potti <
shanmu...@utexas.edu> wrote:

> Hello Group!
>
> I'm trying to leverage various R functions in Zeppelin but am having
> challenges in figuring out how to configure the Spark interpreter/
> SPARK_HOME variable.
>
> I'm going by this
>  documentation
> for now, and specifically have issues with the following steps:
>
>1.
>
>To run R code and visualize plots in Apache Zeppelin, you will need R
>on your master node (or your dev laptop).
>
>For Centos: yum install R R-devel libcurl-devel openssl-devel For
>Ubuntu: apt-get install r-base
>
> How do I figure out the master node and install the R interpreter? Novice
> user here.
>
>
> 2. To run Zeppelin with the R Interpreter, the SPARK_HOME environment
> variable must be set. The best way to do this is by editing
> conf/zeppelin-env.sh. If it is not set, the R Interpreter will not be able
> to interface with Spark. You should also copy
> conf/zeppelin-site.xml.template to conf/zeppelin-site.xml. That will ensure
> that Zeppelin sees the R Interpreter the first time it starts up.
>
> No idea as to how to do step 2 either.
>
> Appreciate your help. If there is a video that you can point me to that
> talks about these steps, that would be fantabulous.
>
> Thanks! Shan
>
> --
> Shan S. Potti,
>
>

Re: Release on 0.7.1 and 0.7.2

[moon soo Lee]

Release process is expected to start on this Friday.

Release process takes minimum 4 days (3 days vote + 1 day mirror sync), It
may take more days depends on how many release candidate we go through
during vote, because each release candidate will restarts vote that takes 3
days.

Hope this helps.

Thanks,
moon

On Wed, Mar 15, 2017 at 12:48 PM Sofiane Cherchalli <sofian...@gmail.com>
wrote:

> Hi Zeppelin team,
>
> What's the release forecast? Shall we expect 0.7.1 or 0.7.2 by Friday?
>
> Thanks.
>
> On Tue, 14 Mar 2017 at 13:09, Jianfeng (Jeff) Zhang <
> jzh...@hortonworks.com> wrote:
>
>
> +1
>
> Best Regard,
> Jeff Zhang
>
>
> From: Jun Kim <i2r@gmail.com>
> Reply-To: "users@zeppelin.apache.org" <users@zeppelin.apache.org>
> Date: Tuesday, March 14, 2017 at 11:38 AM
> To: "users@zeppelin.apache.org" <users@zeppelin.apache.org>
> Subject: Re: Release on 0.7.1 and 0.7.2
>
> Cool! I look forward to it!
>
> 2017년 3월 14일 (화) 오후 12:31, moon soo Lee <m...@apache.org>님이 작성:
>
> Sounds like a plan!
>
>
> On Mon, Mar 13, 2017 at 8:22 PM Xiaohui Liu <hero...@gmail.com> wrote:
>
> This is the right action. In fact, 0.7.0 release bin did not work for my
> team. We almost started to use 0.7.1-snapshot immediately after 0.7.0
> release.
>
> I guess many of us are taking the same route.
>
> But for new zeppelin users, starting with 0.7.0 will give them the wrong
> first impression.
>
>
> On Tue, 14 Mar 2017 at 10:28 AM, Jongyoul Lee <jongy...@gmail.com> wrote:
>
> Hi dev and users,
>
> As we released 0.7.0, most of users and dev reported a lot of bugs which
> were critical. For the reason, community including me started to prepare
> new minor release with umbrella issue[1]. Due to contributors' efforts, we
> have resolved some of issues and have reviewed almost unresolved issues. I
> want to talk about the new minor release at this point. Generally, we have
> resolved all of issues reported as bugs before we release but some issues
> are very critical and it causes serious problem using Apache Zeppelin. Then
> I think, in this time, it's better to release 0.7.1 as soon as we can and
> prepare a new minor release with rest of unresolved issues.
>
> I'd like to start a process this Friday and if some issues are not merged
> until then, I hope they would be included in 0.7.2.
>
> Feel free to talk to me if you have a better plan to improve users'
> experiences.
>
> Regards,
> Jongyoul Lee
>
> [1] https://issues.apache.org/jira/browse/ZEPPELIN-2134
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>
> --
> Taejun Kim
>
> Data Mining Lab.
> School of Electrical and Computer Engineering
> University of Seoul
>
>

Re: Subject: Re: Zeppelin %dev Interpreter

[moon soo Lee]

Each application define what kind of resource the application requires [1]
(e.g. 'Clock' example requires java.util.Date [2]).

Once interpreter pushes required resource into ResourcePool, then Zeppelin
display buttons for available application in paragraph.

SparkInterpreter automatically pushes last evaluation into ResourcePool.
So if you just run '%spark new java.util.Date` then SparkInterpreter pushes
a Date object into ResourcePool and show button to run 'Clock'.

Thanks,
moon

[1]
http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/development/writingzeppelinapplication.html#resources
[2]
https://github.com/apache/zeppelin/blob/master/zeppelin-examples/zeppelin-example-clock/zeppelin-example-clock.json#L23



On Sat, Mar 11, 2017 at 12:45 AM Andreas Weise <andreas.we...@gmail.com>
wrote:

> Thanks. After building with -Phelium-dev I was able to add dev interpreter.
>
> One follow up question:
> Now after running a helium application (e.g. Clock in zeppelin-examples)
> via dev mode, I'm trying to figure out, how a helium application is started
> without %dev interpreter.
>
> While for Helium Visualizations I find addtional chart buttons, there is
> nothing similar available for Helium Applications. Of course I find it
> listed on the /helium Page, where I can enable/disable it. If enabled, I
> can not see, how to actually use the application.
>
> Thanks.
> Andreas
>
>
> ------ Forwarded message --
> From: moon soo Lee <m...@apache.org>
> To: users@zeppelin.apache.org
> Cc:
> Bcc:
> Date: Sat, 11 Mar 2017 01:31:52 +
> Subject: Re: Zeppelin %dev Interpreter
> Have you tried add -Phelium-dev profile in your maven build command?
>
> Thanks,
> moon
>
> On Fri, Mar 10, 2017 at 12:11 PM Andreas Weise <andreas.we...@gmail.com>
> wrote:
>
> Hi,
>
> various documentations describe the usage of %dev interpreter, e.g. when
> developing Helium Applications.
>
> https://zeppelin.apache.org/docs/0.7.0/development/writingzeppelinapplication.html
>
> I'm wondering how the %dev Interpreter gets added to zeppelin a server.
>
> I already tried starting zeppelin as described here:
>
> https://zeppelin.apache.org/docs/0.7.0/development/howtocontribute.html#run-zeppelin-server-in-development-mode
>
>  But without luck. Same for starting an zeppelin-web instance on
> http://localhost:9000
>
> Any advice.
>
> Thanks in advance.
> Andreas
>
>
>
>

Re: Release on 0.7.1 and 0.7.2

[moon soo Lee]

Sounds like a plan!

On Mon, Mar 13, 2017 at 8:22 PM Xiaohui Liu  wrote:

> This is the right action. In fact, 0.7.0 release bin did not work for my
> team. We almost started to use 0.7.1-snapshot immediately after 0.7.0
> release.
>
> I guess many of us are taking the same route.
>
> But for new zeppelin users, starting with 0.7.0 will give them the wrong
> first impression.
>
>
> On Tue, 14 Mar 2017 at 10:28 AM, Jongyoul Lee  wrote:
>
> Hi dev and users,
>
> As we released 0.7.0, most of users and dev reported a lot of bugs which
> were critical. For the reason, community including me started to prepare
> new minor release with umbrella issue[1]. Due to contributors' efforts, we
> have resolved some of issues and have reviewed almost unresolved issues. I
> want to talk about the new minor release at this point. Generally, we have
> resolved all of issues reported as bugs before we release but some issues
> are very critical and it causes serious problem using Apache Zeppelin. Then
> I think, in this time, it's better to release 0.7.1 as soon as we can and
> prepare a new minor release with rest of unresolved issues.
>
> I'd like to start a process this Friday and if some issues are not merged
> until then, I hope they would be included in 0.7.2.
>
> Feel free to talk to me if you have a better plan to improve users'
> experiences.
>
> Regards,
> Jongyoul Lee
>
> [1] https://issues.apache.org/jira/browse/ZEPPELIN-2134
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>
>

Re: Zeppelin %dev Interpreter

[moon soo Lee]

Have you tried add -Phelium-dev profile in your maven build command?

Thanks,
moon

On Fri, Mar 10, 2017 at 12:11 PM Andreas Weise 
wrote:

> Hi,
>
> various documentations describe the usage of %dev interpreter, e.g. when
> developing Helium Applications.
>
> https://zeppelin.apache.org/docs/0.7.0/development/writingzeppelinapplication.html
>
> I'm wondering how the %dev Interpreter gets added to zeppelin a server.
>
> I already tried starting zeppelin as described here:
>
> https://zeppelin.apache.org/docs/0.7.0/development/howtocontribute.html#run-zeppelin-server-in-development-mode
>
>  But without luck. Same for starting an zeppelin-web instance on
> http://localhost:9000
>
> Any advice.
>
> Thanks in advance.
> Andreas
>

Re: Time series interpreter and WebComponent

[moon soo Lee]

Hi,

Thanks for the interest to the project and sharing the problem.
I took a look code in your repo. And i think there're easier way to do.

Since you're trying to add JS dependency and interpreter prints some HTML
and run JS, you can consider either writing new visualization [1] or
writing new spell [2].
Both you can implement in JS and publish to npm. And then Zeppelin users
able to install them in runtime in 'helium' menu.

[1]
http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/development/writingzeppelinvisualization.html
[2]
http://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/development/writingzeppelinspell.html

Let me know if you have any questions.

Thanks,
moon

On Tue, Mar 7, 2017 at 2:01 AM Aurélien Hébert <
aurelien.heb...@cityzendata.com> wrote:

> Hello Apache Zeppelin team,
>
> For our open-source project we built a web component to visualize time
> series data. As I like to develop some demo on Zeppelin I developed a
> Zeppelin interpreter to communicate with it.
>
> Right now, I have to rebuild the web-app to integrate this component (add
> a line in package.json and some in grunt).
>
> I updated the postinstall task in package.json:
>
> ```
> "postinstall": "bower install --silent && bower install
> cityzendata/warp10-quantumviz#~1.6.2 -F
> --config.directory=./bower_quantumviz --silent && grunt googlefonts"
> ```
>
> and I add a Grunt task to copy my component on the Web Application.
> ```
> {
>   expand: true,
>   cwd: 'bower_quantumviz/',
>   src: '**',
>   dest: '<%= yeoman.dist %>/bower_components'
> },
> ```
>
> Is there a proper way to do it (add the components when the interpreter is
> "settings up"), so there is no need to rebuild the WebApp ?
>
> Are you interested in a Pull-Request to integrate this kind of component
> (visualization tool for geo-time series, to plot a graph or a Map) ?
>
> The Zeppelin interpreter is available on github here :
> https://github.com/cityzendata/warp10-zeppelin-quantumviz
>
> Best regards,
>
> ---
> Aurélien Hébert
> Developer at Cityzen Data
>

Re: Can I call sub-function in Zeppelin?

[moon soo Lee]

Hi Jin,

Zeppelin provides interface that each interpreter implement 'completion()'
method. For example, SparkInterpreter implement 'completion()' [1] and
supposed to return available list of method when user enter ctrl+. after
dot (e.g. sc. and ctrl+.)

If it does not work for you, please feel free to open a jira issue.

Thanks,
moon

[1]
https://github.com/apache/zeppelin/blob/v0.7.0/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java#L1012


On Thu, Feb 23, 2017 at 5:50 PM Beth Lee  wrote:

> Zeppelin supports auto-complete function in a notebook.
> So I enter the ctrl+. in my notebook to use it.
> But I can only show the lists which I typed it before.
>
> Is there any way to call sub-function using the auto-complete function in
> python or spark etc.?
>
> (In case of python: When I enter the tab then I can show the sub-function
> lists)
>
> Thanks,
> Jin
>

Re: How can I use anaconda in zeppelin?

[moon soo Lee]

If you're using '%python', not '%pyspark', you can try %python.conda to
change your environment.

Run

%python.conda help

in the notebook will display available command. This allow dynamically
configure conda environment.

Hope this helps.

Thanks,
moon

On Thu, Feb 23, 2017 at 3:23 PM Beth Lee  wrote:

> I want to use anaconda in zeppelin.
>
> So I modify the configure file in /zeppelin/conf/zeppelin-env.sh like
> below.
>
> export SPARK_HOME=/home/jin/spark
> export PYTHONPATH=/home/jin/anaconda3/bin/python
> export PYSPARK_PYTHON=/home/jin/spark/python
>
> Because I think when I modified the PYTHONPATH then I could use the
> anaconda version.
>
> https://zeppelin.apache.org/docs/0.6.2/interpreter/python.html
> Path of the already installed Python binary (could be python2 or python3).If 
> python is not in your $PATH you can set the absolute directory (example 
> :/usr/bin/python)
>
> But python version doesn't be changed. I try to check the python version
> in zeppelin notebook.
>
> %python
> import sys
> print(sys.version)print(sys.version_info)
>
> 2.7.12 (default, Nov 19 2016, 06:48:10)
>
> How can I use anaconda in zeppelin?
>

Re: Binaries for a setup behind corp firewall and proxy

[moon soo Lee]

Hi Mathieu,

Thanks for reporting the problem.
I see no issue tracking this problem in our JIRA [1].

Do you mind add one?
We're preparing 0.7.1 release and hope we can address this problem in
upcoming release.

Thanks,
moon

[1] https://issues.apache.org/jira/browse/ZEPPELIN




On Tue, Feb 21, 2017 at 6:35 PM Mathieu D  wrote:

> Hello Zeppelin folks,
>
> I'm currently deploying Zeppelin for a customer with corporate firewalls
> and proxies everywhere.
> Zeppelin fails at startup because of npm download.
> It seems that proxy parameter at system level (http_proxy env variable) is
> not used when downloading npm.
>
> Essentially, I would like to have a full standalone binary of zeppelin
> without any additional download at startup.
> Where can I find that ?
>
> If this does not exist, I need the Zeppelin startup to follow the proxy
> setting, at least. Possible ?
>
>
> Mathieu
>
>
>
>

Re: [0.7.0] z.show frequently fails to render tables or charts

[moon soo Lee]

The patch is available in latest branch-0.7 as well.

On Sat, Feb 18, 2017 at 11:51 PM moon soo Lee <m...@apache.org> wrote:

> Hi,
>
> ZEPPELIN-2084 [1] addresses the problem.
> Patch [2] is available and merged to master branch.
>
> Thanks,
> moon
>
> [1] https://issues.apache.org/jira/browse/ZEPPELIN-2084
> [2] https://github.com/apache/zeppelin/pull/2005
>
> On Sat, Feb 18, 2017 at 10:51 PM Xiaohui Liu <hero...@gmail.com> wrote:
>
> Hi,
>
> After deploying 0.7.0, z.show often fails to render tables and charts.
> When that happens, if we refresh the page, the tables and charts will show
> up.
>
> In browser side, we observed this error:
>
> TypeError: Cannot read property 'type' of undefined
> at cells (handsonHelper.js:57)
> at y.Core.getCellMeta (vendor.ee876a1abacbe752.js:325)
> at y.Core._getColWidthFromSettings (vendor.ee876a1abacbe752.js:325)
> at getColWidth (vendor.ee876a1abacbe752.js:325)
> at x.getColumnWidth (vendor.ee876a1abacbe752.js:324)
> at Object.columnWidthFn (vendor.ee876a1abacbe752.js:324)
> at e._getColumnWidth (vendor.ee876a1abacbe752.js:322)
> at e.calculate (vendor.ee876a1abacbe752.js:322)
> at new e (vendor.ee876a1abacbe752.js:322)
> at s.createColumnsCalculator (vendor.ee876a1abacbe752.js:324)
> (anonymous) @ vendor.ee876a1abacbe752.js:37
> handsonHelper.js:57
>
> Is there any way to solve this problem?
>
> Regards
> Xiaohui
>
>

Re: [0.7.0] z.show frequently fails to render tables or charts

[moon soo Lee]

Hi,

ZEPPELIN-2084 [1] addresses the problem.
Patch [2] is available and merged to master branch.

Thanks,
moon

[1] https://issues.apache.org/jira/browse/ZEPPELIN-2084
[2] https://github.com/apache/zeppelin/pull/2005

On Sat, Feb 18, 2017 at 10:51 PM Xiaohui Liu  wrote:

> Hi,
>
> After deploying 0.7.0, z.show often fails to render tables and charts.
> When that happens, if we refresh the page, the tables and charts will show
> up.
>
> In browser side, we observed this error:
>
> TypeError: Cannot read property 'type' of undefined
> at cells (handsonHelper.js:57)
> at y.Core.getCellMeta (vendor.ee876a1abacbe752.js:325)
> at y.Core._getColWidthFromSettings (vendor.ee876a1abacbe752.js:325)
> at getColWidth (vendor.ee876a1abacbe752.js:325)
> at x.getColumnWidth (vendor.ee876a1abacbe752.js:324)
> at Object.columnWidthFn (vendor.ee876a1abacbe752.js:324)
> at e._getColumnWidth (vendor.ee876a1abacbe752.js:322)
> at e.calculate (vendor.ee876a1abacbe752.js:322)
> at new e (vendor.ee876a1abacbe752.js:322)
> at s.createColumnsCalculator (vendor.ee876a1abacbe752.js:324)
> (anonymous) @ vendor.ee876a1abacbe752.js:37
> handsonHelper.js:57
>
> Is there any way to solve this problem?
>
> Regards
> Xiaohui
>