Re: maven build fails with spark 2.0

[Nirav Patel]

Never mind.

Found answer in other thread.

I need to run following before running build.

./dev/change_scala_version.sh 2.11



On Mon, Sep 26, 2016 at 3:21 PM, Nirav Patel  wrote:

> Hi,
>
> I am trying to build zeppeling 0.6 branch with spark 2.0 using following:
>
>  mvn clean package -Pbuild-distr -Pmapr41 -Pyarn -Pspark-2.0 -Pscala-2.11
> -DskipTests
>
>
> I am getting following error:
>
>
> [ERROR] Failed to execute goal on project zeppelin-spark_2.10: Could not
> resolve dependencies for project 
> org.apache.zeppelin:zeppelin-spark_2.10:jar:0.6.2-SNAPSHOT:
> The following artifacts could not be resolved: org.apache.zeppelin:zeppelin-
> display_2.11:jar:0.6.2-SNAPSHOT, org.apache.zeppelin:zeppelin-
> spark-dependencies_2.11:jar:0.6.2-SNAPSHOT: Could not find artifact
> org.apache.zeppelin:zeppelin-display_2.11:jar:0.6.2-SNAPSHOT in nexus
>
>
> It works if I don't include '-Pscala-2.11' command line.
>
> Thanks
>
> Nirav
>

-- 


[image: What's New with Xactly] 

  [image: LinkedIn] 
  [image: Twitter] 
  [image: Facebook] 
  [image: YouTube] 

maven build fails with spark 2.0

[Nirav Patel]

Hi,

I am trying to build zeppeling 0.6 branch with spark 2.0 using following:

 mvn clean package -Pbuild-distr -Pmapr41 -Pyarn -Pspark-2.0 -Pscala-2.11
-DskipTests


I am getting following error:


[ERROR] Failed to execute goal on project zeppelin-spark_2.10: Could not
resolve dependencies for project
org.apache.zeppelin:zeppelin-spark_2.10:jar:0.6.2-SNAPSHOT: The following
artifacts could not be resolved:
org.apache.zeppelin:zeppelin-display_2.11:jar:0.6.2-SNAPSHOT,
org.apache.zeppelin:zeppelin-spark-dependencies_2.11:jar:0.6.2-SNAPSHOT:
Could not find artifact
org.apache.zeppelin:zeppelin-display_2.11:jar:0.6.2-SNAPSHOT in nexus


It works if I don't include '-Pscala-2.11' command line.

Thanks

Nirav

-- 


[image: What's New with Xactly] 

  [image: LinkedIn] 
  [image: Twitter] 
  [image: Facebook] 
  [image: YouTube] 

Re: Using R package with hidden credentials

[Alexander Bezzubov]

Hi Kevin,

approach #2 sounds very reasonable and should work indeed.

Ahyoung, how do you think you could you take a quick look at the cases with
SparkR that Kevin described and see if that is documented\expected outcome?

--
Alex

On Fri, Sep 23, 2016 at 9:17 AM, Kevin Niemann 
wrote:

> I'm using the Spark R interpreter with e.g. the mongolite package which
> requires authentication to access MongoDB.
>
> require(mongolite)
> url <- "mongodb://username:passw...@some.server.com/DB"
> m <- mongo(collection = "events", db = "myDB", url = url, verbose = FALSE)
>
> I'm looking at different approaches to hide the username/password
> credentials from the user:
>
> 1. Hide editor from readers. Only allow Writers to see the code. Downside
> is the editor must remember to hide editor if making changes (potential for
> human error) and no one else can see the actual code to produce the report.
>
> 2. Use a form to require input username/password at runtime. I can't
> actually get .z.input("username") working in %spark.r paragraph. Angular
> forms didn't work for me either, not sure how to assign an input or angular
> binding to an R variable.
>
> 3. Use Data source authorization/credential API. Seems pretty much out of
> the question since Spark isn't passing the creds to the R function.
>
> Anything I missed?
>
> Lastly (somewhat unrelated), returning a data frame should show the
> display system, but it's not. e.g.
>
> %spark.r
> data(mtcars)
> mtcars
> Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
>
>
> Thanks,
>

Zeppelin as a modeling platform

[Nirav Patel]

Hi,

Currently I am using apache zeppelin alongside my eclipse based scala
project. So basically I use my scala project to spit various intermediate
files or file I need for analysis and then use zeppelin to create different
visualization on top of those files. However, many times I find myself to
be able to dig more into models that I am using. For that I think it's
easier to just do modeling in zeppelin as well using spark mllib or any
other imported library. Is this a proper use case for zeppelin?

If it is then I think there are some enhancement should be added to
notebook. e.g. Ability to reuse notebook (treat them as a class or package
) so it can be imported into other notebooks at least. That way we can
define common imports, variables, files, objects (filesystem, connection
pool) etc.

Another thing to consider is how to deploy such notebooks in production.
e.g. how to parameterize zeppelin notebook and call it via REST or
something.

Thanks
Nirav

-- 


[image: What's New with Xactly] 

  [image: LinkedIn] 
  [image: Twitter] 
  [image: Facebook] 
  [image: YouTube] 

Re: zeppelin architecture with multiple users

[Eric Charles]

Spark will typically give you that resource (CPU, memory...) sharing and
reuse.
However, the way you configure your Spark REPL (number of cores, memory
size...) is not my way, just because your data and needs are not mine.

There is still a place to design and develop and additional layer of
abstraction which would allow Zeppelin interpreters to run on shared and
distributed resources (think to a Hadoop Slider application).
In that case, a single interpreter configuration could run on any node of a
distributed cluster (Slider would distributed the job for you).

Still, I believe that it would be beneficial to Zeppelin to allow, not
oblige, each user to configure their own separate interpreter settings.


On Mon, Sep 26, 2016 at 2:52 PM, darren  wrote:

> Just throwing my $0.02 at the prior comment by York. In my opinion users
> should not "have their own processing". In cloud environments everything is
> shared because resources are only utilized on demand. Thus users request
> processing and occupy it while using it then it is released for others. If
> there are enough resources to go around then it looks like everyone has
> their own.
>
>
>
> Sent from my Verizon, Samsung Galaxy smartphone
>
>  Original message 
> From: Eric Charles 
> Date: 9/26/16 8:21 AM (GMT-05:00)
> To: users@zeppelin.apache.org
> Subject: Re: zeppelin architecture with multiple users
>
>
>
> On Fri, Sep 23, 2016 at 1:37 AM, York Huang 
> wrote:
>
>> I think Vincent's option 1 is the way to go at this stage. Basically, in
>> a multi-users environment, every users should have their own storage and
>> processing. So far Zeppelin has not seen to be able to isolate users
>> totally.
>>
>
>
> https://github.com/apache/zeppelin/pull/1390 is an attempt to fully
> isolate notes and services (including interpreters) per user.
>
>
>>
>> My question is,
>> 1. I am using windows for client which means Zeppelin is installed on
>> windows. Is their any downside or limitation?
>> 2. Even installed on individual desktop, I still like to have
>> authentication for login as users may access their desktop remotely from
>> browser. Any idea how to set this up?
>> 3. I am using MapR cluster. Does anyone have similar experiences on how
>> to config secure cluster with zeppelin in windows? Also is it possible to
>> set up an admin user for every desktop so that only I can change the
>> configuration?
>>
>> Thanks,
>>
>>
>> On 22 September 2016 at 18:16, vincent gromakowski <
>> vincent.gromakow...@gmail.com> wrote:
>>
>>> It seems credentials are saved per user as described in Shiro, can you
>>> confirm ? I don't find anything in credential API that links the datasource
>>> user to zeppelin user... Or username on datasource should be the same as
>>> Zeppelin username ?
>>>
>>> Can an admin user set credentials for other users ?
>>>
>>> 2016-09-22 10:12 GMT+02:00 vincent gromakowski <
>>> vincent.gromakow...@gmail.com>:
>>>
 Hi,
 From my point of view you have 3 options:

 1. Use a dedicated zeppelin instance per user. Solution I currently
 use. Mesos/marathon launch an instance by user with it's linux UID. A
 service discovery is routing each user based on HTTPS basic auth to his
 instance. Because the configuration file is dedicated per user, it's easy
 to setup credentials for backend. Because the UID is also setup, spark jobs
 are running under each user permissions. This way is totally secure, but no
 possible sharing between users except sending notebooks by mail or git repo

 2. Use a shared instance and configure Shiro permissions which allow to
 manage multi tenancy in Zeppelin (notebooks access) but not in backend as
 all users notebooks will run under the same UID and have the same
 credentials for accessing backend

 3. Use a shared instance and a backend that allows impersonation like
 Livy server. Livy server will execute Spark sessions per user. What is
 unclear is how to deal with backend credentials ? How to configure multiple
 Cassandra credentials and attach each one to a user ? Same thing for Spark
 Livy, How can we configure each Livy session with users cassandra
 credentials ? And finally how credentials are secured in Zeppelin ?

 2016-09-22 8:59 GMT+02:00 York Huang :

> Hi DuyHai,
>
> I would like to know how to set up security (authentication and
> authorization), the architecture, etc.
>
> The users are using windows. I am ok to set up individual zeppelin on
> their desktop or a central zeppelin server. But I want to know the
> complexity, limitation, details, etc.
>
> Many thanks!
>
> On 16 September 2016 at 03:51, DuyHai Doan 
> wrote:
>
>> Right now, you have some options to isolate the notes. Look at the
>> doc about interpreter binding mode  here :
>> http://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/manual/inte
>> rpreters.html#interpreter-bi

Re: zeppelin architecture with multiple users

[darren]

Just throwing my $0.02 at the prior comment by York. In my opinion users should 
not "have their own processing". In cloud environments everything is shared 
because resources are only utilized on demand. Thus users request processing 
and occupy it while using it then it is released for others. If there are 
enough resources to go around then it looks like everyone has their own.


Sent from my Verizon, Samsung Galaxy smartphone
 Original message From: Eric Charles  Date: 
9/26/16  8:21 AM  (GMT-05:00) To: users@zeppelin.apache.org Subject: Re: 
zeppelin architecture with multiple users 


On Fri, Sep 23, 2016 at 1:37 AM, York Huang  wrote:
I think Vincent's option 1 is the way to go at this stage. Basically, in a 
multi-users environment, every users should have their own storage and 
processing. So far Zeppelin has not seen to be able to isolate users totally.


https://github.com/apache/zeppelin/pull/1390 is an attempt to fully isolate 
notes and services (including interpreters) per user.
 
My question is,
1. I am using windows for client which means Zeppelin is installed on windows. 
Is their any downside or limitation?
2. Even installed on individual desktop, I still like to have authentication 
for login as users may access their desktop remotely from browser. Any idea how 
to set this up?
3. I am using MapR cluster. Does anyone have similar experiences on how to 
config secure cluster with zeppelin in windows? Also is it possible to set up 
an admin user for every desktop so that only I can change the configuration?

Thanks,


On 22 September 2016 at 18:16, vincent gromakowski 
 wrote:
It seems credentials are saved per user as described in Shiro, can you confirm 
? I don't find anything in credential API that links the datasource user to 
zeppelin user... Or username on datasource should be the same as Zeppelin 
username ?
Can an admin user set credentials for other users ?
2016-09-22 10:12 GMT+02:00 vincent gromakowski :
Hi,From my point of view you have 3 options:
1. Use a dedicated zeppelin instance per user. Solution I currently use. 
Mesos/marathon launch an instance by user with it's linux UID. A service 
discovery is routing each user based on HTTPS basic auth to his instance. 
Because the configuration file is dedicated per user, it's easy to setup 
credentials for backend. Because the UID is also setup, spark jobs are running 
under each user permissions. This way is totally secure, but no possible 
sharing between users except sending notebooks by mail or git repo
2. Use a shared instance and configure Shiro permissions which allow to manage 
multi tenancy in Zeppelin (notebooks access) but not in backend as all users 
notebooks will run under the same UID and have the same credentials for 
accessing backend
3. Use a shared instance and a backend that allows impersonation like Livy 
server. Livy server will execute Spark sessions per user. What is unclear is 
how to deal with backend credentials ? How to configure multiple Cassandra 
credentials and attach each one to a user ? Same thing for Spark Livy, How can 
we configure each Livy session with users cassandra credentials ? And finally 
how credentials are secured in Zeppelin ?
2016-09-22 8:59 GMT+02:00 York Huang :
Hi DuyHai,

I would like to know how to set up security (authentication and authorization), 
the architecture, etc.

The
 users are using windows. I am ok to set up individual zeppelin on their
 desktop or a central zeppelin server. But I want to know the 
complexity, limitation, details, etc.

Many thanks!

On 16 September 2016 at 03:51, DuyHai Doan  wrote:
Right now, you have some options to isolate the notes. Look at the doc about 
interpreter binding mode  here : 
http://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/manual/interpreters.html#interpreter-binding-mode


On Thu, Sep 15, 2016 at 7:15 AM, York Huang  wrote:
Hi,

I want to set up a environment for a group of users so that they can access 
zeppelin. Each of them should have their own space, should not interfere each 
other.

I install zeppelin on the MapR sandbox. If I access it from different 
computers, even I access different notebooks, the data are still shared. 

What I want is the data should be totally seperate between users and notebooks.

How do I set it up like this?

Thanks,

York Huang

Re: zeppelin architecture with multiple users

[Eric Charles]

On Fri, Sep 23, 2016 at 1:37 AM, York Huang 
wrote:

> I think Vincent's option 1 is the way to go at this stage. Basically, in a
> multi-users environment, every users should have their own storage and
> processing. So far Zeppelin has not seen to be able to isolate users
> totally.
>


https://github.com/apache/zeppelin/pull/1390 is an attempt to fully isolate
notes and services (including interpreters) per user.


>
> My question is,
> 1. I am using windows for client which means Zeppelin is installed on
> windows. Is their any downside or limitation?
> 2. Even installed on individual desktop, I still like to have
> authentication for login as users may access their desktop remotely from
> browser. Any idea how to set this up?
> 3. I am using MapR cluster. Does anyone have similar experiences on how to
> config secure cluster with zeppelin in windows? Also is it possible to set
> up an admin user for every desktop so that only I can change the
> configuration?
>
> Thanks,
>
>
> On 22 September 2016 at 18:16, vincent gromakowski <
> vincent.gromakow...@gmail.com> wrote:
>
>> It seems credentials are saved per user as described in Shiro, can you
>> confirm ? I don't find anything in credential API that links the datasource
>> user to zeppelin user... Or username on datasource should be the same as
>> Zeppelin username ?
>>
>> Can an admin user set credentials for other users ?
>>
>> 2016-09-22 10:12 GMT+02:00 vincent gromakowski <
>> vincent.gromakow...@gmail.com>:
>>
>>> Hi,
>>> From my point of view you have 3 options:
>>>
>>> 1. Use a dedicated zeppelin instance per user. Solution I currently use.
>>> Mesos/marathon launch an instance by user with it's linux UID. A service
>>> discovery is routing each user based on HTTPS basic auth to his instance.
>>> Because the configuration file is dedicated per user, it's easy to setup
>>> credentials for backend. Because the UID is also setup, spark jobs are
>>> running under each user permissions. This way is totally secure, but no
>>> possible sharing between users except sending notebooks by mail or git repo
>>>
>>> 2. Use a shared instance and configure Shiro permissions which allow to
>>> manage multi tenancy in Zeppelin (notebooks access) but not in backend as
>>> all users notebooks will run under the same UID and have the same
>>> credentials for accessing backend
>>>
>>> 3. Use a shared instance and a backend that allows impersonation like
>>> Livy server. Livy server will execute Spark sessions per user. What is
>>> unclear is how to deal with backend credentials ? How to configure multiple
>>> Cassandra credentials and attach each one to a user ? Same thing for Spark
>>> Livy, How can we configure each Livy session with users cassandra
>>> credentials ? And finally how credentials are secured in Zeppelin ?
>>>
>>> 2016-09-22 8:59 GMT+02:00 York Huang :
>>>
 Hi DuyHai,

 I would like to know how to set up security (authentication and
 authorization), the architecture, etc.

 The users are using windows. I am ok to set up individual zeppelin on
 their desktop or a central zeppelin server. But I want to know the
 complexity, limitation, details, etc.

 Many thanks!

 On 16 September 2016 at 03:51, DuyHai Doan 
 wrote:

> Right now, you have some options to isolate the notes. Look at the doc
> about interpreter binding mode  here : http://zeppelin.apache.org/d
> ocs/0.7.0-SNAPSHOT/manual/interpreters.html#interpreter-binding-mode
>
>
>
> On Thu, Sep 15, 2016 at 7:15 AM, York Huang 
> wrote:
>
>> Hi,
>>
>> I want to set up a environment for a group of users so that they can
>> access zeppelin. Each of them should have their own space, should not
>> interfere each other.
>>
>> I install zeppelin on the MapR sandbox. If I access it from different
>> computers, even I access different notebooks, the data are still shared.
>>
>> What I want is the data should be totally seperate between users and
>> notebooks.
>>
>> How do I set it up like this?
>>
>> Thanks,
>>
>> York Huang
>>
>
>

>>>
>>
>

RE: Z-Manager Zeppelin installation

[Vikash Kumar]

Hi Jesang,
The second option worked for me. I used this way because I 
needed to customize the installation with spark version etc. So now I can 
create a script and share with other so they can install just by running the 
script.
Thanks a Lot : )
Thanks & Regards,
Vikash Kuma
From: Jesang Yoon [mailto:yoon...@gmail.com]
Sent: Monday, September 26, 2016 4:56 PM
To: users@zeppelin.apache.org
Subject: Re: Z-Manager Zeppelin installation

Hi Vikash,

From my experience, there is no pre installation required to run Z-Manager.
What about run your modified script into Vagrant Environment?

https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/install/virtual_machine.html#create-a-zeppelin-ready-vm

According to documentation you can execute script like this:


curl -fsSL 
https://raw.githubusercontent.com/NFLabs/z-manager/master/zeppelin-installer.sh 
| bash

or

cat zeppelin-installer.sh | bash

2016-09-26 19:07 GMT+09:00 Vikash Kumar 
mailto:vikash.ku...@resilinc.com>>:
Hi all,
I am trying to install Zeppelin with Z-Manger by using 
zeppelin-installer.sh script. I downloaded that file and made required changes. 
But when I am running from my machine its treating each word as a command. And 
giving errors. Is there any other installation required to run this script or 
how to run this script.


Thanks & Regards,
Vikash Kumar

Re: Z-Manager Zeppelin installation

[Jesang Yoon]

Hi Vikash,

>From my experience, there is no pre installation required to run Z-Manager.
What about run your modified script into Vagrant Environment?

https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/install/virtual_machine.html#create-a-zeppelin-ready-vm

According to documentation you can execute script like this:

curl -fsSL 
https://raw.githubusercontent.com/NFLabs/z-manager/master/zeppelin-installer.sh
| bash


or

cat zeppelin-installer.sh | bash


2016-09-26 19:07 GMT+09:00 Vikash Kumar :

> Hi all,
>
> I am trying to install Zeppelin with Z-Manger by using
> *zeppelin-installer.sh* script. I downloaded that file and made required
> changes. But when I am running from my machine its treating each word as a
> command. And giving errors. Is there any other installation required to run
> this script or how to run this script.
>
>
>
>
>
> Thanks & Regards,
>
> *Vikash Kumar*
>

Zeppelin vs Other Notebooks(Jupyter etc)

[Vikash Kumar]

Hi all,
I am not able to find latest comparison between Zeppelin and 
other open source notebooks. Like Jupyter, beaker etc. Each post is 
almost one year old.
Does anybody know any link or blog ?

Thanks & Regards,
Vikash Kumar

Z-Manager Zeppelin installation

[Vikash Kumar]

Hi all,
I am trying to install Zeppelin with Z-Manger by using 
zeppelin-installer.sh script. I downloaded that file and made required changes. 
But when I am running from my machine its treating each word as a command. And 
giving errors. Is there any other installation required to run this script or 
how to run this script.


Thanks & Regards,
Vikash Kumar