Re: Query about the high availability of Zeppelin

2017-06-29 Thread Vinay Shukla
Here is what I think should be part of HA consideration: 1. Have multiple Zeppelin Instances 2. Have the notebooks storage backed by something like an NFS so all notebooks are visible across all Zeppelin instances 3. Put multiple load balancers infront of Zeppelin to route requests. C

Re: [ANNOUNCE] Apache Zeppelin 0.7.2 released

2017-06-14 Thread Vinay Shukla
Cangatulations Apache Zeppelin community. I am sure our users will appreciate your efforts and this release. Let's make Zeppelin great. On Tue, Jun 13, 2017 at 10:08 AM Jun Kim wrote: > Cool! Thanks for your work Mina :) > > 2017년 6월 14일 (수) 오전 2:06, Mina Lee 님이 작성: > >> The Apache Zeppelin c

Re: Zeppelin Notebook Storage in Google Cloud Storage

2017-04-18 Thread Vinay Shukla
-java > 3. https://github.com/apache/zeppelin/tree/master/bigquery > -- > Alex > > On Tue, Apr 18, 2017 at 3:49 PM, Vinay Shukla > wrote: > >> Does Zeppelin have functionality to store Notebooks in GCS via VFS? >> >> http://zeppelin.apache.org/docs/0.7.1/storage/storage.html Does not >> mention GCS? >> >> Thanks >> > >

Zeppelin Notebook Storage in Google Cloud Storage

2017-04-18 Thread Vinay Shukla
Does Zeppelin have functionality to store Notebooks in GCS via VFS? http://zeppelin.apache.org/docs/0.7.1/storage/storage.html Does not mention GCS? Thanks

Re: Zeppelin should support standard protocols for authN and AuthZ

2017-03-27 Thread Vinay Shukla
I agree, it will be useful to have Zeppelin ATN & ATZ more pluggable. For ATN, if Zeppelin accepted identities vouched for in an upstream system, it can support most ATN protocols. In Hadoop ecosystem, Apache Knox is used for Authentication. Knox already support SAML based identity assertion and

Re: [DISCUSS] Admin feature

2017-03-09 Thread Vinay Shukla
Thanks for starting this thread and I wish I saw this earlier. I filed https://issues.apache.org/jira/browse/ZEPPELIN-2236 recently and much of that feature request is applicable in this blog. >From this JIRA here are some thoughts on what are different areas of Zeppelin where we can put in acces

Re: [ANNOUNCE] Apache Zeppelin 0.7.0 released

2017-02-06 Thread Vinay Shukla
Congratulations Zeppelin users, contributors and committers. This is a great milestone. Before the dust settles on 0.7.0, let's put together our heads together to define features for the next release. On Mon, Feb 6, 2017 at 7:00 AM, Hyung Sung Shim wrote: > Congratulations! > and Thank you a

Re: Monitoring Zeppelin Health Via Rest API

2017-01-25 Thread Vinay Shukla
How about launching Zeppelin JVM with JConsole related config and trying to monitor JVM level stats? On Wed, Jan 25, 2017 at 9:12 AM, Rob Anderson wrote: > Ok, thanks for the reply Jongyoul. > > On Wed, Jan 25, 2017 at 12:13 AM, Jongyoul Lee wrote: > >> AFAIK, Zeppelin doesn't have it for now.

Re: [VOTE] Release Apache Zeppelin 0.7.0 (RC2)

2017-01-22 Thread Vinay Shukla
Downloaded https://dist.apache.org/repos/dist/dev/zeppelin/zeppelin-0.7.0-rc2/ Then ran Basic Feature tutorial successfully and went to *report* mode and it does not re-run the paragraph when the marital or age field value is changed (even after shift + enter) INFO [2017-01-22 13:36:21,924] ({q

Re: Where can I find out if connection passwords are salted?

2016-12-12 Thread Vinay Shukla
Do you mean encrypt password for JDBC/other such connections? On Thu, Dec 8, 2016 at 1:18 AM, Chris M. Balz wrote: > Where can I find out about the security of saved passwords for > connections, such as for jdbc? For example, where can I find out if > connection passwords are salted? I have se

Re: Integrating SSO to Zeepelin Shiro

2016-12-12 Thread Vinay Shukla
Hi Niraj, AFAIK, right now Zeppelin does not support SSO integration. There is a possiblity to integrate Zeppelin with Apache Knox to take advantage of SSO features and integrate any SAML based SSO provider. If you are willing to right custom code, I support SSO integration can be done with curre

Re: Is it possible to add widget on zeppelin UI

2016-08-31 Thread Vinay Shukla
Canan The idea behind Helium is precisely what you mention. Think of Helium as a way to extend Zeppelin UI. Zeppelin had 2 existing extensibility points: interpreters and notebook storage. Helium is adding the 3rd extensibility point, to Zeppelin the UI. See the demo at https://cwiki.apache.org/c

Re: Why the deprecation of %dep?

2016-08-27 Thread Vinay Shukla
Ahmed, Do you find that %dep not work for you post deprecation? Thanks, Vinay On Saturday, August 27, 2016, Ahmed Sobhi wrote: > I've been using the %dep extensively to setup my notebook dependencies for > later use by the Scala Spark paragraphs. > > My main use case for these notebooks involv

Re: is there way to update(push latest data to) Zeppelin dashboard ?

2016-08-24 Thread Vinay Shukla
erhaps some built in visualization widget that can poll a backend for >>> updates?" >>> >>> Surely, but then someone need to code it. And a pre-built solution is >>> never really as flexible as a customized dev. I don't see how we can create >>> a g

Re: is there way to update(push latest data to) Zeppelin dashboard ?

2016-08-24 Thread Vinay Shukla
t >> the updates to be pushed to the dashboard I dont want keep running the >> query over and over again or do some sort of polling. >> >> Thanks! >> >> >> >> On Wed, Aug 24, 2016 3:36 AM, Vinay Shukla vinayshu...@gmail.com >> wrote: >> >

Re: is there way to update(push latest data to) Zeppelin dashboard ?

2016-08-24 Thread Vinay Shukla
Hey Kant, For fast data updates via streaming the normal flow is to put the aggregated results in things like HBase. You could then use Zeppelin's Phoenix interpreter to read that data. Not sure if this fits your requirements. Thanks, Vinay On Wednesday, August 24, 2016, kant kodali wrote: > H

Re: CSV spark package not working in v0.6.1, spark 2.0, scala 2.11

2016-08-17 Thread Vinay Shukla
Abul, Mina is right, until Spark 1.6 csv parsing was bailable as a separate spark package. With Spark 2.0 csv parsing is built in. Zeppelin 0.6.1 ships with Spark 2.0. Thanks, Vinay On Wednesday, August 17, 2016, Mina Lee wrote: > Hi Abul, > > spark-csv is integrated into spark itself so you d

Re: Zeppelin access to remote Spark&Yarn cluster

2016-08-12 Thread Vinay Shukla
Michael, Besides what Moon suggested, the next thing to consider is the security. If the remote cluster in the Data center has Kerberos authentication enabled, then you will need to configure Zeppelin + Livy interpreter for Kerberos and identity propagation to allow the job to run as the end-user

Re: Scratching my head around Zeppelin/Spark & Docker

2016-08-05 Thread Vinay Shukla
Just curious, what is the use case to run Z inside docker? On Friday, August 5, 2016, DuyHai Doan wrote: > No, using Docker compose is easy, what I want is: > > 1) Zeppelin running inside a Docker container > 2) Spark deployed in stand-alone mode, running somewhere on bare-metal / > cloud / Dock

Re: question on livy session in zeppelin

2016-08-04 Thread Vinay Shukla
Hi Chen, I agree this is not very user friendly & there should be no need to restart LivyInterpreter. One short term fix may be to increase the livy session timeout by changing livy.server.session.timeout (takes time in millisecond) Filed https://issues.apache.org/jira/browse/ZEPPELIN-1293 to tra

Re: Can I run different versions of spark interpreter in one zeppelin build ?

2016-08-03 Thread Vinay Shukla
There are a lot of new APIs with Spark 2 and more importantly the new entry point is SparkSession (available as spark variable). I think Z built against Spark 2 should leverage spark variable by default instead of using SC. Please ignore this if the current Spark2 support already includes this fe

Re: Securing ldap password in shiro.ini

2016-08-03 Thread Vinay Shukla
Moon, Polina, Unfortunately the Shiro password hasher won't help. Polina's use case is encrypting password of AD/LDAP that Zeppelin uses to connect to an AD/LDAP. The shiro password hasher encrypts the password that zeppelin stores when AD/LDAP is not used and user accounts are kept in shiro its

Re: spark.jars option for Zeppelin over Livy

2016-08-02 Thread Vinay Shukla
In recent Zeppelin builds with Livy interpreter you can set spark-packages (e.g Databricks csv) as value of property "livy.spark.jars.packages" on Livy interpreter properties. Please see the attached screenshot On Tue, Aug 2, 2016 at 1:43 PM, Felix Cheung wrote: > Have you tried setting it in

Re: UDF does not runs from Zeppelin

2016-07-17 Thread Vinay Shukla
Gilad, I have a simple UDF to convert date running successfully with Zeppelin 0.6.0 & Spark 1.6.1 ## UDF import org.apache.spark.sql.functions._ import org.apache.spark.sql.types._ import java.text.SimpleDateFormat val toDate = udf((t: String) =>{ val sdf = new SimpleDateFormat("MM/dd/y

Re: Forbid creating new notes

2016-07-15 Thread Vinay Shukla
Krzysztof, Today access control is configured at the Note level & not done a-priori in the shiro.ini. You configure authentication in shiro.ini & the notebook creator can set access control by clicking the lock icon on the notebook. Thanks, Vinay On Friday, July 15, 2016, Krzysztof Zarzycki w

Re: Clear results from Zeppelin notebook json

2016-07-11 Thread Vinay Shukla
I tried reproducing it with Zeppelin 0.6.0 and could not. Can you please retry with 0.6.0? Thanks, Vinay On Mon, Jul 11, 2016 at 2:46 PM, Ahmed Sobhi wrote: > Hello, > > I have a use case where I need to export Zeppelin notebooks without the > results being cached in them. > > I tried clearing

Re: Hive interpreter connecting to secure cluster

2016-07-07 Thread Vinay Shukla
We are working on it, expect to fix it in next couple of weeks. Thanks, Vinay On Thursday, July 7, 2016, Rob Anderson wrote: > Does anyone know if/when the Hive interpreter will be able to connect to a > secure, kerberized cluster? > > Thanks, > > Rob >

Re: Phoenix plugin with a secure cluster.

2016-07-06 Thread Vinay Shukla
Don't think today Phoenix interpreter supports Kerberos. Are you using JDBC interpreter by any chance? (Which also doesn't yet support Kerberos, AFAIK). We will try to resolve this soon. Thanks, Vinay On Wed, Jul 6, 2016 at 10:37 AM, Rob Anderson wrote: > Does anyone know if the Phoenix plugi

Re: Build error zengine

2016-06-21 Thread Vinay Shukla
What is the exact build failure? On Tuesday, June 21, 2016, Darren Govoni wrote: > Ubuntu 15.10 > > mvn clean package -Pspark-1.6 -Phadoop-2.4 -Pyarn -Ppyspark > > > On 06/21/2016 09:03 AM, Hyung Sung Shim wrote: > > hi. > What is your build command and please tell me your environments. > > 2016

Re: how to tell Zeppelin version?

2016-06-13 Thread Vinay Shukla
In the menu on the top right an option should show the version. This was implemented in last few weeks. On Monday, June 13, 2016, Leon Katsnelson wrote: > Is there a command or a menu that will tell me what version of Zeppelin I > am running? >

Re: Zeppelin authentication / permissions while using both Shiro and AD

2016-06-02 Thread Vinay Shukla
Thanks Rob. On Thursday, June 2, 2016, Rob Anderson wrote: > Done, thanks. > > https://issues.apache.org/jira/browse/ZEPPELIN-946 > > On Wed, Jun 1, 2016 at 1:06 PM, Vinay Shukla > wrote: > >> Rob, >> >> It appears to be bug, can you please file a JIR

Re: Zeppelin authentication / permissions while using both Shiro and AD

2016-06-01 Thread Vinay Shukla
Rob, It appears to be bug, can you please file a JIRA to track this? Thanks, Vinay On Fri, May 27, 2016 at 7:52 AM, Rob Anderson wrote: > Hey Everyone, > > I'm new to Zeppelin as of this week. I've managed to build and stand up > the *0.6.0-incubating-SNAPSHOT. *I've configured Zeppelin to

Re: the date of the release with livy intercepter

2016-05-26 Thread Vinay Shukla
If you are on HDP, you can try out Zeppelin with Livy today. See https://community.hortonworks.com/articles/34424/apache-zeppelin-on-hdp-242.html In the community we have started a discussion around next release of Zeppelin so some decision about timelines should be coming up in next weeks. On Th

Re: The Apache Software Foundation Announces Apache(R) Zeppelin™ as a Top-Level Project

2016-05-25 Thread Vinay Shukla
Congratulations team. This is all because of your hard work and focus on users. Let's make this community bigger and more successful. Thanks, Vinay On Wed, May 25, 2016 at 6:56 AM, moon soo Lee wrote: > Congratulations and thank you everyone who made it possible! > > -- Forwarded messa

Re: TLS support

2016-05-25 Thread Vinay Shukla
Peter, Looks to me Zeppelin today does leverage underlying Jetty's configuration to set 1. SSL/TLS protocol versions 2. Include & Exclude Ciphers The fix is pretty simple, we will send a pull request soon. Thanks, Vinay On Wed, May 25, 2016 at 8:08 AM, Peter Schwarz wrote: > Thanks