Re: UNSUBSCRIBE

[babeena.kurian]

UNSUBSCRIBE


From: yupeng 
Sent: Thursday, July 28, 2016 11:57:04 AM
To: users
Subject: UNSUBSCRIBE


** This mail has been sent from an external source **

UNSUBSCRIBE

The information contained in this electronic message and any attachments to 
this message are intended for the exclusive use of the addressee(s) and may 
contain proprietary, confidential or privileged information. If you are not the 
intended recipient, you should not disseminate, distribute or copy this e-mail. 
Please notify the sender immediately and destroy all copies of this message and 
any attachments. WARNING: Computer viruses can be transmitted via email. The 
recipient should check this email and any attachments for the presence of 
viruses. The company accepts no liability for any damage caused by any virus 
transmitted by this email. www.wipro.com

Re: Release time frame of 0.6.1?

[Alexander Bezzubov]

Hi,

thanks for asking! We all looking forward and working hard on the next
release of Apache Zeppelin.

You can track the ongoing progress on 0.6.1 release under
https://issues.apache.org/jira/browse/ZEPPELIN-1177

--
Alex

On Thu, Jul 28, 2016, 12:19 Pedro Rodriguez  wrote:

> On similar regards, when is spark 2.0.0 expected to be supported? I know
> it was just released, so this is more to get an idea of when to expect it.
>
> On Wed, Jul 27, 2016 at 8:34 PM, Haopu Wang  wrote:
>
>> Hi, do you know when will Zeppelin 0.6.1 be released? Thanks!
>>
>>
>>
>> Regards,
>>
>> Haopu
>>
>>
>>
>
>
>
> --
> Pedro Rodriguez
> PhD Student in Distributed Machine Learning | CU Boulder
> UC Berkeley AMPLab Alumni
>
> ski.rodrig...@gmail.com | pedrorodriguez.io | 909-353-4423
> Github: github.com/EntilZha | LinkedIn:
> https://www.linkedin.com/in/pedrorodriguezscience
>
>

Output of lm summary

[Abul Basar]

Hello All,

I am trying to run the following R commands using spark R.

*%r #Using SparkR interpreter*

*require(ggplot2)*

*fit = lm(price ~ carat, data = diamonds)*

*summary(fit)*


Expected Output (as displayed by R Studio);

Call:
lm(formula = price ~ carat, data = diamonds)

Residuals:
 Min   1Q   Median   3Q  Max
-18585.3   -804.8-18.9537.4  12731.7

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -2256.36  13.06  -172.8   <2e-16 ***
carat7756.43  14.07   551.4   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 1549 on 53938 degrees of freedom
Multiple R-squared:  0.8493, Adjusted R-squared:  0.8493
F-statistic: 3.041e+05 on 1 and 53938 DF,  p-value: < 2.2e-16


But, below is what I am getting - only first 2 lines of the above output is
shown.

Call:

lm(formula = price ~ carat, data = diamonds)

Currently, as a stop gap solution, I write the following to, but the output
is not good.

output.capture(summary(fit))


Does anyone figured a fix for this?


Thanks!

- AB

Re: Drill JDBC Interpreter: Connection Refused

[Jongyoul Lee]

Hello Kevin,

I've got to know what the problem is.

If you are running query with "%drill ...", you can set

drill.url
drill.user
drill.password

If you set default.*, you should use "%jdbc ..."

It's a little bit confused, and it will be improved by 0.7.0.

Hope this help,
Jongyoul

On Thu, Jul 28, 2016 at 7:56 AM, Kevin Verhoeven 
wrote:

> I have the same problem. I am running Zeppelin version 0.6.0 and Drill
> 1.5. When I attempt a Drill query from Zeppelin I receive a "Connection
> refused" error. The query works successfully in Drill. My Interpreter
> configurations:
>
>
>
> default.url: jdbc:drill:zk=my*ServerName*:8121/drill/drillbits1
>
> default.user:
>
> default.password:
>
>
>
> Dependency: org.apache.drill.exec:drill-jdbc:1.5.0
>
>
>
> The query I run is very simple, but the Zeppelin paragraph runs for a few
> seconds and then returns the connection refused error. I pasted the full
> error below.
>
>
>
> Any recommendations?
>
>
>
> Thanks!
>
>
>
> Kevin
>
>
>
> Zeppelin Paragraph:
>
>
>
> %drill
>
> SELECT * FROM asdf LIMIT 100
>
>
>
> Zeppelin Error:
>
>
>
> java.net.ConnectException: Connection refused
>
> at java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>
> at
> java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
> at java.net.Socket.connect(Socket.java:589)
>
> at
> org.apache.thrift.transport.TSocket.open(TSocket.java:182)
>
> at
> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
>
> at
> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
>
> at
> org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
>
> at
> org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
>
> at
> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
>
> at
> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
>
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:96)
>
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.init(RemoteInterpreter.java:216)
>
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:383)
>
> at
> org.apache.zeppelin.interpreter.LazyOpenInterpreter.getFormType(LazyOpenInterpreter.java:105)
>
> at
> org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:263)
>
> at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
>
> at
> org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:329)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
>
> *From:* Jongyoul Lee [mailto:jongy...@gmail.com]
> *Sent:* Tuesday, July 12, 2016 10:53 PM
> *To:* users@zeppelin.apache.org
> *Subject:* Re: Drill JDBC Interpreter: Connection Refused
>
>
>
> Hello,
>
>
>
> It seem not to launch remote Interpreter properly. Which version are you
> using? I've found the error file and couldn't find the version which you
> are using
>
>
>
> JL
>
>
>
> On Wed, Jul 13, 2016 at 2:32 PM, Krishnaprasad A S <
> krishna.pra...@flytxt.com> wrote:
>
> I tried with,
>  %jdbc(drill)
>
> %drill
>
> %drill(jdbc)
>
> Currently I made the drill interpretor as the default one and ran the sql
> without using a %drill,
> select count(*) from `dfs`.`POC/Sample1.csv`
>
>  Please find attached error log.
>
>
>
>
>
> On Wed, Jul 13, 2016 at 5:26 AM, Jongyoul Lee  wrote:
>
> Hello,
>
>
>
> It looks like a kind of initialization error. Could you tell me your
> script starting from %? And interpreter logs which is located under
> logs/ will help solve this error.
>
>
>
> Hope this help,
>
> JL
>
>
>
> On Wednesday, 13 July 2016, Krishnapra

UNSUBSCRIBE

[Higino Filho com]

UNSUBSCRIBE



Pioneer of Brazilian Data Science movement
BR: +55 (0) 11 98133 3342
UK: +44 (0) 20 3372 5608
E: higino.fi...@higinofilho.com 

[The unexamined life is not worth living]

DataScienceBrazil.com 
Data Science Brazil Sao/Paulo Meetup 
 - Founder & Organiser
Brazil-Sao Paulo Apache Flink Meetup 
 - Founder & 
Organiser
HiginoFilho.com 
LinkedIn 
Twitter 

 +
Este email e quaisquer anexos são confidenciais e destinam-se exclusivamente 
para uso do indivíduo ou entidade a quem se dirigem. Se você recebeu este 
e-mail por engano, por favor notifique o remetente imediatamente e, em seguida, 
exclua-o. Se você não for o destinatário pretendido, você não deve manter, 
usar, divulgar, copiar ou distribuir este e-mail sem autorização prévia do 
autor. As opiniões expressas nesta mensagem de e-mail não representam 
necessariamente os pontos de vista Higino Filho, seus negócios diversos, ou 
afiliados. Muito obrigado!
—
This e-mail and any attachments to it are confidential and are intended solely 
for use of the individual or entity to whom they are addressed. If you have 
received this e-mail in error, please notify the sender immediately and then 
delete it. If you are not the intended recipient, you must not keep, use, 
disclose, copy or distribute this e-mail without the author's prior permission. 
The views expressed in this e-mail message do not necessarily represent the 
views of Higino Filho, its diversified business, or affiliates. Thank you!

Re: Querying an interpreter and retrieving the results

[moon soo Lee]

Hi,

You can use ResourcePool to pass result to another interpreter.
0.7.0-SNAPSHOT, automatically put all table type of result into the
resource pool. (interpreter can manually put any object into resource pool)
And another interpreter can access the data.

I have create an example.

https://www.zeppelinhub.com/viewer/notebooks/bm90ZTovL21vb24vTWFjQm9vay82NzcyNjU0NTI3YTY0YTk1YWE4NDhkMGU2Y2QzYTUzMS9ub3RlLmpzb24

Hope this helps.

Thanks,
moon


On Wed, Jul 27, 2016 at 1:26 PM  wrote:

> Dear community,
>
>
> Is there a way / project to extract the results from an interpreter (let's
> say elasticsearch) and use it in another interpreter (lets say spark?)
> Would be nice to query one interpreter, and then create a dataframe or
> hive table.
> If there is no way, would it be feasible that I manually extend either
> interpreter to drop the results, which approach do you recommend?
>
> Thanks,
> Chris Fred.-
>
>
>

Re: Issue with Spark + Zeppelin on Mesos - Failed to create local dir

[Jongyoul Lee]

It looks like error of Spark's side. Does it works normally by spark shell?

On Thu, Jul 28, 2016 at 7:23 AM, Michael Sells  wrote:

> Trying to get Zeppelin running on Mesos and I'm consistently hitting the
> following error when I try to create a dataframe/rdd from a file.
>
> java.io.IOException: Failed to create local dir in
> /tmp/blockmgr-82f31798-dd17-4907-a039-d1c90bf12a80/0e.
> at
> org.apache.spark.storage.DiskBlockManager.getFile(DiskBlockManager.scala:73)
> at org.apache.spark.storage.DiskStore.contains(DiskStore.scala:161)
> at org.apache.spark.storage.BlockManager.org
> $apache$spark$storage$BlockManager$$getCurrentBlockStatus(BlockManager.scala:391)
> at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:817)
> at
> org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:645)
> at org.apache.spark.storage.BlockManager.putSingle(BlockManager.scala:1003)
>
> Running Mesos 0.28, Zeppelin 0.6.0, Spark 1.6.1. This seems to happen
> whenever I try to read data from any source. Errors out just trying to
> create a dataframe or rdd like:
>
> sc.textFile("s3://filepath")
>
> Any pointers on what might be off here? I've tried changing the temp dir
> around and opening permissions. Everything I see indicates it should be
> able to write there. Any help would be appreciated.
>
> Thanks,
> Mike
>



-- 
이종열, Jongyoul Lee, 李宗烈
http://madeng.net

Re: Setting default interpreter at notebook level

[Abul Basar]

Hello Jongyoul,

I could not find the file interpreter-setting.json, but I found a file
conf/interpreter.json. I added a property "default":"true"
for interpreter "org.apache.zeppelin.spark.SparkRInterpreter". I restarted
zeppelin demon service. But, R did not work as default interpreter.

Then I changed conf/zeppelin-site.xml to alter the sequence of
the interpreter, putting org.apache.zeppelin.spark.SparkRInterpreter as
first interpreter. This worked. But this solution is not practical. If want
to work on a R notebook and a scala notebook in parallel, this mechanism
require me to touch conf/zeppelin-site.xml file or keep on using %r tags on
each cell.

Thanks!

-AB

On Mon, Jul 25, 2016 at 3:30 PM, Jongyoul Lee  wrote:

> Hello Abul,
>
> Changing orders within a group dynamically is not supported yet. You can
> change it by making interpreter-setting.json in a resources directory In a
> interpreter-setting.json, you can find a property named `default`. If it's
> true, That will be a default interpreter in a group. If you don't want to
> compile Zeppelin again, copy interpreter-setting.json into
> interpreter/spark/ and open it and change it. It will also have a same
> effect.
>
> Hope this help,
> Jongyoul
>
> On Mon, Jul 25, 2016 at 4:39 PM, Abul Basar  wrote:
>
>> Hi Krishnaprasad,
>>
>> Yes, I have played around with that feature. What I found is "spark,
>> pyspark, r, sql" are grouped together. I use Zeppelin for Spark projects.
>> So I need to set one of these sub-categories as default. Most often I use
>> scala for Spark. But I should be able to create a notebook using r (which
>> essentially is SparkR) as a default. Please let me know if I am missing
>> something.
>>
>> Thanks!
>> - AB
>>
>> On Mon, Jul 25, 2016 at 12:45 PM, Krishnaprasad A S <
>> krishna.pra...@flytxt.com> wrote:
>>
>>> Hi Abul,
>>>  You can change the default interpreter for each notebook through
>>> zeppelin web UI.
>>> Go to the notebook and then settings(up right corner), there you can
>>> find Interpreter binding option. You can reorder the interpreters by drag
>>> and drop. The first one will be default.
>>>
>>> Hope this helps.
>>>
>>> Regards,
>>> Krishnaprasad
>>>
>>> On Mon, Jul 25, 2016 at 12:01 PM, Abul Basar  wrote:
>>>
 I know there is a way to set up a default interpreter at Zepplin using 
 zeppelin.interpreters
 property in conf/zeppelin-site.xml. The setting is global is nature.

 But, is it possible to create a notebook level setting for interpreter?
 For example, in a notebook I want to set the default interpreter at R so
 that for every code block i do not have to start with "%spark.r", while on
 another notebook, I want to set the default interpreter as Scala.

 I am using v0.6

 AB

>>>
>>>
>>>
>>> --
>>> Krishnaprasad A S
>>> Lead Engineer
>>> Flytxt
>>> Skype: krishnaprasadas
>>> M: +91 8907209454 | O: +91 471.3082753 | F: +91 471.2700202
>>> www.flytxt.com | Visit our blog  | Follow us
>>>  | Connect on LinkedIn
>>> 
>>>
>>
>>
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>

Re: Zeppelin graph off the page to the right (or truncated)

[Abul Basar]

Hello Carmella,

By default Zeppelin pulls 1000 records to draw graphs/tables. You can
change the setting in interpreter configuration to fetch more records. Look
for zeppelin.spark.maxResult if you are using Spark for example. One
caveat, more records you pull, slower will be the responsiveness of the
notebook, so lower is better.

Thanks!

-AB

On Thu, Jul 28, 2016 at 3:27 AM, Carmella Smith <
carmella.sm...@tubemogul.com> wrote:

> Hello Users Group,
>
> I have asked Zeppelin to draw a line graph on data from  3/1/2016 to
> 7/1/2016.  I see 3/1/2016 at the bottom far left, and a lovely graph with
> labels left to right across the bottom at 3/6, 3/11, 3/16, 3/21, 3/26,
> 3/31, 4/6 and 4/11, and then we run out of screen.  I expected a slider to
> move to the right to see the rest, but I am not finding any way to slide
> right.  The small triangle at the bottom right seems to be only for up and
> down, not right and left.
>
> Is it truncated and thus it will be impossible to see it?
> Is it off to the right and I need to learn how to get there?
> Is there a way to squeeze the accordion in the left-right axis to scoot
> each data point closer together and thus fit more in the same amount of
> space?
> Do I have too many data points?  This example has 3,240.
> Something else?
>
> Thanks so much for your help,
> Carmella Smith
>
>
>

Re: Zeppelin graph off the page to the right (or truncated)

[Ahyoung Ryu]

Hi Carmella,

It would be better to provide some gif or png image files if possible. Then
other users can understand your situation and help you more quickly :)

Thanks,
Ahyoung


2016년 7월 28일 (목) 오전 6:57, Carmella Smith 님이
작성:

> Hello Users Group,
>
> I have asked Zeppelin to draw a line graph on data from  3/1/2016 to
> 7/1/2016.  I see 3/1/2016 at the bottom far left, and a lovely graph with
> labels left to right across the bottom at 3/6, 3/11, 3/16, 3/21, 3/26,
> 3/31, 4/6 and 4/11, and then we run out of screen.  I expected a slider to
> move to the right to see the rest, but I am not finding any way to slide
> right.  The small triangle at the bottom right seems to be only for up and
> down, not right and left.
>
> Is it truncated and thus it will be impossible to see it?
> Is it off to the right and I need to learn how to get there?
> Is there a way to squeeze the accordion in the left-right axis to scoot
> each data point closer together and thus fit more in the same amount of
> space?
> Do I have too many data points?  This example has 3,240.
> Something else?
>
> Thanks so much for your help,
> Carmella Smith
>
>
>

Re: How to provide anonymous user view permission without login in secured environment

[Hyung Sung Shim]

Hi.
Did you restart zeppelin after change the shiro.ini?

2016년 7월 27일 수요일, Pankaj Singh님이 작성한 메시지:

> Hi All,
>
> I Uncommented /** = authc in shiro.ini.
> Now the requirement is to  provide view access to anonymous user without
> him getting logged-in.
>
> How  this can be done any hints will be helpful
>
> Thanks,
> Pankaj
>

Seeing output without print in python and pyspark

[Hanan Shteingart]

1. Is it possible to see output without print in python and pyspark?
2. Is it possible to plot a data frame without having to sql it?

Re: How to provide anonymous user view permission without login in secured environment

[moon soo Lee]

Hi Pankaj,

Currently user must login to see notebook when authentication is enabled. I
think there is no way of workaround for now.

Thanks,
moon

On Thu, Jul 28, 2016 at 11:04 PM Hyung Sung Shim  wrote:

> Hi.
> Did you restart zeppelin after change the shiro.ini?
>
> 2016년 7월 27일 수요일, Pankaj Singh님이 작성한 메시지:
>
> Hi All,
>>
>> I Uncommented /** = authc in shiro.ini.
>> Now the requirement is to  provide view access to anonymous user without
>> him getting logged-in.
>>
>> How  this can be done any hints will be helpful
>>
>> Thanks,
>> Pankaj
>>
>

Re: Seeing output without print in python and pyspark

[Jeff Zhang]

The first issue is resolved in https://github.com/apache/zeppelin/pull/1232



On Thu, Jul 28, 2016 at 10:04 PM, Hanan Shteingart 
wrote:

> 1. Is it possible to see output without print in python and pyspark?
> 2. Is it possible to plot a data frame without having to sql it?
>



-- 
Best Regards

Jeff Zhang

Pass Credentials through JDBC

[Benjamin Kim]

How do I pass username and password to JDBC connections such as Phoenix and 
Hive that are my own? Can my credentials be passed from Shiro after logging in? 
Or do I have to set them at the Interpreter level without sharing them? I wish 
there was more information on this.

Thanks,
Ben

Re: Pass Credentials through JDBC

[Jongyoul Lee]

Hi,

In my plan, this is a next step after
https://issues.apache.org/jira/browse/ZEPPELIN-1210. But for now, there's
no way to pass your credentials with hiding them. I hope that would be
included in 0.7.0.

Regards,
Jongyoul

On Thu, Jul 28, 2016 at 11:22 PM, Benjamin Kim  wrote:

> How do I pass username and password to JDBC connections such as Phoenix
> and Hive that are my own? Can my credentials be passed from Shiro after
> logging in? Or do I have to set them at the Interpreter level without
> sharing them? I wish there was more information on this.
>
> Thanks,
> Ben




-- 
이종열, Jongyoul Lee, 李宗烈
http://madeng.net

Re: Pass Credentials through JDBC

[Benjamin Kim]

Hi Jonyoul,

How would I enter credentials with the current version of Zeppelin? Do you know 
of a way to make it work now?

Thanks,
Ben

> On Jul 28, 2016, at 8:06 AM, Jongyoul Lee  wrote:
> 
> Hi,
> 
> In my plan, this is a next step after 
> https://issues.apache.org/jira/browse/ZEPPELIN-1210 
> . But for now, there's 
> no way to pass your credentials with hiding them. I hope that would be 
> included in 0.7.0.
> 
> Regards,
> Jongyoul
> 
> On Thu, Jul 28, 2016 at 11:22 PM, Benjamin Kim  > wrote:
> How do I pass username and password to JDBC connections such as Phoenix and 
> Hive that are my own? Can my credentials be passed from Shiro after logging 
> in? Or do I have to set them at the Interpreter level without sharing them? I 
> wish there was more information on this.
> 
> Thanks,
> Ben
> 
> 
> 
> -- 
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net 

Re: Setting default interpreter at notebook level

[Jongyoul Lee]

Hi Abul,

Concerning "defaultInterpreter", it is a feature of current master and
doesn't work at 0.6.0. Sorry for wrong information. And for now, we don't
have any specific plan for supporting different default interpreter with
same interpreter setting. Thus, in your case, %r tags is a proper way for
now. I also don't think it's the best. I hope Zeppelin support this
feature, too.

Regards,
Jongyoul

On Thu, Jul 28, 2016 at 10:54 PM, Abul Basar  wrote:

> Hello Jongyoul,
>
> I could not find the file interpreter-setting.json, but I found a file
> conf/interpreter.json. I added a property "default":"true"
> for interpreter "org.apache.zeppelin.spark.SparkRInterpreter". I restarted
> zeppelin demon service. But, R did not work as default interpreter.
>
> Then I changed conf/zeppelin-site.xml to alter the sequence of
> the interpreter, putting org.apache.zeppelin.spark.SparkRInterpreter as
> first interpreter. This worked. But this solution is not practical. If want
> to work on a R notebook and a scala notebook in parallel, this mechanism
> require me to touch conf/zeppelin-site.xml file or keep on using %r tags
> on each cell.
>
> Thanks!
>
> -AB
>
> On Mon, Jul 25, 2016 at 3:30 PM, Jongyoul Lee  wrote:
>
>> Hello Abul,
>>
>> Changing orders within a group dynamically is not supported yet. You can
>> change it by making interpreter-setting.json in a resources directory In a
>> interpreter-setting.json, you can find a property named `default`. If it's
>> true, That will be a default interpreter in a group. If you don't want to
>> compile Zeppelin again, copy interpreter-setting.json into
>> interpreter/spark/ and open it and change it. It will also have a same
>> effect.
>>
>> Hope this help,
>> Jongyoul
>>
>> On Mon, Jul 25, 2016 at 4:39 PM, Abul Basar  wrote:
>>
>>> Hi Krishnaprasad,
>>>
>>> Yes, I have played around with that feature. What I found is "spark,
>>> pyspark, r, sql" are grouped together. I use Zeppelin for Spark projects.
>>> So I need to set one of these sub-categories as default. Most often I use
>>> scala for Spark. But I should be able to create a notebook using r (which
>>> essentially is SparkR) as a default. Please let me know if I am missing
>>> something.
>>>
>>> Thanks!
>>> - AB
>>>
>>> On Mon, Jul 25, 2016 at 12:45 PM, Krishnaprasad A S <
>>> krishna.pra...@flytxt.com> wrote:
>>>
 Hi Abul,
  You can change the default interpreter for each notebook through
 zeppelin web UI.
 Go to the notebook and then settings(up right corner), there you can
 find Interpreter binding option. You can reorder the interpreters by drag
 and drop. The first one will be default.

 Hope this helps.

 Regards,
 Krishnaprasad

 On Mon, Jul 25, 2016 at 12:01 PM, Abul Basar  wrote:

> I know there is a way to set up a default interpreter at Zepplin using 
> zeppelin.interpreters
> property in conf/zeppelin-site.xml. The setting is global is nature.
>
> But, is it possible to create a notebook level setting for
> interpreter? For example, in a notebook I want to set the default
> interpreter at R so that for every code block i do not have to start with
> "%spark.r", while on another notebook, I want to set the default
> interpreter as Scala.
>
> I am using v0.6
>
> AB
>



 --
 Krishnaprasad A S
 Lead Engineer
 Flytxt
 Skype: krishnaprasadas
 M: +91 8907209454 | O: +91 471.3082753 | F: +91 471.2700202
 www.flytxt.com | Visit our blog  | Follow us
  | Connect on LinkedIn
 

>>>
>>>
>>
>>
>> --
>> 이종열, Jongyoul Lee, 李宗烈
>> http://madeng.net
>>
>
>


-- 
이종열, Jongyoul Lee, 李宗烈
http://madeng.net

Re: Pass Credentials through JDBC

[Jongyoul Lee]

You can find more information on
https://issues.apache.org/jira/browse/ZEPPELIN-1146

Hope this help,
Jongyoul

On Fri, Jul 29, 2016 at 12:08 AM, Benjamin Kim  wrote:

> Hi Jonyoul,
>
> How would I enter credentials with the current version of Zeppelin? Do you
> know of a way to make it work now?
>
> Thanks,
> Ben
>
> On Jul 28, 2016, at 8:06 AM, Jongyoul Lee  wrote:
>
> Hi,
>
> In my plan, this is a next step after
> https://issues.apache.org/jira/browse/ZEPPELIN-1210. But for now, there's
> no way to pass your credentials with hiding them. I hope that would be
> included in 0.7.0.
>
> Regards,
> Jongyoul
>
> On Thu, Jul 28, 2016 at 11:22 PM, Benjamin Kim  wrote:
>
>> How do I pass username and password to JDBC connections such as Phoenix
>> and Hive that are my own? Can my credentials be passed from Shiro after
>> logging in? Or do I have to set them at the Interpreter level without
>> sharing them? I wish there was more information on this.
>>
>> Thanks,
>> Ben
>
>
>
>
> --
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net
>
>
>


-- 
이종열, Jongyoul Lee, 李宗烈
http://madeng.net

Zeppelin JSON files - Table not found

[Joaquin Alzola]

Hi Guys

Newbie question:

import sys.process._

val jfileRDD = sqlContext.jsonFile(s"file:///mnt/test.json")
val facebook = jfileRDD.toDF()
facebook.show()
facebook.printSchema()
facebook.registerTempTable("facebook")

import sys.process._
warning: there were 1 deprecation warning(s); re-run with -deprecation for 
details
jfileRDD: org.apache.spark.sql.DataFrame = [action: string, timestamp: string]
facebook: org.apache.spark.sql.DataFrame = [action: string, timestamp: string]




|action| timestamp|
+--++
|create|2016-01-07T00:01:17Z|
+--++
Root
|-- action: string (nullable = true)
|-- timestamp: string (nullable = true)

%sql
select action, count(1) value
from facebook
where action = create

ERROR:
org.apache.spark.sql.AnalysisException: Table not found: facebook; line 2 pos 5

What is the matter?
This email is confidential and may be subject to privilege. If you are not the 
intended recipient, please do not copy or disclose its content but contact the 
sender immediately upon receipt.

Re: Filter query results by current user

[moon soo Lee]

Per user interpreter is tracked by
https://issues.apache.org/jira/browse/ZEPPELIN-1210

Thanks,
moon

On Mon, Jul 25, 2016 at 8:52 PM Panayotis Trapatsas <
p.trapat...@pamediakopes.gr> wrote:

> Is there a way to open a separate interpreter for each different user that
> logs in the notebook?
>
> On Mon, Jul 25, 2016 at 12:36 PM, Corneau Damien 
> wrote:
>
>> There was a thread opened not that long ago in the mailing list with a
>> similar use case.
>>
>> In short, there is currently no way to have changes made without those
>> changes being saved.
>>
>> On Mon, Jul 25, 2016 at 6:25 PM, Panayotis Trapatsas <
>> p.trapat...@pamediakopes.gr> wrote:
>>
>>> Thanks moon but this way all users need to have write permissions to run
>>> the report and every time a user runs the reports with his name all other
>>> instances are updated with this users data.
>>>
>>> I am trying to find a way for a user to filter his own data without
>>> affecting other users viewing the report at the same time.
>>>
>>> On Sun, Jul 24, 2016 at 11:15 AM, moon soo Lee  wrote:
>>>
 How about something like

 %spark
 val userName = z.getInterpreterContext.getAuthenticationInfo.getUser
 z.show(sqlContext.sql(s"""select * from data where user="${userName}"
 """))

 Thanks,
 moon

 On Sun, Jul 24, 2016 at 3:25 PM Panayotis Trapatsas <
 p.trapat...@pamediakopes.gr> wrote:

> Does the angular way work even if the user has read-only permissions?
> Is there a way to do this without giving the user write permissions to
> notebook?
>
> On Sat, Jul 23, 2016 at 3:02 AM, Corneau Damien 
> wrote:
>
>> Hi,
>>
>> Maybe Dynamic Forms (
>> https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/manual/dynamicform.html)
>> could help in this case?
>> It allows you to add some conditioning in your queries.
>> In this case, the user would be able to select themselves and rerun
>> the query.
>>
>> If you want to do it on a whole Notebook level, then binding values
>> could be another way (
>> https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/displaysystem/back-end-angular.html
>> )
>>
>> On Sat, Jul 23, 2016 at 1:58 AM, Panayotis Trapatsas <
>> p.trapat...@pamediakopes.gr> wrote:
>>
>>> Hi,
>>>
>>> I know I can get current user with
>>> 'z.getInterpreterContext.getAuthenticationInfo.getUser' but is there a 
>>> way
>>> to filter query results by the username of the current user?
>>>
>>> E.g. there are different salesman and each one wants to log in to
>>> Zeppelin and see only his own sales.
>>>
>>> --
>>>
>>> [image: e-Travel SA] 
>>>
>>> Panayotis Trapatsas / Lead Data Engineer
>>> p.trapat...@pamediakopes.gr / +30 6979493921
>>>
>>> e-Travel SA
>>> Office: +30 213 0184000/ Fax: +30 211 8001181
>>> Leof. Kifisias 7, AmpelokipiAthens 115 23, Greece
>>> pamediakopes.gr |  trip.ru |  airtickets24.com |
>>> fantasticgreece.com
>>>
>>> [image: Google +] 
>>> [image:
>>> Stack Overflow] 
>>>
>>
>>
>
>
> --
>
> [image: e-Travel SA] 
>
> Panayotis Trapatsas / Lead Data Engineer
> p.trapat...@pamediakopes.gr / +30 6979493921
>
> e-Travel SA
> Office: +30 213 0184000/ Fax: +30 211 8001181
> Leof. Kifisias 7, AmpelokipiAthens 115 23, Greece
> pamediakopes.gr |  trip.ru |  airtickets24.com |  fantasticgreece.com
>
> [image: Google +] [image:
> Stack Overflow] 
>

>>>
>>>
>>> --
>>>
>>> [image: e-Travel SA] 
>>>
>>> Panayotis Trapatsas / Lead Data Engineer
>>> p.trapat...@pamediakopes.gr / +30 6979493921
>>>
>>> e-Travel SA
>>> Office: +30 213 0184000/ Fax: +30 211 8001181
>>> Leof. Kifisias 7, AmpelokipiAthens 115 23, Greece
>>> pamediakopes.gr |  trip.ru |  airtickets24.com |  fantasticgreece.com
>>>
>>> [image: Google +] [image:
>>> Stack Overflow] 
>>>
>>
>>
>
>
> --
>
> [image: e-Travel SA] 
>
> Panayotis Trapatsas / Lead Data Engineer
> p.trapat...@pamediakopes.gr / +30 6979493921
>
> e-Travel SA
> Office: +30 213 0184000/ Fax: +30 211 8001181
> Leof. Kifisias 7, AmpelokipiAthens 115 23, Greece
> pamediakopes.gr |  trip.ru |  airtickets24.com |  fantasticgreece.com
>
> [image: Google +] [image:
> Stack Overflow] 
>

Re: Zeppelin multi-user dashboards

[moon soo Lee]

an issue has been created
https://issues.apache.org/jira/browse/ZEPPELIN-1236


On Mon, Jul 25, 2016 at 9:07 PM Panayotis Trapatsas <
p.trapat...@pamediakopes.gr> wrote:

> +1. Do we have any updates on this issue?
>
> On Thu, Jun 30, 2016 at 10:23 AM, Skanda 
> wrote:
>
>> Hi All,
>>
>> I have the same requirement as that of Ashish, wherein the same
>> Dashboard will be used by multiple users and each user will look at the
>> charts by applying different filters. Can somebody from the core Zeppelin
>> team provide the right way to have this feature built in Zeppelin?
>>
>> Regards,
>> Skanda
>>
>> On Thu, Jun 23, 2016 at 2:24 PM, Andrea Santurbano 
>> wrote:
>>
>>> As Ashish said:
>>>
>>> *For Zeppelin, this is true that if the view is different from each
>>> user, then it is not the same notebook anymore. But from
>>> a dashboard perspective, multiple users work on the same dashboard, viewing
>>> different data and we still call it the same dashboard.*
>>>
>>> This is a common use case in most of organizations. We need to find a
>>> way in order to make it work.
>>> There is someone on zeppelin core team that can answer or give a hint on
>>> that?
>>>
>>> Il giorno ven 17 giu 2016 alle ore 10:10 Corneau Damien <
>>> cornead...@gmail.com> ha scritto:
>>>
 That's why I was talking about easy way VS more complicated changes.
 The idea behind the easy way was to bypass the server save and send the
 result to the view directly.

 Honestly, I'm not knowledgeable enough to give an estimation of how
 hard caching of previous results, or other methods would be.



 On Fri, Jun 17, 2016 at 5:04 PM, ashish rawat 
 wrote:

> Yes, that looks like a good way to do it. Also, you mentioned an
> important point that user would have to stay on the page for query to
> finish and the results would be flushed when moving to a new page. This is
> typically the case in dashboard that queries have very small response 
> times
> and users wait for the results. But still, as per my understanding,
> dashboards do retain the last viewed data (through browser cache) and also
> allow people to navigate to some other screen while the data is getting
> loaded.
>
> Is there any limitation in Zeppelin, which would not let browser cache
> the view for the current user session? Is it because Zeppelin web always
> asks for a fresh copy of the notebook from server, whenever user switches
> notebooks? Perhaps, keeping the changed state of the notebook in the user
> session at the server side might resolve this?
>
> Regards,
> Ashish
> On Jun 17, 2016 1:09 PM, "Corneau Damien" 
> wrote:
>
> The easy way to do that would be:
> Have an option on the notebook (automatic-save vs manual save), manual
> save could be set as an Admin command only, and once activated any change
> on the notebook wouldn't be propagated to others or saved in the note.json
>
> One drawback however would be that you would be forced to stay on the
> page until the query is finished, and the modification would be flushed
> once you visit a different page.
>
> Otherwise, it would probably need some heavy changes to the Notebook
> Repo and saving mechanism etc...
>
>
>
> On Fri, Jun 17, 2016 at 4:14 PM, ashish rawat 
> wrote:
>
>> Thanks Vincent. I haven't explore git storage much, but it's
>> important for us that the cost the maintaining a changing notebook and a
>> changing user list shouldn't be high. If we have 20 dashboard users and
>> there are minor changes in dashboard functionality every two weeks, then
>> there should not be a need to make changes to all the 20 copies manually.
>> Apart from taking time, this would invariably result in some human error
>> and variations in copies of the same notebook. Moreover maintaining user
>> permissions, also become complex with so many copies.
>>
>> The overall user management and authorization may become non-trivial
>> to manage if we start creating copies for what should be just one 
>> notebook.
>>
>> Regards,
>> Ashish
>>
>> On Fri, Jun 17, 2016 at 11:37 AM, vincent gromakowski <
>> vincent.gromakow...@gmail.com> wrote:
>>
>>> Isnt it acheivable via git repository ? A kind of git clone to get
>>> an indépendant copy of the notebook?
>>> Le 17 juin 2016 4:18 AM, "Corneau Damien"  a
>>> écrit :
>>>
 It is simple Read/Write
 Some people can edit, some people can see.

 By essence, if the view is different from each user, then it is not
 the same notebook anymore.
 Being able to have a different results for each user saved
 somewhere so that they can read it, seems like a big overhead.

 On Fri, Jun 17, 2016 at 6:35 AM, ashish rawat 
 wrote:

>>

RE: Zeppelin JSON files - Table not found

[Joaquin Alzola]

>ERROR:
>org.apache.spark.sql.AnalysisException: Table not found: facebook; line 2 pos 5

Changing the zeppelin.spark.useHiveContext to false solved the issue.

What format does Zeppelin need the JSON to be?
Playing aroung everything on the same line only picks up the last , while in 
different line it give corrupt JSON.
This email is confidential and may be subject to privilege. If you are not the 
intended recipient, please do not copy or disclose its content but contact the 
sender immediately upon receipt.

RE: Drill JDBC Interpreter: Connection Refused

[Kevin Verhoeven]

Thanks Jongyoul,

I tried your suggestion and used %drill with:

drill.driver
drill.url
drill.user
drill.password

However, the query still failed with a “Connection refused” error.

My drill.url is: jdbc:drill:zk=localhost:2181/drill/drillbits1

ZooKeeper and Drill both work, I’ve run many queries through Drill’s UI. Only 
Zeppelin throws an error.

Do you have any other suggestions?

Thanks,

Kevin

The error I found in the log:

ERROR [2016-07-28 09:13:37,639] ({Thread-97} 
RemoteScheduler.java[getStatus]:256) - Can't get status information
org.apache.zeppelin.interpreter.InterpreterException: 
org.apache.thrift.transport.TTransportException: java.net.ConnectException: 
Connection refused
at 
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at 
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at 
org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at 
org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at 
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at 
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at 
org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:96)
at 
org.apache.zeppelin.scheduler.RemoteScheduler$JobStatusPoller.getStatus(RemoteScheduler.java:254)
at 
org.apache.zeppelin.scheduler.RemoteScheduler$JobStatusPoller.run(RemoteScheduler.java:212)
Caused by: org.apache.thrift.transport.TTransportException: 
java.net.ConnectException: Connection refused
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at 
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 8 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 9 more

From: Jongyoul Lee [mailto:jongy...@gmail.com]
Sent: Thursday, July 28, 2016 3:02 AM
To: users@zeppelin.apache.org
Subject: Re: Drill JDBC Interpreter: Connection Refused

Hello Kevin,

I've got to know what the problem is.

If you are running query with "%drill ...", you can set

drill.url
drill.user
drill.password

If you set default.*, you should use "%jdbc ..."

It's a little bit confused, and it will be improved by 0.7.0.

Hope this help,
Jongyoul

On Thu, Jul 28, 2016 at 7:56 AM, Kevin Verhoeven 
mailto:kevin.verhoe...@ds-iq.com>> wrote:
I have the same problem. I am running Zeppelin version 0.6.0 and Drill 1.5. 
When I attempt a Drill query from Zeppelin I receive a "Connection refused" 
error. The query works successfully in Drill. My Interpreter configurations:

default.url: jdbc:drill:zk=myServerName:8121/drill/drillbits1
default.user:
default.password:

Dependency: org.apache.drill.exec:drill-jdbc:1.5.0

The query I run is very simple, but the Zeppelin paragraph runs for a few 
seconds and then returns the connection refused error. I pasted the full error 
below.

Any recommendations?

Thanks!

Kevin

Zeppelin Paragraph:

%drill
SELECT * FROM asdf LIMIT 100

Zeppelin Error:

java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
at 
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
at 
org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at 
org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at 
org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at 
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at 
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)

Re: Pass Credentials through JDBC

[Benjamin Kim]

Thank you. I’ll take a look.


> On Jul 28, 2016, at 8:16 AM, Jongyoul Lee  wrote:
> 
> You can find more information on 
> https://issues.apache.org/jira/browse/ZEPPELIN-1146 
> 
> 
> Hope this help,
> Jongyoul
> 
> On Fri, Jul 29, 2016 at 12:08 AM, Benjamin Kim  > wrote:
> Hi Jonyoul,
> 
> How would I enter credentials with the current version of Zeppelin? Do you 
> know of a way to make it work now?
> 
> Thanks,
> Ben
> 
>> On Jul 28, 2016, at 8:06 AM, Jongyoul Lee > > wrote:
>> 
>> Hi,
>> 
>> In my plan, this is a next step after 
>> https://issues.apache.org/jira/browse/ZEPPELIN-1210 
>> . But for now, there's 
>> no way to pass your credentials with hiding them. I hope that would be 
>> included in 0.7.0.
>> 
>> Regards,
>> Jongyoul
>> 
>> On Thu, Jul 28, 2016 at 11:22 PM, Benjamin Kim > > wrote:
>> How do I pass username and password to JDBC connections such as Phoenix and 
>> Hive that are my own? Can my credentials be passed from Shiro after logging 
>> in? Or do I have to set them at the Interpreter level without sharing them? 
>> I wish there was more information on this.
>> 
>> Thanks,
>> Ben
>> 
>> 
>> 
>> -- 
>> 이종열, Jongyoul Lee, 李宗烈
>> http://madeng.net 
> 
> 
> 
> 
> -- 
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net 

Re: classnotfoundexception using zeppelin with spark authentication

[Jonathan Esterhazy]

yes, this PR fixes my classnotfoundexceptions. awesome!

any idea when it might make it into a release?

On Wed, Jul 27, 2016 at 11:25 PM, Jeff Zhang  wrote:

> I think it is fixed in https://github.com/apache/zeppelin/pull/1237,
> please try that.
>
> On Thu, Jul 28, 2016 at 1:58 AM, Jonathan Esterhazy <
> jonathan.esterh...@gmail.com> wrote:
>
>> I just tried this with using zeppelin master, and the classnotfound error
>> is still there.
>>
>>
>> On Thu, Jul 7, 2016 at 7:07 AM, Hyung Sung Shim 
>> wrote:
>>
>>> Hi. Jonathan.
>>> If it is possible, could you use the zeppelin master on the github?
>>> Zeppelin is working well with spark authentication when i tested.
>>>
>>>
>>>
>>> 2016-07-02 3:05 GMT+09:00 Jonathan Esterhazy <
>>> jonathan.esterh...@gmail.com>:
>>>
 Hyung, thx for your help. I opened these:

 https://issues.apache.org/jira/browse/ZEPPELIN-1096 (this scala
 problem)
 https://issues.apache.org/jira/browse/ZEPPELIN-1097 (similar looking
 python problem)

 LMK if I can provide more info or help in some way.

 On Fri, Jul 1, 2016 at 5:08 AM, Hyung Sung Shim 
 wrote:

> Hi Jonathan.
> Unfortunately I got same error in my test bed.
> Do you mind create an jira issue for this?
>
> 2016-07-01 3:19 GMT+09:00 Jonathan Esterhazy <
> jonathan.esterh...@gmail.com>:
>
>> I added this and still get the same exception. The same property is
>> also set in spark-defaults.conf.
>>
>> After that didn't work, I also tried adding --conf
>> spark.authenticate=true and --conf
>> spark.authenticate.enableSaslEncryption=true, to match the other related
>> settings in spark-defaults.conf. Still get the same 
>> classnotfoundexception.
>>
>>
>> On Thu, Jun 30, 2016 at 10:45 AM, Hyung Sung Shim 
>> wrote:
>>
>>> Please add the *export SPARK_SUBMIT_OPTIONS="--conf
>>> spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and
>>> restart zeppelin and retry your code.
>>>
>>>
>>> 2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy <
>>> jonathan.esterh...@gmail.com>:
>>>
 yes it does. i only see this problem in zeppelin.

 On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim >>> > wrote:

> Hi Jonathan.
> It's not easy to build the test environments but I am working on
> this.
> I have question for you.
> Does your code working well on spark-shell in the
> spark.authenticate mode?
>
> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <
> jonathan.esterh...@gmail.com>:
>
>> Hyung, did you have any luck w/ zeppelin + spark authentication?
>> I'm quite stumped.
>>
>> thx.
>>
>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim <
>> hss...@nflabs.com> wrote:
>>
>>> Thank you.
>>> Let me try.
>>>
>>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
>>> jonathan.esterh...@gmail.com>:
>>>
 Hyung,

 Yes, here they are.

 zeppelin-env.sh:

 export ZEPPELIN_PORT=8890
 export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
 export ZEPPELIN_LOG_DIR=/var/log/zeppelin
 export ZEPPELIN_PID_DIR=/var/run/zeppelin
 export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
 export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
 export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
 export MASTER=yarn-client
 export SPARK_HOME=/usr/lib/spark
 export HADOOP_CONF_DIR=/etc/hadoop/conf
 export
 CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
 :/usr/share/aws/emr/emrfs/auxlib/*"
 export JAVA_HOME=/usr/lib/jvm/java-1.8.0
 export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
 export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
 export
 ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo

 spark-defaults.conf:

 spark.master yarn
 spark.driver.extraClassPath
  
 /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf

 :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
 spark.driver.extraLibraryPath
  /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
 spark.executor.extraClassPath
  
 /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aw

Re: Zeppelin graph off the page to the right (or truncated)

[Carmella Smith]

Hello Abul,

That was it exactly, the record limit.  Thanks to your tip, I can now get
what I need.

Thanks so much,
Carmella

On Thu, Jul 28, 2016 at 7:00 AM, Abul Basar  wrote:

> Hello Carmella,
>
> By default Zeppelin pulls 1000 records to draw graphs/tables. You can
> change the setting in interpreter configuration to fetch more records. Look
> for zeppelin.spark.maxResult if you are using Spark for example. One
> caveat, more records you pull, slower will be the responsiveness of the
> notebook, so lower is better.
>
> Thanks!
>
> -AB
>
> On Thu, Jul 28, 2016 at 3:27 AM, Carmella Smith <
> carmella.sm...@tubemogul.com> wrote:
>
>> Hello Users Group,
>>
>> I have asked Zeppelin to draw a line graph on data from  3/1/2016 to
>> 7/1/2016.  I see 3/1/2016 at the bottom far left, and a lovely graph with
>> labels left to right across the bottom at 3/6, 3/11, 3/16, 3/21, 3/26,
>> 3/31, 4/6 and 4/11, and then we run out of screen.  I expected a slider to
>> move to the right to see the rest, but I am not finding any way to slide
>> right.  The small triangle at the bottom right seems to be only for up and
>> down, not right and left.
>>
>> Is it truncated and thus it will be impossible to see it?
>> Is it off to the right and I need to learn how to get there?
>> Is there a way to squeeze the accordion in the left-right axis to scoot
>> each data point closer together and thus fit more in the same amount of
>> space?
>> Do I have too many data points?  This example has 3,240.
>> Something else?
>>
>> Thanks so much for your help,
>> Carmella Smith
>>
>>
>>
>


-- 
Thanks,
Carmella Smith

Re: classnotfoundexception using zeppelin with spark authentication

[Jeff Zhang]

I think it is safe to merge into 0.6.1


On Fri, Jul 29, 2016 at 1:08 AM, Jonathan Esterhazy <
jonathan.esterh...@gmail.com> wrote:

> yes, this PR fixes my classnotfoundexceptions. awesome!
>
> any idea when it might make it into a release?
>
> On Wed, Jul 27, 2016 at 11:25 PM, Jeff Zhang  wrote:
>
>> I think it is fixed in https://github.com/apache/zeppelin/pull/1237,
>> please try that.
>>
>> On Thu, Jul 28, 2016 at 1:58 AM, Jonathan Esterhazy <
>> jonathan.esterh...@gmail.com> wrote:
>>
>>> I just tried this with using zeppelin master, and the classnotfound
>>> error is still there.
>>>
>>>
>>> On Thu, Jul 7, 2016 at 7:07 AM, Hyung Sung Shim 
>>> wrote:
>>>
 Hi. Jonathan.
 If it is possible, could you use the zeppelin master on the github?
 Zeppelin is working well with spark authentication when i tested.



 2016-07-02 3:05 GMT+09:00 Jonathan Esterhazy <
 jonathan.esterh...@gmail.com>:

> Hyung, thx for your help. I opened these:
>
> https://issues.apache.org/jira/browse/ZEPPELIN-1096 (this scala
> problem)
> https://issues.apache.org/jira/browse/ZEPPELIN-1097 (similar looking
> python problem)
>
> LMK if I can provide more info or help in some way.
>
> On Fri, Jul 1, 2016 at 5:08 AM, Hyung Sung Shim 
> wrote:
>
>> Hi Jonathan.
>> Unfortunately I got same error in my test bed.
>> Do you mind create an jira issue for this?
>>
>> 2016-07-01 3:19 GMT+09:00 Jonathan Esterhazy <
>> jonathan.esterh...@gmail.com>:
>>
>>> I added this and still get the same exception. The same property is
>>> also set in spark-defaults.conf.
>>>
>>> After that didn't work, I also tried adding --conf
>>> spark.authenticate=true and --conf
>>> spark.authenticate.enableSaslEncryption=true, to match the other related
>>> settings in spark-defaults.conf. Still get the same 
>>> classnotfoundexception.
>>>
>>>
>>> On Thu, Jun 30, 2016 at 10:45 AM, Hyung Sung Shim >> > wrote:
>>>
 Please add the *export SPARK_SUBMIT_OPTIONS="--conf
 spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and
 restart zeppelin and retry your code.


 2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy <
 jonathan.esterh...@gmail.com>:

> yes it does. i only see this problem in zeppelin.
>
> On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim <
> hss...@nflabs.com> wrote:
>
>> Hi Jonathan.
>> It's not easy to build the test environments but I am working on
>> this.
>> I have question for you.
>> Does your code working well on spark-shell in the
>> spark.authenticate mode?
>>
>> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <
>> jonathan.esterh...@gmail.com>:
>>
>>> Hyung, did you have any luck w/ zeppelin + spark authentication?
>>> I'm quite stumped.
>>>
>>> thx.
>>>
>>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim <
>>> hss...@nflabs.com> wrote:
>>>
 Thank you.
 Let me try.

 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
 jonathan.esterh...@gmail.com>:

> Hyung,
>
> Yes, here they are.
>
> zeppelin-env.sh:
>
> export ZEPPELIN_PORT=8890
> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
> export ZEPPELIN_LOG_DIR=/var/log/zeppelin
> export ZEPPELIN_PID_DIR=/var/run/zeppelin
> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
> export MASTER=yarn-client
> export SPARK_HOME=/usr/lib/spark
> export HADOOP_CONF_DIR=/etc/hadoop/conf
> export
> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
> :/usr/share/aws/emr/emrfs/auxlib/*"
> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
> export
> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo
>
> spark-defaults.conf:
>
> spark.master yarn
> spark.driver.extraClassPath
>  
> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>
> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>

Re: Drill JDBC Interpreter: Connection Refused

[Alexander Bezzubov]

Hi Kevin,

this is very strange indeed.

Could you please try to delete ./logs/* restart Zeppelin and try again, and
then post 2 full log files, of both zeppelin server and drill interpreter
I.e on gist or pastebin and share a link here?

This should help to debug the situation.

--
Alex


On Fri, Jul 29, 2016, 01:17 Kevin Verhoeven 
wrote:

> Thanks Jongyoul,
>
>
>
> I tried your suggestion and used %drill with:
>
>
>
> drill.driver
>
> drill.url
>
> drill.user
>
> drill.password
>
>
>
> However, the query still failed with a “Connection refused” error.
>
>
>
> My drill.url is: jdbc:drill:zk=localhost:2181/drill/drillbits1
>
>
>
> ZooKeeper and Drill both work, I’ve run many queries through Drill’s UI.
> Only Zeppelin throws an error.
>
>
>
> Do you have any other suggestions?
>
>
>
> Thanks,
>
>
>
> Kevin
>
>
>
> The error I found in the log:
>
>
>
> ERROR [2016-07-28 09:13:37,639] ({Thread-97}
> RemoteScheduler.java[getStatus]:256) - Can't get status information
>
> org.apache.zeppelin.interpreter.InterpreterException:
> org.apache.thrift.transport.TTransportException: java.net.ConnectException:
> Connection refused
>
> at
> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
>
> at
> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
>
> at
> org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
>
> at
> org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
>
> at
> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
>
> at
> org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
>
> at
> org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:96)
>
> at
> org.apache.zeppelin.scheduler.RemoteScheduler$JobStatusPoller.getStatus(RemoteScheduler.java:254)
>
> at
> org.apache.zeppelin.scheduler.RemoteScheduler$JobStatusPoller.run(RemoteScheduler.java:212)
>
> Caused by: org.apache.thrift.transport.TTransportException:
> java.net.ConnectException: Connection refused
>
> at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
>
> at
> org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
>
> ... 8 more
>
> Caused by: java.net.ConnectException: Connection refused
>
> at java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
> at java.net.Socket.connect(Socket.java:589)
>
> at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
>
> ... 9 more
>
>
>
> *From:* Jongyoul Lee [mailto:jongy...@gmail.com]
> *Sent:* Thursday, July 28, 2016 3:02 AM
>
>
> *To:* users@zeppelin.apache.org
> *Subject:* Re: Drill JDBC Interpreter: Connection Refused
>
>
>
> Hello Kevin,
>
>
>
> I've got to know what the problem is.
>
>
>
> If you are running query with "%drill ...", you can set
>
>
>
> drill.url
>
> drill.user
>
> drill.password
>
>
>
> If you set default.*, you should use "%jdbc ..."
>
>
>
> It's a little bit confused, and it will be improved by 0.7.0.
>
>
>
> Hope this help,
>
> Jongyoul
>
>
>
> On Thu, Jul 28, 2016 at 7:56 AM, Kevin Verhoeven <
> kevin.verhoe...@ds-iq.com> wrote:
>
> I have the same problem. I am running Zeppelin version 0.6.0 and Drill
> 1.5. When I attempt a Drill query from Zeppelin I receive a "Connection
> refused" error. The query works successfully in Drill. My Interpreter
> configurations:
>
>
>
> default.url: jdbc:drill:zk=my*ServerName*:8121/drill/drillbits1
>
> default.user:
>
> default.password:
>
>
>
> Dependency: org.apache.drill.exec:drill-jdbc:1.5.0
>
>
>
> The query I run is very simple, but the Zeppelin paragraph runs for a few
> seconds and then returns the connection refused error. I pasted the full
> error below.
>
>
>
> Any recommendations?
>
>
>
> Thanks!
>
>
>
> Kevin
>
>
>
> Zeppelin Paragraph:
>
>
>
> %drill
>
> SELECT * FROM asdf LIMIT 100
>
>
>
> Zeppelin Error:
>
>
>
> java.net.ConnectException: Connection refused
>
> at java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>
> at
> java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
> at java.net.

Re: Zeppelin JSON files - Table not found

[Alexander Bezzubov]

Hi Joaquin,

Zeppelin itself does not have any restrictions on the format of the input
data, everything depends on the data processing system and interpreter you
choose to use. looking at you example - as you use Apache Spark [1] would
be the place to look for details.

Hope this helps!

1.
http://spark.apache.org/docs/1.6.1/sql-programming-guide.html#json-datasets

--
Alex

On Fri, Jul 29, 2016, 01:00 Joaquin Alzola 
wrote:

>
>
> >ERROR:
> >org.apache.spark.sql.AnalysisException: Table not found: facebook; line
> 2 pos 5
>
>
>
> Changing the zeppelin.spark.useHiveContext to false solved the issue.
>
>
>
> What format does Zeppelin need the JSON to be?
>
> Playing aroung everything on the same line only picks up the last , while
> in different line it give corrupt JSON.
> This email is confidential and may be subject to privilege. If you are not
> the intended recipient, please do not copy or disclose its content but
> contact the sender immediately upon receipt.
>

Re: Zeppelin JSON files - Table not found

[Jeff Zhang]

Is this code snippet your completed code in notebook ? If you create
SQLContext by yourself instead of using the SQLContext created by spark
interpreter, you will hit this issue.


On Fri, Jul 29, 2016 at 9:14 AM, Alexander Bezzubov  wrote:

> Hi Joaquin,
>
> Zeppelin itself does not have any restrictions on the format of the input
> data, everything depends on the data processing system and interpreter you
> choose to use. looking at you example - as you use Apache Spark [1] would
> be the place to look for details.
>
> Hope this helps!
>
> 1.
> http://spark.apache.org/docs/1.6.1/sql-programming-guide.html#json-datasets
>
> --
> Alex
>
> On Fri, Jul 29, 2016, 01:00 Joaquin Alzola 
> wrote:
>
>>
>>
>> >ERROR:
>> >org.apache.spark.sql.AnalysisException: Table not found: facebook; line
>> 2 pos 5
>>
>>
>>
>> Changing the zeppelin.spark.useHiveContext to false solved the issue.
>>
>>
>>
>> What format does Zeppelin need the JSON to be?
>>
>> Playing aroung everything on the same line only picks up the last , while
>> in different line it give corrupt JSON.
>> This email is confidential and may be subject to privilege. If you are
>> not the intended recipient, please do not copy or disclose its content but
>> contact the sender immediately upon receipt.
>>
>


-- 
Best Regards

Jeff Zhang