Re: Obsolete Zeppelin staging repositories

[Luciano Resende]

Thank you, much better to navigate there now.

On Thu, May 31, 2018 at 10:06 PM, Jeff Zhang  wrote:

> Thanks Luciano, I have dropped them.
>
>
>
> Luciano Resende 于2018年6月1日周五 下午12:53写道：
>
> > Looks like in preparation for 0.8 several Zeppelin staging repositories
> > were left behind either open or closed while they should have been
> dropped
> > when the RC failed. Could someone please take a look at that at
> > repository.apache.org and drop the obsolete ones.
> >
> >
> > Thanks
> >
> > --
> > Luciano Resende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
> >
>



-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Obsolete Zeppelin staging repositories

[Luciano Resende]

Looks like in preparation for 0.8 several Zeppelin staging repositories
were left behind either open or closed while they should have been dropped
when the RC failed. Could someone please take a look at that at
repository.apache.org and drop the obsolete ones.


Thanks

-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Windows support ?

[Luciano Resende]

I see a few sporadic issues/questions related to windows, which seems to
get lower priority and I was trying the building Zeppelin in windows
environment and there are some clear issues.

- What is the official word around Zeppelin supporting Windows?

- Is Windows a supported development environment for Zeppelin (e.g.
building)?


-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: Exporting Spark paragraphs as Spark Applications

[Luciano Resende]

Hi Jeff,

While I agree with you that what you mentioned is completely acceptable for
some users, particularly regarding the data science personas. Having said
that, while talking with multiple enterprise companies, that have their own
scheduler infrastructure with different quality of service or just want to
deploy this as an app into their production environment which will have
much more resources for running these apps with complete data sets, and
currently they finish the experiment/development of the application in an
interactive environment and them move their final code into a native spark
application.

Zeppelin is evolving quickly in this area, and I think that export as an
application might be a good option for users that want to actually deploy
their notebooks as native applications into their own Spark cluster.

Having said that, if the community feels that this is not a required
function in Zeppelin anymore, then I can continue with the development of
the tool as a standalone command line tool. I was even thinking about
expanding the functionality and implementing what is described in
ZEPPELIN-1793.

Thoughts ?

On Thu, Jan 5, 2017 at 12:38 AM, Jeff Zhang <zjf...@gmail.com> wrote:

> Thanks Luciano. IIRC, what user want is to run the whole spark app, but
> they don't care about whether it is in zeppelin or through a standard spark
> app jar. I know zeppelin currently doesn't do well in converting note to
> production spark app as Lee mentioned. But exporting note as jar seems a
> short term solution, not a long term solution. I just feel when zeppelin
> improve in this field, user might abandon this solution and transit to
> zeppelin again. Here's some disadvantages I can see of this approach.
>
> 1.  If user want to change the code in iterative development , they have to
> repeat the whole pipeline (change code in zeppelin -> export it to spark
> jar -> redeploy this jar). This process is painful and wasting time.
> 2.  Hard to debug and diagnose as code is changed/restructured when
> exporting to jar
> 3.  User have to leverage several distinct tools for the whole development
> cycle (zeppelin, spark job server, and maybe cron job)
>
> Besides,  the OOM issue of Spark REPL Lee mentioned might not be a problem.
> Because we can shutdown the app (close interpreter) after the app is done.
>
>
>
>
>
> Luciano Resende <luckbr1...@gmail.com>于2017年1月5日周四 下午3:59写道：
>
> Some use cases discussed earlier on this thread:
>
> https://www.mail-archive.com/dev@zeppelin.apache.org/msg06323.html
>
> https://www.mail-archive.com/dev@zeppelin.apache.org/msg06332.html
>
> On Wed, Jan 4, 2017 at 4:51 PM, Jianfeng (Jeff) Zhang <
> jzh...@hortonworks.com> wrote:
>
> >
> > I don¹t understand why user want to export zeppelin note as spark
> > application.
> >
> > If they want to trigger the running of spark app, why not use zeppelin¹s
> > rest api for that. Even user export it as spark application, most of time
> > in reality, they need to submit it through spark job server, so why not
> > use zeppelin as a spark job server.
> > And if the spark app fails, it is pretty hard to debug it, because the
> > exporting tool has changed/restructured the source code.
> >
> >
> > If this is a pretty large and complicated spark application, I don¹t
> think
> > zeppelin is a proper tool for that, they¹d better to use IDE for that
> > project.
> >
> > BTW, After https://github.com/apache/zeppelin/pull/1799, user can define
> > the dependency between paragraphs, and they can run one whole note which
> > contains different interpreters.
> >
> >
> >
> > Best Regard,
> > Jeff Zhang
> >
> >
> >
> >
> >
> > On 1/5/17, 2:25 AM, "Luciano Resende" <luckbr1...@gmail.com> wrote:
> >
> > >I have made some progress with a tool to handle the points discussed in
> > >this thread. It's currently a command line tool and given a Zeppelin
> > >notebook (note.json) it generates a Spark scala application, compiles it
> > >using the compiler embedded in the scala sdk and then package all these
> > >resources into a jar that works with spark-submit command.
> > >
> > >I would like to start prototyping the integration into the Zeppelin UI
> and
> > >I was wondering if it would be ok to use the above jar as a dependency
> > >(e.g. from a maven release) and integrate into zeppelin...
> > >
> > >Thoughts ?
> > >
> > >
> > >On Mon, Sep 19, 2016 at 7:47 AM, Sourav Mazumder <
> > >sourav.mazumde...@gmail.com> wrote:
> > >
> > >> To Moon's point, This is what my 

Re: Exporting Spark paragraphs as Spark Applications

[Luciano Resende]

Some use cases discussed earlier on this thread:

https://www.mail-archive.com/dev@zeppelin.apache.org/msg06323.html

https://www.mail-archive.com/dev@zeppelin.apache.org/msg06332.html

On Wed, Jan 4, 2017 at 4:51 PM, Jianfeng (Jeff) Zhang <
jzh...@hortonworks.com> wrote:

>
> I don¹t understand why user want to export zeppelin note as spark
> application.
>
> If they want to trigger the running of spark app, why not use zeppelin¹s
> rest api for that. Even user export it as spark application, most of time
> in reality, they need to submit it through spark job server, so why not
> use zeppelin as a spark job server.
> And if the spark app fails, it is pretty hard to debug it, because the
> exporting tool has changed/restructured the source code.
>
>
> If this is a pretty large and complicated spark application, I don¹t think
> zeppelin is a proper tool for that, they¹d better to use IDE for that
> project.
>
> BTW, After https://github.com/apache/zeppelin/pull/1799, user can define
> the dependency between paragraphs, and they can run one whole note which
> contains different interpreters.
>
>
>
> Best Regard,
> Jeff Zhang
>
>
>
>
>
> On 1/5/17, 2:25 AM, "Luciano Resende" <luckbr1...@gmail.com> wrote:
>
> >I have made some progress with a tool to handle the points discussed in
> >this thread. It's currently a command line tool and given a Zeppelin
> >notebook (note.json) it generates a Spark scala application, compiles it
> >using the compiler embedded in the scala sdk and then package all these
> >resources into a jar that works with spark-submit command.
> >
> >I would like to start prototyping the integration into the Zeppelin UI and
> >I was wondering if it would be ok to use the above jar as a dependency
> >(e.g. from a maven release) and integrate into zeppelin...
> >
> >Thoughts ?
> >
> >
> >On Mon, Sep 19, 2016 at 7:47 AM, Sourav Mazumder <
> >sourav.mazumde...@gmail.com> wrote:
> >
> >> To Moon's point, This is what my vision is around this feature -
> >>
> >> 1. Use should be able to package 1, more than one, all of the
> >>paragraphs in
> >> a Notebook to create a Jar file which can be used with Spark-Submit.
> >>
> >> 2. The tool should automatically remove the all the interactive
> >>statements
> >> like print, show etc.
> >>
> >> 3. The tool should automatically create a Main class in addition to the
> >>jar
> >> file(s) which will internally call the respective jar. User can then
> >>change
> >> this main class if needed for parameterization through Args.
> >>
> >> Regards,
> >> Sourav
> >>
> >> On Mon, Sep 19, 2016 at 7:33 AM, Sourav Mazumder <
> >> sourav.mazumde...@gmail.com> wrote:
> >>
> >> > I am also pretty much for this.
> >> >
> >> > I have got the similar request from each and every people/group who I
> >> > showcased Zeppelin.Regards,
> >> > Sourav
> >> >
> >> > On Fri, Sep 16, 2016 at 8:06 PM, moon soo Lee <m...@apache.org>
> wrote:
> >> >
> >> >> Hi Luciano,
> >> >>
> >> >> I've also got a lot of questions about "Productize the notebook"
> >>every
> >> >> time
> >> >> i meet users use Zeppelin in their work.
> >> >>
> >> >> I think it's actually about two different problems that Zeppelin
> >>need to
> >> >> address.
> >> >>
> >> >> *1) Provide way that interactive notebook becomes part of production
> >> data
> >> >> pipeline.*
> >> >>
> >> >> Although Zeppelin does have quite convenient cron-like scheduler for
> >> each
> >> >> Note, built-in cron scheduler is not ready for serious use in the
> >> >> production. Because it lacks some features like actions after
> >> >> success/fail,
> >> >> fault-tolerance, history, and so on. I think community is working on
> >> >> improving it, and it's going to take some time.
> >> >>  Meanwhile, any external enterprise level job scheduler can run Note
> >>or
> >> >> Paragraph via REST api. But we don't have any guide and examples for
> >>it,
> >> >> what are the REST APIs user can use for this purpose, and how to use
> >> them
> >> >> in various cases (e.g. with authentication on, dynamic form
> >>parameters,
&g

Re: Exporting Spark paragraphs as Spark Applications

[Luciano Resende]

I have made some progress with a tool to handle the points discussed in
this thread. It's currently a command line tool and given a Zeppelin
notebook (note.json) it generates a Spark scala application, compiles it
using the compiler embedded in the scala sdk and then package all these
resources into a jar that works with spark-submit command.

I would like to start prototyping the integration into the Zeppelin UI and
I was wondering if it would be ok to use the above jar as a dependency
(e.g. from a maven release) and integrate into zeppelin...

Thoughts ?


On Mon, Sep 19, 2016 at 7:47 AM, Sourav Mazumder <
sourav.mazumde...@gmail.com> wrote:

> To Moon's point, This is what my vision is around this feature -
>
> 1. Use should be able to package 1, more than one, all of the paragraphs in
> a Notebook to create a Jar file which can be used with Spark-Submit.
>
> 2. The tool should automatically remove the all the interactive statements
> like print, show etc.
>
> 3. The tool should automatically create a Main class in addition to the jar
> file(s) which will internally call the respective jar. User can then change
> this main class if needed for parameterization through Args.
>
> Regards,
> Sourav
>
> On Mon, Sep 19, 2016 at 7:33 AM, Sourav Mazumder <
> sourav.mazumde...@gmail.com> wrote:
>
> > I am also pretty much for this.
> >
> > I have got the similar request from each and every people/group who I
> > showcased Zeppelin.Regards,
> > Sourav
> >
> > On Fri, Sep 16, 2016 at 8:06 PM, moon soo Lee <m...@apache.org> wrote:
> >
> >> Hi Luciano,
> >>
> >> I've also got a lot of questions about "Productize the notebook" every
> >> time
> >> i meet users use Zeppelin in their work.
> >>
> >> I think it's actually about two different problems that Zeppelin need to
> >> address.
> >>
> >> *1) Provide way that interactive notebook becomes part of production
> data
> >> pipeline.*
> >>
> >> Although Zeppelin does have quite convenient cron-like scheduler for
> each
> >> Note, built-in cron scheduler is not ready for serious use in the
> >> production. Because it lacks some features like actions after
> >> success/fail,
> >> fault-tolerance, history, and so on. I think community is working on
> >> improving it, and it's going to take some time.
> >>  Meanwhile, any external enterprise level job scheduler can run Note or
> >> Paragraph via REST api. But we don't have any guide and examples for it,
> >> what are the REST APIs user can use for this purpose, and how to use
> them
> >> in various cases (e.g. with authentication on, dynamic form parameters,
> >> etc). I think a lot of things need to be improved to make zeppelin
> easier
> >> to be part of production pipeline.
> >>
> >> *2) Provide stable way of run spark paragraphs.*
> >>
> >> Another barrier of using notebook in production pipeline is Scala REPL
> in
> >> SparkInterpreter. SparkInterpreter uses Scala REPL to provide
> interactive
> >> scala session and Scala REPL will eventually hit OOME as it compiles and
> >> runs statements. Current workaround in zeppelin is cron-scheduler inside
> >> of
> >> notebook has checkbox that can restart the Note after scheduler runs it.
> >> Of course that option does not apply when external scheduler runs job
> >> through REST api.
> >>
> >> I think what Luciano suggesting, "Export Spark Paragraph as Spark
> >> application" is interesting. If Spark Paragraphs can be easily packaged
> >> into jar (spark application) that can be one of way to address 1) and
> 2).
> >> In case of user already have stable way to schedule spark application
> jar.
> >>
> >> Actually, Flink interactive shell works in similar way internally as far
> >> as
> >> i know. i.e. package compiled class into jar and submit.
> >>
> >> One idea for prototyping is,
> >> How about make a interpreter inside of spark interpreter group, say it's
> >> %spark.build or some better name.
> >>
> >> And if user runs some command like
> >>
> >> %spark.build
> >> package
> >>
> >> then it builds spark application jar based on spark paragraph in the
> Note.
> >> I think it can be the simplest user interface for the prototype.
> >>
> >> Thanks,
> >> moon
> >>
> >> On Fri, Sep 16, 2016 at 1:11 PM Jeremy Anderson <
> >> jer...@objectadjective.com>
> &

[jira] [Created] (ZEPPELIN-1835) Update Elasticsearch to release 2.4.3

[Luciano Resende (JIRA)]

Luciano Resende created ZEPPELIN-1835:
-

 Summary: Update Elasticsearch to release 2.4.3
 Key: ZEPPELIN-1835
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-1835
 Project: Zeppelin
  Issue Type: Improvement
  Components: Interpreters
Affects Versions: 0.7.0
Reporter: Luciano Resende
Assignee: Luciano Resende


While we wait for supporting ES 5.x, upgrade the current version of ES to 
latest 2.x release.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Interpreter zombie processes

[Luciano Resende]

I have also seen similar issues even using zeppelin-ddeamon but didn't have
much time to investigate the issue when it was happening to me.

On Thu, Dec 15, 2016 at 12:15 PM Ruslan Dautkhanov 
wrote:

> Moon,
>
> > ZeppelinServer try to terminate interpreter process when shutting down
> [1].
>
> Unfortunatally, this does not happen all the time.
> I have seen zombie spark interpreter processes many times.
> As a double conirmation - I see spark yarn application was still running.
>
> > Also bin/zeppelin-deamon.sh script clean up if some processes are left
> [2].
>
> We were not using that script. We were just running like zeppelin.sh
> --config ..
> Will try to switch to zeppelin-deamon.sh and see if it makes a difference.
>
> Thanks!
>
>
>
>
> --
> Ruslan Dautkhanov
>
>
>
> On Thu, Dec 15, 2016 at 12:46 PM, moon  wrote:
>
> Hi,
>
>
>
>
>
> ZeppelinServer try to terminate interpreter process when shutting down [1].
>
>
> Also bin/zeppelin-deamon.sh script clean up if some processes are left [2].
>
>
>
>
>
> If some processes are remained after shutting down Zeppelin, that's not an
>
>
> expected result.
>
>
>
>
>
> Thanks,
>
>
> moon
>
>
>
>
>
> [1]
>
>
>
> https://github.com/apache/zeppelin/blob/master/zeppelin-server/src/main/java/org/apache/zeppelin/server/ZeppelinServer.java#L159
>
>
> [2]
>
>
> https://github.com/apache/zeppelin/blob/master/bin/zeppelin-daemon.sh#L218
>
>
>
>
>
> On Thu, Dec 15, 2016 at 11:25 AM Ruslan Dautkhanov 
>
>
> wrote:
>
>
>
>
>
> > How to make Interpreter processes terminate if Zeppelin server exits?
>
>
> >
>
>
> > When we restart Zeppelin server (main process), in many cases,
> Interpreter
>
>
> > process keeps running, essential becoming a zombie processes.
>
>
> >
>
>
> > In case of Spark interpreter, it also holds SparkContext - consuming
>
>
> > server-side
>
>
> > resources too.
>
>
> >
>
>
> > How do we configure Zeppelin to kill all interpreters before its main
>
>
> > process terminates?
>
>
> > Or is this a bug?
>
>
> >
>
>
> > We're running ~2 weeks old snapshot of 0.7.0 Zeppelin.
>
>
> >
>
>
> >
>
>
> > Thank you,
>
>
> > Ruslan Dautkhanov
>
>
> >
>
>
>
> --
Sent from my Mobile device

Handling spark-submit errors

[Luciano Resende]

I was playing with some new data sources in Zeppelin (master) and had an
issue on my --package declaration in zeppelin-env.sh.

First, the error stack was a little misleading, as it was only reporting
connection issues to Spark trying to retrieve paragraph status.

>From my initial investigation, we pretty much invoke the spark-submit from
interpreter.sh... and I was wondering if there is a good way to trap this
kind of issues and be able to provide a better response to the user (or
even better logging)...

One option is to have something similar to what spark does, and actually,
have a class handling some of the interpreter integration logic.

Thoughts  ? Any other possibilities ?

-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: [VOTE] Release Apache Zeppelin 0.6.2 (RC2)

[Luciano Resende]

+1 (non-binding)

On Wed, Oct 12, 2016 at 3:28 AM, Mina Lee <mina...@apache.org> wrote:

> Hi dev,
>
> I propose the following RC to be released for the Apache Zeppelin 0.6.2
> release.
>
> The commit id is 091086de9400dd1c02ca02acf4180b1bf1e9ede7 which is
> corresponds
> to the tag v0.6.2-rc2:
> *https://git-wip-us.apache.org/repos/asf?p=zeppelin.git;a=commit;h=
> 091086de9400dd1c02ca02acf4180b1bf1e9ede7
> <https://git-wip-us.apache.org/repos/asf?p=zeppelin.git;a=commit;h=
> 091086de9400dd1c02ca02acf4180b1bf1e9ede7>*
>
> The release archives (tgz), signature, and checksums are here
> https://dist.apache.org/repos/dist/dev/zeppelin/zeppelin-0.6.2-rc2/
>
> The release candidate consists of the following source distribution archive
> zeppelin-0.6.2.tgz
>
> In addition, the following supplementary binary distributions are provided
> for user convenience at the same location
> zeppelin-0.6.2-bin-all.tgz
> zeppelin-0.6.2-netinst-all.tgz
>
> The maven artifacts are here
> https://repository.apache
> .org/content/repositories/orgapachezeppelin-1020/org/apache/zeppelin/
>
> You can find the KEYS file here:
> https://dist.apache.org/repos/dist/release/zeppelin/KEYS
>
> Release notes available at
> *https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> version=12336543=Html=12316221
> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> version=12336543=Html=12316221>*
>
> Vote will be open for next 72 hours (close at 4am 15/Oct PDT).
>
> [ ] +1 approve
> [ ] 0 no opinion
> [ ] -1 disapprove (and reason why)
>



-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Exporting Spark paragraphs as Spark Applications

[Luciano Resende]

While talking with a few different users, I have been seeing the use case
of using iterative development in Notebooks or Spark Shell and then copying
and pasting the final solution to a formal application repeating itself
very often.

I was wondering if an "Export Spark Paragraphs as a Spark Application
(jar)" would be a feature that Zeppelin community would think it's useful.
But keep in mind there are some limitation here : we would be constrained
to Spark related paragraphs, etc...  but even so, I think there are
multiple scenarios where I see that the ability to have an application that
directly runs on Spark to be very useful.

If the community is interested, let's use this thread to discuss any
specific requirements or suggestions that others might have, and after a
few days I would like to start prototyping this functionality.

Thoughts ?



-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

[jira] [Created] (ZEPPELIN-1407) Scala 2.11 build fails with empty maven repository

[Luciano Resende (JIRA)]

Luciano Resende created ZEPPELIN-1407:
-

 Summary: Scala 2.11 build fails with empty maven repository
 Key: ZEPPELIN-1407
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-1407
 Project: Zeppelin
  Issue Type: Bug
  Components: build
Reporter: Luciano Resende
Assignee: Luciano Resende
Priority: Blocker


rm -rf ~/.m2/repository/org/apache/zeppelin
git clone g...@github.com:apache/zeppelin.git
cd zeppelin
dev/change_scala_version.sh 2.11
mvn -Pspark-2.0 -Pr -Phadoop-2.6 -Psparkr -Ppyspark -Pscalding -Pexamples 
-Pbuild-distr -DskipTests -Dscala-2.11 clean install

Zeppelin Display will fail, and it seems maven is resolving some scala 
libraries to 2.10 and causing compilation issues

[DEBUG] manageArtifactVersion: 
artifact=org.scala-lang:scala-library:jar:2.11.2:test, 
replacement=org.scala-lang:scala-library:jar:2.10.5
[DEBUG] testArtifact: artifact=org.scala-lang:scala-library:jar:2.10.5:test
[DEBUG] manageArtifactVersion: 
artifact=org.scala-lang:scala-library:jar:2.10.5:test, 
replacement=org.scala-lang:scala-library:jar:2.10.5
[DEBUG] omitForNearer: omitted=org.scala-lang:scala-library:jar:2.10.5:test 
kept=org.scala-lang:scala-library:jar:2.10.5:provided
[DEBUG] testArtifact: artifact=org.scala-lang:scala-reflect:jar:2.11.2:test
[DEBUG] includeArtifact: 
artifact=org.scala-lang:scala-reflect:jar:2.11.2:test
[DEBUG] startProcessChildren: 
artifact=org.scala-lang:scala-reflect:jar:2.11.2:test
[DEBUG]   manageArtifactVersion: 
artifact=org.scala-lang:scala-library:jar:2.11.2:test, 
replacement=org.scala-lang:scala-library:jar:2.10.5
[DEBUG]   testArtifact: 
artifact=org.scala-lang:scala-library:jar:2.10.5:test
[DEBUG]   manageArtifactVersion: 
artifact=org.scala-lang:scala-library:jar:2.10.5:test, 
replacement=org.scala-lang:scala-library:jar:2.10.5
[DEBUG]   omitForNearer: 
omitted=org.scala-lang:scala-library:jar:2.10.5:test 
kept=org.scala-lang:scala-library:jar:2.10.5:provided
[DEBUG] endProcessChildren: 
artifact=org.scala-lang:scala-reflect:jar:2.11.2:test
[DEBUG] testArtifact: 
artifact=org.scala-lang.modules:scala-xml_2.11:jar:1.0.2:test
[DEBUG] includeArtifact: 
artifact=org.scala-lang.modules:scala-xml_2.11:jar:1.0.2:test
[DEBUG] startProcessChildren: 
artifact=org.scala-lang.modules:scala-xml_2.11:jar:1.0.2:test
[DEBUG]   manageArtifactVersion: 
artifact=org.scala-lang:scala-library:jar:2.11.1:test, 
replacement=org.scala-lang:scala-library:jar:2.10.5
[DEBUG]   testArtifact: 
artifact=org.scala-lang:scala-library:jar:2.10.5:test
[DEBUG]   manageArtifactVersion: 
artifact=org.scala-lang:scala-library:jar:2.10.5:test, 
replacement=org.scala-lang:scala-library:jar:2.10.5
[DEBUG]   omitForNearer: 
omitted=org.scala-lang:scala-library:jar:2.10.5:test 
kept=org.scala-lang:scala-library:jar:2.10.5:provided

.

[INFO] --- maven-scala-plugin:2.15.2:testCompile (test-compile) @ 
zeppelin-display_2.11 ---
[INFO] Checking for multiple versions of scala
[INFO] includes = [**/*.java,**/*.scala,]
[INFO] excludes = []
[INFO] 
/Users/lresende/opensource/apache/zeppelin/zeppelin/zeppelin-display/src/test/scala:-1:
 info: compiling
[INFO] Compiling 6 source files to 
/Users/lresende/opensource/apache/zeppelin/zeppelin/zeppelin-display/target/test-classes
 at 1472853163836
[ERROR] error: bad symbolic reference. A signature in package.class refers to 
type compileTimeOnly
[INFO] in package scala.annotation which is not available.
[INFO] It may be completely missing from the current classpath, or the version 
on
[INFO] the classpath might be incompatible with the version used when compiling 
package.class.
[ERROR] 
/Users/lresende/opensource/apache/zeppelin/zeppelin/zeppelin-display/src/test/scala/org/apache/zeppelin/display/angular/AbstractAngularElemTest.scala:32:
 error: Reference to class FlatSpec in package scalatest should not have 
survived past type checking,
[INFO] it should have been processed and eliminated during expansion of an 
enclosing macro.
[INFO]   extends FlatSpec with BeforeAndAfter with BeforeAndAfterEach with 
Eventually with Matchers {
[INFO]   ^
[ERROR] 
/Users/lresende/opensource/apache/zeppelin/zeppelin/zeppelin-display/src/test/scala/org/apache/zeppelin/display/angular/AbstractAngularElemTest.scala:63:
 error: Reference to method onClick in class AbstractAngularElem should not 
have survived past type checking,
[INFO] it should have been processed and eliminated during expansion of an 
enclosing macro.
[INFO] val elem = angularElem().onClick(() => {
[INFO] ^

..









--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Running Zeppelin from my github checkout

[Luciano Resende]

In Spark, I can usually build the source code and run a test cluster from
that checkout that I just built. I was trying the same with Zeppelin, and I
can get things up but with some issues around some interpreters and I
cannot even run the tutorial notebook.  Having said that, the distribution
generated by the same build can be extracted and works fine.

Before I spent more time on this, is this scenario something that works (or
used to work) ? Is there any special profiles or something i need to invoke
to enable this ?

BTW, I was running my build as :

mvn -Pspark-2.0 -Pr -Phadoop-2.6 -Psparkr -Ppyspark -Pscalding -Pexamples
-Pbuild-distr -DskipTests -Dscala-2.11 clean install

-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

[jira] [Created] (ZEPPELIN-1379) Flink interpreter missing scala libraries

[Luciano Resende (JIRA)]

Luciano Resende created ZEPPELIN-1379:
-

 Summary: Flink interpreter missing scala libraries
 Key: ZEPPELIN-1379
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-1379
 Project: Zeppelin
  Issue Type: Bug
  Components: zeppelin-interpreter
Affects Versions: 0.6.1
Reporter: Luciano Resende
Assignee: Luciano Resende


Original post : 
https://www.mail-archive.com/users@zeppelin.apache.org/msg00861.html

I tried the latest zeppelin release (binary with all interpreters), and in 
order to make the flink interpreter work in non-local mode, i had to copy all 
scala libraries from another interpreter to the flink interpreter.

After trying to connect to an existing flink cluster i got

zeppelin_1 | Exception in thread "pool-1-thread-2" 
java.lang.NoClassDefFoundError: scala/collection/Seq
zeppelin_1 |at java.lang.Class.forName0(Native Method)
zeppelin_1 |at java.lang.Class.forName(Class.java:264)
zeppelin_1 |at 
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.createInterpreter(RemoteInterpreterServer.java:148)
zeppelin_1 |at 
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$createInterpreter.getResult(RemoteInterpreterService.java:1409)
zeppelin_1 |at 
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$createInterpreter.getResult(RemoteInterpreterService.java:1394)
zeppelin_1 |at 
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
zeppelin_1 |at 
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
zeppelin_1 |at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
zeppelin_1 |at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
zeppelin_1 |at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
zeppelin_1 |at java.lang.Thread.run(Thread.java:745)
zeppelin_1 | Caused by: java.lang.ClassNotFoundException: 
scala.collection.Seq
zeppelin_1 |at 
java.net.URLClassLoader.findClass(URLClassLoader.java:381)
zeppelin_1 |at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
zeppelin_1 |at 
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
zeppelin_1 |at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
zeppelin_1 |... 11 more

to get it working i did

RUN cp /opt/zeppelin/interpreter/ignite/scala*jar 
/opt/zeppelin/interpreter/flink/

which would copy:

scala-compiler-2.11.7.jar
scala-library-2.11.7.jar
scala-parser-combinators_2.11-1.0.4.jar
scala-reflect-2.11.7.jar
scala-xml_2.11-1.0.4.jar

According to Trevor Grant (who helped me out on flink-user), that is a 
regression of upgrading Zeppelin to spark 2.0/Scala 2.11.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ZEPPELIN-1193) Update Node JS dependencies to avoid RegExp DoS

[Luciano Resende (JIRA)]

Luciano Resende created ZEPPELIN-1193:
-

 Summary: Update Node JS dependencies to avoid RegExp DoS
 Key: ZEPPELIN-1193
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-1193
 Project: Zeppelin
  Issue Type: Bug
  Components: build, front-end
Reporter: Luciano Resende
Assignee: Luciano Resende


I was having some intermittent build issues complaining about some node 
dependencies that needs to be updated to avoid RegExp DoS issues  :

[ERROR] npm WARN deprecated minimatch@0.2.14: Please update to minimatch 3.0.2 
or higher to avoid a RegExp DoS issue

[ERROR] npm WARN deprecated minimatch@2.0.10: Please update to minimatch 3.0.2 
or higher to avoid a RegExp DoS issue

[ERROR] npm WARN deprecated graceful-fs@3.0.8: graceful-fs v3.0.0 and before 
will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon 
as possible. Use 'npm ls graceful-fs' to find it in the tree.

[ERROR] npm WARN deprecated graceful-fs@2.0.3: graceful-fs v3.0.0 and before 
will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon 
as possible. Use 'npm ls graceful-fs' to find it in the tree.

[ERROR] npm WARN deprecated minimatch@0.3.0: Please update to minimatch 3.0.2 
or higher to avoid a RegExp DoS issue

[ERROR] npm WARN deprecated graceful-fs@1.2.3: graceful-fs v3.0.0 and before 
will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon 
as possible. Use 'npm ls graceful-fs' to find it in the tree.

[ERROR] npm WARN deprecated lodash@1.0.2: lodash@<3.0.0 is no longer 
maintained. Upgrade to lodash@^4.0.0.

[ERROR] npm WARN deprecated CSSselect@0.7.0: the module is now available as 
'css-select'

[ERROR] npm WARN optional dep failed, continuing fsevents@1.0.12

[ERROR] npm WARN deprecated CSSwhat@0.4.7: the module is now available as 
'css-what'

[ERROR] npm WARN deprecated minimatch@1.0.0: Please update to minimatch 3.0.2 
or higher to avoid a RegExp DoS issue
 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ZEPPELIN-1179) Prefix pom artifacts with scala version prefix

[Luciano Resende (JIRA)]

Luciano Resende created ZEPPELIN-1179:
-

 Summary: Prefix pom artifacts with scala version prefix
 Key: ZEPPELIN-1179
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-1179
 Project: Zeppelin
  Issue Type: Bug
  Components: build
Reporter: Luciano Resende
Assignee: Luciano Resende






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Zeppelin 0.6.1 release

[Luciano Resende]

For Spark 2.0, do we need to wait for it's official release which is close
by...

On Thu, Jul 14, 2016 at 3:45 PM, moon soo Lee <m...@apache.org> wrote:

> Hi folks,
>
> We have released 0.6.0 July 2nd. During the 0.6.0 release discussion [1],
> we could see consensus that having a release 0.6.1 when Zeppelin have Spark
> 2.0 support.
>
> I have created a issue for 0.6.1 release [2]. The issue currently have
> Spark 2.0 and scala 2.11 support as a blocker.
>
> There were some bug fixes on master branch since 0.6.0 release.
> How about we discuss and backport some important bug fixes into
> branch-0.6.0? (if any)
> And how about we start release process of 0.6.1, as soon as all blockers of
> [2] being resolved?
>
>
> Thanks,
> moon
>
> [1]
>
> https://lists.apache.org/thread.html/e107c44df1a7fc4cd232144f6a24d81b32a7d988d5947093b79f05ee@1464110799@%3Cdev.zeppelin.apache.org%3E
> [2] https://issues.apache.org/jira/browse/ZEPPELIN-1177
>



-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: Node dependency issues trying to build zeppelin-web

[Luciano Resende]

I have some of those updates in a local pr, but i need to finish the
support for scala 2.11 before I can spend more time trying to find all
these depende ies that might need to be updates

On Sunday, July 10, 2016, Alexander Bezzubov <b...@apache.org> wrote:

> Hi Luciano,
>
> I think the problem you are reffering has been reported before and is
> logged under https://issues.apache.org/jira/browse/ZEPPELIN-1133
> It would be great indeed to get those dependencies versions upgraded.
>
> --
> Alex
>
> On Fri, Jul 8, 2016 at 1:32 PM, moon soo Lee <m...@apache.org
> <javascript:;>> wrote:
>
> > I believe the problem is fixed by
> > https://github.com/apache/zeppelin/pull/1149.
> > It's merged in master now. So you can try rebase/merge master to your PR
> > and see if problem disappears.
> >
> > Thanks,
> > moon
> >
> > On Thu, Jul 7, 2016 at 2:57 PM Luciano Resende <luckbr1...@gmail.com
> <javascript:;>>
> > wrote:
> >
> > > I am getting some issues on Travis CI while trying to build some PRs
> [1],
> > > and these errors seem to be unrelated to my changes as they are
> > complaining
> > > about node dependency versions :
> > >
> > > [ERROR] npm WARN deprecated minimatch@0.2.14: Please update to
> minimatch
> > > 3.0.2 or higher to avoid a RegExp DoS issue
> > >
> > > [ERROR] npm WARN deprecated minimatch@2.0.10: Please update to
> minimatch
> > > 3.0.2 or higher to avoid a RegExp DoS issue
> > >
> > > [ERROR] npm WARN deprecated graceful-fs@3.0.8: graceful-fs v3.0.0 and
> > > before will fail on node releases >= v7.0. Please update to
> graceful-fs@
> > > ^4.0.0
> > > as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.
> > >
> > > [ERROR] npm WARN deprecated graceful-fs@2.0.3: graceful-fs v3.0.0 and
> > > before will fail on node releases >= v7.0. Please update to
> graceful-fs@
> > > ^4.0.0
> > > as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.
> > >
> > > [ERROR] npm WARN deprecated minimatch@0.3.0: Please update to
> minimatch
> > > 3.0.2 or higher to avoid a RegExp DoS issue
> > >
> > > [ERROR] npm WARN deprecated graceful-fs@1.2.3: graceful-fs v3.0.0 and
> > > before will fail on node releases >= v7.0. Please update to
> graceful-fs@
> > > ^4.0.0
> > > as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.
> > >
> > > [ERROR] npm WARN deprecated lodash@1.0.2: lodash@<3.0.0 is no longer
> > > maintained. Upgrade to lodash@^4.0.0.
> > >
> > > [ERROR] npm WARN deprecated CSSselect@0.7.0: the module is now
> available
> > > as
> > > 'css-select'
> > >
> > > [ERROR] npm WARN optional dep failed, continuing fsevents@1.0.12
> > >
> > > [ERROR] npm WARN deprecated CSSwhat@0.4.7: the module is now available
> > as
> > > 'css-what'
> > >
> > > [ERROR] npm WARN deprecated minimatch@1.0.0: Please update to
> minimatch
> > > 3.0.2 or higher to avoid a RegExp DoS issue
> > >
> > >
> > > Are others seeing these issues ?
> > >
> > > I am trying to see if I can perform some of these updates, but please
> let
> > > me know if this is a know issue or if someone else already have a fix
> for
> > > this.
> > >
> > > [1] https://travis-ci.org/apache/zeppelin/jobs/143121653
> > >
> > > Thanks
> > >
> > >
> > > --
> > > Luciano Resende
> > > http://twitter.com/lresende1975
> > > http://lresende.blogspot.com/
> > >
> >
>


-- 
Sent from my Mobile device

Node dependency issues trying to build zeppelin-web

[Luciano Resende]

I am getting some issues on Travis CI while trying to build some PRs [1],
and these errors seem to be unrelated to my changes as they are complaining
about node dependency versions :

[ERROR] npm WARN deprecated minimatch@0.2.14: Please update to minimatch
3.0.2 or higher to avoid a RegExp DoS issue

[ERROR] npm WARN deprecated minimatch@2.0.10: Please update to minimatch
3.0.2 or higher to avoid a RegExp DoS issue

[ERROR] npm WARN deprecated graceful-fs@3.0.8: graceful-fs v3.0.0 and
before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0
as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.

[ERROR] npm WARN deprecated graceful-fs@2.0.3: graceful-fs v3.0.0 and
before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0
as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.

[ERROR] npm WARN deprecated minimatch@0.3.0: Please update to minimatch
3.0.2 or higher to avoid a RegExp DoS issue

[ERROR] npm WARN deprecated graceful-fs@1.2.3: graceful-fs v3.0.0 and
before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0
as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.

[ERROR] npm WARN deprecated lodash@1.0.2: lodash@<3.0.0 is no longer
maintained. Upgrade to lodash@^4.0.0.

[ERROR] npm WARN deprecated CSSselect@0.7.0: the module is now available as
'css-select'

[ERROR] npm WARN optional dep failed, continuing fsevents@1.0.12

[ERROR] npm WARN deprecated CSSwhat@0.4.7: the module is now available as
'css-what'

[ERROR] npm WARN deprecated minimatch@1.0.0: Please update to minimatch
3.0.2 or higher to avoid a RegExp DoS issue


Are others seeing these issues ?

I am trying to see if I can perform some of these updates, but please let
me know if this is a know issue or if someone else already have a fix for
this.

[1] https://travis-ci.org/apache/zeppelin/jobs/143121653

Thanks


-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: [VOTE] Apache Zeppelin release 0.6.0 (rc1)

[Luciano Resende]

+1

Looked around legal files
And some basic testing

On Wed, Jun 29, 2016 at 3:00 PM, moon soo Lee <m...@apache.org> wrote:

> Hi folks,
>
> I propose the following RC to be released for the Apache Zeppelin
> 0.6.0 release.
>
> The commit id is fa2c0ff93cca49428df8792e7ee35d2b561669bd which is
> corresponds to the tag v0.6.0-rc1:
>
> https://git-wip-us.apache.org/repos/asf?p=zeppelin.git;a=commit;h=fa2c0ff93cca49428df8792e7ee35d2b561669bd
>
> The release archives (tgz), signature, and checksums are here
> http://home.apache.org/~minalee/zeppelin-releases/zeppelin-0.6.0-rc1/
>
> The release candidate consists of the following source distribution archive
> zeppelin-0.6.0.tgz
>
> In addition, the following supplementary binary distributions are provided
> for user convenience at the same location
> zeppelin-0.6.0-bin-all.tgz
> zeppelin-0.6.0-netinst-all.tgz
>
> The maven artifacts are here
>
> https://repository.apache.org/content/repositories/orgapachezeppelin-1011/org/apache/zeppelin/
>
> You can find the KEYS file here:
> https://people.apache.org/keys/committer/minalee.asc
>
> Release notes available at
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12332761=Html=12316221
>
> Vote will be open for next 72 hours (close at 3pm 2/Jul PDT).
>
> [ ] +1 approve
> [ ] 0 no opinion
> [ ] -1 disapprove (and reason why)
>



-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

[jira] [Created] (ZEPPELIN-1032) Update Apache Flink to 1.0.3 release

[Luciano Resende (JIRA)]

Luciano Resende created ZEPPELIN-1032:
-

 Summary: Update Apache Flink to 1.0.3 release
 Key: ZEPPELIN-1032
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-1032
 Project: Zeppelin
  Issue Type: Bug
Reporter: Luciano Resende


Update to latest Flink release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ZEPPELIN-898) Update resources to use Zeppelin TLP links

[Luciano Resende (JIRA)]

Luciano Resende created ZEPPELIN-898:


 Summary: Update resources to use Zeppelin TLP links
 Key: ZEPPELIN-898
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-898
 Project: Zeppelin
  Issue Type: Bug
  Components: Homepage
Reporter: Luciano Resende
Assignee: Luciano Resende
Priority: Minor


Various links are still pointing to the incubator website, or providing links 
for incubator style mailing lists and should be updated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ZEPPELIN-897) Remove incubator disclaimers

[Luciano Resende (JIRA)]

Luciano Resende created ZEPPELIN-897:


 Summary: Remove incubator disclaimers
 Key: ZEPPELIN-897
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-897
 Project: Zeppelin
  Issue Type: Bug
  Components: build
Reporter: Luciano Resende
Assignee: Luciano Resende
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ZEPPELIN-896) Cleanup pom with Apache best practices

[Luciano Resende (JIRA)]

Luciano Resende created ZEPPELIN-896:


 Summary: Cleanup pom with Apache best practices
 Key: ZEPPELIN-896
 URL: https://issues.apache.org/jira/browse/ZEPPELIN-896
 Project: Zeppelin
  Issue Type: Bug
  Components: build
Reporter: Luciano Resende
Assignee: Luciano Resende
Priority: Minor


Apache doesn't recommend usage of @author and/or individual  on the 
poms.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: The Apache Software Foundation Announces Apache(R) Zeppelin™ as a Top-Level Project

[Luciano Resende]

r for anyone," said
> Dr. Konstantin Boudnik, Founder and CEO of Memcore.io. "I am very excited
> to see Apache Zeppelin graduating as an ASF Top Level Project. This shows
> that more people are joining the community, bringing the project to a new
> level, and adding more integration points with existing data analytics and
> transactional software systems. This directly benefits the community
> at-large."
>
> Apache Zeppelin originated in 2013 at NFLabs as Peloton, a commercial data
> analytics product. Since entering the Apache Incubator in December 2014,
> the project has had three releases, and twice participated in Google Summer
> of Code under the Apache umbrella.
>
> "It was an honor to help with the incubation of Zeppelin," said Ted
> Dunning, Vice President of the Apache Incubator. "I have been very
> impressed with the Zeppelin community and the software they have built. I
> see Apache Zeppelin being adopted all over the place where people need to
> apply a notebook style to a wide variety of kinds of computing."
>
> Catch Apache Zeppelin in action during Berlin Buzzwords, 7 June 2016
> https://s.apache.org/mV8E
>
> Availability and Oversight
> Apache Zeppelin software is released under the Apache License v2.0 and is
> overseen by a self-selected team of active contributors to the project. A
> Project Management Committee (PMC) guides the Project's day-to-day
> operations, including community development and product releases. For
> downloads, documentation, and ways to become involved with Apache Zeppelin,
> visit http://zeppelin.apache.org/ and https://twitter.com/ApacheZeppelin
>
> About the Apache Incubator
> The Apache Incubator is the entry path for projects and codebases wishing
> to become part of the efforts at The Apache Software Foundation. All code
> donations from external organizations and existing external projects
> wishing to join the ASF enter through the Incubator to: 1) ensure all
> donations are in accordance with the ASF legal standards; and 2) develop
> new communities that adhere to our guiding principles. Incubation is
> required of all newly accepted projects until a further review indicates
> that the infrastructure, communications, and decision making process have
> stabilized in a manner consistent with other successful ASF projects. While
> incubation status is not necessarily a reflection of the completeness or
> stability of the code, it does indicate that the project has yet to be
> fully endorsed by the ASF. For more information, visit
> http://incubator.apache.org/
>
> About The Apache Software Foundation (ASF)
> Established in 1999, the all-volunteer Foundation oversees more than 350
> leading Open Source projects, including Apache HTTP Server --the world's
> most popular Web server software. Through the ASF's meritocratic process
> known as "The Apache Way," more than 550 individual Members and 5,300
> Committers successfully collaborate to develop freely available
> enterprise-grade software, benefiting millions of users worldwide:
> thousands of software solutions are distributed under the Apache License;
> and the community actively participates in ASF mailing lists, mentoring
> initiatives, and ApacheCon, the Foundation's official user conference,
> trainings, and expo. The ASF is a US 501(c)(3) charitable organization,
> funded by individual donations and corporate sponsors including Alibaba
> Cloud Computing, ARM, Bloomberg, Budget Direct, Cerner, Cloudera, Comcast,
> Confluent, Facebook, Google, Hortonworks, HP, Huawei, IBM, InMotion
> Hosting, iSigma, LeaseWeb, Microsoft, OPDi, PhoenixNAP, Pivotal, Private
> Internet Access, Produban, Red Hat, Serenata Flowers, WANdisco, and Yahoo.
> For more information, visit http://www.apache.org/ and
> https://twitter.com/TheASF
>
> © The Apache Software Foundation. "Apache", "Zeppelin", "Apache Zeppelin",
> "Ambari", "Apache Ambari", "Flink", "Apache Flink", "Hadoop", "Apache
> Hadoop", "Hive", "Apache Hive", "Spark", "Apache Spark", and "ApacheCon"
> are registered trademarks or trademarks of the Apache Software Foundation
> in the United States and/or other countries. All other brands and
> trademarks are the property of their respective owners.
>
> # # #
>
> NOTE: you are receiving this message because you are subscribed to the
> annou...@apache.org distribution list. To unsubscribe, send email from the
> recipient account to announce-unsubscr...@apache.org with the word
> "Unsubscribe" in the subject line.
>



-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/