For decoupling the build of core Zeppelin vs third-party interpreters, we
can use Maven profiles:
in the root pom.xml, you can define in the default profile all the Maven
modules that should be part of the core Zeppelin build. Additional
interpreters can be defined in separated profiles so that people keen on
using them can build them easily. Ex:
<profiles>
<!-- Core profile -->
<profile>
<id>core</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<modules>
<module>zeppelin-interpreter</module>
<module>zeppelin-zengine</module>
<module>spark</module>
<module>markdown</module>
<module>angular</module>
<module>shell</module>
<module>zeppelin-web</module>
<module>zeppelin-server</module>
<module>zeppelin-distribution</module>
</modules>
</profile>
<!-- Cassandra interpreter -->
<profile>
<id>cassandra</id>
<modules>
<module>cassandra</module>
</modules>
</profile>
<!-- Lens interpreter -->
<profile>
<id>lens</id>
<modules>
<module>lens</module>
</modules>
</profile>
</profiles>
To build Zeppelin with support for Cassandra & Lens:
mvn clean package -Pcore,cassandra,lens -D....
What do you think ?
On Thu, Jul 23, 2015 at 7:38 AM, IT CTO <[email protected]> wrote:
> I volunteer to write a doc for the zeppelin documentation site on step by
> step for adding an "external interpreter" targeted for non developers who
> wants to add someone's else external interpreter.
> The challenge I see is from where to get the compiled interpreter from?
> - my answer is to have a list of external interpreter on our site which
> points to the github of the external interpreter and the user will download
> himself. - I think this should help with licensing issues as well as an
> external site can hold a non apache license interpreter (MySQL, Oracle)
>
> Eran
>
>
> On Thu, Jul 23, 2015 at 2:38 AM moon soo Lee <[email protected]> wrote:
>
> > Thanks for the opinions and feedback.
> >
> > My question was more like simply asking about merging code that does not
> > have test, but apparently went back to recurrent subject. :-)
> >
> > Actually, plug'n'play interpreter is already possible by dropping all the
> > necessary jars under interpreter/[name] directory and add config to
> > zeppelin-site.xml. I think nothing stops making externally managed
> > interpreter.
> >
> > I think having list of all internal + external interpreters in homepage
> or
> > wiki and let user update it would help, as a first step.
> >
> > Best,
> > moon
> >
> > On Wed, Jul 22, 2015 at 9:42 PM Paul Curtis <[email protected]>
> wrote:
> >
> > > +1
> > >
> > > I agree with tog ... the core interpreters should be those that can
> > > complete all the tests and provide the functionality as well. Spark
> local
> > > being the best example. The Apache Drill interpreter I wrote would
> need a
> > > single node Drill installation in order to provide the same testing
> > > capability. I am working to provide this.
> > >
> > > However, I would suggest that review may be needed. As each test and
> > > interpreter adds Zeppelin code, it also adds to the distributable as
> > well.
> > > Currently, the distributable is around ~500MB in size. Is the goal to
> > > provide a completely standalone Zeppelin with all the interpreters and
> > > environments included? Or is the goal to provide a front end to those
> > > environments?
> > >
> > > Maybe that's the decision: which environments and interpreters are
> > included
> > > in a standalone distribution.
> > >
> > > paul
> > >
> > > On Wed, Jul 22, 2015 at 7:52 AM, IT CTO <[email protected]> wrote:
> > >
> > > > I think that we can have the interpreters add in the build system
> using
> > > > compilation parameters (e.g. same as the list in the zeppelin-env.xml
> > > that
> > > > way the basic build builds only the core interpreters and the user
> can
> > > > easily interperters to the build process.
> > > > With regard to a release, this can be just as adding the jar and the
> > > config
> > > > or just the jar with auto-discovery
> > > > Eran
> > > >
> > > > On Wed, Jul 22, 2015 at 1:40 PM DuyHai Doan <[email protected]>
> > > wrote:
> > > >
> > > > > If we want to make the intepreters system modular to decouple souce
> > > code,
> > > > > the process of activating an interpreter should be flexible and
> easy
> > to
> > > > > use.
> > > > >
> > > > > Asking end-users to make a custom build is not a viable strategy.
> > > > Recently
> > > > > I've seen some people trying to make a custom build of Zeppelin and
> > > > facing
> > > > > a lot of issues (incorrect Maven version, no Bower installed,
> > incorrect
> > > > > repository policies settings for Maven etc ...)
> > > > >
> > > > > Ideally, activating an interpreter should be as simple as dropping
> a
> > > jar
> > > > > into the lib directory
> > > > >
> > > > > On Wed, Jul 22, 2015 at 12:28 PM, <[email protected]>
> wrote:
> > > > >
> > > > > > +1 what tog says
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: tog
> > > > > >
> > > > > > Sent: Wednesday, July 22, 12:22 AM
> > > > > >
> > > > > > Subject: Interpreter with no test.
> > > > > >
> > > > > > To: [email protected]
> > > > > >
> > > > > >
> > > > > >
> > > > > > +1
> > > > > >
> > > > > > This seems to be a recurrent topic ;-)
> > > > > >
> > > > > >
> > > > > > It has to be decided what are the core interpreters the Zeppelin
> > want
> > > > to
> > > > > >
> > > > > > support - then I would believe you can have a wiki/webpage
> > dedicated
> > > to
> > > > > >
> > > > > > listing all interpreters that are known to be working with
> > Zeppelin.
> > > > You
> > > > > >
> > > > > > may even consider having recommended plugins and/or ranking in
> > case 2
> > > > > >
> > > > > > plugins are doing the same job. Grails (but it is probably not
> the
> > > only
> > > > > >
> > > > > > one) is implementing this (see https://grails.org/plugins/)
> > > > > >
> > > > > >
> > > > > > In order to keep Zeppelin low in term of footprint - the core
> > > commiters
> > > > > can
> > > > > >
> > > > > > even decide to support some of those plugins (shell for example)
> > > > > >
> > > > > > At one end of the spectrum you could imagine Zeppelin without any
> > > > > >
> > > > > > interpreters and at the other end Zeppelin with all known
> plugins -
> > > the
> > > > > >
> > > > > > truth is probably in the middle.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wednesday, July 22, 2015, Anthony Corbacho <
> > > > > [email protected]
> > > > > >
> > > > > > <javascript:_e(%7B%7D,'cvml','[email protected]');>>
> > wrote:
> > > > > >
> > > > > >
> > > > > > > Hi,
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > IT CTO is right, right now, zeppelin merge every interpreter in
> > the
> > > > > code
> > > > > >
> > > > > > > base and its mean that we (committer) have to take care of
> build
> > > > > falling
> > > > > > or
> > > > > >
> > > > > > > questions for those interpreters. and personally i have no
> > > experience
> > > > > in
> > > > > >
> > > > > > > presto, cassandra etcetc.
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > I think, It can be interesting to detach interpreters from
> > zeppelin
> > > > > code
> > > > > >
> > > > > > > base and create a sortof plug'n'play module where the users can
> > > plug
> > > > > the
> > > > > >
> > > > > > > interpreter he wants to use.
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > With this approach, we can keep zeppelin code base smaller (we
> > > > provide
> > > > > > the
> > > > > >
> > > > > > > core + maybe some interpreters (spark, md, shell) as default).
> > and
> > > > the
> > > > > >
> > > > > > > community can build and manage other interpreters (i assume if
> > they
> > > > > build
> > > > > >
> > > > > > > it they have experiences and probably can answer questions).
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > What do you think?
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > On Wed, Jul 22, 2015 at 2:34 PM, IT CTO <[email protected]>
> > wrote:
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > > I think the question is what\who is going to fix issues in
> > these
> > > > > >
> > > > > > > > Interpreters if something fails?
> > > > > >
> > > > > > > > I am guessing that if one uses these interpreters and
> approach
> > > the
> > > > > >
> > > > > > > > community with questions then we might not have the right
> > support
> > > > for
> > > > > >
> > > > > > > him.
> > > > > >
> > > > > > > >
> > > > > >
> > > > > > > >
> > > > > >
> > > > > > > > On Wed, Jul 22, 2015 at 8:04 AM moon soo Lee <
> [email protected]>
> > > > > wrote:
> > > > > >
> > > > > > > >
> > > > > >
> > > > > > > > > Hi folks,
> > > > > >
> > > > > > > > >
> > > > > >
> > > > > > > > > There're some open pullrequests with complete interpreter
> > > > > >
> > > > > > > implementation
> > > > > >
> > > > > > > > > but no test (eg.
> > > > > > https://github.com/apache/incubator-zeppelin/pull/110
> > > > > >
> > > > > > > ,
> > > > > >
> > > > > > > > > https://github.com/apache/incubator-zeppelin/pull/68).
> > > > > >
> > > > > > > > >
> > > > > >
> > > > > > > > > Personally, I'm feeling not safe having code without test,
> > but
> > > at
> > > > > the
> > > > > >
> > > > > > > > same
> > > > > >
> > > > > > > > > time, keeping these great contributions unmerged sounds not
> > > cool.
> > > > > >
> > > > > > > > >
> > > > > >
> > > > > > > > > I'd like to hear opinions about merging them with some mark
> > > such
> > > > as
> > > > > >
> > > > > > > > 'beta'
> > > > > >
> > > > > > > > > or 'untested', and create issue for adding test.
> > > > > >
> > > > > > > > >
> > > > > >
> > > > > > > > > What do you think?
> > > > > >
> > > > > > > > >
> > > > > >
> > > > > > > > > Thanks,
> > > > > >
> > > > > > > > > moon
> > > > > >
> > > > > > > > >
> > > > > >
> > > > > > > >
> > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > PGP KeyID: 2048R/EA31CFC9 subkeys.pgp.net
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Paul Curtis - *Senior Product Specialist - Field Enablement Team
> > > *O: *+1 203-660-0015 - *M:* +1 203-539-9705
> > >
> > >
> > > Now Available - Free Hadoop On-Demand Training
> > > <
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > > >
> > >
> >
>
