Roman, James, Jim, Jongyoul, thank you guys for a feedback.
James's point on community growing though the interpreter contributors\maintainers sounds great indeed and difficulties supporting external "plugins" with all the versions could overweight the benefits of such separation. And especially valuable is experience with Cloundstack that Jim brought in. I really like the idea of mixed model like the one that Jim has described: more mature interpreters should indeed belong to the root, as it is already now, so we just need to find a technical mean of separating early-stages ones (until the community steps up to say they are important\mature enough, by maintaining them). AFAIK one can also see the same pattern in Apache Spark project. Any thought on how such "zeppelin-extras" might work? On Fri, May 15, 2015 at 1:44 AM, Jim Cooley <[email protected]> wrote: > Forgive me for chiming in as I’m new to the group, but we had a similar > problem with another open source project i worked on: Openstack. There > were three key concerns: 1) some of them must be in the project or else it > is highly likely that they will be inadvertently broken by changes in other > parts of the code and the onus on maintaining core language should be on > the person making the change. a separate project makes this very difficult > to do/enforce. 2) the set is open ended and some are really very > early-stage while others are more mature. pushing the burden to keep > early-stage projects aligned should be on the early-stage project owners > and not the core. 3) some interpreters must be part of the core project as > you need to have at least one, but most likely at least a small group of > canonical interpreters that define the APIs and provide examples for others > to follow. > > Perhaps something of a mix as we arrived at for that project: > + base interpreters and ones that are actively maintained and are > ‘mature’ could be included in the base project. > + have a second project that includes the early-stage interpreters, ones > that are not mature, or do not have a broad base of people supporting them. > + you can of course migrate them from one to the other as they mature, > as the community steps up to say that they are important (by maintaining > them), or as they become ‘deprecated’ or cease to be maintained actively. > > Just a thought, > > > Jim > > > > On May 14, 2015, at 9:16 AM, Roman Shaposhnik <[email protected]> > wrote: > > With my mentor hat on -- huge +1 to what James said. > > Thanks, > Roman. > > On Thu, May 14, 2015 at 6:09 AM, James Carman > <[email protected]> wrote: > > Consider the Apache Camel project. There are a *ton* of components > > available within Camel and most everything (with compatible license) is > > within the Camel project itself, not outside. Keeping the interpreters > > outside and independent gets into a nightmare situation when new versions > > come out, because you don't know which ones work with which versions > (usually > > folks end up combining them into some grouping anyway to avoid the > > fragmentation). You may want to use two different ones in your notebook, > > but they support completely different versions of the core. Also, as I > > said, it would be much tougher to grow a community around the > interpreters > > in isolation. It would be much better to bring them into the fold and > grow > > a large community around all of it together. Now, that doesn't mean that > > you couldn't still keep some outside (zeppelin-extras perhaps?). It also > > doesn't mean that the core has to follow the same lifecycle as the > > "canonical" interpreters. > > > > > > On Wed, May 13, 2015 at 9:34 PM Jongyoul Lee <[email protected]> wrote: > > > >> +1 for me. I think we can use a wiki for maintaining third party > >> interpreters. > >> > >> On Thu, May 14, 2015 at 10:23 AM, Alexander Bezzubov <[email protected]> > >> wrote: > >> > >>> Hi, > >>> > >>> Damien, thanks for bringing the discussion in, and thank everybody for > >>> opinions: it is indeed very timely as we see more and more submodules > >> with > >>> interpreters implementations and having a consistent strategy on how to > >>> deal with them is very important now. > >>> > >>> What I see right now is: > >>> - *number of interpreters is potentially unbounded* > >>> that means that bringing all of them to the root repo is neither > >>> feasible nor a maintainable solution: we need then not only to make > sure > >>> that patches with such contributions are good but also that ALL > >> maintainers > >>> will stick around long enough to support further evolution of API until > >> at > >>> least a stable release like 1.0 > >>> As zeppelin is 0.x and is far from being stable on API level - this > >> will > >>> bring us to the situation when the project does not compile because of > >> some > >>> interpret is not update really quick. > >>> > >>> - *example of other successful extendable systems* > >>> Other systems like Apache Corduva (while being a Phonegap) or > Homebrew > >>> in early days all had a repos with extension code but then moved to > just > >> a > >>> 'registry' model, meaning that they just host a registry in 'npm' > fashion > >>> and only provide a tools and a workflow to use\contribute those > >> extensions. > >>> I.e Ipython does not host all kernels in main repo rather just list > >> all > >>> them in the project wiki > >>> > >>> - *importance of keeping Zeppelin codebase small and simple for fast > >>> iterations* > >>> Last but not least, this is kind of implication from the first > >> statement > >>> but is very important for longevity of the project. The whole field of > >>> large scale data analytics systems is very dynamic, and it makes > perfect > >>> sense, at least to me to, to keep the focus of the project on > delivering > >>> the core value rather then covering all potential applications. > >>> > >>> That being said, if this would be the vote I would support a > >>> "registry-like" model, with an external codebases for interpreters AND > >> some > >>> tools on zeppelin side to simplify management like > install\update\delete > >>> interpreters i.e from Maven or local\remote filesystem URL. > >>> > >>> Those changes would require further discussion in case we have a > >> consensus > >>> here. > >>> > >>> > >>> On Wed, May 13, 2015 at 9:13 PM, James Carman < > >> [email protected]> > >>> wrote: > >>> > >>>> Consider this a huge -1 from me > >>>> On Wed, May 13, 2015 at 7:39 AM James Carman < > >> [email protected] > >>>> > >>>> wrote: > >>>> > >>>>> Why not invite the contributors to be committers? You don't want to > >>>> leave > >>>>> the contributors out in the cold. Likewise, the interpreters are > >> less > >>>>> likely to attract new contributions outside the ASF. You want to > >>> build a > >>>>> community around the code. Vote them in! > >>>>> > >>>>> On Wed, May 13, 2015 at 4:18 AM Corneau Damien < > >> [email protected]> > >>>>> wrote: > >>>>> > >>>>>> Hi, > >>>>>> > >>>>>> I just want to open a discussion about Zeppelin Interpreters. > >>>>>> > >>>>>> Currently we are accepting interpreters and merging them to the > >>> Zeppelin > >>>>>> Branch when people have some done. > >>>>>> > >>>>>> I see more and more issues/post on the mailing list regarding some > >>>>>> problems > >>>>>> with interpreters, while it is a big part of Zeppelin, I think it > >> will > >>>>>> become quickly hard to maintain. > >>>>>> > >>>>>> It could be a better approach to separate Interpreters from the > >> build > >>>>>> (kinda like plugins/package), that people can include easily > >> depending > >>>> on > >>>>>> their needs, and let the creator take care of issues related to it. > >>>>>> > >>>>>> Any thoughts? > >>>>>> > >>>>> > >>>> > >>> > >> > >> > >> > >> -- > >> 이종열, Jongyoul Lee, 李宗烈 > >> http://madeng.net > >> > >
