Re: Integrating vendor-specific code and developing plugins

[Jeremiah D Jordan]

> On May 15, 2017, at 6:57 PM, 大平怜  wrote:
> 
> Thanks for the discussion, all,
> 
>>> * What's included when shipped in tree?
>>> 
>>> Does every idea get merged in? Do we need 30 different Seed providers?  Who
>>> judges what's valuable enough to add?  Who maintains it when it needs
>>> updating?  If the maintainer can't be found, is it removed?  Shipped
>>> broken?  Does the contributed plugins go through the same review process?
>>> Do the contributors need to be committers?  Would CASSANDRA-12627 be merged
>>> in even if nobody saw the value?
> 
> If the rule is to never merge a feature that is pluggable, then it
> would be easy to make a decision, but if not, then we must anyway ask
> ourselves these questions every time a feature is proposed, and I
> think most of the questions are not specific to plugins but generic
> for any new proposals.

Agreed.  Include or not is the same decision we make for any JIRA.

> 
> 
>> In accordance with the idea that the codebase should be better tested, it
>> seems to me like things shouldn't be added that aren't testable.  If
>> there's a million unit tests that are insanely comprehensive but for some
>> reason can never be run, they serve exactly the same value as no tests.
> 
> I think we need to define what the "testable" means.  Does it mean at
> least one of the committers has full access to an environment where
> she can run the tests?  Also, does it mean the environment is
> integrated to the CI?

To me testable means that we can run the tests at the very least for every 
release, but ideally they would be run more often than that.  Especially with 
the push to not release unless the test board is all passing, we should not be 
releasing features that we don’t have a test board for.  Ideally that means we 
have it in ASF CI.  If there is someone that can commit to posting results of 
runs from an outside CI somewhere, then I think that could work as well, but 
that gets pretty cumbersome if we have to check 10 different CI dashboards at 
different locations before every release.

-Jeremiah
-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Integrating vendor-specific code and developing plugins

[大平怜]

Thanks for the discussion, all,

>> * What's included when shipped in tree?
>>
>> Does every idea get merged in? Do we need 30 different Seed providers?  Who
>> judges what's valuable enough to add?  Who maintains it when it needs
>> updating?  If the maintainer can't be found, is it removed?  Shipped
>> broken?  Does the contributed plugins go through the same review process?
>> Do the contributors need to be committers?  Would CASSANDRA-12627 be merged
>> in even if nobody saw the value?

If the rule is to never merge a feature that is pluggable, then it
would be easy to make a decision, but if not, then we must anyway ask
ourselves these questions every time a feature is proposed, and I
think most of the questions are not specific to plugins but generic
for any new proposals.


> In accordance with the idea that the codebase should be better tested, it
> seems to me like things shouldn't be added that aren't testable.  If
> there's a million unit tests that are insanely comprehensive but for some
> reason can never be run, they serve exactly the same value as no tests.

I think we need to define what the "testable" means.  Does it mean at
least one of the committers has full access to an environment where
she can run the tests?  Also, does it mean the environment is
integrated to the CI?


Thanks,
Rei Odaira

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

rebuilding cassandra.apache.org docs

[Jonathan Haddad]

Looks like the docs haven't been rebuilt in a while.  There's a handful of
useful merges recently that I'm aware of that would be great to see on
there.

How do they get rebuilt?  Who can trigger it?

Jon

Re: Integrating vendor-specific code and developing plugins

[benjamin roth]

Absolutely

+ Separate repos for separate modules also separate responsibility. IMHO it
makes a heterogenuous structure more manageable. Both in a technical and a
human or insitutional way.

2017-05-15 13:54 GMT+02:00 Jonathan Haddad :

> There's a handful of issues I can think of with shipping everything
> in-tree.  I'll try to brain dump them here.
>
> * What's included when shipped in tree?
>
> Does every idea get merged in? Do we need 30 different Seed providers?  Who
> judges what's valuable enough to add?  Who maintains it when it needs
> updating?  If the maintainer can't be found, is it removed?  Shipped
> broken?  Does the contributed plugins go through the same review process?
> Do the contributors need to be committers?  Would CASSANDRA-12627 be merged
> in even if nobody saw the value?
>
> * Language support
>
> Cassandra is based on Java 8.  Do we now merge in Scala, Kotlin, Jython?
>
> * Plugins are now tied directly to cassandra release cycle
>
> This one bugs me quite a bit.  With a new plugin, there's going to be a lot
> of rapid iterations.  Spark releases once every 3 months - a lot of the
> packages seem to be released at a much higher frequency.
>
> * In Python, the standard lib is where modules go to die
>
> I forgot where I heard this, but it's pretty accurate.  Including
> everything, "batteries includes", just ends up shipping some pretty
> terrible batteries.  The best stuff for python is on pypi.
>
> Rust deliberately made the decision to limit the std to avoid this
> problem.  There's a "nursery" [1] area for ideas to evolve independently,
> and when some code reaches a high enough level of maturity, it can get
> merged in.  There's also a packages system for third party, non std
> destined code.
>
> Anyways - I'm very +1 on a package system where codebases can independently
> evolve without needing to be part of the project itself.  It's a proven
> model for shipping high quality, useful code, and sometimes is even one of
> the best aspects of a project.  That said, it's quite a bit of work just to
> get going and someone would have to manage that.
>
> Jon
>
> [1] https://github.com/rust-lang-nursery
>
>
> On Sun, May 14, 2017 at 9:03 PM Jeff Jirsa  wrote:
>
> > On Fri, May 12, 2017 at 9:31 PM, J. D. Jordan  >
> > wrote:
> >
> > > In tree would foster more committers which is a good thing.
> >
> >
> > Definitely.
> >
> > But I also agree that not being able to actually run unit tests is a bad
> > > thing. What if we asked people who want to contribute these types of
> > > optimizations to provide the ASF with a Jenkins slave we could test
> them
> > > on, if they want them in tree?
> > >
> >
> > I think SOME FORM of jenkins/unit/dtests need to exist, whether it's ASF
> > puppet controlled or test output explicitly provided by the maintainer.
> >
> >
> > > Otherwise one good thing about out of tree is that the maintainer can
> > > specify "this plugin has been tested and works with Apache Cassandra
> > > version X.Y.Z". If it is in tree it is assumed it will work. If it is
> out
> > > of tree then the plugin can more easily notify a user what version it
> was
> > > last tested with.  And users won't be surprised if they upgrade and
> > things
> > > break.
> > >
> >
> > My main desire is that I'd like to see us do better at helping third
> party
> > contributors be successful in contributing, and to me that means
> something
> > more official. I like the spark packages model. I like the apache httpd
> > model (with LOTS of official extensions in-tree, but a lot externally
> > packaged as well). I'm not a fan of telling people to build and
> distribute
> > their own JARs - it doesn't help the contributors, it doesn't help the
> > users, and it doesn't help the project.
> >
> > - Jeff
> >
>

Re: Integrating vendor-specific code and developing plugins

[Vincent Royer]

Hi all,

I have suggested some modifications in CASSANDRA-13270 to be able to provide 
elasticsearch as a plugin, and these changes are available in a fork at 
https://github.com/strapdata/cassandra 
Of course, i don’t expect to include this in the cassandra tree, but just to be 
able to provide this as an external Cassandra plugin. And even through plugins 
are out of the tree, i would enjoy to contribute to the C* code too !

This plugin is basically a custom secondary index, with hooks to be able to 
start and configure the plugin before playing the commit logs or bootstrap. 
This could probably be improved in a clear plugin API.
Moreover, in my specific case, the major issue is the locale-sensitive behavior 
of Cassandra, there is more than 800 string related functions call with the 
default Locale. So, unit tests with various Locale was not possible until this 
was fixed by a byte-code manipulation with javassit. As a workaround, i have 
develop an ANT task to fix this in the build.xml.
Hope this helps.

Thanks,
Vincent.

On 2017-05-13 04:21 (+0200), Nate McCall  wrote: 
> > This being said, I don't think we should remove support for Windows or> 
> > those snitches. Instead, what I think would be more beneficial, and> 
> > certainly more reflecting the Apache Way, is to see if someone in the> 
> > community would be willing to maintain those components. Looking at 
> > another> 
> > Apache project, Mesos has an interesting breakdown of maintainers for> 
> > specific components [1]. We might consider adopting a similar idea for> 
> > platforms/OSes/architectures/whatevers.> 
> >> 
> 
> This is a good point. Apache Beam does this as well: component> 
> maintainers for specific integrations (Spark, Kafka, HDFS, etc).> 
> Agreed that this is a good way to encourage participation.> 
> 
> -> 
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org> 
> For additional commands, e-mail: dev-h...@cassandra.apache.org> 
> 
>  

Vincent Royer
CEO & Founder
Email: vro...@strapdata.com
Mobile: +33 7 83 18 25 40

Re: Integrating vendor-specific code and developing plugins

[Jonathan Haddad]

There's a handful of issues I can think of with shipping everything
in-tree.  I'll try to brain dump them here.

* What's included when shipped in tree?

Does every idea get merged in? Do we need 30 different Seed providers?  Who
judges what's valuable enough to add?  Who maintains it when it needs
updating?  If the maintainer can't be found, is it removed?  Shipped
broken?  Does the contributed plugins go through the same review process?
Do the contributors need to be committers?  Would CASSANDRA-12627 be merged
in even if nobody saw the value?

* Language support

Cassandra is based on Java 8.  Do we now merge in Scala, Kotlin, Jython?

* Plugins are now tied directly to cassandra release cycle

This one bugs me quite a bit.  With a new plugin, there's going to be a lot
of rapid iterations.  Spark releases once every 3 months - a lot of the
packages seem to be released at a much higher frequency.

* In Python, the standard lib is where modules go to die

I forgot where I heard this, but it's pretty accurate.  Including
everything, "batteries includes", just ends up shipping some pretty
terrible batteries.  The best stuff for python is on pypi.

Rust deliberately made the decision to limit the std to avoid this
problem.  There's a "nursery" [1] area for ideas to evolve independently,
and when some code reaches a high enough level of maturity, it can get
merged in.  There's also a packages system for third party, non std
destined code.

Anyways - I'm very +1 on a package system where codebases can independently
evolve without needing to be part of the project itself.  It's a proven
model for shipping high quality, useful code, and sometimes is even one of
the best aspects of a project.  That said, it's quite a bit of work just to
get going and someone would have to manage that.

Jon

[1] https://github.com/rust-lang-nursery


On Sun, May 14, 2017 at 9:03 PM Jeff Jirsa  wrote:

> On Fri, May 12, 2017 at 9:31 PM, J. D. Jordan 
> wrote:
>
> > In tree would foster more committers which is a good thing.
>
>
> Definitely.
>
> But I also agree that not being able to actually run unit tests is a bad
> > thing. What if we asked people who want to contribute these types of
> > optimizations to provide the ASF with a Jenkins slave we could test them
> > on, if they want them in tree?
> >
>
> I think SOME FORM of jenkins/unit/dtests need to exist, whether it's ASF
> puppet controlled or test output explicitly provided by the maintainer.
>
>
> > Otherwise one good thing about out of tree is that the maintainer can
> > specify "this plugin has been tested and works with Apache Cassandra
> > version X.Y.Z". If it is in tree it is assumed it will work. If it is out
> > of tree then the plugin can more easily notify a user what version it was
> > last tested with.  And users won't be surprised if they upgrade and
> things
> > break.
> >
>
> My main desire is that I'd like to see us do better at helping third party
> contributors be successful in contributing, and to me that means something
> more official. I like the spark packages model. I like the apache httpd
> model (with LOTS of official extensions in-tree, but a lot externally
> packaged as well). I'm not a fan of telling people to build and distribute
> their own JARs - it doesn't help the contributors, it doesn't help the
> users, and it doesn't help the project.
>
> - Jeff
>