Re: Integrating vendor-specific code and developing plugins

[大平怜]

Thanks a lot, Jeremiah, Stefan,

That is exactly what I was looking for.


Rei Odaira

2017-06-03 3:36 GMT-05:00 Stefan Podkowinski :
> I'd suggest to use the git docs for the new pages, so we can accept pull
> requests for adding other plugins. [1]
>
> We can also link there from the main pages. Maybe the community page
> would be a good place for that.
>
> [1] https://cassandra.apache.org/doc/latest/development/documentation.html
>
> On 06/03/2017 02:28 AM, J. D. Jordan wrote:
>> The site is in svn for the main pages.
>>
>> https://svn.apache.org/repos/asf/cassandra/site/src/
>>
>> And in git for the docs.
>> https://github.com/apache/cassandra/tree/trunk/doc/source
>>
>> For suggested changes make a JIRA with proposed changes.
>>
>> -Jeremiah
>>
>>> On Jun 2, 2017, at 5:36 PM, 大平怜  wrote:
>>>
>>> Hi all,
>>>
>>> As for our CAPI Flash enablement code, we are now working on the
>>> plugin approach.  Once it is ready, we would like to propose changes
>>> in some Web pages of http://cassandra.apache.org for better plugin
>>> support.  I don't find any official process to propose such changes,
>>> but could anyone tell us who we should work with?
>>>
>>>
>>> Thanks,
>>> Rei Odaira
>>>
>>> 2017-05-19 16:56 GMT-05:00 大平怜 :
 Hi all,

 Everybody seems to agree with improving the plugin ecosystem (as well
 as not small amount of effort needed to do that), but about
 vendor-specific code integration, let me summarize the issues raised
 so far.

 1) How to test it?  What if my code breaks the vendor-specific build?
 2) How to maintain it?  Who is to maintain the code?
 3) How does it affect the Cassandra release cycle?
 4) How to remove it?  It might be hard to remove once integrated, from
 both technical and markting perspective.

 I think #3 and #4 are rather general issues for any newly proposed
 changes, while #1 and #2 are the most problematic for niche :-)
 platform specific code.  #1 is technically solvable, for example, as
 Jeff (thanks!) showed with the Jenkins slave at ASF and as we are
 trying to connect a ppc machine with a CAPI device to the CI.

 #2 must be socially solved, as a component/platform maintainer system
 should be introduced like some other Apache projects.  Is there any
 chance to have such a system in Cassandra?


 Thanks,
 Rei Odaira

 2017-05-18 12:36 GMT-05:00 Jeff Jirsa :
>
>
>> On Thu, May 18, 2017 at 10:28 AM, Jeff Jirsa  wrote:
>>
>>
>>
>> On Mon, May 15, 2017 at 5:25 PM, Jeremiah D Jordan
>>  wrote:
>>>
>>>
>>>
>>> To me testable means that we can run the tests at the very least for
>>> every release, but ideally they would be run more often than that.
>>> Especially with the push to not release unless the test board is all
>>> passing, we should not be releasing features that we don’t have a test 
>>> board
>>> for.  Ideally that means we have it in ASF CI.  If there is someone 
>>> that can
>>> commit to posting results of runs from an outside CI somewhere, then I 
>>> think
>>> that could work as well, but that gets pretty cumbersome if we have to 
>>> check
>>> 10 different CI dashboards at different locations before every release.
>>
>>
>>
>> It turns out there's a ppc64le jenkins slave @ asf, so I've setup
>> https://builds.apache.org/view/A-D/view/Cassandra/job/cassandra-devbranch-ppc64le-testall/
>> for testing.
>>
>> Like our other devbranch-testall builds, it takes a repo+branch as
>> parameters, and runs unit tests. While the unit tests aren't passing, 
>> this
>> platform should now be considered testable.
>>
>
> (Platform != device, though, the CAPI device obviously isn't there, so the
> row cache implementation still doesn't have public testing)
>
>
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>>
>>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Integrating vendor-specific code and developing plugins

[Stefan Podkowinski]

I'd suggest to use the git docs for the new pages, so we can accept pull
requests for adding other plugins. [1]

We can also link there from the main pages. Maybe the community page
would be a good place for that.

[1] https://cassandra.apache.org/doc/latest/development/documentation.html

On 06/03/2017 02:28 AM, J. D. Jordan wrote:
> The site is in svn for the main pages.
> 
> https://svn.apache.org/repos/asf/cassandra/site/src/
> 
> And in git for the docs.
> https://github.com/apache/cassandra/tree/trunk/doc/source
> 
> For suggested changes make a JIRA with proposed changes.
> 
> -Jeremiah
> 
>> On Jun 2, 2017, at 5:36 PM, 大平怜  wrote:
>>
>> Hi all,
>>
>> As for our CAPI Flash enablement code, we are now working on the
>> plugin approach.  Once it is ready, we would like to propose changes
>> in some Web pages of http://cassandra.apache.org for better plugin
>> support.  I don't find any official process to propose such changes,
>> but could anyone tell us who we should work with?
>>
>>
>> Thanks,
>> Rei Odaira
>>
>> 2017-05-19 16:56 GMT-05:00 大平怜 :
>>> Hi all,
>>>
>>> Everybody seems to agree with improving the plugin ecosystem (as well
>>> as not small amount of effort needed to do that), but about
>>> vendor-specific code integration, let me summarize the issues raised
>>> so far.
>>>
>>> 1) How to test it?  What if my code breaks the vendor-specific build?
>>> 2) How to maintain it?  Who is to maintain the code?
>>> 3) How does it affect the Cassandra release cycle?
>>> 4) How to remove it?  It might be hard to remove once integrated, from
>>> both technical and markting perspective.
>>>
>>> I think #3 and #4 are rather general issues for any newly proposed
>>> changes, while #1 and #2 are the most problematic for niche :-)
>>> platform specific code.  #1 is technically solvable, for example, as
>>> Jeff (thanks!) showed with the Jenkins slave at ASF and as we are
>>> trying to connect a ppc machine with a CAPI device to the CI.
>>>
>>> #2 must be socially solved, as a component/platform maintainer system
>>> should be introduced like some other Apache projects.  Is there any
>>> chance to have such a system in Cassandra?
>>>
>>>
>>> Thanks,
>>> Rei Odaira
>>>
>>> 2017-05-18 12:36 GMT-05:00 Jeff Jirsa :


> On Thu, May 18, 2017 at 10:28 AM, Jeff Jirsa  wrote:
>
>
>
> On Mon, May 15, 2017 at 5:25 PM, Jeremiah D Jordan
>  wrote:
>>
>>
>>
>> To me testable means that we can run the tests at the very least for
>> every release, but ideally they would be run more often than that.
>> Especially with the push to not release unless the test board is all
>> passing, we should not be releasing features that we don’t have a test 
>> board
>> for.  Ideally that means we have it in ASF CI.  If there is someone that 
>> can
>> commit to posting results of runs from an outside CI somewhere, then I 
>> think
>> that could work as well, but that gets pretty cumbersome if we have to 
>> check
>> 10 different CI dashboards at different locations before every release.
>
>
>
> It turns out there's a ppc64le jenkins slave @ asf, so I've setup
> https://builds.apache.org/view/A-D/view/Cassandra/job/cassandra-devbranch-ppc64le-testall/
> for testing.
>
> Like our other devbranch-testall builds, it takes a repo+branch as
> parameters, and runs unit tests. While the unit tests aren't passing, this
> platform should now be considered testable.
>

 (Platform != device, though, the CAPI device obviously isn't there, so the
 row cache implementation still doesn't have public testing)


>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Integrating vendor-specific code and developing plugins

[J. D. Jordan]

The site is in svn for the main pages.

https://svn.apache.org/repos/asf/cassandra/site/src/

And in git for the docs.
https://github.com/apache/cassandra/tree/trunk/doc/source

For suggested changes make a JIRA with proposed changes.

-Jeremiah

> On Jun 2, 2017, at 5:36 PM, 大平怜  wrote:
> 
> Hi all,
> 
> As for our CAPI Flash enablement code, we are now working on the
> plugin approach.  Once it is ready, we would like to propose changes
> in some Web pages of http://cassandra.apache.org for better plugin
> support.  I don't find any official process to propose such changes,
> but could anyone tell us who we should work with?
> 
> 
> Thanks,
> Rei Odaira
> 
> 2017-05-19 16:56 GMT-05:00 大平怜 :
>> Hi all,
>> 
>> Everybody seems to agree with improving the plugin ecosystem (as well
>> as not small amount of effort needed to do that), but about
>> vendor-specific code integration, let me summarize the issues raised
>> so far.
>> 
>> 1) How to test it?  What if my code breaks the vendor-specific build?
>> 2) How to maintain it?  Who is to maintain the code?
>> 3) How does it affect the Cassandra release cycle?
>> 4) How to remove it?  It might be hard to remove once integrated, from
>> both technical and markting perspective.
>> 
>> I think #3 and #4 are rather general issues for any newly proposed
>> changes, while #1 and #2 are the most problematic for niche :-)
>> platform specific code.  #1 is technically solvable, for example, as
>> Jeff (thanks!) showed with the Jenkins slave at ASF and as we are
>> trying to connect a ppc machine with a CAPI device to the CI.
>> 
>> #2 must be socially solved, as a component/platform maintainer system
>> should be introduced like some other Apache projects.  Is there any
>> chance to have such a system in Cassandra?
>> 
>> 
>> Thanks,
>> Rei Odaira
>> 
>> 2017-05-18 12:36 GMT-05:00 Jeff Jirsa :
>>> 
>>> 
 On Thu, May 18, 2017 at 10:28 AM, Jeff Jirsa  wrote:
 
 
 
 On Mon, May 15, 2017 at 5:25 PM, Jeremiah D Jordan
  wrote:
> 
> 
> 
> To me testable means that we can run the tests at the very least for
> every release, but ideally they would be run more often than that.
> Especially with the push to not release unless the test board is all
> passing, we should not be releasing features that we don’t have a test 
> board
> for.  Ideally that means we have it in ASF CI.  If there is someone that 
> can
> commit to posting results of runs from an outside CI somewhere, then I 
> think
> that could work as well, but that gets pretty cumbersome if we have to 
> check
> 10 different CI dashboards at different locations before every release.
 
 
 
 It turns out there's a ppc64le jenkins slave @ asf, so I've setup
 https://builds.apache.org/view/A-D/view/Cassandra/job/cassandra-devbranch-ppc64le-testall/
 for testing.
 
 Like our other devbranch-testall builds, it takes a repo+branch as
 parameters, and runs unit tests. While the unit tests aren't passing, this
 platform should now be considered testable.
 
>>> 
>>> (Platform != device, though, the CAPI device obviously isn't there, so the
>>> row cache implementation still doesn't have public testing)
>>> 
>>> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

Re: Integrating vendor-specific code and developing plugins

[大平怜]

Hi all,

As for our CAPI Flash enablement code, we are now working on the
plugin approach.  Once it is ready, we would like to propose changes
in some Web pages of http://cassandra.apache.org for better plugin
support.  I don't find any official process to propose such changes,
but could anyone tell us who we should work with?


Thanks,
Rei Odaira

2017-05-19 16:56 GMT-05:00 大平怜 :
> Hi all,
>
> Everybody seems to agree with improving the plugin ecosystem (as well
> as not small amount of effort needed to do that), but about
> vendor-specific code integration, let me summarize the issues raised
> so far.
>
> 1) How to test it?  What if my code breaks the vendor-specific build?
> 2) How to maintain it?  Who is to maintain the code?
> 3) How does it affect the Cassandra release cycle?
> 4) How to remove it?  It might be hard to remove once integrated, from
> both technical and markting perspective.
>
> I think #3 and #4 are rather general issues for any newly proposed
> changes, while #1 and #2 are the most problematic for niche :-)
> platform specific code.  #1 is technically solvable, for example, as
> Jeff (thanks!) showed with the Jenkins slave at ASF and as we are
> trying to connect a ppc machine with a CAPI device to the CI.
>
> #2 must be socially solved, as a component/platform maintainer system
> should be introduced like some other Apache projects.  Is there any
> chance to have such a system in Cassandra?
>
>
> Thanks,
> Rei Odaira
>
> 2017-05-18 12:36 GMT-05:00 Jeff Jirsa :
>>
>>
>> On Thu, May 18, 2017 at 10:28 AM, Jeff Jirsa  wrote:
>>>
>>>
>>>
>>> On Mon, May 15, 2017 at 5:25 PM, Jeremiah D Jordan
>>>  wrote:



 To me testable means that we can run the tests at the very least for
 every release, but ideally they would be run more often than that.
 Especially with the push to not release unless the test board is all
 passing, we should not be releasing features that we don’t have a test 
 board
 for.  Ideally that means we have it in ASF CI.  If there is someone that 
 can
 commit to posting results of runs from an outside CI somewhere, then I 
 think
 that could work as well, but that gets pretty cumbersome if we have to 
 check
 10 different CI dashboards at different locations before every release.
>>>
>>>
>>>
>>> It turns out there's a ppc64le jenkins slave @ asf, so I've setup
>>> https://builds.apache.org/view/A-D/view/Cassandra/job/cassandra-devbranch-ppc64le-testall/
>>> for testing.
>>>
>>> Like our other devbranch-testall builds, it takes a repo+branch as
>>> parameters, and runs unit tests. While the unit tests aren't passing, this
>>> platform should now be considered testable.
>>>
>>
>> (Platform != device, though, the CAPI device obviously isn't there, so the
>> row cache implementation still doesn't have public testing)
>>
>>

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Integrating vendor-specific code and developing plugins

[大平怜]

Hi all,

Everybody seems to agree with improving the plugin ecosystem (as well
as not small amount of effort needed to do that), but about
vendor-specific code integration, let me summarize the issues raised
so far.

1) How to test it?  What if my code breaks the vendor-specific build?
2) How to maintain it?  Who is to maintain the code?
3) How does it affect the Cassandra release cycle?
4) How to remove it?  It might be hard to remove once integrated, from
both technical and markting perspective.

I think #3 and #4 are rather general issues for any newly proposed
changes, while #1 and #2 are the most problematic for niche :-)
platform specific code.  #1 is technically solvable, for example, as
Jeff (thanks!) showed with the Jenkins slave at ASF and as we are
trying to connect a ppc machine with a CAPI device to the CI.

#2 must be socially solved, as a component/platform maintainer system
should be introduced like some other Apache projects.  Is there any
chance to have such a system in Cassandra?


Thanks,
Rei Odaira

2017-05-18 12:36 GMT-05:00 Jeff Jirsa :
>
>
> On Thu, May 18, 2017 at 10:28 AM, Jeff Jirsa  wrote:
>>
>>
>>
>> On Mon, May 15, 2017 at 5:25 PM, Jeremiah D Jordan
>>  wrote:
>>>
>>>
>>>
>>> To me testable means that we can run the tests at the very least for
>>> every release, but ideally they would be run more often than that.
>>> Especially with the push to not release unless the test board is all
>>> passing, we should not be releasing features that we don’t have a test board
>>> for.  Ideally that means we have it in ASF CI.  If there is someone that can
>>> commit to posting results of runs from an outside CI somewhere, then I think
>>> that could work as well, but that gets pretty cumbersome if we have to check
>>> 10 different CI dashboards at different locations before every release.
>>
>>
>>
>> It turns out there's a ppc64le jenkins slave @ asf, so I've setup
>> https://builds.apache.org/view/A-D/view/Cassandra/job/cassandra-devbranch-ppc64le-testall/
>> for testing.
>>
>> Like our other devbranch-testall builds, it takes a repo+branch as
>> parameters, and runs unit tests. While the unit tests aren't passing, this
>> platform should now be considered testable.
>>
>
> (Platform != device, though, the CAPI device obviously isn't there, so the
> row cache implementation still doesn't have public testing)
>
>

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Integrating vendor-specific code and developing plugins

[Jeff Jirsa]

On Thu, May 18, 2017 at 10:28 AM, Jeff Jirsa  wrote:

>
>
> On Mon, May 15, 2017 at 5:25 PM, Jeremiah D Jordan <
> jeremiah.jor...@gmail.com> wrote:
>
>>
>>
>> To me testable means that we can run the tests at the very least for
>> every release, but ideally they would be run more often than that.
>> Especially with the push to not release unless the test board is all
>> passing, we should not be releasing features that we don’t have a test
>> board for.  Ideally that means we have it in ASF CI.  If there is someone
>> that can commit to posting results of runs from an outside CI somewhere,
>> then I think that could work as well, but that gets pretty cumbersome if we
>> have to check 10 different CI dashboards at different locations before
>> every release.
>>
>
>
> It turns out there's a ppc64le jenkins slave @ asf, so I've setup
> https://builds.apache.org/view/A-D/view/Cassandra/job/cassandra-devbranch-
> ppc64le-testall/ for testing.
>
> Like our other devbranch-testall builds, it takes a repo+branch as
> parameters, and runs unit tests. While the unit tests aren't passing, this
> platform should now be considered testable.
>
>
(Platform != device, though, the CAPI device obviously isn't there, so the
row cache implementation still doesn't have public testing)

Re: Integrating vendor-specific code and developing plugins

[Michael Kjellman]

That’s epic Jeff. Very cool.

Sent from my iPhone

On May 18, 2017, at 10:28 AM, Jeff Jirsa 
> wrote:

On Mon, May 15, 2017 at 5:25 PM, Jeremiah D Jordan <
jeremiah.jor...@gmail.com> wrote:



To me testable means that we can run the tests at the very least for every
release, but ideally they would be run more often than that.  Especially
with the push to not release unless the test board is all passing, we
should not be releasing features that we don’t have a test board for.
Ideally that means we have it in ASF CI.  If there is someone that can
commit to posting results of runs from an outside CI somewhere, then I
think that could work as well, but that gets pretty cumbersome if we have
to check 10 different CI dashboards at different locations before every
release.



It turns out there's a ppc64le jenkins slave @ asf, so I've setup
https://builds.apache.org/view/A-D/view/Cassandra/job/cassandra-devbranch-ppc64le-testall/
for testing.

Like our other devbranch-testall builds, it takes a repo+branch as
parameters, and runs unit tests. While the unit tests aren't passing, this
platform should now be considered testable.

Re: Integrating vendor-specific code and developing plugins

[Jeff Jirsa]

On Mon, May 15, 2017 at 5:25 PM, Jeremiah D Jordan <
jeremiah.jor...@gmail.com> wrote:

>
>
> To me testable means that we can run the tests at the very least for every
> release, but ideally they would be run more often than that.  Especially
> with the push to not release unless the test board is all passing, we
> should not be releasing features that we don’t have a test board for.
> Ideally that means we have it in ASF CI.  If there is someone that can
> commit to posting results of runs from an outside CI somewhere, then I
> think that could work as well, but that gets pretty cumbersome if we have
> to check 10 different CI dashboards at different locations before every
> release.
>


It turns out there's a ppc64le jenkins slave @ asf, so I've setup
https://builds.apache.org/view/A-D/view/Cassandra/job/cassandra-devbranch-ppc64le-testall/
for testing.

Like our other devbranch-testall builds, it takes a repo+branch as
parameters, and runs unit tests. While the unit tests aren't passing, this
platform should now be considered testable.

Re: Integrating vendor-specific code and developing plugins

[Jeremiah D Jordan]

> On May 15, 2017, at 6:57 PM, 大平怜  wrote:
> 
> Thanks for the discussion, all,
> 
>>> * What's included when shipped in tree?
>>> 
>>> Does every idea get merged in? Do we need 30 different Seed providers?  Who
>>> judges what's valuable enough to add?  Who maintains it when it needs
>>> updating?  If the maintainer can't be found, is it removed?  Shipped
>>> broken?  Does the contributed plugins go through the same review process?
>>> Do the contributors need to be committers?  Would CASSANDRA-12627 be merged
>>> in even if nobody saw the value?
> 
> If the rule is to never merge a feature that is pluggable, then it
> would be easy to make a decision, but if not, then we must anyway ask
> ourselves these questions every time a feature is proposed, and I
> think most of the questions are not specific to plugins but generic
> for any new proposals.

Agreed.  Include or not is the same decision we make for any JIRA.

> 
> 
>> In accordance with the idea that the codebase should be better tested, it
>> seems to me like things shouldn't be added that aren't testable.  If
>> there's a million unit tests that are insanely comprehensive but for some
>> reason can never be run, they serve exactly the same value as no tests.
> 
> I think we need to define what the "testable" means.  Does it mean at
> least one of the committers has full access to an environment where
> she can run the tests?  Also, does it mean the environment is
> integrated to the CI?

To me testable means that we can run the tests at the very least for every 
release, but ideally they would be run more often than that.  Especially with 
the push to not release unless the test board is all passing, we should not be 
releasing features that we don’t have a test board for.  Ideally that means we 
have it in ASF CI.  If there is someone that can commit to posting results of 
runs from an outside CI somewhere, then I think that could work as well, but 
that gets pretty cumbersome if we have to check 10 different CI dashboards at 
different locations before every release.

-Jeremiah
-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Integrating vendor-specific code and developing plugins

[大平怜]

Thanks for the discussion, all,

>> * What's included when shipped in tree?
>>
>> Does every idea get merged in? Do we need 30 different Seed providers?  Who
>> judges what's valuable enough to add?  Who maintains it when it needs
>> updating?  If the maintainer can't be found, is it removed?  Shipped
>> broken?  Does the contributed plugins go through the same review process?
>> Do the contributors need to be committers?  Would CASSANDRA-12627 be merged
>> in even if nobody saw the value?

If the rule is to never merge a feature that is pluggable, then it
would be easy to make a decision, but if not, then we must anyway ask
ourselves these questions every time a feature is proposed, and I
think most of the questions are not specific to plugins but generic
for any new proposals.


> In accordance with the idea that the codebase should be better tested, it
> seems to me like things shouldn't be added that aren't testable.  If
> there's a million unit tests that are insanely comprehensive but for some
> reason can never be run, they serve exactly the same value as no tests.

I think we need to define what the "testable" means.  Does it mean at
least one of the committers has full access to an environment where
she can run the tests?  Also, does it mean the environment is
integrated to the CI?


Thanks,
Rei Odaira

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Integrating vendor-specific code and developing plugins

[benjamin roth]

Absolutely

+ Separate repos for separate modules also separate responsibility. IMHO it
makes a heterogenuous structure more manageable. Both in a technical and a
human or insitutional way.

2017-05-15 13:54 GMT+02:00 Jonathan Haddad :

> There's a handful of issues I can think of with shipping everything
> in-tree.  I'll try to brain dump them here.
>
> * What's included when shipped in tree?
>
> Does every idea get merged in? Do we need 30 different Seed providers?  Who
> judges what's valuable enough to add?  Who maintains it when it needs
> updating?  If the maintainer can't be found, is it removed?  Shipped
> broken?  Does the contributed plugins go through the same review process?
> Do the contributors need to be committers?  Would CASSANDRA-12627 be merged
> in even if nobody saw the value?
>
> * Language support
>
> Cassandra is based on Java 8.  Do we now merge in Scala, Kotlin, Jython?
>
> * Plugins are now tied directly to cassandra release cycle
>
> This one bugs me quite a bit.  With a new plugin, there's going to be a lot
> of rapid iterations.  Spark releases once every 3 months - a lot of the
> packages seem to be released at a much higher frequency.
>
> * In Python, the standard lib is where modules go to die
>
> I forgot where I heard this, but it's pretty accurate.  Including
> everything, "batteries includes", just ends up shipping some pretty
> terrible batteries.  The best stuff for python is on pypi.
>
> Rust deliberately made the decision to limit the std to avoid this
> problem.  There's a "nursery" [1] area for ideas to evolve independently,
> and when some code reaches a high enough level of maturity, it can get
> merged in.  There's also a packages system for third party, non std
> destined code.
>
> Anyways - I'm very +1 on a package system where codebases can independently
> evolve without needing to be part of the project itself.  It's a proven
> model for shipping high quality, useful code, and sometimes is even one of
> the best aspects of a project.  That said, it's quite a bit of work just to
> get going and someone would have to manage that.
>
> Jon
>
> [1] https://github.com/rust-lang-nursery
>
>
> On Sun, May 14, 2017 at 9:03 PM Jeff Jirsa  wrote:
>
> > On Fri, May 12, 2017 at 9:31 PM, J. D. Jordan  >
> > wrote:
> >
> > > In tree would foster more committers which is a good thing.
> >
> >
> > Definitely.
> >
> > But I also agree that not being able to actually run unit tests is a bad
> > > thing. What if we asked people who want to contribute these types of
> > > optimizations to provide the ASF with a Jenkins slave we could test
> them
> > > on, if they want them in tree?
> > >
> >
> > I think SOME FORM of jenkins/unit/dtests need to exist, whether it's ASF
> > puppet controlled or test output explicitly provided by the maintainer.
> >
> >
> > > Otherwise one good thing about out of tree is that the maintainer can
> > > specify "this plugin has been tested and works with Apache Cassandra
> > > version X.Y.Z". If it is in tree it is assumed it will work. If it is
> out
> > > of tree then the plugin can more easily notify a user what version it
> was
> > > last tested with.  And users won't be surprised if they upgrade and
> > things
> > > break.
> > >
> >
> > My main desire is that I'd like to see us do better at helping third
> party
> > contributors be successful in contributing, and to me that means
> something
> > more official. I like the spark packages model. I like the apache httpd
> > model (with LOTS of official extensions in-tree, but a lot externally
> > packaged as well). I'm not a fan of telling people to build and
> distribute
> > their own JARs - it doesn't help the contributors, it doesn't help the
> > users, and it doesn't help the project.
> >
> > - Jeff
> >
>

Re: Integrating vendor-specific code and developing plugins

[Vincent Royer]

Hi all,

I have suggested some modifications in CASSANDRA-13270 to be able to provide 
elasticsearch as a plugin, and these changes are available in a fork at 
https://github.com/strapdata/cassandra 
Of course, i don’t expect to include this in the cassandra tree, but just to be 
able to provide this as an external Cassandra plugin. And even through plugins 
are out of the tree, i would enjoy to contribute to the C* code too !

This plugin is basically a custom secondary index, with hooks to be able to 
start and configure the plugin before playing the commit logs or bootstrap. 
This could probably be improved in a clear plugin API.
Moreover, in my specific case, the major issue is the locale-sensitive behavior 
of Cassandra, there is more than 800 string related functions call with the 
default Locale. So, unit tests with various Locale was not possible until this 
was fixed by a byte-code manipulation with javassit. As a workaround, i have 
develop an ANT task to fix this in the build.xml.
Hope this helps.

Thanks,
Vincent.

On 2017-05-13 04:21 (+0200), Nate McCall  wrote: 
> > This being said, I don't think we should remove support for Windows or> 
> > those snitches. Instead, what I think would be more beneficial, and> 
> > certainly more reflecting the Apache Way, is to see if someone in the> 
> > community would be willing to maintain those components. Looking at 
> > another> 
> > Apache project, Mesos has an interesting breakdown of maintainers for> 
> > specific components [1]. We might consider adopting a similar idea for> 
> > platforms/OSes/architectures/whatevers.> 
> >> 
> 
> This is a good point. Apache Beam does this as well: component> 
> maintainers for specific integrations (Spark, Kafka, HDFS, etc).> 
> Agreed that this is a good way to encourage participation.> 
> 
> -> 
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org> 
> For additional commands, e-mail: dev-h...@cassandra.apache.org> 
> 
>  

Vincent Royer
CEO & Founder
Email: vro...@strapdata.com
Mobile: +33 7 83 18 25 40

Re: Integrating vendor-specific code and developing plugins

[Jonathan Haddad]

There's a handful of issues I can think of with shipping everything
in-tree.  I'll try to brain dump them here.

* What's included when shipped in tree?

Does every idea get merged in? Do we need 30 different Seed providers?  Who
judges what's valuable enough to add?  Who maintains it when it needs
updating?  If the maintainer can't be found, is it removed?  Shipped
broken?  Does the contributed plugins go through the same review process?
Do the contributors need to be committers?  Would CASSANDRA-12627 be merged
in even if nobody saw the value?

* Language support

Cassandra is based on Java 8.  Do we now merge in Scala, Kotlin, Jython?

* Plugins are now tied directly to cassandra release cycle

This one bugs me quite a bit.  With a new plugin, there's going to be a lot
of rapid iterations.  Spark releases once every 3 months - a lot of the
packages seem to be released at a much higher frequency.

* In Python, the standard lib is where modules go to die

I forgot where I heard this, but it's pretty accurate.  Including
everything, "batteries includes", just ends up shipping some pretty
terrible batteries.  The best stuff for python is on pypi.

Rust deliberately made the decision to limit the std to avoid this
problem.  There's a "nursery" [1] area for ideas to evolve independently,
and when some code reaches a high enough level of maturity, it can get
merged in.  There's also a packages system for third party, non std
destined code.

Anyways - I'm very +1 on a package system where codebases can independently
evolve without needing to be part of the project itself.  It's a proven
model for shipping high quality, useful code, and sometimes is even one of
the best aspects of a project.  That said, it's quite a bit of work just to
get going and someone would have to manage that.

Jon

[1] https://github.com/rust-lang-nursery


On Sun, May 14, 2017 at 9:03 PM Jeff Jirsa  wrote:

> On Fri, May 12, 2017 at 9:31 PM, J. D. Jordan 
> wrote:
>
> > In tree would foster more committers which is a good thing.
>
>
> Definitely.
>
> But I also agree that not being able to actually run unit tests is a bad
> > thing. What if we asked people who want to contribute these types of
> > optimizations to provide the ASF with a Jenkins slave we could test them
> > on, if they want them in tree?
> >
>
> I think SOME FORM of jenkins/unit/dtests need to exist, whether it's ASF
> puppet controlled or test output explicitly provided by the maintainer.
>
>
> > Otherwise one good thing about out of tree is that the maintainer can
> > specify "this plugin has been tested and works with Apache Cassandra
> > version X.Y.Z". If it is in tree it is assumed it will work. If it is out
> > of tree then the plugin can more easily notify a user what version it was
> > last tested with.  And users won't be surprised if they upgrade and
> things
> > break.
> >
>
> My main desire is that I'd like to see us do better at helping third party
> contributors be successful in contributing, and to me that means something
> more official. I like the spark packages model. I like the apache httpd
> model (with LOTS of official extensions in-tree, but a lot externally
> packaged as well). I'm not a fan of telling people to build and distribute
> their own JARs - it doesn't help the contributors, it doesn't help the
> users, and it doesn't help the project.
>
> - Jeff
>

Re: Integrating vendor-specific code and developing plugins

[Jeff Jirsa]

On Fri, May 12, 2017 at 9:31 PM, J. D. Jordan 
wrote:

> In tree would foster more committers which is a good thing.


Definitely.

But I also agree that not being able to actually run unit tests is a bad
> thing. What if we asked people who want to contribute these types of
> optimizations to provide the ASF with a Jenkins slave we could test them
> on, if they want them in tree?
>

I think SOME FORM of jenkins/unit/dtests need to exist, whether it's ASF
puppet controlled or test output explicitly provided by the maintainer.


> Otherwise one good thing about out of tree is that the maintainer can
> specify "this plugin has been tested and works with Apache Cassandra
> version X.Y.Z". If it is in tree it is assumed it will work. If it is out
> of tree then the plugin can more easily notify a user what version it was
> last tested with.  And users won't be surprised if they upgrade and things
> break.
>

My main desire is that I'd like to see us do better at helping third party
contributors be successful in contributing, and to me that means something
more official. I like the spark packages model. I like the apache httpd
model (with LOTS of official extensions in-tree, but a lot externally
packaged as well). I'm not a fan of telling people to build and distribute
their own JARs - it doesn't help the contributors, it doesn't help the
users, and it doesn't help the project.

- Jeff

Re: Integrating vendor-specific code and developing plugins

[Nate McCall]

> It may be better to figure out how to foster a plugin ecosystem, which is a
> bit better than "there's an API go deal with it".  This is what Spark is
> doing and it seems like a pretty reasonable approach to me:
> https://spark-packages.org/
>

In thinking about this a bit, we have: Mesos, Beam and Spark as
examples of other Apache projects managing "plugins" (maybe a better
word around?).

Anybody have other examples that come to mind in ASF ecosystem?

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Integrating vendor-specific code and developing plugins

[Jonathan Haddad]

In accordance with the idea that the codebase should be better tested, it
seems to me like things shouldn't be added that aren't testable.  If
there's a million unit tests that are insanely comprehensive but for some
reason can never be run, they serve exactly the same value as no tests.

It may be better to figure out how to foster a plugin ecosystem, which is a
bit better than "there's an API go deal with it".  This is what Spark is
doing and it seems like a pretty reasonable approach to me:
https://spark-packages.org/

On Fri, May 12, 2017 at 9:03 PM Jeff Jirsa  wrote:

> I think the status quo is insufficient - even if it doesn't go in tree, we
> should do more than just say "the API exists, ship your own jar"
>
> What's the real risk of having it in tree? We break it because nobody can
> test it? How's that any worse than breaking it outside the tree? Finger
> pointing?
>
> --
> Jeff Jirsa
>
>
> > On May 12, 2017, at 12:25 PM, Jason Brown  wrote:
> >
> > I agree the plugins route is the safest and best. However, we already
> have
> > platform-specific code in-tree that is semi-unmaintained: the Windows
> > support. To Sylvain's point, I have little to no idea if I'm going to
> break
> > the Windows builds as I don't have access to a Windows machine, nor are
> we
> > as a community (as best as I can tell) actively running unit tests or
> > dtests on Windows.
> >
> > Further, we support snitches for clouds I suspect we don't typically
> > run/test on, as well: CloudstackSnitch, GoogleCloudSnitch.
> >
> > This being said, I don't think we should remove support for Windows or
> > those snitches. Instead, what I think would be more beneficial, and
> > certainly more reflecting the Apache Way, is to see if someone in the
> > community would be willing to maintain those components. Looking at
> another
> > Apache project, Mesos has an interesting breakdown of maintainers for
> > specific components [1]. We might consider adopting a similar idea for
> > platforms/OSes/architectures/whatevers.
> >
> > As for where to put the custom code, there's a few different options:
> >
> > bare minimum: we should have docs pointing to all known third party
> > implementations of pluggable interfaces
> > slightly more involved: contrib/ section of third-party contributed
> plugins
> > even more involved: in tree like gcp / aws snitches
> >
> > I'm not really thrilled on the contribs repo, and in-tree certainly has
> > drawbacks, as well. As I initially stated, it can be on a case-by-case
> > basis.
> >
> > Honestly, I don't want to push away contributors if they can add
> something
> > to the project - as long as it is maintainable.
> >
> > Thanks,
> >
> > -Jason
> >
> >
> > [1] https://mesos.apache.org/documentation/latest/committers/
> >
> > On Fri, May 12, 2017 at 4:35 AM, Sylvain Lebresne 
> > wrote:
> >
> >> On Fri, May 12, 2017 at 12:29 AM, Jason Brown 
> >> wrote:
> >>
> >>> Hey all,
> >>>
> >>> I'm on-board with what Rei is saying. I think we should be open to, and
> >>> encourage, other platforms/architectures for integration. Of course, it
> >>> will come down to specific maintainers/committers to do the testing and
> >>> verification on non-typical platforms. Hopefully those maintainers will
> >>> also contribute to other parts of the code base, as well, so I see
> this as
> >>> another way to bring more folks into the project.
> >>>
> >>
> >> Without going so far as to say we shouldn't merge any
> >> platform/architecture/vendor specific code ever and for no reason, I
> >> personally think we should avoid doing so as much as practical and
> >> encourage the "plugins" route instead. It's just much cleaner imo on
> >> principle and amounts to good software development hygiene.
> >>
> >> I don't want to not be able to commit some changes because it breaks the
> >> build because there is code for X number of super specific
> >> platform/architecture I don't have access to/don't know anything about
> >> and the maintainers are on vacation or hasn't been reachable in a while.
> >> And what if such maintainer do go away? Sure we can have some "process"
> >> to remove the code in such cases, but why add that burden on us? Plus
> >> we, the Apache Cassandra project, would still be seen as the ones that
> >> drop support for said platform/architecture even though we really have
> >> no choice if it's something we don't have access to anyway.
> >>
> >> And sure, I'm painting a bleak picture here, and we would probably have
> >> none of those problems in most cases. But if we do start encourage
> >> actual merge of such code, you can be sure we'll have some of those
> >> problems at some point.
> >>
> >> Encouraging plugins have imo pretty much all the benefits with none of
> >> the risks. In particular, I'm unconvinced that someone will be
> >> much more likely to meaningfully contribute to other part of the code
> >> if his "plugins" is 

Re: Integrating vendor-specific code and developing plugins

[J. D. Jordan]

In tree would foster more committers which is a good thing. But I also agree 
that not being able to actually run unit tests is a bad thing. What if we asked 
people who want to contribute these types of optimizations to provide the ASF 
with a Jenkins slave we could test them on, if they want them in tree?

Otherwise one good thing about out of tree is that the maintainer can specify 
"this plugin has been tested and works with Apache Cassandra version X.Y.Z". If 
it is in tree it is assumed it will work. If it is out of tree then the plugin 
can more easily notify a user what version it was last tested with.  And users 
won't be surprised if they upgrade and things break.

> On May 12, 2017, at 9:02 PM, Jeff Jirsa  wrote:
> 
> What's the real risk of having it in tree? We break it because nobody can 
> test it? How's that any worse than breaking it outside the tree? Finger 
> pointing?

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Integrating vendor-specific code and developing plugins

[Nate McCall]

> This being said, I don't think we should remove support for Windows or
> those snitches. Instead, what I think would be more beneficial, and
> certainly more reflecting the Apache Way, is to see if someone in the
> community would be willing to maintain those components. Looking at another
> Apache project, Mesos has an interesting breakdown of maintainers for
> specific components [1]. We might consider adopting a similar idea for
> platforms/OSes/architectures/whatevers.
>

This is a good point. Apache Beam does this as well: component
maintainers for specific integrations (Spark, Kafka, HDFS, etc).
Agreed that this is a good way to encourage participation.

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Integrating vendor-specific code and developing plugins

[Jeff Jirsa]

I think the status quo is insufficient - even if it doesn't go in tree, we 
should do more than just say "the API exists, ship your own jar"

What's the real risk of having it in tree? We break it because nobody can test 
it? How's that any worse than breaking it outside the tree? Finger pointing? 

-- 
Jeff Jirsa


> On May 12, 2017, at 12:25 PM, Jason Brown  wrote:
> 
> I agree the plugins route is the safest and best. However, we already have
> platform-specific code in-tree that is semi-unmaintained: the Windows
> support. To Sylvain's point, I have little to no idea if I'm going to break
> the Windows builds as I don't have access to a Windows machine, nor are we
> as a community (as best as I can tell) actively running unit tests or
> dtests on Windows.
> 
> Further, we support snitches for clouds I suspect we don't typically
> run/test on, as well: CloudstackSnitch, GoogleCloudSnitch.
> 
> This being said, I don't think we should remove support for Windows or
> those snitches. Instead, what I think would be more beneficial, and
> certainly more reflecting the Apache Way, is to see if someone in the
> community would be willing to maintain those components. Looking at another
> Apache project, Mesos has an interesting breakdown of maintainers for
> specific components [1]. We might consider adopting a similar idea for
> platforms/OSes/architectures/whatevers.
> 
> As for where to put the custom code, there's a few different options:
> 
> bare minimum: we should have docs pointing to all known third party
> implementations of pluggable interfaces
> slightly more involved: contrib/ section of third-party contributed plugins
> even more involved: in tree like gcp / aws snitches
> 
> I'm not really thrilled on the contribs repo, and in-tree certainly has
> drawbacks, as well. As I initially stated, it can be on a case-by-case
> basis.
> 
> Honestly, I don't want to push away contributors if they can add something
> to the project - as long as it is maintainable.
> 
> Thanks,
> 
> -Jason
> 
> 
> [1] https://mesos.apache.org/documentation/latest/committers/
> 
> On Fri, May 12, 2017 at 4:35 AM, Sylvain Lebresne 
> wrote:
> 
>> On Fri, May 12, 2017 at 12:29 AM, Jason Brown 
>> wrote:
>> 
>>> Hey all,
>>> 
>>> I'm on-board with what Rei is saying. I think we should be open to, and
>>> encourage, other platforms/architectures for integration. Of course, it
>>> will come down to specific maintainers/committers to do the testing and
>>> verification on non-typical platforms. Hopefully those maintainers will
>>> also contribute to other parts of the code base, as well, so I see this as
>>> another way to bring more folks into the project.
>>> 
>> 
>> Without going so far as to say we shouldn't merge any
>> platform/architecture/vendor specific code ever and for no reason, I
>> personally think we should avoid doing so as much as practical and
>> encourage the "plugins" route instead. It's just much cleaner imo on
>> principle and amounts to good software development hygiene.
>> 
>> I don't want to not be able to commit some changes because it breaks the
>> build because there is code for X number of super specific
>> platform/architecture I don't have access to/don't know anything about
>> and the maintainers are on vacation or hasn't been reachable in a while.
>> And what if such maintainer do go away? Sure we can have some "process"
>> to remove the code in such cases, but why add that burden on us? Plus
>> we, the Apache Cassandra project, would still be seen as the ones that
>> drop support for said platform/architecture even though we really have
>> no choice if it's something we don't have access to anyway.
>> 
>> And sure, I'm painting a bleak picture here, and we would probably have
>> none of those problems in most cases. But if we do start encourage
>> actual merge of such code, you can be sure we'll have some of those
>> problems at some point.
>> 
>> Encouraging plugins have imo pretty much all the benefits with none of
>> the risks. In particular, I'm unconvinced that someone will be
>> much more likely to meaningfully contribute to other part of the code
>> if his "plugins" is in-tree versus out of it.
>> 
>> *But* I can certainly agree with the part about us not having done a
>> good job to promote plugins and it make sense to me to try to improve on
>> that.
>> 
>> 
>>> 
>>> WRT extensibility, it just requires someone to do the work of making
>>> reasonable abstraction points - and documenting them ;). The interesting
>>> question comes down to how to host/ship any pluggable dependencies. Much
>>> like what we had with jna before they relicensed, we'll probably ship some
>>> things in-tree and some things users will have to fetch and deploy
>>> themselves; it's a case-by-case basis.
>>> 
>>> Thanks,
>>> 
>>> -Jason
>>> 
>>> 
 On Thu, May 11, 2017 at 2:59 PM, 大平怜  wrote:
 
 Hi all,
 
 In 

Re: Integrating vendor-specific code and developing plugins

[Jason Brown]

I agree the plugins route is the safest and best. However, we already have
platform-specific code in-tree that is semi-unmaintained: the Windows
support. To Sylvain's point, I have little to no idea if I'm going to break
the Windows builds as I don't have access to a Windows machine, nor are we
as a community (as best as I can tell) actively running unit tests or
dtests on Windows.

Further, we support snitches for clouds I suspect we don't typically
run/test on, as well: CloudstackSnitch, GoogleCloudSnitch.

This being said, I don't think we should remove support for Windows or
those snitches. Instead, what I think would be more beneficial, and
certainly more reflecting the Apache Way, is to see if someone in the
community would be willing to maintain those components. Looking at another
Apache project, Mesos has an interesting breakdown of maintainers for
specific components [1]. We might consider adopting a similar idea for
platforms/OSes/architectures/whatevers.

As for where to put the custom code, there's a few different options:

bare minimum: we should have docs pointing to all known third party
implementations of pluggable interfaces
slightly more involved: contrib/ section of third-party contributed plugins
even more involved: in tree like gcp / aws snitches

I'm not really thrilled on the contribs repo, and in-tree certainly has
drawbacks, as well. As I initially stated, it can be on a case-by-case
basis.

Honestly, I don't want to push away contributors if they can add something
to the project - as long as it is maintainable.

Thanks,

-Jason


[1] https://mesos.apache.org/documentation/latest/committers/

On Fri, May 12, 2017 at 4:35 AM, Sylvain Lebresne 
wrote:

> On Fri, May 12, 2017 at 12:29 AM, Jason Brown 
> wrote:
>
>> Hey all,
>>
>> I'm on-board with what Rei is saying. I think we should be open to, and
>> encourage, other platforms/architectures for integration. Of course, it
>> will come down to specific maintainers/committers to do the testing and
>> verification on non-typical platforms. Hopefully those maintainers will
>> also contribute to other parts of the code base, as well, so I see this as
>> another way to bring more folks into the project.
>>
>
> Without going so far as to say we shouldn't merge any
> platform/architecture/vendor specific code ever and for no reason, I
> personally think we should avoid doing so as much as practical and
> encourage the "plugins" route instead. It's just much cleaner imo on
> principle and amounts to good software development hygiene.
>
> I don't want to not be able to commit some changes because it breaks the
> build because there is code for X number of super specific
> platform/architecture I don't have access to/don't know anything about
> and the maintainers are on vacation or hasn't been reachable in a while.
> And what if such maintainer do go away? Sure we can have some "process"
> to remove the code in such cases, but why add that burden on us? Plus
> we, the Apache Cassandra project, would still be seen as the ones that
> drop support for said platform/architecture even though we really have
> no choice if it's something we don't have access to anyway.
>
> And sure, I'm painting a bleak picture here, and we would probably have
> none of those problems in most cases. But if we do start encourage
> actual merge of such code, you can be sure we'll have some of those
> problems at some point.
>
> Encouraging plugins have imo pretty much all the benefits with none of
> the risks. In particular, I'm unconvinced that someone will be
> much more likely to meaningfully contribute to other part of the code
> if his "plugins" is in-tree versus out of it.
>
> *But* I can certainly agree with the part about us not having done a
> good job to promote plugins and it make sense to me to try to improve on
> that.
>
>
>>
>> WRT extensibility, it just requires someone to do the work of making
>> reasonable abstraction points - and documenting them ;). The interesting
>> question comes down to how to host/ship any pluggable dependencies. Much
>> like what we had with jna before they relicensed, we'll probably ship some
>> things in-tree and some things users will have to fetch and deploy
>> themselves; it's a case-by-case basis.
>>
>> Thanks,
>>
>> -Jason
>>
>>
>> On Thu, May 11, 2017 at 2:59 PM, 大平怜  wrote:
>>
>> > Hi all,
>> >
>> > In this JIRA ticket https://issues.apache.org/jira
>> /browse/CASSANDRA-13486,
>> > we proposed integrating our code to support a fast flash+FPGA card
>> (called
>> > CAPI Flash) only available in the ppc architecture. Although we will
>> keep
>> > discussing the topics specific to the patch (e.g. documentation,
>> license,
>> > code quality) in the JIRA, we would like to start in this dev list more
>> > general discussion about how to (and how not to) merge
>> > architecture-specific (or vendor-specific) changes.
>> >
>> > I think in the end the problem 

Re: Integrating vendor-specific code and developing plugins

[Aleksey Yeschenko]

I’m with Sylvain on this for essentially same reasons.

If we need to improve our extensibility here, we should do that, and then have 
niche platform specific code
be an external plug-in (and add a link to the plug-in to our docs, to promote 
it).

-- 
AY

On 12 May 2017 at 12:36:33, Sylvain Lebresne (sylv...@datastax.com) wrote:

On Fri, May 12, 2017 at 12:29 AM, Jason Brown  wrote:  

> Hey all,  
>  
> I'm on-board with what Rei is saying. I think we should be open to, and  
> encourage, other platforms/architectures for integration. Of course, it  
> will come down to specific maintainers/committers to do the testing and  
> verification on non-typical platforms. Hopefully those maintainers will  
> also contribute to other parts of the code base, as well, so I see this as  
> another way to bring more folks into the project.  
>  

Without going so far as to say we shouldn't merge any  
platform/architecture/vendor specific code ever and for no reason, I  
personally think we should avoid doing so as much as practical and  
encourage the "plugins" route instead. It's just much cleaner imo on  
principle and amounts to good software development hygiene.  

I don't want to not be able to commit some changes because it breaks the  
build because there is code for X number of super specific  
platform/architecture I don't have access to/don't know anything about  
and the maintainers are on vacation or hasn't been reachable in a while.  
And what if such maintainer do go away? Sure we can have some "process"  
to remove the code in such cases, but why add that burden on us? Plus  
we, the Apache Cassandra project, would still be seen as the ones that  
drop support for said platform/architecture even though we really have  
no choice if it's something we don't have access to anyway.  

And sure, I'm painting a bleak picture here, and we would probably have  
none of those problems in most cases. But if we do start encourage  
actual merge of such code, you can be sure we'll have some of those  
problems at some point.  

Encouraging plugins have imo pretty much all the benefits with none of  
the risks. In particular, I'm unconvinced that someone will be  
much more likely to meaningfully contribute to other part of the code  
if his "plugins" is in-tree versus out of it.  

*But* I can certainly agree with the part about us not having done a  
good job to promote plugins and it make sense to me to try to improve on  
that.  


>  
> WRT extensibility, it just requires someone to do the work of making  
> reasonable abstraction points - and documenting them ;). The interesting  
> question comes down to how to host/ship any pluggable dependencies. Much  
> like what we had with jna before they relicensed, we'll probably ship some  
> things in-tree and some things users will have to fetch and deploy  
> themselves; it's a case-by-case basis.  
>  
> Thanks,  
>  
> -Jason  
>  
>  
> On Thu, May 11, 2017 at 2:59 PM, 大平怜  wrote:  
>  
> > Hi all,  
> >  
> > In this JIRA ticket https://issues.apache.org/  
> jira/browse/CASSANDRA-13486,  
> > we proposed integrating our code to support a fast flash+FPGA card  
> (called  
> > CAPI Flash) only available in the ppc architecture. Although we will keep  
> > discussing the topics specific to the patch (e.g. documentation, license,  
> > code quality) in the JIRA, we would like to start in this dev list more  
> > general discussion about how to (and how not to) merge  
> > architecture-specific (or vendor-specific) changes.  
> >  
> > I think in the end the problem boils down to how to test the  
> > architecture-specific code. The original contributors of the  
> > architecture-specific code can keep "supporting" the code in a sense that  
> > when a problem arises they can fix it and send a patch, but the  
> committers  
> > cannot test it anyway. Are there any other factors we must consider?  
> >  
> > Also, in this particular case, it is relatively easy to turn the code  
> > change into a plugin because it extended the already-pluggable  
> RowCache. I  
> > feel Cassandra has promoted the plugins not so much as other pluggable  
> > software have done like Eclipse, Apache HTTP server, fluentd, etc. For  
> > example, they have a list of plugins in their Web pages. I think if the  
> > community wants to encourage developers to maintain vendor-specific code  
> as  
> > plugins outside of the main source tree, a deeper commitment to the  
> plugin  
> > ecosystem would be appreciated.  
> >  
> > What do you think?  
> >  
> >  
> > Thanks,  
> > Rei Odaira  
> >  
>  

Re: Integrating vendor-specific code and developing plugins

[Sylvain Lebresne]

On Fri, May 12, 2017 at 12:29 AM, Jason Brown  wrote:

> Hey all,
>
> I'm on-board with what Rei is saying. I think we should be open to, and
> encourage, other platforms/architectures for integration. Of course, it
> will come down to specific maintainers/committers to do the testing and
> verification on non-typical platforms. Hopefully those maintainers will
> also contribute to other parts of the code base, as well, so I see this as
> another way to bring more folks into the project.
>

Without going so far as to say we shouldn't merge any
platform/architecture/vendor specific code ever and for no reason, I
personally think we should avoid doing so as much as practical and
encourage the "plugins" route instead. It's just much cleaner imo on
principle and amounts to good software development hygiene.

I don't want to not be able to commit some changes because it breaks the
build because there is code for X number of super specific
platform/architecture I don't have access to/don't know anything about
and the maintainers are on vacation or hasn't been reachable in a while.
And what if such maintainer do go away? Sure we can have some "process"
to remove the code in such cases, but why add that burden on us? Plus
we, the Apache Cassandra project, would still be seen as the ones that
drop support for said platform/architecture even though we really have
no choice if it's something we don't have access to anyway.

And sure, I'm painting a bleak picture here, and we would probably have
none of those problems in most cases. But if we do start encourage
actual merge of such code, you can be sure we'll have some of those
problems at some point.

Encouraging plugins have imo pretty much all the benefits with none of
the risks. In particular, I'm unconvinced that someone will be
much more likely to meaningfully contribute to other part of the code
if his "plugins" is in-tree versus out of it.

*But* I can certainly agree with the part about us not having done a
good job to promote plugins and it make sense to me to try to improve on
that.


>
> WRT extensibility, it just requires someone to do the work of making
> reasonable abstraction points - and documenting them ;). The interesting
> question comes down to how to host/ship any pluggable dependencies. Much
> like what we had with jna before they relicensed, we'll probably ship some
> things in-tree and some things users will have to fetch and deploy
> themselves; it's a case-by-case basis.
>
> Thanks,
>
> -Jason
>
>
> On Thu, May 11, 2017 at 2:59 PM, 大平怜  wrote:
>
> > Hi all,
> >
> > In this JIRA ticket https://issues.apache.org/
> jira/browse/CASSANDRA-13486,
> > we proposed integrating our code to support a fast flash+FPGA card
> (called
> > CAPI Flash) only available in the ppc architecture. Although we will keep
> > discussing the topics specific to the patch (e.g. documentation, license,
> > code quality) in the JIRA, we would like to start in this dev list more
> > general discussion about how to (and how not to) merge
> > architecture-specific (or vendor-specific) changes.
> >
> > I think in the end the problem boils down to how to test the
> > architecture-specific code. The original contributors of the
> > architecture-specific code can keep "supporting" the code in a sense that
> > when a problem arises they can fix it and send a patch, but the
> committers
> > cannot test it anyway.  Are there any other factors we must consider?
> >
> > Also, in this particular case, it is relatively easy to turn the code
> > change into a plugin because it extended the already-pluggable
> RowCache.  I
> > feel Cassandra has promoted the plugins not so much as other pluggable
> > software have done like Eclipse, Apache HTTP server, fluentd, etc.  For
> > example, they have a list of plugins in their Web pages.  I think if the
> > community wants to encourage developers to maintain vendor-specific code
> as
> > plugins outside of the main source tree, a deeper commitment to the
> plugin
> > ecosystem would be appreciated.
> >
> > What do you think?
> >
> >
> > Thanks,
> > Rei Odaira
> >
>

Re: Integrating vendor-specific code and developing plugins

[Jason Brown]

Hey all,

I'm on-board with what Rei is saying. I think we should be open to, and
encourage, other platforms/architectures for integration. Of course, it
will come down to specific maintainers/committers to do the testing and
verification on non-typical platforms. Hopefully those maintainers will
also contribute to other parts of the code base, as well, so I see this as
another way to bring more folks into the project.

WRT extensibility, it just requires someone to do the work of making
reasonable abstraction points - and documenting them ;). The interesting
question comes down to how to host/ship any pluggable dependencies. Much
like what we had with jna before they relicensed, we'll probably ship some
things in-tree and some things users will have to fetch and deploy
themselves; it's a case-by-case basis.

Thanks,

-Jason


On Thu, May 11, 2017 at 2:59 PM, 大平怜  wrote:

> Hi all,
>
> In this JIRA ticket https://issues.apache.org/jira/browse/CASSANDRA-13486,
> we proposed integrating our code to support a fast flash+FPGA card (called
> CAPI Flash) only available in the ppc architecture. Although we will keep
> discussing the topics specific to the patch (e.g. documentation, license,
> code quality) in the JIRA, we would like to start in this dev list more
> general discussion about how to (and how not to) merge
> architecture-specific (or vendor-specific) changes.
>
> I think in the end the problem boils down to how to test the
> architecture-specific code. The original contributors of the
> architecture-specific code can keep "supporting" the code in a sense that
> when a problem arises they can fix it and send a patch, but the committers
> cannot test it anyway.  Are there any other factors we must consider?
>
> Also, in this particular case, it is relatively easy to turn the code
> change into a plugin because it extended the already-pluggable RowCache.  I
> feel Cassandra has promoted the plugins not so much as other pluggable
> software have done like Eclipse, Apache HTTP server, fluentd, etc.  For
> example, they have a list of plugins in their Web pages.  I think if the
> community wants to encourage developers to maintain vendor-specific code as
> plugins outside of the main source tree, a deeper commitment to the plugin
> ecosystem would be appreciated.
>
> What do you think?
>
>
> Thanks,
> Rei Odaira
>

Integrating vendor-specific code and developing plugins

[大平怜]

Hi all,

In this JIRA ticket https://issues.apache.org/jira/browse/CASSANDRA-13486,
we proposed integrating our code to support a fast flash+FPGA card (called
CAPI Flash) only available in the ppc architecture. Although we will keep
discussing the topics specific to the patch (e.g. documentation, license,
code quality) in the JIRA, we would like to start in this dev list more
general discussion about how to (and how not to) merge
architecture-specific (or vendor-specific) changes.

I think in the end the problem boils down to how to test the
architecture-specific code. The original contributors of the
architecture-specific code can keep "supporting" the code in a sense that
when a problem arises they can fix it and send a patch, but the committers
cannot test it anyway.  Are there any other factors we must consider?

Also, in this particular case, it is relatively easy to turn the code
change into a plugin because it extended the already-pluggable RowCache.  I
feel Cassandra has promoted the plugins not so much as other pluggable
software have done like Eclipse, Apache HTTP server, fluentd, etc.  For
example, they have a list of plugins in their Web pages.  I think if the
community wants to encourage developers to maintain vendor-specific code as
plugins outside of the main source tree, a deeper commitment to the plugin
ecosystem would be appreciated.

What do you think?


Thanks,
Rei Odaira