Re: Proposing an Apache Cassandra Management process

[Dinesh Joshi]

Thanks, Jason!

Dinesh

> On Dec 18, 2018, at 5:47 AM, Jason Brown  wrote:
> 
> It looks like we have consensus to move forward with the sidecar
> I will be shepherding the project, and to that end, I'll create a repo 
> against which we can start working.
> 
> Thanks,
> 
> -Jason
> 
>> On Fri, Nov 30, 2018 at 7:54 AM sankalp kohli  wrote:
>> If no one has more feedback, we should create JIRAs for all the subtasks
>> discussed in the proposal. I can see JIRA for Bulk command but not others.
>> 
>> On Mon, Nov 19, 2018 at 2:12 AM dinesh.jo...@yahoo.com.INVALID
>>  wrote:
>> 
>> > Thanks, Mick & Stefan for your inputs. What should we do as next steps to
>> > move this proposal forward?
>> > Dinesh
>> >
>> > On Sunday, November 18, 2018, 11:57:54 AM PST, Stefan Podkowinski <
>> > s...@apache.org> wrote:
>> >
>> >  My goal for a side car would be to enable more people to contribute to
>> > the project, by making it more accessible for anyone who’s not familiar
>> > with the Cassandra code base, or not familiar with Java development in
>> > general. Although most of the functionality described in the proposal
>> > sounds useful to have, I’d already be happy to have a solid REST API for
>> > the existing nodetool and JMX functionality. If an official side car,
>> > installed separately on each node, would provide that, I’m sure we’d see
>> > lots of new tools created by the community (web UIs, cli tools, ..)
>> > based on that. This would also be a good foundation for other existing
>> > tool to converge upon, e.g. by calling the REST APIs for repair
>> > scheduling and progress tracking instead of JMX, or by continually
>> > integrating and sharing useful helper calls. This would also give
>> > Cassandra devs more leeway to replace some of the existing tooling
>> > related code in Cassandra, e.g. by migrating to virtual tables, while at
>> > the same time keep providing a stable API through the side car.
>> >
>> > What I’d also like to point out here is that implementing such a project
>> > as an *official* side car, also implies to me having the same standards
>> > when it comes to release quality. I’d also really prefer having feature
>> > sets matching between Cassandra and the side car, e.g. authentication
>> > and SSL should also be supported in the side car from the beginning,
>> > ideally without any additional configuration.
>> >
>> >
>> > On 06.11.18 10:40, Dinesh Joshi wrote:
>> > > Hi all,
>> > >
>> > > Joey, Vinay & I have fleshed out the Management process proposal as the
>> > very first CIP document (with Jason’s inputs). It is available on the cwiki
>> > -
>> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652224
>> > >
>> > > Please comment on it and provide us any input that you may have. We want
>> > to ideally time-box the period to 2 weeks so we avoid waiting indefinitely.
>> > >
>> > > Thanks,
>> > >
>> > > Dinesh
>> > >
>> > >> On Oct 22, 2018, at 7:30 AM, "dinesh.jo...@yahoo.com.INVALID" <
>> > dinesh.jo...@yahoo.com.INVALID> wrote:
>> > >>
>> > >> Thanks for starting this, Mick. I will flesh it out.
>> > >> Dinesh
>> > >>
>> > >>On Sunday, October 21, 2018, 1:52:10 AM PDT, Mick Semb Wever <
>> > m...@apache.org> wrote:
>> > >>
>> > >>
>> > >>> But I'll try to put together a strawman proposal for the doc(s) over
>> > the
>> > >>> weekend.
>> > >>
>> > >> I've thrown something quickly together here:
>> > >> -
>> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652201
>> > >> -
>> > https://cwiki.apache.org/confluence/display/CASSANDRA/CIP-1%3A+Proposing+an+Apache+Cassandra+Management+process
>> > >>
>> > >> The former is a blatant rip-off from the Kafka and Spark design
>> > proposal pages that Dinesh previously mentioned. I'd hoped to do more of an
>> > analysis of the existing C* habits and precedence on design proposals
>> > (implicit in jira tickets), but in lei of that this is a strawman to start
>> > the discussion.
>> > >>
>> > >> The latter still needs to be fleshed out. Dinesh, can you do this? I
>> > can add a subpage/section that describes the alternative/consuming
>> > third-party tools out there.
>> > >>
>> > >> regards,
>> > >> Mick
>> > >>
>> > >> -
>> > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> > >> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> > >>
>> >
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> > For additional commands, e-mail: dev-h...@cassandra.apache.org
>> >
>> >

Re: Proposing an Apache Cassandra Management process

[Jason Brown]

It looks like we have consensus to move forward with the sidecar
I will be shepherding the project, and to that end, I'll create a repo
against which we can start working.

Thanks,

-Jason

On Fri, Nov 30, 2018 at 7:54 AM sankalp kohli 
wrote:

> If no one has more feedback, we should create JIRAs for all the subtasks
> discussed in the proposal. I can see JIRA for Bulk command but not others.
>
> On Mon, Nov 19, 2018 at 2:12 AM dinesh.jo...@yahoo.com.INVALID
>  wrote:
>
> > Thanks, Mick & Stefan for your inputs. What should we do as next steps to
> > move this proposal forward?
> > Dinesh
> >
> > On Sunday, November 18, 2018, 11:57:54 AM PST, Stefan Podkowinski <
> > s...@apache.org> wrote:
> >
> >  My goal for a side car would be to enable more people to contribute to
> > the project, by making it more accessible for anyone who’s not familiar
> > with the Cassandra code base, or not familiar with Java development in
> > general. Although most of the functionality described in the proposal
> > sounds useful to have, I’d already be happy to have a solid REST API for
> > the existing nodetool and JMX functionality. If an official side car,
> > installed separately on each node, would provide that, I’m sure we’d see
> > lots of new tools created by the community (web UIs, cli tools, ..)
> > based on that. This would also be a good foundation for other existing
> > tool to converge upon, e.g. by calling the REST APIs for repair
> > scheduling and progress tracking instead of JMX, or by continually
> > integrating and sharing useful helper calls. This would also give
> > Cassandra devs more leeway to replace some of the existing tooling
> > related code in Cassandra, e.g. by migrating to virtual tables, while at
> > the same time keep providing a stable API through the side car.
> >
> > What I’d also like to point out here is that implementing such a project
> > as an *official* side car, also implies to me having the same standards
> > when it comes to release quality. I’d also really prefer having feature
> > sets matching between Cassandra and the side car, e.g. authentication
> > and SSL should also be supported in the side car from the beginning,
> > ideally without any additional configuration.
> >
> >
> > On 06.11.18 10:40, Dinesh Joshi wrote:
> > > Hi all,
> > >
> > > Joey, Vinay & I have fleshed out the Management process proposal as the
> > very first CIP document (with Jason’s inputs). It is available on the
> cwiki
> > -
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652224
> > >
> > > Please comment on it and provide us any input that you may have. We
> want
> > to ideally time-box the period to 2 weeks so we avoid waiting
> indefinitely.
> > >
> > > Thanks,
> > >
> > > Dinesh
> > >
> > >> On Oct 22, 2018, at 7:30 AM, "dinesh.jo...@yahoo.com.INVALID" <
> > dinesh.jo...@yahoo.com.INVALID> wrote:
> > >>
> > >> Thanks for starting this, Mick. I will flesh it out.
> > >> Dinesh
> > >>
> > >>    On Sunday, October 21, 2018, 1:52:10 AM PDT, Mick Semb Wever <
> > m...@apache.org> wrote:
> > >>
> > >>
> > >>> But I'll try to put together a strawman proposal for the doc(s) over
> > the
> > >>> weekend.
> > >>
> > >> I've thrown something quickly together here:
> > >> -
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652201
> > >> -
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/CIP-1%3A+Proposing+an+Apache+Cassandra+Management+process
> > >>
> > >> The former is a blatant rip-off from the Kafka and Spark design
> > proposal pages that Dinesh previously mentioned. I'd hoped to do more of
> an
> > analysis of the existing C* habits and precedence on design proposals
> > (implicit in jira tickets), but in lei of that this is a strawman to
> start
> > the discussion.
> > >>
> > >> The latter still needs to be fleshed out. Dinesh, can you do this? I
> > can add a subpage/section that describes the alternative/consuming
> > third-party tools out there.
> > >>
> > >> regards,
> > >> Mick
> > >>
> > >> -
> > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >>
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>

Re: Proposing an Apache Cassandra Management process

[sankalp kohli]

If no one has more feedback, we should create JIRAs for all the subtasks
discussed in the proposal. I can see JIRA for Bulk command but not others.

On Mon, Nov 19, 2018 at 2:12 AM dinesh.jo...@yahoo.com.INVALID
 wrote:

> Thanks, Mick & Stefan for your inputs. What should we do as next steps to
> move this proposal forward?
> Dinesh
>
> On Sunday, November 18, 2018, 11:57:54 AM PST, Stefan Podkowinski <
> s...@apache.org> wrote:
>
>  My goal for a side car would be to enable more people to contribute to
> the project, by making it more accessible for anyone who’s not familiar
> with the Cassandra code base, or not familiar with Java development in
> general. Although most of the functionality described in the proposal
> sounds useful to have, I’d already be happy to have a solid REST API for
> the existing nodetool and JMX functionality. If an official side car,
> installed separately on each node, would provide that, I’m sure we’d see
> lots of new tools created by the community (web UIs, cli tools, ..)
> based on that. This would also be a good foundation for other existing
> tool to converge upon, e.g. by calling the REST APIs for repair
> scheduling and progress tracking instead of JMX, or by continually
> integrating and sharing useful helper calls. This would also give
> Cassandra devs more leeway to replace some of the existing tooling
> related code in Cassandra, e.g. by migrating to virtual tables, while at
> the same time keep providing a stable API through the side car.
>
> What I’d also like to point out here is that implementing such a project
> as an *official* side car, also implies to me having the same standards
> when it comes to release quality. I’d also really prefer having feature
> sets matching between Cassandra and the side car, e.g. authentication
> and SSL should also be supported in the side car from the beginning,
> ideally without any additional configuration.
>
>
> On 06.11.18 10:40, Dinesh Joshi wrote:
> > Hi all,
> >
> > Joey, Vinay & I have fleshed out the Management process proposal as the
> very first CIP document (with Jason’s inputs). It is available on the cwiki
> -
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652224
> >
> > Please comment on it and provide us any input that you may have. We want
> to ideally time-box the period to 2 weeks so we avoid waiting indefinitely.
> >
> > Thanks,
> >
> > Dinesh
> >
> >> On Oct 22, 2018, at 7:30 AM, "dinesh.jo...@yahoo.com.INVALID" <
> dinesh.jo...@yahoo.com.INVALID> wrote:
> >>
> >> Thanks for starting this, Mick. I will flesh it out.
> >> Dinesh
> >>
> >>On Sunday, October 21, 2018, 1:52:10 AM PDT, Mick Semb Wever <
> m...@apache.org> wrote:
> >>
> >>
> >>> But I'll try to put together a strawman proposal for the doc(s) over
> the
> >>> weekend.
> >>
> >> I've thrown something quickly together here:
> >> -
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652201
> >> -
> https://cwiki.apache.org/confluence/display/CASSANDRA/CIP-1%3A+Proposing+an+Apache+Cassandra+Management+process
> >>
> >> The former is a blatant rip-off from the Kafka and Spark design
> proposal pages that Dinesh previously mentioned. I'd hoped to do more of an
> analysis of the existing C* habits and precedence on design proposals
> (implicit in jira tickets), but in lei of that this is a strawman to start
> the discussion.
> >>
> >> The latter still needs to be fleshed out. Dinesh, can you do this? I
> can add a subpage/section that describes the alternative/consuming
> third-party tools out there.
> >>
> >> regards,
> >> Mick
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Proposing an Apache Cassandra Management process

[dinesh.jo...@yahoo.com.INVALID]

Thanks, Mick & Stefan for your inputs. What should we do as next steps to move 
this proposal forward?
Dinesh 

On Sunday, November 18, 2018, 11:57:54 AM PST, Stefan Podkowinski 
 wrote:  
 
 My goal for a side car would be to enable more people to contribute to
the project, by making it more accessible for anyone who’s not familiar
with the Cassandra code base, or not familiar with Java development in
general. Although most of the functionality described in the proposal
sounds useful to have, I’d already be happy to have a solid REST API for
the existing nodetool and JMX functionality. If an official side car,
installed separately on each node, would provide that, I’m sure we’d see
lots of new tools created by the community (web UIs, cli tools, ..)
based on that. This would also be a good foundation for other existing
tool to converge upon, e.g. by calling the REST APIs for repair
scheduling and progress tracking instead of JMX, or by continually
integrating and sharing useful helper calls. This would also give
Cassandra devs more leeway to replace some of the existing tooling
related code in Cassandra, e.g. by migrating to virtual tables, while at
the same time keep providing a stable API through the side car.

What I’d also like to point out here is that implementing such a project
as an *official* side car, also implies to me having the same standards
when it comes to release quality. I’d also really prefer having feature
sets matching between Cassandra and the side car, e.g. authentication
and SSL should also be supported in the side car from the beginning,
ideally without any additional configuration.


On 06.11.18 10:40, Dinesh Joshi wrote:
> Hi all,
>
> Joey, Vinay & I have fleshed out the Management process proposal as the very 
> first CIP document (with Jason’s inputs). It is available on the cwiki - 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652224
>
> Please comment on it and provide us any input that you may have. We want to 
> ideally time-box the period to 2 weeks so we avoid waiting indefinitely.
>
> Thanks,
>
> Dinesh
>
>> On Oct 22, 2018, at 7:30 AM, "dinesh.jo...@yahoo.com.INVALID" 
>>  wrote:
>>
>> Thanks for starting this, Mick. I will flesh it out.
>> Dinesh 
>>
>>    On Sunday, October 21, 2018, 1:52:10 AM PDT, Mick Semb Wever 
>> wrote:  
>>
>>
>>> But I'll try to put together a strawman proposal for the doc(s) over the 
>>> weekend. 
>>
>> I've thrown something quickly together here:
>> - https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652201
>> - 
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CIP-1%3A+Proposing+an+Apache+Cassandra+Management+process
>>
>> The former is a blatant rip-off from the Kafka and Spark design proposal 
>> pages that Dinesh previously mentioned. I'd hoped to do more of an analysis 
>> of the existing C* habits and precedence on design proposals (implicit in 
>> jira tickets), but in lei of that this is a strawman to start the discussion.
>>
>> The latter still needs to be fleshed out. Dinesh, can you do this? I can add 
>> a subpage/section that describes the alternative/consuming third-party tools 
>> out there.
>>
>> regards,
>> Mick
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

  

Re: Proposing an Apache Cassandra Management process

[Stefan Podkowinski]

My goal for a side car would be to enable more people to contribute to
the project, by making it more accessible for anyone who’s not familiar
with the Cassandra code base, or not familiar with Java development in
general. Although most of the functionality described in the proposal
sounds useful to have, I’d already be happy to have a solid REST API for
the existing nodetool and JMX functionality. If an official side car,
installed separately on each node, would provide that, I’m sure we’d see
lots of new tools created by the community (web UIs, cli tools, ..)
based on that. This would also be a good foundation for other existing
tool to converge upon, e.g. by calling the REST APIs for repair
scheduling and progress tracking instead of JMX, or by continually
integrating and sharing useful helper calls. This would also give
Cassandra devs more leeway to replace some of the existing tooling
related code in Cassandra, e.g. by migrating to virtual tables, while at
the same time keep providing a stable API through the side car.

What I’d also like to point out here is that implementing such a project
as an *official* side car, also implies to me having the same standards
when it comes to release quality. I’d also really prefer having feature
sets matching between Cassandra and the side car, e.g. authentication
and SSL should also be supported in the side car from the beginning,
ideally without any additional configuration.


On 06.11.18 10:40, Dinesh Joshi wrote:
> Hi all,
>
> Joey, Vinay & I have fleshed out the Management process proposal as the very 
> first CIP document (with Jason’s inputs). It is available on the cwiki - 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652224
>
> Please comment on it and provide us any input that you may have. We want to 
> ideally time-box the period to 2 weeks so we avoid waiting indefinitely.
>
> Thanks,
>
> Dinesh
>
>> On Oct 22, 2018, at 7:30 AM, "dinesh.jo...@yahoo.com.INVALID" 
>>  wrote:
>>
>> Thanks for starting this, Mick. I will flesh it out.
>> Dinesh 
>>
>>On Sunday, October 21, 2018, 1:52:10 AM PDT, Mick Semb Wever 
>>  wrote:  
>>
>>
>>> But I'll try to put together a strawman proposal for the doc(s) over the 
>>> weekend. 
>>
>> I've thrown something quickly together here:
>> - https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652201
>> - 
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CIP-1%3A+Proposing+an+Apache+Cassandra+Management+process
>>
>> The former is a blatant rip-off from the Kafka and Spark design proposal 
>> pages that Dinesh previously mentioned. I'd hoped to do more of an analysis 
>> of the existing C* habits and precedence on design proposals (implicit in 
>> jira tickets), but in lei of that this is a strawman to start the discussion.
>>
>> The latter still needs to be fleshed out. Dinesh, can you do this? I can add 
>> a subpage/section that describes the alternative/consuming third-party tools 
>> out there.
>>
>> regards,
>> Mick
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Mick Semb Wever]

I've gone through it in more detail. LGTM.


On Wed, 14 Nov 2018, at 00:50, Sankalp Kohli wrote:
> Hi Mick,
>   Any feedback? 
> Also pinging this thread after a week for others to give feedback 
> 
> > On Nov 6, 2018, at 1:40 AM, Dinesh Joshi  
> > wrote:
> > 
> > Hi all,
> > 
> > Joey, Vinay & I have fleshed out the Management process proposal as the 
> > very first CIP document (with Jason’s inputs). It is available on the cwiki 
> > - https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652224
> > 
> > Please comment on it and provide us any input that you may have. We want to 
> > ideally time-box the period to 2 weeks so we avoid waiting indefinitely.
> > 
> > Thanks,
> > 
> > Dinesh
> > 
> >> On Oct 22, 2018, at 7:30 AM, "dinesh.jo...@yahoo.com.INVALID" 
> >>  wrote:
> >> 
> >> Thanks for starting this, Mick. I will flesh it out.
> >> Dinesh 
> >> 
> >>   On Sunday, October 21, 2018, 1:52:10 AM PDT, Mick Semb Wever 
> >>  wrote:  
> >> 
> >> 
> >>> But I'll try to put together a strawman proposal for the doc(s) over the 
> >>> weekend. 
> >> 
> >> 
> >> I've thrown something quickly together here:
> >> - https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652201
> >> - 
> >> https://cwiki.apache.org/confluence/display/CASSANDRA/CIP-1%3A+Proposing+an+Apache+Cassandra+Management+process
> >> 
> >> The former is a blatant rip-off from the Kafka and Spark design proposal 
> >> pages that Dinesh previously mentioned. I'd hoped to do more of an 
> >> analysis of the existing C* habits and precedence on design proposals 
> >> (implicit in jira tickets), but in lei of that this is a strawman to start 
> >> the discussion.
> >> 
> >> The latter still needs to be fleshed out. Dinesh, can you do this? I can 
> >> add a subpage/section that describes the alternative/consuming third-party 
> >> tools out there.
> >> 
> >> regards,
> >> Mick
> >> 
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Sankalp Kohli]

Hi Mick,
  Any feedback? 
Also pinging this thread after a week for others to give feedback 

> On Nov 6, 2018, at 1:40 AM, Dinesh Joshi  
> wrote:
> 
> Hi all,
> 
> Joey, Vinay & I have fleshed out the Management process proposal as the very 
> first CIP document (with Jason’s inputs). It is available on the cwiki - 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652224
> 
> Please comment on it and provide us any input that you may have. We want to 
> ideally time-box the period to 2 weeks so we avoid waiting indefinitely.
> 
> Thanks,
> 
> Dinesh
> 
>> On Oct 22, 2018, at 7:30 AM, "dinesh.jo...@yahoo.com.INVALID" 
>>  wrote:
>> 
>> Thanks for starting this, Mick. I will flesh it out.
>> Dinesh 
>> 
>>   On Sunday, October 21, 2018, 1:52:10 AM PDT, Mick Semb Wever 
>>  wrote:  
>> 
>> 
>>> But I'll try to put together a strawman proposal for the doc(s) over the 
>>> weekend. 
>> 
>> 
>> I've thrown something quickly together here:
>> - https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652201
>> - 
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CIP-1%3A+Proposing+an+Apache+Cassandra+Management+process
>> 
>> The former is a blatant rip-off from the Kafka and Spark design proposal 
>> pages that Dinesh previously mentioned. I'd hoped to do more of an analysis 
>> of the existing C* habits and precedence on design proposals (implicit in 
>> jira tickets), but in lei of that this is a strawman to start the discussion.
>> 
>> The latter still needs to be fleshed out. Dinesh, can you do this? I can add 
>> a subpage/section that describes the alternative/consuming third-party tools 
>> out there.
>> 
>> regards,
>> Mick
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Dinesh Joshi]

Hi all,

Joey, Vinay & I have fleshed out the Management process proposal as the very 
first CIP document (with Jason’s inputs). It is available on the cwiki - 
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652224

Please comment on it and provide us any input that you may have. We want to 
ideally time-box the period to 2 weeks so we avoid waiting indefinitely.

Thanks,

Dinesh

> On Oct 22, 2018, at 7:30 AM, "dinesh.jo...@yahoo.com.INVALID" 
>  wrote:
> 
> Thanks for starting this, Mick. I will flesh it out.
> Dinesh 
> 
>On Sunday, October 21, 2018, 1:52:10 AM PDT, Mick Semb Wever 
>  wrote:  
> 
> 
>> But I'll try to put together a strawman proposal for the doc(s) over the 
>> weekend. 
> 
> 
> I've thrown something quickly together here:
> - https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652201
> - 
> https://cwiki.apache.org/confluence/display/CASSANDRA/CIP-1%3A+Proposing+an+Apache+Cassandra+Management+process
> 
> The former is a blatant rip-off from the Kafka and Spark design proposal 
> pages that Dinesh previously mentioned. I'd hoped to do more of an analysis 
> of the existing C* habits and precedence on design proposals (implicit in 
> jira tickets), but in lei of that this is a strawman to start the discussion.
> 
> The latter still needs to be fleshed out. Dinesh, can you do this? I can add 
> a subpage/section that describes the alternative/consuming third-party tools 
> out there.
> 
> regards,
> Mick
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

Re: Proposing an Apache Cassandra Management process

[dinesh.jo...@yahoo.com.INVALID]

Thanks for starting this, Mick. I will flesh it out.
Dinesh 

On Sunday, October 21, 2018, 1:52:10 AM PDT, Mick Semb Wever 
 wrote:  
 
 
> But I'll try to put together a strawman proposal for the doc(s) over the 
> weekend. 


I've thrown something quickly together here:
 - https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652201
 - 
https://cwiki.apache.org/confluence/display/CASSANDRA/CIP-1%3A+Proposing+an+Apache+Cassandra+Management+process

The former is a blatant rip-off from the Kafka and Spark design proposal pages 
that Dinesh previously mentioned. I'd hoped to do more of an analysis of the 
existing C* habits and precedence on design proposals (implicit in jira 
tickets), but in lei of that this is a strawman to start the discussion.

The latter still needs to be fleshed out. Dinesh, can you do this? I can add a 
subpage/section that describes the alternative/consuming third-party tools out 
there.

regards,
Mick

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

  

Re: Proposing an Apache Cassandra Management process

[Mick Semb Wever]

> But I'll try to put together a strawman proposal for the doc(s) over the 
> weekend. 


I've thrown something quickly together here:
 - https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652201
 - 
https://cwiki.apache.org/confluence/display/CASSANDRA/CIP-1%3A+Proposing+an+Apache+Cassandra+Management+process

The former is a blatant rip-off from the Kafka and Spark design proposal pages 
that Dinesh previously mentioned. I'd hoped to do more of an analysis of the 
existing C* habits and precedence on design proposals (implicit in jira 
tickets), but in lei of that this is a strawman to start the discussion.

The latter still needs to be fleshed out. Dinesh, can you do this? I can add a 
subpage/section that describes the alternative/consuming third-party tools out 
there.

regards,
Mick

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Mick Semb Wever]

>  Can you share the link to cwiki if you have started it ?
> 

I haven't. 
But I'll try to put together a strawman proposal for the doc(s) over the 
weekend. 

Regards, 
Mick 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Carl Mueller]

I too have built a framework over the last year similar to what cstar does
but for our purposes at smartthings. The intention is to OSS it, but it
needs a round of polish, since it really is more of a utility toolbox for
our small cassandra group.

It relies on ssh heavily and performing work on the nodes themselves/using
the installed software that is on the nodes. It also assumes a bash shell.
It is groovy based, uses jsch for the ssh'ing, and we don't use ssh config
(for better or worse). It is fairly aws-centric, although I would like to
abstract out the aws dependence. We use it for backups, restores, some data
collection/triage analysis. The backups can do incremental or fulls, where
incrementals compare the current sstable set against the previous and only
uploads the newer sstables along with manifests that link to the locations
of the already-uploaded sstables. This in particular is very aws/s3 centric
and would be a primary focus to get that dependency abstracted.

Our clusters run behind a variety of different modes, from bastions to
jumpboxes, and once upon a time direct global ssh access, and now global
IPs but with an internal backend access. We will also have ipv6 soon. We
run 2.1.x, 2.2.x, and 3.1 (even some dse community still), so the framework
attempts to deal with all the intricacies and headaches that entails. The
clusters have a lot of variance on what security has been enabled (ssl,
password, password files, etc). Not all operations have been done against
2.2 and especially 3.1, but our big project is a push to 3.x, and this will
be a big tool to enable a lot of that, so I hope to get a lot of the
idiosyncracies for those versions as we go through those upgrades. We will
be running kubernetes soon too, and I will look into abstracting the access
method to the nodes to use maybe kubectl commands once I get to know kuber
better.

I have recently found out about cstar. I'm going to look at that and see if
their way of doing things is better than how we are doing it, especially
with regards to ssh connection maintenance and those types of things. One
thing that I have found is having a "registry" that stores all the
different cluster-specific idiosyncracies, so you can just do 
  for lots of things.

It isn't particularly efficient in some ways, especially not with
connection pooling/caching/conservation. It wastes a lot of reconnecting,
but slow but steady works ok for our backups and not disrupting the work
the clusters need to do. Parallelism helps a lot, but cstar may have a lot
of good ideas for throttling, parallelism, etc.

We also plan on using it to make some dashboards/UI too at some point.

On Thu, Oct 4, 2018 at 7:20 PM Mick Semb Wever  wrote:

> Dinesh / Sankalp,
>
> My suggestion was to document the landscape in hope and an attempt to
> better understand the requirements possible to a side-car.  It wasn't a
> suggestion to patchwork together everything. But rather as part of
> brainstorming, designing, and exercising an inclusive process to see what's
> been done and how it could have been done better.
>
> A well designed side-car could also be a valuable fundamental to some of
> these third-party solutions, not just our own designs and ideals. Maybe, I
> hope, that's already obvious.
>
> It would be really fantastic to see more explorative documentation in
> confluence. Part of that can be to list up all these external tools,
> listing their goals, state, and how a side-car might help them. Reaching
> out to their maintainers to be involved in the process would be awesome
> too. I can start something in the cwiki (but i'm on vacation this week),
> I've also given you write-access Dinesh.
>
> > I also haven't seen a process to propose & discuss larger changes to
> Cassandra. The Cassandra contribution[1] guide possibly needs to be
> updated. Some communities have a process which facilitate things. See Kafka
> Improvement Process[2], Spark Improvement Process[3].
>
> Bringing this up was gold, imho. I would love to see something like this
> exist in the C* community (also in cwiki), and the side-car brainstorming
> and design used to test and flesh it out.
>
> regards,
> Mick
>
>
> On Sun, 30 Sep 2018, at 05:19, Dinesh Joshi wrote:
> > > On Sep 27, 2018, at 7:35 PM, Mick Semb Wever  wrote:
> > >
> > > Reaper,
> >
> > I have looked at this already.
> >
> > > Priam,
> >
> > I have looked at this already.
> >
> > > Marcus Olsson's offering,
> >
> > This isn't OSS.
> >
> > > CStar,
> >
> > I have looked at this already.
> >
> > > OpsCenter.
> >
> > Latest release is only compatible with DSE and not Apache Cassandra[1]
> >
> > > Then there's a host of command line tools like:
> > > ic-tools,
> > > ctop (was awesome, but is it maintained anymore?),
> > > tablesnap.
> >
> > These are interesting tools and I don't think they do what we're
> > interested in doing.
> >
> > > And maybe it's worth including the diy approach people take…
> pssh/dsh/clusterssh/mussh/fabric, etc
> >
> > 

Re: Proposing an Apache Cassandra Management process

[Mick Semb Wever]

Dinesh / Sankalp,

My suggestion was to document the landscape in hope and an attempt to better 
understand the requirements possible to a side-car.  It wasn't a suggestion to 
patchwork together everything. But rather as part of brainstorming, designing, 
and exercising an inclusive process to see what's been done and how it could 
have been done better. 

A well designed side-car could also be a valuable fundamental to some of these 
third-party solutions, not just our own designs and ideals. Maybe, I hope, 
that's already obvious.

It would be really fantastic to see more explorative documentation in 
confluence. Part of that can be to list up all these external tools, listing 
their goals, state, and how a side-car might help them. Reaching out to their 
maintainers to be involved in the process would be awesome too. I can start 
something in the cwiki (but i'm on vacation this week), I've also given you 
write-access Dinesh.

> I also haven't seen a process to propose & discuss larger changes to 
> Cassandra. The Cassandra contribution[1] guide possibly needs to be updated. 
> Some communities have a process which facilitate things. See Kafka 
> Improvement Process[2], Spark Improvement Process[3].

Bringing this up was gold, imho. I would love to see something like this exist 
in the C* community (also in cwiki), and the side-car brainstorming and design 
used to test and flesh it out.

regards,
Mick 


On Sun, 30 Sep 2018, at 05:19, Dinesh Joshi wrote:
> > On Sep 27, 2018, at 7:35 PM, Mick Semb Wever  wrote:
> > 
> > Reaper,
> 
> I have looked at this already.
> 
> > Priam,
> 
> I have looked at this already.
> 
> > Marcus Olsson's offering,
> 
> This isn't OSS.
> 
> > CStar,
> 
> I have looked at this already.
> 
> > OpsCenter.
> 
> Latest release is only compatible with DSE and not Apache Cassandra[1]
> 
> > Then there's a host of command line tools like:
> > ic-tools,
> > ctop (was awesome, but is it maintained anymore?),
> > tablesnap.
> 
> These are interesting tools and I don't think they do what we're 
> interested in doing.
> 
> > And maybe it's worth including the diy approach people take… 
> > pssh/dsh/clusterssh/mussh/fabric, etc
> 
> What's the point? You can definitely add this to the website as helpful 
> documentation.
> 
> The proposal in the original thread was to create something that is 
> supported by the Apache Cassandra project learning from the tooling 
> we've all built over the years. The fact that everyone has a sidecar or 
> their own internal tooling is an indicator that the project has room to 
> grow. It will certainly help this project be more user friendly (at 
> least for operators).
> 
> I, as a user and a developer, do not want to use a patchwork of 
> disparate tools. Does anybody oppose this on technical grounds? If you 
> do, please help me understand why would you prefer using a patchwork of 
> tools vs something that is part of the Cassandra project?
> 
> Thanks,
> 
> Dinesh
> 
> [1] https://docs.datastax.com/en/opscenter/6.0/opsc/opscPolicyChanges.html
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[sankalp kohli]

Hi Mick,
Any other feedback?
Thanks,
Sankalp

On Sun, Sep 30, 2018 at 8:54 AM Sankalp Kohli 
wrote:

> Thank you for the feedback.  There are lot of advantages of doing this in
> Apache and you can read the thread to find more. The main one is to not
> divide the energy between different tools.
> Also we will not be reinventing the wheel as I think this project can
> still get donations along the way.
>
> > On Sep 30, 2018, at 05:31, Rahul Singh 
> wrote:
> >
> > Perfection is the enemy of the good enough. All if not most informed
> open source users understand that the tar they are downloading is
> “unsupported.”
> >
> > Most of the blog posts people read or the documentation they have is in
> place of tools. Open source software let alone Tooling even a nascent
> version comes with a degree of uncertainty.
> >
> > If the question is for me: I would rather use something that exists than
> reinvent the wheel. The same way I’ll use “contrib” packages in any system
> just to see if it works.
> >
> > Example: While working on on a Solr / Kafka / Cassandra integration
> project we must have used a few different “Kafka Connect” variations before
> settling on a solution which was a combination of an existing sink
> connector and created a source connector because what was out there “meh”.
> >
> > If we had scoffed off “unsupported” packages we would have spent more
> time making something than delivering value. Technology exists to serve
> business and people’s lives not technology itself.
> >
> > There is a level of discernment that comes with experience as to what
> works and what doesn’t and what is good and what isn’t. The least we can do
> is help the user community know the difference : as Apache does at a higher
> level.
> >
> >
> > Rahul Singh
> > Chief Executive Officer
> > m 202.905.2818
> >
> > Anant Corporation
> > 1010 Wisconsin Ave NW, Suite 250
> > Washington, D.C. 20007
> >
> > We build and manage digital business technology platforms.
> >> On Sep 29, 2018, 5:29 PM -0400, sankalp kohli ,
> wrote:
> >> Thanks Dinesh for looking at the tools and thanks Mick for listing them
> >> down.
> >> Based on Dinesh response, is it accurate to say that for bulk
> >> functionality, we have evaluated the tools listed by the community? If
> >> anything is missed we should discuss it as we want to make sure we
> looked
> >> at all aspects before implementation starts.
> >>
> >> On Sat, Sep 29, 2018 at 1:19 PM Dinesh Joshi  .invalid>
> >> wrote:
> >>
>  On Sep 29, 2018, at 12:31 PM, Rahul Singh <
> rahul.xavier.si...@gmail.com>
> >>> wrote:
> 
>  All of Apache is a patchwork of tools. All of Open Source is a
> patchwork
> >>> of tools. All of Linux is a patchwork of tools.
> 
>  What works, works.
> >>>
> >>> So there isn't a way to make it any better? You would prefer using an
> >>> unsupported tool vs something that worked out of the box & was well
> tested?
> >>>
> >>> Dinesh
> >>>
> >>>
> >>> -
> >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>>
> >>>
>

Re: Proposing an Apache Cassandra Management process

[Sankalp Kohli]

Thank you for the feedback.  There are lot of advantages of doing this in 
Apache and you can read the thread to find more. The main one is to not divide 
the energy between different tools. 
Also we will not be reinventing the wheel as I think this project can still get 
donations along the way. 

> On Sep 30, 2018, at 05:31, Rahul Singh  wrote:
> 
> Perfection is the enemy of the good enough. All if not most informed open 
> source users understand that the tar they are downloading is “unsupported.”
> 
> Most of the blog posts people read or the documentation they have is in place 
> of tools. Open source software let alone Tooling even a nascent version comes 
> with a degree of uncertainty.
> 
> If the question is for me: I would rather use something that exists than 
> reinvent the wheel. The same way I’ll use “contrib” packages in any system 
> just to see if it works.
> 
> Example: While working on on a Solr / Kafka / Cassandra integration project 
> we must have used a few different “Kafka Connect” variations before settling 
> on a solution which was a combination of an existing sink connector and 
> created a source connector because what was out there “meh”.
> 
> If we had scoffed off “unsupported” packages we would have spent more time 
> making something than delivering value. Technology exists to serve business 
> and people’s lives not technology itself.
> 
> There is a level of discernment that comes with experience as to what works 
> and what doesn’t and what is good and what isn’t. The least we can do is help 
> the user community know the difference : as Apache does at a higher level.
> 
> 
> Rahul Singh
> Chief Executive Officer
> m 202.905.2818
> 
> Anant Corporation
> 1010 Wisconsin Ave NW, Suite 250
> Washington, D.C. 20007
> 
> We build and manage digital business technology platforms.
>> On Sep 29, 2018, 5:29 PM -0400, sankalp kohli , 
>> wrote:
>> Thanks Dinesh for looking at the tools and thanks Mick for listing them
>> down.
>> Based on Dinesh response, is it accurate to say that for bulk
>> functionality, we have evaluated the tools listed by the community? If
>> anything is missed we should discuss it as we want to make sure we looked
>> at all aspects before implementation starts.
>> 
>> On Sat, Sep 29, 2018 at 1:19 PM Dinesh Joshi 
>> wrote:
>> 
 On Sep 29, 2018, at 12:31 PM, Rahul Singh 
>>> wrote:
 
 All of Apache is a patchwork of tools. All of Open Source is a patchwork
>>> of tools. All of Linux is a patchwork of tools.
 
 What works, works.
>>> 
>>> So there isn't a way to make it any better? You would prefer using an
>>> unsupported tool vs something that worked out of the box & was well tested?
>>> 
>>> Dinesh
>>> 
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>>> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Rahul Singh]

Perfection is the enemy of the good enough. All if not most informed open 
source users understand that the tar they are downloading is “unsupported.”

Most of the blog posts people read or the documentation they have is in place 
of tools. Open source software let alone Tooling even a nascent version comes 
with a degree of uncertainty.

If the question is for me: I would rather use something that exists than 
reinvent the wheel. The same way I’ll use “contrib” packages in any system just 
to see if it works.

Example: While working on on a Solr / Kafka / Cassandra integration project we 
must have used a few different “Kafka Connect” variations before settling on a 
solution which was a combination of an existing sink connector and created a 
source connector because what was out there “meh”.

If we had scoffed off “unsupported” packages we would have spent more time 
making something than delivering value. Technology exists to serve business and 
people’s lives not technology itself.

There is a level of discernment that comes with experience as to what works and 
what doesn’t and what is good and what isn’t. The least we can do is help the 
user community know the difference : as Apache does at a higher level.


Rahul Singh
Chief Executive Officer
m 202.905.2818

Anant Corporation
1010 Wisconsin Ave NW, Suite 250
Washington, D.C. 20007

We build and manage digital business technology platforms.
On Sep 29, 2018, 5:29 PM -0400, sankalp kohli , wrote:
> Thanks Dinesh for looking at the tools and thanks Mick for listing them
> down.
> Based on Dinesh response, is it accurate to say that for bulk
> functionality, we have evaluated the tools listed by the community? If
> anything is missed we should discuss it as we want to make sure we looked
> at all aspects before implementation starts.
>
> On Sat, Sep 29, 2018 at 1:19 PM Dinesh Joshi 
> wrote:
>
> > > On Sep 29, 2018, at 12:31 PM, Rahul Singh 
> > wrote:
> > >
> > > All of Apache is a patchwork of tools. All of Open Source is a patchwork
> > of tools. All of Linux is a patchwork of tools.
> > >
> > > What works, works.
> >
> > So there isn't a way to make it any better? You would prefer using an
> > unsupported tool vs something that worked out of the box & was well tested?
> >
> > Dinesh
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >

Re: Proposing an Apache Cassandra Management process

[sankalp kohli]

Thanks Dinesh for looking at the tools and thanks Mick for listing them
down.
Based on Dinesh response, is it accurate to say that for bulk
functionality, we have evaluated the tools listed by the community? If
anything is missed we should discuss it as we want to make sure we looked
at all aspects before implementation starts.

On Sat, Sep 29, 2018 at 1:19 PM Dinesh Joshi 
wrote:

> > On Sep 29, 2018, at 12:31 PM, Rahul Singh 
> wrote:
> >
> > All of Apache is a patchwork of tools. All of Open Source is a patchwork
> of tools. All of Linux is a patchwork of tools.
> >
> > What works, works.
>
> So there isn't a way to make it any better? You would prefer using an
> unsupported tool vs something that worked out of the box & was well tested?
>
> Dinesh
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Proposing an Apache Cassandra Management process

[Dinesh Joshi]

> On Sep 29, 2018, at 12:31 PM, Rahul Singh  
> wrote:
> 
> All of Apache is a patchwork of tools. All of Open Source is a patchwork of 
> tools. All of Linux is a patchwork of tools.
> 
> What works, works.

So there isn't a way to make it any better? You would prefer using an 
unsupported tool vs something that worked out of the box & was well tested?

Dinesh


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Rahul Singh]

Mick,

I think there is someone who’s maintaining CTOP now.. I added it to my 
admin/monitoring list on https://github.com/Anant/awesome-cassandra

Rahul
On Sep 29, 2018, 3:20 PM -0400, Dinesh Joshi , 
wrote:
> > On Sep 27, 2018, at 7:35 PM, Mick Semb Wever  wrote:
> >
> > Reaper,
>
> I have looked at this already.
>
> > Priam,
>
> I have looked at this already.
>
> > Marcus Olsson's offering,
>
> This isn't OSS.
>
> > CStar,
>
> I have looked at this already.
>
> > OpsCenter.
>
> Latest release is only compatible with DSE and not Apache Cassandra[1]
>
> > Then there's a host of command line tools like:
> > ic-tools,
> > ctop (was awesome, but is it maintained anymore?),
> > tablesnap.
>
> These are interesting tools and I don't think they do what we're interested 
> in doing.
>
> > And maybe it's worth including the diy approach people take… 
> > pssh/dsh/clusterssh/mussh/fabric, etc
>
> What's the point? You can definitely add this to the website as helpful 
> documentation.
>
> The proposal in the original thread was to create something that is supported 
> by the Apache Cassandra project learning from the tooling we've all built 
> over the years. The fact that everyone has a sidecar or their own internal 
> tooling is an indicator that the project has room to grow. It will certainly 
> help this project be more user friendly (at least for operators).
>
> I, as a user and a developer, do not want to use a patchwork of disparate 
> tools. Does anybody oppose this on technical grounds? If you do, please help 
> me understand why would you prefer using a patchwork of tools vs something 
> that is part of the Cassandra project?
>
> Thanks,
>
> Dinesh
>
> [1] https://docs.datastax.com/en/opscenter/6.0/opsc/opscPolicyChanges.html
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>

Re: Proposing an Apache Cassandra Management process

[Rahul Singh]

All of Apache is a patchwork of tools. All of Open Source is a patchwork of 
tools. All of Linux is a patchwork of tools.

What works, works.

If it wasn’t a patchwork, open source wouldn’t be what it is. Period.

OSS Cassandra can use all the help it can get — in terms of documentation, and 
tooling. Personally I think starting out with documenting it for people is a 
good start.

Then maybe if there’s a thought later, a “distribution” approach can be taken.

When you think about the application developer or administrator, they are 
different from who we are. They don’t give a shit about “patchwork” or 
perfection, they just care to get their job done and make shit happen.

Who cares if one tool is in Java, and another is in Kotlin, and another is a 
shell script that uses PSSH? Something that works is better than nothing.


On Sep 29, 2018, 3:20 PM -0400, Dinesh Joshi , 
wrote:
> > On Sep 27, 2018, at 7:35 PM, Mick Semb Wever  wrote:
> >
> > Reaper,
>
> I have looked at this already.
>
> > Priam,
>
> I have looked at this already.
>
> > Marcus Olsson's offering,
>
> This isn't OSS.
>
> > CStar,
>
> I have looked at this already.
>
> > OpsCenter.
>
> Latest release is only compatible with DSE and not Apache Cassandra[1]
>
> > Then there's a host of command line tools like:
> > ic-tools,
> > ctop (was awesome, but is it maintained anymore?),
> > tablesnap.
>
> These are interesting tools and I don't think they do what we're interested 
> in doing.
>
> > And maybe it's worth including the diy approach people take… 
> > pssh/dsh/clusterssh/mussh/fabric, etc
>
> What's the point? You can definitely add this to the website as helpful 
> documentation.
>
> The proposal in the original thread was to create something that is supported 
> by the Apache Cassandra project learning from the tooling we've all built 
> over the years. The fact that everyone has a sidecar or their own internal 
> tooling is an indicator that the project has room to grow. It will certainly 
> help this project be more user friendly (at least for operators).
>
> I, as a user and a developer, do not want to use a patchwork of disparate 
> tools. Does anybody oppose this on technical grounds? If you do, please help 
> me understand why would you prefer using a patchwork of tools vs something 
> that is part of the Cassandra project?
>
> Thanks,
>
> Dinesh
>
> [1] https://docs.datastax.com/en/opscenter/6.0/opsc/opscPolicyChanges.html
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>

Re: Proposing an Apache Cassandra Management process

[Dinesh Joshi]

> On Sep 27, 2018, at 7:35 PM, Mick Semb Wever  wrote:
> 
> Reaper,

I have looked at this already.

> Priam,

I have looked at this already.

> Marcus Olsson's offering,

This isn't OSS.

> CStar,

I have looked at this already.

> OpsCenter.

Latest release is only compatible with DSE and not Apache Cassandra[1]

> Then there's a host of command line tools like:
> ic-tools,
> ctop (was awesome, but is it maintained anymore?),
> tablesnap.

These are interesting tools and I don't think they do what we're interested in 
doing.

> And maybe it's worth including the diy approach people take… 
> pssh/dsh/clusterssh/mussh/fabric, etc

What's the point? You can definitely add this to the website as helpful 
documentation.

The proposal in the original thread was to create something that is supported 
by the Apache Cassandra project learning from the tooling we've all built over 
the years. The fact that everyone has a sidecar or their own internal tooling 
is an indicator that the project has room to grow. It will certainly help this 
project be more user friendly (at least for operators).

I, as a user and a developer, do not want to use a patchwork of disparate 
tools. Does anybody oppose this on technical grounds? If you do, please help me 
understand why would you prefer using a patchwork of tools vs something that is 
part of the Cassandra project?

Thanks,

Dinesh

[1] https://docs.datastax.com/en/opscenter/6.0/opsc/opscPolicyChanges.html


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Mick Semb Wever]

> Mick - could you enumerate the list of tools you mentioned earlier?


Dinesh, quickly the following come to mind…

Reaper,
Priam,
Marcus Olsson's offering,
CStar,
OpsCenter.

Then there's a host of command line tools like:
ic-tools,
ctop (was awesome, but is it maintained anymore?),
tablesnap.

And maybe it's worth including the diy approach people take… 
pssh/dsh/clusterssh/mussh/fabric, etc

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[dinesh.jo...@yahoo.com.INVALID]

*bump*
Mick - could you enumerate the list of tools you mentioned earlier?

Dinesh 

On Sunday, September 23, 2018, 6:22:48 PM PDT, Dinesh Joshi 
 wrote:  
 
 Hi Mick,

Thanks for the feedback. Please find my clarifications inline.

> There's also been suggestions to take a step away from any focus on code and 
> go back to documentation and brainstorming. I appreciate the work already 
> done in the gdoc, but there's a lot left to the imagination (or just not 
> properly summarised in one place).

As I mentioned in my previous emails, the gdoc was just a starting point for a 
discussion / brainstorming.

> So could we take a step back from even designing and go to brainstorming. 
> Examples questions i can think of are running through the different scenarios 
> we're thinking of improving/solving, surveying the different C* tooling out 
> there and what they solve, need or could benefit from, ("feature set, 
> maturity, maintainer availability, etc"). I can think of a number of tools 
> already that provide the ability for cluster-wide commands. Such a page 
> listing all the tools and what they offer would be of benefit to the 
> community.

Great point, I would appreciate enumerating the list of tools you know. This is 
the kind of feedback we would've liked on the gdoc. We can look at all the 
features they provide. However, I am curious other than looking at the feature 
set, what is the motivation for surveying these tools?

> But I also thought we agreed to take a piecemeal approach between solutions, 
> rather than jumping in and developing the one and only defined sub-task from 
> scratch.

Yes I think we have a consensus on the piecemeal approach (if something doesn't 
agree, please speak up!). 

> And I can't say I'm a huge fan of google doc. Any chance we could be 
> collaborating in the Cassandra wiki instead?
> https://cwiki.apache.org/confluence/display/CASSANDRA

Sure, I don't think I have a particular preference for gdoc. It's easier to 
collaborate. I am not sure if everyone has write access to the apache wiki. It 
also requires a separate account.

I also haven't seen a process to propose & discuss larger changes to Cassandra. 
The Cassandra contribution[1] guide possibly needs to be updated. Some 
communities have a process which facilitate things. See Kafka Improvement 
Process[2], Spark Improvement Process[3].

Thanks,

Dinesh

[1] http://cassandra.apache.org/doc/latest/development/index.html
[2] 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
[3] http://spark.apache.org/improvement-proposals.html


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org
  

Re: Proposing an Apache Cassandra Management process

[Dinesh Joshi]

Hi Mick,

Thanks for the feedback. Please find my clarifications inline.

> There's also been suggestions to take a step away from any focus on code and 
> go back to documentation and brainstorming. I appreciate the work already 
> done in the gdoc, but there's a lot left to the imagination (or just not 
> properly summarised in one place).

As I mentioned in my previous emails, the gdoc was just a starting point for a 
discussion / brainstorming.

> So could we take a step back from even designing and go to brainstorming. 
> Examples questions i can think of are running through the different scenarios 
> we're thinking of improving/solving, surveying the different C* tooling out 
> there and what they solve, need or could benefit from, ("feature set, 
> maturity, maintainer availability, etc"). I can think of a number of tools 
> already that provide the ability for cluster-wide commands. Such a page 
> listing all the tools and what they offer would be of benefit to the 
> community.

Great point, I would appreciate enumerating the list of tools you know. This is 
the kind of feedback we would've liked on the gdoc. We can look at all the 
features they provide. However, I am curious other than looking at the feature 
set, what is the motivation for surveying these tools?

> But I also thought we agreed to take a piecemeal approach between solutions, 
> rather than jumping in and developing the one and only defined sub-task from 
> scratch.

Yes I think we have a consensus on the piecemeal approach (if something doesn't 
agree, please speak up!). 

> And I can't say I'm a huge fan of google doc. Any chance we could be 
> collaborating in the Cassandra wiki instead?
> https://cwiki.apache.org/confluence/display/CASSANDRA

Sure, I don't think I have a particular preference for gdoc. It's easier to 
collaborate. I am not sure if everyone has write access to the apache wiki. It 
also requires a separate account.

I also haven't seen a process to propose & discuss larger changes to Cassandra. 
The Cassandra contribution[1] guide possibly needs to be updated. Some 
communities have a process which facilitate things. See Kafka Improvement 
Process[2], Spark Improvement Process[3].

Thanks,

Dinesh

[1] http://cassandra.apache.org/doc/latest/development/index.html
[2] 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
[3] http://spark.apache.org/improvement-proposals.html


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Mick Semb Wever]

Dinesh,

> Most of the discussion and work was done off the mailing list - there's a
> big risk involved when folks disappear for months at a time and resurface
> with big pile of code plus an agenda that you failed to loop everyone in
> on.


Given all the discussions that have been had offline at this year's DD 
conference, I hope that we don't repeat the same mistake and fail to put a 
summary (more than just Sankalp's words below) out so that we're all on the 
same page and others have a chance to agree/disagree. "If it didn't happen on 
the mailing list it didn't happen."
 

> In addition, by your own words the design document didn't accurately
> describe what was being built.  


There's been confusion over the different cassandra tickets and what they cover 
and how they overlap. 

There's also been suggestions to take a step away from any focus on code and go 
back to documentation and brainstorming. I appreciate the work already done in 
the gdoc, but there's a lot left to the imagination (or just not properly 
summarised in one place).

So could we take a step back from even designing and go to brainstorming. 
Examples questions i can think of are running through the different scenarios 
we're thinking of improving/solving, surveying the different C* tooling out 
there and what they solve, need or could benefit from, ("feature set, maturity, 
maintainer availability, etc"). I can think of a number of tools already that 
provide the ability for cluster-wide commands. Such a page listing all the 
tools and what they offer would be of benefit to the community. But I also 
thought we agreed to take a piecemeal approach between solutions, rather than 
jumping in and developing the one and only defined sub-task from scratch.

It would be nice to see some horizon on this, to thrash out the landscape, and 
better understand where we're first choosing to head and why. Making it a more 
inclusive approach will also help get consensus, and hopefully post-freeze get 
more people involved. ("measure twice, cut once")

And I can't say I'm a huge fan of google doc. Any chance we could be 
collaborating in the Cassandra wiki instead?
https://cwiki.apache.org/confluence/display/CASSANDRA

regards,
Mick
  

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Jordan West]

I think this feature is important to the community and I don’t want to
stifle that but if committers/contributors are working on the management
process instead of testing 4.0 it takes away from it regardless of where
the code lives. Waiting to merge until after 4.0, at a minimum, would
benefit the testing effort.

Jordan

On Sat, Sep 22, 2018 at 10:06 AM Sankalp Kohli 
wrote:

> This is not part of core database and a separate repo and so my impression
> is that this can continue to make progress. Also we can always make
> progress and not merge it till freeze is lifted.
>
> Open to ideas/suggestions if someone thinks otherwise.
>
> > On Sep 22, 2018, at 03:13, kurt greaves  wrote:
> >
> > Is this something we're moving ahead with despite the feature freeze?
> >
> > On Sat, 22 Sep 2018 at 08:32, dinesh.jo...@yahoo.com.INVALID
> >  wrote:
> >
> >> I have created a sub-task - CASSANDRA-14783. Could we get some feedback
> >> before we begin implementing anything?
> >>
> >> Dinesh
> >>
> >>On Thursday, September 20, 2018, 11:22:33 PM PDT, Dinesh Joshi <
> >> dinesh.jo...@yahoo.com.INVALID> wrote:
> >>
> >> I have updated the doc with a short paragraph providing the
> >> clarification. Sankalp's suggestion is already part of the doc. If there
> >> aren't further objections could we move this discussion over to the jira
> >> (CASSANDRA-14395)?
> >>
> >> Dinesh
> >>
> >>> On Sep 18, 2018, at 10:31 AM, sankalp kohli 
> >> wrote:
> >>>
> >>> How about we start with a few basic features in side car. How about
> >> starting with this
> >>> 1. Bulk nodetool commands: User can curl any sidecar and be able to run
> >> a nodetool command in bulk across the cluster.
> >>>
> >>
> :/bulk/nodetool/tablestats?arg0=keyspace_name.table_name= >> required>
> >>>
> >>> And later
> >>> 2: Health checks.
> >>>
> >>> On Thu, Sep 13, 2018 at 11:34 AM dinesh.jo...@yahoo.com.INVALID <
> >> dinesh.jo...@yahoo.com.invalid> wrote:
> >>> I will update the document to add that point. The document did not mean
> >> to serve as a design or architectural document but rather something that
> >> would spark a discussion on the idea.
> >>> Dinesh
> >>>
> >>>   On Thursday, September 13, 2018, 10:59:34 AM PDT, Jonathan Haddad <
> >> j...@jonhaddad.com > wrote:
> >>>
> >>> Most of the discussion and work was done off the mailing list - there's
> >> a
> >>> big risk involved when folks disappear for months at a time and
> resurface
> >>> with big pile of code plus an agenda that you failed to loop everyone
> in
> >>> on. In addition, by your own words the design document didn't
> accurately
> >>> describe what was being built.  I don't write this to try to argue
> about
> >>> it, I just want to put some perspective for those of us that weren't
> part
> >>> of this discussion on a weekly basis over the last several months.
> Going
> >>> forward let's keep things on the ML so we can avoid confusion and
> >>> frustration for all parties.
> >>>
> >>> With that said - I think Blake made a really good point here and it's
> >>> helped me understand the scope of what's being built better.  Looking
> at
> >> it
> >>> from a different perspective it doesn't seem like there's as much
> overlap
> >>> as I had initially thought.  There's the machinery that runs certain
> >> tasks
> >>> (what Joey has been working on) and the user facing side of exposing
> that
> >>> information in management tool.
> >>>
> >>> I do appreciate (and like) the idea of not trying to boil the ocean,
> and
> >>> working on things incrementally.  Putting a thin layer on top of
> >> Cassandra
> >>> that can perform cluster wide tasks does give us an opportunity to move
> >> in
> >>> the direction of a general purpose user-facing admin tool without
> >>> committing to trying to write the full stack all at once (or even make
> >>> decisions on it now).  We do need a sensible way of doing rolling
> >> restarts
> >>> / scrubs / scheduling and Reaper wasn't built for that, and even though
> >> we
> >>> can add it I'm not sure if it's the best mechanism for the long term.
> >>>
> >>> So if your goal is to add maturity to the project by making cluster
> wide
> >>> tasks easier by providing a framework to build on top of, I'm in favor
> of
> >>> that and I don't see it as antithetical to what I had in mind with
> >> Reaper.
> >>> Rather, the two are more complementary than I had originally realized.
> >>>
> >>> Jon
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, Sep 13, 2018 at 10:39 AM dinesh.jo...@yahoo.com.INVALID
> >>> mailto:dinesh.jo...@yahoo.com>.invalid>
> wrote:
> >>>
>  I have a few clarifications -
>  The scope of the management process is not to simply run repair
>  scheduling. Repair scheduling is one of the many features we could
>  implement or adopt from existing sources. So could we please split the
>  Management Process discussion and the repair scheduling?
>  After re-reading the management process proposal, I see we missed to
> 

Re: Proposing an Apache Cassandra Management process

[Sankalp Kohli]

This is not part of core database and a separate repo and so my impression is 
that this can continue to make progress. Also we can always make progress and 
not merge it till freeze is lifted. 

Open to ideas/suggestions if someone thinks otherwise. 

> On Sep 22, 2018, at 03:13, kurt greaves  wrote:
> 
> Is this something we're moving ahead with despite the feature freeze?
> 
> On Sat, 22 Sep 2018 at 08:32, dinesh.jo...@yahoo.com.INVALID
>  wrote:
> 
>> I have created a sub-task - CASSANDRA-14783. Could we get some feedback
>> before we begin implementing anything?
>> 
>> Dinesh
>> 
>>On Thursday, September 20, 2018, 11:22:33 PM PDT, Dinesh Joshi <
>> dinesh.jo...@yahoo.com.INVALID> wrote:
>> 
>> I have updated the doc with a short paragraph providing the
>> clarification. Sankalp's suggestion is already part of the doc. If there
>> aren't further objections could we move this discussion over to the jira
>> (CASSANDRA-14395)?
>> 
>> Dinesh
>> 
>>> On Sep 18, 2018, at 10:31 AM, sankalp kohli 
>> wrote:
>>> 
>>> How about we start with a few basic features in side car. How about
>> starting with this
>>> 1. Bulk nodetool commands: User can curl any sidecar and be able to run
>> a nodetool command in bulk across the cluster.
>>> 
>> :/bulk/nodetool/tablestats?arg0=keyspace_name.table_name=> required>
>>> 
>>> And later
>>> 2: Health checks.
>>> 
>>> On Thu, Sep 13, 2018 at 11:34 AM dinesh.jo...@yahoo.com.INVALID <
>> dinesh.jo...@yahoo.com.invalid> wrote:
>>> I will update the document to add that point. The document did not mean
>> to serve as a design or architectural document but rather something that
>> would spark a discussion on the idea.
>>> Dinesh
>>> 
>>>   On Thursday, September 13, 2018, 10:59:34 AM PDT, Jonathan Haddad <
>> j...@jonhaddad.com > wrote:
>>> 
>>> Most of the discussion and work was done off the mailing list - there's
>> a
>>> big risk involved when folks disappear for months at a time and resurface
>>> with big pile of code plus an agenda that you failed to loop everyone in
>>> on. In addition, by your own words the design document didn't accurately
>>> describe what was being built.  I don't write this to try to argue about
>>> it, I just want to put some perspective for those of us that weren't part
>>> of this discussion on a weekly basis over the last several months.  Going
>>> forward let's keep things on the ML so we can avoid confusion and
>>> frustration for all parties.
>>> 
>>> With that said - I think Blake made a really good point here and it's
>>> helped me understand the scope of what's being built better.  Looking at
>> it
>>> from a different perspective it doesn't seem like there's as much overlap
>>> as I had initially thought.  There's the machinery that runs certain
>> tasks
>>> (what Joey has been working on) and the user facing side of exposing that
>>> information in management tool.
>>> 
>>> I do appreciate (and like) the idea of not trying to boil the ocean, and
>>> working on things incrementally.  Putting a thin layer on top of
>> Cassandra
>>> that can perform cluster wide tasks does give us an opportunity to move
>> in
>>> the direction of a general purpose user-facing admin tool without
>>> committing to trying to write the full stack all at once (or even make
>>> decisions on it now).  We do need a sensible way of doing rolling
>> restarts
>>> / scrubs / scheduling and Reaper wasn't built for that, and even though
>> we
>>> can add it I'm not sure if it's the best mechanism for the long term.
>>> 
>>> So if your goal is to add maturity to the project by making cluster wide
>>> tasks easier by providing a framework to build on top of, I'm in favor of
>>> that and I don't see it as antithetical to what I had in mind with
>> Reaper.
>>> Rather, the two are more complementary than I had originally realized.
>>> 
>>> Jon
>>> 
>>> 
>>> 
>>> 
>>> On Thu, Sep 13, 2018 at 10:39 AM dinesh.jo...@yahoo.com.INVALID
>>> mailto:dinesh.jo...@yahoo.com>.invalid> wrote:
>>> 
 I have a few clarifications -
 The scope of the management process is not to simply run repair
 scheduling. Repair scheduling is one of the many features we could
 implement or adopt from existing sources. So could we please split the
 Management Process discussion and the repair scheduling?
 After re-reading the management process proposal, I see we missed to
 communicate a basic idea in the document. We wanted to take a pluggable
 approach to various activities that the management process could
>> perform.
 This could accommodate different implementations of common activities
>> such
 as repair. The management process would provide the basic framework
>> and it
 would have default implementations for some of the basic activities.
>> This
 would allow for speedier iteration cycles and keep things extensible.
 Turning to some questions that Jon and others have raised, when I +1,
>> my
 intention is to fully 

Re: Proposing an Apache Cassandra Management process

[kurt greaves]

Is this something we're moving ahead with despite the feature freeze?

On Sat, 22 Sep 2018 at 08:32, dinesh.jo...@yahoo.com.INVALID
 wrote:

> I have created a sub-task - CASSANDRA-14783. Could we get some feedback
> before we begin implementing anything?
>
> Dinesh
>
> On Thursday, September 20, 2018, 11:22:33 PM PDT, Dinesh Joshi <
> dinesh.jo...@yahoo.com.INVALID> wrote:
>
>  I have updated the doc with a short paragraph providing the
> clarification. Sankalp's suggestion is already part of the doc. If there
> aren't further objections could we move this discussion over to the jira
> (CASSANDRA-14395)?
>
> Dinesh
>
> > On Sep 18, 2018, at 10:31 AM, sankalp kohli 
> wrote:
> >
> > How about we start with a few basic features in side car. How about
> starting with this
> > 1. Bulk nodetool commands: User can curl any sidecar and be able to run
> a nodetool command in bulk across the cluster.
> >
> :/bulk/nodetool/tablestats?arg0=keyspace_name.table_name= required>
> >
> > And later
> > 2: Health checks.
> >
> > On Thu, Sep 13, 2018 at 11:34 AM dinesh.jo...@yahoo.com.INVALID <
> dinesh.jo...@yahoo.com.invalid> wrote:
> > I will update the document to add that point. The document did not mean
> to serve as a design or architectural document but rather something that
> would spark a discussion on the idea.
> > Dinesh
> >
> >On Thursday, September 13, 2018, 10:59:34 AM PDT, Jonathan Haddad <
> j...@jonhaddad.com > wrote:
> >
> >  Most of the discussion and work was done off the mailing list - there's
> a
> > big risk involved when folks disappear for months at a time and resurface
> > with big pile of code plus an agenda that you failed to loop everyone in
> > on. In addition, by your own words the design document didn't accurately
> > describe what was being built.  I don't write this to try to argue about
> > it, I just want to put some perspective for those of us that weren't part
> > of this discussion on a weekly basis over the last several months.  Going
> > forward let's keep things on the ML so we can avoid confusion and
> > frustration for all parties.
> >
> > With that said - I think Blake made a really good point here and it's
> > helped me understand the scope of what's being built better.  Looking at
> it
> > from a different perspective it doesn't seem like there's as much overlap
> > as I had initially thought.  There's the machinery that runs certain
> tasks
> > (what Joey has been working on) and the user facing side of exposing that
> > information in management tool.
> >
> > I do appreciate (and like) the idea of not trying to boil the ocean, and
> > working on things incrementally.  Putting a thin layer on top of
> Cassandra
> > that can perform cluster wide tasks does give us an opportunity to move
> in
> > the direction of a general purpose user-facing admin tool without
> > committing to trying to write the full stack all at once (or even make
> > decisions on it now).  We do need a sensible way of doing rolling
> restarts
> > / scrubs / scheduling and Reaper wasn't built for that, and even though
> we
> > can add it I'm not sure if it's the best mechanism for the long term.
> >
> > So if your goal is to add maturity to the project by making cluster wide
> > tasks easier by providing a framework to build on top of, I'm in favor of
> > that and I don't see it as antithetical to what I had in mind with
> Reaper.
> > Rather, the two are more complementary than I had originally realized.
> >
> > Jon
> >
> >
> >
> >
> > On Thu, Sep 13, 2018 at 10:39 AM dinesh.jo...@yahoo.com.INVALID
> > mailto:dinesh.jo...@yahoo.com>.invalid> wrote:
> >
> > > I have a few clarifications -
> > > The scope of the management process is not to simply run repair
> > > scheduling. Repair scheduling is one of the many features we could
> > > implement or adopt from existing sources. So could we please split the
> > > Management Process discussion and the repair scheduling?
> > > After re-reading the management process proposal, I see we missed to
> > > communicate a basic idea in the document. We wanted to take a pluggable
> > > approach to various activities that the management process could
> perform.
> > > This could accommodate different implementations of common activities
> such
> > > as repair. The management process would provide the basic framework
> and it
> > > would have default implementations for some of the basic activities.
> This
> > > would allow for speedier iteration cycles and keep things extensible.
> > > Turning to some questions that Jon and others have raised, when I +1,
> my
> > > intention is to fully contribute and stay with this community. That
> said,
> > > things feel rushed for some but for me it feels like analysis
> paralysis.
> > > We're looking for actionable feedback and to discuss the management
> process
> > > _not_ repair scheduling solutions.
> > > Thanks,
> > > Dinesh
> > >
> > >
> > >
> > > On Sep 12, 2018, at 6:24 PM, sankalp kohli  

Re: Proposing an Apache Cassandra Management process

[dinesh.jo...@yahoo.com.INVALID]

I have created a sub-task - CASSANDRA-14783. Could we get some feedback before 
we begin implementing anything?

Dinesh 

On Thursday, September 20, 2018, 11:22:33 PM PDT, Dinesh Joshi 
 wrote:  
 
 I have updated the doc with a short paragraph providing the clarification. 
Sankalp's suggestion is already part of the doc. If there aren't further 
objections could we move this discussion over to the jira (CASSANDRA-14395)?

Dinesh

> On Sep 18, 2018, at 10:31 AM, sankalp kohli  wrote:
> 
> How about we start with a few basic features in side car. How about starting 
> with this
> 1. Bulk nodetool commands: User can curl any sidecar and be able to run a 
> nodetool command in bulk across the cluster. 
> :/bulk/nodetool/tablestats?arg0=keyspace_name.table_name=  required>
> 
> And later 
> 2: Health checks. 
> 
> On Thu, Sep 13, 2018 at 11:34 AM dinesh.jo...@yahoo.com.INVALID 
>  wrote:
> I will update the document to add that point. The document did not mean to 
> serve as a design or architectural document but rather something that would 
> spark a discussion on the idea.
> Dinesh 
> 
>    On Thursday, September 13, 2018, 10:59:34 AM PDT, Jonathan Haddad 
>mailto:j...@jonhaddad.com>> wrote:  
> 
>  Most of the discussion and work was done off the mailing list - there's a
> big risk involved when folks disappear for months at a time and resurface
> with big pile of code plus an agenda that you failed to loop everyone in
> on. In addition, by your own words the design document didn't accurately
> describe what was being built.  I don't write this to try to argue about
> it, I just want to put some perspective for those of us that weren't part
> of this discussion on a weekly basis over the last several months.  Going
> forward let's keep things on the ML so we can avoid confusion and
> frustration for all parties.
> 
> With that said - I think Blake made a really good point here and it's
> helped me understand the scope of what's being built better.  Looking at it
> from a different perspective it doesn't seem like there's as much overlap
> as I had initially thought.  There's the machinery that runs certain tasks
> (what Joey has been working on) and the user facing side of exposing that
> information in management tool.
> 
> I do appreciate (and like) the idea of not trying to boil the ocean, and
> working on things incrementally.  Putting a thin layer on top of Cassandra
> that can perform cluster wide tasks does give us an opportunity to move in
> the direction of a general purpose user-facing admin tool without
> committing to trying to write the full stack all at once (or even make
> decisions on it now).  We do need a sensible way of doing rolling restarts
> / scrubs / scheduling and Reaper wasn't built for that, and even though we
> can add it I'm not sure if it's the best mechanism for the long term.
> 
> So if your goal is to add maturity to the project by making cluster wide
> tasks easier by providing a framework to build on top of, I'm in favor of
> that and I don't see it as antithetical to what I had in mind with Reaper.
> Rather, the two are more complementary than I had originally realized.
> 
> Jon
> 
> 
> 
> 
> On Thu, Sep 13, 2018 at 10:39 AM dinesh.jo...@yahoo.com.INVALID
> mailto:dinesh.jo...@yahoo.com>.invalid> wrote:
> 
> > I have a few clarifications -
> > The scope of the management process is not to simply run repair
> > scheduling. Repair scheduling is one of the many features we could
> > implement or adopt from existing sources. So could we please split the
> > Management Process discussion and the repair scheduling?
> > After re-reading the management process proposal, I see we missed to
> > communicate a basic idea in the document. We wanted to take a pluggable
> > approach to various activities that the management process could perform.
> > This could accommodate different implementations of common activities such
> > as repair. The management process would provide the basic framework and it
> > would have default implementations for some of the basic activities. This
> > would allow for speedier iteration cycles and keep things extensible.
> > Turning to some questions that Jon and others have raised, when I +1, my
> > intention is to fully contribute and stay with this community. That said,
> > things feel rushed for some but for me it feels like analysis paralysis.
> > We're looking for actionable feedback and to discuss the management process
> > _not_ repair scheduling solutions.
> > Thanks,
> > Dinesh
> >
> >
> >
> > On Sep 12, 2018, at 6:24 PM, sankalp kohli  > > wrote:
> > Here is a list of open discussion points from the voting thread. I think
> > some are already answered but I will still gather these questions here.
> >
> > From several people:
> > 1. Vote is rushed and we need more time for discussion.
> >
> > From Sylvain
> > 2. About the voting process...I think that was addressed by Jeff Jirsa and
> > deserves a separate 

Re: Proposing an Apache Cassandra Management process

[Dinesh Joshi]

I have updated the doc with a short paragraph providing the clarification. 
Sankalp's suggestion is already part of the doc. If there aren't further 
objections could we move this discussion over to the jira (CASSANDRA-14395)?

Dinesh

> On Sep 18, 2018, at 10:31 AM, sankalp kohli  wrote:
> 
> How about we start with a few basic features in side car. How about starting 
> with this
> 1. Bulk nodetool commands: User can curl any sidecar and be able to run a 
> nodetool command in bulk across the cluster. 
> :/bulk/nodetool/tablestats?arg0=keyspace_name.table_name=  required>
> 
> And later 
> 2: Health checks. 
> 
> On Thu, Sep 13, 2018 at 11:34 AM dinesh.jo...@yahoo.com.INVALID 
>  wrote:
> I will update the document to add that point. The document did not mean to 
> serve as a design or architectural document but rather something that would 
> spark a discussion on the idea.
> Dinesh 
> 
> On Thursday, September 13, 2018, 10:59:34 AM PDT, Jonathan Haddad 
> mailto:j...@jonhaddad.com>> wrote:  
> 
>  Most of the discussion and work was done off the mailing list - there's a
> big risk involved when folks disappear for months at a time and resurface
> with big pile of code plus an agenda that you failed to loop everyone in
> on. In addition, by your own words the design document didn't accurately
> describe what was being built.  I don't write this to try to argue about
> it, I just want to put some perspective for those of us that weren't part
> of this discussion on a weekly basis over the last several months.  Going
> forward let's keep things on the ML so we can avoid confusion and
> frustration for all parties.
> 
> With that said - I think Blake made a really good point here and it's
> helped me understand the scope of what's being built better.  Looking at it
> from a different perspective it doesn't seem like there's as much overlap
> as I had initially thought.  There's the machinery that runs certain tasks
> (what Joey has been working on) and the user facing side of exposing that
> information in management tool.
> 
> I do appreciate (and like) the idea of not trying to boil the ocean, and
> working on things incrementally.  Putting a thin layer on top of Cassandra
> that can perform cluster wide tasks does give us an opportunity to move in
> the direction of a general purpose user-facing admin tool without
> committing to trying to write the full stack all at once (or even make
> decisions on it now).  We do need a sensible way of doing rolling restarts
> / scrubs / scheduling and Reaper wasn't built for that, and even though we
> can add it I'm not sure if it's the best mechanism for the long term.
> 
> So if your goal is to add maturity to the project by making cluster wide
> tasks easier by providing a framework to build on top of, I'm in favor of
> that and I don't see it as antithetical to what I had in mind with Reaper.
> Rather, the two are more complementary than I had originally realized.
> 
> Jon
> 
> 
> 
> 
> On Thu, Sep 13, 2018 at 10:39 AM dinesh.jo...@yahoo.com.INVALID
> mailto:dinesh.jo...@yahoo.com>.invalid> wrote:
> 
> > I have a few clarifications -
> > The scope of the management process is not to simply run repair
> > scheduling. Repair scheduling is one of the many features we could
> > implement or adopt from existing sources. So could we please split the
> > Management Process discussion and the repair scheduling?
> > After re-reading the management process proposal, I see we missed to
> > communicate a basic idea in the document. We wanted to take a pluggable
> > approach to various activities that the management process could perform.
> > This could accommodate different implementations of common activities such
> > as repair. The management process would provide the basic framework and it
> > would have default implementations for some of the basic activities. This
> > would allow for speedier iteration cycles and keep things extensible.
> > Turning to some questions that Jon and others have raised, when I +1, my
> > intention is to fully contribute and stay with this community. That said,
> > things feel rushed for some but for me it feels like analysis paralysis.
> > We're looking for actionable feedback and to discuss the management process
> > _not_ repair scheduling solutions.
> > Thanks,
> > Dinesh
> >
> >
> >
> > On Sep 12, 2018, at 6:24 PM, sankalp kohli  > > wrote:
> > Here is a list of open discussion points from the voting thread. I think
> > some are already answered but I will still gather these questions here.
> >
> > From several people:
> > 1. Vote is rushed and we need more time for discussion.
> >
> > From Sylvain
> > 2. About the voting process...I think that was addressed by Jeff Jirsa and
> > deserves a separate thread as it is not directly related to this thread.
> > 3. Does the project need a side car.
> >
> > From Jonathan Haddad
> > 4. Are people doing +1 willing to contribute
> >
> > From Jonathan Ellis
> > 5. 

Re: Proposing an Apache Cassandra Management process

[sankalp kohli]

How about we start with a few basic features in side car. How about
starting with this
1. Bulk nodetool commands: User can curl any sidecar and be able to run a
nodetool command in bulk across the cluster.
:/bulk/nodetool/tablestats?arg0=keyspace_name.table_name=

And later
2: Health checks.

On Thu, Sep 13, 2018 at 11:34 AM dinesh.jo...@yahoo.com.INVALID
 wrote:

> I will update the document to add that point. The document did not mean to
> serve as a design or architectural document but rather something that would
> spark a discussion on the idea.
> Dinesh
>
> On Thursday, September 13, 2018, 10:59:34 AM PDT, Jonathan Haddad <
> j...@jonhaddad.com> wrote:
>
>  Most of the discussion and work was done off the mailing list - there's a
> big risk involved when folks disappear for months at a time and resurface
> with big pile of code plus an agenda that you failed to loop everyone in
> on. In addition, by your own words the design document didn't accurately
> describe what was being built.  I don't write this to try to argue about
> it, I just want to put some perspective for those of us that weren't part
> of this discussion on a weekly basis over the last several months.  Going
> forward let's keep things on the ML so we can avoid confusion and
> frustration for all parties.
>
> With that said - I think Blake made a really good point here and it's
> helped me understand the scope of what's being built better.  Looking at it
> from a different perspective it doesn't seem like there's as much overlap
> as I had initially thought.  There's the machinery that runs certain tasks
> (what Joey has been working on) and the user facing side of exposing that
> information in management tool.
>
> I do appreciate (and like) the idea of not trying to boil the ocean, and
> working on things incrementally.  Putting a thin layer on top of Cassandra
> that can perform cluster wide tasks does give us an opportunity to move in
> the direction of a general purpose user-facing admin tool without
> committing to trying to write the full stack all at once (or even make
> decisions on it now).  We do need a sensible way of doing rolling restarts
> / scrubs / scheduling and Reaper wasn't built for that, and even though we
> can add it I'm not sure if it's the best mechanism for the long term.
>
> So if your goal is to add maturity to the project by making cluster wide
> tasks easier by providing a framework to build on top of, I'm in favor of
> that and I don't see it as antithetical to what I had in mind with Reaper.
> Rather, the two are more complementary than I had originally realized.
>
> Jon
>
>
>
>
> On Thu, Sep 13, 2018 at 10:39 AM dinesh.jo...@yahoo.com.INVALID
>  wrote:
>
> > I have a few clarifications -
> > The scope of the management process is not to simply run repair
> > scheduling. Repair scheduling is one of the many features we could
> > implement or adopt from existing sources. So could we please split the
> > Management Process discussion and the repair scheduling?
> > After re-reading the management process proposal, I see we missed to
> > communicate a basic idea in the document. We wanted to take a pluggable
> > approach to various activities that the management process could perform.
> > This could accommodate different implementations of common activities
> such
> > as repair. The management process would provide the basic framework and
> it
> > would have default implementations for some of the basic activities. This
> > would allow for speedier iteration cycles and keep things extensible.
> > Turning to some questions that Jon and others have raised, when I +1, my
> > intention is to fully contribute and stay with this community. That said,
> > things feel rushed for some but for me it feels like analysis paralysis.
> > We're looking for actionable feedback and to discuss the management
> process
> > _not_ repair scheduling solutions.
> > Thanks,
> > Dinesh
> >
> >
> >
> > On Sep 12, 2018, at 6:24 PM, sankalp kohli 
> wrote:
> > Here is a list of open discussion points from the voting thread. I think
> > some are already answered but I will still gather these questions here.
> >
> > From several people:
> > 1. Vote is rushed and we need more time for discussion.
> >
> > From Sylvain
> > 2. About the voting process...I think that was addressed by Jeff Jirsa
> and
> > deserves a separate thread as it is not directly related to this thread.
> > 3. Does the project need a side car.
> >
> > From Jonathan Haddad
> > 4. Are people doing +1 willing to contribute
> >
> > From Jonathan Ellis
> > 5. List of feature set, maturity, maintainer availability from Reaper or
> > any other project being donated.
> >
> > Mick Semb Wever
> > 6. We should not vote on these things and instead build consensus.
> >
> > Open Questions from this thread
> > 7. What technical debts we are talking about in Reaper. Can someone give
> > concrete examples.
> > 8. What is the timeline of donating Reaper to Apache Cassandra.
> >
> > 

Re: Proposing an Apache Cassandra Management process

[dinesh.jo...@yahoo.com.INVALID]

I will update the document to add that point. The document did not mean to 
serve as a design or architectural document but rather something that would 
spark a discussion on the idea.
Dinesh 

On Thursday, September 13, 2018, 10:59:34 AM PDT, Jonathan Haddad 
 wrote:  
 
 Most of the discussion and work was done off the mailing list - there's a
big risk involved when folks disappear for months at a time and resurface
with big pile of code plus an agenda that you failed to loop everyone in
on. In addition, by your own words the design document didn't accurately
describe what was being built.  I don't write this to try to argue about
it, I just want to put some perspective for those of us that weren't part
of this discussion on a weekly basis over the last several months.  Going
forward let's keep things on the ML so we can avoid confusion and
frustration for all parties.

With that said - I think Blake made a really good point here and it's
helped me understand the scope of what's being built better.  Looking at it
from a different perspective it doesn't seem like there's as much overlap
as I had initially thought.  There's the machinery that runs certain tasks
(what Joey has been working on) and the user facing side of exposing that
information in management tool.

I do appreciate (and like) the idea of not trying to boil the ocean, and
working on things incrementally.  Putting a thin layer on top of Cassandra
that can perform cluster wide tasks does give us an opportunity to move in
the direction of a general purpose user-facing admin tool without
committing to trying to write the full stack all at once (or even make
decisions on it now).  We do need a sensible way of doing rolling restarts
/ scrubs / scheduling and Reaper wasn't built for that, and even though we
can add it I'm not sure if it's the best mechanism for the long term.

So if your goal is to add maturity to the project by making cluster wide
tasks easier by providing a framework to build on top of, I'm in favor of
that and I don't see it as antithetical to what I had in mind with Reaper.
Rather, the two are more complementary than I had originally realized.

Jon




On Thu, Sep 13, 2018 at 10:39 AM dinesh.jo...@yahoo.com.INVALID
 wrote:

> I have a few clarifications -
> The scope of the management process is not to simply run repair
> scheduling. Repair scheduling is one of the many features we could
> implement or adopt from existing sources. So could we please split the
> Management Process discussion and the repair scheduling?
> After re-reading the management process proposal, I see we missed to
> communicate a basic idea in the document. We wanted to take a pluggable
> approach to various activities that the management process could perform.
> This could accommodate different implementations of common activities such
> as repair. The management process would provide the basic framework and it
> would have default implementations for some of the basic activities. This
> would allow for speedier iteration cycles and keep things extensible.
> Turning to some questions that Jon and others have raised, when I +1, my
> intention is to fully contribute and stay with this community. That said,
> things feel rushed for some but for me it feels like analysis paralysis.
> We're looking for actionable feedback and to discuss the management process
> _not_ repair scheduling solutions.
> Thanks,
> Dinesh
>
>
>
> On Sep 12, 2018, at 6:24 PM, sankalp kohli  wrote:
> Here is a list of open discussion points from the voting thread. I think
> some are already answered but I will still gather these questions here.
>
> From several people:
> 1. Vote is rushed and we need more time for discussion.
>
> From Sylvain
> 2. About the voting process...I think that was addressed by Jeff Jirsa and
> deserves a separate thread as it is not directly related to this thread.
> 3. Does the project need a side car.
>
> From Jonathan Haddad
> 4. Are people doing +1 willing to contribute
>
> From Jonathan Ellis
> 5. List of feature set, maturity, maintainer availability from Reaper or
> any other project being donated.
>
> Mick Semb Wever
> 6. We should not vote on these things and instead build consensus.
>
> Open Questions from this thread
> 7. What technical debts we are talking about in Reaper. Can someone give
> concrete examples.
> 8. What is the timeline of donating Reaper to Apache Cassandra.
>
> On Wed, Sep 12, 2018 at 3:49 PM sankalp kohli 
> wrote:
>
>
> (Using this thread and not the vote thread intentionally)
> For folks talking about vote being rushed. I would use the email from
> Joseph to show this is not rushed. There was no email on this thread for 4
> months until I pinged.
>
>
> Dec 2016: Vinay worked with Jon and Alex to try to collaborate on Reaper to
> come up with design goals for a repair scheduler that could work at Netflix
> scale.
>
> ~Feb 2017: Netflix believes that the fundamental design gaps prevented us
> from using Reaper as it relies 

Re: Proposing an Apache Cassandra Management process

[Jonathan Haddad]

Most of the discussion and work was done off the mailing list - there's a
big risk involved when folks disappear for months at a time and resurface
with big pile of code plus an agenda that you failed to loop everyone in
on. In addition, by your own words the design document didn't accurately
describe what was being built.  I don't write this to try to argue about
it, I just want to put some perspective for those of us that weren't part
of this discussion on a weekly basis over the last several months.  Going
forward let's keep things on the ML so we can avoid confusion and
frustration for all parties.

With that said - I think Blake made a really good point here and it's
helped me understand the scope of what's being built better.  Looking at it
from a different perspective it doesn't seem like there's as much overlap
as I had initially thought.  There's the machinery that runs certain tasks
(what Joey has been working on) and the user facing side of exposing that
information in management tool.

I do appreciate (and like) the idea of not trying to boil the ocean, and
working on things incrementally.  Putting a thin layer on top of Cassandra
that can perform cluster wide tasks does give us an opportunity to move in
the direction of a general purpose user-facing admin tool without
committing to trying to write the full stack all at once (or even make
decisions on it now).  We do need a sensible way of doing rolling restarts
/ scrubs / scheduling and Reaper wasn't built for that, and even though we
can add it I'm not sure if it's the best mechanism for the long term.

So if your goal is to add maturity to the project by making cluster wide
tasks easier by providing a framework to build on top of, I'm in favor of
that and I don't see it as antithetical to what I had in mind with Reaper.
Rather, the two are more complementary than I had originally realized.

Jon




On Thu, Sep 13, 2018 at 10:39 AM dinesh.jo...@yahoo.com.INVALID
 wrote:

> I have a few clarifications -
> The scope of the management process is not to simply run repair
> scheduling. Repair scheduling is one of the many features we could
> implement or adopt from existing sources. So could we please split the
> Management Process discussion and the repair scheduling?
> After re-reading the management process proposal, I see we missed to
> communicate a basic idea in the document. We wanted to take a pluggable
> approach to various activities that the management process could perform.
> This could accommodate different implementations of common activities such
> as repair. The management process would provide the basic framework and it
> would have default implementations for some of the basic activities. This
> would allow for speedier iteration cycles and keep things extensible.
> Turning to some questions that Jon and others have raised, when I +1, my
> intention is to fully contribute and stay with this community. That said,
> things feel rushed for some but for me it feels like analysis paralysis.
> We're looking for actionable feedback and to discuss the management process
> _not_ repair scheduling solutions.
> Thanks,
> Dinesh
>
>
>
> On Sep 12, 2018, at 6:24 PM, sankalp kohli  wrote:
> Here is a list of open discussion points from the voting thread. I think
> some are already answered but I will still gather these questions here.
>
> From several people:
> 1. Vote is rushed and we need more time for discussion.
>
> From Sylvain
> 2. About the voting process...I think that was addressed by Jeff Jirsa and
> deserves a separate thread as it is not directly related to this thread.
> 3. Does the project need a side car.
>
> From Jonathan Haddad
> 4. Are people doing +1 willing to contribute
>
> From Jonathan Ellis
> 5. List of feature set, maturity, maintainer availability from Reaper or
> any other project being donated.
>
> Mick Semb Wever
> 6. We should not vote on these things and instead build consensus.
>
> Open Questions from this thread
> 7. What technical debts we are talking about in Reaper. Can someone give
> concrete examples.
> 8. What is the timeline of donating Reaper to Apache Cassandra.
>
> On Wed, Sep 12, 2018 at 3:49 PM sankalp kohli 
> wrote:
>
>
> (Using this thread and not the vote thread intentionally)
> For folks talking about vote being rushed. I would use the email from
> Joseph to show this is not rushed. There was no email on this thread for 4
> months until I pinged.
>
>
> Dec 2016: Vinay worked with Jon and Alex to try to collaborate on Reaper to
> come up with design goals for a repair scheduler that could work at Netflix
> scale.
>
> ~Feb 2017: Netflix believes that the fundamental design gaps prevented us
> from using Reaper as it relies heavily on remote JMX connections and
> central coordination.
>
> Sep. 2017: Vinay gives a lightning talk at NGCC about a highly available
> and distributed repair scheduling sidecar/tool. He is encouraged by
> multiple committers to build repair scheduling into the daemon 

Re: Proposing an Apache Cassandra Management process

[dinesh.jo...@yahoo.com.INVALID]

I have a few clarifications -
The scope of the management process is not to simply run repair scheduling. 
Repair scheduling is one of the many features we could implement or adopt from 
existing sources. So could we please split the Management Process discussion 
and the repair scheduling?
After re-reading the management process proposal, I see we missed to 
communicate a basic idea in the document. We wanted to take a pluggable 
approach to various activities that the management process could perform. This 
could accommodate different implementations of common activities such as 
repair. The management process would provide the basic framework and it would 
have default implementations for some of the basic activities. This would allow 
for speedier iteration cycles and keep things extensible.
Turning to some questions that Jon and others have raised, when I +1, my 
intention is to fully contribute and stay with this community. That said, 
things feel rushed for some but for me it feels like analysis paralysis. We're 
looking for actionable feedback and to discuss the management process _not_ 
repair scheduling solutions.
Thanks,
Dinesh



On Sep 12, 2018, at 6:24 PM, sankalp kohli  wrote:
Here is a list of open discussion points from the voting thread. I think
some are already answered but I will still gather these questions here.

>From several people:
1. Vote is rushed and we need more time for discussion.

>From Sylvain
2. About the voting process...I think that was addressed by Jeff Jirsa and
deserves a separate thread as it is not directly related to this thread.
3. Does the project need a side car.

>From Jonathan Haddad
4. Are people doing +1 willing to contribute

>From Jonathan Ellis
5. List of feature set, maturity, maintainer availability from Reaper or
any other project being donated.

Mick Semb Wever
6. We should not vote on these things and instead build consensus.

Open Questions from this thread
7. What technical debts we are talking about in Reaper. Can someone give
concrete examples.
8. What is the timeline of donating Reaper to Apache Cassandra.

On Wed, Sep 12, 2018 at 3:49 PM sankalp kohli 
wrote:


(Using this thread and not the vote thread intentionally)
For folks talking about vote being rushed. I would use the email from
Joseph to show this is not rushed. There was no email on this thread for 4
months until I pinged.


Dec 2016: Vinay worked with Jon and Alex to try to collaborate on Reaper to
come up with design goals for a repair scheduler that could work at Netflix
scale.

~Feb 2017: Netflix believes that the fundamental design gaps prevented us
from using Reaper as it relies heavily on remote JMX connections and
central coordination.

Sep. 2017: Vinay gives a lightning talk at NGCC about a highly available
and distributed repair scheduling sidecar/tool. He is encouraged by
multiple committers to build repair scheduling into the daemon itself and
not as a sidecar so the database is truly eventually consistent.

~Jun. 2017 - Feb. 2018: Based on internal need and the positive feedback at
NGCC, Vinay and myself prototype the distributed repair scheduler within
Priam and roll it out at Netflix scale.

Mar. 2018: I open a Jira (CASSANDRA-14346) along with a detailed 20 page
design document for adding repair scheduling to the daemon itself and open
the design up for feedback from the community. We get feedback from Alex,
Blake, Nate, Stefan, and Mick. As far as I know there were zero proposals
to contribute Reaper at this point. We hear the consensus that the
community would prefer repair scheduling in a separate distributed sidecar
rather than in the daemon itself and we re-work the design to match this
consensus, re-aligning with our original proposal at NGCC.

Apr 2018: Blake brings the discussion of repair scheduling to the dev list
(

https://lists.apache.org/thread.html/760fbef677f27aa5c2ab4c375c7efeb81304fea428deff986ba1c2eb@%3Cdev.cassandra.apache.org%3E
).
Many community members give positive feedback that we should solve it as
part of Cassandra and there is still no mention of contributing Reaper at
this point. The last message is my attempted summary giving context on how
we want to take the best of all the sidecars (OpsCenter, Priam, Reaper) and
ship them with Cassandra.

Apr. 2018: Dinesh opens CASSANDRA-14395 along with a public design document
for gathering feedback on a general management sidecar. Sankalp and Dinesh
encourage Vinay and myself to kickstart that sidecar using the repair
scheduler patch

Apr 2018: Dinesh reaches out to the dev list (

https://lists.apache.org/thread.html/a098341efd8f344494bcd2761dba5125e971b59b1dd54f282ffda253@%3Cdev.cassandra.apache.org%3E
)
about the general management process to gain further feedback. All feedback
remains positive as it is a potential place for multiple community members
to contribute their various sidecar functionality.

May-Jul 2017: Vinay and I work on creating a basic sidecar for running the
repair scheduler 

Re: Proposing an Apache Cassandra Management process

[Blake Eggleston]

Reading through the history Sankalp posted (I think it was originally posted by 
Joey?), I think part of the problem we’re having here is that we’re trying to 
solve at least 3 problems with a single solution. Also, I don’t think everyone 
has the same goals in mind. The issues we’re trying to solve are:


Repair scheduling - original proposal was for an in-process distributed 
scheduler to make cassandra eventually consistent without relying on external 
tools.

Sidecar - proposed as a helper co-process to make stuff like cluster wide 
nodetool command execution, health check, etc easier. I don’t think the 
original proposal mentioned repair.

Ops center like management application with a ui seems to have made it’s way 
into the mix at some point

These are all intended to make cassandra easier to operate, but they’re really 
separate features. It would be more productive to focus on each one as it’s own 
feature instead of trying to design a one size fits all and does everything 
management tool.

On September 12, 2018 at 6:25:11 PM, sankalp kohli (kohlisank...@gmail.com) 
wrote:

Here is a list of open discussion points from the voting thread. I think  
some are already answered but I will still gather these questions here.  

From several people:  
1. Vote is rushed and we need more time for discussion.  

From Sylvain  
2. About the voting process...I think that was addressed by Jeff Jirsa and  
deserves a separate thread as it is not directly related to this thread.  
3. Does the project need a side car.  

From Jonathan Haddad  
4. Are people doing +1 willing to contribute  

From Jonathan Ellis  
5. List of feature set, maturity, maintainer availability from Reaper or  
any other project being donated.  

Mick Semb Wever  
6. We should not vote on these things and instead build consensus.  

Open Questions from this thread  
7. What technical debts we are talking about in Reaper. Can someone give  
concrete examples.  
8. What is the timeline of donating Reaper to Apache Cassandra.  

On Wed, Sep 12, 2018 at 3:49 PM sankalp kohli   
wrote:  

> (Using this thread and not the vote thread intentionally)  
> For folks talking about vote being rushed. I would use the email from  
> Joseph to show this is not rushed. There was no email on this thread for 4  
> months until I pinged.  
>  
>  
> Dec 2016: Vinay worked with Jon and Alex to try to collaborate on Reaper to  
> come up with design goals for a repair scheduler that could work at Netflix  
> scale.  
>  
> ~Feb 2017: Netflix believes that the fundamental design gaps prevented us  
> from using Reaper as it relies heavily on remote JMX connections and  
> central coordination.  
>  
> Sep. 2017: Vinay gives a lightning talk at NGCC about a highly available  
> and distributed repair scheduling sidecar/tool. He is encouraged by  
> multiple committers to build repair scheduling into the daemon itself and  
> not as a sidecar so the database is truly eventually consistent.  
>  
> ~Jun. 2017 - Feb. 2018: Based on internal need and the positive feedback at  
> NGCC, Vinay and myself prototype the distributed repair scheduler within  
> Priam and roll it out at Netflix scale.  
>  
> Mar. 2018: I open a Jira (CASSANDRA-14346) along with a detailed 20 page  
> design document for adding repair scheduling to the daemon itself and open  
> the design up for feedback from the community. We get feedback from Alex,  
> Blake, Nate, Stefan, and Mick. As far as I know there were zero proposals  
> to contribute Reaper at this point. We hear the consensus that the  
> community would prefer repair scheduling in a separate distributed sidecar  
> rather than in the daemon itself and we re-work the design to match this  
> consensus, re-aligning with our original proposal at NGCC.  
>  
> Apr 2018: Blake brings the discussion of repair scheduling to the dev list  
> (  
>  
> https://lists.apache.org/thread.html/760fbef677f27aa5c2ab4c375c7efeb81304fea428deff986ba1c2eb@%3Cdev.cassandra.apache.org%3E
>   
> ).  
> Many community members give positive feedback that we should solve it as  
> part of Cassandra and there is still no mention of contributing Reaper at  
> this point. The last message is my attempted summary giving context on how  
> we want to take the best of all the sidecars (OpsCenter, Priam, Reaper) and  
> ship them with Cassandra.  
>  
> Apr. 2018: Dinesh opens CASSANDRA-14395 along with a public design document  
> for gathering feedback on a general management sidecar. Sankalp and Dinesh  
> encourage Vinay and myself to kickstart that sidecar using the repair  
> scheduler patch  
>  
> Apr 2018: Dinesh reaches out to the dev list (  
>  
> https://lists.apache.org/thread.html/a098341efd8f344494bcd2761dba5125e971b59b1dd54f282ffda253@%3Cdev.cassandra.apache.org%3E
>   
> )  
> about the general management process to gain further feedback. All feedback  
> remains positive as it is a potential place for multiple community members  
> to 

Re: Proposing an Apache Cassandra Management process

[sankalp kohli]

(Using this thread and not the vote thread intentionally)
For folks talking about vote being rushed. I would use the email from
Joseph to show this is not rushed. There was no email on this thread for 4
months until I pinged.


Dec 2016: Vinay worked with Jon and Alex to try to collaborate on Reaper to
come up with design goals for a repair scheduler that could work at Netflix
scale.

~Feb 2017: Netflix believes that the fundamental design gaps prevented us
from using Reaper as it relies heavily on remote JMX connections and
central coordination.

Sep. 2017: Vinay gives a lightning talk at NGCC about a highly available
and distributed repair scheduling sidecar/tool. He is encouraged by
multiple committers to build repair scheduling into the daemon itself and
not as a sidecar so the database is truly eventually consistent.

~Jun. 2017 - Feb. 2018: Based on internal need and the positive feedback at
NGCC, Vinay and myself prototype the distributed repair scheduler within
Priam and roll it out at Netflix scale.

Mar. 2018: I open a Jira (CASSANDRA-14346) along with a detailed 20 page
design document for adding repair scheduling to the daemon itself and open
the design up for feedback from the community. We get feedback from Alex,
Blake, Nate, Stefan, and Mick. As far as I know there were zero proposals
to contribute Reaper at this point. We hear the consensus that the
community would prefer repair scheduling in a separate distributed sidecar
rather than in the daemon itself and we re-work the design to match this
consensus, re-aligning with our original proposal at NGCC.

Apr 2018: Blake brings the discussion of repair scheduling to the dev list (
https://lists.apache.org/thread.html/760fbef677f27aa5c2ab4c375c7efeb81304fea428deff986ba1c2eb@%3Cdev.cassandra.apache.org%3E
).
Many community members give positive feedback that we should solve it as
part of Cassandra and there is still no mention of contributing Reaper at
this point. The last message is my attempted summary giving context on how
we want to take the best of all the sidecars (OpsCenter, Priam, Reaper) and
ship them with Cassandra.

Apr. 2018: Dinesh opens CASSANDRA-14395 along with a public design document
for gathering feedback on a general management sidecar. Sankalp and Dinesh
encourage Vinay and myself to kickstart that sidecar using the repair
scheduler patch

Apr 2018: Dinesh reaches out to the dev list (
https://lists.apache.org/thread.html/a098341efd8f344494bcd2761dba5125e971b59b1dd54f282ffda253@%3Cdev.cassandra.apache.org%3E
)
about the general management process to gain further feedback. All feedback
remains positive as it is a potential place for multiple community members
to contribute their various sidecar functionality.

May-Jul 2017: Vinay and I work on creating a basic sidecar for running the
repair scheduler based on the feedback from the community in
CASSANDRA-14346 and CASSANDRA-14395

Jun 2018: I bump CASSANDRA-14346 indicating we're still working on this,
nobody objects

Jul 2018: Sankalp asks on the dev list if anyone has feature Jiras anyone
needs review for before 4.0, I mention again that we've nearly got the
basic sidecar and repair scheduling work done and will need help with
review. No one responds.

Aug 2018: We submit a patch that brings a basic distributed sidecar and
robust distributed repair to Cassandra itself. Dinesh mentions that he will
try to review. Now folks appear concerned about it being in tree and
instead maybe it should go in a different repo all together. I don't think
we have consensus on the repo choice yet.

On Sun, Sep 9, 2018 at 9:13 AM sankalp kohli  wrote:

> I agree with Jon and I think folks who are talking about tech debts in
> Reaper should elaborate with examples about these tech debts. Can we be
> more precise and list them down? I see it spread out over this long email
> thread!!
>
> On Sun, Sep 9, 2018 at 6:29 AM Elliott Sims  wrote:
>
>> A big one to add to your list there, IMO as a user:
>> * API for determining detailed repair state (and history?).  Essentially,
>> something beyond just "Is some sort of repair running?" so that tools like
>> Reaper can parallelize better.
>>
>> On Sun, Sep 9, 2018 at 8:30 AM, Stefan Podkowinski 
>> wrote:
>>
>> > Does it have to be a single project with functionality provided by
>> > multiple plugins? Designing a plugin API at this point seems to be a bit
>> > early and comes with additional complexity around managing plugins in
>> > general.
>> >
>> > I was more thinking into the direction of: "what can we do to enable
>> > people to create any kind of side car or tooling solution?". Thinks
>> like:
>> >
>> > Common cluster discovery and management API
>> > * Detect local Cassandra processes
>> > * Discover and receive events on cluster topology
>> > * Get assigned tokens for nodes
>> > * Read node configuration
>> > * Health checks (as already proposed)
>> >
>> > Any side cars should be easy to install on nodes that already run
>> Cassandra
>> > * 

Re: Proposing an Apache Cassandra Management process

[sankalp kohli]

I agree with Jon and I think folks who are talking about tech debts in
Reaper should elaborate with examples about these tech debts. Can we be
more precise and list them down? I see it spread out over this long email
thread!!

On Sun, Sep 9, 2018 at 6:29 AM Elliott Sims  wrote:

> A big one to add to your list there, IMO as a user:
> * API for determining detailed repair state (and history?).  Essentially,
> something beyond just "Is some sort of repair running?" so that tools like
> Reaper can parallelize better.
>
> On Sun, Sep 9, 2018 at 8:30 AM, Stefan Podkowinski 
> wrote:
>
> > Does it have to be a single project with functionality provided by
> > multiple plugins? Designing a plugin API at this point seems to be a bit
> > early and comes with additional complexity around managing plugins in
> > general.
> >
> > I was more thinking into the direction of: "what can we do to enable
> > people to create any kind of side car or tooling solution?". Thinks like:
> >
> > Common cluster discovery and management API
> > * Detect local Cassandra processes
> > * Discover and receive events on cluster topology
> > * Get assigned tokens for nodes
> > * Read node configuration
> > * Health checks (as already proposed)
> >
> > Any side cars should be easy to install on nodes that already run
> Cassandra
> > * Scripts for packaging (tar, deb, rpm)
> > * Templates for systemd support, optionally with auto-startup dependency
> > on the Cassandra main process
> >
> > Integration testing
> > * Provide basic testing framework for mocking cluster state and messages
> >
> > Support for other languages / avoid having to use JMX
> > * JMX bridge (HTTP? gRPC?, already implemented in #14346?)
> >
> > Obviously the whole side car discussion is not moving into a direction
> > everyone's happy with. Would it be an option to take a step back and
> > start implementing such a tooling framework with scripts and libraries
> > for the features described above, as a small GitHub project, instead of
> > putting an existing side-car solution up for vote? If that would work
> > and we get people collaborating on code shared between existing
> > side-cars, then we could take the next step and think about either
> > revisit the "official Cassandra side-car" topic, or add the created
> > client tooling framework as official sub-project to the Cassandra
> > project (maybe via Apache incubator).
> >
> >
> > On 08.09.18 02:49, Joseph Lynch wrote:
> > > On Fri, Sep 7, 2018 at 5:03 PM Jonathan Haddad 
> > wrote:
> > >> We haven’t even defined any requirements for an admin tool. It’s hard
> to
> > >> make a case for anything without agreement on what we’re trying to
> > build.
> > >>
> > > We were/are trying to sketch out scope/requirements in the #14395 and
> > > #14346 tickets as well as their associated design documents. I think
> > > the general proposed direction is a distributed 1:1 management sidecar
> > > process similar in architecture to Netflix's Priam except explicitly
> > > built to be general and pluggable by anyone rather than tightly
> > > coupled to AWS.
> > >
> > > Dinesh, Vinay and I were aiming for low amounts of scope at first and
> > > take things in an iterative approach with just enough upfront design
> > > but not so much we are unable to make any progress at all. For example
> > > maybe something like:
> > >
> > > 1. Get a super simple and non controversial sidecar process that ships
> > > with Cassandra and exposes a lightweight HTTP interface to e.g. some
> > > basic JMX endpoints
> > > 2a. Add a pluggable execution engine for cron/oneshot/scheduled jobs
> > > with the basic interfaces and state store and such
> > > 2b. Start scoping and implementing the full HTTP interface, e.g.
> > > backup status, cluster health status, etc ...
> > > 3a. Start integrating implementations of the jobs from 2a such as
> > > snapshot, backup, cluster restart, daemon + sstable upgrade, repair,
> > > etc
> > > 3b. Start integrating UI components that pair with the HTTP interface
> > from 2b
> > > 4. ?? Perhaps start unlocking next generation operations like moving
> > > "background" activities like compaction, streaming, repair etc into
> > > one or more sidecar contained processes to ensure the main daemon only
> > > handles read+write requests
> > >
> > > There are going to be a lot of questions to answer, and I think trying
> > > to answer them all up front will mean that we get nowhere or make
> > > unfortunate compromises that cripple the project from the start. If
> > > people think we need to do more design and discussion than we have
> > > been doing then we can spend more time on the design, but personally
> > > I'd rather start iterating on code and prove value incrementally. If
> > > it doesn't work out we won't release it GA to the community ...
> > >
> > > -Joey
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > For 

Re: Proposing an Apache Cassandra Management process

[Elliott Sims]

A big one to add to your list there, IMO as a user:
* API for determining detailed repair state (and history?).  Essentially,
something beyond just "Is some sort of repair running?" so that tools like
Reaper can parallelize better.

On Sun, Sep 9, 2018 at 8:30 AM, Stefan Podkowinski  wrote:

> Does it have to be a single project with functionality provided by
> multiple plugins? Designing a plugin API at this point seems to be a bit
> early and comes with additional complexity around managing plugins in
> general.
>
> I was more thinking into the direction of: "what can we do to enable
> people to create any kind of side car or tooling solution?". Thinks like:
>
> Common cluster discovery and management API
> * Detect local Cassandra processes
> * Discover and receive events on cluster topology
> * Get assigned tokens for nodes
> * Read node configuration
> * Health checks (as already proposed)
>
> Any side cars should be easy to install on nodes that already run Cassandra
> * Scripts for packaging (tar, deb, rpm)
> * Templates for systemd support, optionally with auto-startup dependency
> on the Cassandra main process
>
> Integration testing
> * Provide basic testing framework for mocking cluster state and messages
>
> Support for other languages / avoid having to use JMX
> * JMX bridge (HTTP? gRPC?, already implemented in #14346?)
>
> Obviously the whole side car discussion is not moving into a direction
> everyone's happy with. Would it be an option to take a step back and
> start implementing such a tooling framework with scripts and libraries
> for the features described above, as a small GitHub project, instead of
> putting an existing side-car solution up for vote? If that would work
> and we get people collaborating on code shared between existing
> side-cars, then we could take the next step and think about either
> revisit the "official Cassandra side-car" topic, or add the created
> client tooling framework as official sub-project to the Cassandra
> project (maybe via Apache incubator).
>
>
> On 08.09.18 02:49, Joseph Lynch wrote:
> > On Fri, Sep 7, 2018 at 5:03 PM Jonathan Haddad 
> wrote:
> >> We haven’t even defined any requirements for an admin tool. It’s hard to
> >> make a case for anything without agreement on what we’re trying to
> build.
> >>
> > We were/are trying to sketch out scope/requirements in the #14395 and
> > #14346 tickets as well as their associated design documents. I think
> > the general proposed direction is a distributed 1:1 management sidecar
> > process similar in architecture to Netflix's Priam except explicitly
> > built to be general and pluggable by anyone rather than tightly
> > coupled to AWS.
> >
> > Dinesh, Vinay and I were aiming for low amounts of scope at first and
> > take things in an iterative approach with just enough upfront design
> > but not so much we are unable to make any progress at all. For example
> > maybe something like:
> >
> > 1. Get a super simple and non controversial sidecar process that ships
> > with Cassandra and exposes a lightweight HTTP interface to e.g. some
> > basic JMX endpoints
> > 2a. Add a pluggable execution engine for cron/oneshot/scheduled jobs
> > with the basic interfaces and state store and such
> > 2b. Start scoping and implementing the full HTTP interface, e.g.
> > backup status, cluster health status, etc ...
> > 3a. Start integrating implementations of the jobs from 2a such as
> > snapshot, backup, cluster restart, daemon + sstable upgrade, repair,
> > etc
> > 3b. Start integrating UI components that pair with the HTTP interface
> from 2b
> > 4. ?? Perhaps start unlocking next generation operations like moving
> > "background" activities like compaction, streaming, repair etc into
> > one or more sidecar contained processes to ensure the main daemon only
> > handles read+write requests
> >
> > There are going to be a lot of questions to answer, and I think trying
> > to answer them all up front will mean that we get nowhere or make
> > unfortunate compromises that cripple the project from the start. If
> > people think we need to do more design and discussion than we have
> > been doing then we can spend more time on the design, but personally
> > I'd rather start iterating on code and prove value incrementally. If
> > it doesn't work out we won't release it GA to the community ...
> >
> > -Joey
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Proposing an Apache Cassandra Management process

[Stefan Podkowinski]

Does it have to be a single project with functionality provided by
multiple plugins? Designing a plugin API at this point seems to be a bit
early and comes with additional complexity around managing plugins in
general.

I was more thinking into the direction of: "what can we do to enable
people to create any kind of side car or tooling solution?". Thinks like:

Common cluster discovery and management API
* Detect local Cassandra processes
* Discover and receive events on cluster topology
* Get assigned tokens for nodes
* Read node configuration
* Health checks (as already proposed)

Any side cars should be easy to install on nodes that already run Cassandra
* Scripts for packaging (tar, deb, rpm)
* Templates for systemd support, optionally with auto-startup dependency
on the Cassandra main process

Integration testing
* Provide basic testing framework for mocking cluster state and messages

Support for other languages / avoid having to use JMX
* JMX bridge (HTTP? gRPC?, already implemented in #14346?)

Obviously the whole side car discussion is not moving into a direction
everyone's happy with. Would it be an option to take a step back and
start implementing such a tooling framework with scripts and libraries
for the features described above, as a small GitHub project, instead of
putting an existing side-car solution up for vote? If that would work
and we get people collaborating on code shared between existing
side-cars, then we could take the next step and think about either
revisit the "official Cassandra side-car" topic, or add the created
client tooling framework as official sub-project to the Cassandra
project (maybe via Apache incubator).


On 08.09.18 02:49, Joseph Lynch wrote:
> On Fri, Sep 7, 2018 at 5:03 PM Jonathan Haddad  wrote:
>> We haven’t even defined any requirements for an admin tool. It’s hard to
>> make a case for anything without agreement on what we’re trying to build.
>>
> We were/are trying to sketch out scope/requirements in the #14395 and
> #14346 tickets as well as their associated design documents. I think
> the general proposed direction is a distributed 1:1 management sidecar
> process similar in architecture to Netflix's Priam except explicitly
> built to be general and pluggable by anyone rather than tightly
> coupled to AWS.
>
> Dinesh, Vinay and I were aiming for low amounts of scope at first and
> take things in an iterative approach with just enough upfront design
> but not so much we are unable to make any progress at all. For example
> maybe something like:
>
> 1. Get a super simple and non controversial sidecar process that ships
> with Cassandra and exposes a lightweight HTTP interface to e.g. some
> basic JMX endpoints
> 2a. Add a pluggable execution engine for cron/oneshot/scheduled jobs
> with the basic interfaces and state store and such
> 2b. Start scoping and implementing the full HTTP interface, e.g.
> backup status, cluster health status, etc ...
> 3a. Start integrating implementations of the jobs from 2a such as
> snapshot, backup, cluster restart, daemon + sstable upgrade, repair,
> etc
> 3b. Start integrating UI components that pair with the HTTP interface from 2b
> 4. ?? Perhaps start unlocking next generation operations like moving
> "background" activities like compaction, streaming, repair etc into
> one or more sidecar contained processes to ensure the main daemon only
> handles read+write requests
>
> There are going to be a lot of questions to answer, and I think trying
> to answer them all up front will mean that we get nowhere or make
> unfortunate compromises that cripple the project from the start. If
> people think we need to do more design and discussion than we have
> been doing then we can spend more time on the design, but personally
> I'd rather start iterating on code and prove value incrementally. If
> it doesn't work out we won't release it GA to the community ...
>
> -Joey
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Jonathan Haddad]

It's interesting to me that someone would consider features of Reaper as
"technical debt"... an odd way to word it.  When we had proposed donating
Reaper to the project, I had made the assumption people would realize the
benefit of folding in a successful project.  A working codebase with a
large user base gives us an immediate tool, and it would let us migrate
everyone from using our tool to using an official tool.

>From the discussions we've had so far, nobody except Blake seems to care
about those users.  To me, the highest priority of this process is
considering what's best for the Cassandra community.  Providing an easy,
clear migration path forward for the thousands of Reaper users, supporting
all 4 backends and cassandra versions is the logical the way of
transitioning everyone to using the official tool.

If supporting everyone using 2.x with a postgres backend for storing
schedules isn't something anyone cares about, there's not much benefit from
using Reaper.  Gutting a bunch of the project won't help people migrate
over, and if we're not migrating everyone over, then we (TLP) will still
have to maintain Reaper.  That means we'll be indefinitely maintaining a
fork of the official admin, and probably not contributing to the main one,
our time is limited.  That's exactly what's going to happen if we start
with anything other than Reaper.  We've got a lot of teams asking for
features, and if people pay us to keep working on Reaper, we'll be doing
that.

So, with that in mind, if we're putting this to a vote, I'd like to be
clear - taking Reaper as-is about community - not it's technical prowess.
If we're not enabling users to move off our version and directly to the
official tool, I don't see a point in using Reaper at all except as a
reference to correctly run subrange repair and have a nice UI.  There's
plenty of things I'd do differently if I built it from scratch, I'm not
putting the codebase up on a pedestal.

This entire discussion has been incredibly frustrating for me, considering
we're talking about building an alternative to a well tested and widely
deployed codebase that likely won't produce anything of value for at least
a year (Cassandra 5?).  I'm also pretty sure most of the folks that have
cried out "tech debt" have spent less than 5 minutes looking at the
codebase.  I hope your enthusiasm at the moment would carry over if you
build a tech-debt free admin tool from the ground up.

TL;DR: If nobody cares about the Reaper community (which is a very large
subset of the cassandra community) there's no reason to start with Reaper,
it's not about the tech it's about the people.

Jon


On Sat, Sep 8, 2018 at 4:57 PM sankalp kohli  wrote:

> Most of the discussions have been around whether we take some side car or
> do a cherry pick approach where we add a feature at a time to side car and
> use the best implementation.
> We have been discussing this first in April and now for several days. I
> think we all want some progress here. We will start a vote soon..may be
> next week to decide which approach we will take. I don't see any other
> option to make progress besides a vote!!
>
> PS: I think these discussions are very good for the community and it shows
> people care about Apache Cassandra and it is a sign of healthy community.
> Several people offering to donate the side car or help build is showing the
> interest everyone has in it.
>
>
> On Sat, Sep 8, 2018 at 11:17 AM Joseph Lynch 
> wrote:
>
> > On Fri, Sep 7, 2018 at 10:00 PM Blake Eggleston 
> > wrote:
> > >
> > > Right, I understand the arguments for starting a new project. I’m not
> > saying reaper is, technically speaking, the best place to start. The
> point
> > I’m trying to make is that the non-technical advantages of using an
> > existing project as a starting point may outweigh the technical benefits
> of
> > a clean slate. Whether that’s the case or not, it’s not a strictly
> > technical decision, and the non-technical advantages of starting with
> > reaper need to be weighed.
> > >
> >
> > Technical debt doesn't just refer to the technical solutions, having
> > an existing user base means that a project has made promises in the
> > past (in the case of Priam 5+ years ago) that the new project would
> > have to keep if we make keeping users of those sidecars a goal (which
> > for the record I don't think should be a goal, I think the goal is to
> > make Cassandra the database work out of the box in the 4.x series).
> >
> > For example, Priam has to continue supporting the following as users
> > actively use them (including Netflix):
> > * Parallel token assignment and creation allowing parallel bootstrap
> > and parallel doubling of hundred node clusters at once (as long as you
> > use single tokens and run in AWS with asgs).
> > * 3+ backup solutions, as well as assumptions about where in e.g. S3
> > backups are present and what format they're present in.
> > * Numerous configuration options and UI elements (mostly 5 

Re: Proposing an Apache Cassandra Management process

[sankalp kohli]

Most of the discussions have been around whether we take some side car or
do a cherry pick approach where we add a feature at a time to side car and
use the best implementation.
We have been discussing this first in April and now for several days. I
think we all want some progress here. We will start a vote soon..may be
next week to decide which approach we will take. I don't see any other
option to make progress besides a vote!!

PS: I think these discussions are very good for the community and it shows
people care about Apache Cassandra and it is a sign of healthy community.
Several people offering to donate the side car or help build is showing the
interest everyone has in it.


On Sat, Sep 8, 2018 at 11:17 AM Joseph Lynch  wrote:

> On Fri, Sep 7, 2018 at 10:00 PM Blake Eggleston 
> wrote:
> >
> > Right, I understand the arguments for starting a new project. I’m not
> saying reaper is, technically speaking, the best place to start. The point
> I’m trying to make is that the non-technical advantages of using an
> existing project as a starting point may outweigh the technical benefits of
> a clean slate. Whether that’s the case or not, it’s not a strictly
> technical decision, and the non-technical advantages of starting with
> reaper need to be weighed.
> >
>
> Technical debt doesn't just refer to the technical solutions, having
> an existing user base means that a project has made promises in the
> past (in the case of Priam 5+ years ago) that the new project would
> have to keep if we make keeping users of those sidecars a goal (which
> for the record I don't think should be a goal, I think the goal is to
> make Cassandra the database work out of the box in the 4.x series).
>
> For example, Priam has to continue supporting the following as users
> actively use them (including Netflix):
> * Parallel token assignment and creation allowing parallel bootstrap
> and parallel doubling of hundred node clusters at once (as long as you
> use single tokens and run in AWS with asgs).
> * 3+ backup solutions, as well as assumptions about where in e.g. S3
> backups are present and what format they're present in.
> * Numerous configuration options and UI elements (mostly 5 year old JSON
> APIs)
> * Support for Cassandra 2.0, 2.1, 2.2, 3.0, 3.11 and soon 4.0
>
> Reaper has to continue supporting the following as users actively use them:
> * Postgres and h2 storage backends. These were the original storage
> engines and users may not have (probably haven't?) migrated to the
> relatively recently added Cassandra backend (which is probably the
> only one an official sidecar should support imo).
> * The three historical modes of running Reaper [1] all of which
> involve remote JMX (disallowed by many companies security policies
> including Netflix's) and none of which are a sidecar design (although
> Mick says we can add that back in, at which point it has to support
> four).
> * Numerous configuration options and UI elements (e.g. a UI around a
> single Reaper instance controlling many Cassandra clusters instead of
> each cluster having a separate UI more consistent with how Cassandra
> architecture works)
> * Support for Cassandra 2.2, 3.0, 3.11, and soon 4.0
>
> [1] http://cassandra-reaper.io/docs/usage/multi_dc/
> [2] https://github.com/hashbrowncipher/cassandra-mirror
>
> We can't "get the community" of these sidecars and drop support for
> 90+% of the supported configurations and features at the same time ...
> These projects have made promises to users, as per the name technical
> debt these choices made over years have explicit costs that we have to
> take into account if we accept a project as is with the goal of taking
> the community with us. If we don't have the goal of taking the
> existing community with us and are instead aiming to simply make
> Cassandra work out of the box without external tools, then we don't
> have to continue fulfilling these promises.
>
> Instead I think the organizations that made those promises (TLP and
> Netflix in these specific cases) should continue keeping them, and the
> Cassandra management process should be incrementally built by the
> Cassandra community with decisions made as a community and we only GA
> it when the PMC/community is happy with making a promise of support
> for the features that we've merged (and since we're starting from a
> fresh start if people have strong opinions about fundamental
> architecture we can try to take those into account like we did with
> the months of feedback on repair scheduling
> runtime/architecture/design). If we can't prove value over other
> community tools for running 4.x, which is definitely a possibility,
> then we don't mark the management process GA, we re-focus on
> individual community tools, and imo failure is a reasonable result of
> attempted innovation.
>
> -Joey
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional 

Re: Proposing an Apache Cassandra Management process

[Joseph Lynch]

On Fri, Sep 7, 2018 at 10:00 PM Blake Eggleston  wrote:
>
> Right, I understand the arguments for starting a new project. I’m not saying 
> reaper is, technically speaking, the best place to start. The point I’m 
> trying to make is that the non-technical advantages of using an existing 
> project as a starting point may outweigh the technical benefits of a clean 
> slate. Whether that’s the case or not, it’s not a strictly technical 
> decision, and the non-technical advantages of starting with reaper need to be 
> weighed.
>

Technical debt doesn't just refer to the technical solutions, having
an existing user base means that a project has made promises in the
past (in the case of Priam 5+ years ago) that the new project would
have to keep if we make keeping users of those sidecars a goal (which
for the record I don't think should be a goal, I think the goal is to
make Cassandra the database work out of the box in the 4.x series).

For example, Priam has to continue supporting the following as users
actively use them (including Netflix):
* Parallel token assignment and creation allowing parallel bootstrap
and parallel doubling of hundred node clusters at once (as long as you
use single tokens and run in AWS with asgs).
* 3+ backup solutions, as well as assumptions about where in e.g. S3
backups are present and what format they're present in.
* Numerous configuration options and UI elements (mostly 5 year old JSON APIs)
* Support for Cassandra 2.0, 2.1, 2.2, 3.0, 3.11 and soon 4.0

Reaper has to continue supporting the following as users actively use them:
* Postgres and h2 storage backends. These were the original storage
engines and users may not have (probably haven't?) migrated to the
relatively recently added Cassandra backend (which is probably the
only one an official sidecar should support imo).
* The three historical modes of running Reaper [1] all of which
involve remote JMX (disallowed by many companies security policies
including Netflix's) and none of which are a sidecar design (although
Mick says we can add that back in, at which point it has to support
four).
* Numerous configuration options and UI elements (e.g. a UI around a
single Reaper instance controlling many Cassandra clusters instead of
each cluster having a separate UI more consistent with how Cassandra
architecture works)
* Support for Cassandra 2.2, 3.0, 3.11, and soon 4.0

[1] http://cassandra-reaper.io/docs/usage/multi_dc/
[2] https://github.com/hashbrowncipher/cassandra-mirror

We can't "get the community" of these sidecars and drop support for
90+% of the supported configurations and features at the same time ...
These projects have made promises to users, as per the name technical
debt these choices made over years have explicit costs that we have to
take into account if we accept a project as is with the goal of taking
the community with us. If we don't have the goal of taking the
existing community with us and are instead aiming to simply make
Cassandra work out of the box without external tools, then we don't
have to continue fulfilling these promises.

Instead I think the organizations that made those promises (TLP and
Netflix in these specific cases) should continue keeping them, and the
Cassandra management process should be incrementally built by the
Cassandra community with decisions made as a community and we only GA
it when the PMC/community is happy with making a promise of support
for the features that we've merged (and since we're starting from a
fresh start if people have strong opinions about fundamental
architecture we can try to take those into account like we did with
the months of feedback on repair scheduling
runtime/architecture/design). If we can't prove value over other
community tools for running 4.x, which is definitely a possibility,
then we don't mark the management process GA, we re-focus on
individual community tools, and imo failure is a reasonable result of
attempted innovation.

-Joey

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Blake Eggleston]

Right, I understand the arguments for starting a new project. I’m not saying 
reaper is, technically speaking, the best place to start. The point I’m trying 
to make is that the non-technical advantages of using an existing project as a 
starting point may outweigh the technical benefits of a clean slate. Whether 
that’s the case or not, it’s not a strictly technical decision, and the 
non-technical advantages of starting with reaper need to be weighed.

On September 7, 2018 at 8:19:50 PM, Jeff Jirsa (jji...@gmail.com) wrote:

The benefit is that it more closely matched the design doc, from 5 months ago, 
which is decidedly not about coordinating repair - it’s about a general purpose 
management tool, where repair is one of many proposed tasks  

https://docs.google.com/document/d/1UV9pE81NaIUF3g4L1wxq09nT11AkSQcMijgLFwGsY3s/edit
  


By starting with a tool that is built to run repair, you’re sacrificing 
generality and accepting something purpose built for one sub task. It’s an 
important subtask, and it’s a nice tool, but it’s not an implementation of the 
proposal, it’s an alternative that happens to do some of what was proposed.  

--  
Jeff Jirsa  


> On Sep 7, 2018, at 6:53 PM, Blake Eggleston  wrote:  
>  
> What’s the benefit of doing it that way vs starting with reaper and 
> integrating the netflix scheduler? If reaper was just a really inappropriate 
> choice for the cassandra management process, I could see that being a better 
> approach, but I don’t think that’s the case.  
>  
> If our management process isn’t a drop in replacement for reaper, then reaper 
> will continue to exist, which will split the user and developers base between 
> the 2 projects. That won't be good for either project.  
>  
> On September 7, 2018 at 6:12:01 PM, Jeff Jirsa (jji...@gmail.com) wrote:  
>  
> I’d also like to see the end state you describe: reaper UI wrapping the 
> Netflix management process with pluggable scheduling (either as is with 
> reaper now, or using the Netflix scheduler), but I don’t think that means we 
> need to start with reaper - if personally prefer the opposite direction, 
> starting with something small and isolated and layering on top.  
>  
> --  
> Jeff Jirsa  
>  
>  
>> On Sep 7, 2018, at 5:42 PM, Blake Eggleston  wrote:  
>>  
>> I think we should accept the reaper project as is and make that cassandra 
>> management process 1.0, then integrate the netflix scheduler (and other new 
>> features) into that.  
>>  
>> The ultimate goal would be for the netflix scheduler to become the default 
>> repair scheduler, but I think using reaper as the starting point makes it 
>> easier to get there.  
>>  
>> Reaper would bring a prod user base that would realistically take 2-3 years 
>> to build up with a new project. As an operator, switching to a cassandra 
>> management process that’s basically a re-brand of an existing and commonly 
>> used management process isn’t super risky. Asking operators to switch to a 
>> new process is a much harder sell.  
>>  
>> On September 7, 2018 at 4:17:10 PM, Jeff Jirsa (jji...@gmail.com) wrote:  
>>  
>> How can we continue moving this forward?  
>>  
>> Mick/Jon/TLP folks, is there a path here where we commit the  
>> Netflix-provided management process, and you augment Reaper to work with it? 
>>  
>> Is there a way we can make a larger umbrella that's modular that can  
>> support either/both?  
>> Does anyone believe there's a clear, objective argument that one is  
>> strictly better than the other? I haven't seen one.  
>>  
>>  
>>  
>> On Mon, Aug 20, 2018 at 4:14 PM Roopa Tangirala  
>>  wrote:  
>>  
>>> +1 to everything that Joey articulated with emphasis on the fact that  
>>> contributions should be evaluated based on the merit of code and their  
>>> value add to the whole offering. I hope it does not matter whether that  
>>> contribution comes from PMC member or a person who is not a committer. I  
>>> would like the process to be such that it encourages the new members to be  
>>> a part of the community and not shy away from contributing to the code  
>>> assuming their contributions are valued differently than committers or PMC  
>>> members. It would be sad to see the contributions decrease if we go down  
>>> that path.  
>>>  
>>> *Regards,*  
>>>  
>>> *Roopa Tangirala*  
>>>  
>>> Engineering Manager CDE  
>>>  
>>> *(408) 438-3156 - mobile*  
>>>  
>>>  
>>>  
>>>  
>>>  
>>>  
>>> On Mon, Aug 20, 2018 at 2:58 PM Joseph Lynch   
>>> wrote:  
>>>  
> We are looking to contribute Reaper to the Cassandra project.  
>  
 Just to clarify are you proposing contributing Reaper as a project via  
 donation or you are planning on contributing the features of Reaper as  
 patches to Cassandra? If the former how far along are you on the donation  
 process? If the latter, when do you think you would have patches ready  
>>> for  
 consideration / review?  
  
  
> Looking at the patch it's very 

Re: Proposing an Apache Cassandra Management process

[Jeff Jirsa]

The benefit is that it more closely matched the design doc, from 5 months ago, 
which is decidedly not about coordinating repair - it’s about a general purpose 
management tool, where repair is one of many proposed tasks

https://docs.google.com/document/d/1UV9pE81NaIUF3g4L1wxq09nT11AkSQcMijgLFwGsY3s/edit


By starting with a tool that is built to run repair, you’re sacrificing 
generality and accepting something purpose built for one sub task. It’s an 
important subtask, and it’s a nice tool, but it’s not an implementation of the 
proposal, it’s an alternative that happens to do some of what was proposed.

-- 
Jeff Jirsa


> On Sep 7, 2018, at 6:53 PM, Blake Eggleston  wrote:
> 
> What’s the benefit of doing it that way vs starting with reaper and 
> integrating the netflix scheduler? If reaper was just a really inappropriate 
> choice for the cassandra management process, I could see that being a better 
> approach, but I don’t think that’s the case.
> 
> If our management process isn’t a drop in replacement for reaper, then reaper 
> will continue to exist, which will split the user and developers base between 
> the 2 projects. That won't be good for either project.
> 
> On September 7, 2018 at 6:12:01 PM, Jeff Jirsa (jji...@gmail.com) wrote:
> 
> I’d also like to see the end state you describe: reaper UI wrapping the 
> Netflix management process with pluggable scheduling (either as is with 
> reaper now, or using the Netflix scheduler), but I don’t think that means we 
> need to start with reaper - if personally prefer the opposite direction, 
> starting with something small and isolated and layering on top.  
> 
> --  
> Jeff Jirsa  
> 
> 
>> On Sep 7, 2018, at 5:42 PM, Blake Eggleston  wrote:  
>> 
>> I think we should accept the reaper project as is and make that cassandra 
>> management process 1.0, then integrate the netflix scheduler (and other new 
>> features) into that.  
>> 
>> The ultimate goal would be for the netflix scheduler to become the default 
>> repair scheduler, but I think using reaper as the starting point makes it 
>> easier to get there.  
>> 
>> Reaper would bring a prod user base that would realistically take 2-3 years 
>> to build up with a new project. As an operator, switching to a cassandra 
>> management process that’s basically a re-brand of an existing and commonly 
>> used management process isn’t super risky. Asking operators to switch to a 
>> new process is a much harder sell.  
>> 
>> On September 7, 2018 at 4:17:10 PM, Jeff Jirsa (jji...@gmail.com) wrote:  
>> 
>> How can we continue moving this forward?  
>> 
>> Mick/Jon/TLP folks, is there a path here where we commit the  
>> Netflix-provided management process, and you augment Reaper to work with it? 
>>  
>> Is there a way we can make a larger umbrella that's modular that can  
>> support either/both?  
>> Does anyone believe there's a clear, objective argument that one is  
>> strictly better than the other? I haven't seen one.  
>> 
>> 
>> 
>> On Mon, Aug 20, 2018 at 4:14 PM Roopa Tangirala  
>>  wrote:  
>> 
>>> +1 to everything that Joey articulated with emphasis on the fact that  
>>> contributions should be evaluated based on the merit of code and their  
>>> value add to the whole offering. I hope it does not matter whether that  
>>> contribution comes from PMC member or a person who is not a committer. I  
>>> would like the process to be such that it encourages the new members to be  
>>> a part of the community and not shy away from contributing to the code  
>>> assuming their contributions are valued differently than committers or PMC  
>>> members. It would be sad to see the contributions decrease if we go down  
>>> that path.  
>>> 
>>> *Regards,*  
>>> 
>>> *Roopa Tangirala*  
>>> 
>>> Engineering Manager CDE  
>>> 
>>> *(408) 438-3156 - mobile*  
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Mon, Aug 20, 2018 at 2:58 PM Joseph Lynch   
>>> wrote:  
>>> 
> We are looking to contribute Reaper to the Cassandra project.  
> 
 Just to clarify are you proposing contributing Reaper as a project via  
 donation or you are planning on contributing the features of Reaper as  
 patches to Cassandra? If the former how far along are you on the donation  
 process? If the latter, when do you think you would have patches ready  
>>> for  
 consideration / review?  
 
 
> Looking at the patch it's very similar in its base design already, but  
> Reaper does has a lot more to offer. We have all been working hard to  
 move  
> it to also being a side-car so it can be contributed. This raises a  
 number  
> of relevant questions to this thread: would we then accept both works  
>>> in  
> the Cassandra project, and what burden would it put on the current PMC  
>>> to  
> maintain both works.  
> 
 I would hope that we would collaborate on merging the best parts of all  
 into the official Cassandra sidecar, taking the always on, shared 

Re: Proposing an Apache Cassandra Management process

[Joseph Lynch]

> What’s the benefit of doing it that way vs starting with reaper and 
> integrating the netflix scheduler? If reaper was just a really inappropriate 
> choice for the cassandra management process, I could see that being a better 
> approach, but I don’t think that’s the case.
>
The benefit, as Dinesh and I argued earlier, is starting without
technical debt (especially architectural technical debt) and taking
only the best components from the multiple community sidecars for the
Cassandra management sidecar. To be clear, I think Priam is much
closer to the proposed management sidecar than Reaper is (and Priam +
the repair scheduler has basically all proposed scope), but like I
said earlier in the other thread I don't think we should take Priam as
is due to technical debt and I don't think we should take Reaper
either. The community should learn from the many sidecars we've built
and solve the problem once inside the Cassandra sidecar.

> If our management process isn’t a drop in replacement for reaper, then reaper 
> will continue to exist, which will split the user and developers base between 
> the 2 projects. That won't be good for either project.
I think Reaper is a great repair tool for some infrastructures, but I
don't think the management sidecar is about running repairs. It's
about building a general purpose tool that may happen to run repairs
if someone chooses to use that particular plugin. To be honest I think
it's great that there are competing community repair tools ... this is
how we learn from all of them and build the simplest and most narrowly
tailored solution into the database itself...

-Joey

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Vinay Chella]

 > I think we should accept the reaper project as is and make that
cassandra management process 1.0, then integrate the Netflix scheduler (and
other new features) into that.
Integrating Netflix scheduler into reaper is mostly refactoring reaper code
since they are different architectures.

> Reaper would bring a prod user base that would realistically take 2-3
years to build up with a new project.
IMO, it is great if we have that, but this should not be the deciding factor

> As an operator, switching to a cassandra management process that’s
basically a re-brand of an existing and commonly used management process
isn’t super risky. Asking operators to switch to a new process is a much
harder sell.
Reaper is far away from becoming a "cassandra management process", I
understand it does its job in doing repairs and snapshots (and other things
if I missed any), but as a Cassandra mangement sidecar process,
responsibilities of it are far beyond those just mentioned. All the design
goals mentioned in this thread from Joey (pluggable execution engine,
backup, restore, ring health detection, sstable upgrade, rolling restarts,
topology-aware maintenance, replacements of entire fleet without
compromising availability etc.,) are critical operations of a "cassandra
management process" that are hard to add on to a system which is not
architected for that. It basically makes total rework/refactor of reaper,
if we were to go down that path. And don't get me wrong, Reaper is a great
repair tool available for C* community, with a great visualization which
makes it easy to use.

We prefer what Jeff proposes, starting with something small and isolated
and layering the best of all sidecars incrementally on top.

--Vinay Chella


On Fri, Sep 7, 2018 at 6:11 PM Jeff Jirsa  wrote:

> I’d also like to see the end state you describe: reaper UI wrapping the
> Netflix management process with pluggable scheduling (either as is with
> reaper now, or using the Netflix scheduler), but I don’t think that means
> we need to start with reaper - if personally prefer the opposite direction,
> starting with something small and isolated and layering on top.
>
> --
> Jeff Jirsa
>
>
> > On Sep 7, 2018, at 5:42 PM, Blake Eggleston 
> wrote:
> >
> > I think we should accept the reaper project as is and make that
> cassandra management process 1.0, then integrate the netflix scheduler (and
> other new features) into that.
> >
> > The ultimate goal would be for the netflix scheduler to become the
> default repair scheduler, but I think using reaper as the starting point
> makes it easier to get there.
> >
> > Reaper would bring a prod user base that would realistically take 2-3
> years to build up with a new project. As an operator, switching to a
> cassandra management process that’s basically a re-brand of an existing and
> commonly used management process isn’t super risky. Asking operators to
> switch to a new process is a much harder sell.
> >
> > On September 7, 2018 at 4:17:10 PM, Jeff Jirsa (jji...@gmail.com) wrote:
> >
> > How can we continue moving this forward?
> >
> > Mick/Jon/TLP folks, is there a path here where we commit the
> > Netflix-provided management process, and you augment Reaper to work with
> it?
> > Is there a way we can make a larger umbrella that's modular that can
> > support either/both?
> > Does anyone believe there's a clear, objective argument that one is
> > strictly better than the other? I haven't seen one.
> >
> >
> >
> > On Mon, Aug 20, 2018 at 4:14 PM Roopa Tangirala
> >  wrote:
> >
> >> +1 to everything that Joey articulated with emphasis on the fact that
> >> contributions should be evaluated based on the merit of code and their
> >> value add to the whole offering. I hope it does not matter whether
> that
> >> contribution comes from PMC member or a person who is not a committer.
> I
> >> would like the process to be such that it encourages the new members to
> be
> >> a part of the community and not shy away from contributing to the code
> >> assuming their contributions are valued differently than committers or
> PMC
> >> members. It would be sad to see the contributions decrease if we go
> down
> >> that path.
> >>
> >> *Regards,*
> >>
> >> *Roopa Tangirala*
> >>
> >> Engineering Manager CDE
> >>
> >> *(408) 438-3156 - mobile*
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Mon, Aug 20, 2018 at 2:58 PM Joseph Lynch 
> >> wrote:
> >>
>  We are looking to contribute Reaper to the Cassandra project.
> 
> >>> Just to clarify are you proposing contributing Reaper as a project
> via
> >>> donation or you are planning on contributing the features of Reaper
> as
> >>> patches to Cassandra? If the former how far along are you on the
> donation
> >>> process? If the latter, when do you think you would have patches
> ready
> >> for
> >>> consideration / review?
> >>>
> >>>
>  Looking at the patch it's very similar in its base design already,
> but
>  Reaper does has a lot more to offer. We have all been 

Re: Proposing an Apache Cassandra Management process

[Blake Eggleston]

What’s the benefit of doing it that way vs starting with reaper and integrating 
the netflix scheduler? If reaper was just a really inappropriate choice for the 
cassandra management process, I could see that being a better approach, but I 
don’t think that’s the case.

If our management process isn’t a drop in replacement for reaper, then reaper 
will continue to exist, which will split the user and developers base between 
the 2 projects. That won't be good for either project.

On September 7, 2018 at 6:12:01 PM, Jeff Jirsa (jji...@gmail.com) wrote:

I’d also like to see the end state you describe: reaper UI wrapping the Netflix 
management process with pluggable scheduling (either as is with reaper now, or 
using the Netflix scheduler), but I don’t think that means we need to start 
with reaper - if personally prefer the opposite direction, starting with 
something small and isolated and layering on top.  

--  
Jeff Jirsa  


> On Sep 7, 2018, at 5:42 PM, Blake Eggleston  wrote:  
>  
> I think we should accept the reaper project as is and make that cassandra 
> management process 1.0, then integrate the netflix scheduler (and other new 
> features) into that.  
>  
> The ultimate goal would be for the netflix scheduler to become the default 
> repair scheduler, but I think using reaper as the starting point makes it 
> easier to get there.  
>  
> Reaper would bring a prod user base that would realistically take 2-3 years 
> to build up with a new project. As an operator, switching to a cassandra 
> management process that’s basically a re-brand of an existing and commonly 
> used management process isn’t super risky. Asking operators to switch to a 
> new process is a much harder sell.  
>  
> On September 7, 2018 at 4:17:10 PM, Jeff Jirsa (jji...@gmail.com) wrote:  
>  
> How can we continue moving this forward?  
>  
> Mick/Jon/TLP folks, is there a path here where we commit the  
> Netflix-provided management process, and you augment Reaper to work with it?  
> Is there a way we can make a larger umbrella that's modular that can  
> support either/both?  
> Does anyone believe there's a clear, objective argument that one is  
> strictly better than the other? I haven't seen one.  
>  
>  
>  
> On Mon, Aug 20, 2018 at 4:14 PM Roopa Tangirala  
>  wrote:  
>  
>> +1 to everything that Joey articulated with emphasis on the fact that  
>> contributions should be evaluated based on the merit of code and their  
>> value add to the whole offering. I hope it does not matter whether that  
>> contribution comes from PMC member or a person who is not a committer. I  
>> would like the process to be such that it encourages the new members to be  
>> a part of the community and not shy away from contributing to the code  
>> assuming their contributions are valued differently than committers or PMC  
>> members. It would be sad to see the contributions decrease if we go down  
>> that path.  
>>  
>> *Regards,*  
>>  
>> *Roopa Tangirala*  
>>  
>> Engineering Manager CDE  
>>  
>> *(408) 438-3156 - mobile*  
>>  
>>  
>>  
>>  
>>  
>>  
>> On Mon, Aug 20, 2018 at 2:58 PM Joseph Lynch   
>> wrote:  
>>  
 We are looking to contribute Reaper to the Cassandra project.  
  
>>> Just to clarify are you proposing contributing Reaper as a project via  
>>> donation or you are planning on contributing the features of Reaper as  
>>> patches to Cassandra? If the former how far along are you on the donation  
>>> process? If the latter, when do you think you would have patches ready  
>> for  
>>> consideration / review?  
>>>  
>>>  
 Looking at the patch it's very similar in its base design already, but  
 Reaper does has a lot more to offer. We have all been working hard to  
>>> move  
 it to also being a side-car so it can be contributed. This raises a  
>>> number  
 of relevant questions to this thread: would we then accept both works  
>> in  
 the Cassandra project, and what burden would it put on the current PMC  
>> to  
 maintain both works.  
  
>>> I would hope that we would collaborate on merging the best parts of all  
>>> into the official Cassandra sidecar, taking the always on, shared  
>> nothing,  
>>> highly available system that we've contributed a patchset for and adding  
>> in  
>>> many of the repair features (e.g. schedules, a nice web UI) that Reaper  
>>> has.  
>>>  
>>>  
 I share Stefan's concern that consensus had not been met around a  
 side-car, and that it was somehow default accepted before a patch  
>> landed.  
>>>  
>>>  
>>> I feel this is not correct or fair. The sidecar and repair discussions  
>> have  
>>> been anything _but_ "default accepted". The timeline of consensus  
>> building  
>>> involving the management sidecar and repair scheduling plans:  
>>>  
>>> Dec 2016: Vinay worked with Jon and Alex to try to collaborate on Reaper  
>> to  
>>> come up with design goals for a repair scheduler that could work at  
>> 

Re: Proposing an Apache Cassandra Management process

[Mick Semb Wever]

> How can we continue moving this forward?
> 
> Mick/Jon/TLP folks, is there a path here where we commit the
> Netflix-provided management process, and you augment Reaper to work with it?
> Is there a way we can make a larger umbrella that's modular that can
> support either/both?


There seems a reluctance in any collaboration between Reaper and Netflix atm, 
despite efforts from both sides. It worries me that there appears to be this 
whole roadmap of design and implementation ready to go, and reference work to 
it that we don't get to see. When we (Reaper) do have a real reference work out 
in the open. This will ofc make it challenging for the Reaper folk to get 
involved, it leaves us with the feeling that we're just having to re-invent the 
wheel for someone else, when it would be much quicker for us to evolve Reaper 
to the designs proposed.

Such collaboration between us, i reckon needs to happen first ("show, don't 
tell"). So I'd throw out the idea that we start the side-car collaboration 
project as an external github project. From there if we can put something 
together that satisfies people's designs for a new side-car tool, and maintains 
the Reaper user base and feature-list, then we've proven that the 
community-spirit and trust is in place to bring it into the Cassandra project. 
Also, unless we have existing PMC that we know we guardian this project we're 
just presuming everyone will know all the ins and outs of ASF processes and 
requirements. Maybe it's better to be practicing this stuff first, rather than 
presuming it will just happen. 

That's just my two cents, and an attempt at a compromise. My first vote would 
still be to bring in Reaper and retro-fit it step by step (and i'd be more than 
happy to accept and work on someone else's roadmap if we took this approach).

Mick

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Jeff Jirsa]

I’d also like to see the end state you describe: reaper UI wrapping the Netflix 
management process with pluggable scheduling (either as is with reaper now, or 
using the Netflix scheduler), but I don’t think that means we need to start 
with reaper - if personally prefer the opposite direction, starting with 
something small and isolated and layering on top. 

-- 
Jeff Jirsa


> On Sep 7, 2018, at 5:42 PM, Blake Eggleston  wrote:
> 
> I think we should accept the reaper project as is and make that cassandra 
> management process 1.0, then integrate the netflix scheduler (and other new 
> features) into that.
> 
> The ultimate goal would be for the netflix scheduler to become the default 
> repair scheduler, but I think using reaper as the starting point makes it 
> easier to get there. 
> 
> Reaper would bring a prod user base that would realistically take 2-3 years 
> to build up with a new project. As an operator, switching to a cassandra 
> management process that’s basically a re-brand of an existing and commonly 
> used management process isn’t super risky. Asking operators to switch to a 
> new process is a much harder sell. 
> 
> On September 7, 2018 at 4:17:10 PM, Jeff Jirsa (jji...@gmail.com) wrote:
> 
> How can we continue moving this forward?  
> 
> Mick/Jon/TLP folks, is there a path here where we commit the  
> Netflix-provided management process, and you augment Reaper to work with it?  
> Is there a way we can make a larger umbrella that's modular that can  
> support either/both?  
> Does anyone believe there's a clear, objective argument that one is  
> strictly better than the other? I haven't seen one.  
> 
> 
> 
> On Mon, Aug 20, 2018 at 4:14 PM Roopa Tangirala  
>  wrote:  
> 
>> +1 to everything that Joey articulated with emphasis on the fact that  
>> contributions should be evaluated based on the merit of code and their  
>> value add to the whole offering. I hope it does not matter whether that  
>> contribution comes from PMC member or a person who is not a committer. I  
>> would like the process to be such that it encourages the new members to be  
>> a part of the community and not shy away from contributing to the code  
>> assuming their contributions are valued differently than committers or PMC  
>> members. It would be sad to see the contributions decrease if we go down  
>> that path.  
>> 
>> *Regards,*  
>> 
>> *Roopa Tangirala*  
>> 
>> Engineering Manager CDE  
>> 
>> *(408) 438-3156 - mobile*  
>> 
>> 
>> 
>> 
>> 
>> 
>> On Mon, Aug 20, 2018 at 2:58 PM Joseph Lynch   
>> wrote:  
>> 
 We are looking to contribute Reaper to the Cassandra project.  
 
>>> Just to clarify are you proposing contributing Reaper as a project via  
>>> donation or you are planning on contributing the features of Reaper as  
>>> patches to Cassandra? If the former how far along are you on the donation  
>>> process? If the latter, when do you think you would have patches ready  
>> for  
>>> consideration / review?  
>>> 
>>> 
 Looking at the patch it's very similar in its base design already, but  
 Reaper does has a lot more to offer. We have all been working hard to  
>>> move  
 it to also being a side-car so it can be contributed. This raises a  
>>> number  
 of relevant questions to this thread: would we then accept both works  
>> in  
 the Cassandra project, and what burden would it put on the current PMC  
>> to  
 maintain both works.  
 
>>> I would hope that we would collaborate on merging the best parts of all  
>>> into the official Cassandra sidecar, taking the always on, shared  
>> nothing,  
>>> highly available system that we've contributed a patchset for and adding  
>> in  
>>> many of the repair features (e.g. schedules, a nice web UI) that Reaper  
>>> has.  
>>> 
>>> 
 I share Stefan's concern that consensus had not been met around a  
 side-car, and that it was somehow default accepted before a patch  
>> landed.  
>>> 
>>> 
>>> I feel this is not correct or fair. The sidecar and repair discussions  
>> have  
>>> been anything _but_ "default accepted". The timeline of consensus  
>> building  
>>> involving the management sidecar and repair scheduling plans:  
>>> 
>>> Dec 2016: Vinay worked with Jon and Alex to try to collaborate on Reaper  
>> to  
>>> come up with design goals for a repair scheduler that could work at  
>> Netflix  
>>> scale.  
>>> 
>>> ~Feb 2017: Netflix believes that the fundamental design gaps prevented us  
>>> from using Reaper as it relies heavily on remote JMX connections and  
>>> central coordination.  
>>> 
>>> Sep. 2017: Vinay gives a lightning talk at NGCC about a highly available  
>>> and distributed repair scheduling sidecar/tool. He is encouraged by  
>>> multiple committers to build repair scheduling into the daemon itself and  
>>> not as a sidecar so the database is truly eventually consistent.  
>>> 
>>> ~Jun. 2017 - Feb. 2018: Based on internal need and the positive feedback  

Re: Proposing an Apache Cassandra Management process

[Joseph Lynch]

On Fri, Sep 7, 2018 at 5:03 PM Jonathan Haddad  wrote:
>
> We haven’t even defined any requirements for an admin tool. It’s hard to
> make a case for anything without agreement on what we’re trying to build.
>
We were/are trying to sketch out scope/requirements in the #14395 and
#14346 tickets as well as their associated design documents. I think
the general proposed direction is a distributed 1:1 management sidecar
process similar in architecture to Netflix's Priam except explicitly
built to be general and pluggable by anyone rather than tightly
coupled to AWS.

Dinesh, Vinay and I were aiming for low amounts of scope at first and
take things in an iterative approach with just enough upfront design
but not so much we are unable to make any progress at all. For example
maybe something like:

1. Get a super simple and non controversial sidecar process that ships
with Cassandra and exposes a lightweight HTTP interface to e.g. some
basic JMX endpoints
2a. Add a pluggable execution engine for cron/oneshot/scheduled jobs
with the basic interfaces and state store and such
2b. Start scoping and implementing the full HTTP interface, e.g.
backup status, cluster health status, etc ...
3a. Start integrating implementations of the jobs from 2a such as
snapshot, backup, cluster restart, daemon + sstable upgrade, repair,
etc
3b. Start integrating UI components that pair with the HTTP interface from 2b
4. ?? Perhaps start unlocking next generation operations like moving
"background" activities like compaction, streaming, repair etc into
one or more sidecar contained processes to ensure the main daemon only
handles read+write requests

There are going to be a lot of questions to answer, and I think trying
to answer them all up front will mean that we get nowhere or make
unfortunate compromises that cripple the project from the start. If
people think we need to do more design and discussion than we have
been doing then we can spend more time on the design, but personally
I'd rather start iterating on code and prove value incrementally. If
it doesn't work out we won't release it GA to the community ...

-Joey

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Blake Eggleston]

I think we should accept the reaper project as is and make that cassandra 
management process 1.0, then integrate the netflix scheduler (and other new 
features) into that.

The ultimate goal would be for the netflix scheduler to become the default 
repair scheduler, but I think using reaper as the starting point makes it 
easier to get there. 

Reaper would bring a prod user base that would realistically take 2-3 years to 
build up with a new project. As an operator, switching to a cassandra 
management process that’s basically a re-brand of an existing and commonly used 
management process isn’t super risky. Asking operators to switch to a new 
process is a much harder sell. 

On September 7, 2018 at 4:17:10 PM, Jeff Jirsa (jji...@gmail.com) wrote:

How can we continue moving this forward?  

Mick/Jon/TLP folks, is there a path here where we commit the  
Netflix-provided management process, and you augment Reaper to work with it?  
Is there a way we can make a larger umbrella that's modular that can  
support either/both?  
Does anyone believe there's a clear, objective argument that one is  
strictly better than the other? I haven't seen one.  



On Mon, Aug 20, 2018 at 4:14 PM Roopa Tangirala  
 wrote:  

> +1 to everything that Joey articulated with emphasis on the fact that  
> contributions should be evaluated based on the merit of code and their  
> value add to the whole offering. I hope it does not matter whether that  
> contribution comes from PMC member or a person who is not a committer. I  
> would like the process to be such that it encourages the new members to be  
> a part of the community and not shy away from contributing to the code  
> assuming their contributions are valued differently than committers or PMC  
> members. It would be sad to see the contributions decrease if we go down  
> that path.  
>  
> *Regards,*  
>  
> *Roopa Tangirala*  
>  
> Engineering Manager CDE  
>  
> *(408) 438-3156 - mobile*  
>  
>  
>  
>  
>  
>  
> On Mon, Aug 20, 2018 at 2:58 PM Joseph Lynch   
> wrote:  
>  
> > > We are looking to contribute Reaper to the Cassandra project.  
> > >  
> > Just to clarify are you proposing contributing Reaper as a project via  
> > donation or you are planning on contributing the features of Reaper as  
> > patches to Cassandra? If the former how far along are you on the donation  
> > process? If the latter, when do you think you would have patches ready  
> for  
> > consideration / review?  
> >  
> >  
> > > Looking at the patch it's very similar in its base design already, but  
> > > Reaper does has a lot more to offer. We have all been working hard to  
> > move  
> > > it to also being a side-car so it can be contributed. This raises a  
> > number  
> > > of relevant questions to this thread: would we then accept both works  
> in  
> > > the Cassandra project, and what burden would it put on the current PMC  
> to  
> > > maintain both works.  
> > >  
> > I would hope that we would collaborate on merging the best parts of all  
> > into the official Cassandra sidecar, taking the always on, shared  
> nothing,  
> > highly available system that we've contributed a patchset for and adding  
> in  
> > many of the repair features (e.g. schedules, a nice web UI) that Reaper  
> > has.  
> >  
> >  
> > > I share Stefan's concern that consensus had not been met around a  
> > > side-car, and that it was somehow default accepted before a patch  
> landed.  
> >  
> >  
> > I feel this is not correct or fair. The sidecar and repair discussions  
> have  
> > been anything _but_ "default accepted". The timeline of consensus  
> building  
> > involving the management sidecar and repair scheduling plans:  
> >  
> > Dec 2016: Vinay worked with Jon and Alex to try to collaborate on Reaper  
> to  
> > come up with design goals for a repair scheduler that could work at  
> Netflix  
> > scale.  
> >  
> > ~Feb 2017: Netflix believes that the fundamental design gaps prevented us  
> > from using Reaper as it relies heavily on remote JMX connections and  
> > central coordination.  
> >  
> > Sep. 2017: Vinay gives a lightning talk at NGCC about a highly available  
> > and distributed repair scheduling sidecar/tool. He is encouraged by  
> > multiple committers to build repair scheduling into the daemon itself and  
> > not as a sidecar so the database is truly eventually consistent.  
> >  
> > ~Jun. 2017 - Feb. 2018: Based on internal need and the positive feedback  
> at  
> > NGCC, Vinay and myself prototype the distributed repair scheduler within  
> > Priam and roll it out at Netflix scale.  
> >  
> > Mar. 2018: I open a Jira (CASSANDRA-14346) along with a detailed 20 page  
> > design document for adding repair scheduling to the daemon itself and  
> open  
> > the design up for feedback from the community. We get feedback from Alex,  
> > Blake, Nate, Stefan, and Mick. As far as I know there were zero proposals  
> > to contribute Reaper at this point. 

Re: Proposing an Apache Cassandra Management process

[Jonathan Haddad]

We haven’t even defined any requirements for an admin tool. It’s hard to
make a case for anything without agreement on what we’re trying to build.

On Fri, Sep 7, 2018 at 7:17 PM Jeff Jirsa  wrote:

> How can we continue moving this forward?
>
> Mick/Jon/TLP folks, is there a path here where we commit the
> Netflix-provided management process, and you augment Reaper to work with
> it?
> Is there a way we can make a larger umbrella that's modular that can
> support either/both?
> Does anyone believe there's a clear, objective argument that one is
> strictly better than the other? I haven't seen one.
>
>
>
> On Mon, Aug 20, 2018 at 4:14 PM Roopa Tangirala
>  wrote:
>
> > +1 to everything that Joey articulated with emphasis on the fact that
> > contributions should be evaluated based on the merit of code and their
> > value add to the whole offering. I  hope it does not matter whether that
> > contribution comes from PMC member or a person who is not a committer. I
> > would like the process to be such that it encourages the new members to
> be
> > a part of the community and not shy away from contributing to the code
> > assuming their contributions are valued differently than committers or
> PMC
> > members. It would be sad to see the contributions decrease if we go down
> > that path.
> >
> > *Regards,*
> >
> > *Roopa Tangirala*
> >
> > Engineering Manager CDE
> >
> > *(408) 438-3156 - mobile*
> >
> >
> >
> >
> >
> >
> > On Mon, Aug 20, 2018 at 2:58 PM Joseph Lynch 
> > wrote:
> >
> > > > We are looking to contribute Reaper to the Cassandra project.
> > > >
> > > Just to clarify are you proposing contributing Reaper as a project via
> > > donation or you are planning on contributing the features of Reaper as
> > > patches to Cassandra? If the former how far along are you on the
> donation
> > > process? If the latter, when do you think you would have patches ready
> > for
> > > consideration / review?
> > >
> > >
> > > > Looking at the patch it's very similar in its base design already,
> but
> > > > Reaper does has a lot more to offer. We have all been working hard to
> > > move
> > > > it to also being a side-car so it can be contributed. This raises a
> > > number
> > > > of relevant questions to this thread: would we then accept both works
> > in
> > > > the Cassandra project, and what burden would it put on the current
> PMC
> > to
> > > > maintain both works.
> > > >
> > > I would hope that we would collaborate on merging the best parts of all
> > > into the official Cassandra sidecar, taking the always on, shared
> > nothing,
> > > highly available system that we've contributed a patchset for and
> adding
> > in
> > > many of the repair features (e.g. schedules, a nice web UI) that Reaper
> > > has.
> > >
> > >
> > > > I share Stefan's concern that consensus had not been met around a
> > > > side-car, and that it was somehow default accepted before a patch
> > landed.
> > >
> > >
> > > I feel this is not correct or fair. The sidecar and repair discussions
> > have
> > > been anything _but_ "default accepted". The timeline of consensus
> > building
> > > involving the management sidecar and repair scheduling plans:
> > >
> > > Dec 2016: Vinay worked with Jon and Alex to try to collaborate on
> Reaper
> > to
> > > come up with design goals for a repair scheduler that could work at
> > Netflix
> > > scale.
> > >
> > > ~Feb 2017: Netflix believes that the fundamental design gaps prevented
> us
> > > from using Reaper as it relies heavily on remote JMX connections and
> > > central coordination.
> > >
> > > Sep. 2017: Vinay gives a lightning talk at NGCC about a highly
> available
> > > and distributed repair scheduling sidecar/tool. He is encouraged by
> > > multiple committers to build repair scheduling into the daemon itself
> and
> > > not as a sidecar so the database is truly eventually consistent.
> > >
> > > ~Jun. 2017 - Feb. 2018: Based on internal need and the positive
> feedback
> > at
> > > NGCC, Vinay and myself prototype the distributed repair scheduler
> within
> > > Priam and roll it out at Netflix scale.
> > >
> > > Mar. 2018: I open a Jira (CASSANDRA-14346) along with a detailed 20
> page
> > > design document for adding repair scheduling to the daemon itself and
> > open
> > > the design up for feedback from the community. We get feedback from
> Alex,
> > > Blake, Nate, Stefan, and Mick. As far as I know there were zero
> proposals
> > > to contribute Reaper at this point. We hear the consensus that the
> > > community would prefer repair scheduling in a separate distributed
> > sidecar
> > > rather than in the daemon itself and we re-work the design to match
> this
> > > consensus, re-aligning with our original proposal at NGCC.
> > >
> > > Apr 2018: Blake brings the discussion of repair scheduling to the dev
> > list
> > > (
> > >
> > >
> >
> https://lists.apache.org/thread.html/760fbef677f27aa5c2ab4c375c7efeb81304fea428deff986ba1c2eb@%3Cdev.cassandra.apache.org%3E
> > > ).
> > > 

Re: Proposing an Apache Cassandra Management process

[Jeff Jirsa]

How can we continue moving this forward?

Mick/Jon/TLP folks, is there a path here where we commit the
Netflix-provided management process, and you augment Reaper to work with it?
Is there a way we can make a larger umbrella that's modular that can
support either/both?
Does anyone believe there's a clear, objective argument that one is
strictly better than the other? I haven't seen one.



On Mon, Aug 20, 2018 at 4:14 PM Roopa Tangirala
 wrote:

> +1 to everything that Joey articulated with emphasis on the fact that
> contributions should be evaluated based on the merit of code and their
> value add to the whole offering. I  hope it does not matter whether that
> contribution comes from PMC member or a person who is not a committer. I
> would like the process to be such that it encourages the new members to be
> a part of the community and not shy away from contributing to the code
> assuming their contributions are valued differently than committers or PMC
> members. It would be sad to see the contributions decrease if we go down
> that path.
>
> *Regards,*
>
> *Roopa Tangirala*
>
> Engineering Manager CDE
>
> *(408) 438-3156 - mobile*
>
>
>
>
>
>
> On Mon, Aug 20, 2018 at 2:58 PM Joseph Lynch 
> wrote:
>
> > > We are looking to contribute Reaper to the Cassandra project.
> > >
> > Just to clarify are you proposing contributing Reaper as a project via
> > donation or you are planning on contributing the features of Reaper as
> > patches to Cassandra? If the former how far along are you on the donation
> > process? If the latter, when do you think you would have patches ready
> for
> > consideration / review?
> >
> >
> > > Looking at the patch it's very similar in its base design already, but
> > > Reaper does has a lot more to offer. We have all been working hard to
> > move
> > > it to also being a side-car so it can be contributed. This raises a
> > number
> > > of relevant questions to this thread: would we then accept both works
> in
> > > the Cassandra project, and what burden would it put on the current PMC
> to
> > > maintain both works.
> > >
> > I would hope that we would collaborate on merging the best parts of all
> > into the official Cassandra sidecar, taking the always on, shared
> nothing,
> > highly available system that we've contributed a patchset for and adding
> in
> > many of the repair features (e.g. schedules, a nice web UI) that Reaper
> > has.
> >
> >
> > > I share Stefan's concern that consensus had not been met around a
> > > side-car, and that it was somehow default accepted before a patch
> landed.
> >
> >
> > I feel this is not correct or fair. The sidecar and repair discussions
> have
> > been anything _but_ "default accepted". The timeline of consensus
> building
> > involving the management sidecar and repair scheduling plans:
> >
> > Dec 2016: Vinay worked with Jon and Alex to try to collaborate on Reaper
> to
> > come up with design goals for a repair scheduler that could work at
> Netflix
> > scale.
> >
> > ~Feb 2017: Netflix believes that the fundamental design gaps prevented us
> > from using Reaper as it relies heavily on remote JMX connections and
> > central coordination.
> >
> > Sep. 2017: Vinay gives a lightning talk at NGCC about a highly available
> > and distributed repair scheduling sidecar/tool. He is encouraged by
> > multiple committers to build repair scheduling into the daemon itself and
> > not as a sidecar so the database is truly eventually consistent.
> >
> > ~Jun. 2017 - Feb. 2018: Based on internal need and the positive feedback
> at
> > NGCC, Vinay and myself prototype the distributed repair scheduler within
> > Priam and roll it out at Netflix scale.
> >
> > Mar. 2018: I open a Jira (CASSANDRA-14346) along with a detailed 20 page
> > design document for adding repair scheduling to the daemon itself and
> open
> > the design up for feedback from the community. We get feedback from Alex,
> > Blake, Nate, Stefan, and Mick. As far as I know there were zero proposals
> > to contribute Reaper at this point. We hear the consensus that the
> > community would prefer repair scheduling in a separate distributed
> sidecar
> > rather than in the daemon itself and we re-work the design to match this
> > consensus, re-aligning with our original proposal at NGCC.
> >
> > Apr 2018: Blake brings the discussion of repair scheduling to the dev
> list
> > (
> >
> >
> https://lists.apache.org/thread.html/760fbef677f27aa5c2ab4c375c7efeb81304fea428deff986ba1c2eb@%3Cdev.cassandra.apache.org%3E
> > ).
> > Many community members give positive feedback that we should solve it as
> > part of Cassandra and there is still no mention of contributing Reaper at
> > this point. The last message is my attempted summary giving context on
> how
> > we want to take the best of all the sidecars (OpsCenter, Priam, Reaper)
> and
> > ship them with Cassandra.
> >
> > Apr. 2018: Dinesh opens CASSANDRA-14395 along with a public design
> document
> > for gathering feedback on a general 

Re: Proposing an Apache Cassandra Management process

[Mick Semb Wever]

> > contributions should be evaluated based on the merit of code and their
> > value add to the whole offering. I  hope it does not matter whether that
> > contribution comes from PMC member or a person who is not a committer.
> 
> 
> I hope this goes without saying.


Absolutely.

Joseph and Roopa, the words that you quoted me on were only intended to raise 
my questions and concerns around longevity and ongoing maintenance that the pmc 
bears. 

regards,
Mick

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Jeff Jirsa]

On Mon, Aug 20, 2018 at 4:14 PM Roopa Tangirala
 wrote:

> contributions should be evaluated based on the merit of code and their
> value add to the whole offering. I  hope it does not matter whether that
> contribution comes from PMC member or a person who is not a committer.


I hope this goes without saying.

Re: Proposing an Apache Cassandra Management process

[Roopa Tangirala]

+1 to everything that Joey articulated with emphasis on the fact that
contributions should be evaluated based on the merit of code and their
value add to the whole offering. I  hope it does not matter whether that
contribution comes from PMC member or a person who is not a committer. I
would like the process to be such that it encourages the new members to be
a part of the community and not shy away from contributing to the code
assuming their contributions are valued differently than committers or PMC
members. It would be sad to see the contributions decrease if we go down
that path.

*Regards,*

*Roopa Tangirala*

Engineering Manager CDE

*(408) 438-3156 - mobile*






On Mon, Aug 20, 2018 at 2:58 PM Joseph Lynch  wrote:

> > We are looking to contribute Reaper to the Cassandra project.
> >
> Just to clarify are you proposing contributing Reaper as a project via
> donation or you are planning on contributing the features of Reaper as
> patches to Cassandra? If the former how far along are you on the donation
> process? If the latter, when do you think you would have patches ready for
> consideration / review?
>
>
> > Looking at the patch it's very similar in its base design already, but
> > Reaper does has a lot more to offer. We have all been working hard to
> move
> > it to also being a side-car so it can be contributed. This raises a
> number
> > of relevant questions to this thread: would we then accept both works in
> > the Cassandra project, and what burden would it put on the current PMC to
> > maintain both works.
> >
> I would hope that we would collaborate on merging the best parts of all
> into the official Cassandra sidecar, taking the always on, shared nothing,
> highly available system that we've contributed a patchset for and adding in
> many of the repair features (e.g. schedules, a nice web UI) that Reaper
> has.
>
>
> > I share Stefan's concern that consensus had not been met around a
> > side-car, and that it was somehow default accepted before a patch landed.
>
>
> I feel this is not correct or fair. The sidecar and repair discussions have
> been anything _but_ "default accepted". The timeline of consensus building
> involving the management sidecar and repair scheduling plans:
>
> Dec 2016: Vinay worked with Jon and Alex to try to collaborate on Reaper to
> come up with design goals for a repair scheduler that could work at Netflix
> scale.
>
> ~Feb 2017: Netflix believes that the fundamental design gaps prevented us
> from using Reaper as it relies heavily on remote JMX connections and
> central coordination.
>
> Sep. 2017: Vinay gives a lightning talk at NGCC about a highly available
> and distributed repair scheduling sidecar/tool. He is encouraged by
> multiple committers to build repair scheduling into the daemon itself and
> not as a sidecar so the database is truly eventually consistent.
>
> ~Jun. 2017 - Feb. 2018: Based on internal need and the positive feedback at
> NGCC, Vinay and myself prototype the distributed repair scheduler within
> Priam and roll it out at Netflix scale.
>
> Mar. 2018: I open a Jira (CASSANDRA-14346) along with a detailed 20 page
> design document for adding repair scheduling to the daemon itself and open
> the design up for feedback from the community. We get feedback from Alex,
> Blake, Nate, Stefan, and Mick. As far as I know there were zero proposals
> to contribute Reaper at this point. We hear the consensus that the
> community would prefer repair scheduling in a separate distributed sidecar
> rather than in the daemon itself and we re-work the design to match this
> consensus, re-aligning with our original proposal at NGCC.
>
> Apr 2018: Blake brings the discussion of repair scheduling to the dev list
> (
>
> https://lists.apache.org/thread.html/760fbef677f27aa5c2ab4c375c7efeb81304fea428deff986ba1c2eb@%3Cdev.cassandra.apache.org%3E
> ).
> Many community members give positive feedback that we should solve it as
> part of Cassandra and there is still no mention of contributing Reaper at
> this point. The last message is my attempted summary giving context on how
> we want to take the best of all the sidecars (OpsCenter, Priam, Reaper) and
> ship them with Cassandra.
>
> Apr. 2018: Dinesh opens CASSANDRA-14395 along with a public design document
> for gathering feedback on a general management sidecar. Sankalp and Dinesh
> encourage Vinay and myself to kickstart that sidecar using the repair
> scheduler patch
>
> Apr 2018: Dinesh reaches out to the dev list (
>
> https://lists.apache.org/thread.html/a098341efd8f344494bcd2761dba5125e971b59b1dd54f282ffda253@%3Cdev.cassandra.apache.org%3E
> )
> about the general management process to gain further feedback. All feedback
> remains positive as it is a potential place for multiple community members
> to contribute their various sidecar functionality.
>
> May-Jul 2017: Vinay and I work on creating a basic sidecar for running the
> repair scheduler based on the feedback from the community in
> 

Re: Proposing an Apache Cassandra Management process

[dinesh.jo...@yahoo.com.INVALID]

On Mon, Aug 20, 2018 at 4:23 AM Mick Semb Wever  wrote:
>
> We are looking to contribute Reaper to the Cassandra project.
>
> Looking at the patch it's very similar in its base design already, but
> Reaper does has a lot more to offer. We have all been working hard to move
> it to also being a side-car so it can be contributed. This raises a number
> of relevant questions to this thread: would we then accept both works in
> the Cassandra project, and what burden would it put on the current PMC to
> maintain both works.
I think the comparison is not fair. The patch that has landed is new and is the 
beginning of a sidecar within Cassandra. It would be unfair to compare its 
features with Reaper which has been around for some time now. 
I proposed a management process (not exactly a sidecar) for Cassandra about 4 
months ago. Had you guys indicated interest in contributing Reaper, we would 
not be discussing two separate implementations. Don't get me wrong, I'm happy 
that we're talking about this right now.
> This seems at odds when we're already struggling to keep up with the
> incoming patches/contributions, and there could be other git repos in the
> project we will need to support in the future too. 
I think this is a great problem to have for a project. This is a sign that the 
pool of contributors is greater than reviewers / committers. I personally have 
been volunteering my time reviewing tickets, fixing flaky tests and generally 
helping out. I definitely think we need more people actively contributing.
> The Reaper project has worked hard in building both its user and
> contributor base. And I would have thought these, including having the
> contributor base overlap with the C* PMC, were prerequisites before moving
> a larger body of work into the project (separate git repo or not). I guess
You're talking about donating a body of code i.e. Reaper which is different 
from building a new feature from scratch.
> There's been little effort in evaluating these two bodies of work, one
> which is largely unknown to us, and my concern is how we would fairly
> support both going into the future?
I don't think we should have two separate implementations as part of the 
project. It would be best if we could have a single sidecar that could have 
features from Reaper as well as the proposed patch.
Thanks,
Dinesh  

Re: Proposing an Apache Cassandra Management process

[Joseph Lynch]

> We are looking to contribute Reaper to the Cassandra project.
>
Just to clarify are you proposing contributing Reaper as a project via
donation or you are planning on contributing the features of Reaper as
patches to Cassandra? If the former how far along are you on the donation
process? If the latter, when do you think you would have patches ready for
consideration / review?


> Looking at the patch it's very similar in its base design already, but
> Reaper does has a lot more to offer. We have all been working hard to move
> it to also being a side-car so it can be contributed. This raises a number
> of relevant questions to this thread: would we then accept both works in
> the Cassandra project, and what burden would it put on the current PMC to
> maintain both works.
>
I would hope that we would collaborate on merging the best parts of all
into the official Cassandra sidecar, taking the always on, shared nothing,
highly available system that we've contributed a patchset for and adding in
many of the repair features (e.g. schedules, a nice web UI) that Reaper has.


> I share Stefan's concern that consensus had not been met around a
> side-car, and that it was somehow default accepted before a patch landed.


I feel this is not correct or fair. The sidecar and repair discussions have
been anything _but_ "default accepted". The timeline of consensus building
involving the management sidecar and repair scheduling plans:

Dec 2016: Vinay worked with Jon and Alex to try to collaborate on Reaper to
come up with design goals for a repair scheduler that could work at Netflix
scale.

~Feb 2017: Netflix believes that the fundamental design gaps prevented us
from using Reaper as it relies heavily on remote JMX connections and
central coordination.

Sep. 2017: Vinay gives a lightning talk at NGCC about a highly available
and distributed repair scheduling sidecar/tool. He is encouraged by
multiple committers to build repair scheduling into the daemon itself and
not as a sidecar so the database is truly eventually consistent.

~Jun. 2017 - Feb. 2018: Based on internal need and the positive feedback at
NGCC, Vinay and myself prototype the distributed repair scheduler within
Priam and roll it out at Netflix scale.

Mar. 2018: I open a Jira (CASSANDRA-14346) along with a detailed 20 page
design document for adding repair scheduling to the daemon itself and open
the design up for feedback from the community. We get feedback from Alex,
Blake, Nate, Stefan, and Mick. As far as I know there were zero proposals
to contribute Reaper at this point. We hear the consensus that the
community would prefer repair scheduling in a separate distributed sidecar
rather than in the daemon itself and we re-work the design to match this
consensus, re-aligning with our original proposal at NGCC.

Apr 2018: Blake brings the discussion of repair scheduling to the dev list (
https://lists.apache.org/thread.html/760fbef677f27aa5c2ab4c375c7efeb81304fea428deff986ba1c2eb@%3Cdev.cassandra.apache.org%3E).
Many community members give positive feedback that we should solve it as
part of Cassandra and there is still no mention of contributing Reaper at
this point. The last message is my attempted summary giving context on how
we want to take the best of all the sidecars (OpsCenter, Priam, Reaper) and
ship them with Cassandra.

Apr. 2018: Dinesh opens CASSANDRA-14395 along with a public design document
for gathering feedback on a general management sidecar. Sankalp and Dinesh
encourage Vinay and myself to kickstart that sidecar using the repair
scheduler patch

Apr 2018: Dinesh reaches out to the dev list (
https://lists.apache.org/thread.html/a098341efd8f344494bcd2761dba5125e971b59b1dd54f282ffda253@%3Cdev.cassandra.apache.org%3E)
about the general management process to gain further feedback. All feedback
remains positive as it is a potential place for multiple community members
to contribute their various sidecar functionality.

May-Jul 2017: Vinay and I work on creating a basic sidecar for running the
repair scheduler based on the feedback from the community in
CASSANDRA-14346 and CASSANDRA-14395

Jun 2018: I bump CASSANDRA-14346 indicating we're still working on this,
nobody objects

Jul 2018: Sankalp asks on the dev list if anyone has feature Jiras anyone
needs review for before 4.0, I mention again that we've nearly got the
basic sidecar and repair scheduling work done and will need help with
review. No one responds.

Aug 2018: We submit a patch that brings a basic distributed sidecar and
robust distributed repair to Cassandra itself. Dinesh mentions that he will
try to review. Now folks appear concerned about it being in tree and
instead maybe it should go in a different repo all together. I don't think
we have consensus on the repo choice yet.

This seems at odds when we're already struggling to keep up with the
> incoming patches/contributions, and there could be other git repos in the
> project we will need to support in the future too. But 

Re: Proposing an Apache Cassandra Management process

[sankalp kohli]

Hi,
Here is how I think we can make progress
1. Consensus is reached that we need side car as this thread is months old
and I do not see any objections. I bumped it again and it is good to see it
being active.
2. There is no consensus on new repo vs not. I will start a thread on it
and lets discuss it.
3. We have 2 implementations now for running repair via side car. This is
actually awesome to see for the community. We should work on JIRA(like we
did for virtual table) to get the best out of both the implementation.

Thanks,
Sankalp



On Mon, Aug 20, 2018 at 4:23 AM Mick Semb Wever  wrote:

>
> On Fri, 17 Aug 2018, at 14:27, Sankalp Kohli wrote:
> > I am bumping this thread because patch has landed for this with repair
> > functionality.
>
>
> We are looking to contribute Reaper to the Cassandra project.
>
> Looking at the patch it's very similar in its base design already, but
> Reaper does has a lot more to offer. We have all been working hard to move
> it to also being a side-car so it can be contributed. This raises a number
> of relevant questions to this thread: would we then accept both works in
> the Cassandra project, and what burden would it put on the current PMC to
> maintain both works.
>
> I share Stefan's concern that consensus had not been met around a
> side-car, and that it was somehow default accepted before a patch landed.
> This seems at odds when we're already struggling to keep up with the
> incoming patches/contributions, and there could be other git repos in the
> project we will need to support in the future too. But I'm also curious
> about the whole "Community over Code" angle to this, how do we encourage
> multiple external works to collaborate together building value in both the
> technical and community.
>
> The Reaper project has worked hard in building both its user and
> contributor base. And I would have thought these, including having the
> contributor base overlap with the C* PMC, were prerequisites before moving
> a larger body of work into the project (separate git repo or not). I guess
> this isn't so much "Community over Code", but it illustrates a concern
> regarding abandoned code when there's no existing track record of
> maintaining it as OSS, as opposed to expecting an existing "show, don't
> tell" culture. Reaper for example has stronger indicators for ongoing
> support and an existing OSS user base: today C* committers having
> contributed to Reaper are Jon, Stefan, Nate, and myself, amongst the 40
> contributors in total. And we've been making steps to involve it more into
> the C* community (eg users ML), without being too presumptuous. On the
> technical side: Reaper supports (or can easily) all the concerns that the
> proposal here raises: distributed nodetool commands, centralising jmx
> interfacing, scheduling ops (repairs, snapshots, compactions, cleanups,
> etc), monitoring and diagnostics, etc etc. It's designed so that it can be
> a single instance, instance-per-datacenter, or side-car (per process). When
> there are multiple instances in a datacenter you get HA. You have a choice
> of different storage backends (memory, postgres, c*). You can ofc use a
> separate C* cluster as a backend so to separate infrastructure data from
> production data. And it's got an UI for C* Diagnostics already (which
> imposes a different jmx interface of polling for events rather than
> subscribing to jmx notifications which we know is problematic, thanks to
> Stefan). Anyway, that's my plug for Reaper :-)
>
> There's been little effort in evaluating these two bodies of work, one
> which is largely unknown to us, and my concern is how we would fairly
> support both going into the future?
>
> Another option would be that this side-car patch first exists as a github
> project for a period of time, on par to how Reaper has been. This will help
> evaluate its use and to first build up its contributors. This makes it
> easier for the C* PMC to choose which projects it would want to formally
> maintain, and to do so based on factors beyond merits of the technical. We
> may even see it converge (or collaborate more) with Reaper, a win for
> everyone.
>
> regards,
> Mick
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Proposing an Apache Cassandra Management process

[Mick Semb Wever]

On Fri, 17 Aug 2018, at 14:27, Sankalp Kohli wrote:
> I am bumping this thread because patch has landed for this with repair 
> functionality. 


We are looking to contribute Reaper to the Cassandra project. 

Looking at the patch it's very similar in its base design already, but Reaper 
does has a lot more to offer. We have all been working hard to move it to also 
being a side-car so it can be contributed. This raises a number of relevant 
questions to this thread: would we then accept both works in the Cassandra 
project, and what burden would it put on the current PMC to maintain both works.

I share Stefan's concern that consensus had not been met around a side-car, and 
that it was somehow default accepted before a patch landed. This seems at odds 
when we're already struggling to keep up with the incoming 
patches/contributions, and there could be other git repos in the project we 
will need to support in the future too. But I'm also curious about the whole 
"Community over Code" angle to this, how do we encourage multiple external 
works to collaborate together building value in both the technical and 
community.  
 
The Reaper project has worked hard in building both its user and contributor 
base. And I would have thought these, including having the contributor base 
overlap with the C* PMC, were prerequisites before moving a larger body of work 
into the project (separate git repo or not). I guess this isn't so much 
"Community over Code", but it illustrates a concern regarding abandoned code 
when there's no existing track record of maintaining it as OSS, as opposed to 
expecting an existing "show, don't tell" culture. Reaper for example has 
stronger indicators for ongoing support and an existing OSS user base: today C* 
committers having contributed to Reaper are Jon, Stefan, Nate, and myself, 
amongst the 40 contributors in total. And we've been making steps to involve it 
more into the C* community (eg users ML), without being too presumptuous. On 
the technical side: Reaper supports (or can easily) all the concerns that the 
proposal here raises: distributed nodetool commands, centralising jmx 
interfacing, scheduling ops (repairs, snapshots, compactions, cleanups, etc), 
monitoring and diagnostics, etc etc. It's designed so that it can be a single 
instance, instance-per-datacenter, or side-car (per process). When there are 
multiple instances in a datacenter you get HA. You have a choice of different 
storage backends (memory, postgres, c*). You can ofc use a separate C* cluster 
as a backend so to separate infrastructure data from production data. And it's 
got an UI for C* Diagnostics already (which imposes a different jmx interface 
of polling for events rather than subscribing to jmx notifications which we 
know is problematic, thanks to Stefan). Anyway, that's my plug for Reaper :-)

There's been little effort in evaluating these two bodies of work, one which is 
largely unknown to us, and my concern is how we would fairly support both going 
into the future? 

Another option would be that this side-car patch first exists as a github 
project for a period of time, on par to how Reaper has been. This will help 
evaluate its use and to first build up its contributors. This makes it easier 
for the C* PMC to choose which projects it would want to formally maintain, and 
to do so based on factors beyond merits of the technical. We may even see it 
converge (or collaborate more) with Reaper, a win for everyone.

regards,
Mick

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Sankalp Kohli]

That’s a good point. Let’s review the patch and when it is ready to commit, we 
can create the repo then. 

> On Aug 18, 2018, at 14:04, Stefan Podkowinski  wrote:
> 
> I think we do have some consensus that 1) we should give it a try to
> have one or many side-car processes for non-essential features, and that
> 2) they should be developed in a separate repo. I'm also open to the
> idea of accepting the proposed implementation as a possible side-car
> solution and really appreciate effort. But my point is that creating a
> new repo, just for the patch, seems to imply that it's also going to
> become the de facto official side-car solution, which doesn't feel right
> to me, given that the proposed patch isn't even reviewed and hasn't
> received much feedback yet.
> 
> 
>> On 18.08.18 17:44, Sankalp Kohli wrote:
>> The thread for side car is months old and no one has opposed to it and hence 
>> someone developed it. I am not sure how else you get consensus. 
>> 
>> Regarding separate repo, how do we get consensus? 
>> 
>>> On Aug 18, 2018, at 05:19, Stefan Podkowinski  wrote:
>>> 
>>> I don't see that we have reached sufficient consensus on this yet. We've
>>> had a brief discussion about the pros and cons of in-tree cassandra vs
>>> separate ASF repo here, but let's not frame it like it's either or. From
>>> my perspective, there hasn't been any final decision yet whether the
>>> proposed side-car solution should be further developed as part of the
>>> Cassandra project, or not.
>>> 
>>> 
 On 18.08.18 03:12, Dinesh Joshi wrote:
 Thanks, Nate. I’ll create this request.
 
 Dinesh
 
 On Aug 17, 2018, at 5:09 PM, Nate McCall  wrote:
 
>> I'm not sure logistically how we get a new repo created and licensing and
>> such, but if someone helps make it we can cut the patch against it
>> 
> This is pretty straight forward. For precedent, see:
> https://issues.apache.org/jira/browse/CASSANDRA-13634
> 
> We currently have three repositories:
> https://git-wip-us.apache.org/repos/asf
> 
> I'm +0 on what approach we take.
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 
 -
 To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
 For additional commands, e-mail: dev-h...@cassandra.apache.org
 
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Stefan Podkowinski]

I think we do have some consensus that 1) we should give it a try to
have one or many side-car processes for non-essential features, and that
2) they should be developed in a separate repo. I'm also open to the
idea of accepting the proposed implementation as a possible side-car
solution and really appreciate effort. But my point is that creating a
new repo, just for the patch, seems to imply that it's also going to
become the de facto official side-car solution, which doesn't feel right
to me, given that the proposed patch isn't even reviewed and hasn't
received much feedback yet.


On 18.08.18 17:44, Sankalp Kohli wrote:
> The thread for side car is months old and no one has opposed to it and hence 
> someone developed it. I am not sure how else you get consensus. 
>
> Regarding separate repo, how do we get consensus? 
>
>> On Aug 18, 2018, at 05:19, Stefan Podkowinski  wrote:
>>
>> I don't see that we have reached sufficient consensus on this yet. We've
>> had a brief discussion about the pros and cons of in-tree cassandra vs
>> separate ASF repo here, but let's not frame it like it's either or. From
>> my perspective, there hasn't been any final decision yet whether the
>> proposed side-car solution should be further developed as part of the
>> Cassandra project, or not.
>>
>>
>>> On 18.08.18 03:12, Dinesh Joshi wrote:
>>> Thanks, Nate. I’ll create this request.
>>>
>>> Dinesh
>>>
>>> On Aug 17, 2018, at 5:09 PM, Nate McCall  wrote:
>>>
> I'm not sure logistically how we get a new repo created and licensing and
> such, but if someone helps make it we can cut the patch against it
>
 This is pretty straight forward. For precedent, see:
 https://issues.apache.org/jira/browse/CASSANDRA-13634

 We currently have three repositories:
 https://git-wip-us.apache.org/repos/asf

 I'm +0 on what approach we take.

 -
 To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
 For additional commands, e-mail: dev-h...@cassandra.apache.org

>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Sankalp Kohli]

The thread for side car is months old and no one has opposed to it and hence 
someone developed it. I am not sure how else you get consensus. 

Regarding separate repo, how do we get consensus? 

> On Aug 18, 2018, at 05:19, Stefan Podkowinski  wrote:
> 
> I don't see that we have reached sufficient consensus on this yet. We've
> had a brief discussion about the pros and cons of in-tree cassandra vs
> separate ASF repo here, but let's not frame it like it's either or. From
> my perspective, there hasn't been any final decision yet whether the
> proposed side-car solution should be further developed as part of the
> Cassandra project, or not.
> 
> 
>> On 18.08.18 03:12, Dinesh Joshi wrote:
>> Thanks, Nate. I’ll create this request.
>> 
>> Dinesh
>> 
>> On Aug 17, 2018, at 5:09 PM, Nate McCall  wrote:
>> 
 I'm not sure logistically how we get a new repo created and licensing and
 such, but if someone helps make it we can cut the patch against it
 
>>> This is pretty straight forward. For precedent, see:
>>> https://issues.apache.org/jira/browse/CASSANDRA-13634
>>> 
>>> We currently have three repositories:
>>> https://git-wip-us.apache.org/repos/asf
>>> 
>>> I'm +0 on what approach we take.
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Stefan Podkowinski]

I don't see that we have reached sufficient consensus on this yet. We've
had a brief discussion about the pros and cons of in-tree cassandra vs
separate ASF repo here, but let's not frame it like it's either or. From
my perspective, there hasn't been any final decision yet whether the
proposed side-car solution should be further developed as part of the
Cassandra project, or not.


On 18.08.18 03:12, Dinesh Joshi wrote:
> Thanks, Nate. I’ll create this request.
>
> Dinesh
>
> On Aug 17, 2018, at 5:09 PM, Nate McCall  wrote:
>
>>> I'm not sure logistically how we get a new repo created and licensing and
>>> such, but if someone helps make it we can cut the patch against it
>>>
>> This is pretty straight forward. For precedent, see:
>> https://issues.apache.org/jira/browse/CASSANDRA-13634
>>
>> We currently have three repositories:
>> https://git-wip-us.apache.org/repos/asf
>>
>> I'm +0 on what approach we take.
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Dinesh Joshi]

Thanks, Nate. I’ll create this request.

Dinesh

On Aug 17, 2018, at 5:09 PM, Nate McCall  wrote:

>> I'm not sure logistically how we get a new repo created and licensing and
>> such, but if someone helps make it we can cut the patch against it
>> 
> This is pretty straight forward. For precedent, see:
> https://issues.apache.org/jira/browse/CASSANDRA-13634
> 
> We currently have three repositories:
> https://git-wip-us.apache.org/repos/asf
> 
> I'm +0 on what approach we take.
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Nate McCall]

> I'm not sure logistically how we get a new repo created and licensing and
> such, but if someone helps make it we can cut the patch against it
>
This is pretty straight forward. For precedent, see:
https://issues.apache.org/jira/browse/CASSANDRA-13634

We currently have three repositories:
https://git-wip-us.apache.org/repos/asf

I'm +0 on what approach we take.

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Vinay Chella]

You can still use the same repo and go with different release cadence as
Jeremiah mentioned, and yes, you can just pull the management sidecar,
test, and rollout without rolling out the server.

It looks like there are merits to both approaches, but it is going to come
down to the appetite of the team to manage several projects, it's releases,
coordination and without compromising on quality. we are fine with either
of the approaches. If this does not work out in the future, we always have
a decision log to refer, learn and move forward.

I'm not sure logistically how we get a new repo created and licensing and
such, but if someone helps make it we can cut the patch against it

Regards,
Vinay Chella


On Fri, Aug 17, 2018 at 10:46 AM Rahul Singh 
wrote:

> I understand the issues of managing different versions of two correlated
> components — but it is possible to create unit tests with core components
> of both. It takes more effort but it is possible.
>
> That being said, in my experience using Reaper and in the DataStax
> distribution , using OpsCenter , I prefer a separate project that is
> loosely tied to the system and not connected at the hips. Whenever there is
> an update to Reaper or OpsCenter, I can always pull it down and test it
> before rolling it out - and this is much more frequently than if I were
> rolling out updates to a C* cluster.
>
>
> Rahul
> On Aug 17, 2018, 9:41 AM -0700, Jonathan Haddad ,
> wrote:
> > Speaking from experience (Reaper), I can say that developing a sidecar
> > admin / repair tool out of tree that's compatible with multiple versions
> > really isn't that big of a problem. We've been doing it for a while now.
> >
> >
> https://github.com/thelastpickle/cassandra-reaper/blob/master/.travis.yml
> >
> > On Fri, Aug 17, 2018 at 9:39 AM Joseph Lynch 
> wrote:
> >
> > > While I would love to use a different build system (e.g. gradle) for
> the
> > > sidecar, I agree with Dinesh that a separate repo would make sidecar
> > > development much harder to verify, especially on the testing and
> > > compatibility front. As Jeremiah mentioned we can always choose later
> to
> > > release the sidecar artifact separately or more frequently than the
> main
> > > server regardless of repo choice and as per Roopa's point having a
> separate
> > > release artifact (jar or deb/rpm) is probably a good idea until the
> default
> > > Cassandra packages don't automatically stop and start Cassandra on
> install.
> > >
> > > While we were developing the repair scheduler in a separate repo we
> had a
> > > number of annoying issues that only started surfacing once we started
> > > merging it directly into the trunk tree:
> > > 1. It is hard to compile/test against unreleased versions of Cassandra
> > > (e.g. the JMX interfaces changed a lot with 4.x, and it was pretty
> tricky
> > > to compile against that as the main project doesn't release nightly
> > > snapshots or anything like that, so we had to build local trunk jars
> and
> > > then manually dep them).
> > > 2. Integration testing and cross version compatibility is extremely
> hard.
> > > The sidecar is going to be involved in multi node coordination (e.g.
> > > monitoring, metrics, maintenance) and will be tightly coupled to JMX
> > > interface choices in the server and trying to make sure that it all
> works
> > > with multiple versions of Cassandra is much harder if it's a separate
> repo
> > > that has to have a mirroring release cycle to Cassandra. It seems much
> > > easier to have it in tree and just be like "the in tree sidecar is
> tested
> > > against that version of Cassandra". Every time we cut a Cassandra
> server
> > > branch the sidecar branches with it.
> > >
> > > We experience these pains all the time with Priam being in a separate
> repo,
> > > where every time we support a new Cassandra version a bunch of JMX
> > > endpoints break and we have to refactor the code to either call JMX
> methods
> > > by string or cut a new Priam branch. A separate artifact is pretty
> > > important, but a separate repo just allows drifts. Furthermore from the
> > > Priam experience I also don't think it's realistic to say one version
> of a
> > > sidecar artifact can actually support multiple versions.
> > >
> > > -Joey
> > >
> > > On Fri, Aug 17, 2018 at 12:00 PM Jeremiah D Jordan <
> jerem...@datastax.com>
> > > wrote:
> > >
> > > > Not sure why the two things being in the same repo means they need
> the
> > > > same release process. You can always do interim releases of the
> > > management
> > > > artifact between server releases, or even have completely decoupled
> > > > releases.
> > > >
> > > > -Jeremiah
> > > >
> > > > > On Aug 17, 2018, at 10:52 AM, Blake Eggleston <
> beggles...@apple.com>
> > > > wrote:
> > > > >
> > > > > I'd be more in favor of making it a separate project, basically
> for all
> > > > the reasons listed below. I'm assuming we'd want a management
> process to
> > > > work across different versions, which will 

Re: Proposing an Apache Cassandra Management process

[dinesh.jo...@yahoo.com.INVALID]

Both approaches have merits. However the general consensus seems to be that we 
should put this in a separate repo and I agree.
Dinesh 

On Friday, August 17, 2018, 10:46:04 AM PDT, Rahul Singh 
 wrote:  
 
 I understand the issues of managing different versions of two correlated 
components — but it is possible to create unit tests with core components of 
both. It takes more effort but it is possible.

That being said, in my experience using Reaper and in the DataStax distribution 
, using OpsCenter , I prefer a separate project that is loosely tied to the 
system and not connected at the hips. Whenever there is an update to Reaper or 
OpsCenter, I can always pull it down and test it before rolling it out - and 
this is much more frequently than if I were rolling out updates to a C* cluster.


Rahul
On Aug 17, 2018, 9:41 AM -0700, Jonathan Haddad , wrote:
> Speaking from experience (Reaper), I can say that developing a sidecar
> admin / repair tool out of tree that's compatible with multiple versions
> really isn't that big of a problem. We've been doing it for a while now.
>
> https://github.com/thelastpickle/cassandra-reaper/blob/master/.travis.yml
>
> On Fri, Aug 17, 2018 at 9:39 AM Joseph Lynch  wrote:
>
> > While I would love to use a different build system (e.g. gradle) for the
> > sidecar, I agree with Dinesh that a separate repo would make sidecar
> > development much harder to verify, especially on the testing and
> > compatibility front. As Jeremiah mentioned we can always choose later to
> > release the sidecar artifact separately or more frequently than the main
> > server regardless of repo choice and as per Roopa's point having a separate
> > release artifact (jar or deb/rpm) is probably a good idea until the default
> > Cassandra packages don't automatically stop and start Cassandra on install.
> >
> > While we were developing the repair scheduler in a separate repo we had a
> > number of annoying issues that only started surfacing once we started
> > merging it directly into the trunk tree:
> > 1. It is hard to compile/test against unreleased versions of Cassandra
> > (e.g. the JMX interfaces changed a lot with 4.x, and it was pretty tricky
> > to compile against that as the main project doesn't release nightly
> > snapshots or anything like that, so we had to build local trunk jars and
> > then manually dep them).
> > 2. Integration testing and cross version compatibility is extremely hard.
> > The sidecar is going to be involved in multi node coordination (e.g.
> > monitoring, metrics, maintenance) and will be tightly coupled to JMX
> > interface choices in the server and trying to make sure that it all works
> > with multiple versions of Cassandra is much harder if it's a separate repo
> > that has to have a mirroring release cycle to Cassandra. It seems much
> > easier to have it in tree and just be like "the in tree sidecar is tested
> > against that version of Cassandra". Every time we cut a Cassandra server
> > branch the sidecar branches with it.
> >
> > We experience these pains all the time with Priam being in a separate repo,
> > where every time we support a new Cassandra version a bunch of JMX
> > endpoints break and we have to refactor the code to either call JMX methods
> > by string or cut a new Priam branch. A separate artifact is pretty
> > important, but a separate repo just allows drifts. Furthermore from the
> > Priam experience I also don't think it's realistic to say one version of a
> > sidecar artifact can actually support multiple versions.
> >
> > -Joey
> >
> > On Fri, Aug 17, 2018 at 12:00 PM Jeremiah D Jordan 
> > wrote:
> >
> > > Not sure why the two things being in the same repo means they need the
> > > same release process. You can always do interim releases of the
> > management
> > > artifact between server releases, or even have completely decoupled
> > > releases.
> > >
> > > -Jeremiah
> > >
> > > > On Aug 17, 2018, at 10:52 AM, Blake Eggleston 
> > > wrote:
> > > >
> > > > I'd be more in favor of making it a separate project, basically for all
> > > the reasons listed below. I'm assuming we'd want a management process to
> > > work across different versions, which will be more awkward if it's in
> > tree.
> > > Even if that's not the case, keeping it in a different repo at this point
> > > will make iteration easier than if it were in tree. I'd imagine (or at
> > > least hope) that validating the management process for release would be
> > > less difficult than the main project, so tying them to the Cassandra
> > > release cycle seems unnecessarily restrictive.
> > > >
> > > >
> > > > On August 17, 2018 at 12:07:18 AM, Dinesh Joshi (
> > dinesh.jo...@yahoo.com.invalid)
> > > wrote:
> > > >
> > > > > On Aug 16, 2018, at 9:27 PM, Sankalp Kohli 
> > > wrote:
> > > > >
> > > > > I am bumping this thread because patch has landed for this with repair
> > > functionality.
> > > > >
> > > > > I have a following proposal for this which I can put in the 

Re: Proposing an Apache Cassandra Management process

[Rahul Singh]

I understand the issues of managing different versions of two correlated 
components — but it is possible to create unit tests with core components of 
both. It takes more effort but it is possible.

That being said, in my experience using Reaper and in the DataStax distribution 
, using OpsCenter , I prefer a separate project that is loosely tied to the 
system and not connected at the hips. Whenever there is an update to Reaper or 
OpsCenter, I can always pull it down and test it before rolling it out - and 
this is much more frequently than if I were rolling out updates to a C* cluster.


Rahul
On Aug 17, 2018, 9:41 AM -0700, Jonathan Haddad , wrote:
> Speaking from experience (Reaper), I can say that developing a sidecar
> admin / repair tool out of tree that's compatible with multiple versions
> really isn't that big of a problem. We've been doing it for a while now.
>
> https://github.com/thelastpickle/cassandra-reaper/blob/master/.travis.yml
>
> On Fri, Aug 17, 2018 at 9:39 AM Joseph Lynch  wrote:
>
> > While I would love to use a different build system (e.g. gradle) for the
> > sidecar, I agree with Dinesh that a separate repo would make sidecar
> > development much harder to verify, especially on the testing and
> > compatibility front. As Jeremiah mentioned we can always choose later to
> > release the sidecar artifact separately or more frequently than the main
> > server regardless of repo choice and as per Roopa's point having a separate
> > release artifact (jar or deb/rpm) is probably a good idea until the default
> > Cassandra packages don't automatically stop and start Cassandra on install.
> >
> > While we were developing the repair scheduler in a separate repo we had a
> > number of annoying issues that only started surfacing once we started
> > merging it directly into the trunk tree:
> > 1. It is hard to compile/test against unreleased versions of Cassandra
> > (e.g. the JMX interfaces changed a lot with 4.x, and it was pretty tricky
> > to compile against that as the main project doesn't release nightly
> > snapshots or anything like that, so we had to build local trunk jars and
> > then manually dep them).
> > 2. Integration testing and cross version compatibility is extremely hard.
> > The sidecar is going to be involved in multi node coordination (e.g.
> > monitoring, metrics, maintenance) and will be tightly coupled to JMX
> > interface choices in the server and trying to make sure that it all works
> > with multiple versions of Cassandra is much harder if it's a separate repo
> > that has to have a mirroring release cycle to Cassandra. It seems much
> > easier to have it in tree and just be like "the in tree sidecar is tested
> > against that version of Cassandra". Every time we cut a Cassandra server
> > branch the sidecar branches with it.
> >
> > We experience these pains all the time with Priam being in a separate repo,
> > where every time we support a new Cassandra version a bunch of JMX
> > endpoints break and we have to refactor the code to either call JMX methods
> > by string or cut a new Priam branch. A separate artifact is pretty
> > important, but a separate repo just allows drifts. Furthermore from the
> > Priam experience I also don't think it's realistic to say one version of a
> > sidecar artifact can actually support multiple versions.
> >
> > -Joey
> >
> > On Fri, Aug 17, 2018 at 12:00 PM Jeremiah D Jordan 
> > wrote:
> >
> > > Not sure why the two things being in the same repo means they need the
> > > same release process. You can always do interim releases of the
> > management
> > > artifact between server releases, or even have completely decoupled
> > > releases.
> > >
> > > -Jeremiah
> > >
> > > > On Aug 17, 2018, at 10:52 AM, Blake Eggleston 
> > > wrote:
> > > >
> > > > I'd be more in favor of making it a separate project, basically for all
> > > the reasons listed below. I'm assuming we'd want a management process to
> > > work across different versions, which will be more awkward if it's in
> > tree.
> > > Even if that's not the case, keeping it in a different repo at this point
> > > will make iteration easier than if it were in tree. I'd imagine (or at
> > > least hope) that validating the management process for release would be
> > > less difficult than the main project, so tying them to the Cassandra
> > > release cycle seems unnecessarily restrictive.
> > > >
> > > >
> > > > On August 17, 2018 at 12:07:18 AM, Dinesh Joshi (
> > dinesh.jo...@yahoo.com.invalid)
> > > wrote:
> > > >
> > > > > On Aug 16, 2018, at 9:27 PM, Sankalp Kohli 
> > > wrote:
> > > > >
> > > > > I am bumping this thread because patch has landed for this with repair
> > > functionality.
> > > > >
> > > > > I have a following proposal for this which I can put in the JIRA or
> > doc
> > > > >
> > > > > 1. We should see if we can keep this in a separate repo like Dtest.
> > > >
> > > > This would imply a looser coupling between the two. Keeping things
> > > in-tree is my 

Re: Proposing an Apache Cassandra Management process

[Jonathan Haddad]

Speaking from experience (Reaper), I can say that developing a sidecar
admin / repair tool out of tree that's compatible with multiple versions
really isn't that big of a problem.  We've been doing it for a while now.

https://github.com/thelastpickle/cassandra-reaper/blob/master/.travis.yml

On Fri, Aug 17, 2018 at 9:39 AM Joseph Lynch  wrote:

> While I would love to use a different build system (e.g. gradle) for the
> sidecar, I agree with Dinesh that a separate repo would make sidecar
> development much harder to verify, especially on the testing and
> compatibility front. As Jeremiah mentioned we can always choose later to
> release the sidecar artifact separately or more frequently than the main
> server regardless of repo choice and as per Roopa's point having a separate
> release artifact (jar or deb/rpm) is probably a good idea until the default
> Cassandra packages don't automatically stop and start Cassandra on install.
>
> While we were developing the repair scheduler in a separate repo we had a
> number of annoying issues that only started surfacing once we started
> merging it directly into the trunk tree:
> 1. It is hard to compile/test against unreleased versions of Cassandra
> (e.g. the JMX interfaces changed a lot with 4.x, and it was pretty tricky
> to compile against that as the main project doesn't release nightly
> snapshots or anything like that, so we had to build local trunk jars and
> then manually dep them).
> 2. Integration testing and cross version compatibility is extremely hard.
> The sidecar is going to be involved in multi node coordination (e.g.
> monitoring, metrics, maintenance) and will be tightly coupled to JMX
> interface choices in the server and trying to make sure that it all works
> with multiple versions of Cassandra is much harder if it's a separate repo
> that has to have a mirroring release cycle to Cassandra. It seems much
> easier to have it in tree and just be like "the in tree sidecar is tested
> against that version of Cassandra". Every time we cut a Cassandra server
> branch the sidecar branches with it.
>
> We experience these pains all the time with Priam being in a separate repo,
> where every time we support a new Cassandra version a bunch of JMX
> endpoints break and we have to refactor the code to either call JMX methods
> by string or cut a new Priam branch. A separate artifact is pretty
> important, but a separate repo just allows drifts. Furthermore from the
> Priam experience I also don't think it's realistic to say one version of a
> sidecar artifact can actually support multiple versions.
>
> -Joey
>
> On Fri, Aug 17, 2018 at 12:00 PM Jeremiah D Jordan 
> wrote:
>
> > Not sure why the two things being in the same repo means they need the
> > same release process.  You can always do interim releases of the
> management
> > artifact between server releases, or even have completely decoupled
> > releases.
> >
> > -Jeremiah
> >
> > > On Aug 17, 2018, at 10:52 AM, Blake Eggleston 
> > wrote:
> > >
> > > I'd be more in favor of making it a separate project, basically for all
> > the reasons listed below. I'm assuming we'd want a management process to
> > work across different versions, which will be more awkward if it's in
> tree.
> > Even if that's not the case, keeping it in a different repo at this point
> > will make iteration easier than if it were in tree. I'd imagine (or at
> > least hope) that validating the management process for release would be
> > less difficult than the main project, so tying them to the Cassandra
> > release cycle seems unnecessarily restrictive.
> > >
> > >
> > > On August 17, 2018 at 12:07:18 AM, Dinesh Joshi (
> dinesh.jo...@yahoo.com.invalid)
> > wrote:
> > >
> > >> On Aug 16, 2018, at 9:27 PM, Sankalp Kohli 
> > wrote:
> > >>
> > >> I am bumping this thread because patch has landed for this with repair
> > functionality.
> > >>
> > >> I have a following proposal for this which I can put in the JIRA or
> doc
> > >>
> > >> 1. We should see if we can keep this in a separate repo like Dtest.
> > >
> > > This would imply a looser coupling between the two. Keeping things
> > in-tree is my preferred approach. It makes testing, dependency management
> > and code sharing easier.
> > >
> > >> 2. It should have its own release process.
> > >
> > > This means now there would be two releases that need to be managed and
> > coordinated.
> > >
> > >> 3. It should have integration tests for different versions of
> Cassandra
> > it will support.
> > >
> > > Given the lack of test infrastructure - this will be hard especially if
> > you have to qualify a matrix of builds.
> > >
> > > Dinesh
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >
> >
> >
> > -
> > To unsubscribe, e-mail: 

Re: Proposing an Apache Cassandra Management process

[Joseph Lynch]

While I would love to use a different build system (e.g. gradle) for the
sidecar, I agree with Dinesh that a separate repo would make sidecar
development much harder to verify, especially on the testing and
compatibility front. As Jeremiah mentioned we can always choose later to
release the sidecar artifact separately or more frequently than the main
server regardless of repo choice and as per Roopa's point having a separate
release artifact (jar or deb/rpm) is probably a good idea until the default
Cassandra packages don't automatically stop and start Cassandra on install.

While we were developing the repair scheduler in a separate repo we had a
number of annoying issues that only started surfacing once we started
merging it directly into the trunk tree:
1. It is hard to compile/test against unreleased versions of Cassandra
(e.g. the JMX interfaces changed a lot with 4.x, and it was pretty tricky
to compile against that as the main project doesn't release nightly
snapshots or anything like that, so we had to build local trunk jars and
then manually dep them).
2. Integration testing and cross version compatibility is extremely hard.
The sidecar is going to be involved in multi node coordination (e.g.
monitoring, metrics, maintenance) and will be tightly coupled to JMX
interface choices in the server and trying to make sure that it all works
with multiple versions of Cassandra is much harder if it's a separate repo
that has to have a mirroring release cycle to Cassandra. It seems much
easier to have it in tree and just be like "the in tree sidecar is tested
against that version of Cassandra". Every time we cut a Cassandra server
branch the sidecar branches with it.

We experience these pains all the time with Priam being in a separate repo,
where every time we support a new Cassandra version a bunch of JMX
endpoints break and we have to refactor the code to either call JMX methods
by string or cut a new Priam branch. A separate artifact is pretty
important, but a separate repo just allows drifts. Furthermore from the
Priam experience I also don't think it's realistic to say one version of a
sidecar artifact can actually support multiple versions.

-Joey

On Fri, Aug 17, 2018 at 12:00 PM Jeremiah D Jordan 
wrote:

> Not sure why the two things being in the same repo means they need the
> same release process.  You can always do interim releases of the management
> artifact between server releases, or even have completely decoupled
> releases.
>
> -Jeremiah
>
> > On Aug 17, 2018, at 10:52 AM, Blake Eggleston 
> wrote:
> >
> > I'd be more in favor of making it a separate project, basically for all
> the reasons listed below. I'm assuming we'd want a management process to
> work across different versions, which will be more awkward if it's in tree.
> Even if that's not the case, keeping it in a different repo at this point
> will make iteration easier than if it were in tree. I'd imagine (or at
> least hope) that validating the management process for release would be
> less difficult than the main project, so tying them to the Cassandra
> release cycle seems unnecessarily restrictive.
> >
> >
> > On August 17, 2018 at 12:07:18 AM, Dinesh Joshi 
> > (dinesh.jo...@yahoo.com.invalid)
> wrote:
> >
> >> On Aug 16, 2018, at 9:27 PM, Sankalp Kohli 
> wrote:
> >>
> >> I am bumping this thread because patch has landed for this with repair
> functionality.
> >>
> >> I have a following proposal for this which I can put in the JIRA or doc
> >>
> >> 1. We should see if we can keep this in a separate repo like Dtest.
> >
> > This would imply a looser coupling between the two. Keeping things
> in-tree is my preferred approach. It makes testing, dependency management
> and code sharing easier.
> >
> >> 2. It should have its own release process.
> >
> > This means now there would be two releases that need to be managed and
> coordinated.
> >
> >> 3. It should have integration tests for different versions of Cassandra
> it will support.
> >
> > Given the lack of test infrastructure - this will be hard especially if
> you have to qualify a matrix of builds.
> >
> > Dinesh
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Proposing an Apache Cassandra Management process

[Roopa]

+1 in maintaining a separate project and release cycle for side car. We have 
been running side car in production for 6+ years and the rate of changes to the 
side car is much higher than to the actual data store. This will enable faster 
iteration needed for the side car and help folks roll out maintenance fixes 
easily. 

Thanks
Roopa



> On Aug 17, 2018, at 8:52 AM, Blake Eggleston  wrote:
> 
> I'd be more in favor of making it a separate project, basically for all the 
> reasons listed below. I'm assuming we'd want a management process to work 
> across different versions, which will be more awkward if it's in tree. Even 
> if that's not the case, keeping it in a different repo at this point will 
> make iteration easier than if it were in tree. I'd imagine (or at least hope) 
> that validating the management process for release would be less difficult 
> than the main project, so tying them to the Cassandra release cycle seems 
> unnecessarily restrictive.
> 
> 
>> On August 17, 2018 at 12:07:18 AM, Dinesh Joshi 
>> (dinesh.jo...@yahoo.com.invalid) wrote:
>> 
>> On Aug 16, 2018, at 9:27 PM, Sankalp Kohli  wrote: 
>> 
>> I am bumping this thread because patch has landed for this with repair 
>> functionality. 
>> 
>> I have a following proposal for this which I can put in the JIRA or doc 
>> 
>> 1. We should see if we can keep this in a separate repo like Dtest. 
> 
> This would imply a looser coupling between the two. Keeping things in-tree is 
> my preferred approach. It makes testing, dependency management and code 
> sharing easier. 
> 
>> 2. It should have its own release process. 
> 
> This means now there would be two releases that need to be managed and 
> coordinated. 
> 
>> 3. It should have integration tests for different versions of Cassandra it 
>> will support. 
> 
> Given the lack of test infrastructure - this will be hard especially if you 
> have to qualify a matrix of builds. 
> 
> Dinesh 
> - 
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org 
> For additional commands, e-mail: dev-h...@cassandra.apache.org 
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Jeremiah D Jordan]

Not sure why the two things being in the same repo means they need the same 
release process.  You can always do interim releases of the management artifact 
between server releases, or even have completely decoupled releases.

-Jeremiah

> On Aug 17, 2018, at 10:52 AM, Blake Eggleston  wrote:
> 
> I'd be more in favor of making it a separate project, basically for all the 
> reasons listed below. I'm assuming we'd want a management process to work 
> across different versions, which will be more awkward if it's in tree. Even 
> if that's not the case, keeping it in a different repo at this point will 
> make iteration easier than if it were in tree. I'd imagine (or at least hope) 
> that validating the management process for release would be less difficult 
> than the main project, so tying them to the Cassandra release cycle seems 
> unnecessarily restrictive.
> 
> 
> On August 17, 2018 at 12:07:18 AM, Dinesh Joshi 
> (dinesh.jo...@yahoo.com.invalid) wrote:
> 
>> On Aug 16, 2018, at 9:27 PM, Sankalp Kohli  wrote: 
>> 
>> I am bumping this thread because patch has landed for this with repair 
>> functionality. 
>> 
>> I have a following proposal for this which I can put in the JIRA or doc 
>> 
>> 1. We should see if we can keep this in a separate repo like Dtest. 
> 
> This would imply a looser coupling between the two. Keeping things in-tree is 
> my preferred approach. It makes testing, dependency management and code 
> sharing easier. 
> 
>> 2. It should have its own release process. 
> 
> This means now there would be two releases that need to be managed and 
> coordinated. 
> 
>> 3. It should have integration tests for different versions of Cassandra it 
>> will support. 
> 
> Given the lack of test infrastructure - this will be hard especially if you 
> have to qualify a matrix of builds. 
> 
> Dinesh 
> - 
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org 
> For additional commands, e-mail: dev-h...@cassandra.apache.org 
> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Blake Eggleston]

I'd be more in favor of making it a separate project, basically for all the 
reasons listed below. I'm assuming we'd want a management process to work 
across different versions, which will be more awkward if it's in tree. Even if 
that's not the case, keeping it in a different repo at this point will make 
iteration easier than if it were in tree. I'd imagine (or at least hope) that 
validating the management process for release would be less difficult than the 
main project, so tying them to the Cassandra release cycle seems unnecessarily 
restrictive.


On August 17, 2018 at 12:07:18 AM, Dinesh Joshi 
(dinesh.jo...@yahoo.com.invalid) wrote:

> On Aug 16, 2018, at 9:27 PM, Sankalp Kohli  wrote: 
> 
> I am bumping this thread because patch has landed for this with repair 
> functionality. 
> 
> I have a following proposal for this which I can put in the JIRA or doc 
> 
> 1. We should see if we can keep this in a separate repo like Dtest. 

This would imply a looser coupling between the two. Keeping things in-tree is 
my preferred approach. It makes testing, dependency management and code sharing 
easier. 

> 2. It should have its own release process. 

This means now there would be two releases that need to be managed and 
coordinated. 

> 3. It should have integration tests for different versions of Cassandra it 
> will support. 

Given the lack of test infrastructure - this will be hard especially if you 
have to qualify a matrix of builds. 

Dinesh 
- 
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org 
For additional commands, e-mail: dev-h...@cassandra.apache.org 

Re: Proposing an Apache Cassandra Management process

[Dinesh Joshi]

> On Aug 16, 2018, at 9:27 PM, Sankalp Kohli  wrote:
> 
> I am bumping this thread because patch has landed for this with repair 
> functionality. 
> 
> I have a following proposal for this which I can put in the JIRA or doc 
> 
> 1. We should see if we can keep this in a separate repo like Dtest. 

This would imply a looser coupling between the two. Keeping things in-tree is 
my preferred approach. It makes testing, dependency management and code sharing 
easier.

> 2. It should have its own release process.

This means now there would be two releases that need to be managed and 
coordinated.

> 3. It should have integration tests for different versions of Cassandra it 
> will support. 

Given the lack of test infrastructure - this will be hard especially if you 
have to qualify a matrix of builds. 

Dinesh
-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Sankalp Kohli]

I am bumping this thread because patch has landed for this with repair 
functionality. 

I have a following proposal for this which I can put in the JIRA or doc 

1. We should see if we can keep this in a separate repo like Dtest. 
2. It should have its own release process.
3. It should have integration tests for different versions of Cassandra it will 
support. 

Does anyone has any ideas on this ? 

Thanks
Sankalp 

> On Apr 18, 2018, at 19:20, Dinesh Joshi  
> wrote:
> 
> Thank you all for the feedback. Nate made a Google doc with the proposal in 
> it and is linked off of the ticket. I will try to flesh it out as I gather 
> your input.
> Dinesh 
> 
>On Wednesday, April 18, 2018, 5:12:49 PM PDT, kurt greaves 
>  wrote:  
> 
> For anyone looking Dinesh made a ticket already: CASSANDRA-14395
> 
> 
>> On 18 April 2018 at 18:14, Vinay Chella  wrote:
>> 
>> This is a good initiative. We have advocated for and run a sidecar for the
>> past 5+ years, and we've learned and improved it a lot. We look forward to
>> moving features from Priam (such as backup, HTTP -> JMX, etc) incrementally
>> to this sidecar as they make sense.
>> 
>> 
>> Thanks,
>> Vinay Chella
>> 
>> On Fri, Apr 13, 2018 at 7:01 AM, Eric Evans 
>> wrote:
>> 
>>> On Thu, Apr 12, 2018 at 4:41 PM, Dinesh Joshi
>>>  wrote:
 Hey all -
 With the uptick in discussion around Cassandra operability and after
>>> discussing potential solutions with various members of the community, we
>>> would like to propose the addition of a management process/sub-project
>> into
>>> Apache Cassandra. The process would be responsible for common operational
>>> tasks like bulk execution of nodetool commands, backup/restore, and
>> health
>>> checks, among others. We feel we have a proposal that will garner some
>>> discussion and debate but is likely to reach consensus.
 While the community, in large part, agrees that these features should
>>> exist “in the database”, there is debate on how they should be
>> implemented.
>>> Primarily, whether or not to use an external process or build on
>>> CassandraDaemon. This is an important architectural decision but we feel
>>> the most critical aspect is not where the code runs but that the operator
>>> still interacts with the notion of a single database. Multi-process
>>> databases are as old as Postgres and continue to be common in newer
>> systems
>>> like Druid. As such, we propose a separate management process for the
>>> following reasons:
 
 - Resource isolation & Safety: Features in the management process
>>> will not affect C*'s read/write path which is critical for stability. An
>>> isolated process has several technical advantages including preventing
>> use
>>> of unnecessary dependencies in CassandraDaemon, separation of JVM
>> resources
>>> like thread pools and heap, and preventing bugs from adversely affecting
>>> the main process. In particular, GC tuning can be done separately for the
>>> two processes, hopefully helping to improve, or at least not adversely
>>> affect, tail latencies of the main process.
 
 - Health Checks & Recovery: Currently users implement health checks
>>> in their own sidecar process. Implementing them in the serving process
>> does
>>> not make sense because if the JVM running the CassandraDaemon goes south,
>>> the healthchecks and potentially any recovery code may not be able to
>> run.
>>> Having a management process running in isolation opens up the possibility
>>> to not only report the health of the C* process such as long GC pauses or
>>> stuck JVM but also to recover from it. Having a list of basic health
>> checks
>>> that are tested with every C* release and officially supported will help
>>> boost confidence in C* quality and make it easier to operate.
 
 - Reduced Risk: By having a separate Daemon we open the possibility
>>> to contribute features that otherwise would not have been considered
>> before
>>> eg. a UI. A library that started many background threads and is operated
>>> completely differently would likely be considered too risky for
>>> CassandraDaemon but is a good candidate for the management process.
>>> 
>>> Makes sense IMO.
>>> 
 What can go into the management process?
 - Features that are non-essential for serving reads & writes for eg.
>>> Backup/Restore or Running Health Checks against the CassandraDaemon, etc.
 
 - Features that do not make the management process critical for
>>> functioning of the serving process. In other words, if someone does not
>>> wish to use this management process, they are free to disable it.
 
 We would like to initially build minimal set of features such as health
>>> checks and bulk commands into the first iteration of the management
>>> process. We would use the same software stack that is used to build the
>>> current CassandraDaemon binary. This would be critical for sharing code

Re: Proposing an Apache Cassandra Management process

[Dinesh Joshi]

Thank you all for the feedback. Nate made a Google doc with the proposal in it 
and is linked off of the ticket. I will try to flesh it out as I gather your 
input.
Dinesh 

On Wednesday, April 18, 2018, 5:12:49 PM PDT, kurt greaves 
 wrote:  
 
 For anyone looking Dinesh made a ticket already: CASSANDRA-14395


On 18 April 2018 at 18:14, Vinay Chella  wrote:

> This is a good initiative. We have advocated for and run a sidecar for the
> past 5+ years, and we've learned and improved it a lot. We look forward to
> moving features from Priam (such as backup, HTTP -> JMX, etc) incrementally
> to this sidecar as they make sense.
>
>
> Thanks,
> Vinay Chella
>
> On Fri, Apr 13, 2018 at 7:01 AM, Eric Evans 
> wrote:
>
> > On Thu, Apr 12, 2018 at 4:41 PM, Dinesh Joshi
> >  wrote:
> > > Hey all -
> > > With the uptick in discussion around Cassandra operability and after
> > discussing potential solutions with various members of the community, we
> > would like to propose the addition of a management process/sub-project
> into
> > Apache Cassandra. The process would be responsible for common operational
> > tasks like bulk execution of nodetool commands, backup/restore, and
> health
> > checks, among others. We feel we have a proposal that will garner some
> > discussion and debate but is likely to reach consensus.
> > > While the community, in large part, agrees that these features should
> > exist “in the database”, there is debate on how they should be
> implemented.
> > Primarily, whether or not to use an external process or build on
> > CassandraDaemon. This is an important architectural decision but we feel
> > the most critical aspect is not where the code runs but that the operator
> > still interacts with the notion of a single database. Multi-process
> > databases are as old as Postgres and continue to be common in newer
> systems
> > like Druid. As such, we propose a separate management process for the
> > following reasons:
> > >
> > >    - Resource isolation & Safety: Features in the management process
> > will not affect C*'s read/write path which is critical for stability. An
> > isolated process has several technical advantages including preventing
> use
> > of unnecessary dependencies in CassandraDaemon, separation of JVM
> resources
> > like thread pools and heap, and preventing bugs from adversely affecting
> > the main process. In particular, GC tuning can be done separately for the
> > two processes, hopefully helping to improve, or at least not adversely
> > affect, tail latencies of the main process.
> > >
> > >    - Health Checks & Recovery: Currently users implement health checks
> > in their own sidecar process. Implementing them in the serving process
> does
> > not make sense because if the JVM running the CassandraDaemon goes south,
> > the healthchecks and potentially any recovery code may not be able to
> run.
> > Having a management process running in isolation opens up the possibility
> > to not only report the health of the C* process such as long GC pauses or
> > stuck JVM but also to recover from it. Having a list of basic health
> checks
> > that are tested with every C* release and officially supported will help
> > boost confidence in C* quality and make it easier to operate.
> > >
> > >    - Reduced Risk: By having a separate Daemon we open the possibility
> > to contribute features that otherwise would not have been considered
> before
> > eg. a UI. A library that started many background threads and is operated
> > completely differently would likely be considered too risky for
> > CassandraDaemon but is a good candidate for the management process.
> >
> > Makes sense IMO.
> >
> > > What can go into the management process?
> > >    - Features that are non-essential for serving reads & writes for eg.
> > Backup/Restore or Running Health Checks against the CassandraDaemon, etc.
> > >
> > >    - Features that do not make the management process critical for
> > functioning of the serving process. In other words, if someone does not
> > wish to use this management process, they are free to disable it.
> > >
> > > We would like to initially build minimal set of features such as health
> > checks and bulk commands into the first iteration of the management
> > process. We would use the same software stack that is used to build the
> > current CassandraDaemon binary. This would be critical for sharing code
> > between CassandraDaemon & management processes. The code should live
> > in-tree to make this easy.
> > > With regards to more in-depth features like repair scheduling and
> > discussions around compaction in or out of CassandraDaemon, while the
> > management process may be a suitable host, it is not our goal to decide
> > that at this time. The management process could be used in these cases,
> as
> > they meet 

Re: Proposing an Apache Cassandra Management process

[kurt greaves]

For anyone looking Dinesh made a ticket already: CASSANDRA-14395


On 18 April 2018 at 18:14, Vinay Chella  wrote:

> This is a good initiative. We have advocated for and run a sidecar for the
> past 5+ years, and we've learned and improved it a lot. We look forward to
> moving features from Priam (such as backup, HTTP -> JMX, etc) incrementally
> to this sidecar as they make sense.
>
>
> Thanks,
> Vinay Chella
>
> On Fri, Apr 13, 2018 at 7:01 AM, Eric Evans 
> wrote:
>
> > On Thu, Apr 12, 2018 at 4:41 PM, Dinesh Joshi
> >  wrote:
> > > Hey all -
> > > With the uptick in discussion around Cassandra operability and after
> > discussing potential solutions with various members of the community, we
> > would like to propose the addition of a management process/sub-project
> into
> > Apache Cassandra. The process would be responsible for common operational
> > tasks like bulk execution of nodetool commands, backup/restore, and
> health
> > checks, among others. We feel we have a proposal that will garner some
> > discussion and debate but is likely to reach consensus.
> > > While the community, in large part, agrees that these features should
> > exist “in the database”, there is debate on how they should be
> implemented.
> > Primarily, whether or not to use an external process or build on
> > CassandraDaemon. This is an important architectural decision but we feel
> > the most critical aspect is not where the code runs but that the operator
> > still interacts with the notion of a single database. Multi-process
> > databases are as old as Postgres and continue to be common in newer
> systems
> > like Druid. As such, we propose a separate management process for the
> > following reasons:
> > >
> > >- Resource isolation & Safety: Features in the management process
> > will not affect C*'s read/write path which is critical for stability. An
> > isolated process has several technical advantages including preventing
> use
> > of unnecessary dependencies in CassandraDaemon, separation of JVM
> resources
> > like thread pools and heap, and preventing bugs from adversely affecting
> > the main process. In particular, GC tuning can be done separately for the
> > two processes, hopefully helping to improve, or at least not adversely
> > affect, tail latencies of the main process.
> > >
> > >- Health Checks & Recovery: Currently users implement health checks
> > in their own sidecar process. Implementing them in the serving process
> does
> > not make sense because if the JVM running the CassandraDaemon goes south,
> > the healthchecks and potentially any recovery code may not be able to
> run.
> > Having a management process running in isolation opens up the possibility
> > to not only report the health of the C* process such as long GC pauses or
> > stuck JVM but also to recover from it. Having a list of basic health
> checks
> > that are tested with every C* release and officially supported will help
> > boost confidence in C* quality and make it easier to operate.
> > >
> > >- Reduced Risk: By having a separate Daemon we open the possibility
> > to contribute features that otherwise would not have been considered
> before
> > eg. a UI. A library that started many background threads and is operated
> > completely differently would likely be considered too risky for
> > CassandraDaemon but is a good candidate for the management process.
> >
> > Makes sense IMO.
> >
> > > What can go into the management process?
> > >- Features that are non-essential for serving reads & writes for eg.
> > Backup/Restore or Running Health Checks against the CassandraDaemon, etc.
> > >
> > >- Features that do not make the management process critical for
> > functioning of the serving process. In other words, if someone does not
> > wish to use this management process, they are free to disable it.
> > >
> > > We would like to initially build minimal set of features such as health
> > checks and bulk commands into the first iteration of the management
> > process. We would use the same software stack that is used to build the
> > current CassandraDaemon binary. This would be critical for sharing code
> > between CassandraDaemon & management processes. The code should live
> > in-tree to make this easy.
> > > With regards to more in-depth features like repair scheduling and
> > discussions around compaction in or out of CassandraDaemon, while the
> > management process may be a suitable host, it is not our goal to decide
> > that at this time. The management process could be used in these cases,
> as
> > they meet the criteria above, but other technical/architectural reasons
> may
> > exists for why it should not be.
> > > We are looking forward to your comments on our proposal,
> >
> > Sounds good to me.
> >
> > Personally, I'm a little less interested in things like
> > 

Re: Proposing an Apache Cassandra Management process

[Vinay Chella]

This is a good initiative. We have advocated for and run a sidecar for the
past 5+ years, and we've learned and improved it a lot. We look forward to
moving features from Priam (such as backup, HTTP -> JMX, etc) incrementally
to this sidecar as they make sense.


Thanks,
Vinay Chella

On Fri, Apr 13, 2018 at 7:01 AM, Eric Evans 
wrote:

> On Thu, Apr 12, 2018 at 4:41 PM, Dinesh Joshi
>  wrote:
> > Hey all -
> > With the uptick in discussion around Cassandra operability and after
> discussing potential solutions with various members of the community, we
> would like to propose the addition of a management process/sub-project into
> Apache Cassandra. The process would be responsible for common operational
> tasks like bulk execution of nodetool commands, backup/restore, and health
> checks, among others. We feel we have a proposal that will garner some
> discussion and debate but is likely to reach consensus.
> > While the community, in large part, agrees that these features should
> exist “in the database”, there is debate on how they should be implemented.
> Primarily, whether or not to use an external process or build on
> CassandraDaemon. This is an important architectural decision but we feel
> the most critical aspect is not where the code runs but that the operator
> still interacts with the notion of a single database. Multi-process
> databases are as old as Postgres and continue to be common in newer systems
> like Druid. As such, we propose a separate management process for the
> following reasons:
> >
> >- Resource isolation & Safety: Features in the management process
> will not affect C*'s read/write path which is critical for stability. An
> isolated process has several technical advantages including preventing use
> of unnecessary dependencies in CassandraDaemon, separation of JVM resources
> like thread pools and heap, and preventing bugs from adversely affecting
> the main process. In particular, GC tuning can be done separately for the
> two processes, hopefully helping to improve, or at least not adversely
> affect, tail latencies of the main process.
> >
> >- Health Checks & Recovery: Currently users implement health checks
> in their own sidecar process. Implementing them in the serving process does
> not make sense because if the JVM running the CassandraDaemon goes south,
> the healthchecks and potentially any recovery code may not be able to run.
> Having a management process running in isolation opens up the possibility
> to not only report the health of the C* process such as long GC pauses or
> stuck JVM but also to recover from it. Having a list of basic health checks
> that are tested with every C* release and officially supported will help
> boost confidence in C* quality and make it easier to operate.
> >
> >- Reduced Risk: By having a separate Daemon we open the possibility
> to contribute features that otherwise would not have been considered before
> eg. a UI. A library that started many background threads and is operated
> completely differently would likely be considered too risky for
> CassandraDaemon but is a good candidate for the management process.
>
> Makes sense IMO.
>
> > What can go into the management process?
> >- Features that are non-essential for serving reads & writes for eg.
> Backup/Restore or Running Health Checks against the CassandraDaemon, etc.
> >
> >- Features that do not make the management process critical for
> functioning of the serving process. In other words, if someone does not
> wish to use this management process, they are free to disable it.
> >
> > We would like to initially build minimal set of features such as health
> checks and bulk commands into the first iteration of the management
> process. We would use the same software stack that is used to build the
> current CassandraDaemon binary. This would be critical for sharing code
> between CassandraDaemon & management processes. The code should live
> in-tree to make this easy.
> > With regards to more in-depth features like repair scheduling and
> discussions around compaction in or out of CassandraDaemon, while the
> management process may be a suitable host, it is not our goal to decide
> that at this time. The management process could be used in these cases, as
> they meet the criteria above, but other technical/architectural reasons may
> exists for why it should not be.
> > We are looking forward to your comments on our proposal,
>
> Sounds good to me.
>
> Personally, I'm a little less interested in things like
> health/availability checks and metrics collection, because there are
> already tools to solve this problem (and most places will already be
> using them).  I'm more interested in things like cluster status,
> streaming, repair, etc.  Something to automate/centralize
> database-specific command and control, and improve visibility.
>
> In-tree also makes sense (tools/ maybe?), but I would 

Re: Proposing an Apache Cassandra Management process

[Eric Evans]

On Thu, Apr 12, 2018 at 4:41 PM, Dinesh Joshi
 wrote:
> Hey all -
> With the uptick in discussion around Cassandra operability and after 
> discussing potential solutions with various members of the community, we 
> would like to propose the addition of a management process/sub-project into 
> Apache Cassandra. The process would be responsible for common operational 
> tasks like bulk execution of nodetool commands, backup/restore, and health 
> checks, among others. We feel we have a proposal that will garner some 
> discussion and debate but is likely to reach consensus.
> While the community, in large part, agrees that these features should exist 
> “in the database”, there is debate on how they should be implemented. 
> Primarily, whether or not to use an external process or build on 
> CassandraDaemon. This is an important architectural decision but we feel the 
> most critical aspect is not where the code runs but that the operator still 
> interacts with the notion of a single database. Multi-process databases are 
> as old as Postgres and continue to be common in newer systems like Druid. As 
> such, we propose a separate management process for the following reasons:
>
>- Resource isolation & Safety: Features in the management process will not 
> affect C*'s read/write path which is critical for stability. An isolated 
> process has several technical advantages including preventing use of 
> unnecessary dependencies in CassandraDaemon, separation of JVM resources like 
> thread pools and heap, and preventing bugs from adversely affecting the main 
> process. In particular, GC tuning can be done separately for the two 
> processes, hopefully helping to improve, or at least not adversely affect, 
> tail latencies of the main process.
>
>- Health Checks & Recovery: Currently users implement health checks in 
> their own sidecar process. Implementing them in the serving process does not 
> make sense because if the JVM running the CassandraDaemon goes south, the 
> healthchecks and potentially any recovery code may not be able to run. Having 
> a management process running in isolation opens up the possibility to not 
> only report the health of the C* process such as long GC pauses or stuck JVM 
> but also to recover from it. Having a list of basic health checks that are 
> tested with every C* release and officially supported will help boost 
> confidence in C* quality and make it easier to operate.
>
>- Reduced Risk: By having a separate Daemon we open the possibility to 
> contribute features that otherwise would not have been considered before eg. 
> a UI. A library that started many background threads and is operated 
> completely differently would likely be considered too risky for 
> CassandraDaemon but is a good candidate for the management process.

Makes sense IMO.

> What can go into the management process?
>- Features that are non-essential for serving reads & writes for eg. 
> Backup/Restore or Running Health Checks against the CassandraDaemon, etc.
>
>- Features that do not make the management process critical for 
> functioning of the serving process. In other words, if someone does not wish 
> to use this management process, they are free to disable it.
>
> We would like to initially build minimal set of features such as health 
> checks and bulk commands into the first iteration of the management process. 
> We would use the same software stack that is used to build the current 
> CassandraDaemon binary. This would be critical for sharing code between 
> CassandraDaemon & management processes. The code should live in-tree to make 
> this easy.
> With regards to more in-depth features like repair scheduling and discussions 
> around compaction in or out of CassandraDaemon, while the management process 
> may be a suitable host, it is not our goal to decide that at this time. The 
> management process could be used in these cases, as they meet the criteria 
> above, but other technical/architectural reasons may exists for why it should 
> not be.
> We are looking forward to your comments on our proposal,

Sounds good to me.

Personally, I'm a little less interested in things like
health/availability checks and metrics collection, because there are
already tools to solve this problem (and most places will already be
using them).  I'm more interested in things like cluster status,
streaming, repair, etc.  Something to automate/centralize
database-specific command and control, and improve visibility.

In-tree also makes sense (tools/ maybe?), but I would suggest working
out of a branch initially, and seeking inclusion when there is
something more concrete to discuss.


-- 
Eric Evans
john.eric.ev...@gmail.com

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Jaydeep Chovatia]

In my opinion this will be a great addition to the Cassandra and will take
overall Cassandra project to next level. This will also improve user
experience especially for new users.

Jaydeep

On Thu, Apr 12, 2018 at 2:42 PM Dinesh Joshi 
wrote:

> Hey all -
> With the uptick in discussion around Cassandra operability and after
> discussing potential solutions with various members of the community, we
> would like to propose the addition of a management process/sub-project into
> Apache Cassandra. The process would be responsible for common operational
> tasks like bulk execution of nodetool commands, backup/restore, and health
> checks, among others. We feel we have a proposal that will garner some
> discussion and debate but is likely to reach consensus.
> While the community, in large part, agrees that these features should
> exist “in the database”, there is debate on how they should be implemented.
> Primarily, whether or not to use an external process or build on
> CassandraDaemon. This is an important architectural decision but we feel
> the most critical aspect is not where the code runs but that the operator
> still interacts with the notion of a single database. Multi-process
> databases are as old as Postgres and continue to be common in newer systems
> like Druid. As such, we propose a separate management process for the
> following reasons:
>
>- Resource isolation & Safety: Features in the management process will
> not affect C*'s read/write path which is critical for stability. An
> isolated process has several technical advantages including preventing use
> of unnecessary dependencies in CassandraDaemon, separation of JVM resources
> like thread pools and heap, and preventing bugs from adversely affecting
> the main process. In particular, GC tuning can be done separately for the
> two processes, hopefully helping to improve, or at least not adversely
> affect, tail latencies of the main process.
>
>- Health Checks & Recovery: Currently users implement health checks in
> their own sidecar process. Implementing them in the serving process does
> not make sense because if the JVM running the CassandraDaemon goes south,
> the healthchecks and potentially any recovery code may not be able to run.
> Having a management process running in isolation opens up the possibility
> to not only report the health of the C* process such as long GC pauses or
> stuck JVM but also to recover from it. Having a list of basic health checks
> that are tested with every C* release and officially supported will help
> boost confidence in C* quality and make it easier to operate.
>
>- Reduced Risk: By having a separate Daemon we open the possibility to
> contribute features that otherwise would not have been considered before
> eg. a UI. A library that started many background threads and is operated
> completely differently would likely be considered too risky for
> CassandraDaemon but is a good candidate for the management process.
>
>
> What can go into the management process?
>- Features that are non-essential for serving reads & writes for eg.
> Backup/Restore or Running Health Checks against the CassandraDaemon, etc.
>
>- Features that do not make the management process critical for
> functioning of the serving process. In other words, if someone does not
> wish to use this management process, they are free to disable it.
>
> We would like to initially build minimal set of features such as health
> checks and bulk commands into the first iteration of the management
> process. We would use the same software stack that is used to build the
> current CassandraDaemon binary. This would be critical for sharing code
> between CassandraDaemon & management processes. The code should live
> in-tree to make this easy.
> With regards to more in-depth features like repair scheduling and
> discussions around compaction in or out of CassandraDaemon, while the
> management process may be a suitable host, it is not our goal to decide
> that at this time. The management process could be used in these cases, as
> they meet the criteria above, but other technical/architectural reasons may
> exists for why it should not be.
> We are looking forward to your comments on our proposal,
> Dinesh Joshi and Jordan West

Proposing an Apache Cassandra Management process

[Dinesh Joshi]

Hey all -
With the uptick in discussion around Cassandra operability and after discussing 
potential solutions with various members of the community, we would like to 
propose the addition of a management process/sub-project into Apache Cassandra. 
The process would be responsible for common operational tasks like bulk 
execution of nodetool commands, backup/restore, and health checks, among 
others. We feel we have a proposal that will garner some discussion and debate 
but is likely to reach consensus.
While the community, in large part, agrees that these features should exist “in 
the database”, there is debate on how they should be implemented. Primarily, 
whether or not to use an external process or build on CassandraDaemon. This is 
an important architectural decision but we feel the most critical aspect is not 
where the code runs but that the operator still interacts with the notion of a 
single database. Multi-process databases are as old as Postgres and continue to 
be common in newer systems like Druid. As such, we propose a separate 
management process for the following reasons:
   
   - Resource isolation & Safety: Features in the management process will not 
affect C*'s read/write path which is critical for stability. An isolated 
process has several technical advantages including preventing use of 
unnecessary dependencies in CassandraDaemon, separation of JVM resources like 
thread pools and heap, and preventing bugs from adversely affecting the main 
process. In particular, GC tuning can be done separately for the two processes, 
hopefully helping to improve, or at least not adversely affect, tail latencies 
of the main process.    

   - Health Checks & Recovery: Currently users implement health checks in their 
own sidecar process. Implementing them in the serving process does not make 
sense because if the JVM running the CassandraDaemon goes south, the 
healthchecks and potentially any recovery code may not be able to run. Having a 
management process running in isolation opens up the possibility to not only 
report the health of the C* process such as long GC pauses or stuck JVM but 
also to recover from it. Having a list of basic health checks that are tested 
with every C* release and officially supported will help boost confidence in C* 
quality and make it easier to operate.   

   - Reduced Risk: By having a separate Daemon we open the possibility to 
contribute features that otherwise would not have been considered before eg. a 
UI. A library that started many background threads and is operated completely 
differently would likely be considered too risky for CassandraDaemon but is a 
good candidate for the management process.   


What can go into the management process?   
   - Features that are non-essential for serving reads & writes for eg. 
Backup/Restore or Running Health Checks against the CassandraDaemon, etc.   

   - Features that do not make the management process critical for functioning 
of the serving process. In other words, if someone does not wish to use this 
management process, they are free to disable it.

We would like to initially build minimal set of features such as health checks 
and bulk commands into the first iteration of the management process. We would 
use the same software stack that is used to build the current CassandraDaemon 
binary. This would be critical for sharing code between CassandraDaemon & 
management processes. The code should live in-tree to make this easy.
With regards to more in-depth features like repair scheduling and discussions 
around compaction in or out of CassandraDaemon, while the management process 
may be a suitable host, it is not our goal to decide that at this time. The 
management process could be used in these cases, as they meet the criteria 
above, but other technical/architectural reasons may exists for why it should 
not be.
We are looking forward to your comments on our proposal,
Dinesh Joshi and Jordan West