Re: Audit logging to tables.

[Jonathan Haddad]

Sagar,

There isn't going to be much in the way of docs, since it's brand new and
not really a public facing thing yet.  As Chris pointed out, there's other
work that would need to be done to work on virtual tables for large
datasets.

Jon

On Mon, Mar 4, 2019 at 6:42 AM Chris Lohfink  wrote:

> While you probably could put a virtual table wrapper over the binlogs, you
> would want to wait for something like
> https://issues.apache.org/jira/browse/CASSANDRA-14629 to get in so you
> would not OOM. The current virtual table implementation requires you have
> the entire result set to be returned at once.
>
> Chris
>
> On Mon, Mar 4, 2019 at 5:29 AM Sagar  wrote:
>
> > Hi Jonathan,
> >
> > I couldn't find much literature on Virtual tables apart from this ticket:
> >
> > https://issues.apache.org/jira/browse/CASSANDRA-7622
> >
> > Any insights would be helpful.
> >
> > Thanks!
> > Sagar.
> >
> > On Sat, Mar 2, 2019 at 7:23 AM Jonathan Haddad 
> wrote:
> >
> > > Instead of logging to tables, putting a virtual table around the audit
> /
> > > query logs might be an option. Same with the commit log for cdc
> > >
> > > On Fri, Mar 1, 2019 at 5:25 PM Sagar 
> wrote:
> > >
> > > > Thanks all for the pointers. Really insightful.
> > > >
> > > > Subroto I think that’s part of the enterprise version but yeah even I
> > > have
> > > > seen it. Again not sure of the performance implications.
> > > >
> > > > Sagar.
> > > >
> > > > On Sat, 2 Mar 2019 at 5:15 AM, Subroto Barua
> >  > > >
> > > > wrote:
> > > >
> > > > > Datastax version has an option to store audit info to
> > > dse_audit.audit_log
> > > > > table; I do not know the performance impact since I use the file
> > option
> > > > >
> > > > > Subroto
> > > > >
> > > > > > On Mar 1, 2019, at 9:40 AM, Jeremiah D Jordan <
> > > > jeremiah.jor...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > AFAIK the Full Query Logging binary format was already made more
> > > > general
> > > > > in order to support using that format for the audit logging.
> > > > > >
> > > > > > -Jeremiah
> > > > > >
> > > > > >> On Mar 1, 2019, at 11:38 AM, Joshua McKenzie <
> > jmcken...@apache.org>
> > > > > wrote:
> > > > > >>
> > > > > >> Is there a world in which a general purpose, side-channel file
> > > storage
> > > > > >> format for transient things like this (hints, batches, audit
> logs,
> > > > etc)
> > > > > >> could be useful as a first class citizen in the codebase? i.e. a
> > > world
> > > > > in
> > > > > >> which we refactored some of the hints-specific reader/writer
> code
> > to
> > > > be
> > > > > >> used for things like this if/when they come up?
> > > > > >>
> > > > > >>> On Thu, Feb 28, 2019 at 12:04 PM Jonathan Haddad <
> > > j...@jonhaddad.com
> > > > > <mailto:j...@jonhaddad.com>> wrote:
> > > > > >>>
> > > > > >>> Agreed with Dinesh and Josh.  I would *never* put the audit log
> > > back
> > > > in
> > > > > >>> Cassandra.
> > > > > >>>
> > > > > >>> This is extendable, Sagar, so you're free to do as you want,
> but
> > > I'm
> > > > > very
> > > > > >>> opposed to putting a ticking time bomb in Cassandra proper.
> > > > > >>>
> > > > > >>> Jon
> > > > > >>>
> > > > > >>>
> > > > > >>> On Thu, Feb 28, 2019 at 8:38 AM Dinesh Joshi
> > > > > 
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> I strongly echo Josh’s sentiment. Imagine losing audit entries
> > > > > because C*
> > > > > >>>> is overloaded? It’s fine if you don’t care about losing audit
> > > > entries.
> > > > > >>>>
> > > > > >>>> Dinesh
> > > > > >>>>
> > > > > >>>>> On Fe

Re: CASSANDRA-14482

[Jonathan Haddad]

I plan on doing some more thorough testing, it’s just a matter of finding
time.


On Fri, Mar 1, 2019 at 7:17 PM Dinesh Joshi 
wrote:

> Thanks for testing this out Jon. Apart from the compression ratio could
> you also test the insertion rate and GC? 14482 allows you to use full range
> of Zstd compression levels.
>
> Dinesh
>
> > On Mar 1, 2019, at 6:41 PM, Jonathan Haddad  wrote:
> >
> > Hey all,
> >
> > I finally got around to doing some testing.  Nothing too crazy, I had it
> > run on my laptop while I did other things around the house.
> >
> > Test 1: Inserting Random Data in a K/V table, 10 million inserts
> >
> > LZ4 compression rate: 0.909857609644112
> > ZStd: 0.6136099401596449
> >
> > Test 2: Inserting fairly compressable text data into K/V table: 20
> million
> > inserts
> > LZ4: 0.6950432247957942
> > ZStd: 0.4797311577845362
> >
> > Definitely worth it at first glance.  More testing to come.
> >
> > Jon
> >
> >
> >
> >
> > On Sun, Feb 17, 2019 at 8:46 PM dinesh.jo...@yahoo.com.INVALID
> >  wrote:
> >
> >> Thanks all for your input. The consensus is to go forward with this
> ticket.
> >> Dinesh
> >>
> >>On Friday, February 15, 2019, 12:54:20 PM PST, Sumanth Pasupuleti <
> >> spasupul...@netflix.com.INVALID> wrote:
> >>
> >> +1
> >>
> >>> On Fri, Feb 15, 2019 at 12:14 PM Dikang Gu  wrote:
> >>>
> >>> +1
> >>>
> >>> On Fri, Feb 15, 2019 at 10:27 AM Vinay Chella  >
> >>> wrote:
> >>>
> >>>> We have been using Zstd compressor across different products/services
> >>> here
> >>>> and have seen significant improvements, getting this in 4.0 would be a
> >>> big
> >>>> win.
> >>>>
> >>>> +1
> >>>>
> >>>> Thanks,
> >>>> Vinay Chella
> >>>>
> >>>>
> >>>>> On Fri, Feb 15, 2019 at 10:19 AM Jeff Jirsa 
> wrote:
> >>>>>
> >>>>> +1
> >>>>>
> >>>>> --
> >>>>> Jeff Jirsa
> >>>>>
> >>>>>
> >>>>>> On Feb 15, 2019, at 9:35 AM, Jonathan Ellis 
> >>> wrote:
> >>>>>>
> >>>>>> IMO "add a new compression class that has demonstrable benefits to
> >>>> Sushma
> >>>>>> and Joseph" is sufficiently noninvasive that we should allow it
> >> into
> >>>> 4.0.
> >>>>>>
> >>>>>> On Fri, Feb 15, 2019 at 10:48 AM Dinesh Joshi
> >>>>>>  wrote:
> >>>>>>
> >>>>>>> Hey folks,
> >>>>>>>
> >>>>>>> Just wanted to get a pulse on whether we can proceed with ZStd
> >>>> support.
> >>>>>>> The consensus on the ticket was that it’s a very valuable addition
> >>>>> without
> >>>>>>> any risk of destabilizing 4.0. It’s ready to go if there aren’t
> >> any
> >>>>>>> objections.
> >>>>>>>
> >>>>>>> Dinesh
> >>>>>>>
> >>>>>>>
> >>> -
> >>>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>>>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Jonathan Ellis
> >>>>>> co-founder, http://www.datastax.com
> >>>>>> @spyced
> >>>>>
> >>>>> -
> >>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Dikang
> >>>
> >
> >
> >
> > --
> > Jon Haddad
> > http://www.rustyrazorblade.com
> > twitter: rustyrazorblade
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
> --
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: CASSANDRA-14482

[Jonathan Haddad]

Hey all,

I finally got around to doing some testing.  Nothing too crazy, I had it
run on my laptop while I did other things around the house.

Test 1: Inserting Random Data in a K/V table, 10 million inserts

LZ4 compression rate: 0.909857609644112
ZStd: 0.6136099401596449

Test 2: Inserting fairly compressable text data into K/V table: 20 million
inserts
LZ4: 0.6950432247957942
ZStd: 0.4797311577845362

Definitely worth it at first glance.  More testing to come.

Jon




On Sun, Feb 17, 2019 at 8:46 PM dinesh.jo...@yahoo.com.INVALID
 wrote:

> Thanks all for your input. The consensus is to go forward with this ticket.
> Dinesh
>
> On Friday, February 15, 2019, 12:54:20 PM PST, Sumanth Pasupuleti <
> spasupul...@netflix.com.INVALID> wrote:
>
>  +1
>
> On Fri, Feb 15, 2019 at 12:14 PM Dikang Gu  wrote:
>
> > +1
> >
> > On Fri, Feb 15, 2019 at 10:27 AM Vinay Chella 
> > wrote:
> >
> > > We have been using Zstd compressor across different products/services
> > here
> > > and have seen significant improvements, getting this in 4.0 would be a
> > big
> > > win.
> > >
> > > +1
> > >
> > > Thanks,
> > > Vinay Chella
> > >
> > >
> > > On Fri, Feb 15, 2019 at 10:19 AM Jeff Jirsa  wrote:
> > >
> > > > +1
> > > >
> > > > --
> > > > Jeff Jirsa
> > > >
> > > >
> > > > > On Feb 15, 2019, at 9:35 AM, Jonathan Ellis 
> > wrote:
> > > > >
> > > > > IMO "add a new compression class that has demonstrable benefits to
> > > Sushma
> > > > > and Joseph" is sufficiently noninvasive that we should allow it
> into
> > > 4.0.
> > > > >
> > > > > On Fri, Feb 15, 2019 at 10:48 AM Dinesh Joshi
> > > > >  wrote:
> > > > >
> > > > >> Hey folks,
> > > > >>
> > > > >> Just wanted to get a pulse on whether we can proceed with ZStd
> > > support.
> > > > >> The consensus on the ticket was that it’s a very valuable addition
> > > > without
> > > > >> any risk of destabilizing 4.0. It’s ready to go if there aren’t
> any
> > > > >> objections.
> > > > >>
> > > > >> Dinesh
> > > > >>
> > > > >>
> > -
> > > > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > > >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> > > > >>
> > > > >>
> > > > >
> > > > > --
> > > > > Jonathan Ellis
> > > > > co-founder, http://www.datastax.com
> > > > > @spyced
> > > >
> > > > -
> > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > > >
> > > >
> > >
> >
> >
> > --
> > Dikang
> >



-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Audit logging to tables.

[Jonathan Haddad]

Instead of logging to tables, putting a virtual table around the audit /
query logs might be an option. Same with the commit log for cdc

On Fri, Mar 1, 2019 at 5:25 PM Sagar  wrote:

> Thanks all for the pointers. Really insightful.
>
> Subroto I think that’s part of the enterprise version but yeah even I have
> seen it. Again not sure of the performance implications.
>
> Sagar.
>
> On Sat, 2 Mar 2019 at 5:15 AM, Subroto Barua 
> wrote:
>
> > Datastax version has an option to store audit info to dse_audit.audit_log
> > table; I do not know the performance impact since I use the file option
> >
> > Subroto
> >
> > > On Mar 1, 2019, at 9:40 AM, Jeremiah D Jordan <
> jeremiah.jor...@gmail.com>
> > wrote:
> > >
> > > AFAIK the Full Query Logging binary format was already made more
> general
> > in order to support using that format for the audit logging.
> > >
> > > -Jeremiah
> > >
> > >> On Mar 1, 2019, at 11:38 AM, Joshua McKenzie 
> > wrote:
> > >>
> > >> Is there a world in which a general purpose, side-channel file storage
> > >> format for transient things like this (hints, batches, audit logs,
> etc)
> > >> could be useful as a first class citizen in the codebase? i.e. a world
> > in
> > >> which we refactored some of the hints-specific reader/writer code to
> be
> > >> used for things like this if/when they come up?
> > >>
> > >>> On Thu, Feb 28, 2019 at 12:04 PM Jonathan Haddad  > <mailto:j...@jonhaddad.com>> wrote:
> > >>>
> > >>> Agreed with Dinesh and Josh.  I would *never* put the audit log back
> in
> > >>> Cassandra.
> > >>>
> > >>> This is extendable, Sagar, so you're free to do as you want, but I'm
> > very
> > >>> opposed to putting a ticking time bomb in Cassandra proper.
> > >>>
> > >>> Jon
> > >>>
> > >>>
> > >>> On Thu, Feb 28, 2019 at 8:38 AM Dinesh Joshi
> > 
> > >>> wrote:
> > >>>
> > >>>> I strongly echo Josh’s sentiment. Imagine losing audit entries
> > because C*
> > >>>> is overloaded? It’s fine if you don’t care about losing audit
> entries.
> > >>>>
> > >>>> Dinesh
> > >>>>
> > >>>>> On Feb 28, 2019, at 6:41 AM, Joshua McKenzie  >
> > >>>> wrote:
> > >>>>>
> > >>>>> One of the things we've run into historically, on a *lot* of axes,
> is
> > >>>> that
> > >>>>> "just put it in C*" for various functionality looks great from a
> user
> > >>> and
> > >>>>> usability perspective, and proves to be something of a nightmare
> from
> > >>> an
> > >>>>> admin / cluster behavior perspective.
> > >>>>>
> > >>>>> i.e. - cluster suffering so you're writing hints? Write them to C*
> > >>> tables
> > >>>>> and watch the cluster suffer more! :)
> > >>>>> Same thing probably holds true for audit logging - at a time frame
> > when
> > >>>>> things are getting hairy w/a cluster, if you're writing that audit
> > >>>> logging
> > >>>>> into C* proper (and dealing with ser/deser, compaction pressure,
> > >>> flushing
> > >>>>> pressure, etc) from that, there's a compounding effect of pressure
> > and
> > >>>> pain
> > >>>>> on the cluster.
> > >>>>>
> > >>>>> So the TL;DR we as a project kind of philosophically have been
> moving
> > >>>>> towards (I think that's valid to say?) is: use C* for the things
> it's
> > >>>>> absolutely great at, and try to side-channel other recovery
> > operations
> > >>> as
> > >>>>> much as you can (see: file-based hints) to stay out of its way.
> > >>>>>
> > >>>>> Same thing held true w/design of CDC - I debated "materialize in
> > memory
> > >>>> for
> > >>>>> consumer to take over socket", and "keep the data in another C*
> > table",
> > >>>> but
> > >>>>> the ramifications to perf and core I/O operations in C* the moment
> > >>&

Re: Audit logging to tables.

[Jonathan Haddad]

Agreed with Dinesh and Josh.  I would *never* put the audit log back in
Cassandra.

This is extendable, Sagar, so you're free to do as you want, but I'm very
opposed to putting a ticking time bomb in Cassandra proper.

Jon


On Thu, Feb 28, 2019 at 8:38 AM Dinesh Joshi 
wrote:

> I strongly echo Josh’s sentiment. Imagine losing audit entries because C*
> is overloaded? It’s fine if you don’t care about losing audit entries.
>
> Dinesh
>
> > On Feb 28, 2019, at 6:41 AM, Joshua McKenzie 
> wrote:
> >
> > One of the things we've run into historically, on a *lot* of axes, is
> that
> > "just put it in C*" for various functionality looks great from a user and
> > usability perspective, and proves to be something of a nightmare from an
> > admin / cluster behavior perspective.
> >
> > i.e. - cluster suffering so you're writing hints? Write them to C* tables
> > and watch the cluster suffer more! :)
> > Same thing probably holds true for audit logging - at a time frame when
> > things are getting hairy w/a cluster, if you're writing that audit
> logging
> > into C* proper (and dealing with ser/deser, compaction pressure, flushing
> > pressure, etc) from that, there's a compounding effect of pressure and
> pain
> > on the cluster.
> >
> > So the TL;DR we as a project kind of philosophically have been moving
> > towards (I think that's valid to say?) is: use C* for the things it's
> > absolutely great at, and try to side-channel other recovery operations as
> > much as you can (see: file-based hints) to stay out of its way.
> >
> > Same thing held true w/design of CDC - I debated "materialize in memory
> for
> > consumer to take over socket", and "keep the data in another C* table",
> but
> > the ramifications to perf and core I/O operations in C* the moment things
> > start to go badly were significant enough that the route we went was "do
> no
> > harm". For better or for worse, as there's obvious tradeoffs there.
> >
> >> On Thu, Feb 28, 2019 at 7:46 AM Sagar 
> wrote:
> >>
> >> Thanks all for the pointers.
> >>
> >> @Joseph,
> >>
> >> I have gone through the links shared by you. Also, I have been looking
> at
> >> the code base.
> >>
> >> I understand the fact that pushing the logs to ES or Solr is a lot
> easier
> >> to do. Having said that, the only reason I thought having something like
> >> this might help is, if I don't want to add more pieces and still
> provide a
> >> central piece of audit logging within Cassandra itself and still be
> >> queryable.
> >>
> >> In terms of usages, one of them could definitely be CDC related use
> cases.
> >> With data being stored in tables and being queryable, it can become a
> lot
> >> more easier to expose this data to external systems like Kafka Connect,
> >> Debezium which have the ability to push data to Kafka for example. Note
> >> that pushing data to Kafka is just an example, but what I mean is, if we
> >> can have data in tables, then instead of everyone writing custom custom
> >> loggers, they can hook into this table info and take action.
> >>
> >> Regarding the infinite loop question, I have done some analysis, and in
> my
> >> opinion, instead of tweaking the behaviour of Binlog and the way it
> >> functions currently, we can actually spin up another tailer thread to
> the
> >> same Chronicle Queue which can do the needful. This way the config
> options
> >> etc all remain the same(apart from the logger ofcourse).
> >>
> >> Let me know if any of it makes sense :D
> >>
> >> Thanks!
> >> Sagar.
> >>
> >>
> >> On Thu, Feb 28, 2019 at 1:09 AM Dinesh Joshi  >
> >> wrote:
> >>
> >>>
> >>>
>  On Feb 27, 2019, at 10:41 AM, Joseph Lynch 
> >>> wrote:
> 
>  Vinay can confirm, but as far as I am aware we have no current plans
> to
>  implement audit logging to a table directly, but the implementation is
>  fully pluggable (like compaction, compression, etc ...). Check out the
> >>> blog
>  post [1] and documentation [2] Vinay wrote for more details, but the
> >>> short
> >>>
> >>> +1. I am still curious as to why you'd want to store audit log entries
> >>> back in Cassandra? Depending on the scale it can generate a lot of load
> >> and
> >>> I think you'd end up in an infinite loop because as you're inserting
> the
> >>> audit log entry you'll generate a new one and so on unless you black
> list
> >>> audits to that table / keyspace.
> >>>
> >>> Ideally you'd insert this data into ElasticSearch / Solr or some other
> >>> place that can be then used for analytics or search.
> >>>
> >>> Dinesh
> >>> -
> >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>>
> >>>
> >>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad

Re: CASSANDRA-14482

[Jonathan Haddad]

Seems low risk, potentially high reward.

I can run some tests next week to get a rough idea of how compression
ratios differ as well as if there's a difference in performance.

I won't be testing correctness, just looking at the performance profile.

Jon



On Fri, Feb 15, 2019 at 9:56 AM Michael Shuler 
wrote:

> +0.5
>
> I skimmed the jira and github diff and a few things came to mind:
> - There are multiple comments about using an older jar than the latest
> version.
> - I did not see any performance test results to form an opinion on any
> gains/caveats as a user. This was the first thing I looked for.
> - I did not see any conf/cassandra.yaml comment in the diff for the
> valid class_name configuration option to use.
>
> Seems interesting, and there are comments about 4.0 being in a freeze
> and all, but OK on it being non-default.
>
> --
> Michael
>
> On 2/15/19 11:43 AM, Ariel Weisberg wrote:
> > Hi,
> >
> > I am +1 since it's an additional compressor and not the default.
> >
> > Ariel
> >
> > On Fri, Feb 15, 2019, at 11:41 AM, Dinesh Joshi wrote:
> >> Hey folks,
> >>
> >> Just wanted to get a pulse on whether we can proceed with ZStd support.
> >> The consensus on the ticket was that it’s a very valuable addition
> >> without any risk of destabilizing 4.0. It’s ready to go if there aren’t
> >> any objections.
> >>
> >> Dinesh
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: [VOTE] Release Apache Cassandra 3.11.4

[Jonathan Haddad]

+1

On Sun, Feb 3, 2019 at 9:35 AM Nate McCall  wrote:

> On Sat, Feb 2, 2019 at 4:38 PM Michael Shuler 
> wrote:
> >
> > I propose the following artifacts for release as 3.11.4.
> >
> > sha1: fd47391aae13bcf4ee995abcde1b0e180372d193
> > Git:
> >
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/3.11.4-tentative
> > Artifacts:
> >
> https://repository.apache.org/content/repositories/orgapachecassandra-1170/org/apache/cassandra/apache-cassandra/3.11.4/
> > Staging repository:
> >
> https://repository.apache.org/content/repositories/orgapachecassandra-1170/
> >
> > The Debian and RPM packages are available here:
> > http://people.apache.org/~mshuler
> >
> > The vote will be open for 72 hours (longer if needed).
> >
> > [1]: CHANGES.txt:
> >
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/3.11.4-tentative
> > [2]: NEWS.txt:
> >
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/3.11.4-tentative
> >
>
> +1
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: [VOTE] Release Apache Cassandra 3.0.18

[Jonathan Haddad]

+1

On Sun, Feb 3, 2019 at 9:44 AM Nate McCall  wrote:

> On Sat, Feb 2, 2019 at 4:32 PM Michael Shuler 
> wrote:
> >
> > I propose the following artifacts for release as 3.0.18.
> >
> > sha1: edd52cef50a6242609a20d0d84c8eb74c580035e
> > Git:
> >
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/3.0.18-tentative
> > Artifacts:
> >
> https://repository.apache.org/content/repositories/orgapachecassandra-1171/org/apache/cassandra/apache-cassandra/3.0.18/
> > Staging repository:
> >
> https://repository.apache.org/content/repositories/orgapachecassandra-1171/
> >
> > The Debian and RPM packages are available here:
> > http://people.apache.org/~mshuler
> >
> > The vote will be open for 72 hours (longer if needed).
> >
> > [1]: CHANGES.txt:
> >
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/3.0.18-tentative
> > [2]: NEWS.txt:
> >
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/3.0.18-tentative
> >
>
> +1
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
> --
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: SSTable exclusion from read path based on sstable metadata marked by custom compaction strategies

[Jonathan Haddad]

In addition to what Jeff mentioned, there was an optimization in 3.4 that
can significantly reduce the number of sstables accessed when a LIMIT
clause was used.  This can be a pretty big win with TWCS.

http://thelastpickle.com/blog/2017/03/07/The-limit-clause-in-cassandra-might-not-work-as-you-think.html

On Thu, Jan 31, 2019 at 5:50 PM Jeff Jirsa  wrote:

> In my original TWCS talk a few years back, I suggested that people make
> the partitions match the time window to avoid exactly what you’re
> describing. I added that to the talk because my first team that used TWCS
> (the team for which I built TWCS) had a data model not unlike yours, and
> the read-every-sstable thing turns out not to work that well if you have
> lots of windows (or very large partitions). If you do this, you can fan out
> a bunch of async reads for the first few days and ask for more as you need
> to fill the page - this means the reads are more distributed, too, which is
> an extra bonus when you have noisy partitions.
>
> In 3.0 and newer (I think, don’t quote me in the specific version), the
> sstable metadata has the min and max clustering which helps exclude
> sstables from the read path quite well if everything in the table is using
> timestamp clustering columns. I know there was some issue with this and RTs
> recently, so I’m not sure if it’s current state, but worth considering that
> this may be much better on 3.0+
>
>
>
> --
> Jeff Jirsa
>
>
> > On Jan 31, 2019, at 1:56 PM, Carl Mueller 
> > 
> wrote:
> >
> > Situation:
> >
> > We use TWCS for a task history table (partition is user, column key is
> > timeuuid of task, TWCS is used due to tombstone TTLs that rotate out the
> > tasks every say month. )
> >
> > However, if we want to get a "slice" of tasks (say, tasks in the last two
> > days and we are using TWCS sstable blocks of 12 hours).
> >
> > The problem is, this is a frequent user and they have tasks in ALL the
> > sstables that are organized by the TWCS into time-bucketed sstables.
> >
> > So Cassandra has to first read in, say 80 sstables to reconstruct the
> row,
> > THEN it can exclude/slice on the column key.
> >
> > Question:
> >
> > Or am I wrong that the read path needs to grab all relevant sstables
> before
> > applying column key slicing and this is possible? Admittedly we are in
> 2.1
> > for this table (we in the process of upgrading now that we have an
> > automated upgrading program that seems to work pretty well)
> >
> > If my assumption is correct, then the compaction strategy knows as it
> > writes the sstables what it is bucketing them as (and could encode in
> > sstable metadata?). If my assumption about slicing is that the whole row
> > needs reconstruction, if we had a perfect infinite monkey coding team
> that
> > could generate whatever we wanted within some feasibility, could we
> provide
> > special hooks to do sstable exclusion based on metadata if we know that
> > that the metadata will indicate exclusion/inclusion of columns based on
> > metadata?
> >
> > Goal:
> >
> > The overall goal would be to support exclusion of sstables from a read
> > path, in case we had compaction strategies hand-tailored for other
> queries.
> > Essentially we would be doing a first-pass bucketsort exclusion with the
> > sstable metadata marking the buckets. This might aid support of superwide
> > rows and paging through column keys if we allowed the table creator to
> > specify bucketing as flushing occurs. In general it appears query
> > performance quickly degrades based on # sstables required for a lookup.
> >
> > I still don't know the code nearly well enough to do patches, it would
> seem
> > based on my looking at custom compaction strategies and the basic read
> path
> > that this would be a useful extension for advanced users.
> >
> > The fallback would be a set of tables to serve as buckets and we span the
> > buckets with queries when one bucket runs out. The tables rotate.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Warn about SASI usage and allow to disable them

[Jonathan Haddad]

I'm +1 on the warning for two reasons.

> A cqlsh warning only applies to those that create the sasi via cqlsh.

1. When people are creating their schemas in development, this is usually
the first step.  You use the REPL to figure out what you need, then you
copy your schema somewhere else.  The warning here should prevent a lot of
folks from making a serious mistake.

2. It's consistent with how we warn when people try to use materialized
views.




On Wed, Jan 16, 2019 at 2:15 PM Mick Semb Wever  wrote:

> Regarding the warning, we might add it at least in 3.11, since for that
> version the property to enable SASI is going to be present but not disabled
> by default. WDYT?
>
>
> I'm  -0 on this.
>
> A single line warning in the logs on the sasi creation won't be noticed by
> many users.
> A cqlsh warning only applies to those that create the sasi via cqlsh.
> And we're not talking about patching client drivers to generate a warning
> there.
>
> So I'd be happy with a yaml comment on the config flag explaining that
> it's a beta feature and that users should check open tickets and understand
> current limitations on sasi before using them.
>
> regards,
> Mick
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: [DISCUSS] releasing next 3.0 & 3.11

[Jonathan Haddad]

Ping on this.

On Mon, Jan 7, 2019 at 5:58 PM Michael Shuler 
wrote:

> No problem, thanks for asking :)
>
> Michael
>
> On 1/7/19 6:20 PM, Jonathan Haddad wrote:
> > It's been 5 months and 30+ bug fixes to each branch.
> >
> > Here's the two changelogs:
> >
> > https://github.com/apache/cassandra/blob/cassandra-3.0/CHANGES.txt
> > https://github.com/apache/cassandra/blob/cassandra-3.11/CHANGES.txt
> >
> > How's everyone feel about getting a release out this week / early next
> > week?  Some of these bugs are show stoppers, causing OOMs and data loss.
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Warn about SASI usage and allow to disable them

[Jonathan Haddad]

I'm very much in favor of a warning, and I lean towards disabling them (and
MVs, while we're at it) by default as well.

I've seen both features be the death of clusters, and are a major risk for
teams that are brand new to Cassandra.



On Mon, Jan 14, 2019 at 11:19 AM Andrés de la Peña 
wrote:

> Hello all,
>
> It is my understanding that SASI is still to be considered an
> experimental/beta feature, and they apparently are not being very actively
> developed. Some higlighted problems in SASI are:
>
> - OOMs during flush, as it is described in CASSANDRA-12662
> - General secondary index consistency problems described in CASSANDRA-8272.
> There is a pending-review patch addressing the problem for regular 2i.
> However, the proposed solution is based on indexing tombstones. SASI
> doesn't index tombstones, so it wouldn't be enterely trivial to extend the
> approach to SASI.
> - Probably insufficient testing. As far as I know, we don't have a single
> dtest for SASI nor tests dealing with large SSTables.
>
> Similarly to what CASSANDRA-13959 did with materialized views,
> CASSANDRA-14866 aims to throw a native protocol warning about SASI
> experimental state, and to add a config property to disable them. Perhaps
> this property could be disabled by default in trunk. This should raise
> awareness about SASI maturity until we let them in a more stable state.
>
> The purpose for this thread is discussing whether we want to add this
> warning, the config property and, more controversially, if we want to set
> SASI as disabled by default in trunk.
>
> WDYT?
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

[DISCUSS] releasing next 3.0 & 3.11

[Jonathan Haddad]

It's been 5 months and 30+ bug fixes to each branch.

Here's the two changelogs:

https://github.com/apache/cassandra/blob/cassandra-3.0/CHANGES.txt
https://github.com/apache/cassandra/blob/cassandra-3.11/CHANGES.txt

How's everyone feel about getting a release out this week / early next
week?  Some of these bugs are show stoppers, causing OOMs and data loss.
-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Who should be in our distribution KEYS file?

[Jonathan Haddad]

> I don't understand how adding keys changes release frequency. Did
someone request a release to be made or are we on some assumed date
interval?

I don't know if it would (especially by itself), I just know that if more
people are able to do releases that's more opportunity to do so.

I think getting more folks involved in the release process is a good idea
for other reasons.  People take vacations, there's job conflicts, there's
life stuff (kids usually take priority), etc.

The last release of 3.11 was almost half a year ago, and there's 30+ bug
fixes in the 3.11 branch.

> Did someone request a release to be made or are we on some assumed date
interval?

I can't recall (and a search didn't find) anyone asking for a 3.11.4
release, but I think part of the point is that requesting a release from a
static release manager is a sign of a flaw in the release process.

On a human, note, it feels a little awkward asking for a release.  I might
be alone on this though.

Jon


On Mon, Jan 7, 2019 at 1:16 PM Michael Shuler 
wrote:

> Mick and I have discussed this previously, but I don't recall if it was
> email or irc. Apologies if I was unable to describe the problem to a
> point of general understanding.
>
> To reiterate the problem, changing gpg signature keys screws our debian
> and redhat package repositories for all users. Tarballs are not
> installed with a client that checks signatures in a known trust
> database. When gpg key signer changes, users need to modify their trust
> on every node, importing new key(s), in order for packages to
> install/upgrade with apt or yum.
>
> I don't understand how adding keys changes release frequency. Did
> someone request a release to be made or are we on some assumed date
> interval?
>
> Michael
>
> On 1/7/19 2:30 PM, Jonathan Haddad wrote:
> > That's a good point.  Looking at the ASF docs I had assumed the release
> > manager was per-project, but on closer inspection it appears to be
> > per-release.  You're right, it does say that it can be any committer.
> >
> > http://www.apache.org/dev/release-publishing.html#release_manager
> >
> > We definitely need more frequent releases, if this is the first step
> > towards that goal, I think it's worth it.
> >
> > Glad you brought this up!
> > Jon
> >
> >
> > On Mon, Jan 7, 2019 at 11:58 AM Mick Semb Wever  wrote:
> >
> >>
> >>
> >>> I don't see any reason to have any keys in there, except from release
> >>> managers who are signing releases.
> >>
> >>
> >> Shouldn't any PMC (or committer) should be able to be a release manager?
> >>
> >> The release process should be reliable and reproducible enough to be
> safe
> >> for rotating release managers every release. I would have thought
> security
> >> concerns were better addressed by a more tested process? And AFAIK no
> other
> >> asf projects are as restrictive on who can be the release manager role
> (but
> >> i've only checked a few projects).
> >>
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>
> >>
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Who should be in our distribution KEYS file?

[Jonathan Haddad]

That's a good point.  Looking at the ASF docs I had assumed the release
manager was per-project, but on closer inspection it appears to be
per-release.  You're right, it does say that it can be any committer.

http://www.apache.org/dev/release-publishing.html#release_manager

We definitely need more frequent releases, if this is the first step
towards that goal, I think it's worth it.

Glad you brought this up!
Jon


On Mon, Jan 7, 2019 at 11:58 AM Mick Semb Wever  wrote:

>
>
> > I don't see any reason to have any keys in there, except from release
> > managers who are signing releases.
>
>
> Shouldn't any PMC (or committer) should be able to be a release manager?
>
> The release process should be reliable and reproducible enough to be safe
> for rotating release managers every release. I would have thought security
> concerns were better addressed by a more tested process? And AFAIK no other
> asf projects are as restrictive on who can be the release manager role (but
> i've only checked a few projects).
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Who should be in our distribution KEYS file?

[Jonathan Haddad]

I agree with Stefan, if someone isn't a release manager there's no reason
to add them, and it just increases the surface area for potential attack or
issue.

On Mon, Jan 7, 2019 at 11:35 AM Stefan Podkowinski  wrote:

> I don't see any reason to have any keys in there, except from release
> managers who are signing releases. Additional keys from other developers
> may even harm security, by creating more opportunities for compromising
> keys.
>
> On 07.01.19 11:29, Mick Semb Wever wrote:
> > And when should it get updated?
> >
> > Currently our KEYS file: that contains the public keys of those that can
> signed released binary artifacts; only contains a few of the PMC. My
> understanding is that we've avoid updating it because it causes headache
> for operators in having to validate the authenticity of a new key that's
> signed a binary when upgrading.
> >
> > If this is accurate, how prevalent is this problem actually on
> operators? Do some operators download the KEYS fresh from apache.org
> every release? Are the keys of our PMCs already in the existing web of
> trust?
> >
> > I'm not knowledgeable on the precedence here for operators, and curious
> to what's the community's stance (and why)… And whether it is the time
> right to add all/more our PMC to the file? And whether we should always add
> new PMC to the file (if they're already in the web of trust?)?
> >
> > cheers,
> > Mick
> >
> > https://www.apache.org/info/verification.html#Validating
> > https://www.apache.org/dyn/closer.cgi#verify
> > https://dist.apache.org/repos/dist/release/cassandra/KEYS
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Git Repo Migration

[Jonathan Haddad]

Great news, we can also remove this line: We cannot merge or close any pull
requests opened for the apache/cassandra repository, so please don't open
them.

On Fri, Jan 4, 2019 at 1:27 PM Michael Shuler 
wrote:

> Patches:
> cassandra - https://github.com/apache/cassandra/pull/297
> cassandra-builds - https://github.com/apache/cassandra-builds/pull/8
> cassandra-dtest
> <https://github.com/apache/cassandra-builds/pull/8cassandra-dtest> - no
> git-wip usage found.
>
> URL update for the DSL seed job repo should be the only change needed
> there, after the above cassandra-builds commit.
> - https://builds.apache.org/job/Cassandra-Job-DSL/
>
> The only other consideration I can think of after migration is checking
> the git.a.o bare and github mirror post-commit hooks work as expected.
> - http://git.apache.org/cassandra.git/
>
> Michael
>
> On 1/4/19 2:00 PM, Jonathan Haddad wrote:
> > Silence can be interpreted either as non-awareness or implicit approval.
> >
> > I interpreted (and meant) +1 as "go for it now".
> >
> > On Fri, Jan 4, 2019 at 8:48 AM Sam Tunnicliffe  wrote:
> >
> >> Yeah, I wasn’t really proposing a vote as like you said, it’s happening
> >> anyway. I was just going to give a few days for people to chime in about
> >> timings, seeing as some may still be afk after holidays etc. I’ll give
> it a
> >> little while to at least maintain the illusion of waiting for consensus,
> >> but filing the JIRA early next week SGTM.
> >>
> >> Thanks,
> >> Sam
> >>
> >>> On 4 Jan 2019, at 16:24, Michael Shuler 
> wrote:
> >>>
> >>> This change will be forced on Feb 7, so a vote isn't really relevant :)
> >>>
> >>> I'm in favor of sooner is better. Things are relatively quiet right
> now,
> >>> post-holidays. Let's do it Monday, for example, and we can work through
> >>> the config changes while most people are fresh from a break.
> >>>
> >>> The longer we wait, the more things people have in flight and they get
> >>> bothered about being interrupted. Just interrupt ASAP, IMO.
> >>>
> >>> Michael
> >>>
> >>> On 1/4/19 4:49 AM, Sam Tunnicliffe wrote:
> >>>> As per the announcement on 7th December 2018[1], ASF infra are
> >>>> planning to shutdown the service behind git-wip-us.apache.org and
> >>>> migrate all existing repos to gitbox.apache.org
> >>>>
> >>>> There are further details in the original mail, but apparently one of
> >>>> the benefits of the migration is that we'll have full write access
> >>>> via Github, including the ability finally to close PRs. This affects
> >>>> the cassandra, cassandra-dtest and cassandra-build repos (but not the
> >>>> new cassandra-sidecar repo).
> >>>>
> >>>> A pre-requisite of the migration is to demonstrate consensus within
> >>>> the community, so to satisfy that formality I'm starting this thread
> >>>> to gather any objections or specific requests regarding the timing of
> >>>> the move.
> >>>>
> >>>> I'll collate responses in a week or so and file the necessary INFRA
> >>>> Jira.
> >>>>
> >>>> Thanks, Sam
> >>>>
> >>>> [1]
> >>>>
> >>
> https://lists.apache.org/thread.html/667772efdabf49a0a23d585539c127f335477e033f1f9b6f5079aced@%3Cdev.cassandra.apache.org%3E
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Git Repo Migration

[Jonathan Haddad]

+1

On Fri, Jan 4, 2019 at 8:13 AM Ariel Weisberg  wrote:

> +1
>
> On Fri, Jan 4, 2019, at 5:49 AM, Sam Tunnicliffe wrote:
> > As per the announcement on 7th December 2018[1], ASF infra are planning
> > to shutdown the service behind git-wip-us.apache.org and migrate all
> > existing repos to gitbox.apache.org
> >
> > There are further details in the original mail, but apparently one of
> > the benefits of the migration is that we'll have full write access via
> > Github, including the ability finally to close PRs. This affects the
> > cassandra, cassandra-dtest and cassandra-build repos (but not the new
> > cassandra-sidecar repo).
> >
> > A pre-requisite of the migration is to demonstrate consensus within the
> > community, so to satisfy that formality I'm starting this thread to
> > gather any objections or specific requests regarding the timing of the
> > move.
> >
> > I'll collate responses in a week or so and file the necessary INFRA Jira.
> >
> > Thanks,
> > Sam
> >
> > [1]
> >
> https://lists.apache.org/thread.html/667772efdabf49a0a23d585539c127f335477e033f1f9b6f5079aced@%3Cdev.cassandra.apache.org%3E
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
> --
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: [VOTE] Change Jira Workflow

[Jonathan Haddad]

+1

On Mon, Dec 17, 2018 at 9:50 AM Blake Eggleston
 wrote:

> +1
>
> > On Dec 17, 2018, at 9:31 AM, jay.zhu...@yahoo.com.INVALID wrote:
> >
> > +1
> >
> >On Monday, December 17, 2018, 9:10:55 AM PST, Jason Brown <
> jasedbr...@gmail.com> wrote:
> >
> > +1.
> >
> > On Mon, Dec 17, 2018 at 7:36 AM Michael Shuler 
> > wrote:
> >
> >> +1
> >>
> >> --
> >> Michael
> >>
> >> On 12/17/18 9:19 AM, Benedict Elliott Smith wrote:
> >>> I propose these changes <
> >>
> https://cwiki.apache.org/confluence/display/CASSANDRA/JIRA+Workflow+Proposals
> >*
> >> to the Jira Workflow for the project.  The vote will be open for 72
> hours**.
> >>>
> >>> I am, of course, +1.
> >>>
> >>> * With the addendum of the mailing list discussion <
> >>
> https://lists.apache.org/thread.html/e4668093169aa4ef52f2bea779333f04a0afde8640c9a79a8c86ee74@%3Cdev.cassandra.apache.org%3E
> >;
> >> in case of any conflict arising from a mistake on my part in the wiki,
> the
> >> consensus reached by polling the mailing list will take precedence.
> >>> ** I won’t be around to close the vote, as I will be on vacation.
> >> Everyone is welcome to ignore the result until I get back in a couple of
> >> weeks, or if anybody is eager feel free to close the vote and take some
> >> steps towards implementation.
> >>>
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: JIRA Workflow Proposals

[Jonathan Haddad]

My personal preference is to remove labels, but I don't see it as a huge
issue if they stay.

1. A
2. prefer to remove (I suppose I'm a -.1?)
3. +1
4. +1
5. No preference
6. +1



On Wed, Dec 5, 2018 at 11:43 AM jay.zhu...@yahoo.com.INVALID
 wrote:

>  1: Component. (A) Multi-select
> 2: Labels: leave alone +1
> 3: No workflow changes for first/second review: +1
> 4: Priorities Including High: -1
> 5: Mandatory Platform and Feature: +1
> 6: Remove Environment field: +1
>
> On Wednesday, December 5, 2018, 2:58:21 AM PST, Benedict Elliott Smith
>  wrote:
>
>  Thanks for the feedback and further questions.  I’m sure there will be
> more, particularly around permissions and roles, so it’s good to get some
> of these other discussions moving.
>
> No doubt we’ll do a second poll once the first one concludes.  Please,
> everyone, keep your first poll answers coming!
>
> > On 5 Dec 2018, at 09:50, Sylvain Lebresne  wrote:
> >
> > Thanks for all those that contributed to the proposal, and specifically
> to
> > Benedict for leading the discussion.
> >
> > On the general proposal, I suspect there is a few details we may have to
> > tweak going forward, but hard to be sure before trying and as I do think
> > it's progress, I'm personally happy to move forward with this. But for
> the
> > sake of discussions, a minor remarks that I don't think have been
> mentioned
> > above:
> > - at least the platform and feature fields will be moving targets (the
> > features hopefully more so than the platform, but new java version gets
> > release regularly for instance). Do we know technically if we can get
> those
> > easily updated without requiring an infra JIRA ticket?
>
> This is actually a really good question.  I had assumed this would be
> treated much like Component, Version, etc
>
> However, I was wrong:
>
> https://community.atlassian.com/t5/Jira-questions/Adding-options-to-a-custom-field-as-project-administrator/qaq-p/113311
> <
> https://community.atlassian.com/t5/Jira-questions/Adding-options-to-a-custom-field-as-project-administrator/qaq-p/113311
> >
>
> There are some possible workarounds, but none of them seem great.  There’s
> a plugin, but not sure if Infra would be OK with this:
> https://marketplace.atlassian.com/apps/1212096/customfield-editor-plugin?hosting=server=overview
> <
> https://marketplace.atlassian.com/apps/1212096/customfield-editor-plugin?hosting=server=overview
> >
>
> This is potentially a real blocker for these two fields.
>
> So, for feature an alternative is to merge Feature/[Feature] into
> Component.  This would implicitly make it non-mandatory, however.  This
> could potentially be fixed with a custom field validator, but this might
> need a plugin.
>
> For Platform, I don’t have a good alternative idea right now.  This is
> something that is best curated, but definitely needs to be easily updated.
>
>
> > - I'd personally probably remove the condition of being "jira
> contributor"
> > to move tickets out of triage. Being a "jira contributor" is a pretty
> > arbitrary in practice. Anyone that ever asked has been added, no question
> > asked, but it can actually be annoying to get added if no-one that knows
> > how to do it is around. So if, as I assume, the goal is to ensure that
> > fields are properly fielded, I don't think it will help much, and so I
> > suspect it would just be an annoyance to new, not-yet-"jira contributor"
> > users that are willing to fill all the required fields seriously (but
> will
> > see their ticket stuck in triage until they either get added, or some
> other
> > random user flip the switch). And why assume people will mis-field
> stuffs?
> > So I'd vote for just letting anyone transition out of triage as long as
> all
> > required field are there. Side-note: fwiw, I'd very much be in favor of
> > removing the "jira contributor" concept for the reasons exposed above: I
> > never felt it was anything else than an hindrance.
>
> I realise we have no real barrier to becoming a contributor, and that many
> of these permissions are going to have limited impact.  But I think a
> really easy to jump gate can still be helpful, and I don’t recall
> contributors overstepping their role.  But this isn’t a critical point, and
> it would be great to hear other’s opinions.
>
> > - Once we have 'complexity' and 'severity', I'm not entirely sure what
> > 'priority' really means in a voluntary-based project? Is it the
> gut-feeling
> > for how useful the ticket is of whomever triage the ticket? If that,
> > outside of not being convinced of the value, I'd argue that 1) it doesn't
> > make sense for bugs and 2) it's sufficiently imprecise that it's imo
> worth
> > keeping it simple, something like low/normal/high ought to be enough. If
> > it's not that, then what is it, cause it's genuinely not immediately
> > obvious to me?
>
> Well, if nothing else Priority is the thing I sort my outstanding work by,
> and for this reason I have a preference for it being present for 

Re: Implicit Casts for Arithmetic Operators

[Jonathan Haddad]

I can’t agree more. We should be able to make changes in a manner that
improves the DB In the long term, rather than live with the technical debt
of arbitrary decisions made by a handful of people.

I also agree that putting a knob in place to let people migrate over is a
reasonable decision.

Jon

On Wed, Nov 21, 2018 at 4:54 PM Benedict Elliott Smith 
wrote:

> The goal is simply to agree on a set of well-defined principles for how we
> should behave.  If we don’t like the implications that arise, we’ll have
> another vote?  A democracy cannot bind itself, so I never understood this
> fear of a decision.
>
> A database also has a thousand toggles.  If we absolutely need to, we can
> introduce one more.
>
> We should be doing this upfront a great deal more often.  Doing it
> retrospectively sucks, but in my opinion it's a bad reason to bind
> ourselves to whatever made it in.
>
> Do we anywhere define the principles of our current behaviour?  I couldn’t
> find it.
>
>
> > On 21 Nov 2018, at 21:08, Sylvain Lebresne  wrote:
> >
> > On Tue, Nov 20, 2018 at 5:02 PM Benedict Elliott Smith <
> bened...@apache.org>
> > wrote:
> >
> >> FWIW, my meaning of arithmetic in this context extends to any features
> we
> >> have already released (such as aggregates, and perhaps other built-in
> >> functions) that operate on the same domain.  We should be consistent,
> after
> >> all.
> >>
> >> Whether or not we need to revisit any existing functionality we can
> figure
> >> out after the fact, once we have agreed what our behaviour should be.
> >>
> >
> > I'm not sure I correctly understand the process suggested, but I don't
> > particularly like/agree with what I understand. What I understand is a
> > suggestion for voting on agreeing to be ANSI SQL 92 compliant, with no
> real
> > evaluation of what that entails (at least I haven't seen one), and that
> > this vote, if passed, would imply we'd then make any backward
> incompatible
> > change necessary to achieve compliance ("my meaning of arithmetic in this
> > context extends to any features we have already released" and "Whether or
> > not we need to revisit any existing functionality we can figure out after
> > the fact, once we have agreed what our behaviour should be").
> >
> > This might make sense of a new product, but at our stage that seems
> > backward to me. I think we owe our users to first make the effort of
> > identifying what "inconsistencies" our existing arithmetic has[1] and
> > _then_ consider what options we have to fix those, with their pros and
> cons
> > (including how bad they break backward compatibility). And if _then_
> > getting ANSI SQL 92 compliant proves to not be disruptive (or at least
> > acceptably so), then sure, that's great.
> >
> > [1]: one possibly efficient way to do that could actually be to compare
> our
> > arithmetic to ANSI SQL 92. Not that all differences found would imply
> > inconsistencies/wrongness of our arithmetic, but still, it should be
> > helpful. And I guess my whole point is that we should that analysis
> first,
> > and then maybe decide that being ANSI SQL 92 is a reasonable option, not
> > decide first and live with the consequences no matter what they are.
> >
> > --
> > Sylvain
> >
> >
> >> I will make this more explicit for the vote, but just to clarify the
> >> intention so that we are all discussing the same thing.
> >>
> >>
> >>> On 20 Nov 2018, at 14:18, Ariel Weisberg  wrote:
> >>>
> >>> Hi,
> >>>
> >>> +1
> >>>
> >>> This is a public API so we will be much better off if we get it right
> >> the first time.
> >>>
> >>> Ariel
> >>>
> >>>> On Nov 16, 2018, at 10:36 AM, Jonathan Haddad 
> >> wrote:
> >>>>
> >>>> Sounds good to me.
> >>>>
> >>>> On Fri, Nov 16, 2018 at 5:09 AM Benedict Elliott Smith <
> >> bened...@apache.org>
> >>>> wrote:
> >>>>
> >>>>> So, this thread somewhat petered out.
> >>>>>
> >>>>> There are still a number of unresolved issues, but to make progress I
> >>>>> wonder if it would first be helpful to have a vote on ensuring we are
> >> ANSI
> >>>>> SQL 92 compliant for our arithmetic?  This seems like a sensible
> >> baseline,
> >>>>> since we will hopefully minimise surprise to op

Re: Request to review feature-freeze proposed tickets

[Jonathan Haddad]

If nobody has any objections to CASSANDRA-14303 (default RF) being merged
in I can take care of it.  It's a significant usability improvement, looks
well tested and will prevent a number of headaches.

I'll try to get to it tomorrow.

Thanks for bringing these up, Vinay.

Jon

On Tue, Nov 20, 2018 at 5:59 PM kurt greaves  wrote:

> Thanks Vinay. While I suspect this will be subject to heated debate, I'm
> also for this. The time to review for this project is incredibly
> demotivating, and it stems from a lack of contributors that are interested
> in the general health of the project. I think this can be quite easily
> remedied by making more committers/PMC, however there is a catch-22 that to
> achieve this our existing set of committers needs to be dedicated to
> reviewing contributions from non-committers.
>
> If we can get dedicated reviewers for the listed tickets I'll take on some
> of the work to get the tickets up to scratch.
>
> On Wed, 21 Nov 2018 at 02:12, Ariel Weisberg  wrote:
>
> > Hi,
> >
> > I would like to get as many of these as is feasible in. Before the
> feature
> > freeze started 1 out of 17 JIRAs that were patch available were reviewed
> > and committed.
> >
> > If you didn’t have access reviewers and committers, as the one out of the
> > 17 did, it has been essentially impossible to get your problems with
> > Cassandra fixed in 4.0.
> >
> > This is basically the same as saying that despite the fact Cassandra is
> > open source it does you no good because it will be years before the
> issues
> > impacting you get fixed even if you contribute the fixes yourself.
> >
> > Pulling up the ladder after getting “your own” fixes in is a sure fire
> way
> > to fracture the community into a collection of private forks containing
> the
> > fixes people can’t live without, and pushing people to look at
> alternatives.
> >
> > Private forks are a serious threat to the project. The people on them are
> > at risk of getting left behind and Cassandra stagnates for them and
> becomes
> > uncompetitive. Those with the resources to maintain a seriously diverged
> > fork are also the ones better positioned to be active contributors.
> >
> > Regards,
> > Ariel
> >
> > > On Nov 18, 2018, at 9:18 PM, Vinay Chella 
> > wrote:
> > >
> > > Hi,
> > >
> > > We still have 15 Patch Available/ open tickets which were requested for
> > > reviews before the Sep 1, 2018 freeze. I am starting this email thread
> to
> > > resurface and request a review of community tickets as most of these
> > > tickets address vital correctness, performance, and usability bugs that
> > > help avoid critical production issues. I tried to provide context on
> why
> > we
> > > feel these tickets are important to get into 4.0. If you would like to
> > > discuss the technical details of a particular ticket, let's try to do
> > that
> > > in JIRA.
> > >
> > > CASSANDRA-14525: Cluster enters an inconsistent state after bootstrap
> > > failures. (Correctness bug, Production impact, Ready to Commit)
> > >
> > > CASSANDRA-14459: DES sends requests to the wrong nodes routinely. (SLA
> > > breaking latencies, Production impact, Review in progress)
> > >
> > > CASSANDRA-14303 and CASSANDRA-14557: Currently production 3.0+ clusters
> > > cannot be rebuilt after node failure due to 3.0’s introduction of the
> > > system_auth keyspace with rf of 1. These tickets both fix the
> regression
> > > introduced in 3.0 by letting operators configure rf=3 and prevent
> future
> > > outages (Usability bug, Production impact, Patch Available).
> > >
> > > CASSANDRA-14096: Cassandra 3.11.1 Repair Causes Out of Memory. We
> believe
> > > this may also impact 3.0 (Title says it all, Production impact, Patch
> > > Available)
> > >
> > > CASSANDRA-10023: It is impossible to accurately determine local
> > read/write
> > > calls on C*. This patch allows users to detect when they are choosing
> > > incorrect coordinators. (Usability bug (troubleshoot), Review in
> > progress)
> > >
> > > CASSANDRA-10789: There is no way to safely stop bad clients bringing
> down
> > > C* nodes. This patch would give operators a very important tool to use
> > > during production incidents to mitigate impact. (Usability bug,
> > Production
> > > Impact (recovery), Patch Available)
> > >
> > > CASSANDRA-13010: No visibility into which disk is being compacted to.
> > > (Usability bug, Production Impact (troubleshoot), Review in progress)
> > >
> > > CASSANDRA-12783 - Break up large MV mutations to prevent OOMs (Title
> says
> > > it all, Production Impact, Patch InProgress/ Awaiting Feedback)
> > >
> > > CASSANDRA-14319 - nodetool rebuild from DC lets you pass invalid
> > > datacenters (Usability bug, Production impact, Patch available)
> > >
> > > CASSANDRA-13841 - Smarter nodetool rebuild. Kind of a bug but would be
> > nice
> > > to get it in 4.0. (Production Impact (recovery), Patch Available)
> > >
> > > CASSANDRA-9452: Cleanup of old configuration, confusing to new C*
> > > 

Re: Cassandra 3.11.x and Java 11

[Jonathan Haddad]

Correct.

On Mon, Nov 19, 2018 at 1:02 PM Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

> Hello,
>
> am I right that Java 11 support will only be in Cassandra 4.0 and not in
> future releases of the 3.11.x series? Just want to be sure.
>
> Thanks,
> Thomas
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freist?dterstra?e 313
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Implicit Casts for Arithmetic Operators

[Jonathan Haddad]

Sounds good to me.

On Fri, Nov 16, 2018 at 5:09 AM Benedict Elliott Smith 
wrote:

> So, this thread somewhat petered out.
>
> There are still a number of unresolved issues, but to make progress I
> wonder if it would first be helpful to have a vote on ensuring we are ANSI
> SQL 92 compliant for our arithmetic?  This seems like a sensible baseline,
> since we will hopefully minimise surprise to operators this way.
>
> If people largely agree, I will call a vote, and we can pick up a couple
> of more focused discussions afterwards on how we interpret the leeway it
> gives.
>
>
> > On 12 Oct 2018, at 18:10, Ariel Weisberg  wrote:
> >
> > Hi,
> >
> > From reading the spec. Precision is always implementation defined. The
> spec specifies scale in several cases, but never precision for any type or
> operation (addition/subtraction, multiplication, division).
> >
> > So we don't implement anything remotely approaching precision and scale
> in CQL when it comes to numbers I think? So we aren't going to follow the
> spec for scale. We are already pretty far down that road so I would leave
> it alone.
> >
> > I don't think the spec is asking for the most approximate type. It's
> just saying the result is approximate, and the precision is implementation
> defined. We could return either float or double. I think if one of the
> operands is a double we should return a double because clearly the schema
> thought a double was required to represent that number. I would also be in
> favor of returning a double all the time so that people can expect a
> consistent type from expressions involving approximate numbers.
> >
> > I am a big fan of widening for arithmetic expressions in a database to
> avoid having to error on overflow. You can go to the trouble of only
> widening the minimum amount, but I think it's simpler if we always widen to
> bigint and double. This would be something the spec allows.
> >
> > Definitely if we can make overflow not occur we should and the spec
> allows that. We should also not return different types for the same operand
> types just to work around overflow if we detect we need more precision.
> >
> > Ariel
> > On Fri, Oct 12, 2018, at 12:45 PM, Benedict Elliott Smith wrote:
> >> If it’s in the SQL spec, I’m fairly convinced.  Thanks for digging this
> >> out (and Mike for getting some empirical examples).
> >>
> >> We still have to decide on the approximate data type to return; right
> >> now, we have float+bigint=double, but float+int=float.  I think this is
> >> fairly inconsistent, and either the approximate type should always win,
> >> or we should always upgrade to double for mixed operands.
> >>
> >> The quoted spec also suggests that decimal+float=float, and decimal
> >> +double=double, whereas we currently have decimal+float=decimal, and
> >> decimal+double=decimal
> >>
> >> If we’re going to go with an approximate operand implying an
> approximate
> >> result, I think we should do it consistently (and consistent with the
> >> SQL92 spec), and have the type of the approximate operand always be the
> >> return type.
> >>
> >> This would still leave a decision for float+double, though.  The most
> >> consistent behaviour with that stated above would be to always take the
> >> most approximate type to return (i.e. float), but this would seem to me
> >> to be fairly unexpected for the user.
> >>
> >>
> >>> On 12 Oct 2018, at 17:23, Ariel Weisberg  wrote:
> >>>
> >>> Hi,
> >>>
> >>> I agree with what's been said about expectations regarding expressions
> involving floating point numbers. I think that if one of the inputs is
> approximate then the result should be approximate.
> >>>
> >>> One thing we could look at for inspiration is the SQL spec. Not to
> follow dogmatically necessarily.
> >>>
> >>> From the SQL 92 spec regarding assignment
> http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt section 4.6:
> >>> "
> >>>Values of the data types NUMERIC, DECIMAL, INTEGER, SMALLINT,
> >>>FLOAT, REAL, and DOUBLE PRECISION are numbers and are all
> mutually
> >>>comparable and mutually assignable. If an assignment would
> result
> >>>in a loss of the most significant digits, an exception condition
> >>>is raised. If least significant digits are lost, implementation-
> >>>defined rounding or truncating occurs with no exception
> condition
> >>>being raised. The rules for arithmetic are generally governed by
> >>>Subclause 6.12, "".
> >>> "
> >>>
> >>> Section 6.12 numeric value expressions:
> >>> "
> >>>1) If the data type of both operands of a dyadic arithmetic
> opera-
> >>>   tor is exact numeric, then the data type of the result is
> exact
> >>>   numeric, with precision and scale determined as follows:
> >>> ...
> >>>2) If the data type of either operand of a dyadic arithmetic op-
> >>>   erator is approximate numeric, then the data type of the re-
> >>>   sult is 

Re: CASSANDRA-13241 lower default chunk_length_in_kb

[Jonathan Haddad]

Looks straightforward, I can review today.

On Mon, Oct 29, 2018 at 12:25 PM Ariel Weisberg  wrote:

> Hi,
>
> Seeing too many -'s for changing the representation and essentially no +1s
> so I submitted a patch for just changing the default. I could use a
> reviewer for https://issues.apache.org/jira/browse/CASSANDRA-13241
>
> I created https://issues.apache.org/jira/browse/CASSANDRA-14857  "Use a
> more space efficient representation for compressed chunk offsets" for post
> 4.0.
>
> Regards,
> Ariel
>
> On Tue, Oct 23, 2018, at 11:46 AM, Ariel Weisberg wrote:
> > Hi,
> >
> > To summarize who we have heard from so far
> >
> > WRT to changing just the default:
> >
> > +1:
> > Jon Haddadd
> > Ben Bromhead
> > Alain Rodriguez
> > Sankalp Kohli (not explicit)
> >
> > -0:
> > Sylvaine Lebresne
> > Jeff Jirsa
> >
> > Not sure:
> > Kurt Greaves
> > Joshua Mckenzie
> > Benedict Elliot Smith
> >
> > WRT to change the representation:
> >
> > +1:
> > There are only conditional +1s at this point
> >
> > -0:
> > Sylvaine Lebresne
> >
> > -.5:
> > Jeff Jirsa
> >
> > This
> > (
> https://github.com/aweisberg/cassandra/commit/a9ae85daa3ede092b9a1cf84879fb1a9f25b9dce)
>
> > is a rough cut of the change for the representation. It needs better
> > naming, unit tests, javadoc etc. but it does implement the change.
> >
> > Ariel
> > On Fri, Oct 19, 2018, at 3:42 PM, Jonathan Haddad wrote:
> > > Sorry, to be clear - I'm +1 on changing the configuration default, but
> I
> > > think changing the compression in memory representations warrants
> further
> > > discussion and investigation before making a case for or against it
> yet.
> > > An optimization that reduces in memory cost by over 50% sounds pretty
> good
> > > and we never were really explicit that those sort of optimizations
> would be
> > > excluded after our feature freeze.  I don't think they should
> necessarily
> > > be excluded at this time, but it depends on the size and risk of the
> patch.
> > >
> > > On Sat, Oct 20, 2018 at 8:38 AM Jonathan Haddad 
> wrote:
> > >
> > > > I think we should try to do the right thing for the most people that
> we
> > > > can.  The number of folks impacted by 64KB is huge.  I've worked on
> a lot
> > > > of clusters created by a lot of different teams, going from brand
> new to
> > > > pretty damn knowledgeable.  I can't think of a single time over the
> last 2
> > > > years that I've seen a cluster use non-default settings for
> compression.
> > > > With only a handful of exceptions, I've lowered the chunk size
> considerably
> > > > (usually to 4 or 8K) and the impact has always been very noticeable,
> > > > frequently resulting in hardware reduction and cost savings.  Of all
> the
> > > > poorly chosen defaults we have, this is one of the biggest offenders
> that I
> > > > see.  There's a good reason ScyllaDB  claims they're so much faster
> than
> > > > Cassandra - we ship a DB that performs poorly for 90+% of teams
> because we
> > > > ship for a specific use case, not a general one (time series on
> memory
> > > > constrained boxes being the specific use case)
> > > >
> > > > This doesn't impact existing tables, just new ones.  More and more
> teams
> > > > are using Cassandra as a general purpose database, we should
> acknowledge
> > > > that adjusting our defaults accordingly.  Yes, we use a little bit
> more
> > > > memory on new tables if we just change this setting, and what we get
> out of
> > > > it is a massive performance win.
> > > >
> > > > I'm +1 on the change as well.
> > > >
> > > >
> > > >
> > > > On Sat, Oct 20, 2018 at 4:21 AM Sankalp Kohli <
> kohlisank...@gmail.com>
> > > > wrote:
> > > >
> > > >> (We should definitely harden the definition for freeze in a separate
> > > >> thread)
> > > >>
> > > >> My thinking is that this is the best time to do this change as we
> have
> > > >> not even cut alpha or beta. All the people involved in the test will
> > > >> definitely be testing it again when we have these releases.
> > > >>
> > > >> > On Oct 19, 2018, at 8:00 AM, Michael Shuler <
> mich...@pbandjelly.org>

Re: CASSANDRA-13241 lower default chunk_length_in_kb

[Jonathan Haddad]

Sorry, to be clear - I'm +1 on changing the configuration default, but I
think changing the compression in memory representations warrants further
discussion and investigation before making a case for or against it yet.
An optimization that reduces in memory cost by over 50% sounds pretty good
and we never were really explicit that those sort of optimizations would be
excluded after our feature freeze.  I don't think they should necessarily
be excluded at this time, but it depends on the size and risk of the patch.

On Sat, Oct 20, 2018 at 8:38 AM Jonathan Haddad  wrote:

> I think we should try to do the right thing for the most people that we
> can.  The number of folks impacted by 64KB is huge.  I've worked on a lot
> of clusters created by a lot of different teams, going from brand new to
> pretty damn knowledgeable.  I can't think of a single time over the last 2
> years that I've seen a cluster use non-default settings for compression.
> With only a handful of exceptions, I've lowered the chunk size considerably
> (usually to 4 or 8K) and the impact has always been very noticeable,
> frequently resulting in hardware reduction and cost savings.  Of all the
> poorly chosen defaults we have, this is one of the biggest offenders that I
> see.  There's a good reason ScyllaDB  claims they're so much faster than
> Cassandra - we ship a DB that performs poorly for 90+% of teams because we
> ship for a specific use case, not a general one (time series on memory
> constrained boxes being the specific use case)
>
> This doesn't impact existing tables, just new ones.  More and more teams
> are using Cassandra as a general purpose database, we should acknowledge
> that adjusting our defaults accordingly.  Yes, we use a little bit more
> memory on new tables if we just change this setting, and what we get out of
> it is a massive performance win.
>
> I'm +1 on the change as well.
>
>
>
> On Sat, Oct 20, 2018 at 4:21 AM Sankalp Kohli 
> wrote:
>
>> (We should definitely harden the definition for freeze in a separate
>> thread)
>>
>> My thinking is that this is the best time to do this change as we have
>> not even cut alpha or beta. All the people involved in the test will
>> definitely be testing it again when we have these releases.
>>
>> > On Oct 19, 2018, at 8:00 AM, Michael Shuler 
>> wrote:
>> >
>> >> On 10/19/18 9:16 AM, Joshua McKenzie wrote:
>> >>
>> >> At the risk of hijacking this thread, when are we going to transition
>> from
>> >> "no new features, change whatever else you want including refactoring
>> and
>> >> changing years-old defaults" to "ok, we think we have something that's
>> >> stable, time to start testing"?
>> >
>> > Creating a cassandra-4.0 branch would allow trunk to, for instance, get
>> > a default config value change commit and get more testing. We might
>> > forget again, from what I understand of Benedict's last comment :)
>> >
>> > --
>> > Michael
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> > For additional commands, e-mail: dev-h...@cassandra.apache.org
>> >
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>
>>
>
> --
> Jon Haddad
> http://www.rustyrazorblade.com
> twitter: rustyrazorblade
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: CASSANDRA-13241 lower default chunk_length_in_kb

[Jonathan Haddad]

I think we should try to do the right thing for the most people that we
can.  The number of folks impacted by 64KB is huge.  I've worked on a lot
of clusters created by a lot of different teams, going from brand new to
pretty damn knowledgeable.  I can't think of a single time over the last 2
years that I've seen a cluster use non-default settings for compression.
With only a handful of exceptions, I've lowered the chunk size considerably
(usually to 4 or 8K) and the impact has always been very noticeable,
frequently resulting in hardware reduction and cost savings.  Of all the
poorly chosen defaults we have, this is one of the biggest offenders that I
see.  There's a good reason ScyllaDB  claims they're so much faster than
Cassandra - we ship a DB that performs poorly for 90+% of teams because we
ship for a specific use case, not a general one (time series on memory
constrained boxes being the specific use case)

This doesn't impact existing tables, just new ones.  More and more teams
are using Cassandra as a general purpose database, we should acknowledge
that adjusting our defaults accordingly.  Yes, we use a little bit more
memory on new tables if we just change this setting, and what we get out of
it is a massive performance win.

I'm +1 on the change as well.



On Sat, Oct 20, 2018 at 4:21 AM Sankalp Kohli 
wrote:

> (We should definitely harden the definition for freeze in a separate
> thread)
>
> My thinking is that this is the best time to do this change as we have not
> even cut alpha or beta. All the people involved in the test will definitely
> be testing it again when we have these releases.
>
> > On Oct 19, 2018, at 8:00 AM, Michael Shuler 
> wrote:
> >
> >> On 10/19/18 9:16 AM, Joshua McKenzie wrote:
> >>
> >> At the risk of hijacking this thread, when are we going to transition
> from
> >> "no new features, change whatever else you want including refactoring
> and
> >> changing years-old defaults" to "ok, we think we have something that's
> >> stable, time to start testing"?
> >
> > Creating a cassandra-4.0 branch would allow trunk to, for instance, get
> > a default config value change commit and get more testing. We might
> > forget again, from what I understand of Benedict's last comment :)
> >
> > --
> > Michael
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

cluster launching tool for dev work

[Jonathan Haddad]

Recently I reached an inflection point where my annoyance of launching
clusters finally overcame my laziness.  I wanted something similar to CCM,
so I wrote it.

The tool was designed for our usage at TLP, which usually means quickly
firing up clusters for running tests.  It started out as some scripts,
added a little docker, and eventually reached a point where it's generic
enough to OSS.

I don't expect (or want, for that matter) this to be a tool for normal
Cassandra users to launch clusters.  It's designed for taking a C*
codebase, turn it into a deb package and push it onto a handful of
servers.  It configures some stuff for you automatically, like seeds.  It
has bugs, you'll probably need to read the code to figure it out while I
work on improving the docs.  It definitely needs more features & polish as
well, but I figured hey, might as well share what I have maybe someone will
find it useful.

Code: https://github.com/thelastpickle/tlp-cluster
Docs: http://thelastpickle.com/tlp-cluster/

It's Apache licensed, so feel free to do what you want with it.  I'll try
to update the docs to make it a little friendlier to use.
-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Implicit Casts for Arithmetic Operators

[Jonathan Haddad]

Thanks for bringing this up, it definitely needs to be discussed.

Last surprise is difficult here, since all major databases have their own
way of doing things and people will just assume that their way is the right
way.  On that note, some people will be surprised no matter what we do.

I'd rather avoid the pitfalls of returning incorrect results, so either
option 2 or 3 sound reasonable, but leaning towards the Postgres approach
of always returning a decimal for those cases.

Jon



On Tue, Oct 2, 2018 at 10:54 AM Benedict Elliott Smith 
wrote:

> I agree, in broad strokes at least.  Interested to hear others’ positions.
>
>
>
> > On 2 Oct 2018, at 16:44, Ariel Weisberg  wrote:
> >
> > Hi,
> >
> > I think overflow and the role of widening conversions are pretty linked
> so I'll continue to inject that into this discussion. Also overflow is much
> worse since most applications won't be impacted by a loss of precision when
> an expression involves an int and float, but will care quite a bit if they
> get some nonsense wrapped number in an integer only expression.
> >
> > For VoltDB in practice we didn't run into issues with applications not
> making progress due to exceptions with real data due to the widening
> conversions. The range of double and long are pretty big and that hides
> wrap around/infinity.
> >
> > I think the proposal of having all operations return a decimal is
> attractive in that these expressions always result in a consistent type.
> Two pain points might be whether client languages have decimal support and
> whether there is a performance issue? The nice thing about always returning
> decimal is we can sidestep the issue of overflow.
> >
> > I would start with seeing if that's acceptable, and if it isn't then
> look at other approaches like returning a variety of types such when doing
> int + int return a bigint or int + float return a double.
> >
> > If we take an approach that allows overflow the ideal end state IMO
> would be to get all users to run Cassandra in way that overflow results in
> an error even in the context of aggregation. The road to get there is
> tricky, but maybe start by having it as an opt in tunable in
> cassandra.yaml. I don't know how/when we could ever change that as a
> default and it's unfortunate having an option like this that 99% won't know
> they should flip.
> >
> > It seems like having the default throw on overflow is not as bad as it
> sounds if you do the widening conversions since most people won't run into
> them. The change in the column types of results sets actually sounds worse
> if we want to also improve aggregrations. Many applications won't notice if
> the client library abstracts that away, but I think there are still cases
> where people would notice the type changing.
> >
> > Ariel
> >
> >> On Tue, Oct 2, 2018, at 11:09 AM, Benedict Elliott Smith wrote:
> >> This (overflow) is an excellent point, but this also affects
> >> aggregations which were introduced a long time ago.  They already
> >> inherit Java semantics for all of the relevant types (silent wrap
> >> around).  We probably want to be consistent, meaning either changing
> >> aggregations (which incurs a cost for changing API) or continuing the
> >> java semantics here.
> >>
> >> This is why having these discussions explicitly in the community before
> >> a release is so critical, in my view.  It’s very easy for these
> semantic
> >> changes to go unnoticed on a JIRA, and then ossify.
> >>
> >>
> >>> On 2 Oct 2018, at 15:48, Ariel Weisberg  wrote:
> >>>
> >>> Hi,
> >>>
> >>> I think we should decide based on what is least surprising as you
> mention, but isn't overridden by some other concern.
> >>>
> >>> It seems to me the priorities are
> >>>
> >>> * Correctness
> >>> * Performance
> >>> * User visible complexity
> >>> * Developer visible complexity
> >>>
> >>> Defaulting to silent implicit data loss is not ideal from a
> correctness standpoint.
> >>>
> >>> Doing something better like using wider types doesn't seem like a
> performance issue.
> >>>
> >>> From a user standpoint doing something less lossy doesn't look more
> complex as long as it's consistent, and documented and doesn't change from
> version to version.
> >>>
> >>> There is some developer complexity, but this is a public API and we
> only get one shot at this.
> >>>
> >>> I wonder about how overflow is handled as well. In VoltDB I think we
> threw on overflow and tended to just do widening conversions to make that
> less common. We didn't imitate another database (as far as I know) we just
> went with what least likely to silently corrupt data.
> >>>
> https://github.com/VoltDB/voltdb/blob/master/src/ee/common/NValue.hpp#L2213
> <
> https://github.com/VoltDB/voltdb/blob/master/src/ee/common/NValue.hpp#L2213
> >
> >>>
> https://github.com/VoltDB/voltdb/blob/master/src/ee/common/NValue.hpp#L3764
> <
> https://github.com/VoltDB/voltdb/blob/master/src/ee/common/NValue.hpp#L3764
> >
> >>>
> >>> Ariel
> >>>
>  

Re: [DISCUSS] changing default token behavior for 4.0

[Jonathan Haddad]

Is there a use case for random allocation? How does it help with testing? I
can’t see a reason to keep it around.

On Sat, Sep 22, 2018 at 3:06 AM kurt greaves  wrote:

> +1. I've been making a case for this for some time now, and was actually a
> focus of my talk last week. I'd be very happy to get this into 4.0.
>
> We've tested various num_tokens with the algorithm on various sized
> clusters and we've found that typically 16 works best. With lower numbers
> we found that balance is good initially but as a cluster gets larger you
> have some problems. E.g We saw that on a 60 node cluster with 8 tokens per
> node we were seeing a difference of 22% in token ownership, but on a <=12
> node cluster a difference of only 12%. 16 tokens on the other hand wasn't
> perfect but generally gave a better balance regardless of cluster size at
> least up to 100 nodes. TBH we should probably do some proper testing and
> record all the results for this before we pick a default (I'm happy to do
> this - think we can use the original testing script for this).
>
> But anyway, I'd say Jon is on the right track. Personally how I'd like to
> see it is that we:
>
>1. Change allocate_tokens_for_keyspace to allocate_tokens_for_rf in the
>same way that DSE does it. Allowing a user to specify a RF to allocate
>from, and allowing multiple DC's.
>2. Add a new boolean property random_token_allocation, defaults to
> false.
>3. Make allocate_tokens_for_rf default to *unset**.
>4. Make allocate_tokens_for_rf *required*** if num_tokens > 1 and
>random_token_allocation != true.
>5. Default num_tokens to 16 (or whatever we find appropriate)
>
> * I think setting a default is asking for trouble. When people are going to
> add new DC's/nodes we don't want to risk them adding a node with the wrong
> RF. I think it's safe to say that a user should have to think about this
> before they spin up their cluster.
> ** Following above, it should be required to be set so that we don't have
> people accidentally using random allocation. I think we should really be
> aiming to get rid of random allocation completely, but provide a new
> property to enable it for backwards compatibility (also for testing).
>
> It's worth noting that a smaller number of tokens *theoretically* decreases
> the time for replacement/rebuild, so if we're considering QUORUM
> availability with vnodes there's an argument against having a very low
> num_tokens. I think it's better to utilise NTS and racks to reduce the
> chance of a QUORUM outage over banking on having a lower number of tokens,
> as with just a low number of tokens unless you go all the way to 1 you are
> just relying on luck that 2 nodes don't overlap. Guess what I'm saying is
> that I think we should be choosing a num_tokens that gives the best
> distribution for most cluster sizes rather than choosing one that
> "decreases" the probability of an outage.
>
> Also I think we should continue using CASSANDRA-13701 to track this. TBH I
> think in general we should be a bit better at searching for and using
> existing tickets...
>
> On Sat, 22 Sep 2018 at 18:13, Stefan Podkowinski  wrote:
>
> > There already have been some discussions on this here:
> > https://issues.apache.org/jira/browse/CASSANDRA-13701
> >
> > The mentioned blocker there on the token allocation shouldn't exist
> > anymore. Although it would be good to get more feedback on it, in case
> > we want to enable it by default, along with new defaults for number of
> > tokens.
> >
> >
> > On 22.09.18 06:30, Dinesh Joshi wrote:
> > > Jon, thanks for starting this thread!
> > >
> > > I have created CASSANDRA-14784 to track this.
> > >
> > > Dinesh
> > >
> > >> On Sep 21, 2018, at 9:18 PM, Sankalp Kohli 
> > wrote:
> > >>
> > >> Putting it on JIRA is to make sure someone is assigned to it and it is
> > tracked. Changes should be discussed over ML like you are saying.
> > >>
> > >> On Sep 21, 2018, at 21:02, Jonathan Haddad  wrote:
> > >>
> > >>>> We should create a JIRA to find what other defaults we need revisit.
> > >>> Changing a default is a pretty big deal, I think we should discuss
> any
> > >>> changes to defaults here on the ML before moving it into JIRA.  It's
> > nice
> > >>> to get a bit more discussion around the change than what happens in
> > JIRA.
> > >>>
> > >>> We (TLP) did some testing on 4 tokens and found it to work
> surprisingly
> > >>> well.   It wasn't particularly formal, but we verified the load stays
> &

Re: [DISCUSS] changing default token behavior for 4.0

[Jonathan Haddad]

> We should create a JIRA to find what other defaults we need revisit.

Changing a default is a pretty big deal, I think we should discuss any
changes to defaults here on the ML before moving it into JIRA.  It's nice
to get a bit more discussion around the change than what happens in JIRA.

We (TLP) did some testing on 4 tokens and found it to work surprisingly
well.   It wasn't particularly formal, but we verified the load stays
pretty even with only 4 tokens as we added nodes to the cluster.  Higher
token count hurts availability by increasing the number of nodes any given
node is a neighbor with, meaning any 2 nodes that fail have an increased
chance of downtime when using QUORUM.  In addition, with the recent
streaming optimization it seems the token counts will give a greater chance
of a node streaming entire sstables (with LCS), meaning we'll do a better
job with node density out of the box.

Next week I can try to put together something a little more convincing.
Weekend time.

Jon


On Fri, Sep 21, 2018 at 8:45 PM sankalp kohli 
wrote:

> +1 to lowering it.
> Thanks Jon for starting this.We should create a JIRA to find what other
> defaults we need revisit. (Please keep this discussion for "default token"
> only.  )
>
> On Fri, Sep 21, 2018 at 8:26 PM Jeff Jirsa  wrote:
>
> > Also agree it should be lowered, but definitely not to 1, and probably
> > something closer to 32 than 4.
> >
> > --
> > Jeff Jirsa
> >
> >
> > > On Sep 21, 2018, at 8:24 PM, Jeremy Hanna 
> > wrote:
> > >
> > > I agree that it should be lowered. What I’ve seen debated a bit in the
> > past is the number but I don’t think anyone thinks that it should remain
> > 256.
> > >
> > >> On Sep 21, 2018, at 7:05 PM, Jonathan Haddad 
> wrote:
> > >>
> > >> One thing that's really, really bothered me for a while is how we
> > default
> > >> to 256 tokens still.  There's no experienced operator that leaves it
> as
> > is
> > >> at this point, meaning the only people using 256 are the poor folks
> that
> > >> just got started using C*.  I've worked with over a hundred clusters
> in
> > the
> > >> last couple years, and I think I only worked with one that had lowered
> > it
> > >> to something else.
> > >>
> > >> I think it's time we changed the default to 4 (or 8, up for debate).
> > >>
> > >> To improve the behavior, we need to change a couple other things.  The
> > >> allocate_tokens_for_keyspace setting is... odd.  It requires you have
> a
> > >> keyspace already created, which doesn't help on new clusters.  What
> I'd
> > >> like to do is add a new setting, allocate_tokens_for_rf, and set it to
> > 3 by
> > >> default.
> > >>
> > >> To handle clusters that are already using 256 tokens, we could prevent
> > the
> > >> new node from joining unless a -D flag is set to explicitly allow
> > >> imbalanced tokens.
> > >>
> > >> We've agreed to a trunk freeze, but I feel like this is important
> enough
> > >> (and pretty trivial) to do now.  I'd also personally characterize this
> > as a
> > >> bug fix since 256 is horribly broken when the cluster gets to any
> > >> reasonable size, but maybe I'm alone there.
> > >>
> > >> I honestly can't think of a use case where random tokens is a good
> > choice
> > >> anymore, so I'd be fine / ecstatic with removing it completely and
> > >> requiring either allocate_tokens_for_keyspace (for existing clusters)
> > >> or allocate_tokens_for_rf
> > >> to be set.
> > >>
> > >> Thoughts?  Objections?
> > >> --
> > >> Jon Haddad
> > >> http://www.rustyrazorblade.com
> > >> twitter: rustyrazorblade
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

[DISCUSS] changing default token behavior for 4.0

[Jonathan Haddad]

One thing that's really, really bothered me for a while is how we default
to 256 tokens still.  There's no experienced operator that leaves it as is
at this point, meaning the only people using 256 are the poor folks that
just got started using C*.  I've worked with over a hundred clusters in the
last couple years, and I think I only worked with one that had lowered it
to something else.

I think it's time we changed the default to 4 (or 8, up for debate).

To improve the behavior, we need to change a couple other things.  The
allocate_tokens_for_keyspace setting is... odd.  It requires you have a
keyspace already created, which doesn't help on new clusters.  What I'd
like to do is add a new setting, allocate_tokens_for_rf, and set it to 3 by
default.

To handle clusters that are already using 256 tokens, we could prevent the
new node from joining unless a -D flag is set to explicitly allow
imbalanced tokens.

We've agreed to a trunk freeze, but I feel like this is important enough
(and pretty trivial) to do now.  I'd also personally characterize this as a
bug fix since 256 is horribly broken when the cluster gets to any
reasonable size, but maybe I'm alone there.

I honestly can't think of a use case where random tokens is a good choice
anymore, so I'd be fine / ecstatic with removing it completely and
requiring either allocate_tokens_for_keyspace (for existing clusters)
or allocate_tokens_for_rf
to be set.

Thoughts?  Objections?
-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: QA signup

[Jonathan Haddad]

Sure - I'm not disagreeing with you that pre-built packages would be nice
to have.  That said, if someone's gone through the trouble of building an
entire testing infrastructure and has hundreds of machines available,
running `docker-compose up build-deb` is likely not a major issue.  If I'm
trying to decide between solving the 2 problems I'd prefer to make builds
easier as very few people actually know how to do it.  I'm also biased
because I'm working on a tool that does _exactly_ that (build arbitrary C*
debs and deploy them to AWS for perf testing with tlp-stress which we've
already open sourced https://github.com/thelastpickle/tlp-stress).

I'll building it for internal TLP use but there's not much TLP specific
stuff, we'll be open sourcing it as soon as we can.

TL;DR: we need both things

On Thu, Sep 20, 2018 at 2:12 PM Scott Andreas  wrote:

> Mick – Got it, thanks and sorry to have misunderstood. No fault in your
> writing at all; that was my misreading.
>
> Agreed with you and Kurt; I can’t think of a pressing need or immediate
> use for the Maven artifacts. As you mentioned, all of the examples I’d
> listed require binary artifacts only.
>
> Re: Jon’s question:
> > It seems to me that improving / simplifying the process of building the
> packages might solve this problem better.
>
> Agreed that making builds easy is important, and that manually-applied
> patches were involved in a couple cases I’d cited. My main motivation is
> toward making it easier for developers who’d like to produce
> fully-automated test pipelines to do so using common artifacts, rather than
> each replicating the build/packaging step for tarball artifacts themselves.
>
> Publishing binary artifacts in a common location would enable developers
> to configure testing and benchmarking pipelines to pick up those artifacts
> on a daily basis without intervention. In the case of a build landing DOA
> due to an issue with a commit, it’d be enough for zero-touch automation to
> pick up a new build with the fix the following day and run an extended
> suite across a large number of machines and publish results, for example.
>
>
> On September 19, 2018 at 8:17:05 PM, kurt greaves (k...@instaclustr.com
> ) wrote:
>
> It's pretty much only third party plugins. I need it for the LDAP
> authenticator, and StratIO's lucene plugin will also need it. I know there
> are users out there with their own custom plugins that would benefit from
> it as well (and various other open source projects). It would make it
> easier, however it certainly is feasible for these devs to just build the
> jars themselves (and I've done this so far). If it's going to be easy I
> think there's value in generating and hosting nightly jars, but if it's
> difficult I can just write some docs for DIY.
>
> On Thu, 20 Sep 2018 at 12:20, Mick Semb Wever  wrote:
>
> > Sorry about the terrible english in my last email.
> >
> >
> > > On the target audience:
> > >
> > > [snip]
> > > For developers building automation around testing and
> > > validation, it’d be great to have a common build to work from rather
> > > than each developer producing these builds themselves.
> >
> >
> > Sure. My question was only in context of maven artefacts.
> > It seems to me all the use-cases you highlight would be for the binary
> > artefacts.
> >
> > If that's the case we don't need to worry about publishing snapshots
> maven
> > artefacts, and can just focus on uploading nightly builds to
> > https://dist.apache.org/repos/dist/dev/cassandra/
> >
> > Or is there a use-case I'm missing that needs the maven artefacts?
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: QA signup

[Jonathan Haddad]

It seems to me that improving / simplifying the process of building the
packages might solve this problem better.  For example, in the tests you
linked to, they were using a custom build that hadn't been rolled into
trunk.  I expect we're going to see a lot of that.

If we make building a deb package as easy as running `docker-compose run
build-deb` then that addresses the problem of testing all branches.

This sort of exists already in cassandra-builds, but it's doing a little
more than just building a basic deb package.  Might be easier if it was
directly in the Cassandra repo.

On Wed, Sep 19, 2018 at 5:00 PM Scott Andreas  wrote:

> Got it, thanks!
>
> On the target audience:
>
> This would primarily be C* developers who are working on development,
> testing, and validation of the release. The performance testing exercise
> Joey, Vinay, Sumanth, Jason, Jordan, and Dinesh completed yesterday is a
> good example [1]. For developers building automation around testing and
> validation, it’d be great to have a common build to work from rather than
> each developer producing these builds themselves.
>
> Some purposes that could serve:
> – Ensuring we’re testing the same artifact. While a build produced from a
> given SHA should be ~identical, we can make stronger statements about
> particular build artifacts produced by common infrastructure than local
> builds produced by individual developers.
> – Automation: the ability to automate test suite runs on publication of a
> new build (e.g., perf, replay, traffic shadowing, upgrade tests, etc).
> – Faster dev/test/validation cycles. The perf tests in [1] identified two
> issues whose fixes will land in C-14503. Being able to pick up these fixes
> on commit via automation provides quicker feedback than waiting for a new
> build to be manually cut.
> – Fewer developers experiencing blocked automation. In a case where a
> regression is identified in a build produced by a commit (e.g., a snapshot
> build is “DOA” for testing purposes), a quick fix could resolve the issue
> with a new testable artifact produced within a day.
> – Better delineation between developer builds and those we recommend to
> the user community. Our ability to produce snapshot/nightly artifacts
> reduces pressure to cut an alpha for testing, reducing pressure to nominate
> community-facing releases in order to further the developer-focused goals
> above.
>
> ––
> [1]
> https://cwiki.apache.org/confluence/display/CASSANDRA/4.0+Performance+Testing
>
>
> On September 18, 2018 at 5:47:18 PM, Mick Semb Wever (m...@apache.org
> ) wrote:
>
> Scott,
>
> > @Mick, thanks for your reply re: publishing snapshots/nightlies. In
> > terms of what’s needed to configure these, would it be automation around
> > building release artifacts, publishing jars to the Maven snapshots repo,
> …
>
>
> Maven artefacts are deployed to the ASF snapshot repository in Nexus.
> The short of it is to add credentials for `apache.snapshots.https` to
> ~/.m2/settings.xml and run `ant publish`.
>
> It looks like `ant publish` won't run when it's not a release, but
> otherwise the maven deploy properties in `build.xml` look correct for
> snapshots.
>
> I haven't looked into how to automate this in Jenkins in regards to the
> settings.xml credentials and the gpg signing.
>
> For info at: http://www.apache.org/dev/publishing-maven-artifacts.html
>
> I question I have is who are we targeting with maven snapshots? Is this an
> audience that can easily enough be building the jars themselves to test
> during the feature freeze period?
>
>
> > and to dist/dev/cassandra on dist.apache.org for binary artifacts?
>
>
> This is a simpler task, just upload (`svn commit`) the nightly binaries to
> https://dist.apache.org/repos/dist/dev/cassandra/
>
> See https://www.apache.org/legal/release-policy.html#host-rc
>
> regards,
> Mick
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Proposing an Apache Cassandra Management process

[Jonathan Haddad]

Most of the discussion and work was done off the mailing list - there's a
big risk involved when folks disappear for months at a time and resurface
with big pile of code plus an agenda that you failed to loop everyone in
on. In addition, by your own words the design document didn't accurately
describe what was being built.  I don't write this to try to argue about
it, I just want to put some perspective for those of us that weren't part
of this discussion on a weekly basis over the last several months.  Going
forward let's keep things on the ML so we can avoid confusion and
frustration for all parties.

With that said - I think Blake made a really good point here and it's
helped me understand the scope of what's being built better.  Looking at it
from a different perspective it doesn't seem like there's as much overlap
as I had initially thought.  There's the machinery that runs certain tasks
(what Joey has been working on) and the user facing side of exposing that
information in management tool.

I do appreciate (and like) the idea of not trying to boil the ocean, and
working on things incrementally.  Putting a thin layer on top of Cassandra
that can perform cluster wide tasks does give us an opportunity to move in
the direction of a general purpose user-facing admin tool without
committing to trying to write the full stack all at once (or even make
decisions on it now).  We do need a sensible way of doing rolling restarts
/ scrubs / scheduling and Reaper wasn't built for that, and even though we
can add it I'm not sure if it's the best mechanism for the long term.

So if your goal is to add maturity to the project by making cluster wide
tasks easier by providing a framework to build on top of, I'm in favor of
that and I don't see it as antithetical to what I had in mind with Reaper.
Rather, the two are more complementary than I had originally realized.

Jon




On Thu, Sep 13, 2018 at 10:39 AM dinesh.jo...@yahoo.com.INVALID
 wrote:

> I have a few clarifications -
> The scope of the management process is not to simply run repair
> scheduling. Repair scheduling is one of the many features we could
> implement or adopt from existing sources. So could we please split the
> Management Process discussion and the repair scheduling?
> After re-reading the management process proposal, I see we missed to
> communicate a basic idea in the document. We wanted to take a pluggable
> approach to various activities that the management process could perform.
> This could accommodate different implementations of common activities such
> as repair. The management process would provide the basic framework and it
> would have default implementations for some of the basic activities. This
> would allow for speedier iteration cycles and keep things extensible.
> Turning to some questions that Jon and others have raised, when I +1, my
> intention is to fully contribute and stay with this community. That said,
> things feel rushed for some but for me it feels like analysis paralysis.
> We're looking for actionable feedback and to discuss the management process
> _not_ repair scheduling solutions.
> Thanks,
> Dinesh
>
>
>
> On Sep 12, 2018, at 6:24 PM, sankalp kohli  wrote:
> Here is a list of open discussion points from the voting thread. I think
> some are already answered but I will still gather these questions here.
>
> From several people:
> 1. Vote is rushed and we need more time for discussion.
>
> From Sylvain
> 2. About the voting process...I think that was addressed by Jeff Jirsa and
> deserves a separate thread as it is not directly related to this thread.
> 3. Does the project need a side car.
>
> From Jonathan Haddad
> 4. Are people doing +1 willing to contribute
>
> From Jonathan Ellis
> 5. List of feature set, maturity, maintainer availability from Reaper or
> any other project being donated.
>
> Mick Semb Wever
> 6. We should not vote on these things and instead build consensus.
>
> Open Questions from this thread
> 7. What technical debts we are talking about in Reaper. Can someone give
> concrete examples.
> 8. What is the timeline of donating Reaper to Apache Cassandra.
>
> On Wed, Sep 12, 2018 at 3:49 PM sankalp kohli 
> wrote:
>
>
> (Using this thread and not the vote thread intentionally)
> For folks talking about vote being rushed. I would use the email from
> Joseph to show this is not rushed. There was no email on this thread for 4
> months until I pinged.
>
>
> Dec 2016: Vinay worked with Jon and Alex to try to collaborate on Reaper to
> come up with design goals for a repair scheduler that could work at Netflix
> scale.
>
> ~Feb 2017: Netflix believes that the fundamental design gaps prevented us
> from using Reaper as it relies heavily on remote JMX connections and
> central coordination.
>
> Sep

Re: [VOTE] Development Approach for Apache Cassandra Management process

[Jonathan Haddad]

This voting process feels a bit rushed and frankly not well thought out.
In addition to Sylvain's valid points, which you (Sankalp) didn't address
at all, the discussion in the other threads seemed to be ongoing.  The last
email you wrote on one of them was asking for additional feedback, that
indicates the discussion is still open.

Out of principal I vote for none of the options (inaction).  You're
deliberately trying to ram *something* through, and that's not how this is
supposed to work.

For those of you unfamiliar with the process - please read
https://www.apache.org/foundation/voting.html.

I'd like to ask those of you that are +1'ing, are you willing to contribute
or are you just voting we start an admin tool from scratch because you
think it'll somehow produce a perfect codebase?





On Wed, Sep 12, 2018 at 12:50 PM sankalp kohli 
wrote:

> Hi Sylvain,
> I would appreciate if we can give feedback on the
> discussion threads and not wait for vote threads. I made it clear in the
> discussion thread that we will start a vote!!
> Thanks,
> Sankalp
>
> On Wed, Sep 12, 2018 at 12:47 PM Jeff Jirsa  wrote:
>
> > On Wed, Sep 12, 2018 at 12:41 PM Sylvain Lebresne 
> > wrote:
> >
> > > That's probably a stupid question, and excuse me if it is, but what
> does
> > > those votes on the dev mailing list even mean?
> > >
> > > How do you count votes at the end? Just by counting all votes cast,
> > > irregardless of whomever cast it? Or are we intending to only count PMC
> > > members, or maybe committers votes?
> > >
> >
> > I believe the intent is to try to see if there exists consensus.
> > Ultimately, PMC is going to matter more than random email addresses from
> > people nobody recognizes. This should be in public, though, not private,
> so
> > seeing what feedback is beyond the PMC is useful (primarily because it
> will
> > matter when it comes time to extend and maintain it - if people strongly
> > prefer one or the other, then maintenance is going to be a problem).
> >
> > If there's 100 random non-contributor votes for one option and 20 pmc
> votes
> > for another options, I think the real answer will be "we don't have
> > consensus, and either we don't do it, or we do it the way the PMC thinks
> is
> > best", for all of the reasons you describe in the paragraphs below.
> >
> >
> > > If the former, that is a bit weird to me because we simply don't know
> who
> > > votes. And I don't mean to be rude towards anyone, but 1) someone could
> > > easily create 10 email addresses to vote 10 times (and sure, you could
> > > invoke trust, and I'm not entirely against trust in general, but it's
> the
> > > internet...) and 2) this kind of decision will have non-trivial
> > > consequences for the project, particularly on those that maintain it,
> so
> > I
> > > admit I'm not entirely comfortable with "anyone's voice has the same
> > > weight".
> > > If the latter, then this makes more sense to me (why are we even
> > bothering
> > > voting PMC members in if it's not to handle these kinds of decisions,
> > which
> > > are very "project management" related), but we should be very clear
> about
> > > this from the get go (we could still use the dev list for transparency
> > > sake, that I don't mind)? We should probably also have some deadline to
> > the
> > > vote, one that isn't too short.
> > >
> >
> > Like releases, I think PMC votes count
> >
> >
> > >
> > > Anyway, fwiw, my opinion on this vote is not far from the one on the
> > golang
> > > driver acceptance vote (for which my remark above also apply btw): no
> yet
> > > 100% convinced adding more pieces and scope to the project is what the
> > > project needs just right now, but not strongly opposed if people really
> > > wants this (and this one makes more sense to me than the golang driver
> > > actually). But if I'm to pick between a) and b), I'm leaning b).
> > >
> >
> > FWIW, two of the main reasons I'm in favor is as a way to lower barrier
> to
> > entry to both using the software AND contributing to the project, so I
> > think your points are valid (both on gocql thread and on this note
> above),
> > but I think that's also part of why we should be encouraging both.
> >
> > - Jeff
> >
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: [VOTE] Accept GoCQL driver donation and begin incubation process

[Jonathan Haddad]

I'm +0, and I share the same concerns as Sylvain.

For those of you that have +1'ed, are you planning on contributing to the
driver?  Docs, code, QA?  It's easy to throw a +1 down to make the driver
the responsibility of the project if you're asking others to do the work.
I vote this way because I already know I won't contribute to it and it's
irresponsible to vote something in if I have no intent on helping maintain
it.

On Wed, Sep 12, 2018 at 10:45 AM Sumanth Pasupuleti
 wrote:

> +1
>
> On Wed, Sep 12, 2018 at 10:37 AM Dinesh Joshi
>  wrote:
>
> > +1
> >
> > Dinesh
> >
> > > On Sep 12, 2018, at 10:23 AM, Jaydeep Chovatia <
> > chovatia.jayd...@gmail.com> wrote:
> > >
> > > +1
> > >
> > > On Wed, Sep 12, 2018 at 10:00 AM Roopa Tangirala
> > >  wrote:
> > >
> > >> +1
> > >>
> > >>
> > >> *Regards,*
> > >>
> > >> *Roopa Tangirala*
> > >>
> > >> Engineering Manager CDE
> > >>
> > >> *(408) 438-3156 - mobile*
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Wed, Sep 12, 2018 at 8:51 AM Sylvain Lebresne 
> > >> wrote:
> > >>
> > >>> -0
> > >>>
> > >>> The project seems to have a hard time getting on top of reviewing his
> > >>> backlog
> > >>> of 'patch available' issues, so that I'm skeptical adopting more code
> > to
> > >>> maintain is the thing the project needs the most right now. Besides,
> > I'm
> > >>> also
> > >>> generally skeptical that augmenting the scope of a project makes it
> > >> better:
> > >>> I feel
> > >>> keeping this project focused on the core server is better. I see
> risks
> > >>> here, but
> > >>> the upsides haven't been made very clear for me, even for end users:
> > yes,
> > >>> it
> > >>> may provide a tiny bit more clarity around which Golang driver to
> > choose
> > >> by
> > >>> default, but I'm not sure users are that lost, and I think there is
> > other
> > >>> ways to
> > >>> solve that if we really want.
> > >>>
> > >>> Anyway, I reckon I may be overly pessimistic here and it's not that
> > >> strong
> > >>> of
> > >>> an objection if a large majority is on-board, so giving my opinion
> but
> > >> not
> > >>> opposing.
> > >>>
> > >>> --
> > >>> Sylvain
> > >>>
> > >>>
> > >>> On Wed, Sep 12, 2018 at 5:36 PM Jeremiah D Jordan <
> > >>> jeremiah.jor...@gmail.com>
> > >>> wrote:
> > >>>
> >  +1
> > 
> >  But I also think getting this through incubation might take a
> while/be
> >  impossible given how large the contributor list looks…
> > 
> > > On Sep 12, 2018, at 10:22 AM, Jeff Jirsa  wrote:
> > >
> > > +1
> > >
> > > (Incubation looks like it may be challenging to get acceptance from
> > >> all
> >  existing contributors, though)
> > >
> > > --
> > > Jeff Jirsa
> > >
> > >
> > >> On Sep 12, 2018, at 8:12 AM, Nate McCall 
> > >> wrote:
> > >>
> > >> This will be the same process used for dtest. We will need to walk
> > >> this through the incubator per the process outlined here:
> > >>
> > >>
> > 
> > >>>
> > >>
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__incubator.apache.org_guides_ip-5Fclearance.html=DwIFAg=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=g-MlYFZVJ7j5Dj_ZfPfa0Ik8Nxco7QsJhTG1TnJH7xI=rk5T_t1HZY6PAhN5XgflBhfEtNrcZkVTIvQxixDlw9o=
> > >>
> > >> Pending the outcome of this vote, we will create the JIRA issues
> for
> > >> tracking and after we go through the process, and discuss adding
> > >> committers in a separate thread (we need to do this atomically
> > >> anyway
> > >> per general ASF committer adding processes).
> > >>
> > >> Thanks,
> > >> -Nate
> > >>
> > >>
> > >> -
> > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >>
> > >
> > >
> -
> > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >
> > 
> > 
> > 
> -
> >  To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >  For additional commands, e-mail: dev-h...@cassandra.apache.org
> > 
> > 
> > >>>
> > >>
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Proposing an Apache Cassandra Management process

[Jonathan Haddad]

It's interesting to me that someone would consider features of Reaper as
"technical debt"... an odd way to word it.  When we had proposed donating
Reaper to the project, I had made the assumption people would realize the
benefit of folding in a successful project.  A working codebase with a
large user base gives us an immediate tool, and it would let us migrate
everyone from using our tool to using an official tool.

>From the discussions we've had so far, nobody except Blake seems to care
about those users.  To me, the highest priority of this process is
considering what's best for the Cassandra community.  Providing an easy,
clear migration path forward for the thousands of Reaper users, supporting
all 4 backends and cassandra versions is the logical the way of
transitioning everyone to using the official tool.

If supporting everyone using 2.x with a postgres backend for storing
schedules isn't something anyone cares about, there's not much benefit from
using Reaper.  Gutting a bunch of the project won't help people migrate
over, and if we're not migrating everyone over, then we (TLP) will still
have to maintain Reaper.  That means we'll be indefinitely maintaining a
fork of the official admin, and probably not contributing to the main one,
our time is limited.  That's exactly what's going to happen if we start
with anything other than Reaper.  We've got a lot of teams asking for
features, and if people pay us to keep working on Reaper, we'll be doing
that.

So, with that in mind, if we're putting this to a vote, I'd like to be
clear - taking Reaper as-is about community - not it's technical prowess.
If we're not enabling users to move off our version and directly to the
official tool, I don't see a point in using Reaper at all except as a
reference to correctly run subrange repair and have a nice UI.  There's
plenty of things I'd do differently if I built it from scratch, I'm not
putting the codebase up on a pedestal.

This entire discussion has been incredibly frustrating for me, considering
we're talking about building an alternative to a well tested and widely
deployed codebase that likely won't produce anything of value for at least
a year (Cassandra 5?).  I'm also pretty sure most of the folks that have
cried out "tech debt" have spent less than 5 minutes looking at the
codebase.  I hope your enthusiasm at the moment would carry over if you
build a tech-debt free admin tool from the ground up.

TL;DR: If nobody cares about the Reaper community (which is a very large
subset of the cassandra community) there's no reason to start with Reaper,
it's not about the tech it's about the people.

Jon


On Sat, Sep 8, 2018 at 4:57 PM sankalp kohli  wrote:

> Most of the discussions have been around whether we take some side car or
> do a cherry pick approach where we add a feature at a time to side car and
> use the best implementation.
> We have been discussing this first in April and now for several days. I
> think we all want some progress here. We will start a vote soon..may be
> next week to decide which approach we will take. I don't see any other
> option to make progress besides a vote!!
>
> PS: I think these discussions are very good for the community and it shows
> people care about Apache Cassandra and it is a sign of healthy community.
> Several people offering to donate the side car or help build is showing the
> interest everyone has in it.
>
>
> On Sat, Sep 8, 2018 at 11:17 AM Joseph Lynch 
> wrote:
>
> > On Fri, Sep 7, 2018 at 10:00 PM Blake Eggleston 
> > wrote:
> > >
> > > Right, I understand the arguments for starting a new project. I’m not
> > saying reaper is, technically speaking, the best place to start. The
> point
> > I’m trying to make is that the non-technical advantages of using an
> > existing project as a starting point may outweigh the technical benefits
> of
> > a clean slate. Whether that’s the case or not, it’s not a strictly
> > technical decision, and the non-technical advantages of starting with
> > reaper need to be weighed.
> > >
> >
> > Technical debt doesn't just refer to the technical solutions, having
> > an existing user base means that a project has made promises in the
> > past (in the case of Priam 5+ years ago) that the new project would
> > have to keep if we make keeping users of those sidecars a goal (which
> > for the record I don't think should be a goal, I think the goal is to
> > make Cassandra the database work out of the box in the 4.x series).
> >
> > For example, Priam has to continue supporting the following as users
> > actively use them (including Netflix):
> > * Parallel token assignment and creation allowing parallel bootstrap
> > and parallel doubling of hundred node clusters at once (as long as you
> > use single tokens and run in AWS with asgs).
> > * 3+ backup solutions, as well as assumptions about where in e.g. S3
> > backups are present and what format they're present in.
> > * Numerous configuration options and UI elements (mostly 5 

Re: Proposing an Apache Cassandra Management process

[Jonathan Haddad]

We haven’t even defined any requirements for an admin tool. It’s hard to
make a case for anything without agreement on what we’re trying to build.

On Fri, Sep 7, 2018 at 7:17 PM Jeff Jirsa  wrote:

> How can we continue moving this forward?
>
> Mick/Jon/TLP folks, is there a path here where we commit the
> Netflix-provided management process, and you augment Reaper to work with
> it?
> Is there a way we can make a larger umbrella that's modular that can
> support either/both?
> Does anyone believe there's a clear, objective argument that one is
> strictly better than the other? I haven't seen one.
>
>
>
> On Mon, Aug 20, 2018 at 4:14 PM Roopa Tangirala
>  wrote:
>
> > +1 to everything that Joey articulated with emphasis on the fact that
> > contributions should be evaluated based on the merit of code and their
> > value add to the whole offering. I  hope it does not matter whether that
> > contribution comes from PMC member or a person who is not a committer. I
> > would like the process to be such that it encourages the new members to
> be
> > a part of the community and not shy away from contributing to the code
> > assuming their contributions are valued differently than committers or
> PMC
> > members. It would be sad to see the contributions decrease if we go down
> > that path.
> >
> > *Regards,*
> >
> > *Roopa Tangirala*
> >
> > Engineering Manager CDE
> >
> > *(408) 438-3156 - mobile*
> >
> >
> >
> >
> >
> >
> > On Mon, Aug 20, 2018 at 2:58 PM Joseph Lynch 
> > wrote:
> >
> > > > We are looking to contribute Reaper to the Cassandra project.
> > > >
> > > Just to clarify are you proposing contributing Reaper as a project via
> > > donation or you are planning on contributing the features of Reaper as
> > > patches to Cassandra? If the former how far along are you on the
> donation
> > > process? If the latter, when do you think you would have patches ready
> > for
> > > consideration / review?
> > >
> > >
> > > > Looking at the patch it's very similar in its base design already,
> but
> > > > Reaper does has a lot more to offer. We have all been working hard to
> > > move
> > > > it to also being a side-car so it can be contributed. This raises a
> > > number
> > > > of relevant questions to this thread: would we then accept both works
> > in
> > > > the Cassandra project, and what burden would it put on the current
> PMC
> > to
> > > > maintain both works.
> > > >
> > > I would hope that we would collaborate on merging the best parts of all
> > > into the official Cassandra sidecar, taking the always on, shared
> > nothing,
> > > highly available system that we've contributed a patchset for and
> adding
> > in
> > > many of the repair features (e.g. schedules, a nice web UI) that Reaper
> > > has.
> > >
> > >
> > > > I share Stefan's concern that consensus had not been met around a
> > > > side-car, and that it was somehow default accepted before a patch
> > landed.
> > >
> > >
> > > I feel this is not correct or fair. The sidecar and repair discussions
> > have
> > > been anything _but_ "default accepted". The timeline of consensus
> > building
> > > involving the management sidecar and repair scheduling plans:
> > >
> > > Dec 2016: Vinay worked with Jon and Alex to try to collaborate on
> Reaper
> > to
> > > come up with design goals for a repair scheduler that could work at
> > Netflix
> > > scale.
> > >
> > > ~Feb 2017: Netflix believes that the fundamental design gaps prevented
> us
> > > from using Reaper as it relies heavily on remote JMX connections and
> > > central coordination.
> > >
> > > Sep. 2017: Vinay gives a lightning talk at NGCC about a highly
> available
> > > and distributed repair scheduling sidecar/tool. He is encouraged by
> > > multiple committers to build repair scheduling into the daemon itself
> and
> > > not as a sidecar so the database is truly eventually consistent.
> > >
> > > ~Jun. 2017 - Feb. 2018: Based on internal need and the positive
> feedback
> > at
> > > NGCC, Vinay and myself prototype the distributed repair scheduler
> within
> > > Priam and roll it out at Netflix scale.
> > >
> > > Mar. 2018: I open a Jira (CASSANDRA-14346) along with a detailed 20
> page
> > > design document for adding repair scheduling to the daemon itself and
> > open
> > > the design up for feedback from the community. We get feedback from
> Alex,
> > > Blake, Nate, Stefan, and Mick. As far as I know there were zero
> proposals
> > > to contribute Reaper at this point. We hear the consensus that the
> > > community would prefer repair scheduling in a separate distributed
> > sidecar
> > > rather than in the daemon itself and we re-work the design to match
> this
> > > consensus, re-aligning with our original proposal at NGCC.
> > >
> > > Apr 2018: Blake brings the discussion of repair scheduling to the dev
> > list
> > > (
> > >
> > >
> >
> https://lists.apache.org/thread.html/760fbef677f27aa5c2ab4c375c7efeb81304fea428deff986ba1c2eb@%3Cdev.cassandra.apache.org%3E
> > > ).
> > > 

Re: QA signup

[Jonathan Haddad]

Really good idea JD. Keeping all the tests under an umbrella ticket for the
feature with everything linked back makes a lot of sense.

On Thu, Sep 6, 2018 at 11:09 PM J. D. Jordan 
wrote:

> I would suggest that JIRA’s tagged as 4.0 blockers be created for the list
> once it is fleshed out.  Test plans and results could be posted to said
> JIRAs, to be closed once a given test passes. Any bugs found can also then
> be related back to such a ticket for tracking them as well.
>
> -Jeremiah
>
> > On Sep 6, 2018, at 12:27 PM, Jonathan Haddad  wrote:
> >
> > I completely agree with you, Sankalp.  I didn't want to dig too deep into
> > the underlying testing methodology (and I still think we shouldn't just
> > yet) but if the goal is to have confidence in the release, our QA process
> > needs to be comprehensive.
> >
> > I believe that having focused teams for each component with a team leader
> > with support from committers & contributors gives us the best shot at
> > defining large scale functional tests that can be used to form both
> > progress and bug reports.  (A person could / hopefully will be on more
> than
> > one team).  Coming up with those comprehensive tests will be the jobs of
> > the teams, getting frequent bidirectional feedback on the dev ML.  Bugs
> go
> > in JIRA as per usual.
> >
> > Hopefully we can continue this process after the release, giving the
> > project more structure, and folding more people in over time as
> > contributors and ideally committers / PMC.
> >
> > Jon
> >
> >
> >> On Thu, Sep 6, 2018 at 1:15 PM sankalp kohli 
> wrote:
> >>
> >> Thanks for starting this Jon.
> >> Instead of saying "I tested streaming", we should define what all was
> >> tested like was all data transferred, what happened when stream failed,
> >> etc.
> >> Based on talking to a few users, looks like most testing is done by
> doing
> >> an operation or running a load and seeing if it "worked" and no errors
> in
> >> logs.
> >>
> >> Another important thing will be to fix bugs asap ahead of testing,  as
> >> fixes can lead to more bugs :)
> >>
> >>>> On Thu, Sep 6, 2018 at 7:52 AM Jonathan Haddad 
> wrote:
> >>>
> >>> I was thinking along the same lines.  For this to be successful I think
> >>> either weekly or bi-weekly summary reports back to the mailing list by
> >> the
> >>> team lead for each subsection on what's been tested and how it's been
> >>> tested will help keep things moving along.
> >>>
> >>> In my opinion the lead for each team should *not* be the contributor
> that
> >>> wrote the feature, but someone who's very interested in it and can use
> >> the
> >>> contributor as a resource.  I think it would be difficult for the
> >>> contributor to poke holes in their own work - if they could do that it
> >>> would have been done already.  This should be a verification process
> >> that's
> >>> independent as possible from the original work.
> >>>
> >>> In addition to the QA process, it would be great if we could get a docs
> >>> team together.  We've got quite a bit of undocumented features and
> nuance
> >>> still, I think hammering that out would be a good idea.  Mick brought
> up
> >>> updating the website docs in the thread on testing different JDK's [1],
> >> if
> >>> we could figure that out in the process we'd be in a really great
> >> position
> >>> from the user perspective.
> >>>
> >>> Jon
> >>>
> >>> [1]
> >>
> https://lists.apache.org/thread.html/5645178efb57939b96e73ab9c298e80ad8e76f11a563b4d250c1ae38@%3Cdev.cassandra.apache.org%3E
> >>>
> >>>>> On Thu, Sep 6, 2018 at 10:35 AM Jordan West 
> wrote:
> >>>>
> >>>> Thanks for staring this thread Jon!
> >>>>
> >>>>> On Thu, Sep 6, 2018 at 5:51 AM Jonathan Haddad 
> >>>> wrote:
> >>>>
> >>>>> For 4.0, I'm thinking it would be a good idea to put together a list
> >> of
> >>>> the
> >>>>> things that need testing and see if people are willing to help test /
> >>>> break
> >>>>> those things.  My goal here is to get as much coverage as possible,
> >> and
> >>>> let
> >>>>> folks focus on really hammering on specific things ra

Re: QA signup

[Jonathan Haddad]

I completely agree with you, Sankalp.  I didn't want to dig too deep into
the underlying testing methodology (and I still think we shouldn't just
yet) but if the goal is to have confidence in the release, our QA process
needs to be comprehensive.

I believe that having focused teams for each component with a team leader
with support from committers & contributors gives us the best shot at
defining large scale functional tests that can be used to form both
progress and bug reports.  (A person could / hopefully will be on more than
one team).  Coming up with those comprehensive tests will be the jobs of
the teams, getting frequent bidirectional feedback on the dev ML.  Bugs go
in JIRA as per usual.

Hopefully we can continue this process after the release, giving the
project more structure, and folding more people in over time as
contributors and ideally committers / PMC.

Jon


On Thu, Sep 6, 2018 at 1:15 PM sankalp kohli  wrote:

> Thanks for starting this Jon.
> Instead of saying "I tested streaming", we should define what all was
> tested like was all data transferred, what happened when stream failed,
> etc.
> Based on talking to a few users, looks like most testing is done by doing
> an operation or running a load and seeing if it "worked" and no errors in
> logs.
>
> Another important thing will be to fix bugs asap ahead of testing,  as
> fixes can lead to more bugs :)
>
> On Thu, Sep 6, 2018 at 7:52 AM Jonathan Haddad  wrote:
>
> > I was thinking along the same lines.  For this to be successful I think
> > either weekly or bi-weekly summary reports back to the mailing list by
> the
> > team lead for each subsection on what's been tested and how it's been
> > tested will help keep things moving along.
> >
> > In my opinion the lead for each team should *not* be the contributor that
> > wrote the feature, but someone who's very interested in it and can use
> the
> > contributor as a resource.  I think it would be difficult for the
> > contributor to poke holes in their own work - if they could do that it
> > would have been done already.  This should be a verification process
> that's
> > independent as possible from the original work.
> >
> > In addition to the QA process, it would be great if we could get a docs
> > team together.  We've got quite a bit of undocumented features and nuance
> > still, I think hammering that out would be a good idea.  Mick brought up
> > updating the website docs in the thread on testing different JDK's [1],
> if
> > we could figure that out in the process we'd be in a really great
> position
> > from the user perspective.
> >
> > Jon
> >
> > [1]
> >
> >
> https://lists.apache.org/thread.html/5645178efb57939b96e73ab9c298e80ad8e76f11a563b4d250c1ae38@%3Cdev.cassandra.apache.org%3E
> >
> > On Thu, Sep 6, 2018 at 10:35 AM Jordan West  wrote:
> >
> > > Thanks for staring this thread Jon!
> > >
> > > On Thu, Sep 6, 2018 at 5:51 AM Jonathan Haddad 
> > wrote:
> > >
> > > > For 4.0, I'm thinking it would be a good idea to put together a list
> of
> > > the
> > > > things that need testing and see if people are willing to help test /
> > > break
> > > > those things.  My goal here is to get as much coverage as possible,
> and
> > > let
> > > > folks focus on really hammering on specific things rather than just
> > > firing
> > > > up a cluster and rubber stamping it.  If we're going to be able to
> > > > confidently deploy 4.0 quickly after it's release we're going to
> need a
> > > > high attention to detail.
> > > >
> > > >
> > > +1 to a more coordinated effort. I think we could use the Confluence
> that
> > > was set up a little bit ago since it was setup for this purpose, at
> least
> > > for finalized plans and results:
> > > https://cwiki.apache.org/confluence/display/CASSANDRA.
> > >
> > >
> > > > In addition to a signup sheet, I think providing some guidance on how
> > to
> > > QA
> > > > each thing that's being tested would go a long way.  Throwing "hey
> > please
> > > > test sstable streaming" over the wall will only get quality feedback
> > from
> > > > folks that are already heavily involved in the development process.
> It
> > > > would be nice to bring some new faces into the project by providing a
> > > > little guidance.
> > > >
> > >
> > > > We could help facilitate this even further by considering the people
> > > >

Re: QA signup

[Jonathan Haddad]

I was thinking along the same lines.  For this to be successful I think
either weekly or bi-weekly summary reports back to the mailing list by the
team lead for each subsection on what's been tested and how it's been
tested will help keep things moving along.

In my opinion the lead for each team should *not* be the contributor that
wrote the feature, but someone who's very interested in it and can use the
contributor as a resource.  I think it would be difficult for the
contributor to poke holes in their own work - if they could do that it
would have been done already.  This should be a verification process that's
independent as possible from the original work.

In addition to the QA process, it would be great if we could get a docs
team together.  We've got quite a bit of undocumented features and nuance
still, I think hammering that out would be a good idea.  Mick brought up
updating the website docs in the thread on testing different JDK's [1], if
we could figure that out in the process we'd be in a really great position
from the user perspective.

Jon

[1]
https://lists.apache.org/thread.html/5645178efb57939b96e73ab9c298e80ad8e76f11a563b4d250c1ae38@%3Cdev.cassandra.apache.org%3E

On Thu, Sep 6, 2018 at 10:35 AM Jordan West  wrote:

> Thanks for staring this thread Jon!
>
> On Thu, Sep 6, 2018 at 5:51 AM Jonathan Haddad  wrote:
>
> > For 4.0, I'm thinking it would be a good idea to put together a list of
> the
> > things that need testing and see if people are willing to help test /
> break
> > those things.  My goal here is to get as much coverage as possible, and
> let
> > folks focus on really hammering on specific things rather than just
> firing
> > up a cluster and rubber stamping it.  If we're going to be able to
> > confidently deploy 4.0 quickly after it's release we're going to need a
> > high attention to detail.
> >
> >
> +1 to a more coordinated effort. I think we could use the Confluence that
> was set up a little bit ago since it was setup for this purpose, at least
> for finalized plans and results:
> https://cwiki.apache.org/confluence/display/CASSANDRA.
>
>
> > In addition to a signup sheet, I think providing some guidance on how to
> QA
> > each thing that's being tested would go a long way.  Throwing "hey please
> > test sstable streaming" over the wall will only get quality feedback from
> > folks that are already heavily involved in the development process.  It
> > would be nice to bring some new faces into the project by providing a
> > little guidance.
> >
>
> > We could help facilitate this even further by considering the people
> > signing up to test a particular feature as a team, with seasoned
> Cassandra
> > veterans acting as team leads.
> >
>
> +1 to this as well. I am always a fan of folks learning about a
> subsystem/project through testing. It can be challenging to get folks new
> to a project excited about testing first but for those that do, or for
> committers who want to learn another part of the db, its a great way to
> learn.
>
> Another thing we can do here is make sure teams are writing about the
> testing they are doing and their results. This will help share knowledge
> about techniques and approaches that others can then apply. This knowledge
> can be shared on the mailing list, a blog post, or in JIRA.
>
>  Jordan
>
>
> > Any thoughts?  I'm happy to take the lead on this.
> > --
> > Jon Haddad
> > http://www.rustyrazorblade.com
> > twitter: rustyrazorblade
> >
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

QA signup

[Jonathan Haddad]

For 4.0, I'm thinking it would be a good idea to put together a list of the
things that need testing and see if people are willing to help test / break
those things.  My goal here is to get as much coverage as possible, and let
folks focus on really hammering on specific things rather than just firing
up a cluster and rubber stamping it.  If we're going to be able to
confidently deploy 4.0 quickly after it's release we're going to need a
high attention to detail.

In addition to a signup sheet, I think providing some guidance on how to QA
each thing that's being tested would go a long way.  Throwing "hey please
test sstable streaming" over the wall will only get quality feedback from
folks that are already heavily involved in the development process.  It
would be nice to bring some new faces into the project by providing a
little guidance.

We could help facilitate this even further by considering the people
signing up to test a particular feature as a team, with seasoned Cassandra
veterans acting as team leads.

Any thoughts?  I'm happy to take the lead on this.
-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: NGCC 2018?

[Jonathan Haddad]

I'm thinking a month or two after 4.0 would give us time to unwind after
the release and start to give real thought to big changes coming in the
next release.  Let's focus on one thing at a time.

On Wed, Sep 5, 2018 at 12:29 PM sankalp kohli 
wrote:

> A good time for NGCC will be closer to 4.0 release where we can plan what
> we can put it on 4.0-next. I am not sure doing it now is going to help when
> we are months away from 4.0 release.
>
> On Fri, Aug 31, 2018 at 7:42 AM Jay Zhuang  wrote:
>
> > Are we going to have a dev event next month? Or anything this year? We
> may
> > also be able to provide space in bay area and help to organize it.
> (Please
> > let us know, so we could get final approval for that).
> >
> > On Fri, Jul 27, 2018 at 10:05 AM Jonathan Haddad 
> > wrote:
> >
> > > My interpretation of Nate's statement was that since there would be a
> > bunch
> > > of us at Lynn's event, we might as well do NGCC at the same time.
> > >
> > > On Thu, Jul 26, 2018 at 9:03 PM Ben Bromhead 
> > wrote:
> > >
> > > > It sounds like there may be an appetite for something, but the NGCC
> in
> > > its
> > > > current format is likely to not be that useful?
> > > >
> > > > Is a bay area event focused on C* developers something that is
> > > interesting
> > > > for the broader dev community? In whatever format that may be?
> > > >
> > > > On Tue, Jul 24, 2018 at 5:02 PM Nate McCall 
> > wrote:
> > > >
> > > > > This was discussed amongst the PMC recently. We did not come to a
> > > > > conclusion and there were not terribly strong feelings either way.
> > > > >
> > > > > I don't feel like we need to hustle to get "NGCC" in place,
> > > > > particularly given our decided focus on 4.0. However, that should
> not
> > > > > stop us from doing an additional 'c* developer' event in sept. to
> > > > > coincide with distributed data summit.
> > > > >
> > > > > On Wed, Jul 25, 2018 at 5:03 AM, Patrick McFadin <
> pmcfa...@gmail.com
> > >
> > > > > wrote:
> > > > > > Ben,
> > > > > >
> > > > > > Lynn Bender had offered a space the day before Distributed Data
> > > Summit
> > > > in
> > > > > > September (http://distributeddatasummit.com/) since we are both
> > > > platinum
> > > > > > sponsors. I thought he and Nate had talked about that being a
> good
> > > > place
> > > > > > for NGCC since many of us will be in town already.
> > > > > >
> > > > > > Nate, now that I've spoken for you, you can clarify, :D
> > > > > >
> > > > > > Patrick
> > > > > >
> > > > >
> > > > >
> -
> > > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > > > >
> > > > > --
> > > > Ben Bromhead
> > > > CTO | Instaclustr <https://www.instaclustr.com/>
> > > > +1 650 284 9692
> > > > Reliability at Scale
> > > > Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
> > > >
> > >
> > >
> > > --
> > > Jon Haddad
> > > http://www.rustyrazorblade.com
> > > twitter: rustyrazorblade
> > >
> >
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: [Discuss] Accept GoCQL driver donation

[Jonathan Haddad]

Definitely does not belong in the same repo.

I’m all for folding drivers in / writing our own, just needs active
committers.

On Fri, Aug 31, 2018 at 7:45 AM Michael Shuler 
wrote:

> On 08/31/2018 09:34 AM, Jay Zhuang wrote:
> > That's great. Could that be in the same repo as Cassandra or a
> > separate repo?
>
> For similar reasons as discussed for an admin tool, separate
> repositories are quick and simple to create, as well as allow handling
> contribution, CI, build, release, etc. mechanics more flexibly. I see
> little to no advantage to including allthethings in-tree. If there comes
> a time that particular contrib items do make sense to include in the
> main repository, dropping them in would be no problem.
>
> --
> Michael
>
> > On Fri, Aug 31, 2018 at 7:14 AM Nate McCall  wrote:
> >
> >> Hi folks,
> >> So I was recently talking with, Chris Bannister  the gocql [0]
> >> maintainer, and he expressed an interest in donating the driver to the
> >> ASF.
> >>
> >> We could accept this along the same lines as how we took in the dtest
> >> donation - going through the incubator IP clearance process [1], but
> >> in this case it's much simpler as an individual (Chris) owns the
> >> copyright.
> >>
> >> I think the end goal here is to have a reference protocol
> >> implementation controlled by the project at the least, potentially
> >> replace cqlsh with a GoLang statically compiled binary eventually (?).
> >>
> >> What are other folks' thoughts about this? (we are discussing, not
> voting).
> >>
> >> [0] https://github.com/gocql/gocql
> >> [1] https://incubator.apache.org/guides/ip_clearance.html
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>
> >>
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
> --
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Java 11 Z garbage collector

[Jonathan Haddad]

Advertised, yes, but so far I haven't found it to be any better than
ParNew + CMS or G1 in the performance tests I did when writing
http://thelastpickle.com/blog/2018/08/16/java11.html.

That said, I didn't try it with a huge heap (i think it was 16 or 24GB), so
maybe it'll do better if I throw 50 GB RAM at it.



On Thu, Aug 30, 2018 at 8:42 AM Carl Mueller
 wrote:

> https://www.opsian.com/blog/javas-new-zgc-is-very-exciting/
>
> .. max of 4ms for stop the world, large terabyte heaps, seems promising.
>
> Will this be a major boon to cassandra p99 times? Anyone know the aspects
> of cassandra that cause the most churn and lead to StopTheWorld GC? I was
> under the impression that bloom filters, caches, etc are statically
> allocated at startup.
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Yet another repair solution

[Jonathan Haddad]

I'm also very interested.


On Tue, Aug 28, 2018 at 8:47 AM Dinesh Joshi 
wrote:

> > On Aug 28, 2018, at 6:33 AM, Marcus Olsson 
> wrote:
> >
> > Hi,
> >
> > With the risk of stirring the repair/side-car topic  even further I'd
> just like to mention that we have recently gotten approval to contribute
> our repair management side-car solution.
> > It's based on the proposal in
> https://issues.apache.org/jira/browse/CASSANDRA-10070 as a standalone
> application sitting next to each instance.
> > With the recent discussions in mind I'd just like to hear the thoughts
> from the community on this before we put in the effort of bringing our
> solution into open source.
> >
> > Would there be an interest of having yet another repair solution in the
> discussion?
>
> I personally think looking at multiple options is important. So yes, it
> would be interesting to see your solution.
>
> Dinesh
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Reaper as cassandra-admin

[Jonathan Haddad]

I don't believe #1 should be an issue, Mick has been reaching out.

Alex and Mick are putting together some architecture documentation, I won't
step on their toes.  Currently you can run Reaper as a single instance that
connects to your entire cluster, multiple instances in HA mode, and we're
finishing up the rework of the code to run it as a sidecar.

On Mon, Aug 27, 2018 at 6:02 PM Jeff Jirsa  wrote:

> Can you get all of the contributors cleared?
> What’s the architecture? Is it centralized? Is there a sidecar?
>
>
> > On Aug 27, 2018, at 5:36 PM, Jonathan Haddad  wrote:
> >
> > Hey folks,
> >
> > Mick brought this up in the sidecar thread, but I wanted to have a clear
> /
> > separate discussion about what we're thinking with regard to contributing
> > Reaper to the C* project.  In my mind, starting with Reaper is a great
> way
> > of having an admin right now, that we know works well at the kind of
> scale
> > we need.  We've worked with a lot of companies putting Reaper in prod (at
> > least 50), running on several hundred clusters.  The codebase has evolved
> > as a direct result of production usage, and we feel it would be great to
> > pair it with the 4.0 release.  There was a LOT of work done on the repair
> > logic to make things work across every supported version of Cassandra,
> with
> > a great deal of documentation as well.
> >
> > In case folks aren't aware, in addition to one off and scheduled repairs,
> > Reaper also does cluster wide snapshots, exposes thread pool stats, and
> > visualizes streaming (in trunk).
> >
> > We're hoping to get some feedback on our side if that's something people
> > are interested in.  We've gone back and forth privately on our own
> > preferences, hopes, dreams, etc, but I feel like a public discussion
> would
> > be healthy at this point.  Does anyone share the view of using Reaper as
> a
> > starting point?  What concerns to people have?
> > --
> > Jon Haddad
> > http://www.rustyrazorblade.com
> > twitter: rustyrazorblade
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Reaper as cassandra-admin

[Jonathan Haddad]

Hey folks,

Mick brought this up in the sidecar thread, but I wanted to have a clear /
separate discussion about what we're thinking with regard to contributing
Reaper to the C* project.  In my mind, starting with Reaper is a great way
of having an admin right now, that we know works well at the kind of scale
we need.  We've worked with a lot of companies putting Reaper in prod (at
least 50), running on several hundred clusters.  The codebase has evolved
as a direct result of production usage, and we feel it would be great to
pair it with the 4.0 release.  There was a LOT of work done on the repair
logic to make things work across every supported version of Cassandra, with
a great deal of documentation as well.

In case folks aren't aware, in addition to one off and scheduled repairs,
Reaper also does cluster wide snapshots, exposes thread pool stats, and
visualizes streaming (in trunk).

We're hoping to get some feedback on our side if that's something people
are interested in.  We've gone back and forth privately on our own
preferences, hopes, dreams, etc, but I feel like a public discussion would
be healthy at this point.  Does anyone share the view of using Reaper as a
starting point?  What concerns to people have?
-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Side Car New Repo vs not

[Jonathan Haddad]

Strongly agree with Blake.  In my mind supporting multiple versions is
mandatory.  As I've stated before, we already do it with Reaper, I'd
consider it a major misstep if we couldn't support multiple with the
project - provided admin tool.  It's the same reason dtests are separate -
they work with multiple versions.

The number of repos does not affect distribution - if we want to ship
Cassandra with the admin / repair tool (we should, imo), that can be part
of the build process.




On Mon, Aug 20, 2018 at 9:21 PM Blake Eggleston 
wrote:

> If the sidecar is going to be on a different release cadence, or support
> interacting with mixed mode clusters, then it should definitely be in a
> separate repo. I don’t even know how branching and merging would work in a
> repo that supports 2 separate release targets and/or mixed mode
> compatibility, but I’m pretty sure it would be a mess.
>
> As a cluster management tool, mixed mode is probably going to be a goal at
> some point. As a new project, it will benefit from not being tied to the C*
> release cycle (which would probably delay any sidecar release until
> whenever 4.1 is cut).
>
>
> On August 20, 2018 at 3:22:54 PM, Joseph Lynch (joe.e.ly...@gmail.com)
> wrote:
>
> I think that the pros of incubating the sidecar in tree as a tool first
> outweigh the alternatives at this point of time. Rough tradeoffs that I
> see:
>
> Unique pros of in tree sidecar:
> * Faster iteration speed in general. For example when we need to add a
> new
> JMX endpoint that the sidecar needs, or change something from JMX to a
> virtual table (e.g. for repair, or monitoring) we can do all changes
> including tests as one commit within the main repository and don't have
> to
> commit to main repo, sidecar repo, and dtest repo (juggling version
> compatibility along the way).
> * We can in the future more easily move serious background functionality
> like compaction or repair itself (not repair scheduling, actual
> repairing)
> into the sidecar with a single atomic commit, we don't have to do two
> phase
> commits where we add some IPC mechanism to allow us to support it in
> both,
> then turn it on in the sidecar, then turn it off in the server, etc...
> * I think that the verification is much easier (sounds like Jonathan
> disagreed on the other thread, I could certainly be wrong), and we don't
> have to worry about testing matrices to assure that the sidecar works
> with
> various versions as the version of the sidecar that is released with that
> version of Cassandra is the only one we have to certify works. If people
> want to pull in new versions or maintain backports they can do that at
> their discretion/testing.
> * We can iterate and prove value before committing to a choice. Since it
> will be a separate artifact from the start we can always move the
> artifact
> to a separate repo later (but moving the other way is harder).
> * Users will get the sidecar "for free" when they install the daemon,
> they
> don't need to take affirmative action to e.g. be able to restart their
> cluster, run repair, or back their data up; it just comes out of the box
> for free.
>
> Unique pros of a separate repository sidecar:
> * We can use a more modern build system like gradle instead of ant
> * Merging changes is less "scary" I guess (I feel like if you're not
> touching the daemon this is already true but I could see this being less
> worrisome for some).
> * Releasing a separate artifact is somewhat easier from a separate repo
> (especially if we have gradle which makes e.g. building debs and rpms
> trivial).
> * We could backport to previous versions without getting into arguments
> about bug fixes vs features.
> * Committers could be different from the main repo, which ... may be a
> useful thing
>
> Non unique pros of a sidecar (could be achieved in the main repo or in a
> separate repo):
> * A separate build artifact .jar/.deb/.rpm that can be installed
> separately. It's slightly easier with a separate repo but certainly not
> out
> of reach within a single repo (indeed the current patch already creates a
> separate jar, and we could create a separate .deb reasonably easily).
> Personally I think having a separate .deb/.rpm is premature at this point
> (for companies that really want it they can build their own packages
> using
> the .jars), but I think it really is a distracting issue from where the
> patch should go as we can always choose to remove experimental .jar files
> that the main daemon doesn't touch.
> * A separate process lifecycle. No matter where the sidecar goes, we get
> the benefit of restarting it being less dangerous for availability than
> restarting the main daemon.
>
> That all being said, these are strong opinions weakly held and I would
> rather get something actually committed so that we can prove value one
> way
> or the other and am therefore, of course, happy to put sidecar patches
> wherever someone can review and commit it.
>
> -Joey
>
> On Mon, Aug 

Re: Proposing an Apache Cassandra Management process

[Jonathan Haddad]

Speaking from experience (Reaper), I can say that developing a sidecar
admin / repair tool out of tree that's compatible with multiple versions
really isn't that big of a problem.  We've been doing it for a while now.

https://github.com/thelastpickle/cassandra-reaper/blob/master/.travis.yml

On Fri, Aug 17, 2018 at 9:39 AM Joseph Lynch  wrote:

> While I would love to use a different build system (e.g. gradle) for the
> sidecar, I agree with Dinesh that a separate repo would make sidecar
> development much harder to verify, especially on the testing and
> compatibility front. As Jeremiah mentioned we can always choose later to
> release the sidecar artifact separately or more frequently than the main
> server regardless of repo choice and as per Roopa's point having a separate
> release artifact (jar or deb/rpm) is probably a good idea until the default
> Cassandra packages don't automatically stop and start Cassandra on install.
>
> While we were developing the repair scheduler in a separate repo we had a
> number of annoying issues that only started surfacing once we started
> merging it directly into the trunk tree:
> 1. It is hard to compile/test against unreleased versions of Cassandra
> (e.g. the JMX interfaces changed a lot with 4.x, and it was pretty tricky
> to compile against that as the main project doesn't release nightly
> snapshots or anything like that, so we had to build local trunk jars and
> then manually dep them).
> 2. Integration testing and cross version compatibility is extremely hard.
> The sidecar is going to be involved in multi node coordination (e.g.
> monitoring, metrics, maintenance) and will be tightly coupled to JMX
> interface choices in the server and trying to make sure that it all works
> with multiple versions of Cassandra is much harder if it's a separate repo
> that has to have a mirroring release cycle to Cassandra. It seems much
> easier to have it in tree and just be like "the in tree sidecar is tested
> against that version of Cassandra". Every time we cut a Cassandra server
> branch the sidecar branches with it.
>
> We experience these pains all the time with Priam being in a separate repo,
> where every time we support a new Cassandra version a bunch of JMX
> endpoints break and we have to refactor the code to either call JMX methods
> by string or cut a new Priam branch. A separate artifact is pretty
> important, but a separate repo just allows drifts. Furthermore from the
> Priam experience I also don't think it's realistic to say one version of a
> sidecar artifact can actually support multiple versions.
>
> -Joey
>
> On Fri, Aug 17, 2018 at 12:00 PM Jeremiah D Jordan 
> wrote:
>
> > Not sure why the two things being in the same repo means they need the
> > same release process.  You can always do interim releases of the
> management
> > artifact between server releases, or even have completely decoupled
> > releases.
> >
> > -Jeremiah
> >
> > > On Aug 17, 2018, at 10:52 AM, Blake Eggleston 
> > wrote:
> > >
> > > I'd be more in favor of making it a separate project, basically for all
> > the reasons listed below. I'm assuming we'd want a management process to
> > work across different versions, which will be more awkward if it's in
> tree.
> > Even if that's not the case, keeping it in a different repo at this point
> > will make iteration easier than if it were in tree. I'd imagine (or at
> > least hope) that validating the management process for release would be
> > less difficult than the main project, so tying them to the Cassandra
> > release cycle seems unnecessarily restrictive.
> > >
> > >
> > > On August 17, 2018 at 12:07:18 AM, Dinesh Joshi (
> dinesh.jo...@yahoo.com.invalid)
> > wrote:
> > >
> > >> On Aug 16, 2018, at 9:27 PM, Sankalp Kohli 
> > wrote:
> > >>
> > >> I am bumping this thread because patch has landed for this with repair
> > functionality.
> > >>
> > >> I have a following proposal for this which I can put in the JIRA or
> doc
> > >>
> > >> 1. We should see if we can keep this in a separate repo like Dtest.
> > >
> > > This would imply a looser coupling between the two. Keeping things
> > in-tree is my preferred approach. It makes testing, dependency management
> > and code sharing easier.
> > >
> > >> 2. It should have its own release process.
> > >
> > > This means now there would be two releases that need to be managed and
> > coordinated.
> > >
> > >> 3. It should have integration tests for different versions of
> Cassandra
> > it will support.
> > >
> > > Given the lack of test infrastructure - this will be hard especially if
> > you have to qualify a matrix of builds.
> > >
> > > Dinesh
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >
> >
> >
> > -
> > To unsubscribe, e-mail: 

Re: upgrade guava on trunk before 9/1?

[Jonathan Haddad]

Pushing it back means it’s a bigger risk later on. I’m +.5 on upgrading now

On Wed, Aug 15, 2018 at 11:46 PM dinesh.jo...@yahoo.com.INVALID
 wrote:

> Jason,
> Given that we're so close to the 9/1 date, I would err on the side of
> caution especially given the low value prop. If someone does run into Guava
> compatibility issues (and someone somewhere will), we can revisit this
> question then.
> Dinesh
>
> On Wednesday, August 15, 2018, 11:42:31 PM PDT, Dinesh A. Joshi <
> dinesh.jo...@gatech.edu> wrote:
>
>  Jason,
> Given that we're so close to the 9/1 date, I would err on the side of
> caution especially given the low value prop. If someone does run into Guava
> compatibility issues (and someone somewhere will), we can revisit this
> question then.
> Dinesh
>
> On Wednesday, August 15, 2018, 8:22:28 AM PDT, Salih Gedik <
> m...@salih.xyz> wrote:
>
>  Hi,
>
> Change logs are on Github releases page. It seems like only hash flooding
> protection which is added to ImmutableMap is relevant to Cassandra code. I
> haven’t checked whether we use deprecated APIs. But there isn’t much on
> table from what I see.
>
> Salih
> On 15 Aug 2018 17:55 +0300, Ariel Weisberg , wrote:
> > Hi,
> >
> > They don't even do release notes after 23. Also no API diffs. I mean I'm
> fine with it, but it's mostly just changing to another arbitrary version
> that won't match what is in apps.
> >
> > Ariel
> >
> > On Wed, Aug 15, 2018, at 10:48 AM, Jason Brown wrote:
> > > Hey Ariel,
> > >
> > > Tbqh, not that much. I was mostly thinking from the "I have conflicts
> on
> > > guava versions in my app because I pull in cassandra and XYZ
> libraries, and
> > > the transitive dependencies on guava use different versions" POV.
> Further,
> > > we'll be on this version of guava for 4.0 for at least two years from
> now.
> > >
> > > As I asked, "does anybody feeling strongly?". Personally, I'm sorta +0
> to
> > > +0.5, but I was just throwing this out there in case someone does
> really
> > > think it best we upgrade (and wants to make a contribution).
> > >
> > > -Jason
> > >
> > >
> > >
> > >
> > > On Wed, Aug 15, 2018 at 7:25 AM, Ariel Weisberg 
> wrote:
> > >
> > > > Hi,
> > > >
> > > > What do we get from Guava in exchange for upgrading?
> > > >
> > > > Ariel
> > > >
> > > > On Wed, Aug 15, 2018, at 10:19 AM, Jason Brown wrote:
> > > > > Hey all,
> > > > >
> > > > > Does anyone feel strongly about upgrading guava on trunk before
> the 9/1
> > > > > feature freeze for 4.0? We are currently at 23.3 (thanks to
> > > > > CASSANDRA-13997), and the current is 26.0.
> > > > >
> > > > > I took a quick look, and there's about 17 compilation errors. They
> fall
> > > > > into two categories, both of which appear not too difficult to
> resolve (I
> > > > > didn't look too closely, tbh).
> > > > >
> > > > > If anyone wants to tackle this LHF I can rustle up some review
> time.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > -Jason
> > > >
> > > > -
> > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > > >
> > > >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

G1GC is now default

[Jonathan Haddad]

I fired up trunk to check something, and noticed this:

INFO  [Service Thread] 2018-08-08 15:01:36,723 GCInspector.java:290 - G1
Young Generation GC in 339ms.  G1 Eden Space: 4634705920 -> 0; G1 Old Gen:
1190138352 -> 1435504616; G1 Survivor Space: 406847488 -> 301989888;

which I thought was a bit weird, since I was using trunk without making any
changes and didn't remember seeing a JIRA where we agreed to make that
change.  I looked back and saw it made it in as a side effect of
CASSANDRA-9608, but wasn't explicitly discussed in the ticket, and there's
no note of it in CHANGES.

I'm personally OK with this change as G1 is a safer bet for anyone who uses
the defaults, but we should be explicit about the change.  Can anyone think
of a reason why we'd need to revert this back to ParNew / CMS?

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: GitHub PR ticket spam

[Jonathan Haddad]

+1 to worklog

On Mon, Aug 6, 2018 at 9:44 AM Ariel Weisberg  wrote:

> Hi,
>
> Great idea. +1 to moving it to the work log.
>
> Thanks,
> Ariel
>
> On Mon, Aug 6, 2018, at 12:40 PM, Aleksey Yeshchenko wrote:
> > Nice indeed. I assume it also doesn’t spam commits@ when done this way,
> > in which case double +1 from me.
> >
> > —
> > AY
> >
> > On 6 August 2018 at 17:18:36, Jeremiah D Jordan
> > (jeremiah.jor...@gmail.com) wrote:
> >
> > Oh nice. I like the idea of keeping it but moving it to the worklog tab.
> > +1 on that from me.
> >
> > > On Aug 6, 2018, at 5:34 AM, Stefan Podkowinski 
> wrote:
> > >
> > > +1 for worklog option
> > >
> > > Here's an example ticket from Arrow, where they seem to be using the
> > > same approach:
> > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_ARROW-2D2583=DwICaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=wYZwHSze6YITTXgzOrEvfr_onojahtjeJRzGAt8ByzM=KWt0xsOv9ESaieg432edGvPhktGkWHxVuLAdNyORiYY=
> <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_ARROW-2D2583=DwICaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=wYZwHSze6YITTXgzOrEvfr_onojahtjeJRzGAt8ByzM=KWt0xsOv9ESaieg432edGvPhktGkWHxVuLAdNyORiYY=>
>
> > >
> > >
> > > On 05.08.2018 09:56, Mick Semb Wever wrote:
> > >>> I find this a bit annoying while subscribed to commits@,
> > >>> especially since we created pr@ for these kind of messages. Also I
> don't
> > >>> really see any value in mirroring all github comments to the
> ticket.
> > >>
> > >>
> > >> I agree with you Stefan. It makes the jira tickets quite painful to
> read. And I tend to make comments on the commits rather than the PRs so to
> avoid spamming back to the jira ticket.
> > >>
> > >> But the linking to the PR is invaluable. And I can see Ariel's point
> about a chronological historical archive.
> > >>
> > >>
> > >>> Ponies would be for this to be mirrored to a tab
> > >>> separate from comments in JIRA.
> > >>
> > >>
> > >> Ariel, that would be the the "worklog" option.
> > >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__reference.apache.org_pmc_github=DwICaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=wYZwHSze6YITTXgzOrEvfr_onojahtjeJRzGAt8ByzM=1lWQawAO9fITzakpnmdzERuCbZs6IGQsUH_EEIMCMqs=
> <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__reference.apache.org_pmc_github=DwICaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=wYZwHSze6YITTXgzOrEvfr_onojahtjeJRzGAt8ByzM=1lWQawAO9fITzakpnmdzERuCbZs6IGQsUH_EEIMCMqs=>
>
> > >>
> > >> If this works for you, and others, I can open a INFRA to switch to
> worklog.
> > >> wdyt?
> > >>
> > >>
> > >> Mick.
> > >>
> > >>
> -
> > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org  dev-unsubscr...@cassandra.apache.org>
> > >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 
> > >>
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org  dev-unsubscr...@cassandra.apache.org>
> > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> 
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: NGCC 2018?

[Jonathan Haddad]

My interpretation of Nate's statement was that since there would be a bunch
of us at Lynn's event, we might as well do NGCC at the same time.

On Thu, Jul 26, 2018 at 9:03 PM Ben Bromhead  wrote:

> It sounds like there may be an appetite for something, but the NGCC in its
> current format is likely to not be that useful?
>
> Is a bay area event focused on C* developers something that is interesting
> for the broader dev community? In whatever format that may be?
>
> On Tue, Jul 24, 2018 at 5:02 PM Nate McCall  wrote:
>
> > This was discussed amongst the PMC recently. We did not come to a
> > conclusion and there were not terribly strong feelings either way.
> >
> > I don't feel like we need to hustle to get "NGCC" in place,
> > particularly given our decided focus on 4.0. However, that should not
> > stop us from doing an additional 'c* developer' event in sept. to
> > coincide with distributed data summit.
> >
> > On Wed, Jul 25, 2018 at 5:03 AM, Patrick McFadin 
> > wrote:
> > > Ben,
> > >
> > > Lynn Bender had offered a space the day before Distributed Data Summit
> in
> > > September (http://distributeddatasummit.com/) since we are both
> platinum
> > > sponsors. I thought he and Nate had talked about that being a good
> place
> > > for NGCC since many of us will be in town already.
> > >
> > > Nate, now that I've spoken for you, you can clarify, :D
> > >
> > > Patrick
> > >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> > --
> Ben Bromhead
> CTO | Instaclustr 
> +1 650 284 9692
> Reliability at Scale
> Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: [VOTE] Release Apache Cassandra 2.2.13

[Jonathan Haddad]

+1

On Wed, Jul 25, 2018 at 11:35 AM Jeff Jirsa  wrote:

> +1
>
> On Wed, Jul 25, 2018 at 12:17 AM, Michael Shuler 
> wrote:
>
> > I propose the following artifacts for release as 2.2.13.
> >
> > sha1: 3482370df5672c9337a16a8a52baba53b70a4fe8
> > Git:
> > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> > shortlog;h=refs/tags/2.2.13-tentative
> > Artifacts:
> > https://repository.apache.org/content/repositories/
> > orgapachecassandra-1167/org/apache/cassandra/apache-cassandra/2.2.13/
> > Staging repository:
> > https://repository.apache.org/content/repositories/
> > orgapachecassandra-1167/
> >
> > The Debian and RPM packages are available here:
> > http://people.apache.org/~mshuler
> >
> > The vote will be open for 72 hours (longer if needed).
> >
> > [1]: CHANGES.txt:
> > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> > blob_plain;f=CHANGES.txt;hb=refs/tags/2.2.13-tentative
> > [2]: NEWS.txt:
> > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> > blob_plain;f=CHANGES.txt;hb=refs/tags/2.2.13-tentative
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: [VOTE] Release Apache Cassandra 3.11.3 (Take 2)

[Jonathan Haddad]

+1

On Wed, Jul 25, 2018 at 11:35 AM Jeff Jirsa  wrote:

> +1
>
> On Wed, Jul 25, 2018 at 12:16 AM, Michael Shuler 
> wrote:
>
> > I propose the following artifacts for release as 3.11.3.
> >
> > sha1: 31d5d870f9f5b56391db46ba6cdf9e0882d8a5c0
> > Git:
> > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> > shortlog;h=refs/tags/3.11.3-tentative
> > Artifacts:
> > https://repository.apache.org/content/repositories/
> > orgapachecassandra-1164/org/apache/cassandra/apache-cassandra/3.11.3/
> > Staging repository:
> > https://repository.apache.org/content/repositories/
> > orgapachecassandra-1164/
> >
> > The Debian and RPM packages are available here:
> > http://people.apache.org/~mshuler
> >
> > The vote will be open for 72 hours (longer if needed).
> >
> > [1]: CHANGES.txt:
> > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> > blob_plain;f=CHANGES.txt;hb=refs/tags/3.11.3-tentative
> > [2]: NEWS.txt:
> > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> > blob_plain;f=CHANGES.txt;hb=refs/tags/3.11.3-tentative
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: [VOTE] Release Apache Cassandra 3.0.17 (Take 2)

[Jonathan Haddad]

+1

On Wed, Jul 25, 2018 at 11:35 AM Jeff Jirsa  wrote:

> +1
>
> On Wed, Jul 25, 2018 at 12:17 AM, Michael Shuler 
> wrote:
>
> > I propose the following artifacts for release as 3.0.17.
> >
> > sha1: d52c7b8c595cc0d06fc3607bf16e3f595f016bb6
> > Git:
> > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> > shortlog;h=refs/tags/3.0.17-tentative
> > Artifacts:
> > https://repository.apache.org/content/repositories/
> > orgapachecassandra-1165/org/apache/cassandra/apache-cassandra/3.0.17/
> > Staging repository:
> > https://repository.apache.org/content/repositories/
> > orgapachecassandra-1165/
> >
> > The Debian and RPM packages are available here:
> > http://people.apache.org/~mshuler
> >
> > The vote will be open for 72 hours (longer if needed).
> >
> > [1]: CHANGES.txt:
> > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> > blob_plain;f=CHANGES.txt;hb=refs/tags/3.0.17-tentative
> > [2]: NEWS.txt:
> > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> > blob_plain;f=CHANGES.txt;hb=refs/tags/3.0.17-tentative
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: [VOTE] Branching Change for 4.0 Freeze

[Jonathan Haddad]

+1,

I've come around on this, I think the long and short term benefits
will be worth it.
On Fri, Jul 13, 2018 at 11:17 AM Vinay Chella
 wrote:
>
> +1 nb
>
> Regards,
> Vinay Chella
>
>
> On Fri, Jul 13, 2018 at 11:15 AM Michael Shuler 
> wrote:
>
> > +0
> >
> > There are pros and cons. I do hope the pros work out and the cons aren't
> > too impactful. I thought about just abstaining, but figured a "meh,
> > whatever" vote was at least worth voicing.
> >
> > Michael
> >
> > On 07/11/2018 04:46 PM, sankalp kohli wrote:
> > > Hi,
> > > As discussed in the thread[1], we are proposing that we will not
> > branch
> > > on 1st September but will only allow following merges into trunk.
> > >
> > > a. Bug and Perf fixes to 4.0.
> > > b. Critical bugs in any version of C*.
> > > c. Testing changes to help test 4.0
> > >
> > > If someone has a change which does not fall under these three, we can
> > > always discuss it and have an exception.
> > >
> > > Vote will be open for 72 hours.
> > >
> > > Thanks,
> > > Sankalp
> > >
> > > [1]
> > >
> > https://lists.apache.org/thread.html/494c3ced9e83ceeb53fa127e44eec6e2588a01b769896b25867fd59f@%3Cdev.cassandra.apache.org%3E
> > >
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >



-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Scratch an itch

[Jonathan Haddad]

>
> Deferring huge amount of commits gives rebase/redo hell. That's the
> biggest impact and the order in which these deferred commits are then
> actually committed can make it more painful or less painful depending on
> the commit. And that in turn will have to then wait for each contributor
> to rebase/redo their commit and those timings might make more rebase
> issues. If those committers will want to rebase something after n-months
> or have time at that point.


This was a concern I had initially as well, but the more I thought about
it, the less concerned I became.  If trunk drifts far away from a 4.0
branch, we would *still* have to deal with the pain of merging everything
forward.  It would just change who is responsible for doing the work.


On Thu, Jul 12, 2018 at 10:54 AM Michael Burman  wrote:

> On 07/12/2018 07:38 PM, Stefan Podkowinski wrote:
> > this point? Also, if we tell someone that their contribution will be
> > reviewed and committed later after 4.0-beta, how is that actually making
> > a difference for that person, compared to committing it now for a 4.x
> > version. It may be satisfying to get a patch committed, but what matters
> > more is when the code will actually be released and deferring committing
> > contributions after 4.0-beta doesn't necessarily mean that there's any
> > disadvantage when it comes to that.
> >
> Deferring huge amount of commits gives rebase/redo hell. That's the
> biggest impact and the order in which these deferred commits are then
> actually committed can make it more painful or less painful depending on
> the commit. And that in turn will have to then wait for each contributor
> to rebase/redo their commit and those timings might make more rebase
> issues. If those committers will want to rebase something after n-months
> or have time at that point.
>
> That's a problem for all Cassandra patches that take huge time to commit
> and if this block takes a lot of time, then that will for sure be even
> more painful. I know products such as Kubernetes does the same (I guess
> that's where this idea might have come from) "trunk patches only", but
> their block is quite short.
>
> My wish is that this freeze does not last too long to kill enthusiasm
> towards committing to Cassandra. There are (I assume) many hobbyist who
> do this as a side-project instead of their daily work and might not have
> the capabilities to test 4.0 in a way that will trigger bugs (easy bugs
> are fixed quite quickly I hope). And if they feel like it's not worth
> the time at this point to invest time to Cassandra (because nothing they
> do will get merged) - they might move to another project. And there's no
> guarantee they will return. Getting stuff to the product is part of the
> satisfaction and without satisfaction there's no interest in continuing.
>
>- Micke
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Testing 4.0 Post-Freeze

[Jonathan Haddad]

I guess I look at the idea of changing the branching strategy as a
means of blocking work as a very odd way of solving a human problem.
Having a majority of votes temporarily block potentially good work
might be a good thing, and it might not matter at all.  It might also
frustrate some folks who have been around for a really long time.

I'm also making the assumption that I don't know every plausible
reason someone might need / want to merge into trunk and considering
that there's valid cases for it that we'll be blocking.

With regard to "the process has been broken for years" I've already
said my bit on what I considered to different now, nobody has
responded to that yet.  I think I've said all I need to on this, it's
probably just noise now, so I won't respond any further on this
thread.  I don't anticipate having a personal need to commit to a
future 5.0 release before 4.0 is out, so it won't impact me
personally.

On Tue, Jul 10, 2018 at 10:32 AM Benedict Elliott Smith
 wrote:
>
> That’s a peculiar way of looking at it.
>
> Committer status is not an absolute right to autonomy over the codebase.  
> It’s an embodiment of trust that you will follow the community's prevailing 
> rules around commit, and that you’re competent to do so.
>
> If the community wants to change those rules you’re trusted to follow, how 
> does this modify your right, or the trust emplaced in you?
>
>
>
>
>
> > On 10 Jul 2018, at 10:18, Jonathan Haddad  wrote:
> >
> > I guess I look at the initial voting in of committers as the process
> > by which people are trusted to merge things in.  This proposed process
> > revokes that trust. If Jonathan Ellis or Dave Brosius (arbitrarily
> > picked) wants to merge a new feature into trunk during the freeze, now
> > they're not allowed?  That's absurd.  People have already met the bar
> > and have been voted in by merit, they should not have their privilege
> > revoked.
> > On Tue, Jul 10, 2018 at 10:14 AM Ben Bromhead  wrote:
> >>
> >> Well put Mick
> >>
> >> +1
> >>
> >> On Tue, Jul 10, 2018 at 1:06 PM Aleksey Yeshchenko 
> >> wrote:
> >>
> >>> +1 from me too.
> >>>
> >>> —
> >>> AY
> >>>
> >>> On 10 July 2018 at 04:17:26, Mick Semb Wever (m...@apache.org) wrote:
> >>>
> >>>
> >>>> We have done all this for previous releases and we know it has not
> >>> worked
> >>>> well. So how giving it one more try is going to help here. Can someone
> >>>> outline what will change for 4.0 which will make it more successful?
> >>>
> >>>
> >>> I (again) agree with you Sankalp :-)
> >>>
> >>> Why not try something new?
> >>> It's easier to discuss these things more genuinely after trying it out.
> >>>
> >>> One of the differences in the branching approaches: to feature-freeze on a
> >>> 4.0 branch or on trunk; is who it is that has to then merge and work with
> >>> multiple branches.
> >>>
> >>> Where that small but additional effort is placed I think becomes a signal
> >>> to what the community values most: new features or stability.
> >>>
> >>> I think most folk would vote for stability, so why not give this approach
> >>> a go and to learn from it.
> >>> It also creates an incentive to make the feature-freeze period as short as
> >>> possible, moving us towards an eventual goal of not needing to
> >>> feature-freeze at all.
> >>>
> >>> regards,
> >>> Mick
> >>>
> >>> -
> >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>>
> >>> --
> >> Ben Bromhead
> >> CTO | Instaclustr <https://www.instaclustr.com/>
> >> +1 650 284 9692
> >> Reliability at Scale
> >> Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
> >
> >
> >
> > --
> > Jon Haddad
> > http://www.rustyrazorblade.com
> > twitter: rustyrazorblade
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Testing 4.0 Post-Freeze

[Jonathan Haddad]

I guess I look at the initial voting in of committers as the process
by which people are trusted to merge things in.  This proposed process
revokes that trust. If Jonathan Ellis or Dave Brosius (arbitrarily
picked) wants to merge a new feature into trunk during the freeze, now
they're not allowed?  That's absurd.  People have already met the bar
and have been voted in by merit, they should not have their privilege
revoked.
On Tue, Jul 10, 2018 at 10:14 AM Ben Bromhead  wrote:
>
> Well put Mick
>
> +1
>
> On Tue, Jul 10, 2018 at 1:06 PM Aleksey Yeshchenko 
> wrote:
>
> > +1 from me too.
> >
> > —
> > AY
> >
> > On 10 July 2018 at 04:17:26, Mick Semb Wever (m...@apache.org) wrote:
> >
> >
> > > We have done all this for previous releases and we know it has not
> > worked
> > > well. So how giving it one more try is going to help here. Can someone
> > > outline what will change for 4.0 which will make it more successful?
> >
> >
> > I (again) agree with you Sankalp :-)
> >
> > Why not try something new?
> > It's easier to discuss these things more genuinely after trying it out.
> >
> > One of the differences in the branching approaches: to feature-freeze on a
> > 4.0 branch or on trunk; is who it is that has to then merge and work with
> > multiple branches.
> >
> > Where that small but additional effort is placed I think becomes a signal
> > to what the community values most: new features or stability.
> >
> > I think most folk would vote for stability, so why not give this approach
> > a go and to learn from it.
> > It also creates an incentive to make the feature-freeze period as short as
> > possible, moving us towards an eventual goal of not needing to
> > feature-freeze at all.
> >
> > regards,
> > Mick
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> > --
> Ben Bromhead
> CTO | Instaclustr 
> +1 650 284 9692
> Reliability at Scale
> Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer



-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Testing 4.0 Post-Freeze

[Jonathan Haddad]

The most impactful change that I can see is the people that are
involved with development who are willing to -1 a release.  Those of
us with votes are TLP have a big incentive for 4.0 to be stable - we
want people to upgrade.  I assume folks at Netflix, Apple are also
very incentivized to see a very stable 4.0. That's a lot of committer
head count already.

I don't remember much community call to action in the past with regard
to getting folks testing releases with real workloads.  If we want
help, it's on us to ask.  Making it as easy as possible to test stuff
out and get feedback is also on us.  Banning folks from committing to
trunk isn't solving the main problem: it's still pretty difficult to
get involved.  We need to lower the barrier to entry for setting &
tearing down test clusters.  We also need to recruit community members
to be part of a QA feedback cycle.  Once folks are in, keeping them
involved for multiple iterations will improve their ability to give
feedback.

The nice part is, if we do it right, eventually maybe those folks will
commit some code and become committers down the line.

On Mon, Jul 9, 2018 at 11:31 AM sankalp kohli  wrote:
>
> We have done all this for previous releases and we know it has not worked
> well. So how giving it one more try is going to help here. Can someone
> outline what will change for 4.0 which will make it more successful?
>
> Not branching is one idea but we should discuss what will change now than
> iterating what has already happened.
>
> On Mon, Jul 9, 2018 at 11:25 AM Jeremy Hanna 
> wrote:
>
> > I wholeheartedly agree with efforts to make releases more stable and the
> > more contributors that add tests or test within the context of their own
> > use cases, the better.  +1 non-binding to the idea of better, more complete
> > testing for releases.
> >
> > When it comes to how to do this, I think that’s where there is
> > disagreement.  I personally don’t think that branching detracts from the
> > goal of stabilization.  There are a couple of personas involved here:
> >
> > * Longtime contributors/committers: I think it’s sufficient to generally
> > ask that whatever time/effort they can contribute be towards stabilizing,
> > testing, and especially testing their production scenarios to cover more
> > surface area when it comes to usage edge cases.  That along with testing
> > longer running things in those scenarios for other types of edge cases.
> >
> > * New contributors: They likely won’t know about the strategy.  They’re
> > just poking around and looking at lhf tickets or tickets that they need
> > done.  Those are typically low risk but at the same time we don’t want to
> > compromise stability by merging those in.  Reviewing low risk stuff for
> > inclusion doesn’t take a ton of time and gives them a sense that they can
> > be of help and empowers them.  After they have that first experience, then
> > a longer term contributor could share with them a blog post or tldr thread
> > summary about the 4.0 stabilization efforts (would be nice to have one to
> > point people too once we're done).  We could also start creating lhf
> > tickets with stabilization and testing in mind.
> >
> > Unless we want to make a fundamental change to the project’s process
> > (making trunk stable at all times going forward), then I don’t think
> > there’s a reason why branching would detract from these efforts.  In other
> > words if we’re just trying to say that we want to focus on stabilization
> > for the alpha/beta/rc of 4.0, then I would prefer branching along with
> > clear messaging.
> >
> > Jeremy
> >
> > > On Jul 9, 2018, at 12:43 PM, sankalp kohli 
> > wrote:
> > >
> > > How is this preventing someone from working and collaborating on an
> > Apache
> > > Project? You can still collaborate on making 4.0 more stable.
> > >
> > > Why will these committers not work on making 4.0 more stable and fix
> > broken
> > > tests? Considering  we cannot release without test passing, how will
> > these
> > > features be used?
> > >
> > > On Mon, Jul 9, 2018 at 10:03 AM Jonathan Haddad 
> > wrote:
> > >
> > >> I don't see how changing the process and banning feature commits is
> > >> going to be any help to the project. There may be a couple committers
> > >> who are interested in committing new features.
> > >>
> > >> I'm a -1 on changing the branching strategy in a way that prevents
> > >> people from working and collaborating on an Apache project.
> > >>
> > >> On Mon, Jul 9, 2018 at 9:56 AM sankalp ko

Re: Testing 4.0 Post-Freeze

[Jonathan Haddad]

My proposal is that we implement everything you're suggesting, except
we branch off as we have in the past rather than locking down trunk.
I'd encourage everyone to work to improve 4.0 in some way or another,
whether it be fixing bugs, testing in a staging environment
(feedback), writing tooling (data loaders, stress testing, cluster
deployment tools), improving unit / dtests tests and writing docs.
But outright blocking folks from committing a feature they may have
been working on for months makes me very uncomfortable.  I don't think
there's going to be much of this, but it seems a little authoritarian
to me to mandate that it's not allowed.

My personal preference is that everyone commit to making 4.0 stable on
day 1, and we work towards that goal as a community, but not in a
manner that's so heavy handed.

On Mon, Jul 9, 2018 at 11:06 AM sankalp kohli  wrote:
>
> If some committers want to bring in features without a path forward for
> releasing(as tests are broken), is that an acceptable state for you? How do
> we get out of this state.
>
> Like I said in my original email, this is a "proposal" and I am trying to
> see how we can improve things here. So if you are -1 on this, please help
> us propose something else to get these tests pass?
>
> And thanks for giving your feedback.
>
> On Mon, Jul 9, 2018 at 10:50 AM Jonathan Haddad  wrote:
>
> > Not everyone wants to work and collaborate the way you do.  It seems
> > absurd to me to force everyone to hold off on merging into a single
> > branch just because a lot of us want to prioritize testing 4.0.
> >
> > I think most committers are going to want to contribute to the 4.0
> > testing process more than they want to commit new features to trunk,
> > but I also think people shouldn't be banned from new bringing in new
> > features if that's what they want to do.
> >
> > You're essentially giving a blanket -1 to all new features for a 3-6
> > month period.
> > On Mon, Jul 9, 2018 at 10:44 AM sankalp kohli 
> > wrote:
> > >
> > > How is this preventing someone from working and collaborating on an
> > Apache
> > > Project? You can still collaborate on making 4.0 more stable.
> > >
> > > Why will these committers not work on making 4.0 more stable and fix
> > broken
> > > tests? Considering  we cannot release without test passing, how will
> > these
> > > features be used?
> > >
> > > On Mon, Jul 9, 2018 at 10:03 AM Jonathan Haddad 
> > wrote:
> > >
> > > > I don't see how changing the process and banning feature commits is
> > > > going to be any help to the project. There may be a couple committers
> > > > who are interested in committing new features.
> > > >
> > > > I'm a -1 on changing the branching strategy in a way that prevents
> > > > people from working and collaborating on an Apache project.
> > > >
> > > > On Mon, Jul 9, 2018 at 9:56 AM sankalp kohli 
> > > > wrote:
> > > > >
> > > > > I did not see a -1 but all +0s and a few +1s.
> > > > >
> > > > > On Mon, Jul 9, 2018 at 5:49 AM Josh McKenzie 
> > > > wrote:
> > > > >
> > > > > > What did we land on? Sounds like we're pretty split without
> > consensus
> > > > on
> > > > > > "change the old branching strategy and reject new things on trunk
> > > > during
> > > > > > stabilization" vs. "keep doing things the way we did but message
> > > > strongly
> > > > > > that we won't be reviewing new things until 4.0 is stable".
> > > > > >
> > > > > > On Sat, Jul 7, 2018 at 5:01 PM Sankalp Kohli <
> > kohlisank...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Does anyone has any more feedback on this?
> > > > > > >
> > > > > > > > On Jul 4, 2018, at 05:36, Aleksey Yeshchenko <
> > alek...@apple.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > For what it’s worth, I’m fine with both formal branching-level
> > > > freeze
> > > > > > > and informal ‘let people commit to trunk if they can find a
> > > > reviewer’ one
> > > > > > > and will support either.
> > > > > > > >
> > > > > > > > So long as either/both are communicated to the contributors,
> > the
> > > > only
&g

Re: Testing 4.0 Post-Freeze

[Jonathan Haddad]

Not everyone wants to work and collaborate the way you do.  It seems
absurd to me to force everyone to hold off on merging into a single
branch just because a lot of us want to prioritize testing 4.0.

I think most committers are going to want to contribute to the 4.0
testing process more than they want to commit new features to trunk,
but I also think people shouldn't be banned from new bringing in new
features if that's what they want to do.

You're essentially giving a blanket -1 to all new features for a 3-6
month period.
On Mon, Jul 9, 2018 at 10:44 AM sankalp kohli  wrote:
>
> How is this preventing someone from working and collaborating on an Apache
> Project? You can still collaborate on making 4.0 more stable.
>
> Why will these committers not work on making 4.0 more stable and fix broken
> tests? Considering  we cannot release without test passing, how will these
> features be used?
>
> On Mon, Jul 9, 2018 at 10:03 AM Jonathan Haddad  wrote:
>
> > I don't see how changing the process and banning feature commits is
> > going to be any help to the project. There may be a couple committers
> > who are interested in committing new features.
> >
> > I'm a -1 on changing the branching strategy in a way that prevents
> > people from working and collaborating on an Apache project.
> >
> > On Mon, Jul 9, 2018 at 9:56 AM sankalp kohli 
> > wrote:
> > >
> > > I did not see a -1 but all +0s and a few +1s.
> > >
> > > On Mon, Jul 9, 2018 at 5:49 AM Josh McKenzie 
> > wrote:
> > >
> > > > What did we land on? Sounds like we're pretty split without consensus
> > on
> > > > "change the old branching strategy and reject new things on trunk
> > during
> > > > stabilization" vs. "keep doing things the way we did but message
> > strongly
> > > > that we won't be reviewing new things until 4.0 is stable".
> > > >
> > > > On Sat, Jul 7, 2018 at 5:01 PM Sankalp Kohli 
> > > > wrote:
> > > >
> > > > > Does anyone has any more feedback on this?
> > > > >
> > > > > > On Jul 4, 2018, at 05:36, Aleksey Yeshchenko 
> > > > wrote:
> > > > > >
> > > > > > For what it’s worth, I’m fine with both formal branching-level
> > freeze
> > > > > and informal ‘let people commit to trunk if they can find a
> > reviewer’ one
> > > > > and will support either.
> > > > > >
> > > > > > So long as either/both are communicated to the contributors, the
> > only
> > > > > difference is in where new feature work gets accumulated: will stay
> > a bit
> > > > > longer in personal branches in the latter way.
> > > > > >
> > > > > > —
> > > > > > AY
> > > > > >
> > > > > > On 4 July 2018 at 01:30:40, sankalp kohli (kohlisank...@gmail.com)
> > > > > wrote:
> > > > > >
> > > > > > Having an explicit way to tell the community that we all will work
> > on
> > > > > > testing is better than writing a patch which will sit without
> > review
> > > > > for
> > > > > > months. I think not having your patch reviewed for months is more
> > > > > > discouraging than following the community and helping with
> > stability of
> > > > > > 4.0.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Jul 3, 2018 at 3:02 PM Josh McKenzie  > >
> > > > > wrote:
> > > > > >
> > > > > >>>
> > > > > >>> We propose that between the September freeze date and beta, a new
> > > > > branch
> > > > > >>> would not be created and trunk would only have bug fixes and
> > > > > performance
> > > > > >>> improvements committed to it.
> > > > > >>
> > > > > >>
> > > > > >> This is more of a call to action and announcement of intent than
> > an
> > > > > attempt
> > > > > >>> to
> > > > > >>> enforce policy; we can and probably will branch off 4.0, and keep
> > > > > trunk
> > > > > >>> technically active.
> > > > > >>
> > > > > >> These are two very different statements. :) Which is it?
> > > > > >>

Re: Testing 4.0 Post-Freeze

[Jonathan Haddad]

I don't see how changing the process and banning feature commits is
going to be any help to the project. There may be a couple committers
who are interested in committing new features.

I'm a -1 on changing the branching strategy in a way that prevents
people from working and collaborating on an Apache project.

On Mon, Jul 9, 2018 at 9:56 AM sankalp kohli  wrote:
>
> I did not see a -1 but all +0s and a few +1s.
>
> On Mon, Jul 9, 2018 at 5:49 AM Josh McKenzie  wrote:
>
> > What did we land on? Sounds like we're pretty split without consensus on
> > "change the old branching strategy and reject new things on trunk during
> > stabilization" vs. "keep doing things the way we did but message strongly
> > that we won't be reviewing new things until 4.0 is stable".
> >
> > On Sat, Jul 7, 2018 at 5:01 PM Sankalp Kohli 
> > wrote:
> >
> > > Does anyone has any more feedback on this?
> > >
> > > > On Jul 4, 2018, at 05:36, Aleksey Yeshchenko 
> > wrote:
> > > >
> > > > For what it’s worth, I’m fine with both formal branching-level freeze
> > > and informal ‘let people commit to trunk if they can find a reviewer’ one
> > > and will support either.
> > > >
> > > > So long as either/both are communicated to the contributors, the only
> > > difference is in where new feature work gets accumulated: will stay a bit
> > > longer in personal branches in the latter way.
> > > >
> > > > —
> > > > AY
> > > >
> > > > On 4 July 2018 at 01:30:40, sankalp kohli (kohlisank...@gmail.com)
> > > wrote:
> > > >
> > > > Having an explicit way to tell the community that we all will work on
> > > > testing is better than writing a patch which will sit without review
> > > for
> > > > months. I think not having your patch reviewed for months is more
> > > > discouraging than following the community and helping with stability of
> > > > 4.0.
> > > >
> > > >
> > > >
> > > > On Tue, Jul 3, 2018 at 3:02 PM Josh McKenzie 
> > > wrote:
> > > >
> > > >>>
> > > >>> We propose that between the September freeze date and beta, a new
> > > branch
> > > >>> would not be created and trunk would only have bug fixes and
> > > performance
> > > >>> improvements committed to it.
> > > >>
> > > >>
> > > >> This is more of a call to action and announcement of intent than an
> > > attempt
> > > >>> to
> > > >>> enforce policy; we can and probably will branch off 4.0, and keep
> > > trunk
> > > >>> technically active.
> > > >>
> > > >> These are two very different statements. :) Which is it?
> > > >>
> > > >> On Tue, Jul 3, 2018 at 5:57 PM Aleksey Yeshchenko 
> > > >> wrote:
> > > >>
> > > >>> If we want to have a stable, usable 4.0.0 release out in the next
> > > 6-12
> > > >>> months, there needs to be a focused effort on getting it out - or
> > > else
> > > >>> it’ll just never happen.
> > > >>>
> > > >>> This is more of a call to action and announcement of intent than an
> > > >>> attempt to enforce policy; we can and probably will branch off 4.0,
> > > and
> > > >>> keep trunk technically active. But so long as there is a critical
> > mass
> > > of
> > > >>> active contributors who are on board with only/mostly working on
> > > >> stability,
> > > >>> bug fixes, and validation - both as assignees and reviewers - we’ll
> > > have
> > > >> a
> > > >>> de-facto freeze.
> > > >>>
> > > >>> And I have a feeling that there is such a critical mass.
> > > >>>
> > > >>> —
> > > >>> AY
> > > >>>
> > > >>> On 3 July 2018 at 22:23:38, Jeff Jirsa (jji...@gmail.com) wrote:
> > > >>>
> > > >>> I think there's value in the psychological commitment that if someone
> > > has
> > > >>> time to contribute, their contributions should be focused on
> > > validating a
> > > >>> release, not pushing future features.
> > > >>>
> > > >>>
&

Re: Testing 4.0 Post-Freeze

[Jonathan Haddad]

I agree with Josh. I don’t see how changing the convention around trunk
will improve the process, seems like it’ll only introduce a handful of
rollback commits when people forget.

Other than that, it all makes sense to me.

I’ve been working on a workload centric stress tool on and off for a little
bit in an effort to create something that will help with wider adoption in
stress testing. It differs from the stress we ship by including fully
functional stress workloads as well as a validation process. The idea being
to be flexible enough to test both performance and correctness in LWT and
MVs as well as other arbitrary workloads.

https://github.com/thelastpickle/tlp-stress

Jon


On Tue, Jul 3, 2018 at 12:28 PM Josh McKenzie  wrote:

> Why not just branch a 4.0-rel and bugfix there and merge up while still
> accepting new features or improvements on trunk?
>
> I don't think the potential extra engagement in testing will balance out
> the atrophy and discouraging contributions / community engagement we'd get
> by deferring all improvements and new features in an open-ended way.
>
> On Tue, Jul 3, 2018 at 1:33 PM sankalp kohli 
> wrote:
>
> > Hi cassandra-dev@,
> >
> > With the goal of making Cassandra's 4.0 the most stable major release to
> > date, we would like all committers of the project to consider joining us
> in
> > dedicating their time and attention to testing, running, and fixing
> issues
> > in 4.0 between the September freeze and the 4.0 beta release. This would
> > result in a freeze of new feature development on trunk or branches during
> > this period, instead focusing on writing, improving, and running tests or
> > fixing and reviewing bugs or performance regressions found in 4.0 or
> > earlier.
> >
> > How would this work?
> >
> > We propose that between the September freeze date and beta, a new branch
> > would not be created and trunk would only have bug fixes and performance
> > improvements committed to it. At the same time we do not want to
> discourage
> > community contributions. Not all contributors can be expected to be aware
> > of such a decision or may be new to the project. In cases where new
> > features are contributed during this time, the contributor can be
> informed
> > of the current status of the release process, be encouraged to contribute
> > to testing or bug fixing, and have their feature reviewed after the beta
> is
> > reached.
> >
> >
> > What happens when beta is reached?
> >
> > Ideally, contributors who have made significant contributions to the
> > release will stick around to continue testing between beta and final
> > release. Any additional folks who continue this focus would also be
> greatly
> > appreciated.
> >
> > What about before the freeze?
> >
> > Testing new features is of course important. This isn't meant to
> discourage
> > development – only to enable us to focus on testing and hardening 4.0 to
> > deliver Cassandra's most stable major release. We would like to see
> > adoption of 4.0 happen much more quickly than its predecessor.
> >
> > Thanks for considering this proposal,
> > Sankalp Kohli
>
-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Planning to port cqlsh to Python 3 (CASSANDRA-10190)

[Jonathan Haddad]

And that's why I said supporting both with six is the right path
forward, later dropping support for 2.  I'm not advocating we drop 2
support now, and I'm not asking for any sort of commitment.  I didn't
think adding support for 3 would be so controversial.
On Fri, Jun 1, 2018 at 9:40 AM Jeremiah D Jordan
 wrote:
>
> The community of people doing python development and the community of people 
> running Cassandra servers are not the same.  I am not fine riding the coat 
> tails of libraries used in python development.  As others have stated we need 
> to be following the lead of the OS vendors that people will be deploying 
> Cassandra on top of.  And those will not be dropping Python 2 at the end of 
> the year.
>
> -Jeremiah
>
> > On Jun 1, 2018, at 12:37 PM, Jonathan Haddad  wrote:
> >
> > Both can work.  I did a lot of the work on the port of the Python
> > driver's object mapper (formerly cqlengine) to Python 3.  It's
> > reasonably straightforward if you use the six library.
> >
> > Both pandas and numpy are dropping support for Python 2 at the end of
> > this year.  I'm fine with riding on their coattails.
> > On Fri, Jun 1, 2018 at 9:21 AM Russell Bateman  
> > wrote:
> >>
> >> Support for, but not the very script, right? Because, as gently pointed
> >> out by several realists here, Python 2 is far from dead and arguably
> >> still the majority usage. That's only just now beginning to change. I
> >> think it will be more than 2 years before people begin asking what
> >> Python 2 was.
> >>
> >>
> >> On 06/01/2018 10:10 AM, Jonathan Haddad wrote:
> >>> Supporting both as a next step is logical, removing support for 2 in the
> >>> next year or two seems reasonable enough. Gotta rip the band aid off at
> >>> some point.
> >>>
> >>> On Fri, Jun 1, 2018 at 2:34 AM Michael Burman  wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> Deprecating in this context does not mean removing it or it being
> >>>> replaced by 3 (RHEL 7.x will remain with Python 2.x as default). It
> >>>> refers to future versions (>7), but there are none at this point. It
> >>>> appears Ubuntu has deviated from Debian in this sense, but Debian has
> >>>> not changed yet (likely Debian 10 will, but that's not out yet and has
> >>>> no announced release date).
> >>>>
> >>>> Thus, 2.x still remains the most used version for servers. And servers
> >>>> deployed at this point of time will use these versions for years.
> >>>>
> >>>>- Micke
> >>>>
> >>>>
> >>>> On 06/01/2018 10:52 AM, Murukesh Mohanan wrote:
> >>>>> On 2018/06/01 07:40:04, Michael Burman  wrote:
> >>>>>> IIRC, there's no major distribution yet that defaults to Python 3 (I
> >>>>>> think Ubuntu & Debian are still defaulting to Python 2 also). This will
> >>>>>> happen eventually (maybe), but not yet. Discarding Python 2 support
> >>>>>> would mean more base-OS work for most people wanting to run Cassandra
> >>>>>> and that's not a positive thing.
> >>>>>>
> >>>>> Ubuntu since 16.04 defaults to Python 3:
> >>>>>
> >>>>>> Python2 is not installed anymore by default on the server, cloud and
> >>>> the touch images, long live Python3! Python3 itself has been upgraded to
> >>>> the 3.5 series. -
> >>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.ubuntu.com_XenialXerus_ReleaseNotes-23Python-5F3=DwIBaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=J5Su6wvm91QrOBcici7HyIiFiyzjrg8UnamYu8qtSRA=9OWAbO26grwiI2ly_-gAGBqJP9Mv6KPAKJyQu_OEDPc=
> >>>>> RHEL 7.5 deprecates Python 2 (
> >>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__access.redhat.com_documentation_en-2Dus_red-5Fhat-5Fenterprise-5Flinux_7_html_7.5-5Frelease-5Fnotes_chap-2Dred-5Fhat-5Fenterprise-5Flinux-2D7.5-5Frelease-5Fnotes-2Ddeprecated-5Ffunctionality=DwIBaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=J5Su6wvm91QrOBcici7HyIiFiyzjrg8UnamYu8qtSRA=CDFufWbcvq6VpoLJQVbCQP9rpvIv3ssNtKMQce-1vwU=
> >>>> ).
> >>>>>
> >>>>>
> >>>>> -
> >>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>>>> For additional commands, e-mail: dev-h...

Re: Planning to port cqlsh to Python 3 (CASSANDRA-10190)

[Jonathan Haddad]

Both can work.  I did a lot of the work on the port of the Python
driver's object mapper (formerly cqlengine) to Python 3.  It's
reasonably straightforward if you use the six library.

Both pandas and numpy are dropping support for Python 2 at the end of
this year.  I'm fine with riding on their coattails.
On Fri, Jun 1, 2018 at 9:21 AM Russell Bateman  wrote:
>
> Support for, but not the very script, right? Because, as gently pointed
> out by several realists here, Python 2 is far from dead and arguably
> still the majority usage. That's only just now beginning to change. I
> think it will be more than 2 years before people begin asking what
> Python 2 was.
>
>
> On 06/01/2018 10:10 AM, Jonathan Haddad wrote:
> > Supporting both as a next step is logical, removing support for 2 in the
> > next year or two seems reasonable enough. Gotta rip the band aid off at
> > some point.
> >
> > On Fri, Jun 1, 2018 at 2:34 AM Michael Burman  wrote:
> >
> >> Hi,
> >>
> >> Deprecating in this context does not mean removing it or it being
> >> replaced by 3 (RHEL 7.x will remain with Python 2.x as default). It
> >> refers to future versions (>7), but there are none at this point. It
> >> appears Ubuntu has deviated from Debian in this sense, but Debian has
> >> not changed yet (likely Debian 10 will, but that's not out yet and has
> >> no announced release date).
> >>
> >> Thus, 2.x still remains the most used version for servers. And servers
> >> deployed at this point of time will use these versions for years.
> >>
> >> - Micke
> >>
> >>
> >> On 06/01/2018 10:52 AM, Murukesh Mohanan wrote:
> >>> On 2018/06/01 07:40:04, Michael Burman  wrote:
> >>>> IIRC, there's no major distribution yet that defaults to Python 3 (I
> >>>> think Ubuntu & Debian are still defaulting to Python 2 also). This will
> >>>> happen eventually (maybe), but not yet. Discarding Python 2 support
> >>>> would mean more base-OS work for most people wanting to run Cassandra
> >>>> and that's not a positive thing.
> >>>>
> >>> Ubuntu since 16.04 defaults to Python 3:
> >>>
> >>>> Python2 is not installed anymore by default on the server, cloud and
> >> the touch images, long live Python3! Python3 itself has been upgraded to
> >> the 3.5 series. -
> >> https://wiki.ubuntu.com/XenialXerus/ReleaseNotes#Python_3
> >>> RHEL 7.5 deprecates Python 2 (
> >> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.5_release_notes/chap-red_hat_enterprise_linux-7.5_release_notes-deprecated_functionality
> >> ).
> >>>
> >>>
> >>> -
> >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>>
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>
> >> --
> > Jon Haddad
> > http://www.rustyrazorblade.com
> > twitter: rustyrazorblade
> >
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Planning to port cqlsh to Python 3 (CASSANDRA-10190)

[Jonathan Haddad]

Supporting both as a next step is logical, removing support for 2 in the
next year or two seems reasonable enough. Gotta rip the band aid off at
some point.

On Fri, Jun 1, 2018 at 2:34 AM Michael Burman  wrote:

> Hi,
>
> Deprecating in this context does not mean removing it or it being
> replaced by 3 (RHEL 7.x will remain with Python 2.x as default). It
> refers to future versions (>7), but there are none at this point. It
> appears Ubuntu has deviated from Debian in this sense, but Debian has
> not changed yet (likely Debian 10 will, but that's not out yet and has
> no announced release date).
>
> Thus, 2.x still remains the most used version for servers. And servers
> deployed at this point of time will use these versions for years.
>
>- Micke
>
>
> On 06/01/2018 10:52 AM, Murukesh Mohanan wrote:
> > On 2018/06/01 07:40:04, Michael Burman  wrote:
> >> IIRC, there's no major distribution yet that defaults to Python 3 (I
> >> think Ubuntu & Debian are still defaulting to Python 2 also). This will
> >> happen eventually (maybe), but not yet. Discarding Python 2 support
> >> would mean more base-OS work for most people wanting to run Cassandra
> >> and that's not a positive thing.
> >>
> > Ubuntu since 16.04 defaults to Python 3:
> >
> >> Python2 is not installed anymore by default on the server, cloud and
> the touch images, long live Python3! Python3 itself has been upgraded to
> the 3.5 series. -
> https://wiki.ubuntu.com/XenialXerus/ReleaseNotes#Python_3
> > RHEL 7.5 deprecates Python 2 (
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.5_release_notes/chap-red_hat_enterprise_linux-7.5_release_notes-deprecated_functionality
> ).
> >
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
> --
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: [DISCUSS] Cassandra and future Java

[Jonathan Haddad]

Personally I don’t mind dropping support for previsous java versions.

On Fri, May 25, 2018 at 6:33 AM J. D. Jordan 
wrote:

> +1 for “Option 3: both 8 + 11” it shouldn’t be too hard to maintain code
> wise, and leaves people’s options open.
>
> -Jeremiah
>
> > On May 25, 2018, at 6:31 AM, Robert Stupp  wrote:
> >
> > I'd like to bring up the C*/Java discussion again. It's been a while
> since we've discussed this.
> >
> > To me it sounds like there's still the question about which version(s)
> of Java we want to support beginning with C* 4.0.
> >
> > I assume, that it's legit (and probably very necessary) to assume that
> OpenJDK is now (i.e. after Java 6) considered as "production ready" for C*.
> The public (and legal and free) availability of Oracle's Java 8 will end in
> January 2019 (unless you're using it privately on your desktop). Java 9 and
> 10 are not a thing, as both will be EOL when the C* 4.0 branch is about to
> be cut. The most recent available Java version will be 11, which is meant
> to be publicly available from Oracle until March 2019 and should get LTS
> support for OpenJDK 11 from major Linux distros (RHEL and derivates,
> Ubuntu, Azul Zulu).
> >
> > (Side note: adoptopenjdk is different here, because it does not include
> the patch version in the version banner (java.version=1.8.0-adoptopenjdk),
> so difficult to check the minimum patch version on startup of C*.)
> >
> > (Attn, rant: I'm not particularly happy with the new release and support
> model for Java, because developing something now, that's about to release
> end of the year on a Java version that has not even reached
> feature-complete status, is, gently speaking, difficult. But sticking to an
> "antique" Java version (8) has its own risks as well.)
> >
> > I'm silently ignoring any Java release, that's not aimed to get any
> LTS(-ish?) support from anybody - so only Java 8 + 11 remain.
> >
> > There are generally three (IMO legit) options here: only support Java 8,
> only support Java 11, support both Java 8 and Java 11. All three options
> have a bunch of pros and cons.
> >
> > Option 1, only Java 8: Probably the safest option. Considering the
> potential lifetimes of Java 8 and C* 4.0, even the most enthusiastic
> maintainers may stop backporting security or bug fixes to OpenJDK 8. It
> might not be an issue in practice, but if there's for example a severe
> issue in the SSL/TLS area and nobody fixes it in 8, well, good luck.
> >
> > Option 2, only Java 11: The option with the most risks IMO. Java 11 is
> not even feature complete, and there a bunch of big projects that still may
> make it into 11 (think: Valhalla). There's no guarantee whether the C* code
> or any included library will actually work with Java 11 (think: if it works
> now, it may not work with the final Java version). However, it leaves the
> door wide open for all the neat and geeky things in Java 11.
> >
> > Option 3: both 8 + 11: The idea here is to default to Java 8, but the
> code also runs on 11. It leaves the option to benefit from optimizations
> that are only available on 11 while maintaining the known stability of 8.
> Initially, only the combination of C* 4.0 + Java 8 would be labeled as
> "stable" and the combination of C* 4.0 + Java 11 as "experimental". But it
> gives us time to "evaluate" 4.0 on 11. When we have enough experience with
> 11, C* on 11 can be labeled as "stable" as well. The downside of this
> "hybrid" is, that it's a bit more difficult to introduce features, that
> depend on 11.
> >
> > I think, 3) gives the best of both worlds: stability of 8 and an upgrade
> path to 11 in the future, that people can actually test with C* 4.0. Happy
> to make the patch for #9608 ready for option 3. But it would be great to
> get a consensus here for either option before we review #9608 and commit it.
> >
> > Another proposal, for both options 1+3: Raise the minimum supported
> version of 8 for C* 4.0 to something more recent than 8u40, which is quite
> from the stone-age. It could be 8u171 or whatever will be recent in autumn.
> >
> > Robert
> >
> > --
> > Robert Stupp
> > @snazy
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
> --
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Evolving the client protocol

[Jonathan Haddad]

DataStax invested millions of dollars into Cassandra, tens of thousands of
man hours, hosted hundreds of events and has been a major factor in the
success of the project.

ScyllaDB wants us to change the C* protocol in order to improve features in
a competing database which contributes nothing back to the Cassandra
community.

Seems a little different to me.

On Tue, Apr 24, 2018 at 8:30 AM Eric Stevens  wrote:

> Let met just say that as an observer to this conversation -- and someone
> who believes that compatibility, extensibility, and frankly competition
> bring out the best in products -- I'm fairly surprised and disappointed
> with the apparent hostility many community members have shown toward a
> sincere attempt by another open source product to find common ground here.
>
> Yes, Scylla has a competing OSS project (albeit under a different
> license).  They also have a business built around it.  It's hard for me to
> see that as dramatically different than the DataStax relationship to this
> community.  Though I would love to be shown why.
>

Re: Evolving the client protocol

[Jonathan Haddad]

>From where I stand it looks like you've got only two options for any
feature that involves updating the protocol:

1. Don't built the feature
2. Built it in Cassanda & scylladb, update the drivers accordingly

I don't think you have a third option, which is built it only in ScyllaDB,
because that means you have to fork *all* the drivers and make it work,
then maintain them.  Your business model appears to be built on not doing
any of the driver work yourself, and you certainly aren't giving back to
the open source community via a permissive license on ScyllaDB itself, so
I'm a bit lost here.

To me it looks like you're asking a bunch of volunteers that work on
Cassandra to accommodate you.  What exactly do we get out of this
relationship?  What incentive do I or anyone else have to spend time
helping you instead of working on something that interests me?

Jon


On Mon, Apr 23, 2018 at 7:59 AM Ben Bromhead  wrote:

> > >> This doesn't work without additional changes, for RF>1. The token ring
> > could place two replicas of the same token range on the same physical
> > server, even though those are two separate cores of the same server. You
> > could add another element to the hierarchy (cluster -> datacenter -> rack
> > -> node -> core/shard), but that generates unneeded range movements when
> a
> > node is added.
> > > I have seen rack awareness used/abused to solve this.
> > >
> >
> > But then you lose real rack awareness. It's fine for a quick hack, but
> > not a long-term solution.
> >
> > (it also creates a lot more tokens, something nobody needs)
> >
>
> I'm having trouble understanding how you loose "real" rack awareness, as
> these shards are in the same rack anyway, because the address and port are
> on the same server in the same rack. So it behaves as expected. Could you
> explain a situation where the shards on a single server would be in
> different racks (or fault domains)?
>
> If you wanted to support a situation where you have a single rack per DC
> for simple deployments, extending NetworkTopologyStrategy to behave the way
> it did before https://issues.apache.org/jira/browse/CASSANDRA-7544 with
> respect to treating InetAddresses as servers rather than the address and
> port would be simple. Both this implementation in Apache Cassandra and the
> respective load balancing classes in the drivers are explicitly designed to
> be pluggable so that would be an easier integration point for you.
>
> I'm not sure how it creates more tokens? If a server normally owns 256
> tokens, each shard on a different port would just advertise ownership of
> 256/# of cores (e.g. 4 tokens if you had 64 cores).
>
>
> >
> > > Regards,
> > > Ariel
> > >
> > >> On Apr 22, 2018, at 8:26 AM, Avi Kivity  wrote:
> > >>
> > >>
> > >>
> > >>> On 2018-04-19 21:15, Ben Bromhead wrote:
> > >>> Re #3:
> > >>>
> > >>> Yup I was thinking each shard/port would appear as a discrete server
> > to the
> > >>> client.
> > >> This doesn't work without additional changes, for RF>1. The token ring
> > could place two replicas of the same token range on the same physical
> > server, even though those are two separate cores of the same server. You
> > could add another element to the hierarchy (cluster -> datacenter -> rack
> > -> node -> core/shard), but that generates unneeded range movements when
> a
> > node is added.
> > >>
> > >>> If the per port suggestion is unacceptable due to hardware
> > requirements,
> > >>> remembering that Cassandra is built with the concept scaling
> > *commodity*
> > >>> hardware horizontally, you'll have to spend your time and energy
> > convincing
> > >>> the community to support a protocol feature it has no (current) use
> > for or
> > >>> find another interim solution.
> > >> Those servers are commodity servers (not x86, but still commodity). In
> > any case 60+ logical cores are common now (hello AWS i3.16xlarge or even
> > i3.metal), and we can only expect logical core count to continue to
> > increase (there are 48-core ARM processors now).
> > >>
> > >>> Another way, would be to build support and consensus around a clear
> > >>> technical need in the Apache Cassandra project as it stands today.
> > >>>
> > >>> One way to build community support might be to contribute an Apache
> > >>> licensed thread per core implementation in Java that matches the
> > protocol
> > >>> change and shard concept you are looking for ;P
> > >> I doubt I'll survive the egregious top-posting that is going on in
> this
> > list.
> > >>
> > >>>
> >  On Thu, Apr 19, 2018 at 1:43 PM Ariel Weisberg 
> > wrote:
> > 
> >  Hi,
> > 
> >  So at technical level I don't understand this yet.
> > 
> >  So you have a database consisting of single threaded shards and a
> > socket
> >  for accept that is generating TCP connections and in advance you
> > don't know
> >  which connection is going to send messages to which shard.
> > 
> > 

Re: Evolving the client protocol

[Jonathan Haddad]

Avi, if this is something that matters to you, you're more than welcome to
submit a patch to both this project and to the different drivers.  Getting
the first 2 optimizations into 4.0 would be nice, I'm sure someone would be
happy to work with you on it.

The third, I can't see why we'd need that right now.  It's going to be more
of an uphill battle, since we currently would have no notion of a shard in
Cassandra.  If you want to work on Thread Per Core for Cassandra 5.0 it
seems like it would be a reasonable addition to the protocol.

On Wed, Apr 18, 2018 at 1:24 PM sankalp kohli 
wrote:

> Do we have driver authors who wish to support both projects?
>
> On Wed, Apr 18, 2018 at 9:25 AM, Jeff Jirsa  wrote:
>
> > Removed other lists (please don't cross post)
> >
> >
> >
> >
> >
> > On Wed, Apr 18, 2018 at 3:47 AM, Avi Kivity  wrote:
> >
> > > Hello Cassandra developers,
> > >
> > >
> > > We're starting to see client protocol limitations impact performance,
> and
> > > so we'd like to evolve the protocol to remove the limitations. In order
> > to
> > > avoid fragmenting the driver ecosystem and reduce work duplication for
> > > driver authors, we'd like to avoid forking the protocol. Since these
> > issues
> > > affect Cassandra, either now or in the future, I'd like to cooperate on
> > > protocol development.
> > >
> > >
> > > Some issues that we'd like to work on near-term are:
> > >
> > >
> > > 1. Token-aware range queries
> > >
> > >
> > > When the server returns a page in a range query, it will also return a
> > > token to continue on. In case that token is on a different node, the
> > client
> > > selects a new coordinator based on the token. This eliminates a network
> > hop
> > > for range queries.
> > >
> > >
> > > For the first page, the PREPARE message returns information allowing
> the
> > > client to compute where the first page is held, given the query
> > parameters.
> > > This is just information identifying how to compute the token, given
> the
> > > query parameters (non-range queries already do this).
> > >
> > >
> > > https://issues.apache.org/jira/browse/CASSANDRA-14311
> > >
> > >
> > > 2. Per-request timeouts
> > >
> > >
> > > Allow each request to have its own timeout. This allows the user to set
> > > short timeouts on business-critical queries that are invalid if not
> > served
> > > within a short time, long timeouts for scanning or indexed queries, and
> > > even longer timeouts for administrative tasks like TRUNCATE and DROP.
> > >
> > >
> > > https://issues.apache.org/jira/browse/CASSANDRA-2848
> > >
> > >
> > > 3. Shard-aware driver
> > >
> > >
> > > This admittedly is a burning issue for ScyllaDB, but not so much for
> > > Cassandra at this time.
> > >
> > >
> > > In the same way that drivers are token-aware, they can be shard-aware -
> > > know how many shards each node has, and the sharding algorithm. They
> can
> > > then open a connection per shard and send cql requests directly to the
> > > shard that will serve them, instead of requiring cross-core
> communication
> > > to happen on the server.
> > >
> > >
> > > https://issues.apache.org/jira/browse/CASSANDRA-10989
> > >
> > >
> > > I see three possible modes of cooperation:
> > >
> > >
> > > 1. The protocol change is developed using the Cassandra process in a
> JIRA
> > > ticket, culminating in a patch to doc/native_protocol*.spec when
> > consensus
> > > is achieved.
> > >
> > >
> > > The advantage to this mode is that Cassandra developers can verify that
> > > the change is easily implementable; when they are ready to implement
> the
> > > feature, drivers that were already adapted to support it will just
> work.
> > >
> > >
> > > 2. The protocol change is developed outside the Cassandra process.
> > >
> > >
> > > In this mode, we develop the change in a forked version of
> > > native_protocol*.spec; Cassandra can still retroactively merge that
> > change
> > > when (and if) it is implemented, but the ability to influence the
> change
> > > during development is reduced.
> > >
> > >
> > > If we agree on this, I'd like to allocate a prefix for feature names in
> > > the SUPPORTED message for our use.
> > >
> > >
> > > 3. No cooperation.
> > >
> > >
> > > This requires the least amount of effort from Cassandra developers
> (just
> > > enough to reach this point in this email), but will cause duplication
> of
> > > effort for driver authors who wish to support both projects, and may
> > cause
> > > Cassandra developers to redo work that we already did.
> > >
> > >
> > > Looking forward to your views.
> > >
> > >
> > > Avi
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >
> > >
> >
>

Re: Repair scheduling tools

[Jonathan Haddad]

Off the top of my head I can remember clusters with 600 or 700 nodes with
256 tokens.

Not the best situation, but it’s real. 256 has been the default for better
or worse.

On Thu, Apr 5, 2018 at 7:41 PM Joseph Lynch  wrote:

> >
> > We see this in larger clusters regularly. Usually folks have just
> > 'grown into it' because it was the default.
> >
>
> I could understand a few dozen nodes with 256 vnodes, but hundreds is
> surprising. I have a whitepaper draft lying around showing how vnodes
> decrease availability in large clusters by orders of magnitude, I'll polish
> it up and send it out to the list when I get a second.
>
> In the meantime, sorry for de-railing a conversation about repair
> scheduling to talk about vnodes, let's chat about that in a different
> thread :-)
>
> -Joey
>

Re: Roadmap for 4.0

[Jonathan Haddad]

That’s exactly what I was thinking too.

There’s also nothing preventing features from being merged into trunk after
we create the 4.0 branch, which in my opinion is a better approach than
trying to jam everything in right before the release.
On Thu, Apr 5, 2018 at 12:06 PM Jason Brown  wrote:

> My understanding, from Nate's summary, was June 1 is the freeze date for
> features. I expect we would go for at least 4 months (if not longer)
> testing, fixing bugs, early dogfooding, and so on. I also equated June 1
> with the data which we would create a 'cassandra-4.0' branch, and thus the
> merge order becomes: 3.0->3,11->4.0->trunk.
>
> Is this different from what others are thinking? I'm open to shifting the
> actual date, but what about the rest?
>
>
> On Thu, Apr 5, 2018 at 11:39 AM, Aleksey Yeshchenko 
> wrote:
>
> > June feels a bit too early to me as well.
> >
> > I personally would go prefer end of August / beginning of September.
> >
> > +1 to the idea of having a fixed date, though, just not this one.
> >
> > —
> > AY
> >
> > On 5 April 2018 at 19:20:12, Stefan Podkowinski (s...@apache.org) wrote:
> >
> > June is too early.
> >
> >
> > On 05.04.18 19:32, Josh McKenzie wrote:
> > > Just as a matter of perspective, I'm personally mentally diffing from
> > > when 3.0 hit, not 3.10.
> > >
> > >> commit 96f407bce56b98cd824d18e32ee012dbb99a0286
> > >> Author: T Jake Luciani 
> > >> Date: Fri Nov 6 14:38:34 2015 -0500
> > >> 3.0 release versions
> > > While June feels close to today relative to momentum for a release
> > > before this discussion, it's certainly long enough from when the
> > > previous traditional major released that it doesn't feel "too soon" to
> > > me.
> > >
> > > On Thu, Apr 5, 2018 at 12:46 PM, sankalp kohli  >
> > wrote:
> > >> We can take a look on 1st June how things are then decide if we want
> to
> > >> freeze it and whats in and whats out.
> > >>
> > >> On Thu, Apr 5, 2018 at 9:31 AM, Ariel Weisberg 
> > wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> +1 to having a feature freeze date. June 1st is earlier than I would
> > have
> > >>> picked.
> > >>>
> > >>> Ariel
> > >>>
> > >>> On Thu, Apr 5, 2018, at 10:57 AM, Josh McKenzie wrote:
> >  +1 here for June 1.
> > 
> >  On Thu, Apr 5, 2018 at 9:50 AM, Jason Brown 
> > >>> wrote:
> > > +1
> > >
> > > On Wed, Apr 4, 2018 at 8:31 PM, Blake Eggleston <
> > beggles...@apple.com>
> > > wrote:
> > >
> > >> +1
> > >>
> > >> On 4/4/18, 5:48 PM, "Jeff Jirsa"  wrote:
> > >>
> > >> Earlier than I’d have personally picked, but I’m +1 too
> > >>
> > >>
> > >>
> > >> --
> > >> Jeff Jirsa
> > >>
> > >>
> > >> > On Apr 4, 2018, at 5:06 PM, Nate McCall 
> > > wrote:
> > >> >
> > >> > Top-posting as I think this summary is on point - thanks,
> > >>> Scott!
> > > (And
> > >> > great to have you back, btw).
> > >> >
> > >> > It feels to me like we are coalescing on two points:
> > >> > 1. June 1 as a freeze for alpha
> > >> > 2. "Stable" is the new "Exciting" (and the testing and
> > >>> dogfooding
> > >> > implied by such before a GA)
> > >> >
> > >> > How do folks feel about the above points?
> > >> >
> > >> >
> > >> >> Re-raising a point made earlier in the thread by Jeff and
> > >>> affirmed
> > >> by Josh:
> > >> >>
> > >> >> –––
> > >> >> Jeff:
> > >>  A hard date for a feature freeze makes sense, a hard date
> > >>> for a
> > >> release
> > >>  does not.
> > >> >>
> > >> >> Josh:
> > >> >>> Strongly agree. We should also collectively define what
> > >>> "Done"
> > >> looks like
> > >> >>> post freeze so we don't end up in bike-shedding hell like we
> > >>> have
> > >> in the
> > >> >>> past.
> > >> >> –––
> > >> >>
> > >> >> Another way of saying this: ensuring that the 4.0 release is
> > >>> of
> > >> high quality is more important than cutting the release on a
> > specific
> > > date.
> > >> >>
> > >> >> If we adopt Sylvain's suggestion of freezing features on a
> > > "feature
> > >> complete" date (modulo a "definition of done" as Josh suggested),
> > >>> that
> > > will
> > >> help us align toward the polish, performance work, and dog-fooding
> > >>> needed
> > >> to feel great about shipping 4.0. It's a good time to start
> thinking
> > > about
> > >> the approaches to testing, profiling, and dog-fooding various
> > > contributors
> > >> will want to take on before release.
> > >> >>
> > >> >> I love how Ben put it:
> > >> >>
> > >> >>> An "exciting" 4.0 release to me is one that is stable and
> > >>> usable
> > >> >>> with no perf regressions on day 1 and includes some of the
> > >>> big
> > 

Re: Repair scheduling tools

[Jonathan Haddad]

To be fair, reaper in 2016 only worked with 2.0 and was just sitting
around, more or less.

Since then we've had 401 commits changing tens of thousands of lines of
code, dealing with fault tolerance, repair retries, scalability, etc.
We've had 1 reaper node managing repairs across dozens of clusters and
thousands of nodes.  It's a totally different situation today.


On Thu, Apr 5, 2018 at 11:17 AM benjamin roth  wrote:

> That would be totally awesome!
>
> Not sure if it helps here but for completeness:
> We completely "dumped" regular repairs - no matter if 'nodetool repair' or
> reaper - and run our own tool that does simply CL_ALL scraping over the
> whole cluster.
> It runs now for over a year in production and the only problem we
> encountered was that we got timeouts when scraping (too) large / tombstoned
> partitions. It turned out that the large partitions weren't even readable
> with CQL / cqlsh / DevCenter. So that wasn't a problem of the repair. It
> was rather a design problem. Storing data that can't be read doesn't make
> sense anyway.
>
> What I can tell from our experience:
> - It works much more reliable than what we had before - also more reliable
> than reaper (state of 2016)
> - It runs totally smooth and much faster than regular repairs as it only
> streams what needs to be streamed
> - It's easily manageable, interruptible, resumable on a very fine-grained
> level. The only thing you need to do is to store state (KS/CF/Last Token)
> in a simple storage like redis
> - It works even pretty well when populating a empty node e.g. when changing
> RFs / bootstrapping DCs
> - You can easily control the cluster-load by tuning the concurrency of the
> scrape process
>
> I don't see a reason for us to ever go back to built-in repairs if they
> don't improve immensely. In many cases (especially with MVs) they are true
> resource killers.
>
> Just my 2 cent and experience.
>
> 2018-04-04 17:00 GMT+02:00 Ben Bromhead :
>
> > +1 to including the implementation in Cassandra itself. Makes managed
> > repair a first-class citizen, it nicely rounds out Cassandra's
> consistency
> > story and makes it 1000x more likely that repairs will get run.
> >
> >
> >
> >
> > On Wed, Apr 4, 2018 at 10:45 AM Jon Haddad  wrote:
> >
> > > Implementation details aside, I’m firmly in the “it would be nice of C*
> > > could take care of it” camp.  Reaper is pretty damn easy to use and
> > people
> > > *still* don’t put it in prod.
> > >
> > >
> > > > On Apr 4, 2018, at 4:16 AM, Rahul Singh <
> rahul.xavier.si...@gmail.com>
> > > wrote:
> > > >
> > > > I understand the merits of both approaches. In working with other DBs
> > In
> > > the “old country” of SQL, we often had to write indexing sequences
> > manually
> > > for important tables. It was “built into the product” but in order to
> > > leverage the maximum benefits of indices we had to have different
> indices
> > > other than the clustered (physical index). The process still sucked.
> It’s
> > > never perfect.
> > > >
> > > > The JVM is already fraught with GC issues and putting another process
> > > being managed in the same heapspace is what I’m worried about.
> > Technically
> > > the process could be in the same binary but started as a side Car or in
> > the
> > > same main process.
> > > >
> > > > Consider a process called “cassandra-agent” that’s sitting around
> with
> > a
> > > scheduler based on config or a Cassandra table. Distributed in the same
> > > release. Shell / service scripts would start it. The end user knows it
> > only
> > > by examining the .sh files. This opens possibilities of including a GUI
> > > hosted in the same process without cluttering the core coolness of
> > > Cassandra.
> > > >
> > > > Best,
> > > >
> > > > --
> > > > Rahul Singh
> > > > rahul.si...@anant.us
> > > >
> > > > Anant Corporation
> > > >
> > > > On Apr 4, 2018, 2:50 AM -0400, Dor Laor , wrote:
> > > >> We at Scylla, implemented repair in a similar way to the Cassandra
> > > reaper.
> > > >> We do
> > > >> that using an external application, written in go that manages
> repair
> > > for
> > > >> multiple clusters
> > > >> and saves the data in an external Scylla cluster. The logic
> resembles
> > > the
> > > >> reaper one with
> > > >> some specific internal sharding optimizations and uses the Scylla
> rest
> > > api.
> > > >>
> > > >> However, I have doubts it's the ideal way. After playing a bit with
> > > >> CockroachDB, I realized
> > > >> it's super nice to have a single binary that repairs itself,
> provides
> > a
> > > GUI
> > > >> and is the core DB.
> > > >>
> > > >> Even while distributed, you can elect a leader node to manage the
> > > repair in
> > > >> a consistent
> > > >> way so the complexity can be reduced to a minimum. Repair can write
> > its
> > > >> status to the
> > > >> system tables and to provide an api for progress, rate control, etc.
> > > >>
> > > >> The big 

Re: Paying off tech debt and correctly naming things

[Jonathan Haddad]

As requested, I'm alerting the ML that I've got the first patch of several
to come.  This one only addresses the ColumnFamilyStore class - and only
changes the name.  There's follow up tickets to change method and variable
names.  As we agreed, I'm doing this incrementally, so please let's keep
that in mind, I have no interest in producing a 10K patch that changes all
the things.

Since the patch only touches lines explicitly naming ColumnFamilyStore this
patch should be reasonably non-intrusive.

https://issues.apache.org/jira/browse/CASSANDRA-14340

Jon

On Thu, Mar 22, 2018 at 8:09 AM Jon Haddad  wrote:

> Cool.  I think there’s general agreement that doing this in as small bites
> as possible is going to be the best approach.  I have no interest in mega
> patches.
>
> >  The combined approach takes a
> > change that's already non-trivially dealing with complex subsystem
> > changes and injects a bunch of trivial renaming noise across unrelated
> > subsystems into the signal of an actual logic refactor.
>
> I agree.  This is why I like the idea of proactively working to improve
> the readability of the codebase as a specific goal, rather than being
> wrapped into some other unrelated patch.  Keeping the scope in check is the
> challenge.  Simple class and method renames, as several have pointed out,
> is easy enough with IDEA.
>
> I’ll start with class renames, as individual patches for each of them.
> I’ll be sure to call it out on the ML.  First one will be ColumnFamilyStore
> -> TableStore.
>
> Jon
>
> > On Mar 22, 2018, at 7:13 AM, Jason Brown  wrote:
> >
> > Jon,
> >
> > Thanks for bringing up this topic. I'll admit that I've been around this
> > code base for long enough, and have enough accumulated history, that I
> > probably can't fully appreciate the impact for a newcomer wrt naming.
> > However, as Josh points out, this situation probably happens to "every
> > non-trivially aged code-base ever".
> >
> > One thing I'd like to add is that with these types of large refactoring
> > changes, the review effort is non-trivial. This is because the review
> still
> > has to ensure that correctness is preserved and it's easy to overlook a
> > seemingly innocuous change.
> >
> > That being said, I am supportive of this effort. However, I believe it's
> > going to be best, for contributor and reviewer, to break it up into
> > smaller, more digestible pieces. I'd also like to request that we not go
> > whole hog and try to do everything in a compressed time frame; reviewer
> > availability is already stretched thin and I'm afraid of deepening the
> > review queue, especially mine :)
> >
> > Thanks,
> >
> > -Jason
> >
> >
> >
> >
> > On Thu, Mar 22, 2018 at 6:41 AM, Josh McKenzie 
> wrote:
> >
> >>> Some of us have big patches in flight, things that actually
> >>> pay off some technical debt, and dealing with such renames is rebase
> >> hell :\
> >> For sure, but with a code-base this old / organically grown, I expect
> >> this will always be the case. If we're talking something as simple as
> >> an intellij rename refactor, while menial, couldn't someone with a
> >> giant patch just do the same thing on their side and spend half an
> >> hour of their life clicking next? ;)
> >>
> >>> That said, there is good time for such renames - it’s during
> >>> those major refactors and rewrites. When you are
> >>> changing a subsystem, might as well do the appropriate renames.
> >> Does that hold true for a code-base with as much static state and
> >> abstraction leaking / bad factoring as we have? (i.e. every
> >> non-trivially aged code-base ever) The combined approach takes a
> >> change that's already non-trivially dealing with complex subsystem
> >> changes and injects a bunch of trivial renaming noise across unrelated
> >> subsystems into the signal of an actual logic refactor.
> >>
> >> On Thu, Mar 22, 2018 at 9:31 AM, Aleksey Yeshchenko 
> >> wrote:
> >>> Poor and out-of-date naming of things is probably the least serious
> part
> >> of our technical debt. Bad factoring, and straight-up
> >>> poorly written components is where it’s really at.
> >>>
> >>> Doing a big rename for rename sake alone does more harm than it is
> good,
> >> sometimes. Some of us have big patches
> >>> in flight, things that actually pay off some technical debt, and
> dealing
> >> with such renames is rebase hell :\
> >>>
> >>> That said, there is good time for such renames - it’s during those
> major
> >> refactors and rewrites. When you are
> >>> changing a subsystem, might as well do the appropriate renames.
> >>>
> >>> —
> >>> AY
> >>>
> >>> On 20 March 2018 at 22:04:48, Jon Haddad (j...@jonhaddad.com) wrote:
> >>>
> >>> Whenever I hop around in the codebase, one thing that always manages to
> >> slow me down is needing to understand the context of the variable names
> >> that I’m looking at. We’ve now removed thrift the transport, but the
> >> 

Re: [DISCUSS] java 9 and the future of cassandra on the jdk

[Jonathan Haddad]

Java 8 was marked as EOL in the middle of last year, I hope we wouldn't
require it for Cassandra 4.  At this point I feel like we should already be
targeting Java 10 at a minimum.

Personally I'd prefer not to tie our releases to any vendor / product /
package's release schedule.


On Fri, Mar 23, 2018 at 6:49 AM Jason Brown  wrote:

> I'm coming to be on-board with #3.
>
> One thing to watch out for (we can't account for it now) is how our
> dependencies choose to move forward. If we need to upgrade a jar (netty,
> for example) due to some leak or vulnerability, and it only runs on a
> higher version, we may be forced to upgrade the base java version. Like, I
> said we can't possibly foresee these things, and we'll just have to make a
> hard decision if the situation arises, but just something to keep in mind.
>
>
> On Fri, Mar 23, 2018 at 5:39 AM, Josh McKenzie 
> wrote:
>
> > >
> > > 3) Release 4.0 for Java 8, *optionally* branch 4.1 for Java 11 later
> >
> > This seems like the best of our bad options, with the addition of
> > "optionally".
> >
> >
> > On Fri, Mar 23, 2018 at 8:12 AM, Gerald Henriksen 
> > wrote:
> >
> > > On Fri, 23 Mar 2018 04:54:23 +, you wrote:
> > >
> > > >I think Michael is right. It would be impossible to make everyone
> follow
> > > >such a fast release scheme, and supporting it will be pressured onto
> the
> > > >various distributions, M$ and Apple.
> > > >On the other hand https://adoptopenjdk.net has already done a lot of
> > the
> > > >work and it's already rumoured they may take up backporting of
> > > security/bug
> > > >fixes. I'd fully expect a lot of users to collaborate around this (or
> > > >similar), and there's no reason we couldn't do our part to contribute.
> > >
> > > A posting on Reddit yesterday from someone from adoptopenjdk claimes
> > > that they will be doing LTS releases starting with Java 11, and there
> > > should be updates to their website to reflect that soon:
> > >
> > > https://www.reddit.com/r/java/comments/86ce66/java_long_term_support/
> > >
> > > So I guess a wait and see to what they commit could be worthwhile.
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >
> > >
> >
>

Re: Making RF4 useful aka primary and secondary ranges

[Jonathan Haddad]

You could use a load balancing policy at the driver level to do what you
want, mixed with the existing consistency levels as Jeff suggested.

On Wed, Mar 14, 2018 at 3:47 PM Carl Mueller 
wrote:

> But we COULD have CL2 write (for RF4)
>
> The extension to this idea is multiple backup/secondary replicas. So you
> have RF5 or RF6 or higher, but still are performing CL2 against the
> preferred first three for both read and write.
>
> You could also ascertain the general write health of affected ranges before
> taking a node down for maintenance from the primary, and then know the
> switchover is in good shape. Yes there are CAP limits and race conditions
> there, but you could get pretty good assurances (all repaired, low/zero
> queued hinted handoffs, etc).
>
> This is essentially like if you had two datacenters, but are doing
> local_quorum on the one datacenter. Well, except switchover is a bit more
> granular if you run out of replicas in the local.
>
>
>
> On Wed, Mar 14, 2018 at 5:17 PM, Jeff Jirsa  wrote:
>
> > Write at CL 3 and read at CL 2
> >
> > --
> > Jeff Jirsa
> >
> >
> > > On Mar 14, 2018, at 2:40 PM, Carl Mueller <
> carl.muel...@smartthings.com>
> > wrote:
> > >
> > > Currently there is little use for RF4. You're getting the requirements
> of
> > > QUORUM-3 but only one extra backup.
> > >
> > > I'd like to propose something that would make RF4 a sort of more
> heavily
> > > backed up RF3.
> > >
> > > A lot of this is probably achievable with strictly driver-level logic,
> so
> > > perhaps it would belong more there.
> > >
> > > Basically the idea is to have four replicas of the data, but only have
> to
> > > practically do QUORUM with three nodes. We consider the first three
> > > replicas the "primary replicas". On an ongoing basis for QUORUM reads
> and
> > > writes, we would rely on only those three replicas to satisfy
> > > two-out-of-three QUORUM. Writes are persisted to the fourth replica in
> > the
> > > normal manner of cassandra, it just doesn't count towards the QUORUM
> > write.
> > >
> > > On reads, with token and node health awareness by the driver, if the
> > > primaries are all healthy, two-of-three QUORUM is calculated from
> those.
> > >
> > > If however one of the three primaries is down, read QUORUM is a bit
> > > different:
> > > 1) if the first two replies come from the two remaining primaries and
> > > agree, the is returned
> > > 2) if the first two replies are a primary and the "hot spare" and those
> > > agree, that is returned
> > > 3) if the primary and hot spare disagree, wait for the next primary to
> > > return, and then take the agreement (hopefully) that results
> > >
> > > Then once the previous primary comes back online, the read quorum goes
> > back
> > > to preferring that set, with the assuming hinted handoff and repair
> will
> > > get it back up to snuff.
> > >
> > > There could also be some mechanism examining the hinted handoff status
> of
> > > the four to determine when to reactivate the primary that was down.
> > >
> > > For mutations, one could prefer a "QUORUM plus" that was a quorum of
> the
> > > primaries plus the hot spare.
> > >
> > > Of course one could do multiple hot spares, so RF5 could still be
> treated
> > > as RF3 + hot spares.
> > >
> > > The goal here is more data resiliency but not having to rely on as many
> > > nodes for resiliency.
> > >
> > > Since the data is ring-distributed, the fact there are primary owners
> of
> > > ranges should still be evenly distributed and no hot nodes should
> result
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>

Re: Why isn't there a separate JVM per table?

[Jonathan Haddad]

There's an incredible amount of work that would need to be done in order to
make any of this happen.  Basically a full rewrite of the entire codebase.
Years of effort.

The codebase would have to move to a shared-nothing actor & message based
communication mechanism before any of this is possible.  Fun in theory, but
considering removing singletons has been a multi-year, many failure effort,
I suspect we might need 10 years to refactor Cassandra to use multiple
JVMs.  By then maybe we'll have a pauseless / low pause collector and it
won't matter.

On Thu, Feb 22, 2018 at 3:59 PM kurt greaves  wrote:

> >
> >  ... compaction on its own jvm was also something I was thinking about,
> but
> > then I realized even more JVM sharding could be done at the table level.
>
>
> Compaction in it's own JVM makes sense. At the table level I'm not so sure
> about. Gotta be some serious overheads from running that many JVM's.
> Keyspace might be reasonable purely to isolate bad tables, but for the most
> part I'd think isolating every table isn't that beneficial and pretty
> complicated. In most cases people just fix their modelling so that they
> don't generate large amounts of GC, and hopefully test enough so they know
> how it will behave in production.
>
> If we did at the table level we would inevitable have to make each
> individual table incredibly tune-able which would be a bit tedious IMO.
> There's no way for us to smartly decide how much heap/memtable space/etc
> each table should use (not without some decent AI, anyway).
> ​
>

Re: Expensive metrics?

[Jonathan Haddad]

Hey Micke, very cool you're looking to improve C*'s performance, we would
absolutely benefit from it.

Have you done any other benchmarks beside the micro one to determine the
total effect of these metrics on the system overall?  Microbenchmarks are a
great way to tune small sections of code but they aren't a great starting
point.  It would be good if we could put some context around the idea by
benchmarking a tuned, single node (so there's less network overhead)
running on fast disks with compaction disabled so we can see what kind of
impact these metrics are adding.  Ideally we'd look at GC promotion and CPU
time using something like YourKit to identify the overall effect of the
metrics, so we can set our expectations and goals in a reasonable manner.
Happy to coordinate with you on this!

On Thu, Feb 22, 2018 at 8:08 AM Jeremiah D Jordan 
wrote:

> re: nanoTime vs currentTimeMillis there is a good blog post here about the
> timing of both and how your choice of Linux clock source can drastically
> effect the speed of the calls, and also showing that in general on linux
> there is no perf improvement for one over the other.
> http://pzemtsov.github.io/2017/07/23/the-slow-currenttimemillis.html
>
> > On Feb 22, 2018, at 11:01 AM, Blake Eggleston 
> wrote:
> >
> > Hi Micke,
> >
> > This is really cool, thanks for taking the time to investigate this. I
> believe the metrics around memtable insert time come in handy in
> identifying high partition contention in the memtable. I know I've been
> involved in a situation over the past year where we got actionable info
> from this metric. Reducing resolution to milliseconds is probably a no go
> since most things in this path should complete in less than a millisecond.
> >
> > Revisiting the use of the codahale metrics in the hot path like this
> definitely seems like a good idea though. I don't think it's been something
> we've talked about a lot, and it definitely looks like we could benefit
> from using something more specialized here. I think it's worth doing,
> especially since there won't be any major changes to how we do threading in
> 4.0. It's probably also worth opening a JIRA and investigating the calls to
> nano time. We at least need microsecond resolution here, and there could be
> something we haven't thought of? It's worth a look at least.
> >
> > Thanks,
> >
> > Blake
> >
> > On 2/22/18, 6:10 AM, "Michael Burman"  wrote:
> >
> >Hi,
> >
> >I wanted to get some input from the mailing list before making a JIRA
> >and potential fixes. I'll touch the performance more on latter part,
> but
> >there's one important question regarding the write latency metric
> >recording place. Currently we measure the writeLatency (and metric
> write
> >sampler..) in ColumnFamilyStore.apply() and this is also the metric we
> >then replicate to Keyspace metrics etc.
> >
> >This is an odd place for writeLatency. Not to mention it is in a
> >hot-path of Memtable-modifications, but it also does not measure the
> >real write latency, since it completely ignores the CommitLog latency
> in
> >that same process. Is the intention really to measure
> >Memtable-modification latency only or the actual write latencies?
> >
> >Then the real issue.. this single metric is a cause of huge overhead
> in
> >Memtable processing. There are several metrics / events in the CFS
> apply
> >method, including metric sampler, storageHook reportWrite,
> >colUpdateTimeDeltaHistogram and metric.writeLatency. These are not
> free
> >at all when it comes to the processing. I made a small JMH benchmark
> >here:
> https://gist.github.com/burmanm/b5b284bc9f1d410b1d635f6d3dac3ade
> >that I'll be referring to.
> >
> >The most offending of all these metrics is the writeLatency metric.
> What
> >it does is update the latency in codahale's timer, doing a histogram
> >update and then going through all the parent metrics also which update
> >the keyspace writeLatency and globalWriteLatency. When measuring the
> >performance of Memtable.put with parameter of 1 partition (to reduce
> the
> >ConcurrentSkipListMap search speed impact - that's separate issue and
> >takes a little bit longer to solve although I've started to prototype
> >something..) on my machine I see 1.3M/s performance with the metric
> and
> >when it is disabled the performance climbs to 4M/s. So the overhead
> for
> >this single metric is ~2/3 of total performance. That's insane. My
> perf
> >stats indicate that the CPU is starved as it can't get enough data in.
> >
> >Removing the replication from TableMetrics to the Keyspace & global
> >latencies in the write time (and doing this when metrics are requested
> >instead) improves the performance to 2.1M/s on my machine. It's an
> >improvement, but it's still huge amount. Even when we pressure 

Re: [VOTE] (Take 2) Release Apache Cassandra 3.11.2

[Jonathan Haddad]

+1
On Mon, Feb 12, 2018 at 9:51 PM mck  wrote:

>
>
> > I propose the following artifacts for release as 3.11.2.
>
>
> Thanks Michael for the recut, very much appreciated.
>
> +1
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Getting partition min/max timestamp

[Jonathan Haddad]

Do you need to support TTLs? That might be a bit of an issue.
On Sat, Jan 13, 2018 at 12:41 PM Arthur Kushka  wrote:

> Hi folks,
>
> Currently, I working on custom CQL operator that should return the max
> timestamp for some partition.
>
> I don't think that scanning of partition for that kind of data is a nice
> idea. Instead of it, I thinking about adding a metadata to the partition. I
> want to store minTimestamp and maxTimestamp for every partition as it
> already done in Memtable`s. That timestamps will be updated on each
> mutation operation, that is quite cheap in comparison to full scan.
>
> I quite new to Cassandra codebase and want to get some critics and ideas,
> maybe that kind of data already stored somewhere or you have better ideas.
> Is my assumption right?
>
> Best,
> Artur
>

Re: Do it need to run 'nodetool upgradesstables' when updating from 2.0.9 to 2.1.19?

[Jonathan Haddad]

After an upgrade I recommend running upgrade sstables no matter what the
version change is. If it's not needed, nothing will happen.
On Fri, Oct 13, 2017 at 4:30 AM Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

> And extremely useful/important in the field not being a strict requirement
> immediately upgrading sstables, especially for not closely monitored
> environments, e.g. unattended deployments like development machines or even
> customer on-prem installations etc.
>
> Cassandra's data backwards compatibility is one of the best I have seen.
>
> Thanks a lot for that!
>
> T.
>
>
> -Original Message-
> From: J. D. Jordan [mailto:jeremiah.jor...@gmail.com]
> Sent: Freitag, 13. Oktober 2017 13:13
> To: dev@cassandra.apache.org
> Subject: Re: Do it need to run 'nodetool upgradesstables' when updating
> from 2.0.9 to 2.1.19?
>
> Old restrictions that have been lifted Jeff :) Ever since 2.0 we have
> supported streaming old sstables, see here for the 2.0 ticket:
> https://issues.apache.org/jira/browse/CASSANDRA-5772
>
> That was broken in early 3.0, but had since been fixed by the ticket
> Marcus linked https://issues.apache.org/jira/browse/CASSANDRA-10990
>
> So it should be fine in all latest 3.x versions.
>
> -Jeremiah
>
>
> > On Oct 13, 2017, at 3:34 AM, Steinmaurer, Thomas <
> thomas.steinmau...@dynatrace.com> wrote:
> >
> > Same here, regarding sincere question to know the corner cases from 2.1
> => 3.11.1, but Marcus already provided the JIRA ticket.
> >
> > Thanks!
> >
> >
> > -Original Message-
> > From: Per Otterström [mailto:per.otterst...@ericsson.com]
> > Sent: Freitag, 13. Oktober 2017 09:01
> > To: dev@cassandra.apache.org
> > Subject: RE: Do it need to run 'nodetool upgradesstables' when updating
> from 2.0.9 to 2.1.19?
> >
> > Hepp!
> >
> > Just to be clear, I'm not claiming this to be the case, it is a sincere
> question. My team is planning an upgrade from 2.2 to 3.0 which is why I'm
> asking. Some initial tests indicate that repairs etc work well before
> running upgradesstables (assuming all nodes are upgraded to 3.0). But if
> there are known corner cases I'd be interested to know.
> >
> > We need to support reading old sstables in order to support rolling
> upgrades. This seem to be a general assumption that people agree on. I'm
> not sure there is a clear agreement on the streaming part. Would it make
> sense to clarify this in documentation?
> >
> > /pelle
> >
> > -Original Message-
> > From: Jeff Jirsa [mailto:jji...@gmail.com]
> > Sent: den 13 oktober 2017 08:09
> > To: dev@cassandra.apache.org
> > Subject: Re: Do it need to run 'nodetool upgradesstables' when updating
> from 2.0.9 to 2.1.19?
> >
> > Two people say I’m wrong, I’ll believe it - must be imagining
> restrictions that don’t exist.
> >
> > --
> > Jeff Jirsa
> >
> >
> >> On Oct 12, 2017, at 10:55 PM, Per Otterström <
> per.otterst...@ericsson.com> wrote:
> >>
> >> Hi!
> >>
> >> This was news to me. I had the impression that we are maintaining
> backwards compatibility for streaming non-upgraded sstables. Is that not
> the case?
> >>
> >> However, I believe it is a good idea to run upgradesstables at some
> point _before_ upgrading Cassandra next time to make sure there are no old
> sstables around before you take the next step.
> >>
> >> /pelle
> >>
> >> -Original Message-
> >> From: Jeff Jirsa [mailto:jji...@gmail.com]
> >> Sent: den 12 oktober 2017 23:11
> >> To: Cassandra DEV 
> >> Subject: Re: Do it need to run 'nodetool upgradesstables' when updating
> from 2.0.9 to 2.1.19?
> >>
> >> You won't notice any issues (all versions of cassandra are backwards
> compatible at least 1 version), and sstables will be slowly migrated to the
> new format as you compact, but you should still run it (upgradesstables)
> explicitly because when it comes time to run
> repair/boostrap/decommission/etc, you'll need all sstables on the
> same/current format.
> >>
> >>
> >>
> >> On Thu, Oct 12, 2017 at 1:59 PM, Li, Guangxing
> >> 
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> I recently upgraded my test cluster from C* 2.0.9 to 2.1.19 and I
> >>> did not run 'nodetool upgradesstables' at all. Per my confirmation
> >>> afterwards, everything is fine. But according to documentation at
> >>> https://support.datastax.com/hc/en-us/articles/208040036-
> >>> Nodetool-upgradesstables-FAQ.
> >>> I need to do that.
> >>> So can someone tell me if I must do 'nodetool upgradesstables' after
> >>> upgrading each node from 2.0.9 to 2.1.19?
> >>>
> >>> Thanks.
> >>>
> >>> George.
> >>>
> >>  B‹
> >> CB• È [œà XœØÜšX™K  K[XZ[
> > ˆ  ]‹][œà XœØÜšX™P Ø\ÜØ[™ ˜K˜\ XÚ K›Ü™ÃB‘›
> > ܈ Y  ] [Û˜[  ÛÛ[X[™ Ë  K[XZ[
> > ˆ  ]‹Z [   Ø\ÜØ[™ ˜K˜\ XÚ K›Ü™ÃBƒB
> >
> > -

Re: [VOTE] Release Apache Cassandra 2.1.19

[Jonathan Haddad]

+1
On Thu, Sep 28, 2017 at 1:46 PM Nate McCall  wrote:

> +1
>
> On Sep 29, 2017 7:12 AM, "Michael Shuler"  wrote:
>
> I propose the following artifacts for release as 2.1.19.
>
> sha1: 428eaa3e37cab7227c81fdf124d29dfc1db4257c
> Git:
> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> shortlog;h=refs/tags/2.1.19-tentative
> Artifacts:
> https://repository.apache.org/content/repositories/
> orgapachecassandra-1148/org/apache/cassandra/apache-cassandra/2.1.19/
> Staging repository:
> https://repository.apache.org/content/repositories/orgapachecassandra-1148/
>
> The Debian and RPM packages are available here:
> http://people.apache.org/~mshuler
>
> The vote will be open for 72 hours (longer if needed).
>
> [1]: (CHANGES.txt) https://goo.gl/1sZLdP (also RPM file ownership fix)
> [2]: (NEWS.txt) https://goo.gl/YKEuRc
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>

Re: Apache cassandra wiki access

[Jonathan Haddad]

The wiki isn't really used anymore. You can submit patches to improve the
docs in tree now. Fee free to tag me as a reviewer and I'll get to it this
week.
On Sat, Sep 16, 2017 at 10:26 AM Vusal Ahmadoglu 
wrote:

> Hello,
>
> Could you please give access me to the apache cassandra  wiki?
> e-mail: vusal.ahmado...@gmail.com
> Wiki name: Vusal Ahmadoglu
>
> Have a lovely weekend!
>
> Many thanks,
> Vusal
>

Re: NGCC Proposal (Was Re: NGCC?)

[Jonathan Haddad]

Agreed with Jeff & Jason.

On Tue, Jun 13, 2017 at 11:45 AM Jeff Jirsa  wrote:

> Looks great to me - especially the venue.  Date wise, Tuesday (19th) lets
> people fly in on Monday instead of costing a weekend, so selfishly that
> seems better to me.
>
>
>
> On Mon, Jun 12, 2017 at 1:30 PM, Gary Dusbabek 
> wrote:
>
> > ## Cassandra list email
> >
> > Hi everybody,
> >
> > Here are current thoughts about structure and timing. Your feedback is
> > welcome.
> >
> > Date: One of 18, 19, or 22 September 2017. We are aware the Tokyo user
> > group is planning an event the first week in October. We’re doing our
> best
> > to give a buffer there.
> >
> > Venue: After evaluating a few options in San Antonio, the event space at
> > Geekdom seems like a good fit and the right balance of cost and services
> (
> > goo.gl/tViC72).
> >
> > Format: This will be a one day event, starting in the morning and running
> > through the evening. Here is a proposed agenda we can iterate on:
> >
> > * 8-9am - catered breakfast, conversation
> > * 9-12 - presentations, including a coffee/snack break
> > * 12-1:30pm - catered lunch
> > * 1:30-4:00pm - presentations, including a coffee/snack break
> > * 4-5pm - lightning talks
> > * 5-6:30pm - happy hour and a half
> > * 6:30-9:30pm - dinner at a local restaurant
> >
> > We hope to film the presentations and make them available on Youtube.
> >
> > A few have already reached out offering various resources or assistance.
> > Thank you! Eric or I will be reaching out to coordinate as soon as we are
> > ready to finalize plans.
> >
> > Please chime in if you have suggestions, especially those relating to
> > format.
> >
> > Cheers,
> >
> > Gary
> >
>

Re: NGCC?

[Jonathan Haddad]

+1.

Are there guidelines for how to set something like this up or is it tribal
knowledge?

On Wed, May 31, 2017 at 3:05 PM Gary Dusbabek  wrote:

> +1. I'm happy to help with logistics.
>
> Mid-to-late September is good, but so is October.
>
> Gary.
>
> On Wed, May 31, 2017 at 2:19 PM, Eric Evans 
> wrote:
>
> > Hi,
> >
> > Is anyone working on an NGCC event for this year? Given all of the
> > recent changes, it seems like it could be really useful. If not, is
> > there interest? When would be good, September? October? What about
> > location? I think we could get the space to host such an even in
> > either San Antonio (centrally located), or San Francisco (San
> > Franciscoly located).
> >
> > Thoughts?
> >
> > --
> > Eric Evans
> > john.eric.ev...@gmail.com
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>

rebuilding cassandra.apache.org docs

[Jonathan Haddad]

Looks like the docs haven't been rebuilt in a while.  There's a handful of
useful merges recently that I'm aware of that would be great to see on
there.

How do they get rebuilt?  Who can trigger it?

Jon

Re: Integrating vendor-specific code and developing plugins

[Jonathan Haddad]

There's a handful of issues I can think of with shipping everything
in-tree.  I'll try to brain dump them here.

* What's included when shipped in tree?

Does every idea get merged in? Do we need 30 different Seed providers?  Who
judges what's valuable enough to add?  Who maintains it when it needs
updating?  If the maintainer can't be found, is it removed?  Shipped
broken?  Does the contributed plugins go through the same review process?
Do the contributors need to be committers?  Would CASSANDRA-12627 be merged
in even if nobody saw the value?

* Language support

Cassandra is based on Java 8.  Do we now merge in Scala, Kotlin, Jython?

* Plugins are now tied directly to cassandra release cycle

This one bugs me quite a bit.  With a new plugin, there's going to be a lot
of rapid iterations.  Spark releases once every 3 months - a lot of the
packages seem to be released at a much higher frequency.

* In Python, the standard lib is where modules go to die

I forgot where I heard this, but it's pretty accurate.  Including
everything, "batteries includes", just ends up shipping some pretty
terrible batteries.  The best stuff for python is on pypi.

Rust deliberately made the decision to limit the std to avoid this
problem.  There's a "nursery" [1] area for ideas to evolve independently,
and when some code reaches a high enough level of maturity, it can get
merged in.  There's also a packages system for third party, non std
destined code.

Anyways - I'm very +1 on a package system where codebases can independently
evolve without needing to be part of the project itself.  It's a proven
model for shipping high quality, useful code, and sometimes is even one of
the best aspects of a project.  That said, it's quite a bit of work just to
get going and someone would have to manage that.

Jon

[1] https://github.com/rust-lang-nursery


On Sun, May 14, 2017 at 9:03 PM Jeff Jirsa  wrote:

> On Fri, May 12, 2017 at 9:31 PM, J. D. Jordan 
> wrote:
>
> > In tree would foster more committers which is a good thing.
>
>
> Definitely.
>
> But I also agree that not being able to actually run unit tests is a bad
> > thing. What if we asked people who want to contribute these types of
> > optimizations to provide the ASF with a Jenkins slave we could test them
> > on, if they want them in tree?
> >
>
> I think SOME FORM of jenkins/unit/dtests need to exist, whether it's ASF
> puppet controlled or test output explicitly provided by the maintainer.
>
>
> > Otherwise one good thing about out of tree is that the maintainer can
> > specify "this plugin has been tested and works with Apache Cassandra
> > version X.Y.Z". If it is in tree it is assumed it will work. If it is out
> > of tree then the plugin can more easily notify a user what version it was
> > last tested with.  And users won't be surprised if they upgrade and
> things
> > break.
> >
>
> My main desire is that I'd like to see us do better at helping third party
> contributors be successful in contributing, and to me that means something
> more official. I like the spark packages model. I like the apache httpd
> model (with LOTS of official extensions in-tree, but a lot externally
> packaged as well). I'm not a fan of telling people to build and distribute
> their own JARs - it doesn't help the contributors, it doesn't help the
> users, and it doesn't help the project.
>
> - Jeff
>

Re: Integrating vendor-specific code and developing plugins

[Jonathan Haddad]

In accordance with the idea that the codebase should be better tested, it
seems to me like things shouldn't be added that aren't testable.  If
there's a million unit tests that are insanely comprehensive but for some
reason can never be run, they serve exactly the same value as no tests.

It may be better to figure out how to foster a plugin ecosystem, which is a
bit better than "there's an API go deal with it".  This is what Spark is
doing and it seems like a pretty reasonable approach to me:
https://spark-packages.org/

On Fri, May 12, 2017 at 9:03 PM Jeff Jirsa  wrote:

> I think the status quo is insufficient - even if it doesn't go in tree, we
> should do more than just say "the API exists, ship your own jar"
>
> What's the real risk of having it in tree? We break it because nobody can
> test it? How's that any worse than breaking it outside the tree? Finger
> pointing?
>
> --
> Jeff Jirsa
>
>
> > On May 12, 2017, at 12:25 PM, Jason Brown  wrote:
> >
> > I agree the plugins route is the safest and best. However, we already
> have
> > platform-specific code in-tree that is semi-unmaintained: the Windows
> > support. To Sylvain's point, I have little to no idea if I'm going to
> break
> > the Windows builds as I don't have access to a Windows machine, nor are
> we
> > as a community (as best as I can tell) actively running unit tests or
> > dtests on Windows.
> >
> > Further, we support snitches for clouds I suspect we don't typically
> > run/test on, as well: CloudstackSnitch, GoogleCloudSnitch.
> >
> > This being said, I don't think we should remove support for Windows or
> > those snitches. Instead, what I think would be more beneficial, and
> > certainly more reflecting the Apache Way, is to see if someone in the
> > community would be willing to maintain those components. Looking at
> another
> > Apache project, Mesos has an interesting breakdown of maintainers for
> > specific components [1]. We might consider adopting a similar idea for
> > platforms/OSes/architectures/whatevers.
> >
> > As for where to put the custom code, there's a few different options:
> >
> > bare minimum: we should have docs pointing to all known third party
> > implementations of pluggable interfaces
> > slightly more involved: contrib/ section of third-party contributed
> plugins
> > even more involved: in tree like gcp / aws snitches
> >
> > I'm not really thrilled on the contribs repo, and in-tree certainly has
> > drawbacks, as well. As I initially stated, it can be on a case-by-case
> > basis.
> >
> > Honestly, I don't want to push away contributors if they can add
> something
> > to the project - as long as it is maintainable.
> >
> > Thanks,
> >
> > -Jason
> >
> >
> > [1] https://mesos.apache.org/documentation/latest/committers/
> >
> > On Fri, May 12, 2017 at 4:35 AM, Sylvain Lebresne 
> > wrote:
> >
> >> On Fri, May 12, 2017 at 12:29 AM, Jason Brown 
> >> wrote:
> >>
> >>> Hey all,
> >>>
> >>> I'm on-board with what Rei is saying. I think we should be open to, and
> >>> encourage, other platforms/architectures for integration. Of course, it
> >>> will come down to specific maintainers/committers to do the testing and
> >>> verification on non-typical platforms. Hopefully those maintainers will
> >>> also contribute to other parts of the code base, as well, so I see
> this as
> >>> another way to bring more folks into the project.
> >>>
> >>
> >> Without going so far as to say we shouldn't merge any
> >> platform/architecture/vendor specific code ever and for no reason, I
> >> personally think we should avoid doing so as much as practical and
> >> encourage the "plugins" route instead. It's just much cleaner imo on
> >> principle and amounts to good software development hygiene.
> >>
> >> I don't want to not be able to commit some changes because it breaks the
> >> build because there is code for X number of super specific
> >> platform/architecture I don't have access to/don't know anything about
> >> and the maintainers are on vacation or hasn't been reachable in a while.
> >> And what if such maintainer do go away? Sure we can have some "process"
> >> to remove the code in such cases, but why add that burden on us? Plus
> >> we, the Apache Cassandra project, would still be seen as the ones that
> >> drop support for said platform/architecture even though we really have
> >> no choice if it's something we don't have access to anyway.
> >>
> >> And sure, I'm painting a bleak picture here, and we would probably have
> >> none of those problems in most cases. But if we do start encourage
> >> actual merge of such code, you can be sure we'll have some of those
> >> problems at some point.
> >>
> >> Encouraging plugins have imo pretty much all the benefits with none of
> >> the risks. In particular, I'm unconvinced that someone will be
> >> much more likely to meaningfully contribute to other part of the code
> >> if his "plugins" is 

Re: Guidelines on testing

[Jonathan Haddad]

+1

On Tue, Apr 25, 2017 at 2:21 AM Stefan Podkowinski  wrote:

> I don't see any reasons not to make this part of our guidelines. The
> idea of having a list of what should be tested in each kind of test
> makes sense. I also like the examples how to improve tests dealing with
> global state.
>
> Some of the integration test cases, such as "dry
> start"/"restart"/"shutdown"/"upgrade", could use some further
> description and how-to examples. Are there any existing tests we can
> link for reference?
>
> We also already have a testing related page in our documentation:
> http://cassandra.apache.org/doc/latest/development/testing.html
> Not sure if it would make sense to merge or create an additional document.
>
>
> On 24.04.2017 18:13, Blake Eggleston wrote:
> > About a month ago, in the ‘Code quality, principles and rules’ thread,
> I’d proposed adding some testing standards to the project in lieu of
> revisiting the idea of removing singletons. The idea was that we could
> drive incremental improvement of the test coverage and testability
> situation that could be applied in day to day work. I’ve pushed a first
> draft to my repo here:
> >
> > https://github.com/bdeggleston/cassandra/blob/testing-doc/TESTING.md
> >
> > Please take a look and let me know what you think. With the blessing of
> the pmc, I’d like this, or something like it, to be adopted as the
> reference for contributors and reviewers when deciding if a contribution is
> properly tested.
> >
> > Blake
> >
>

Re: [VOTE] self-assignment of jira tickets

[Jonathan Haddad]

Non binding +1
On Wed, Mar 29, 2017 at 7:22 AM Jeff Jirsa  wrote:

> +1
>
> --
> Jeff Jirsa
>
>
> > On Mar 29, 2017, at 6:21 AM, Jason Brown  wrote:
> >
> > Hey all,
> >
> > Following up my thread from a week or two ago (
> >
> https://lists.apache.org/thread.html/0665f40c7213654e99817141972c003a2131aba7a1c63d6765db75c5@%3Cdev.cassandra.apache.org%3E
> ),
> > I'd like to propose a vote to change to allow any potential contributor
> to
> > assign a jira to themselves without needing to be added to the
> contributors
> > group first.
> >
> > https://issues.apache.org/jira/browse/INFRA-11950 is an example of how
> to
> > get this done with INFRA.
> >
> > Vote will be open for 72 hours.
> >
> > Thanks,
> >
> > -Jason Brown
>

splitting CQL parser & spec into separate repo

[Jonathan Haddad]

I created CASSANDRA-13284 a few days ago with the intent of starting a
discussion around the topic of breaking the CQL parser out into a separate
project.  I see a few benefits to doing it and was wondering what the folks
here thought as well.

First off, the Java CQL parser would obviously continue to be the reference
parser.  I'd love to see other languages have CQL parsers as well, but the
intent here isn't for the OSS C* team to be responsible for maintaining
that.  My vision here is simply the ability to have some high level
CQLParser.parse(statement) call that returns the parse tree, nothing more.

It would be nice to be able to leverage that parser in other projects such
as IDEs, code gen tools, etc.  It would be outstanding to be able to create
the parser tests in such a way that they can be referenced by other parsers
in other languages.  Yay code reuse.  It also has the benefit of making the
codebase a little more modular and a bit easier to understand.

Thoughts?

Jon

Re: [VOTE] Ask Infra to move github notification emails to pr@

[Jonathan Haddad]

+1
On Mon, Mar 20, 2017 at 6:33 PM Jason Brown  wrote:

> +1
> On Mon, Mar 20, 2017 at 18:21 Anthony Grasso 
> wrote:
>
> > +1
> >
> > On 21 March 2017 at 09:32, Jeff Jirsa  wrote:
> >
> > > There's no reason for the dev list to get spammed everytime there's a
> > > github PR. We know most of the time we prefer JIRAs for real code PRs,
> > but
> > > with docs being in tree and low barrier to entry, we may want to accept
> > > docs through PRs ( see https://issues.apache.org/
> > > jira/browse/CASSANDRA-13256
> > > , and comment on it if you disagree).
> > >
> > > To make that viable, we should make it not spam dev@ with every
> comment.
> > > Therefore I propose we move github PR comments/actions to pr@ so as
> > > not to clutter the dev@ list.
> > >
> > > Voting to remain open for 72 hours.
> > >
> > > - Jeff
> > >
> >
>

Re: [VOTE] Ask Infra to move github notification emails to commits@

[Jonathan Haddad]

+1

On Mon, Mar 20, 2017 at 3:02 PM Brandon Williams  wrote:

> +1, we've had to explain this a thousand times here.
>
> On Mon, Mar 20, 2017 at 5:00 PM, Jeff Jirsa  wrote:
>
> > There's no reason for the dev list to get spammed everytime there's a
> > github PR. We know most of the time we prefer JIRAs for real code PRs,
> but
> > with docs being in tree and low barrier to entry, we may want to accept
> > docs through PRs ( see https://issues.apache.org/
> > jira/browse/CASSANDRA-13256
> > , and comment on it if you disagree).
> >
> > To make that viable, we should make it not spam dev@ with every comment.
> > Therefore I propose we move github PR comments/actions to commits@ so as
> > not to clutter the dev@ list.
> >
> > Voting to remain open for 72 hours.
> >
> > - Jeff
> >
>

Re: Can we kill the wiki?

[Jonathan Haddad]

If there was a single instance of collaborative design in the wiki I would
agree with you.

Google docs have been used a bit recently and have worked well.
On Sun, Mar 19, 2017 at 8:17 AM Edward Capriolo <edlinuxg...@gmail.com>
wrote:

> Wikis are still good for collaberative design etc. Its a burden to edit the
> docs and its not the place for all info.
>
> On Friday, March 17, 2017, Murukesh Mohanan <murukesh.moha...@gmail.com>
> wrote:
>
> > I wonder if the recent influx has anything to do with GSoC. The student
> > application period begins in a few days. I don't see any Cassandra issues
> > on the GSoC ideas list, though.
> >
> > On Sat, 18 Mar 2017 at 10:40 Anthony Grasso <anthony.gra...@gmail.com
> > <javascript:;>>
> > wrote:
> >
> > +1 to killing the wiki as well. If that is not possible, we should at
> least
> > put a note on there saying it is deprecated and point people to the new
> > docs.
> >
> > On 18 March 2017 at 08:09, Jonathan Haddad <j...@jonhaddad.com
> > <javascript:;>> wrote:
> >
> > > +1 to killing the wiki.
> > >
> > > On Fri, Mar 17, 2017 at 2:08 PM Blake Eggleston <beggles...@apple.com
> > <javascript:;>>
> > > wrote:
> > >
> > > > With CASSANDRA-8700, docs were moved in tree, with the intention that
> > > they
> > > > would replace the wiki. However, it looks like we’re still getting
> > > regular
> > > > requests to edit the wiki. It seems like we should be directing these
> > > folks
> > > > to the in tree docs and either disabling edits for the wiki, or just
> > > > removing it entirely, and replacing it with a link to the hosted
> docs.
> > > I'd
> > > > prefer we just remove it myself, makes things less confusing for
> > > newcomers.
> > > >
> > > > Does that seem reasonable to everyone?
> > >
> >
> > --
> >
> > Murukesh Mohanan,
> > Yahoo! Japan
> >
>
>
> --
> Sorry this was sent from mobile. Will do less grammar and spell check than
> usual.
>

Re: Code quality, principles and rules

[Jonathan Haddad]

I'd like to think that if someone refactors existing code, making it more
testable (with tests, of course) it should be acceptable on it's own
merit.  In fact, in my opinion it sometimes makes more sense to do these
types of refactorings for the sole purpose of improving stability and
testability as opposed to mixing them with features.

You referenced the issue I fixed in one of the early emails.  The fix
itself was a couple lines of code.  Refactoring the codebase to make it
testable would have been a huge effort.  I wish I had time to do it.  I
created CASSANDRA-13007 as a follow up with the intent of working on
compaction from a purely architectural standpoint.  I think this type of
thing should be done throughout the codebase.

Removing the singletons is a good first step, my vote is we just rip off
the bandaid, do it, and move forward.

On Fri, Mar 17, 2017 at 2:20 PM Edward Capriolo 
wrote:

> On Fri, Mar 17, 2017 at 2:31 PM, Jason Brown  wrote:
>
> > To François's point about code coverage for new code, I think this makes
> a
> > lot of sense wrt large features (like the current work on
> 8457/12229/9754).
> > It's much simpler to (mentally, at least) isolate those changed sections
> > and it'll show up better in a code coverage report. With small patches,
> > that might be harder to achieve - however, as the patch should come with
> > *some* tests (unless it's a truly trivial patch), it might just work
> itself
> > out.
> >
> > On Fri, Mar 17, 2017 at 11:19 AM, Jason Brown 
> > wrote:
> >
> > > As someone who spent a lot of time looking at the singletons topic in
> the
> > > past, Blake brings a great perspective here. Figuring out and
> > communicating
> > > how best to test with the system we have (and of course incrementally
> > > making that system easier to work with/test) seems like an achievable
> > goal.
> > >
> > > On Fri, Mar 17, 2017 at 10:17 AM, Edward Capriolo <
> edlinuxg...@gmail.com
> > >
> > > wrote:
> > >
> > >> On Fri, Mar 17, 2017 at 12:33 PM, Blake Eggleston <
> beggles...@apple.com
> > >
> > >> wrote:
> > >>
> > >> > I think we’re getting a little ahead of ourselves talking about DI
> > >> > frameworks. Before that even becomes something worth talking about,
> > we’d
> > >> > need to have made serious progress on un-spaghettifying Cassandra in
> > the
> > >> > first place. It’s an extremely tall order. Adding a DI framework
> right
> > >> now
> > >> > would be like throwing gasoline on a raging tire fire.
> > >> >
> > >> > Removing singletons seems to come up every 6-12 months, and usually
> > >> > abandoned once people figure out how difficult they are to remove
> > >> properly.
> > >> > I do think removing them *should* be a long term goal, but we really
> > >> need
> > >> > something more immediately actionable. Otherwise, nothing’s going to
> > >> > happen, and we’ll be having this discussion again in a year or so
> when
> > >> > everyone’s angry that Cassandra 5.0 still isn’t ready for
> production,
> > a
> > >> > year after it’s release.
> > >> >
> > >> > That said, the reason singletons regularly get brought up is because
> > >> doing
> > >> > extensive testing of anything in Cassandra is pretty much
> impossible,
> > >> since
> > >> > the code is basically this big web of interconnected global state.
> > >> Testing
> > >> > anything in isolation can’t be done, which, for a distributed
> > database,
> > >> is
> > >> > crazy. It’s a chronic problem that handicaps our ability to release
> a
> > >> > stable database.
> > >> >
> > >> > At this point, I think a more pragmatic approach would be to draft
> and
> > >> > enforce some coding standards that can be applied in day to day
> > >> development
> > >> > that drive incremental improvement of the testing and testability of
> > the
> > >> > project. What should be tested, how it should be tested. How to
> write
> > >> new
> > >> > code that talks to the rest of Cassandra and is testable. How to fix
> > >> bugs
> > >> > in old code in a way that’s testable. We should also have some
> > >> guidelines
> > >> > around refactoring the wildly untested sections, how to get started,
> > >> what
> > >> > to do, what not to do, etc.
> > >> >
> > >> > Thoughts?
> > >>
> > >>
> > >> To make the conversation practical. There is one class I personally
> > really
> > >> want to refactor so it can be tested:
> > >>
> > >> https://github.com/apache/cassandra/blob/trunk/src/java/org/
> > >> apache/cassandra/net/OutboundTcpConnection.java
> > >>
> > >> There is little coverage here. Questions like:
> > >> what errors cause the connection to restart?
> > >> when are undropable messages are dropped?
> > >> what happens when the queue fills up?
> > >> Infamous throw new AssertionError(ex); (which probably bubble up to
> > >> nowhere)
> > >> what does the COALESCED strategy do in case XYZ.
> > >> A nifty label (wow a label you just never see those much!)
> > >> outer:
> > >> 

Re: Can we kill the wiki?

[Jonathan Haddad]

+1 to killing the wiki.

On Fri, Mar 17, 2017 at 2:08 PM Blake Eggleston 
wrote:

> With CASSANDRA-8700, docs were moved in tree, with the intention that they
> would replace the wiki. However, it looks like we’re still getting regular
> requests to edit the wiki. It seems like we should be directing these folks
> to the in tree docs and either disabling edits for the wiki, or just
> removing it entirely, and replacing it with a link to the hosted docs. I'd
> prefer we just remove it myself, makes things less confusing for newcomers.
>
> Does that seem reasonable to everyone?