incrementally acquiring tokens

[Jonathan Haddad]

I've been thinking a bit about the problem of fat nodes (say, 20gb per
node).  My understanding is that the amount of overhead of adding a new
node into the ring is massive with fat nodes due to the fact that you have
to stream in 20TB which takes forever.

In a scenario where a given node only has a single token, my assumption is
this forces Cassandra to send the entire token range to the new node.

As a potential approach, would it be possible for a node to incrementally
acquire tokens, and as a result incrementally stream?  You could have a
node serving requests after acquiring 1 token, and it would gradually take
ownership of more and more of the ring as it bootstraps.

Thoughts?
Jon

Re: 3.0 and the Cassandra release process

[Jonathan Haddad]

If every other release is a bug fix release, would the versioning go:

3.1.0 -- feature release
3.1.1 -- bug fix release

Eventually it seems like it might be possible to be able to push out a bug
fix release more frequently than once a month?

On Wed, Mar 18, 2015 at 7:59 AM Josh McKenzie josh.mcken...@datastax.com
wrote:

 +1

 On Wed, Mar 18, 2015 at 7:54 AM, Jake Luciani jak...@gmail.com wrote:

  +1
 
  On Tue, Mar 17, 2015 at 5:06 PM, Jonathan Ellis jbel...@gmail.com
 wrote:
   Cassandra 2.1 was released in September, which means that if we were on
   track with our stated goal of six month releases, 3.0 would be done
 about
   now.  Instead, we haven't even delivered a beta.  The immediate cause
  this
   time is blocking for 8099
   https://issues.apache.org/jira/browse/CASSANDRA-8099, but the
 reality
  is
   that nobody should really be surprised.  Something always comes up --
  we've
   averaged about nine months since 1.0, with 2.1 taking an entire year.
  
   We could make theory align with reality by acknowledging, if nine
 months
   is our 'natural' release schedule, then so be it.  But I think we can
 do
   better.
  
   Broadly speaking, we have two constituencies with Cassandra releases:
  
   First, we have the users who are building or porting an application on
   Cassandra.  These users want the newest features to make their job
  easier.
   If 2.1.0 has a few bugs, it's not the end of the world.  They have time
  to
   wait for 2.1.x to stabilize while they write their code.  They would
 like
   to see us deliver on our six month schedule or even faster.
  
   Second, we have the users who have an application in production.  These
   users, or their bosses, want Cassandra to be as stable as possible.
   Assuming they deploy on a stable release like 2.0.12, they don't want
 to
   touch it.  They would like to see us release *less* often.  (Because
 that
   means they have to do less upgrades while remaining in our backwards
   compatibility window.)
  
   With our current big release every X months model, these users' needs
  are
   in tension.
  
   We discussed this six months ago, and ended up with this:
  
   What if we tried a [four month] release cycle, BUT we would guarantee
  that
   you could do a rolling upgrade until we bump the supermajor version?
 So
  2.0
   could upgrade to 3.0 without having to go through 2.1.  (But to go to
  3.1
   or 4.0 you would have to go through 3.0.)
  
  
   Crucially, I added
  
   Whether this is reasonable depends on how fast we can stabilize
 releases.
   2.1.0 will be a good test of this.
  
  
   Unfortunately, even after DataStax hired half a dozen full-time test
   engineers, 2.1.0 continued the proud tradition of being unready for
   production use, with wait for .5 before upgrading once again looking
  like
   a good guideline.
  
   I’m starting to think that the entire model of “write a bunch of new
   features all at once and then try to stabilize it for release” is
 broken.
   We’ve been trying that for years and empirically speaking the evidence
 is
   that it just doesn’t work, either from a stability standpoint or even
  just
   shipping on time.
  
   A big reason that it takes us so long to stabilize new releases now is
   that, because our major release cycle is so long, it’s super tempting
 to
   slip in “just one” new feature into bugfix releases, and I’m as guilty
 of
   that as anyone.
  
   For similar reasons, it’s difficult to do a meaningful freeze with big
   feature releases.  A look at 3.0 shows why: we have 8099 coming, but we
   also have significant work done (but not finished) on 6230, 7970, 6696,
  and
   6477, all of which are meaningful improvements that address
 demonstrated
   user pain.  So if we keep doing what we’ve been doing, our choices are
 to
   either delay 3.0 further while we finish and stabilize these, or we
 wait
   nine months to a year for the next release.  Either way, one of our
   constituencies gets disappointed.
  
   So, I’d like to try something different.  I think we were on the right
   track with shorter releases with more compatibility.  But I’d like to
  throw
   in a twist.  Intel cuts down on risk with a “tick-tock” schedule for
 new
   architectures and process shrinks instead of trying to do both at once.
  We
   can do something similar here:
  
   One month releases.  Period.  If it’s not done, it can wait.
   *Every other release only accepts bug fixes.*
  
   By itself, one-month releases are going to dramatically reduce the
   complexity of testing and debugging new releases -- and bugs that do
 slip
   past us will only affect a smaller percentage of users, avoiding the
 “big
   release has a bunch of bugs no one has seen before and pretty much
  everyone
   is hit by something” scenario.  But by adding in the second rule, I
 think
   we have a real chance to make a quantum leap here: stable,
  production-ready
   releases every two months.
  
   So here is my 

Re: 3.0 and the Cassandra release process

[Jonathan Haddad]

In this tick tock cycle, is there still a long term release that's
maintained, meant for production?  Will bug fixes be back ported to 3.0
(stable) with new stuff going forward to 3.x?

On Thu, Mar 26, 2015 at 6:50 AM Aleksey Yeschenko alek...@apache.org
wrote:

 Hey Jason. I think pretty much everybody is on board with:

 1) A monthly release cycle
 2) Keeping trunk releasable all the times

 And that’s what my personal +1 was for.

 The tick-tock mechanism details and bug fix policy for the maintained
 stable lines should be fleshed out before we proceed. I believe that once
 they are explained better, the concerns will mostly, or entirely, go away.

 --
 AY

 On Mon, Mar 23, 2015 at 11:15 PM, Jason Brown jasedbr...@gmail.com
 wrote:

  Hey all,
 
  I had a hallway conversation with some folks here last week, and they
  expressed some concerns with this proposal. I will not attempt to
 summarize
  their arguments as I don't believe I could do them ample justice, but I
  strongly encouraged those individuals to speak up and be heard on this
  thread (I know they are watching!).
 
  Thanks,
 
  -Jason
 
  On Mon, Mar 23, 2015 at 6:32 AM, 曹志富 cao.zh...@gmail.com wrote:
 
   +1
  
   --
   Ranger Tsao
  
   2015-03-20 22:57 GMT+08:00 Ryan McGuire r...@datastax.com:
  
I'm taking notes from the infrastructure doc and wrote down some
 action
items for my team:
   
https://gist.github.com/EnigmaCurry/d53eccb55f5d0986c976
   
   
--
   
[image: datastax_logo.png] http://www.datastax.com/
   
Ryan McGuire
   
Software Engineering Manager in Test | r...@datastax.com
   
[image: linkedin.png] https://www.linkedin.com/in/enigmacurry
  [image:
twitter.png] http://twitter.com/enigmacurry
http://github.com/enigmacurry
   
   
On Thu, Mar 19, 2015 at 1:08 PM, Ariel Weisberg 
ariel.weisb...@datastax.com
 wrote:
   
 Hi,

 I realized one of the documents we didn't send out was the
   infrastructure
 side changes I am looking for. This one is maybe a little rougher
 as
  it
was
 the first one I wrote on the subject.



   
  
  https://docs.google.com/document/d/1Seku0vPwChbnH3uYYxon0UO-
 b6LDtSqluZiH--sWWi0/edit?usp=sharing

 The goal is to have infrastructure that gives developers as close
 to
 immediate feedback as possible on their code before they merge.
   Feedback
 that is delayed to after merging to trunk should come in a day or
 two
   and
 there is a product owner (Michael Shuler) responsible for making
 sure
that
 issues are addressed quickly.

 QA is going to help by providing developers with a better tools for
writing
 higher level functional tests that explore all of the functions
   together
 along with the configuration space without developers having to do
  any
work
 other then plugging in functionality to exercise and then validate
 something specific. This kind of harness is hard to get right and
  make
 reliable and expressive so they have their work cut out for them.

 It's going to be an iterative process where the tests improve as
 new
   work
 introduces missing coverage and as bugs/regressions drive the
introduction
 of new tests. The monthly retrospective (planning on doing that
 first
   of
 the month) is also going to help us refine the testing and
  development
 process.

 Ariel

 On Thu, Mar 19, 2015 at 7:23 AM, Jason Brown jasedbr...@gmail.com
 
wrote:

  +1 to this general proposal. I think the time has finally come
 for
  us
to
  try something new, and this sounds legit. Thanks!
 
  On Thu, Mar 19, 2015 at 12:49 AM, Phil Yang ud1...@gmail.com
   wrote:
 
   Can I regard the odd version as the development preview and
 the
even
   version as the production ready?
  
   IMO, as a database infrastructure project, stable is more
   important
  than
   other kinds of projects. LTS is a good idea, but if we don't
   support
   non-LTS releases for enough time to fix their bugs, users on
   non-LTS
   release may have to upgrade a new major release to fix the bugs
  and
may
   have to handle some new bugs by the new features. I'm afraid
 that
   eventually people would only think about the LTS one.
  
  
   2015-03-19 8:48 GMT+08:00 Pavel Yaskevich pove...@gmail.com:
  
+1
   
On Wed, Mar 18, 2015 at 3:50 PM, Michael Kjellman 
mkjell...@internalcircle.com wrote:
   
 For most of my life I’ve lived on the software bleeding
 edge
   both
 personally and professionally. Maybe it’s a personal
  weakness,
but
 I
guess
 I get a thrill out of the problem solving aspect?

 Recently I came to a bit of an epiphany — the closer I keep
  to
the
   daily
 build — generally the happier 

Re: DateTieredCompactionStrategy and static columns

[Jonathan Haddad]

If you want it in a separate sstable, just use a separate table.  There's
nothing that warrants making the codebase more complex to accomplish
something it already does.

On Thu, Apr 30, 2015 at 5:07 PM graham sanderson gra...@vast.com wrote:

 Anyone here have an opinion; how realistic would it be to have a separate
 memtable/sstable for static columns?

 Begin forwarded message:

 *From: *Jonathan Haddad j...@jonhaddad.com
 *Subject: **Re: DateTieredCompactionStrategy and static columns*
 *Date: *April 30, 2015 at 3:55:46 PM CDT
 *To: *u...@cassandra.apache.org
 *Reply-To: *u...@cassandra.apache.org


 I suspect this will kill the benefit of DTCS, but haven't tested it to be
 100% here.

 The benefit of DTCS is that sstables are selected for compaction based on
 the age of the data, not their size.  When you mix TTL'ed data and non
 TTL'ed data, you end up screwing with the drop the entire SSTable
 optimization.  I don't believe this is any different just because you're
 mixing in static columns.  What I think will happen is you'll end up with
 an sstable that's almost entirely TTL'ed with a few static columns that
 will never get compacted or dropped.  Pretty much the worst scenario I can
 think of.



 On Thu, Apr 30, 2015 at 11:21 AM graham sanderson gra...@vast.com wrote:

 I have a potential use case I haven’t had a chance to prototype yet,
 which would normally be a good candidate for DTCS (i.e. data delivered in
 order and a fixed TTL), however with every write we’d also be updating some
 static cells (namely a few key/values in a static maptext.text CQL
 column). There could also be explicit deletes of keys in the static map,
 though that’s not 100% necessary.

 Since those columns don’t have TTL, without reading thru the code code
 and/or trying it, I have no idea what effect this has on DTCS (perhaps it
 needs to use separate sstables for static columns). Has anyone tried this.
 If not I eventually will and will report back.

Re: DateTieredCompactionStrategy and static columns

[Jonathan Haddad]

I think what Benedict has described feels very much like a very specialized
version of the following:

1. Updates to different tables in a batch become atomic if the node is a
replica for the partition
2. Supporting Inner joins if the partition key is the same in both tables.

I'd rather see join support personally :)

Jon

On Fri, May 1, 2015 at 6:38 AM graham sanderson gra...@vast.com wrote:

 I 100% agree with Benedict, but just to be clear about my use case

 1) We have state of lets say real estate listings
 2) We get field level deltas for them
 3) Previously we would store the base state all the deltas in partition
 and roll them up from the beginning of time (this was a prototype and silly
 since there was no expiration strategy)
 4) Preferred plan is to keep current state in a static map (i.e. one delta
 field only updates one cell) - we are MVCC but in the common case the
 latest version will be what we want
 5) However we require history, so we’d use the partition to keep TTL
 deltas going backwards from the now state - this seems like a common
 pattern people would want. Note also that sometimes we might need to apply
 reverse deltas if C* is ahead of our SOLR indexes

 The static columns and the regular columns ARE completely different in
 behavior/lifecycle, so I’d definitely vote for them being treated as such.


  On May 1, 2015, at 7:27 AM, Benedict Elliott Smith 
 belliottsm...@datastax.com wrote:
 
 
  How would it be different from creating an actual real extra table
 instead?
 
 
  There's nothing that warrants making the codebase more complex to
  accomplish something it already does.
 
 
  As far as I was aware, the only point of static columns was to support
 the
  thrift ability to mutate and read them in the same expression, with
  atomicity and isolation. As to whether or not it is more complex, I'm not
  at all convinced that it would be. We have had a lot of unexpected
 special
  casing added to ensure they behave correctly (e.g. paging is broken), and
  have complicated the comparison/slice logic to accommodate them, so that
 it
  is harder to reason about (and to optimise). They also have very
 different
  compaction characteristics, so the complexity on the user is increased
  without their necessarily realising it. All told, it introduces a lot
 more
  subtlety of behaviour than there would be with a separate set of
 sstables,
  or perhaps a separate file attached to each sstable.
 
  Of course, we've already implemented it as a specialisation of the
  slice/comparator, I think because it seemed like the least frictional
 path
  to do so, but that doesn't mean it is the least complex. It does mean
 it's
  the least work (assuming we're now on top of the bugs), which is its own
  virtue.
 
  There are some advantages to having them managed separately, and
 advantages
  to having them combined. Combined, for small partitions, they can be read
  in the same seek. However for large partitions this is no longer true,
 and
  we may behave much worse by polluting the page cache with lots of
 unwanted
  data that is adjacent to the static columns. If they were managed
  separately, the page cache would be populated mostly with other static
  columns, which may be more likely of use. We could quite easily have a
  static column cache, also, and completely avoid merging them. Or at
 least
  we could easily read them with collectTimeOrderedData instead of
  collectAllData semantics.
 
  All told, it certainly isn't a terrible idea, and shouldn't be dismissed
 so
  readily. Personally I think in the long run whether or not we manage
 static
  columns together with non-static columns is dependent on if we intend to
  add tiered static columns (i.e., if each level of clustering component
  can have columns associated with it). If we do, we should definitely keep
  it all inline. If not, it probably permits a lot better behaviour to
  separate them, since it's easier to reason about and improve their
 distinct
  characteristics.
 
 
  On Fri, May 1, 2015 at 1:24 AM, graham sanderson gra...@vast.com
 wrote:
 
  Well you lose the atomicity and isolation, but in this case that is
  probably fine
 
  That said, in every interaction I’ve had with static columns, they seem
 to
  be an odd duck (e.g. adding or complicating range slices), perhaps
 worthy
  of their own code path and sstables. Just food for thought.
 
  On Apr 30, 2015, at 7:13 PM, Jonathan Haddad j...@jonhaddad.com
 wrote:
 
  If you want it in a separate sstable, just use a separate table.
 There's
  nothing that warrants making the codebase more complex to accomplish
  something it already does.
 
  On Thu, Apr 30, 2015 at 5:07 PM graham sanderson gra...@vast.com
  wrote:
 
  Anyone here have an opinion; how realistic would it be to have a
  separate
  memtable/sstable for static columns?
 
  Begin forwarded message:
 
  *From: *Jonathan Haddad j...@jonhaddad.com
  *Subject: **Re: DateTieredCompactionStrategy and static

Re: NewBie Question

[Jonathan Haddad]

Maybe some brave soul will document the 3.0 on disk format as part of
https://issues.apache.org/jira/browse/CASSANDRA-8700.

On Wed, Jun 15, 2016 at 7:02 AM Christopher Bradford 
wrote:

> Consider taking a look at Aaron Morton's dive into the C* 3.0 storage
> engine.
>
>
> http://thelastpickle.com/blog/2016/03/04/introductiont-to-the-apache-cassandra-3-storage-engine.html
>
> On Wed, Jun 15, 2016 at 9:38 AM Jim Witschey 
> wrote:
>
> > > http://wiki.apache.org/cassandra/ArchitectureSSTable
> >
> > Be aware that this page hasn't been updated since 2013, so it doesn't
> > reflect any changes to the SSTable format since then, including the
> > new storage engine introduced in 3.0 (see CASSANDRA-8099).
> >
> > That said, I believe the linked Apache wiki page is the best
> > documentation for the format. Unfortunately, if you want a better or
> > more current understanding, you'll have to read the code and read some
> > SSTables.
> >
>

Re: NewBie Question

[Jonathan Haddad]

Since the specs change with the code id argue everything belongs in the git
repo including deep technical specs.
On Wed, Jun 15, 2016 at 10:26 AM J. D. Jordan <jeremiah.jor...@gmail.com>
wrote:

> I think high level concepts of how data is stored should be in user facing
> documentation. Storage format affects schema design. But low level
> specifics should be kept to contributor documentation.
>
> > On Jun 15, 2016, at 12:20 PM, Jonathan Ellis <jbel...@gmail.com> wrote:
> >
> > I agree that it should be documented but I don't think it should be in
> user
> > level docs. Let's keep it in the wiki for contributors.
> >> On Jun 15, 2016 7:04 PM, "Jonathan Haddad" <j...@jonhaddad.com> wrote:
> >>
> >> Definitely required reading for anyone getting into it, plus Aaron's
> post.
> >> I think ideally one day something like this should live in the docs:
> >>
> >> https://www.postgresql.org/docs/9.0/static/storage-page-layout.html
> >>
> >> Definitely not suggesting everyone drop what they're doing and document
> the
> >> 3.0 format but ultimately I think there should be specs for any wire
> >> protocol or on disk format.  For wire protocols network diagrams can
> easily
> >> be generated via the nwdiag plugin.
> >>
> >> On Wed, Jun 15, 2016 at 9:55 AM Michael Kjellman <
> >> mkjell...@internalcircle.com> wrote:
> >>
> >>> This was forwarded to me yesterday... a helpful first step
> >>> https://github.com/apache/cassandra/blob/cassandra-3.0.0/guide_8099.md
> >>>
> >>>> On Jun 15, 2016, at 9:54 AM, Jonathan Haddad <j...@jonhaddad.com>
> >> wrote:
> >>>>
> >>>> Maybe some brave soul will document the 3.0 on disk format as part of
> >>>> https://issues.apache.org/jira/browse/CASSANDRA-8700.
> >>>>
> >>>> On Wed, Jun 15, 2016 at 7:02 AM Christopher Bradford <
> >>> bradfor...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Consider taking a look at Aaron Morton's dive into the C* 3.0 storage
> >>>>> engine.
> >>
> http://thelastpickle.com/blog/2016/03/04/introductiont-to-the-apache-cassandra-3-storage-engine.html
> >>>>>
> >>>>> On Wed, Jun 15, 2016 at 9:38 AM Jim Witschey <
> >> jim.witsc...@datastax.com
> >>>>
> >>>>> wrote:
> >>>>>
> >>>>>>> http://wiki.apache.org/cassandra/ArchitectureSSTable
> >>>>>>
> >>>>>> Be aware that this page hasn't been updated since 2013, so it
> doesn't
> >>>>>> reflect any changes to the SSTable format since then, including the
> >>>>>> new storage engine introduced in 3.0 (see CASSANDRA-8099).
> >>>>>>
> >>>>>> That said, I believe the linked Apache wiki page is the best
> >>>>>> documentation for the format. Unfortunately, if you want a better or
> >>>>>> more current understanding, you'll have to read the code and read
> >> some
> >>>>>> SSTables.
> >>
>

Re: [Proposal] Mandatory comments

[Jonathan Haddad]

I've been discouraged from contributing to the codebase due to it's lack of
comments, so I +1 this big time.

Some people argue against comments by saying "the code should be self
describing", but that misses the point.  The comments should describe the
*intent* of the code, the problem it's mean to solve, not what it does.
It's hard to look at code with a bug and ask yourself "what is this
supposed to do?", that should have already been answered for you.

On Mon, May 2, 2016 at 9:40 AM Josh McKenzie  wrote:

> Fighting comment atrophy will become a more pressing concern for us in the
> future if we standardize this so we might want to add something about that
> in CodeStyle as well. This is fundamental best-practices behavior for the
> maintainability of a complex code-base with multiple contributors so I'm
> strongly +1 on this.
>
> It would help to have some examples on the CodeStyle page to go along with
> this for both a class javadoc and a method javadoc.
>
> On Mon, May 2, 2016 at 12:26 PM, Sylvain Lebresne 
> wrote:
>
> > There could be disagreement on this, but I think that we are in general
> not
> > very good at commenting Cassandra code and I would suggest that we make a
> > collective effort to improve on this. And to help with that goal, I would
> > like
> > to suggest that we add the following rule to our code style guide
> > (https://wiki.apache.org/cassandra/CodeStyle):
> >   - Every public class and method must have a quality javadoc. That
> > javadoc, on
> > top of describing what the class/method does, should call particular
> > attention to the assumptions that the class/method does, if any.
> >
> > And of course, we'd also start enforcing that rule by not +1ing code
> unless
> > it
> > adheres to this rule.
> >
> > Note that I'm not pretending this arguably simplistic rule will magically
> > make
> > comments perfect, it won't. It's still up to us to write good and
> complete
> > comments, and it doesn't even talk about comments within methods that are
> > important too. But I think some basic rule here would be beneficial and
> > that
> > one feels sensible to me.
> >
> > Looking forward to other's opinions and feedbacks on this proposal.
> >
> > --
> > Sylvain
> >
>

Re: A proposal to move away from Jira-centric development

[Jonathan Haddad]

I don't know about everyone else, but a big deterrent in contributing code
to Cassandra for me (over the last 4 years or so) is the massive amount of
ramp up that needs to happen in order to get started working on something
meaningful.  This comes in a variety of forms - understanding how test
failures aren't actually failures (being corrected now), lack of comments
(for example:
https://github.com/PaytmLabs/cassandra/blob/master/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java),
out of date wiki (now semi retired but still a source of misinformation),
and class names that don't make sense.  Historically there has been a
massive amount of tribal knowledge required to contribute in a significant
way.  Going through JIRA history and asking people who wrote the feature is
the only way to find out what is supposed to be going on.  Let's not forget
the fact that it's a distributed database, so add to all the above issues
the fact that getting this thing right at all is a miraculous achievement.
In order to get more people contributing to the project there needs to be a
significant effort in making it more approachable.

I suspect that as the project continues to move faster, it'll become ever
harder for new contributors to join unless they are paid to work on
Cassandra full time.  Very few people are deploying Tick Tock releases on
purpose - 1 bug fix release doesn't exactly scream reassurance, sorry - so
the folks that are going to scratch their own itch by fixing bugs and
eventually contributing features they need is likely going to drop over
time.  This leaves full time paid positions such as those employed by
DataStax as the logical place to go if you are interested in working on
Cassandra.


On Tue, Aug 16, 2016 at 10:47 AM Benedict Elliott Smith 
wrote:

> This is a much more useful focusing of the discussion, in my opinion.  It
> seemed to me that city hall was focusing on a very narrow definition of
> project health.
>
> I would be the first to say the project need to improve here, but doing so
> will be challenging;  I'm not sure anyone really knows how to go about it.
> Which is why we end up in these local minima of discussions about the
> minutiae of JIRA replication.
>
> What this project really needs, and the board is chomping at the bit about,
> is diversity.  The fact is, right now DataStax does 95% of the substantive
> development on the project, and so they make all the decisions.  As such,
> their internal community outweighs the Apache community.  I will emphasise
> clearly for my ex-colleagues, I'm not making any value judgement about
> this, just clarifying the crux of the discussion that everyone seems to be
> dancing around.
>
> The question is, what can be done about it?  The project needs a lot of new
> highly productive and independent contributors who are capable of
> meaningfully shaping project direction.  The problem is we don't know how
> to achieve that.
>
>
>
> On 16 August 2016 at 17:24, Dennis E. Hamilton 
> wrote:
>
> >
> >
> > > -Original Message-
> > > From: Eric Stevens [mailto:migh...@gmail.com]
> > > Sent: Tuesday, August 16, 2016 06:10
> > > To: dev@cassandra.apache.org
> > > Subject: Re: A proposal to move away from Jira-centric development
> > >
> > > I agree with Benedict that we really shouldn't be getting into a
> > > legalese
> > > debate on this subject, however "it didn't happen" has been brought up
> > > as a
> > > hammer in this conversation multiple times, and I think it's important
> > > that
> > > we put it to rest.  It's pretty clear cut that projects are free to
> > > disregard this advice.  "It didn't happen" is a motto, not a rule.
> > [orcmid]
> >
> > <
> http://community.apache.org/apache-way/apache-project-maturity-model.html
> > >
> >
> > Please read them all, but especially the sections on Community, Consensus
> > Building, and Independence.
> >
> > Apache projects are expected to govern themselves, PMCs are responsible
> > for it, and PMC Chairs (officers of the foundation) are accountable to
> the
> > Board on how the project is striving toward maturity.
> >
> > On occasion, deviations are so notable that there is objection.  It is
> not
> > that folks run around policing the projects.  But there are occasions
> where
> > there are concerns that a project has gone astray.
> >
> > One maturity factor that might not be emphasized enough is
> > Sustainability.  It is about the transparency of project conduct, the
> > inclusiveness of operation and visibility, and the ways that growth and
> > turnover are accommodated.  Since we are looking at mottos, "community
> over
> > code" comes to mind.
> >
> > Project freedom is a bit like the freedom to drive at 100mph on an
> > arterial highway.  Occassionally, the infraction becomes worthy of
> > attention and even a road block and spike strips.
> >
> > While individual preferences are being discussed here, and I agree it is

Re: A proposal to move away from Jira-centric development

[Jonathan Haddad]

(non binding) +1

On Mon, Aug 15, 2016 at 7:23 AM Jonathan Ellis  wrote:

> A long time ago, I was a proponent of keeping most development discussions
> on Jira, where tickets can be self contained and the threadless nature
> helps keep discussions from getting sidetracked.
>
> But Cassandra was a lot smaller then, and as we've grown it has become
> necessary to separate out the signal (discussions of new features and major
> changes) from the noise of routine bug reports.
>
> I propose that we take advantage of the dev list to perform that
> separation.  Major new features and architectural improvements should be
> discussed first here, then when consensus on design is achieved, moved to
> Jira for implementation and review.
>
> I think this will also help with the problem when the initial idea proves
> to be unworkable and gets revised substantially later after much
> discussion.  It can be difficult to figure out what the conclusion was, as
> review comments start to pile up afterwards.  Having that discussion on the
> list, and summarizing on Jira, would mitigate this.
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder, http://www.datastax.com
> @spyced
>

Re: Performance issue in 3.0.9

[Jonathan Haddad]

Also, if you have been tracking your performance metrics in graph form
(before and after the upgrade), that would be extremely helpful.
On Thu, Feb 2, 2017 at 5:29 AM Romain Hardouin 
wrote:

> Yes you should provide more context."Lots of timeouts": read? write?
> both?Did you run sstableupgrade? Java version ? (C* 3.0 requires Java 8u40
> or later)What is your data model? Lots of counters? Compression enabled on
> tables? "No updates/deletes": no deletes but is there TTL on data?
>  etc.
> Best,
> Romain
> Le Jeudi 2 février 2017 9h52, Benjamin Lerer <
> benjamin.le...@datastax.com> a écrit :
>
>
>  Guys,
>
> If you really want us to improve the things you need to be a bit more
> helpfull.
> We have no clue of what are the problems or changes in preformance that you
> see.
> So, if you could provide more context and facts it would be great.
> The more help and clarity you can provide, the easier it will be for us to
> investigate and solve those problems.
>
> By helping us you will help yourselves.
>
> On Thu, Feb 2, 2017 at 9:38 AM, Matija Gobec  wrote:
>
> > We ran for months with the same highly tuned setup on 2.1 and once we
> > switched to 3.0.9 the performance with the same configuration was crap.
> > Leveled compaction but a bit more nodes. There are differences in how 2.1
> > and 3.0 work so I guess you need to revisit your cassandra.yaml and os
> > settings.
> > Next to everything Jeff mentioned, is there any reason you have data on
> all
> > nodes and use QUORUM?
> > Also, is there any reason you are not using G1 with 3.0?
> >
> > On Thu, Feb 2, 2017 at 6:28 AM, Jeff Jirsa  wrote:
> >
> > > Can you quantify "major"?
> > >
> > > Latency or throughput?
> > > GC pauses?
> > > What did you see before? What do you see now?
> > > Do you have a stack dump?
> > >
> > >
> > > --
> > > Jeff Jirsa
> > >
> > >
> > > > On Feb 1, 2017, at 4:23 PM, Shashank Joshi <
> > shashank.jo...@ericsson.com>
> > > wrote:
> > > >
> > > > We are seeing major performance issues with about 100 GB of data in
> > > 3.0.9-E001. The exact same app runs very well in 2.1.
> > > >
> > > >
> > > >
> > > > It feels to us like something is wrong with our configuration because
> > of
> > > the severity of the issues. Thanks in advance for any recommendations
> or
> > > suggestions.
> > > >
> > > >
> > > >
> > > > Details:
> > > >
> > > > Size of data: 100 GB+  all in one table, with a simple schema, couple
> > of
> > > bigints and a double
> > > >
> > > > Cluster: 3 nodes with RF of 3
> > > >
> > > > Client: App uses read and write CL of QUORUM and we have a lots of
> > > timeouts due to inability to reach quorum
> > > >
> > > > Compaction: Leveled
> > > >
> > > > Nature of data usage: No updates/deletes, High reads, relatively low
> > > writes
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > JVM:
> > > >
> > > > Using CMS GC and around 8 GB of max heap
> > > >
> > >
> >
>
>
>

Re: Any idea when 3.0.11 will be released?

[Jonathan Haddad]

You can't go by the DSE version. Datastax backports patches all the time.
It's possible they shipped whatever is in trunk, or backported something
that hasn't been released yet. Or did a fix that's not even in OSS yet.


On Thu, Jan 26, 2017 at 5:49 AM Matija Gobec  wrote:

> By looking at last DSE version it's already out
>
> [cqlsh 5.0.1 | Cassandra 3.0.11.1485 | DSE 5.0.5 | CQL spec 3.4.0 | Native
> protocol v4]
>
> Matija
>
> On Tue, Jan 24, 2017 at 6:24 PM, Johnny Miller 
> wrote:
>
> > Just wondering when 3.0.11 will be released?
> >
> > Thanks,
> >
> > Johnny
> >
>

Re: New committers announcement

[Jonathan Haddad]

Congratulations! Definitely a lot of great contributions from everyone on
the list.
On Tue, Feb 14, 2017 at 1:31 PM Jason Brown  wrote:

> Hello all,
>
> It's raining new committers here in Apache Cassandra!  I'd like to announce
> the following individuals are now committers for the project:
>
> Branimir Lambov
> Paulo Motta
> Stefan Pokowinski
> Ariel Weisberg
> Blake Eggleston
> Alex Petrov
> Joel Knighton
>
> Congratulations all! Please keep the excellent contributions coming.
>
> Thanks,
>
> -Jason Brown
>

Re: Proposal - 3.5.1

[Jonathan Haddad]

As I follow up, I suppose I'm only advocating for a fix to the odd
releases.  Sadly, Tick Tock versioning is misleading.

If tick tock were to continue (and I'm very much against how it currently
works) the whole even-features odd-fixes thing needs to stop ASAP, all it
does it confuse people.

The follow up to 3.4 (3.5) should have been 3.4.1, following semver, so
people know it's bug fixes only to 3.4.

Jon

On Wed, Sep 14, 2016 at 10:37 PM Jonathan Haddad <j...@jonhaddad.com> wrote:

> In this particular case, I'd say adding a bug fix release for every
> version that's affected would be the right thing.  The issue is so easily
> reproducible and will likely result in massive data loss for anyone on 3.X
> WHERE X < 6 and uses the "date" type.
>
> This is how easy it is to reproduce:
>
> 1. Start Cassandra 3.5
> 2. create KEYSPACE test WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': 1};
> 3. use test;
> 4. create table fail (id int primary key, d date);
> 5. delete d from fail where id = 1;
> 6. Stop Cassandra
> 7. Start Cassandra
>
> You will get this, and startup will fail:
>
> ERROR 05:32:09 Exiting due to error while processing commit log during
> initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
> Unexpected error deserializing mutation; saved to
> /var/folders/0l/g2p6cnyd5kx_1wkl83nd3y4rgn/T/mutation6313332720566971713dat.
> This may be caused by replaying a mutation against a table with the same
> name but incompatible schema.  Exception follows:
> org.apache.cassandra.serializers.MarshalException: Expected 4 byte long for
> date (0)
>
> I mean.. come on.  It's an easy fix.  It cleanly merges against 3.5 (and
> probably the other releases) and requires very little investment from
> anyone.
>
>
> On Wed, Sep 14, 2016 at 9:40 PM Jeff Jirsa <jeff.ji...@crowdstrike.com>
> wrote:
>
>> We did 3.1.1 and 3.2.1, so there’s SOME precedent for emergency fixes,
>> but we certainly didn’t/won’t go back and cut new releases from every
>> branch for every critical bug in future releases, so I think we need to
>> draw the line somewhere. If it’s fixed in 3.7 and 3.0.x (x >= 6), it seems
>> like you’ve got options (either stay on the tick and go up to 3.7, or bail
>> down to 3.0.x)
>>
>> Perhaps, though, this highlights the fact that tick/tock may not be the
>> best option long term. We’ve tried it for a year, perhaps we should instead
>> discuss whether or not it should continue, or if there’s another process
>> that gives us a better way to get useful patches into versions people are
>> willing to run in production.
>>
>>
>>
>> On 9/14/16, 8:55 PM, "Jonathan Haddad" <j...@jonhaddad.com> wrote:
>>
>> >Common sense is what prevents someone from upgrading to yet another
>> >completely unknown version with new features which have probably broken
>> >even more stuff that nobody is aware of.  The folks I'm helping right
>> >deployed 3.5 when they got started because
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__cassandra.apache.org=DQIBaQ=08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M=yfYEBHVkX6l0zImlOIBID0gmhluYPD5Jje-3CtaT3ow=MZ9nLcNNhQZkuXyH0NBbP1kSEE2M-SYgyVqZ88IJcXY=pLP3udocOcAG6k_sAb9p8tcAhtOhpFm6JB7owGhPQEs=
>> suggests
>> >it's acceptable for production.  It turns out using 4 of the built in
>> >datatypes of the database result in the server being unable to restart
>> >without clearing out the commit logs and running a repair.  That screams
>> >critical to me.  You shouldn't even be able to install 3.5 without the
>> >patch I've supplied - that bug is a ticking time bomb for anyone that
>> >installs it.
>> >
>> >On Wed, Sep 14, 2016 at 8:12 PM Michael Shuler <mich...@pbandjelly.org>
>> >wrote:
>> >
>> >> What's preventing the use of the 3.6 or 3.7 releases where this bug is
>> >> already fixed? This is also fixed in the 3.0.6/7/8 releases.
>> >>
>> >> Michael
>> >>
>> >> On 09/14/2016 08:30 PM, Jonathan Haddad wrote:
>> >> > Unfortunately CASSANDRA-11618 was fixed in 3.6 but was not back
>> ported to
>> >> > 3.5 as well, and it makes Cassandra effectively unusable if someone
>> is
>> >> > using any of the 4 types affected in any of their schema.
>> >> >
>> >> > I have cherry picked & merged the patch back to here and will put it
>> in a
>> >> > JIRA as well tonight, I just wanted to get the ball rolling asap on
>> this.
>> >> >
>> >> >
>> >>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_rustyrazorblade_cassandra_tree_fix-5Fcommitlog-5Fexception=DQIBaQ=08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M=yfYEBHVkX6l0zImlOIBID0gmhluYPD5Jje-3CtaT3ow=MZ9nLcNNhQZkuXyH0NBbP1kSEE2M-SYgyVqZ88IJcXY=ktY5tkT-nO1jtyc0EicbgZHXJYl03DvzuxqzyyOgzII=
>> >> >
>> >> > Jon
>> >> >
>> >>
>> >>
>>
>

Proposal - 3.5.1

[Jonathan Haddad]

Unfortunately CASSANDRA-11618 was fixed in 3.6 but was not back ported to
3.5 as well, and it makes Cassandra effectively unusable if someone is
using any of the 4 types affected in any of their schema.

I have cherry picked & merged the patch back to here and will put it in a
JIRA as well tonight, I just wanted to get the ball rolling asap on this.

https://github.com/rustyrazorblade/cassandra/tree/fix_commitlog_exception

Jon

Re: Proposal - 3.5.1

[Jonathan Haddad]

ll major bugs being found for months – the first
> production-ready
> > > version of 2.2 is probably in the 2.2.5 or 2.2.6 range.
> > >
> > > For version 3, we moved to an alternate release, modeled on Intel’s
> > > tick/tock https://en.wikipedia.org/wiki/Tick-Tock_model
> > >
> > > The intention was to allow new features into 3.even releases (3.0, 3.2,
> > > 3.4, 3.6, and so on), with bugfixes in 3.odd releases (3.1, … ). The
> hope
> > > was to allow more frequent releases to address the first big negative
> > > (flood of new features that blocked releases), while also helping to
> > > address the second – with fewer major features in a release, they
> better
> > > get more/better test coverage.
> > >
> > > In the tick/tock model, anyone running 3.odd (like 3.5) should be
> looking
> > > for bugfixes in 3.7. It’s certainly true that 3.5 is horribly broken
> (as
> > is
> > > 3.3, and 3.4, etc), but with this release model, the bugfix SHOULD BE
> in
> > > 3.7. As I mentioned previously, we have precedent for backporting
> > critical
> > > fixes, but we don’t have a well defined bar (that I see) for what’s
> > > critical enough for a backport.
> > >
> > > Jon is noting (and what many of us who run Cassandra in production have
> > > really known for a very long time) is that nobody wants to run 3.newest
> > > (even or odd), because 3.newest is likely broken (because it’s a
> complex
> > > distributed database, and testing is hard, and it takes time and
> complex
> > > workloads to find bugs). In the tick/tock model, because new features
> > went
> > > into 3.6, there are new features that may not be adequately
> > > tested/validated in 3.7 a user of 3.5 doesn’t want, and isn’t willing
> to
> > > accept the risk.
> > >
> > > The bottom line here is that tick/tock is probably a well intentioned
> but
> > > failed attempt to bring stability to Cassandra’s releases. The problems
> > > tick/tock was meant to solve are real problems, but tick/tock doesn’t
> > seem
> > > to be addressing them – new features invalidate old testing, which
> makes
> > it
> > > difficult/impossible for real users to sit on the 3.odd versions.
> > >
> > > We’re due for cutting 3.9 and 3.0.9, and we have limited RE manpower to
> > > get those out. Only after those are out would I be +1 on a 3.5.1, and
> > then
> > > only because if I were running 3.5, and I hit this bug, I wouldn’t want
> > to
> > > spend the ~$100k it would cost my organization to validate 3.7 prior to
> > > upgrading, and I don’t think it’s reasonable to ask users to recompile
> a
> > > release for a ~10 line fix for a very nasty bug.
> > >
> > > I’m also very strongly recommend we (committers/PMC) reconsider
> tick/tock
> > > for 4.x releases, because this is exactly the type of problem that will
> > > continue to happen as we move forward. I suggest that we either need to
> > go
> > > back to the old model and do a better job of dealing with feature creep
> > and
> > > testing, or we need to better define what gets backported, because the
> > > community needs a stable version to run, and running latest odd release
> > of
> > > tick/tock isn’t it.
> > >
> > > - Jeff
> > >
> > >
> > > On 9/15/16, 10:31 AM, "dave_les...@apple.com on behalf of Dave
> Lester" <
> > > dave_les...@apple.com> wrote:
> > >
> > > >How would cutting a 3.5.1 release possibly confuse users of the
> > software?
> > > It would be easy to document the change and to send release notes.
> > > >
> > > >Given the bug’s critical nature and that it's a minor fix, I’m +1
> > > (non-binding) to a new release.
> > > >
> > > >Dave
> > > >
> > > >> On Sep 15, 2016, at 7:18 AM, Jeremiah D Jordan <https://urldefense.
> > >
> proofpoint.com/v2/url?u=http-3A__jeremiah.jordan-40gmail.com=DQIFaQ=
> > > 08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M=
> > > yfYEBHVkX6l0zImlOIBID0gmhluYPD5Jje-3CtaT3ow=
> > > srNzKwrs8hKPoJMZ4Ao18CYaMYKnbWaCHou6ui5tqdM=iM_
> > > LKKIhaiC0w6uz3lhK1lob4gJbKhLPqGNfPPLye6w= > wrote:
> > > >>
> > > >> I’m with Jeff on this, 3.7 (bug fixes on 3.6) has already been
> > released
> > > with the fix.  Since the fix applies cleanly anyone is free to put it
> on
> > > top of 3.5 on their 

Re: Proposal - 3.5.1

[Jonathan Haddad]

If the releases can be tagged as alpha / beta so that people don't
accidentally put it in prod (or at least, will do so less), that would be
totally reasonable.

On Thu, Sep 15, 2016 at 12:27 PM Tyler Hobbs  wrote:

> On Thu, Sep 15, 2016 at 2:22 PM, Benedict Elliott Smith <
> bened...@apache.org
> > wrote:
>
> > Feature releases don't have to be on the same cadence as bug fixes.
> They're
> > naturally different beasts.
> >
>
> With the exception of critical bug fixes (which can warrant an immediate
> release), I think keeping a regular cadence makes us less likely to slip
> and fall behind on releases.
>
>
> >
> > Why not stick with monthly feature releases, but mark every third (or
> > sixth) as a supported release that gets quarterly updates for 2-3
> quarters?
> >
>
> That's also a good idea.
>
> --
> Tyler Hobbs
> DataStax 
>

Re: cassandra-3.9 (and 3.8) release

[Jonathan Haddad]

(non-binding) -1 on releasing 2 versions with the same version number.
Everything that's been communicated to the world has been that there would
be a feature release, then a bug fix release a month later.  This breaks
that promise.

On Fri, Sep 23, 2016 at 4:23 PM Michael Shuler 
wrote:

> Thanks! I'll do these release builds and start votes, first thing Monday
> morning, unless I find some time on Sunday.
>
> --
> Michael
>
> On 09/23/2016 05:15 PM, Aleksey Yeschenko wrote:
> > Branch 3.8 off 3.9 with a commit that only changes the version in all
> appropriate places.
> >
> > Two separate votes works.
> >
> > --
> > AY
> >
> > On 23 September 2016 at 12:36:54, Michael Shuler (mich...@pbandjelly.org)
> wrote:
> >
> > The cassandra-3.9 branch HEAD, commit bb371ea, looks good to release
> > (which will also be released as 3.8, changing just the version number).
> > I'm re-running a couple jobs right now, but overall, I think we hit the
> > goal of a clean board: http://cassci.datastax.com/view/cassandra-3.9/
> >
> > If there are no objections, I'd like to roll up 3.9/3.8 and get them out
> > the door. Should this be on one vote, since they are really the same, or
> > do 2 votes? I'm actually not entirely sure how the build for 3.8 will
> > work, since the branch was deleted. Should I create new branch again for
> > 3.8 with the version edit? This sounds the most reasonable and workable
> > with the release build process. This actually does sound like it should
> > be 2 votes, since the commit sha will be different.. Thanks!
> >
> > --
> > Kind regards,
> > Michael
> >
>
>

Re: Proposal - 3.5.1

[Jonathan Haddad]

Common sense is what prevents someone from upgrading to yet another
completely unknown version with new features which have probably broken
even more stuff that nobody is aware of.  The folks I'm helping right
deployed 3.5 when they got started because cassandra.apache.org suggests
it's acceptable for production.  It turns out using 4 of the built in
datatypes of the database result in the server being unable to restart
without clearing out the commit logs and running a repair.  That screams
critical to me.  You shouldn't even be able to install 3.5 without the
patch I've supplied - that bug is a ticking time bomb for anyone that
installs it.

On Wed, Sep 14, 2016 at 8:12 PM Michael Shuler <mich...@pbandjelly.org>
wrote:

> What's preventing the use of the 3.6 or 3.7 releases where this bug is
> already fixed? This is also fixed in the 3.0.6/7/8 releases.
>
> Michael
>
> On 09/14/2016 08:30 PM, Jonathan Haddad wrote:
> > Unfortunately CASSANDRA-11618 was fixed in 3.6 but was not back ported to
> > 3.5 as well, and it makes Cassandra effectively unusable if someone is
> > using any of the 4 types affected in any of their schema.
> >
> > I have cherry picked & merged the patch back to here and will put it in a
> > JIRA as well tonight, I just wanted to get the ball rolling asap on this.
> >
> >
> https://github.com/rustyrazorblade/cassandra/tree/fix_commitlog_exception
> >
> > Jon
> >
>
>

Re: Proposal - 3.5.1

[Jonathan Haddad]

@Sylvain - I see what you're saying now on the branches.  I suppose a
branching strategy like that does give some flexibility to have multiple
things in the pipeline so it does give some additional flexibility there.


On Mon, Sep 19, 2016 at 9:06 AM Eric Evans 
wrote:

> On Fri, Sep 16, 2016 at 5:05 AM, Sylvain Lebresne 
> wrote:
> > In light of all this, my suggesting for a release cycle woud be:
> > - To have 3 branches: 'features', 'testing' and 'stable', with an X month
> >   rotation: 'features' becomes 'testing' after X months and then 'stable'
> > after
> >   X more, before getting EOL X months later.
> > - The feature branch gets everything. The testing branch only gets bug
> > fixes.
> >   The stable branch only gets critical bug fixes. And imo, we should be
> very
> >   strict on this (I acknowledge that there is sometimes a bit of
> > subjectivity on
> >   whether something is a bug or an improvement, and if it's critical or
> > not, but
> >   I think it's not that hard to get consensus if we're all reasonable
> > (though it
> >   might worth agreeing on some rough but written guideline upfront)).
> > - We release on a short and fixed cadence of Y month(s) for both the
> > feature and
> >   testing branch. For the stable branch, given that it already had X
> months
> > of
> >   only bug fixes during the testing phase, one can hope critical fixes
> will
> > be
> >   fairly rare, less than 1 per Y period on average). Further, it's
> supposed
> > to
> >   be stable and fixes are supposed to be critical, so doing hot-fix
> releases
> >   probably makes the most sense (though it probably only work if we're
> > indeed
> >   strict on what is considered critical).
>
> This seems pretty close to what Mck suggested; I think this could work.
>
>
> --
> Eric Evans
> john.eric.ev...@gmail.com
>

Re: Proposal - 3.5.1

[Jonathan Haddad]

I've worked on a few projects where we've had a branch that new stuff went
in before merging to master / trunk.  What you've described reminds me a
lot of git-flow (http://nvie.com/posts/a-successful-git-branching-model/)
although not quite the same.  I'll be verbose in this email to minimize the
reader's assumptions.

The goals of the release cycle should be (in descending order of priority):

1. Minimize bugs introduced through change
2. Allow the codebase to iterate quickly
3. Not get caught up in a ton of back porting bug fixes

There is significant benefit to having a releasable trunk.  This is
different from a trunk which is constantly released.  A releasable trunk
simply means all tests should *always* pass and PMC & committers should
feel confident that they could actually put it in prod for a project that
actually matters.  Having it always be releasable (all tests pass, etc)
means people can at least test the DB on sample data or evaluate it before
the release happens, and get feedback to the team when there are bugs.

This is a different mentality from having a "features" branch, where it's
implied that at times it's acceptable that it not be stable.  The
historical trend with the Cassandra codebase has been to test minimally,
throw the code over the wall, and get feedback from people putting it in
prod who run into issues.  In my experience I have found a general purpose
"features" branch to result in poorly quality codebases.  It's shares a lot
of the same problems as the 1+ year release cycle did previously, with
things getting merged in and then an attempt to stabilize later.

Improving the state of testing in trunk will catch more bugs, satisfying
#1, which naturally leads to #2, and by reducing bugs before they get
released #3 will happen over time.

My suggestion for a *supported* feature release every 3 months (could just
as well be 4 or 6) mixed with Benedict's idea of frequent non-supported
releases (tagged as alpha).  Supported releases should get ~6 months worth
of bug fixes, which if done right, will decrease over time due to a
hopefully more stable codebase.  I 100% agree with Mick that semver makes
sense here, it's not just for frameworks.  Major.Minor.Patch is well
understood and is pretty standard throughout the world, I don't think we
need to reinvent versioning.

TL;DR:
Release every 3 months
Support for 6
Keep a stable trunk
New features get merged into trunk but the standard for code quality and
testing needs to be property defined as something closer to "production
ready" rather than "let the poor user figure it out"

Jon







On Fri, Sep 16, 2016 at 3:05 AM Sylvain Lebresne 
wrote:

> As probably pretty much everyone at this point, I agree the tick-tock
> experiment
> isn't working as well as it should and that it's probably worth course
> correcting. I happen to have been thinking about this quite a bit already
> as it
> turns out so I'm going to share my reasoning and suggestion below, even
> though
> it's going to be pretty long, in the hope it can be useful (and if it
> isn't, so
> be it).
>
> My current thinking is that a good cycle should accommodate 2 main
> constraints:
>   1) be useful for users
>   2) be realistic/limit friction on the development side
> and let me develop what I mean by both points slightly first.
>
> I think users mostly want 2 things out of the release schedule: they want a
> clearly labeled stable branch to know what they should run into production,
> and
> they want new features and improvements. Let me clarify that different
> users
> will want those 2 in different degrees and with variation over time, but I
> believe it's mainly some combination of those. On the development side, I
> don't
> think it's realistic to expect more than 2/3 branches/series to be
> supported at
> any one time (not going to argue that, let's call it a professional
> opinion). I
> also think accumulating new work for any meaningful length of time before
> releasing, as we used to do, is bad as it pushes devs to rush things to
> meet a
> given release deadline as they don't want to wait for the next one. This in
> turn
> impacts quality and creates unnecessary drama. It's also good imo to have a
> clear policy regarding where a given work can go (having to debate on each
> ticket on which branch it should go is a waste of dev time).
>
> With those "goals" in mind, I'll note that:
> - the fixed _and_ short cadence of tick-tock is imo very good, in
> particular in
>   (but not limited to) avoiding the 'accumulate unreleased stuffs' problem.
> - we have ample evidence that stuffs don't get truly stable until they get
> only
>   bug fixes for a few months. Which doesn't mean at all that we shouldn't
>   continue to make progress on increasing the quality of new code btw.
> - Simple is also a great quality of a release cycle. I think we should try
> to
>   define what's truly important to achieve (my opinion on that is above)
> and do
>   the 

Re: Proposal - 3.5.1

[Jonathan Haddad]

What I was trying to suggest is that the *goal* of trunk should always be
releasable, and the alpha releases would be the means of testing that.  If
the goal is to always be releasable, we move towards achieving that goal by
improving modularity, test coverage and test granularity.

Yes, it's very difficult to prove a piece of software is completely free of
bugs and I wouldn't expect NASA to put Cassandra on the space shuttle.
That said, by prioritizing stability in the software development process up
front, the cost of maintaining older branches over time will decrease and
the velocity of the project will increase - which was the original goal of
Tick Tock.

Jon

On Fri, Sep 16, 2016 at 9:04 AM Sylvain Lebresne <sylv...@datastax.com>
wrote:

> On Fri, Sep 16, 2016 at 5:18 PM, Jonathan Haddad <j...@jonhaddad.com>
> wrote:
>
> >
> > This is a different mentality from having a "features" branch, where it's
> > implied that at times it's acceptable that it not be stable.
>
>
> I absolutely never implied that, though I willingly admit my choice of
> branch
> names may be to blame. I 100% agree that no releases should be done
> without a green test board moving forward and if something was implicit
> in my 'feature' branch proposal, it was that.
>
> Where we might not be in the same page is that I just don't believe it's
> reasonable to expect the project will get any time soon in a state where
> even a green test board release (with new features) meets the "can be
> confidently put into production". I'm not even sure it's reasonable to
> expect from *any* software, and even less so for an open-source
> project based on volunteering. Not saying it wouldn't be amazing, it
> would, I just don't believe it's realistic. In a way, the reason why I
> think
> tick-tock doesn't work is *exactly* because it's based on that unrealistic
> assumption.
>
> Of course, I suppose that's kind of my opinion. I'm sure some will think
> that the "historical trend" of release instability is simply due to a lack
> of
> effort (obviously Cassandra developers don't give a shit about users, that
> must the simplest explanation).
>

Re: Proposal - 3.5.1

[Jonathan Haddad]

Yep - the progress that's been made on trunk recently has been excellent
and should continue.  The spirit of tick tock - stable trunk - should not
change, just that the release cycle did not support what humans are
comfortable with maintaining or deploying.

On Fri, Sep 16, 2016 at 10:08 AM Jonathan Ellis <jbel...@gmail.com> wrote:

> On Fri, Sep 16, 2016 at 11:36 AM, Jonathan Haddad <j...@jonhaddad.com>
> wrote:
>
> > What I was trying to suggest is that the *goal* of trunk should always be
> > releasable, and the alpha releases would be the means of testing that.
> If
> > the goal is to always be releasable, we move towards achieving that goal
> by
> > improving modularity, test coverage and test granularity.
> >
> > Yes, it's very difficult to prove a piece of software is completely free
> of
> > bugs and I wouldn't expect NASA to put Cassandra on the space shuttle.
> > That said, by prioritizing stability in the software development process
> up
> > front, the cost of maintaining older branches over time will decrease and
> > the velocity of the project will increase - which was the original goal
> of
> > Tick Tock.
> >
>
> And we *did* make substantial progress on this.  Not nearly as quickly as I
> originally hoped, but our CI is worlds cleaner and more useful than it was
> this time last year.
>
> --
> Jonathan Ellis
> co-founder, http://www.datastax.com
> @spyced
>

Re: Github pull requests

[Jonathan Haddad]

+1 to officially supporting GitHub PRs.

On Fri, Aug 26, 2016 at 11:07 AM Jason Brown  wrote:

> It seems to me we might get more contributions if we can lower the barrier
> to participation. (see Jeff Beck's statement above)
>
> +1 to Aleksey's sentiment about the Docs contributions.
>
> On Fri, Aug 26, 2016 at 9:48 AM, Mark Thomas  wrote:
>
> > On 26/08/2016 17:11, Aleksey Yeschenko wrote:
> > > Mark, I, for one, will be happy with the level of GitHub integration
> > that Spark has, formal or otherwise.
> >
> > If Cassandra doesn't already have it, that should be a simple request to
> > infra.
> >
> > > As it stands right now, none of the committers/PMC members have any
> > control over Cassandra Github mirror.
> > >
> > > Which, among other things, means that we cannot even close the
> > erroneously opened PRs ourselves,
> > > they just accumulate unless the PR authors is kind enough to close
> them.
> > That’s really frustrating.
> >
> > No PMC currently has the ability to directly close PRs on GitHub. This
> > is one of the things on the infra TODO list that is being looked at. You
> > can close them via a commit comment that the ASF GitHub tooling picks up.
> >
> > Mark
> >
> >
> > >
> > > --
> > > AY
> > >
> > > On 26 August 2016 at 17:07:29, Mark Thomas (ma...@apache.org) wrote:
> > >
> > > On 26/08/2016 16:33, Jonathan Ellis wrote:
> > >> Hi all,
> > >>
> > >> Historically we've insisted that people go through the process of
> > creating
> > >> a Jira issue and attaching a patch or linking a branch to demonstrate
> > >> intent-to-contribute and to make sure we have a unified record of
> > changes
> > >> in Jira.
> > >>
> > >> But I understand that other Apache projects are now recognizing a
> github
> > >> pull request as intent-to-contribute [1] and some are even making
> github
> > >> the official repo, with an Apache mirror, rather than the other way
> > >> around. (Maybe this is required to accept pull requests, I am not
> sure.)
> > >>
> > >> Should we revisit our policy here?
> > >
> > > At the moment, the ASF Git repo is always the master, with GitHub as a
> > > mirror. That allows push requests to be made via GitHub.
> > >
> > > Infra is exploring options for giving PMCs greater control over GitHub
> > > config (including allowing GitHub to be the master with a golden copy
> > > held at the ASF) but that is a work in progress.
> > >
> > > As far as intent to contribute goes, there does appear to be a trend
> > > that the newer a project is to the ASF, the more formal the project
> > > makes process around recording intent to contribute. (The same can be
> > > said for other processes as well like Jira config.)
> > >
> > > It is worth noting that all the ASF requires is that there is an intent
> > > to contribute. Anything that can be reasonably read that way is fine.
> > > Many PMCs happily accept patches sent to the dev list (although they
> may
> > > ask them to be attached to issues more so they don't get forgotten than
> > > anything else). Pull requests are certainly acceptable.
> > >
> > > My personal recommendation is don't put more formal process in place
> > > than you actually need. Social controls are a lot more flexible than
> > > technical ones and generally have a much lower overhead.
> > >
> > > Mark
> > >
> > >>
> > >> [1] e.g. https://github.com/apache/spark/pulls?q=is%3Apr+is%3Aclosed
> > >
> > >
> >
> >
>

Re: Proposal: create a mailing list for just the newly created Jira ticket notifications

[Jonathan Haddad]

Wow, I am embarrassed to say that I had no idea that existed.  Thanks Russ!

On Mon, Aug 29, 2016 at 9:58 AM Russell Hatch <rha...@datastax.com> wrote:

> This is currently possible using jira saved search + subscription. Create a
> new saved search with a query like:
>
> project = CASSANDRA AND created >= -24h ORDER BY createdDate DESC
>
> When viewing your filter, go to Details > New Subscription and set it up to
> run once a day and you're set!
>
> Hope it helps,
>
> Russ
>
> On Mon, Aug 29, 2016 at 10:45 AM, Jonathan Haddad <j...@jonhaddad.com>
> wrote:
>
> > A daily digest of new issues would be incredible.  A non-binding +1 from
> > this guy.
> >
> >
> > On Mon, Aug 29, 2016 at 8:58 AM Jeremy Hanna <jeremy.hanna1...@gmail.com
> >
> > wrote:
> >
> > > Currently we have a mailing list (commits@) that includes commit
> > messages
> > > as well as all updates to Jira tickets.  However for the purpose of
> > staying
> > > up to date with new tickets, it’s something of a firehose with the
> rapid
> > > rate of change.
> > >
> > > Some people (like me) filter out just the new ticket creation emails
> and
> > > watch/vote for those that seem interesting.  Others create Jira filters
> > and
> > > subscribe to them.  Personally I prefer an email per Jira creation.
> For
> > > those that prefer that route, would anyone mind if I created an infra
> > > ticket to create a mailing list to just publish those emails?  Are
> there
> > > others like me out there that prefer that to doing a daily digest or
> > > something similar?
> > >
> > > Thanks,
> > >
> > > Jeremy
> >
>

Re: Proposal: create a mailing list for just the newly created Jira ticket notifications

[Jonathan Haddad]

A daily digest of new issues would be incredible.  A non-binding +1 from
this guy.


On Mon, Aug 29, 2016 at 8:58 AM Jeremy Hanna 
wrote:

> Currently we have a mailing list (commits@) that includes commit messages
> as well as all updates to Jira tickets.  However for the purpose of staying
> up to date with new tickets, it’s something of a firehose with the rapid
> rate of change.
>
> Some people (like me) filter out just the new ticket creation emails and
> watch/vote for those that seem interesting.  Others create Jira filters and
> subscribe to them.  Personally I prefer an email per Jira creation.  For
> those that prefer that route, would anyone mind if I created an infra
> ticket to create a mailing list to just publish those emails?  Are there
> others like me out there that prefer that to doing a daily digest or
> something similar?
>
> Thanks,
>
> Jeremy

Re: Rough roadmap for 4.0

[Jonathan Haddad]

epaxos would be nice, it's been about 2 years since it was started, and
Blake asked for a first review well over a year ago.  the benchmarks and
thorough test suite look like a huge improvement over the current situation.

https://issues.apache.org/jira/browse/CASSANDRA-6246

On Fri, Nov 4, 2016 at 3:28 AM Edward Capriolo 
wrote:

> I would like to propose features around seeds:
> https://issues.apache.org/jira/browse/CASSANDRA-12627
>
> I have other follow up issues like getting seeds from Amazon API, or from
> JNDI/ DNS, etc.
>
> I was hoping 12627 was an easy way to grease the wheels.
>
>
> On Fri, Nov 4, 2016 at 8:39 AM, Jason Brown  wrote:
>
> > Hey Nate,
> >
> > I'd like to add CASSANDRA-11559 (Enhance node representation) to the
> list,
> > including the comment I made on the ticket (different storage ports for
> > each node). For us, it's a great "would really like to have" as our group
> > will probably need this in production within the next 1 year or less.
> > However since it hasn't gotten much attention, I'm not sure if we should
> > add it to the list of "must haves" for 4.0. I'm planning on bringing it
> up
> > internally today, as well.
> >
> > Also, from the previous 4.0 email thread that Jonathan started back in
> July
> > (
> > https://mail-archives.apache.org/mod_mbox/cassandra-dev/
> > 201607.mbox/%3CCALdd-zhW3qJ%3DOWida9nMXPj0JOsru7guOYh6-
> > 7uTjqEBvacrgQ%40mail.gmail.com%3E
> > )
> >
> > - CASSANDRA-5 (thrift removal) - ticket not mentioned explicitly in
> the
> > email, but the notion of removing thrift was
> > - CASSANDRA-10857 (Allow dropping COMPACT STORAGE flag)
> >
> > Thanks,
> >
> > -Jason
> >
> > On Thu, Nov 3, 2016 at 8:43 PM, sankalp kohli 
> > wrote:
> >
> > > List looks really good. I will let you know if there is something else
> we
> > > plan to add to this list.
> > >
> > > On Thu, Nov 3, 2016 at 7:47 PM, Nate McCall 
> wrote:
> > >
> > > > It was brought up recently at the PMC level that our goals as a
> > > > project are not terribly clear.
> > > >
> > > > This is a pretty good point as outside of Jira 'Fix Version'
> labelling
> > > > (which we actually suck less at compared to a lot of other ASF
> > > > projects) this really isnt tracked anywhere outside of general tribal
> > > > knowledge about who is working on what.
> > > >
> > > > I would like to see us change this for two reasons:
> > > > - it's important we are clear with our community about where we are
> > going
> > > > - we need to start working more closely together
> > > >
> > > > To that end, i've put together a list (in no particular order) of the
> > > > *major* features in which I know folks are interested, have patches
> > > > coming, are awaiting design review, etc.:
> > > >
> > > > - CASSANDRA-9425 Immutable node-local schema
> > > > - CASSANDRA-10699 Strongly consistent schema alterations
> > > > - CASSANDRA-12229 NIO streaming
> > > > - CASSANDRA-8457 NIO messaging
> > > > - CASSANDRA-12345 Gossip 2.0
> > > > - CASSANDRA-9754 Birch trees
> > > >
> > > > What did I miss? What else would folks like to see? Specifically,
> this
> > > > should be "new stuff that could/will break things" given we are
> upping
> > > > the major version.
> > > >
> > > > To be clear, it's not my intention to set this in stone and then beat
> > > > people about the head with it. More to have it there to point it at a
> > > > high level and foster better communication with our users from the
> > > > perspective of an open source project.
> > > >
> > > > Please keep in mind that given everything else going on, I think it's
> > > > a fantastic idea to keep this list small and spend some time focusing
> > > > on stability particularly as we transition to a new release process.
> > > >
> > > > -Nate
> > > >
> > >
> >
>

Re: Moderation

[Jonathan Haddad]

I took a look at https://www.apache.org/dev/pmc.html, and it doesn't seem
to give any guidelines on who should be on the PMC.  My assumption has
always been the most active committers become PMC members, but it sounds
like that's not the case on other projects.  Is the process to be added to
the PMC supposed to be the same everywhere, or is it up to the project?
Can you be on the PMC but not have commit access?

On Sun, Nov 6, 2016 at 5:04 AM Chris Mattmann  wrote:

> Sorry one typo below:
>
> Where I said:
>
> “The Cassandra MVP comment was also not a diss on you as much as it was me
> saying – ideally – I would hope that
> the Apache Cassandra MVP people promote the concept of their community
> leaders becoming “ASF members”,
> and that Cassandra MVPs are great – but secondary – to the
> responsibilities of the PMC to move towards ensuring
> its community understands the Apache Way.”
>
> I meant to say:
>
> “The Cassandra MVP comment was also not a diss on you as much as it was me
> saying – ideally – I would hope that
> the Apache Cassandra *PMC* people promote the concept of their community
> leaders becoming “ASF members”,
> and that Cassandra MVPs are great – but secondary – to the
> responsibilities of the PMC to move towards ensuring
> its community understands the Apache Way.”
>
> Thanks.
>
> Cheers,
> Chris
>
>
> On 11/6/16, 6:53 AM, "Chris Mattmann"  wrote:
>
> For the record, your breakdown of the email trying to decipher what I
> meant is not
> correct. It’s not your fault, but email doesn’t convey tone, nor do
> you know what I am
> thinking or what I was trying to say. In fact, I was actually saying
> the PMC wasn’t doing its job,
> because (as I stated to you months ago), you (and many other community
> members of
> Cassandra) *should* have a binding vote. It wasn’t discrediting to you
> to point out that
> you don’t have the PMC or committer credentials; it was an example
> trying to point out
> that you *should* have them. And that you clearly care about the
> project as I believe you
> have developed a book on the subject of Apache Cassandra a while back
> IIRC which in Tika,
> Nutch, OODT, and a number of other projects would have earned you the
> ability to have a
> direct say in those Apache projects. And a lot of others.
>
> It’s these systematic fracturing of the community under the guise of a
> single vendor who
> has stated that they care about Cassandra (note the omission of
> Apache), but by demonstration
> has shown they either don’t understand, or don’t care about the Apache
> part of the equation.
> That’s what caused me to become frustrated when the following sequence
> of events
> happened:
>
> 1. After the Board meeting Mark Thomas one of our Directors took point
> on engaging
> the Apache Cassandra PMC with some of the concerns brought up over the
> past 6
> months and the role I was filling there became a back seat for me.
> 2. I saw over the past few days on a Twitter feed retweeted by an ASF
> member that
> Kelly Sommers (whom I have never met in person and do not know
> previously) was asking
> questions and stating negative things about the ASF that I believed
> could be much better
> understood by bringing them here to the ASF mailing lists for Apache
> Cassandra. I suggested
> on Twitter that she bring her concerns to the Apache lists and told
> her which email address
> to send it to. Some of the same people that eventually came onto the
> thread were people
> who were communicating with her on Twitter – this was disappointing as
> they could have
> done the same thing, and suggested Kelly come to the lists, Apache
> Cassandra PMC or not.
> 3. After 12 hours I checked back with Kelly and the Twitter dialogue
> had continued with several
> ASF members and even some Board members getting involved. Again, I
> asked Kelly why talk
> there, and why not just talk to the email list which is the canonical
> home for Apache Cassandra?
> She told me she sent the mail the prior night.
> 4. So of course I checked (after having already guessed it was stuck
> in moderation) and yes it
> was. What ensued was both frustration by my part and also email
> conversation that was heated
> on both sides. I felt swiped on by a few emails where I had good
> intentions but I felt we were
> wasting time debating whether we *should* moderate something through –
> which to me was
> a clear answer (yes). Where I failed there was to recognize that the
> real answer was that the Apache
> Cassandra PMC did not have enough moderators and the people I was
> mostly going back and forth
> with were not the moderators of the mailing lists.
> 5. One positive thing that came from #4 was that at least there are
> more moderators now. I’m not sure
> the reason for the lack of geographically diverse 

Re: Moderation

[Jonathan Haddad]

I agree with Paul. Same boat, not a PMC / Datastax, just someone that cares
a lot about this community.
On Sat, Nov 5, 2016 at 3:04 PM paul cannon  wrote:

> I'm not a stakeholder here- I don't know Russell, I don't work for
> Datastax, and I'm not a member of the ASF.
>
> For what little it's probably worth since I haven't "been elected to have a
> binding voice within the project", Russell's is exactly how I read the
> message from Chris Mattmann. Whether or not it was intended to be so
> aggressive and dismissive and patronizing, I almost can't even believe
> something that *might* be taken this way is tolerated in a board member's
> *public* communications.
>
> In the end, I *can* believe it, though, as it reinforces my perception of
> the Foundation in general. :(
>
> p
>
>
> On Sat, Nov 5, 2016 at 5:16 PM, Russell Bradberry 
> wrote:
>
> > For the record, I never said anyone was attempting to make me “look bad”.
> > I simply stated that his method of argument was to discredit me.  Below I
> > will break down his response, as I see it, and as others who have
> messaged
> > me off list see it as well:
> >
> > “… You see I’ve been around since 2004 and elected by the membership to
> > the Board for the last three years based on merit …”
> >
> > Here he is showing his superiority by way of tenure, or merit.
> >
> > “You see I actually understand…”
> >
> > The use of the term “actually” in this sense is to provide an attack
> > against me in an effort to prove that I do not understand.
> >
> > “…unfortunately you do not have a voice …”
> >
> > Again, this is a blatant attempt to discredit me and provide proof that
> my
> > word is of no worth because I am not on the PMC, nor a committer.
> >
> > “You won’t have a vote in the next Apache Board election.”
> >
> > Again
> >
> > “You won’t have a vote in the next Members election.”
> >
> > Again
> >
> > “why haven’t you been elected to have a binding voice within the project?
> > Please ask yourself that”
> >
> > This is either an attempt to discredit me, in that I have not done enough
> > to be elected, or an attempt to state the PMC hasn’t been doing their job
> > in recognizing my efforts.
> >
> > “please ask yourself – what is a “Cassandra MVP” compared to a member of
> > the ASF which is home to the project””
> >
> > This is not only insinuating that MVP is less than being a member of the
> > ASF, and because I was given the MVP title, that somehow I am less than
> as
> > well. (for the record, I have not asked for the MVP title, it was
> awarded,
> > and I do not think that it should have any effect on the project from an
> > Apache standpoint. Quite simply put, it is just another bullet point on a
> > resume)
> >
> > “I’ve been privy and voted on granting membership to within the
> foundation
> > since 2011”
> >
> > More attempts to discredit me by showing tenure.
> >
> > Literally, the first portion of the response was a campaign to discredit
> > me in order to demonstrate his merit.  The rest of the email goes to
> defend
> > a point that I did not make.
> >
> > Again, I will assert that the complaints the board has are valid.
> > Datastax may have overstepped bounds and, as a result, put the project
> and
> > ASF at risk.  I am not an authority on the subject and have not been
> privy
> > to the private messages between the board, PMC, and Datastax.  What I
> will
> > say, is that the tone, vitriol, ad-hominem responses and other
> > unprofessional conduct has caused a rift in this community.  Most of this
> > is coming directly from the board, specifically Chris.  Furthermore, as
> > Aleksey has pointed out, this occurs in the private lists as well.  This
> is
> > a form of toxic-leadership and is proven to not only be ineffective, but
> > also be directly harmful.  These issues can, and should, be resolved
> > amicably.
> >
> > Professionalism and Respect, if aren’t, should be of the core tenets of
> > any foundation, especially one of the caliber of Apache.
> >
> >
> >
> > On 11/5/16, 9:38 AM, "Mark Struberg"  wrote:
> >
> > Russel, I don't read that out of Chris' answer.
> > He just tried to show how community development might look like if
> > done a bit more openly.
> >
> > Do you mind going back to Chris' original reply and re-read it again?
> > I've not interpreted it as anyone trying to make you look bad. Au
> > contraire!
> >
> >
> > txs and LieGrue,
> > strub
> >
> >
> >
> >
> >
> > > On Saturday, 5 November 2016, 13:56, Russell Bradberry <
> > rbradbe...@gmail.com> wrote:
> > > > It seems that your tactic of argument is to discredit me at every
> > level in order
> > > to show your superiority of sorts.  Let me set this straight, I am
> > not
> > > attempting to say that I am an authority on ASF or that I know how
> > things should
> > > be run.  I also was not attempting to vilify you in front of the
> > board or vilify
> >   

Re: Low hanging fruit crew

[Jonathan Haddad]

t; >> perfect (and as Jeremiah points out, how will you know that without
> > >> reading
> > >> >> the code?), you can still pass tests with bad code.  And is
> expecting
> > >> >> perfect tests really realistic for multithreaded code?
> > >> >>
> > >> >> But besides correctness, reviews should deal with
> > >> >>
> > >> >> 1. Efficiency.  Is there a quadratic loop that will blow up when
> the
> > >> number
> > >> >> of nodes in a cluster gets large?  Is there a linear amount of
> memory
> > >> used
> > >> >> that will cause problems when a partition gets large?  These are
> not
> > >> >> theoretical problems.
> > >> >>
> > >> >> 2. Maintainability.  Is this the simplest way to accomplish your
> > >> goals?  Is
> > >> >> there a method in SSTableReader that would make your life easier if
> > you
> > >> >> refactored it a bit instead of reinventing it?  Does this reduce
> > >> technical
> > >> >> debt or add to it?
> > >> >>
> > >> >> 3. "Bus factor."  If only the author understands the code and all
> > >> anyone
> > >> >> else knows is that tests pass, the project will quickly be in bad
> > >> shape.
> > >> >> Review should ensure that at least one other person understand the
> > >> code,
> > >> >> what it does, and why, at a level that s/he could fix bugs later in
> > it
> > >> if
> > >> >> necessary.
> > >> >>
> > >> >> On Wed, Oct 19, 2016 at 10:52 AM, Jonathan Haddad <
> j...@jonhaddad.com
> > >
> > >> wrote:
> > >> >>
> > >> >>> Shouldn't the tests test the code for correctness?
> > >> >>>
> > >> >>> On Wed, Oct 19, 2016 at 8:34 AM Jonathan Ellis <jbel...@gmail.com
> >
> > >> wrote:
> > >> >>>
> > >> >>>> On Wed, Oct 19, 2016 at 8:27 AM, Benjamin Lerer <
> > >> >>>> benjamin.le...@datastax.com
> > >> >>>>> wrote:
> > >> >>>>
> > >> >>>>> Having the test passing does not mean that a patch is fine.
> Which
> > is
> > >> >>> why
> > >> >>>> we
> > >> >>>>> have a review check list.
> > >> >>>>> I never put a patch available without having the tests passing
> but
> > >> most
> > >> >>>> of
> > >> >>>>> my patches never pass on the first try. We always make mistakes
> no
> > >> >>> matter
> > >> >>>>> how hard we try.
> > >> >>>>> The reviewer job is to catch those mistakes by looking at the
> > patch
> > >> >>> from
> > >> >>>>> another angle. Of course, sometime, both of them fail.
> > >> >>>>>
> > >> >>>>
> > >> >>>> Agreed.  Review should not just be a "tests pass, +1" rubber
> stamp,
> > >> but
> > >> >>>> actually checking the code for correctness.  The former is just
> > >> process;
> > >> >>>> the latter actually catches problems that the tests would not.
> > (And
> > >> this
> > >> >>>> is true even if the tests are much much better than ours.)
> > >> >>>>
> > >> >>>> --
> > >> >>>> Jonathan Ellis
> > >> >>>> co-founder, https://urldefense.proofpoint.
> > >> com/v2/url?u=http-3A__www.datastax.com=DQIFaQ=08AGY6txKs
> > >> vMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M=yfYEBHVkX6l0zImlOIBID
> > >> 0gmhluYPD5Jje-3CtaT3ow=eXI0TPp0DM06kmTuJQRNcUX5zy_O_KhWcDK
> > >> MA-jxww0=D28xk3JpCIOAQnCXJAVky0lJJv_mPnYwy4gKxLKsSqw=
> > >> >>>> @spyced
> > >> >>>>
> > >> >>>
> > >> >>
> > >> >>
> > >> >>
> > >> >> --
> > >> >> Jonathan Ellis
> > >> >> co-founder, https://urldefense.proofpoint.
> > >> com/v2/url?u=http-3A__www.datastax.com=DQIFaQ=08AGY6txKs
> > >> vMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M=yfYEBHVkX6l0zImlOIBID
> > >> 0gmhluYPD5Jje-3CtaT3ow=eXI0TPp0DM06kmTuJQRNcUX5zy_O_KhWcDK
> > >> MA-jxww0=D28xk3JpCIOAQnCXJAVky0lJJv_mPnYwy4gKxLKsSqw=
> > >> >> @spyced
> > >> >
> > >>
> > >> 
> > >> CONFIDENTIALITY NOTE: This e-mail and any attachments are confidential
> > >> and may be legally privileged. If you are not the intended recipient,
> do
> > >> not disclose, copy, distribute, or use this email or any attachments.
> If
> > >> you have received this in error please let the sender know and then
> > delete
> > >> the email and all attachments.
> > >>
> > >
> > >
> > > --
> > > Sorry this was sent from mobile. Will do less grammar and spell check
> > than
> > > usual.
> > >
> >
> >
> > --
> > Sorry this was sent from mobile. Will do less grammar and spell check
> than
> > usual.
> >
>

Re: Proposal - 3.5.1

[Jonathan Haddad]

And +1 to ditching the "tick tock" alternating release thing nobody
understands it anyway.

On Thu, Oct 20, 2016 at 3:38 PM Jonathan Haddad <j...@jonhaddad.com> wrote:

> From a community perspective a supported release every 6 months would be
> much more attractive than yearly, as having to wait ~9-10 months for
> something like SASI is kind of frustrating.
>
> Monthly dev releases is awesome.
>
> On Thu, Oct 20, 2016 at 3:18 PM Nate McCall <zznat...@gmail.com> wrote:
>
> > I’m not sure it makes sense to have separate features/stability releases
> in that world. 4.0.x will be stable, every 4.x will be a dev release on the
> road to 5.0.
> >
> +1. Much easier to understand and it's 'backwards compatible' (sort
> of) with wherever we leave 3.x.
>
> Still keeping 4.x on monthly cadence should keep the small, incremental
> focus.
>
>

Re: Proposal - 3.5.1

[Jonathan Haddad]

>From a community perspective a supported release every 6 months would be
much more attractive than yearly, as having to wait ~9-10 months for
something like SASI is kind of frustrating.

Monthly dev releases is awesome.

On Thu, Oct 20, 2016 at 3:18 PM Nate McCall  wrote:

> > I’m not sure it makes sense to have separate features/stability releases
> in that world. 4.0.x will be stable, every 4.x will be a dev release on the
> road to 5.0.
> >
> +1. Much easier to understand and it's 'backwards compatible' (sort
> of) with wherever we leave 3.x.
>
> Still keeping 4.x on monthly cadence should keep the small, incremental
> focus.
>

Re: Low hanging fruit crew

[Jonathan Haddad]

Shouldn't the tests test the code for correctness?

On Wed, Oct 19, 2016 at 8:34 AM Jonathan Ellis  wrote:

> On Wed, Oct 19, 2016 at 8:27 AM, Benjamin Lerer <
> benjamin.le...@datastax.com
> > wrote:
>
> > Having the test passing does not mean that a patch is fine. Which is why
> we
> > have a review check list.
> > I never put a patch available without having the tests passing but most
> of
> > my patches never pass on the first try. We always make mistakes no matter
> > how hard we try.
> > The reviewer job is to catch those mistakes by looking at the patch from
> > another angle. Of course, sometime, both of them fail.
> >
>
> Agreed.  Review should not just be a "tests pass, +1" rubber stamp, but
> actually checking the code for correctness.  The former is just process;
> the latter actually catches problems that the tests would not.  (And this
> is true even if the tests are much much better than ours.)
>
> --
> Jonathan Ellis
> co-founder, http://www.datastax.com
> @spyced
>

Re: [VOTE] Release Apache Cassandra 3.10 (Take 3)

[Jonathan Haddad]

-1 (non-binding) for the failing tests as well.

On Fri, Nov 25, 2016 at 10:35 AM Sam Tunnicliffe  wrote:

> I'll register another (non-binding) -1 due to the failing tests. In
> particular, CASSANDRA-12651 is due to a legit bug with potential corruption
> and data loss (albeit in indexes only). The same bug is also the likely
> cause of CASSANDRA-12590, so it'd be good to include the fix in 3.10.
>
> On Mon, Nov 21, 2016 at 7:27 PM, sankalp kohli 
> wrote:
>
> > We voted in the dev list discussion[1] that we wont cut a release till
> all
> > unit tests are passing. I can see that testall are failing for trunk and
> > 3.X. Hence I need to do a -1 on this release. My -1 is not due to Dtest
> > failing as we discussed about unit tests only in that thread. I will
> start
> > another thread for making Dtest a blocker for release.
> >
> > -1 due to testall failing.
> >
> > 1. http://mail-archives.apache.org/mod_mbox/incubator-
> > cassandra-dev/201607.mbox/%3cCAEPxca1TGo6S1O8DcpM7dmXsBe4
> > 4EniSezkpq5_=qksctk3...@mail.gmail.com%3e
> >
> >
> >
> > On Mon, Nov 21, 2016 at 9:19 AM, Sankalp Kohli 
> > wrote:
> >
> > > We had decided in an email thread initiated by me that we won't cut
> > > releases till all tests are passing.
> > >
> > > Why not wait till these patches are merged? Is there any critical fix
> in
> > > it which need to be rushed out ?
> > >
> > > > On Nov 20, 2016, at 23:36, Oleksandr Petrov <
> > oleksandr.pet...@gmail.com>
> > > wrote:
> > > >
> > > > From just a quick glance, I can say at least some of the tests are
> > either
> > > > PA or are getting there:
> > > >
> > > > For example:
> > > > http://cassci.datastax.com/job/cassandra-3.X_novnode_
> > > dtest/lastCompletedBuild/testReport/paging_test/
> > > TestPagingData/test_paging_with_filtering_on_partition_key/
> > > > Should be fixed by
> > > > https://issues.apache.org/jira/browse/CASSANDRA-12666
> > > >
> > > > http://cassci.datastax.com/job/cassandra-3.X_testall/
> > > lastCompletedBuild/testReport/org.apache.cassandra.cql3.
> > > validation.entities/SecondaryIndexTest/testAllowFilteringOnPartitionK
> > > eyWithSecondaryIndex/
> > > > Should be fixed by
> > > > https://issues.apache.org/jira/browse/CASSANDRA-12651
> > > >
> > > > Although there can be more tests in PA.
> > > >
> > > > On Sun, Nov 20, 2016 at 10:35 PM sankalp kohli <
> kohlisank...@gmail.com
> > >
> > > > wrote:
> > > >
> > > >> Hi,
> > > >>I see the following test runs are failing. Are they for this
> > release?
> > > >>
> > > >> http://cassci.datastax.com/job/cassandra-3.X_utest_cdc/
> > > >> http://cassci.datastax.com/job/cassandra-3.X_testall/
> > > >> http://cassci.datastax.com/job/cassandra-3.X_offheap_dtest/
> > > >> http://cassci.datastax.com/job/cassandra-3.X_novnode_dtest/
> > > >> http://cassci.datastax.com/job/cassandra-3.X_dtest/40/
> > > >>
> > > >> Thanks,
> > > >> Sankalp
> > > >>
> > > >>
> > > >>> On Sun, Nov 20, 2016 at 12:50 PM, Nate McCall 
> > > wrote:
> > > >>>
> > > >>> +1
> > > >>>
> > > >>> On Sat, Nov 19, 2016 at 7:08 AM, Michael Shuler <
> > > mich...@pbandjelly.org>
> > > >>> wrote:
> > >  I propose the following artifacts for release as 3.10.
> > > 
> > >  sha1: 96d67b109a2ef858c2753bbb9853d01460cb8f8e
> > >  Git:
> > >  http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> > > >>> shortlog;h=refs/tags/3.10-tentative
> > >  Artifacts:
> > >  https://repository.apache.org/content/repositories/
> > > >>> orgapachecassandra-1135/org/apache/cassandra/apache-cassandra/3.10/
> > >  Staging repository:
> > >  https://repository.apache.org/content/repositories/
> > > >>> orgapachecassandra-1135/
> > > 
> > >  The Debian packages are available here:
> http://people.apache.org/~
> > > >>> mshuler
> > > 
> > >  The vote will be open for 72 hours (longer if needed).
> > > 
> > >  [1]: (CHANGES.txt) https://goo.gl/vah74X
> > >  [2]: (NEWS.txt) https://goo.gl/In3vp7
> > > 
> > > >>>
> > > >>
> > > > --
> > > > Alex Petrov
> > >
> >
>

Re: Proposals for releases - 4.0 and beyond

[Jonathan Haddad]

+1 as well

On Mon, Nov 21, 2016 at 10:59 AM kurt Greaves  wrote:

> yep +1 to this. The LTS release solves my previous concern
>

Re: Failed Dtest will block cutting releases

[Jonathan Haddad]

+1.  Kind of silly to put advise people to put something in prod which is
known to be broken in obvious ways

On Mon, Nov 21, 2016 at 11:31 AM sankalp kohli 
wrote:

> Hi,
> We should not cut a releases if Dtest are not passing. I won't block
> 3.10 on this since we are just discussing this.
>
> Please provide feedback on this.
>
> Thanks,
> Sankalp
>

Re: Rough roadmap for 4.0

[Jonathan Haddad]

I think it might be worth considering adopting the release strategy before
4.0 release.  Are any PMC members putting tick tock in prod? Does anyone
even trust it?  What's the downside of changing the release cycle
independently from 4.0?

On Thu, Nov 17, 2016 at 9:03 AM Jason Brown  wrote:

Jason,

That's a separate topic, but we will have a different vote on how the
branching/release strategy should be for the future.

On Thursday, November 17, 2016, jason zhao yang 
wrote:

> Hi,
>
> Will we still use tick-tock release for 4.x and 4.0.x ?
>
> Stefan Podkowinski >于2016年11月16日周三
> 下午4:52写道：
>
> > From my understanding, this will also effect EOL dates of other
branches.
> >
> > "We will maintain the 2.2 stability series until 4.0 is released, and
3.0
> > for six months after that.".
> >
> >
> > On Wed, Nov 16, 2016 at 5:34 AM, Nate McCall  > wrote:
> >
> > > Agreed. As long as we have a goal I don't see why we have to adhere to
> > > arbitrary date for 4.0.
> > >
> > > On Nov 16, 2016 1:45 PM, "Aleksey Yeschenko"  >
> > wrote:
> > >
> > > > I’ll comment on the broader issue, but right now I want to elaborate
> on
> > > > 3.11/January/arbitrary cutoff date.
> > > >
> > > > Doesn’t matter what the original plan was. We should continue with
> 3.X
> > > > until all the 4.0 blockers have been
> > > > committed - and there are quite a few of them remaining yet.
> > > >
> > > > So given all the holidays, and the tickets remaining, I’ll
personally
> > be
> > > > surprised if 4.0 comes out before
> > > > February/March and 3.13/3.14. Nor do I think it’s an issue.
> > > >
> > > > —
> > > > AY
> > > >
> > > > On 16 November 2016 at 00:39:03, Mick Semb Wever (
> > m...@thelastpickle.com 
> > > )
> > > > wrote:
> > > >
> > > > On 4 November 2016 at 13:47, Nate McCall  > wrote:
> > > >
> > > > > Specifically, this should be "new stuff that could/will break
> things"
> > > > > given we are upping
> > > > > the major version.
> > > > >
> > > >
> > > >
> > > > How does this co-ordinate with the tick-tock versioning¹ leading up
> to
> > > the
> > > > 4.0 release?
> > > >
> > > > To just stop tick-tock and then say yeehaa let's jam in all the
> > breaking
> > > > changes we really want seems to be throwing away some of the learnt
> > > wisdom,
> > > > and not doing a very sane transition from tick-tock to
> > > > features/testing/stable². I really hope all this is done in a way
> that
> > > > continues us down the path towards a stable-master.
> > > >
> > > > For example, are we fixing the release of 4.0 to November? or
> > continuing
> > > > tick-tocks until we complete the 4.0 roadmap? or starting the
> > > > features/testing/stable branching approach with 3.11?
> > > >
> > > >
> > > > Background:
> > > > ¹) Sylvain wrote in an earlier thread titled "A Home for 4.0"
> > > >
> > > > > And as 4.0 was initially supposed to come after 3.11, which is
> > coming,
> > > > it's probably time to have a home for those tickets.
> > > >
> > > > ²) The new versioning scheme slated for 4.0, per the "Proposal -
> 3.5.1"
> > > > thread
> > > >
> > > > > three branch plan with “features”, “testing”, and “stable”
starting
> > > with
> > > > 4.0?
> > > >
> > > >
> > > > Mick
> > > >
> > >
> >
>

Re: Bootstrapping data from Cassandra 2.2.5 datacenter to 3.0.8 datacenter fails because of streaming errors

[Jonathan Haddad]

You can't stream between major versions. Don't tear down your first data
center, upgrade it instead.
On Mon, Oct 10, 2016 at 4:35 PM Abhishek Verma  wrote:

> Hi Cassandra users,
>
> We are trying to upgrade our Cassandra version from 2.2.5 to 3.0.8
> (running on Mesos, but that's besides the point). We have two datacenters,
> so in order to preserve our data, we are trying to upgrade one datacenter
> at a time.
>
> Initially both DCs (dc1 and dc2) are running 2.2.5. The idea is to tear
> down dc1 completely (delete all the data in it), bring it up with 3.0.8,
> let data replicate from dc2 to dc1, and then tear down dc2, bring it up
> with 3.0.8 and replicate data from dc1.
>
> I am able to reproduce the problem on bare metal clusters running on 3
> nodes. I am using Oracle's server-jre-8u74-linux-x64 JRE.
>
> *Node A*: Downloaded 2.2.5-bin.tar.gz, changed the seeds to include its
> own IP address, changed listen_address and rpc_address to its own IP and
> changed endpoint_snitch to GossipingPropertyFileSnitch. I
> changed conf/cassandra-rackdc.properties to
> dc=dc2
> rack=rack2
> This node started up fine and is UN in nodetool status in dc2.
>
> I used CQL shell to create a table and insert 3 rows:
> verma@x:~/apache-cassandra-2.2.5$ bin/cqlsh $HOSTNAME
> Connected to Test Cluster at x:9042.
> [cqlsh 5.0.1 | Cassandra 2.2.5 | CQL spec 3.3.1 | Native protocol v4]
> Use HELP for help.
> cqlsh> desc tmp
>
> CREATE KEYSPACE tmp WITH replication = {'class':
> 'NetworkTopologyStrategy', 'dc1': '1', 'dc2': '1'}  AND durable_writes =
> true;
>
> CREATE TABLE tmp.map (
> key text PRIMARY KEY,
> value text
> )...;
> cqlsh> select * from tmp.map;
>
>  key | value
> -+---
>   k1 |v1
>   k3 |v3
>   k2 |v2
>
>
> *Node B:* Downloaded 3.0.8-bin.tar.gz, changed the seeds to include
> itself and node A, changed listen_address and rpc_address to its own IP,
> changed endpoint_snitch to GossipingPropertyFileSnitch. I did not change
> conf/cassandra-rackdc.properties and its contents are
> dc=dc1
> rack=rack1
>
> In the logs, I see:
> INFO  [main] 2016-10-10 22:42:42,850 MessagingService.java:557 - Starting
> Messaging Service on /10.164.32.29:7000 (eth0)
> INFO  [main] 2016-10-10 22:42:42,864 StorageService.java:784 - This node
> will not auto bootstrap because it is configured to be a seed node.
>
> So I start a third node:
> *Node C:* Downloaded 3.0.8-bin.tar.gz, changed the seeds to include node
> A and node B, changed listen_address and rpc_address to its own IP, changed
> endpoint_snitch to GossipingPropertyFileSnitch. I did not change
> conf/cassandra-rackdc.properties.
> Now, nodetool status shows:
>
> verma@xxx:~/apache-cassandra-3.0.8$ bin/nodetool status
> Datacenter: dc1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address   Load   Tokens   Owns (effective)  Host ID
> Rack
> UJ 87.81 KB   256  ?
> 9064832d-ed5c-4c42-ad5a-f754b52b670c  rack1
> UN107.72 KB  256  100.0%
>  28b1043f-115b-46a5-b6b6-8609829cde76  rack1
> Datacenter: dc2
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address   Load   Tokens   Owns (effective)  Host ID
> Rack
> UN  73.2 KB256  100.0%
>  09cc542c-2299-45a5-a4d1-159c239ded37  rack2
>
> Nodetool describe cluster shows:
> verma@xxx:~/apache-cassandra-3.0.8$ bin/nodetool describecluster
> Cluster Information:
> Name: Test Cluster
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> c2a2bb4f-7d31-3fb8-a216-00b41a643650: [, ]
>
> 9770e3c5-3135-32e2-b761-65a0f6d8824e: []
>
> Note that there are two schema versions and they don't match.
>
> I see the following in the system.log:
>
> INFO  [InternalResponseStage:1] 2016-10-10 22:48:36,055
> ColumnFamilyStore.java:390 - Initializing system_auth.roles
> INFO  [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
> waiting for schema information to complete
> INFO  [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
> schema complete, ready to bootstrap
> INFO  [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
> waiting for pending range calculation
> INFO  [main] 2016-10-10 22:48:36,317 StorageService.java:1149 - JOINING:
> calculation complete, ready to bootstrap
> INFO  [main] 2016-10-10 22:48:36,319 StorageService.java:1149 - JOINING:
> getting bootstrap token
> INFO  [main] 2016-10-10 22:48:36,357 StorageService.java:1149 - JOINING:
> sleeping 3 ms for pending range setup
> INFO  [main] 2016-10-10 22:49:06,358 StorageService.java:1149 - JOINING:
> Starting to bootstrap...
> INFO  [main] 2016-10-10 22:49:06,494 StreamResultFuture.java:87 - [Stream
> #bfb5e470-8f3b-11e6-b69a-1b451159408e] Executing streaming plan for
> Bootstrap
> INFO  [StreamConnectionEstablisher:1] 

Re: Collecting slow queries

[Jonathan Haddad]

Love the slow query log, very nice that got done.  I've created some follow
up JIRAs I intend to contribute:

CASSANDRA-13000 - slow query log analysis (basically mysqldumpslow)
CASSANDRA-13001 - pluggable slow query handling
CASSANDRA-13002 - per table slow query threshold

Jon

On Tue, Dec 6, 2016 at 8:40 AM Jeremiah D Jordan 
wrote:

> Per the Fix Version on the ticket it is going to be in 3.10 when that is
> released.  Probably in the next week provided we don’t find any more show
> stopper bugs.
>
> -Jeremiah
>
> > On Dec 6, 2016, at 10:38 AM, Jan  wrote:
> >
> > Hello Yoshi-san;
> > is this fix rolled into  Cassandra 3.7.0 ?  I do not see it in the
> cassandra.yaml file.
> > Is there anything special that needs to be downloaded for this feature
> to show up that I am missing in  Cassandra 3.7.0.
> > Thank you for your prompt response earlier, Jan
> >
> >
> >On Monday, December 5, 2016 3:42 PM, Yoshi Kimoto <
> yy.kim...@gmail.com> wrote:
> >
> >
> > This? : https://issues.apache.org/jira/browse/CASSANDRA-12403
> >
> > 2016-12-06 6:36 GMT+09:00 Jeff Jirsa :
> >
> >> Should we reopen 6226? Tracing 0.1% doesn’t help find the outliers that
> >> are slow but don’t time out (slow query log could help find large
> >> partitions for users with infrequent but painful large partitions, far
> >> easier than dumping sstables to json to identify them).
> >>
> >>
> >> On 12/5/16, 1:28 PM, "sankalp kohli"  wrote:
> >>
> >>> This is duped by a JIRA which is fixed in 3.2
> >>>
> >>> https://issues.apache.org/jira/browse/CASSANDRA-6226
> >>>
> >>> On Mon, Dec 5, 2016 at 12:15 PM, Jan  wrote:
> >>>
>  HI Folks;
>  is there a way for 'Collecting slow queries'  in the Apache Cassandra.
> >> ?I
>  am aware of the DSE product offering such an option, but need the
> >> solution
>  on Apache Cassandra.
>  ThanksJan
> >>
> >
>
>

Re: Wrapping up tick-tock

[Jonathan Haddad]

Based on the rate of change in Tick Tock, I really doubt it'll be a problem.

On Fri, Jan 13, 2017 at 11:07 AM sankalp kohli <kohlisank...@gmail.com>
wrote:

> + to Jason idea. We should have a minimum of 6 months between a major
> version.
>
> On Fri, Jan 13, 2017 at 10:57 AM, Jason Brown <jasedbr...@gmail.com>
> wrote:
>
> > It's fine to limit the minimum time between major releases to six months,
> > but I do not think we should force a major just because n months have
> > passed. I think we should up the major only when we have significant
> > (possibly breaking) changes/features. It would seem odd to have a 6.0
> > that's basically the same as 4.0 (in terms of features and
> protocol/format
> > compatibility).
> >
> > Thoughts?
> >
> > On Wed, Jan 11, 2017 at 1:58 AM, Stefan Podkowinski <spo...@gmail.com>
> > wrote:
> >
> > > I honestly don't understand the release cadence discussion. The 3.x
> > branch
> > > is far from production ready. Is this really the time to plan the next
> > > major feature releases on top of it, instead of focusing to stabilize
> 3.x
> > > first? Who knows how long that would take, even if everyone would
> > > exclusively work on bug fixing (which I think should happen).
> > >
> > > On Tue, Jan 10, 2017 at 4:29 PM, Jonathan Haddad <j...@jonhaddad.com>
> > > wrote:
> > >
> > > > I don't see why it has to be one extreme (yearly) or another
> (monthly).
> > > > When you had originally proposed Tick Tock, you wrote:
> > > >
> > > > "The primary goal is to improve release quality.  Our current major
> > “dot
> > > > zero” releases require another five or six months to make them stable
> > > > enough for production.  This is directly related to how we pile
> > features
> > > in
> > > > for 9 to 12 months and release all at once.  The interactions between
> > the
> > > > new features are complex and not always obvious.  2.1 was no
> exception,
> > > > despite DataStax hiring a full tme test engineering team specifically
> > for
> > > > Apache Cassandra."
> > > >
> > > > I agreed with you at the time that the yearly cycle was too long to
> be
> > > > adding features before cutting a release, and still do now.  Instead
> of
> > > > elastic banding all the way back to a process which wasn't working
> > > before,
> > > > why not try somewhere in the middle?  A release every 6 months (with
> > > > monthly bug fixes for a year) gives:
> > > >
> > > > 1. long enough time to stabilize (1 year vs 1 month)
> > > > 2. not so long things sit around untested forever
> > > > 3. only 2 releases (current and previous) to do bug fix support at
> any
> > > > given time.
> > > >
> > > > Jon
> > > >
> > > > On Tue, Jan 10, 2017 at 6:56 AM Jonathan Ellis <jbel...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > We’ve had a few threads now about the successes and failures of the
> > > > > tick-tock release process and what to do to replace it, but they
> all
> > > died
> > > > > out without reaching a robust consensus.
> > > > >
> > > > > In those threads we saw several reasonable options proposed, but
> from
> > > my
> > > > > perspective they all operated in a kind of theoretical fantasy land
> > of
> > > > > testing and development resources.  In particular, it takes around
> a
> > > > > person-week of effort to verify that a release is ready.  That is,
> > > going
> > > > > through all the test suites, inspecting and re-running failing
> tests
> > to
> > > > see
> > > > > if there is a product problem or a flaky test.
> > > > >
> > > > > (I agree that in a perfect world this wouldn’t be necessary because
> > > your
> > > > > test ci is always green, but see my previous framing of the perfect
> > > world
> > > > > as a fantasy land.  It’s also worth noting that this is a common
> > > problem
> > > > > for large OSS projects, not necessarily something to beat ourselves
> > up
> > > > > over, but in any case, that's our reality right now.)
> > > > >
> > > > > I submit that any process that assume

Re: [VOTE] 3.X branch feature freeze

[Jonathan Haddad]

+1 (non binding) to feature freeze.

I also like the idea of stabilizing MVs.  Ben, you've probably been the
most vocal about the issues, have you made any progress towards making them
work any better during bootstrap / etc?  Any idea of fixing them is a major
undertaking?

Jon

On Fri, Jan 13, 2017 at 9:39 AM Benjamin Roth 
wrote:

+1 also I appreciate any effort on MV stability. It is an official 3.x
feature but not production ready for the masses.

Am 13.01.2017 18:34 schrieb "Jonathan Ellis" :

> +1
>
> On Fri, Jan 13, 2017 at 11:21 AM, Aleksey Yeschenko 
> wrote:
>
> > Hi all!
> >
> > It seems like we have a general consensus on ending tick-tock at 3.11,
> and
> > moving
> > on to stabilisation-only for 3.11.x series.
> >
> > In light of this, I suggest immediate feature freeze in the 3.X branch.
> >
> > Meaning that only bug fixes go to the 3.11/3.X branch from now on.
> >
> > All new features that haven’t be committed yet should go to trunk only
> > (4.0), if the vote passes.
> >
> > What do you think?
> >
> > Thanks.
> >
> > --
> > AY
>
>
>
>
> --
> Jonathan Ellis
> co-founder, http://www.datastax.com
> @spyced
>

Re: Rollback procedure for Cassandra Upgrade.

[Jonathan Haddad]

There's no downgrade procedure. You either upgrade or you go back to a
snapshot from the previous version.
On Mon, Jan 9, 2017 at 8:13 PM Prakash Chauhan 
wrote:

> Hi All ,
>
> Do we have an official procedure to rollback the upgrade of C* from 2.0.x
> to 2.1.x ?
>
>
> Description:
> I have upgraded C* from 2.0.x to 2.1.x . As a part of upgrade procedure ,
> I have to run nodetool upgradesstables .
> What if the command fails in the middle ? Some of the sstables will be in
> newer format (*-ka-*) where as other might be in older format(*-jb-*).
>
> Do we have a standard procedure to do rollback in such cases?
>
>
>
> Regards,
> Prakash Chauhan.
>
>

Re: Wrapping up tick-tock

[Jonathan Haddad]

I don't see why it has to be one extreme (yearly) or another (monthly).
When you had originally proposed Tick Tock, you wrote:

"The primary goal is to improve release quality.  Our current major “dot
zero” releases require another five or six months to make them stable
enough for production.  This is directly related to how we pile features in
for 9 to 12 months and release all at once.  The interactions between the
new features are complex and not always obvious.  2.1 was no exception,
despite DataStax hiring a full tme test engineering team specifically for
Apache Cassandra."

I agreed with you at the time that the yearly cycle was too long to be
adding features before cutting a release, and still do now.  Instead of
elastic banding all the way back to a process which wasn't working before,
why not try somewhere in the middle?  A release every 6 months (with
monthly bug fixes for a year) gives:

1. long enough time to stabilize (1 year vs 1 month)
2. not so long things sit around untested forever
3. only 2 releases (current and previous) to do bug fix support at any
given time.

Jon

On Tue, Jan 10, 2017 at 6:56 AM Jonathan Ellis  wrote:

> Hi all,
>
> We’ve had a few threads now about the successes and failures of the
> tick-tock release process and what to do to replace it, but they all died
> out without reaching a robust consensus.
>
> In those threads we saw several reasonable options proposed, but from my
> perspective they all operated in a kind of theoretical fantasy land of
> testing and development resources.  In particular, it takes around a
> person-week of effort to verify that a release is ready.  That is, going
> through all the test suites, inspecting and re-running failing tests to see
> if there is a product problem or a flaky test.
>
> (I agree that in a perfect world this wouldn’t be necessary because your
> test ci is always green, but see my previous framing of the perfect world
> as a fantasy land.  It’s also worth noting that this is a common problem
> for large OSS projects, not necessarily something to beat ourselves up
> over, but in any case, that's our reality right now.)
>
> I submit that any process that assumes a monthly release cadence is not
> realistic from a resourcing standpoint for this validation.  Notably, we
> have struggled to marshal this for 3.10 for two months now.
>
> Therefore, I suggest first that we collectively roll up our sleeves to vet
> 3.10 as the last tick-tock release.  Stick a fork in it, it’s done.  No
> more tick-tock.
>
> I further suggest that in place of tick tock we go back to our old model of
> yearly-ish releases with as-needed bug fix releases on stable branches,
> probably bi-monthly.  This amortizes the release validation problem over a
> longer development period.  And of course we remain free to ramp back up to
> the more rapid cadence envisioned by the other proposals if we increase our
> pool of QA effort or we are able to eliminate flakey tests to the point
> that a long validation process becomes unnecessary.
>
> (While a longer dev period could mean a correspondingly more painful test
> validation process at the end, my experience is that most of the validation
> cost is “fixed” in the form of flaky tests and thus does not increase
> proportionally to development time.)
>
> Thoughts?
>
> --
> Jonathan Ellis
> co-founder, http://www.datastax.com
> @spyced
>

splitting CQL parser & spec into separate repo

[Jonathan Haddad]

I created CASSANDRA-13284 a few days ago with the intent of starting a
discussion around the topic of breaking the CQL parser out into a separate
project.  I see a few benefits to doing it and was wondering what the folks
here thought as well.

First off, the Java CQL parser would obviously continue to be the reference
parser.  I'd love to see other languages have CQL parsers as well, but the
intent here isn't for the OSS C* team to be responsible for maintaining
that.  My vision here is simply the ability to have some high level
CQLParser.parse(statement) call that returns the parse tree, nothing more.

It would be nice to be able to leverage that parser in other projects such
as IDEs, code gen tools, etc.  It would be outstanding to be able to create
the parser tests in such a way that they can be referenced by other parsers
in other languages.  Yay code reuse.  It also has the benefit of making the
codebase a little more modular and a bit easier to understand.

Thoughts?

Jon

Re: [VOTE] self-assignment of jira tickets

[Jonathan Haddad]

Non binding +1
On Wed, Mar 29, 2017 at 7:22 AM Jeff Jirsa  wrote:

> +1
>
> --
> Jeff Jirsa
>
>
> > On Mar 29, 2017, at 6:21 AM, Jason Brown  wrote:
> >
> > Hey all,
> >
> > Following up my thread from a week or two ago (
> >
> https://lists.apache.org/thread.html/0665f40c7213654e99817141972c003a2131aba7a1c63d6765db75c5@%3Cdev.cassandra.apache.org%3E
> ),
> > I'd like to propose a vote to change to allow any potential contributor
> to
> > assign a jira to themselves without needing to be added to the
> contributors
> > group first.
> >
> > https://issues.apache.org/jira/browse/INFRA-11950 is an example of how
> to
> > get this done with INFRA.
> >
> > Vote will be open for 72 hours.
> >
> > Thanks,
> >
> > -Jason Brown
>

Re: Can we kill the wiki?

[Jonathan Haddad]

If there was a single instance of collaborative design in the wiki I would
agree with you.

Google docs have been used a bit recently and have worked well.
On Sun, Mar 19, 2017 at 8:17 AM Edward Capriolo <edlinuxg...@gmail.com>
wrote:

> Wikis are still good for collaberative design etc. Its a burden to edit the
> docs and its not the place for all info.
>
> On Friday, March 17, 2017, Murukesh Mohanan <murukesh.moha...@gmail.com>
> wrote:
>
> > I wonder if the recent influx has anything to do with GSoC. The student
> > application period begins in a few days. I don't see any Cassandra issues
> > on the GSoC ideas list, though.
> >
> > On Sat, 18 Mar 2017 at 10:40 Anthony Grasso <anthony.gra...@gmail.com
> > <javascript:;>>
> > wrote:
> >
> > +1 to killing the wiki as well. If that is not possible, we should at
> least
> > put a note on there saying it is deprecated and point people to the new
> > docs.
> >
> > On 18 March 2017 at 08:09, Jonathan Haddad <j...@jonhaddad.com
> > <javascript:;>> wrote:
> >
> > > +1 to killing the wiki.
> > >
> > > On Fri, Mar 17, 2017 at 2:08 PM Blake Eggleston <beggles...@apple.com
> > <javascript:;>>
> > > wrote:
> > >
> > > > With CASSANDRA-8700, docs were moved in tree, with the intention that
> > > they
> > > > would replace the wiki. However, it looks like we’re still getting
> > > regular
> > > > requests to edit the wiki. It seems like we should be directing these
> > > folks
> > > > to the in tree docs and either disabling edits for the wiki, or just
> > > > removing it entirely, and replacing it with a link to the hosted
> docs.
> > > I'd
> > > > prefer we just remove it myself, makes things less confusing for
> > > newcomers.
> > > >
> > > > Does that seem reasonable to everyone?
> > >
> >
> > --
> >
> > Murukesh Mohanan,
> > Yahoo! Japan
> >
>
>
> --
> Sorry this was sent from mobile. Will do less grammar and spell check than
> usual.
>

Re: Can we kill the wiki?

[Jonathan Haddad]

+1 to killing the wiki.

On Fri, Mar 17, 2017 at 2:08 PM Blake Eggleston 
wrote:

> With CASSANDRA-8700, docs were moved in tree, with the intention that they
> would replace the wiki. However, it looks like we’re still getting regular
> requests to edit the wiki. It seems like we should be directing these folks
> to the in tree docs and either disabling edits for the wiki, or just
> removing it entirely, and replacing it with a link to the hosted docs. I'd
> prefer we just remove it myself, makes things less confusing for newcomers.
>
> Does that seem reasonable to everyone?

Re: Code quality, principles and rules

[Jonathan Haddad]

I'd like to think that if someone refactors existing code, making it more
testable (with tests, of course) it should be acceptable on it's own
merit.  In fact, in my opinion it sometimes makes more sense to do these
types of refactorings for the sole purpose of improving stability and
testability as opposed to mixing them with features.

You referenced the issue I fixed in one of the early emails.  The fix
itself was a couple lines of code.  Refactoring the codebase to make it
testable would have been a huge effort.  I wish I had time to do it.  I
created CASSANDRA-13007 as a follow up with the intent of working on
compaction from a purely architectural standpoint.  I think this type of
thing should be done throughout the codebase.

Removing the singletons is a good first step, my vote is we just rip off
the bandaid, do it, and move forward.

On Fri, Mar 17, 2017 at 2:20 PM Edward Capriolo 
wrote:

> On Fri, Mar 17, 2017 at 2:31 PM, Jason Brown  wrote:
>
> > To François's point about code coverage for new code, I think this makes
> a
> > lot of sense wrt large features (like the current work on
> 8457/12229/9754).
> > It's much simpler to (mentally, at least) isolate those changed sections
> > and it'll show up better in a code coverage report. With small patches,
> > that might be harder to achieve - however, as the patch should come with
> > *some* tests (unless it's a truly trivial patch), it might just work
> itself
> > out.
> >
> > On Fri, Mar 17, 2017 at 11:19 AM, Jason Brown 
> > wrote:
> >
> > > As someone who spent a lot of time looking at the singletons topic in
> the
> > > past, Blake brings a great perspective here. Figuring out and
> > communicating
> > > how best to test with the system we have (and of course incrementally
> > > making that system easier to work with/test) seems like an achievable
> > goal.
> > >
> > > On Fri, Mar 17, 2017 at 10:17 AM, Edward Capriolo <
> edlinuxg...@gmail.com
> > >
> > > wrote:
> > >
> > >> On Fri, Mar 17, 2017 at 12:33 PM, Blake Eggleston <
> beggles...@apple.com
> > >
> > >> wrote:
> > >>
> > >> > I think we’re getting a little ahead of ourselves talking about DI
> > >> > frameworks. Before that even becomes something worth talking about,
> > we’d
> > >> > need to have made serious progress on un-spaghettifying Cassandra in
> > the
> > >> > first place. It’s an extremely tall order. Adding a DI framework
> right
> > >> now
> > >> > would be like throwing gasoline on a raging tire fire.
> > >> >
> > >> > Removing singletons seems to come up every 6-12 months, and usually
> > >> > abandoned once people figure out how difficult they are to remove
> > >> properly.
> > >> > I do think removing them *should* be a long term goal, but we really
> > >> need
> > >> > something more immediately actionable. Otherwise, nothing’s going to
> > >> > happen, and we’ll be having this discussion again in a year or so
> when
> > >> > everyone’s angry that Cassandra 5.0 still isn’t ready for
> production,
> > a
> > >> > year after it’s release.
> > >> >
> > >> > That said, the reason singletons regularly get brought up is because
> > >> doing
> > >> > extensive testing of anything in Cassandra is pretty much
> impossible,
> > >> since
> > >> > the code is basically this big web of interconnected global state.
> > >> Testing
> > >> > anything in isolation can’t be done, which, for a distributed
> > database,
> > >> is
> > >> > crazy. It’s a chronic problem that handicaps our ability to release
> a
> > >> > stable database.
> > >> >
> > >> > At this point, I think a more pragmatic approach would be to draft
> and
> > >> > enforce some coding standards that can be applied in day to day
> > >> development
> > >> > that drive incremental improvement of the testing and testability of
> > the
> > >> > project. What should be tested, how it should be tested. How to
> write
> > >> new
> > >> > code that talks to the rest of Cassandra and is testable. How to fix
> > >> bugs
> > >> > in old code in a way that’s testable. We should also have some
> > >> guidelines
> > >> > around refactoring the wildly untested sections, how to get started,
> > >> what
> > >> > to do, what not to do, etc.
> > >> >
> > >> > Thoughts?
> > >>
> > >>
> > >> To make the conversation practical. There is one class I personally
> > really
> > >> want to refactor so it can be tested:
> > >>
> > >> https://github.com/apache/cassandra/blob/trunk/src/java/org/
> > >> apache/cassandra/net/OutboundTcpConnection.java
> > >>
> > >> There is little coverage here. Questions like:
> > >> what errors cause the connection to restart?
> > >> when are undropable messages are dropped?
> > >> what happens when the queue fills up?
> > >> Infamous throw new AssertionError(ex); (which probably bubble up to
> > >> nowhere)
> > >> what does the COALESCED strategy do in case XYZ.
> > >> A nifty label (wow a label you just never see those much!)
> > >> outer:
> > >> 

Re: [VOTE] Ask Infra to move github notification emails to commits@

[Jonathan Haddad]

+1

On Mon, Mar 20, 2017 at 3:02 PM Brandon Williams  wrote:

> +1, we've had to explain this a thousand times here.
>
> On Mon, Mar 20, 2017 at 5:00 PM, Jeff Jirsa  wrote:
>
> > There's no reason for the dev list to get spammed everytime there's a
> > github PR. We know most of the time we prefer JIRAs for real code PRs,
> but
> > with docs being in tree and low barrier to entry, we may want to accept
> > docs through PRs ( see https://issues.apache.org/
> > jira/browse/CASSANDRA-13256
> > , and comment on it if you disagree).
> >
> > To make that viable, we should make it not spam dev@ with every comment.
> > Therefore I propose we move github PR comments/actions to commits@ so as
> > not to clutter the dev@ list.
> >
> > Voting to remain open for 72 hours.
> >
> > - Jeff
> >
>

Re: [VOTE] Ask Infra to move github notification emails to pr@

[Jonathan Haddad]

+1
On Mon, Mar 20, 2017 at 6:33 PM Jason Brown  wrote:

> +1
> On Mon, Mar 20, 2017 at 18:21 Anthony Grasso 
> wrote:
>
> > +1
> >
> > On 21 March 2017 at 09:32, Jeff Jirsa  wrote:
> >
> > > There's no reason for the dev list to get spammed everytime there's a
> > > github PR. We know most of the time we prefer JIRAs for real code PRs,
> > but
> > > with docs being in tree and low barrier to entry, we may want to accept
> > > docs through PRs ( see https://issues.apache.org/
> > > jira/browse/CASSANDRA-13256
> > > , and comment on it if you disagree).
> > >
> > > To make that viable, we should make it not spam dev@ with every
> comment.
> > > Therefore I propose we move github PR comments/actions to pr@ so as
> > > not to clutter the dev@ list.
> > >
> > > Voting to remain open for 72 hours.
> > >
> > > - Jeff
> > >
> >
>

Re: Testing and jira tickets

[Jonathan Haddad]

No problem, I'll start a new thread.

On Thu, Mar 9, 2017 at 11:48 AM Jason Brown <jasedbr...@gmail.com> wrote:

> Jon and Brandon,
>
> I'd actually like to narrow the discussion, and keep it focused to my
> original topic. Those are two excellent topics that should be addressed,
> and the solution(s) might be the same or similar as the outcome of this.
> However, I feel they deserve their own message thread.
>
> Thanks for understanding,
>
> -Jason
>
> On Thu, Mar 9, 2017 at 11:27 AM, Brandon Williams <dri...@gmail.com>
> wrote:
>
> > Let me further broaden this discussion to include github branches, which
> > are often linked on tickets, and then later deleted.  This forces a
> person
> > to search through git to actually see the patch, and that process can be
> a
> > little rough (especially since we all know if you're gonna make a typo,
> > it's going to be in the commit, and it's probably going to be the ticket
> > number.)
> >
> > On Thu, Mar 9, 2017 at 1:00 PM, Jonathan Haddad <j...@jonhaddad.com>
> wrote:
> >
> > > If you don't mind, I'd like to broaden the discussion a little bit to
> > also
> > > discuss performance related patches.  For instance, CASSANDRA-13271
> was a
> > > performance / optimization related patch that included *zero*
> information
> > > on if there was any perf improvement or a regression as a result of the
> > > change, even though I've asked twice for that information.
> > >
> > > In addition to "does this thing break anything" we should be asking
> "how
> > > does this patch affect performance?" (and were the appropriate docs
> > > included, but that's another topic altogether)
> > >
> > > On Thu, Mar 9, 2017 at 10:51 AM Jason Brown <jasedbr...@gmail.com>
> > wrote:
> > >
> > > > Hey all,
> > > >
> > > > A nice convention we've stumbled into wrt to patches submitted via
> Jira
> > > is
> > > > to post the results of unit test and dtest runs to the ticket (to
> show
> > > the
> > > > patch doesn't break things). Many contributors have used the
> > > > DataStax-provided cassci system, but that's not the best long term
> > > > solution. To that end, I'd like to start a conversation about what is
> > the
> > > > best way to proceed going forward, and then add it to the "How to
> > > > contribute" docs.
> > > >
> > > > As an example, should contributors/committers run dtests and unit
> tests
> > > on
> > > > *some* machine (publicly available or otherwise), and then post those
> > > > results to the ticket? This could be a link to a build system, like
> > what
> > > we
> > > > have with cassci, or just  upload the output of the test run(s).
> > > >
> > > > I don't have any fixed notions, and am looking forward to hearing
> > other's
> > > > ideas.
> > > >
> > > > Thanks,
> > > >
> > > > -Jason
> > > >
> > > > p.s. a big thank you to DataStax for providing the cassci system
> > > >
> > >
> >
>

Re: Testing and jira tickets

[Jonathan Haddad]

If you don't mind, I'd like to broaden the discussion a little bit to also
discuss performance related patches.  For instance, CASSANDRA-13271 was a
performance / optimization related patch that included *zero* information
on if there was any perf improvement or a regression as a result of the
change, even though I've asked twice for that information.

In addition to "does this thing break anything" we should be asking "how
does this patch affect performance?" (and were the appropriate docs
included, but that's another topic altogether)

On Thu, Mar 9, 2017 at 10:51 AM Jason Brown  wrote:

> Hey all,
>
> A nice convention we've stumbled into wrt to patches submitted via Jira is
> to post the results of unit test and dtest runs to the ticket (to show the
> patch doesn't break things). Many contributors have used the
> DataStax-provided cassci system, but that's not the best long term
> solution. To that end, I'd like to start a conversation about what is the
> best way to proceed going forward, and then add it to the "How to
> contribute" docs.
>
> As an example, should contributors/committers run dtests and unit tests on
> *some* machine (publicly available or otherwise), and then post those
> results to the ticket? This could be a link to a build system, like what we
> have with cassci, or just  upload the output of the test run(s).
>
> I don't have any fixed notions, and am looking forward to hearing other's
> ideas.
>
> Thanks,
>
> -Jason
>
> p.s. a big thank you to DataStax for providing the cassci system
>

committing performance patches without performance numbers

[Jonathan Haddad]

I'd like to discuss what I consider to be a pretty important matter -
patches which are written for the sole purpose of improving performance
without including a single performance benchmark in the JIRA.

My original email was in "Testing and Jira Tickets", i'll copy it here
for posterity:

If you don't mind, I'd like to broaden the discussion a little bit to also
discuss performance related patches.  For instance, CASSANDRA-13271 was a
performance / optimization related patch that included *zero* information
on if there was any perf improvement or a regression as a result of the
change, even though I've asked twice for that information.

In addition to "does this thing break anything" we should be asking "how
does this patch affect performance?" (and were the appropriate docs
included, but that's another topic altogether)

There's a minor note about perf related stuff here:
http://cassandra.apache.org/doc/latest/development/testing.html#performance-testing


"Performance tests for Cassandra are a special breed of tests that are not
part of the usual patch contribution process. In fact you can contribute
tons of patches to Cassandra without ever running performance tests. They
are important however when working on performance improvements, as such
improvements must be measurable."

I think performance numbers aren't just important, but should be a
*requirement* to merge a performance patch.

Thoughts?
Jon

Re: committing performance patches without performance numbers

[Jonathan Haddad]

I don't expect everyone to run a 500 node cluster off to the side to test
their patches, but at least some indication that the contributor started
Cassandra on their laptop would be a good sign.  The JIRA I referenced was
an optimization around List, Set and Map serialization.  Would it really
have been that crazy to run a handful of benchmarks locally and post those
results?

On Thu, Mar 9, 2017 at 12:26 PM Ariel Weisberg <ar...@weisberg.ws> wrote:

> Hi,
>
> I think there are issues around the availability of hardware sufficient
> to demonstrate the performance concerns under test. It's an open source
> project without centralized infrastructure. A lot of performance
> contributions come from people running C* in production. They are
> already running these changes and have seen the improvement, but
> communicating that to the outside world in a convincing way can be hard.
>
> Right now we don't even have performance in continuous integration. I
> think we are putting the cart before the horse in that respect. What
> about all the commits that don't intend to have a performance impact but
> do? Even if we had performance metrics in CI who is going to triage the
> results religiously?
>
> We also to my knowledge don't have benchmarks for key functionality in
> cassandra-stress. Can cassandra-stress benchmark CAS? My recollection is
> every time I looked is that it wasn't there.
>
> We can only set the bar as high as contributors are able to meet.
> Certainly if they can't justify why they can't benchmark the thing they
> want to contribute then reviewers should make them go and benchmark it.
>
> Regards,
> Ariel
>
> On Thu, Mar 9, 2017, at 03:11 PM, Jeff Jirsa wrote:
> > Agree. Anything that's meant to increase performance should demonstrate
> > it
> > actually does that. We have microbench available in recent versions -
> > writing a new microbenchmark isn't all that onerous. Would be great if we
> > had perf tests included in the normal testall/dtest workflow for ALL
> > patches so we could quickly spot regressions, but that gets pretty
> > expensive in terms of running long enough tests to actually see most
> > common
> > code paths.
> >
> >
> > On Thu, Mar 9, 2017 at 12:00 PM, Jonathan Haddad <j...@jonhaddad.com>
> > wrote:
> >
> > > I'd like to discuss what I consider to be a pretty important matter -
> > > patches which are written for the sole purpose of improving performance
> > > without including a single performance benchmark in the JIRA.
> > >
> > > My original email was in "Testing and Jira Tickets", i'll copy it here
> > > for posterity:
> > >
> > > If you don't mind, I'd like to broaden the discussion a little bit to
> also
> > > discuss performance related patches.  For instance, CASSANDRA-13271
> was a
> > > performance / optimization related patch that included *zero*
> information
> > > on if there was any perf improvement or a regression as a result of the
> > > change, even though I've asked twice for that information.
> > >
> > > In addition to "does this thing break anything" we should be asking
> "how
> > > does this patch affect performance?" (and were the appropriate docs
> > > included, but that's another topic altogether)
> > >
> > > There's a minor note about perf related stuff here:
> > > http://cassandra.apache.org/doc/latest/development/
> > > testing.html#performance-testing
> > >
> > >
> > > "Performance tests for Cassandra are a special breed of tests that are
> not
> > > part of the usual patch contribution process. In fact you can
> contribute
> > > tons of patches to Cassandra without ever running performance tests.
> They
> > > are important however when working on performance improvements, as such
> > > improvements must be measurable."
> > >
> > > I think performance numbers aren't just important, but should be a
> > > *requirement* to merge a performance patch.
> > >
> > > Thoughts?
> > > Jon
> > >
>

Re: Contribute to the Cassandra wiki

[Jonathan Haddad]

Ugh... Let's put a few facts out in the open before we start pushing to
move back to the wiki.

First off, take a look at CASSANDRA-8700.  There's plenty of reasoning for
why the docs are now located in tree.  The TL;DR is:

1. Nobody used the wiki.  Like, ever.  A handful of edits per year.
2. Docs in the wiki were out of sync w/ cassandra.  Trying to outline the
difference in implementations w/ nuanced behavior was difficult /
impossible.  With in-tree, you just check the docs that come w/ the version
you installed.  And you get them locally.  Huzzah!
3. The in-tree docs are a million times better quality than the wiki *ever*
was.

I urge you to try giving the in-tree docs a chance.  It may not be the way
*you* want it but I have to point out that they're the best we've seen in
Cassandra world.  Making them prettier won't help anything.

I do agree that the process needs to be a bit smoother for people to add
stuff to the in tree side.  For instance, maybe for every features that's
written we start creating a corresponding JIRA for the documentation.  Not
every developer wants to write docs, and that's fair.  The accompanying
JIRA would serve as a way for 2 or more people to collaborate on the
feature & the docs in tandem.  It may also be beneficial to use the dev-ml
to say "hey, i'm working on feature X, anyone want to help me write the
docs for it?  check out CASSANDRA-XYZ"

Part of CASSANDRA-8700 was to shut down the wiki.  I still advocate for
this. At the very minimum we should make it read only with a big notice
that points people to the in-tree docs.

On Mon, Mar 13, 2017 at 8:49 AM Jeremy Hanna 
wrote:

> The moinmoin wiki was preferred but because of spam, images couldn’t be
> attached.  The options were to use confluence or have a moderated list of
> individuals be approved to update the wiki.  The decision was made to go
> with the latter because of the preference to stick with moinmoin rather
> than confluence.  That’s my understanding of the history there.  I don’t
> know if people would like to revisit using one or the other at this point,
> though it would take a bit of work to convert.
>
> > On Mar 13, 2017, at 9:42 AM, Nate McCall  wrote:
> >
> >> Isn't there a way to split tech docs (aka reference) and more
> >> user-generated and use-case related/content oriented docs? And maybe to
> use
> >> a more modern WIKI software or scheme. The CS wiki looks like 1998.
> >
> > The wiki is what ASF Infra provides by default. Agree that it is a bit
> > "old-school."
> >
> > I'll ask around about what other projects are doing (or folks who are
> > involved in other ASF projects, please chime in).
>
>

Re: Integrating vendor-specific code and developing plugins

[Jonathan Haddad]

There's a handful of issues I can think of with shipping everything
in-tree.  I'll try to brain dump them here.

* What's included when shipped in tree?

Does every idea get merged in? Do we need 30 different Seed providers?  Who
judges what's valuable enough to add?  Who maintains it when it needs
updating?  If the maintainer can't be found, is it removed?  Shipped
broken?  Does the contributed plugins go through the same review process?
Do the contributors need to be committers?  Would CASSANDRA-12627 be merged
in even if nobody saw the value?

* Language support

Cassandra is based on Java 8.  Do we now merge in Scala, Kotlin, Jython?

* Plugins are now tied directly to cassandra release cycle

This one bugs me quite a bit.  With a new plugin, there's going to be a lot
of rapid iterations.  Spark releases once every 3 months - a lot of the
packages seem to be released at a much higher frequency.

* In Python, the standard lib is where modules go to die

I forgot where I heard this, but it's pretty accurate.  Including
everything, "batteries includes", just ends up shipping some pretty
terrible batteries.  The best stuff for python is on pypi.

Rust deliberately made the decision to limit the std to avoid this
problem.  There's a "nursery" [1] area for ideas to evolve independently,
and when some code reaches a high enough level of maturity, it can get
merged in.  There's also a packages system for third party, non std
destined code.

Anyways - I'm very +1 on a package system where codebases can independently
evolve without needing to be part of the project itself.  It's a proven
model for shipping high quality, useful code, and sometimes is even one of
the best aspects of a project.  That said, it's quite a bit of work just to
get going and someone would have to manage that.

Jon

[1] https://github.com/rust-lang-nursery


On Sun, May 14, 2017 at 9:03 PM Jeff Jirsa  wrote:

> On Fri, May 12, 2017 at 9:31 PM, J. D. Jordan 
> wrote:
>
> > In tree would foster more committers which is a good thing.
>
>
> Definitely.
>
> But I also agree that not being able to actually run unit tests is a bad
> > thing. What if we asked people who want to contribute these types of
> > optimizations to provide the ASF with a Jenkins slave we could test them
> > on, if they want them in tree?
> >
>
> I think SOME FORM of jenkins/unit/dtests need to exist, whether it's ASF
> puppet controlled or test output explicitly provided by the maintainer.
>
>
> > Otherwise one good thing about out of tree is that the maintainer can
> > specify "this plugin has been tested and works with Apache Cassandra
> > version X.Y.Z". If it is in tree it is assumed it will work. If it is out
> > of tree then the plugin can more easily notify a user what version it was
> > last tested with.  And users won't be surprised if they upgrade and
> things
> > break.
> >
>
> My main desire is that I'd like to see us do better at helping third party
> contributors be successful in contributing, and to me that means something
> more official. I like the spark packages model. I like the apache httpd
> model (with LOTS of official extensions in-tree, but a lot externally
> packaged as well). I'm not a fan of telling people to build and distribute
> their own JARs - it doesn't help the contributors, it doesn't help the
> users, and it doesn't help the project.
>
> - Jeff
>

rebuilding cassandra.apache.org docs

[Jonathan Haddad]

Looks like the docs haven't been rebuilt in a while.  There's a handful of
useful merges recently that I'm aware of that would be great to see on
there.

How do they get rebuilt?  Who can trigger it?

Jon

Re: Integrating vendor-specific code and developing plugins

[Jonathan Haddad]

In accordance with the idea that the codebase should be better tested, it
seems to me like things shouldn't be added that aren't testable.  If
there's a million unit tests that are insanely comprehensive but for some
reason can never be run, they serve exactly the same value as no tests.

It may be better to figure out how to foster a plugin ecosystem, which is a
bit better than "there's an API go deal with it".  This is what Spark is
doing and it seems like a pretty reasonable approach to me:
https://spark-packages.org/

On Fri, May 12, 2017 at 9:03 PM Jeff Jirsa  wrote:

> I think the status quo is insufficient - even if it doesn't go in tree, we
> should do more than just say "the API exists, ship your own jar"
>
> What's the real risk of having it in tree? We break it because nobody can
> test it? How's that any worse than breaking it outside the tree? Finger
> pointing?
>
> --
> Jeff Jirsa
>
>
> > On May 12, 2017, at 12:25 PM, Jason Brown  wrote:
> >
> > I agree the plugins route is the safest and best. However, we already
> have
> > platform-specific code in-tree that is semi-unmaintained: the Windows
> > support. To Sylvain's point, I have little to no idea if I'm going to
> break
> > the Windows builds as I don't have access to a Windows machine, nor are
> we
> > as a community (as best as I can tell) actively running unit tests or
> > dtests on Windows.
> >
> > Further, we support snitches for clouds I suspect we don't typically
> > run/test on, as well: CloudstackSnitch, GoogleCloudSnitch.
> >
> > This being said, I don't think we should remove support for Windows or
> > those snitches. Instead, what I think would be more beneficial, and
> > certainly more reflecting the Apache Way, is to see if someone in the
> > community would be willing to maintain those components. Looking at
> another
> > Apache project, Mesos has an interesting breakdown of maintainers for
> > specific components [1]. We might consider adopting a similar idea for
> > platforms/OSes/architectures/whatevers.
> >
> > As for where to put the custom code, there's a few different options:
> >
> > bare minimum: we should have docs pointing to all known third party
> > implementations of pluggable interfaces
> > slightly more involved: contrib/ section of third-party contributed
> plugins
> > even more involved: in tree like gcp / aws snitches
> >
> > I'm not really thrilled on the contribs repo, and in-tree certainly has
> > drawbacks, as well. As I initially stated, it can be on a case-by-case
> > basis.
> >
> > Honestly, I don't want to push away contributors if they can add
> something
> > to the project - as long as it is maintainable.
> >
> > Thanks,
> >
> > -Jason
> >
> >
> > [1] https://mesos.apache.org/documentation/latest/committers/
> >
> > On Fri, May 12, 2017 at 4:35 AM, Sylvain Lebresne 
> > wrote:
> >
> >> On Fri, May 12, 2017 at 12:29 AM, Jason Brown 
> >> wrote:
> >>
> >>> Hey all,
> >>>
> >>> I'm on-board with what Rei is saying. I think we should be open to, and
> >>> encourage, other platforms/architectures for integration. Of course, it
> >>> will come down to specific maintainers/committers to do the testing and
> >>> verification on non-typical platforms. Hopefully those maintainers will
> >>> also contribute to other parts of the code base, as well, so I see
> this as
> >>> another way to bring more folks into the project.
> >>>
> >>
> >> Without going so far as to say we shouldn't merge any
> >> platform/architecture/vendor specific code ever and for no reason, I
> >> personally think we should avoid doing so as much as practical and
> >> encourage the "plugins" route instead. It's just much cleaner imo on
> >> principle and amounts to good software development hygiene.
> >>
> >> I don't want to not be able to commit some changes because it breaks the
> >> build because there is code for X number of super specific
> >> platform/architecture I don't have access to/don't know anything about
> >> and the maintainers are on vacation or hasn't been reachable in a while.
> >> And what if such maintainer do go away? Sure we can have some "process"
> >> to remove the code in such cases, but why add that burden on us? Plus
> >> we, the Apache Cassandra project, would still be seen as the ones that
> >> drop support for said platform/architecture even though we really have
> >> no choice if it's something we don't have access to anyway.
> >>
> >> And sure, I'm painting a bleak picture here, and we would probably have
> >> none of those problems in most cases. But if we do start encourage
> >> actual merge of such code, you can be sure we'll have some of those
> >> problems at some point.
> >>
> >> Encouraging plugins have imo pretty much all the benefits with none of
> >> the risks. In particular, I'm unconvinced that someone will be
> >> much more likely to meaningfully contribute to other part of the code
> >> if his "plugins" is 

Re: NGCC Proposal (Was Re: NGCC?)

[Jonathan Haddad]

Agreed with Jeff & Jason.

On Tue, Jun 13, 2017 at 11:45 AM Jeff Jirsa  wrote:

> Looks great to me - especially the venue.  Date wise, Tuesday (19th) lets
> people fly in on Monday instead of costing a weekend, so selfishly that
> seems better to me.
>
>
>
> On Mon, Jun 12, 2017 at 1:30 PM, Gary Dusbabek 
> wrote:
>
> > ## Cassandra list email
> >
> > Hi everybody,
> >
> > Here are current thoughts about structure and timing. Your feedback is
> > welcome.
> >
> > Date: One of 18, 19, or 22 September 2017. We are aware the Tokyo user
> > group is planning an event the first week in October. We’re doing our
> best
> > to give a buffer there.
> >
> > Venue: After evaluating a few options in San Antonio, the event space at
> > Geekdom seems like a good fit and the right balance of cost and services
> (
> > goo.gl/tViC72).
> >
> > Format: This will be a one day event, starting in the morning and running
> > through the evening. Here is a proposed agenda we can iterate on:
> >
> > * 8-9am - catered breakfast, conversation
> > * 9-12 - presentations, including a coffee/snack break
> > * 12-1:30pm - catered lunch
> > * 1:30-4:00pm - presentations, including a coffee/snack break
> > * 4-5pm - lightning talks
> > * 5-6:30pm - happy hour and a half
> > * 6:30-9:30pm - dinner at a local restaurant
> >
> > We hope to film the presentations and make them available on Youtube.
> >
> > A few have already reached out offering various resources or assistance.
> > Thank you! Eric or I will be reaching out to coordinate as soon as we are
> > ready to finalize plans.
> >
> > Please chime in if you have suggestions, especially those relating to
> > format.
> >
> > Cheers,
> >
> > Gary
> >
>

Re: Guidelines on testing

[Jonathan Haddad]

+1

On Tue, Apr 25, 2017 at 2:21 AM Stefan Podkowinski  wrote:

> I don't see any reasons not to make this part of our guidelines. The
> idea of having a list of what should be tested in each kind of test
> makes sense. I also like the examples how to improve tests dealing with
> global state.
>
> Some of the integration test cases, such as "dry
> start"/"restart"/"shutdown"/"upgrade", could use some further
> description and how-to examples. Are there any existing tests we can
> link for reference?
>
> We also already have a testing related page in our documentation:
> http://cassandra.apache.org/doc/latest/development/testing.html
> Not sure if it would make sense to merge or create an additional document.
>
>
> On 24.04.2017 18:13, Blake Eggleston wrote:
> > About a month ago, in the ‘Code quality, principles and rules’ thread,
> I’d proposed adding some testing standards to the project in lieu of
> revisiting the idea of removing singletons. The idea was that we could
> drive incremental improvement of the test coverage and testability
> situation that could be applied in day to day work. I’ve pushed a first
> draft to my repo here:
> >
> > https://github.com/bdeggleston/cassandra/blob/testing-doc/TESTING.md
> >
> > Please take a look and let me know what you think. With the blessing of
> the pmc, I’d like this, or something like it, to be adopted as the
> reference for contributors and reviewers when deciding if a contribution is
> properly tested.
> >
> > Blake
> >
>

Re: NGCC?

[Jonathan Haddad]

+1.

Are there guidelines for how to set something like this up or is it tribal
knowledge?

On Wed, May 31, 2017 at 3:05 PM Gary Dusbabek  wrote:

> +1. I'm happy to help with logistics.
>
> Mid-to-late September is good, but so is October.
>
> Gary.
>
> On Wed, May 31, 2017 at 2:19 PM, Eric Evans 
> wrote:
>
> > Hi,
> >
> > Is anyone working on an NGCC event for this year? Given all of the
> > recent changes, it seems like it could be really useful. If not, is
> > there interest? When would be good, September? October? What about
> > location? I think we could get the space to host such an even in
> > either San Antonio (centrally located), or San Francisco (San
> > Franciscoly located).
> >
> > Thoughts?
> >
> > --
> > Eric Evans
> > john.eric.ev...@gmail.com
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>

Re: Apache cassandra wiki access

[Jonathan Haddad]

The wiki isn't really used anymore. You can submit patches to improve the
docs in tree now. Fee free to tag me as a reviewer and I'll get to it this
week.
On Sat, Sep 16, 2017 at 10:26 AM Vusal Ahmadoglu 
wrote:

> Hello,
>
> Could you please give access me to the apache cassandra  wiki?
> e-mail: vusal.ahmado...@gmail.com
> Wiki name: Vusal Ahmadoglu
>
> Have a lovely weekend!
>
> Many thanks,
> Vusal
>

Re: Do it need to run 'nodetool upgradesstables' when updating from 2.0.9 to 2.1.19?

[Jonathan Haddad]

After an upgrade I recommend running upgrade sstables no matter what the
version change is. If it's not needed, nothing will happen.
On Fri, Oct 13, 2017 at 4:30 AM Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

> And extremely useful/important in the field not being a strict requirement
> immediately upgrading sstables, especially for not closely monitored
> environments, e.g. unattended deployments like development machines or even
> customer on-prem installations etc.
>
> Cassandra's data backwards compatibility is one of the best I have seen.
>
> Thanks a lot for that!
>
> T.
>
>
> -Original Message-
> From: J. D. Jordan [mailto:jeremiah.jor...@gmail.com]
> Sent: Freitag, 13. Oktober 2017 13:13
> To: dev@cassandra.apache.org
> Subject: Re: Do it need to run 'nodetool upgradesstables' when updating
> from 2.0.9 to 2.1.19?
>
> Old restrictions that have been lifted Jeff :) Ever since 2.0 we have
> supported streaming old sstables, see here for the 2.0 ticket:
> https://issues.apache.org/jira/browse/CASSANDRA-5772
>
> That was broken in early 3.0, but had since been fixed by the ticket
> Marcus linked https://issues.apache.org/jira/browse/CASSANDRA-10990
>
> So it should be fine in all latest 3.x versions.
>
> -Jeremiah
>
>
> > On Oct 13, 2017, at 3:34 AM, Steinmaurer, Thomas <
> thomas.steinmau...@dynatrace.com> wrote:
> >
> > Same here, regarding sincere question to know the corner cases from 2.1
> => 3.11.1, but Marcus already provided the JIRA ticket.
> >
> > Thanks!
> >
> >
> > -Original Message-
> > From: Per Otterström [mailto:per.otterst...@ericsson.com]
> > Sent: Freitag, 13. Oktober 2017 09:01
> > To: dev@cassandra.apache.org
> > Subject: RE: Do it need to run 'nodetool upgradesstables' when updating
> from 2.0.9 to 2.1.19?
> >
> > Hepp!
> >
> > Just to be clear, I'm not claiming this to be the case, it is a sincere
> question. My team is planning an upgrade from 2.2 to 3.0 which is why I'm
> asking. Some initial tests indicate that repairs etc work well before
> running upgradesstables (assuming all nodes are upgraded to 3.0). But if
> there are known corner cases I'd be interested to know.
> >
> > We need to support reading old sstables in order to support rolling
> upgrades. This seem to be a general assumption that people agree on. I'm
> not sure there is a clear agreement on the streaming part. Would it make
> sense to clarify this in documentation?
> >
> > /pelle
> >
> > -Original Message-
> > From: Jeff Jirsa [mailto:jji...@gmail.com]
> > Sent: den 13 oktober 2017 08:09
> > To: dev@cassandra.apache.org
> > Subject: Re: Do it need to run 'nodetool upgradesstables' when updating
> from 2.0.9 to 2.1.19?
> >
> > Two people say I’m wrong, I’ll believe it - must be imagining
> restrictions that don’t exist.
> >
> > --
> > Jeff Jirsa
> >
> >
> >> On Oct 12, 2017, at 10:55 PM, Per Otterström <
> per.otterst...@ericsson.com> wrote:
> >>
> >> Hi!
> >>
> >> This was news to me. I had the impression that we are maintaining
> backwards compatibility for streaming non-upgraded sstables. Is that not
> the case?
> >>
> >> However, I believe it is a good idea to run upgradesstables at some
> point _before_ upgrading Cassandra next time to make sure there are no old
> sstables around before you take the next step.
> >>
> >> /pelle
> >>
> >> -Original Message-
> >> From: Jeff Jirsa [mailto:jji...@gmail.com]
> >> Sent: den 12 oktober 2017 23:11
> >> To: Cassandra DEV 
> >> Subject: Re: Do it need to run 'nodetool upgradesstables' when updating
> from 2.0.9 to 2.1.19?
> >>
> >> You won't notice any issues (all versions of cassandra are backwards
> compatible at least 1 version), and sstables will be slowly migrated to the
> new format as you compact, but you should still run it (upgradesstables)
> explicitly because when it comes time to run
> repair/boostrap/decommission/etc, you'll need all sstables on the
> same/current format.
> >>
> >>
> >>
> >> On Thu, Oct 12, 2017 at 1:59 PM, Li, Guangxing
> >> 
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> I recently upgraded my test cluster from C* 2.0.9 to 2.1.19 and I
> >>> did not run 'nodetool upgradesstables' at all. Per my confirmation
> >>> afterwards, everything is fine. But according to documentation at
> >>> https://support.datastax.com/hc/en-us/articles/208040036-
> >>> Nodetool-upgradesstables-FAQ.
> >>> I need to do that.
> >>> So can someone tell me if I must do 'nodetool upgradesstables' after
> >>> upgrading each node from 2.0.9 to 2.1.19?
> >>>
> >>> Thanks.
> >>>
> >>> George.
> >>>
> >>  B‹
> >> CB• È [œà XœØÜšX™K  K[XZ[
> > ˆ  ]‹][œà XœØÜšX™P Ø\ÜØ[™ ˜K˜\ XÚ K›Ü™ÃB‘›
> > ܈ Y  ] [Û˜[  ÛÛ[X[™ Ë  K[XZ[
> > ˆ  ]‹Z [   Ø\ÜØ[™ ˜K˜\ XÚ K›Ü™ÃBƒB
> >
> > -

Re: [VOTE] Release Apache Cassandra 2.1.19

[Jonathan Haddad]

+1
On Thu, Sep 28, 2017 at 1:46 PM Nate McCall  wrote:

> +1
>
> On Sep 29, 2017 7:12 AM, "Michael Shuler"  wrote:
>
> I propose the following artifacts for release as 2.1.19.
>
> sha1: 428eaa3e37cab7227c81fdf124d29dfc1db4257c
> Git:
> http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> shortlog;h=refs/tags/2.1.19-tentative
> Artifacts:
> https://repository.apache.org/content/repositories/
> orgapachecassandra-1148/org/apache/cassandra/apache-cassandra/2.1.19/
> Staging repository:
> https://repository.apache.org/content/repositories/orgapachecassandra-1148/
>
> The Debian and RPM packages are available here:
> http://people.apache.org/~mshuler
>
> The vote will be open for 72 hours (longer if needed).
>
> [1]: (CHANGES.txt) https://goo.gl/1sZLdP (also RPM file ownership fix)
> [2]: (NEWS.txt) https://goo.gl/YKEuRc
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>

Re: [DISCUSS] Cassandra and future Java

[Jonathan Haddad]

Personally I don’t mind dropping support for previsous java versions.

On Fri, May 25, 2018 at 6:33 AM J. D. Jordan 
wrote:

> +1 for “Option 3: both 8 + 11” it shouldn’t be too hard to maintain code
> wise, and leaves people’s options open.
>
> -Jeremiah
>
> > On May 25, 2018, at 6:31 AM, Robert Stupp  wrote:
> >
> > I'd like to bring up the C*/Java discussion again. It's been a while
> since we've discussed this.
> >
> > To me it sounds like there's still the question about which version(s)
> of Java we want to support beginning with C* 4.0.
> >
> > I assume, that it's legit (and probably very necessary) to assume that
> OpenJDK is now (i.e. after Java 6) considered as "production ready" for C*.
> The public (and legal and free) availability of Oracle's Java 8 will end in
> January 2019 (unless you're using it privately on your desktop). Java 9 and
> 10 are not a thing, as both will be EOL when the C* 4.0 branch is about to
> be cut. The most recent available Java version will be 11, which is meant
> to be publicly available from Oracle until March 2019 and should get LTS
> support for OpenJDK 11 from major Linux distros (RHEL and derivates,
> Ubuntu, Azul Zulu).
> >
> > (Side note: adoptopenjdk is different here, because it does not include
> the patch version in the version banner (java.version=1.8.0-adoptopenjdk),
> so difficult to check the minimum patch version on startup of C*.)
> >
> > (Attn, rant: I'm not particularly happy with the new release and support
> model for Java, because developing something now, that's about to release
> end of the year on a Java version that has not even reached
> feature-complete status, is, gently speaking, difficult. But sticking to an
> "antique" Java version (8) has its own risks as well.)
> >
> > I'm silently ignoring any Java release, that's not aimed to get any
> LTS(-ish?) support from anybody - so only Java 8 + 11 remain.
> >
> > There are generally three (IMO legit) options here: only support Java 8,
> only support Java 11, support both Java 8 and Java 11. All three options
> have a bunch of pros and cons.
> >
> > Option 1, only Java 8: Probably the safest option. Considering the
> potential lifetimes of Java 8 and C* 4.0, even the most enthusiastic
> maintainers may stop backporting security or bug fixes to OpenJDK 8. It
> might not be an issue in practice, but if there's for example a severe
> issue in the SSL/TLS area and nobody fixes it in 8, well, good luck.
> >
> > Option 2, only Java 11: The option with the most risks IMO. Java 11 is
> not even feature complete, and there a bunch of big projects that still may
> make it into 11 (think: Valhalla). There's no guarantee whether the C* code
> or any included library will actually work with Java 11 (think: if it works
> now, it may not work with the final Java version). However, it leaves the
> door wide open for all the neat and geeky things in Java 11.
> >
> > Option 3: both 8 + 11: The idea here is to default to Java 8, but the
> code also runs on 11. It leaves the option to benefit from optimizations
> that are only available on 11 while maintaining the known stability of 8.
> Initially, only the combination of C* 4.0 + Java 8 would be labeled as
> "stable" and the combination of C* 4.0 + Java 11 as "experimental". But it
> gives us time to "evaluate" 4.0 on 11. When we have enough experience with
> 11, C* on 11 can be labeled as "stable" as well. The downside of this
> "hybrid" is, that it's a bit more difficult to introduce features, that
> depend on 11.
> >
> > I think, 3) gives the best of both worlds: stability of 8 and an upgrade
> path to 11 in the future, that people can actually test with C* 4.0. Happy
> to make the patch for #9608 ready for option 3. But it would be great to
> get a consensus here for either option before we review #9608 and commit it.
> >
> > Another proposal, for both options 1+3: Raise the minimum supported
> version of 8 for C* 4.0 to something more recent than 8u40, which is quite
> from the stone-age. It could be 8u171 or whatever will be recent in autumn.
> >
> > Robert
> >
> > --
> > Robert Stupp
> > @snazy
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
> --
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Planning to port cqlsh to Python 3 (CASSANDRA-10190)

[Jonathan Haddad]

Supporting both as a next step is logical, removing support for 2 in the
next year or two seems reasonable enough. Gotta rip the band aid off at
some point.

On Fri, Jun 1, 2018 at 2:34 AM Michael Burman  wrote:

> Hi,
>
> Deprecating in this context does not mean removing it or it being
> replaced by 3 (RHEL 7.x will remain with Python 2.x as default). It
> refers to future versions (>7), but there are none at this point. It
> appears Ubuntu has deviated from Debian in this sense, but Debian has
> not changed yet (likely Debian 10 will, but that's not out yet and has
> no announced release date).
>
> Thus, 2.x still remains the most used version for servers. And servers
> deployed at this point of time will use these versions for years.
>
>- Micke
>
>
> On 06/01/2018 10:52 AM, Murukesh Mohanan wrote:
> > On 2018/06/01 07:40:04, Michael Burman  wrote:
> >> IIRC, there's no major distribution yet that defaults to Python 3 (I
> >> think Ubuntu & Debian are still defaulting to Python 2 also). This will
> >> happen eventually (maybe), but not yet. Discarding Python 2 support
> >> would mean more base-OS work for most people wanting to run Cassandra
> >> and that's not a positive thing.
> >>
> > Ubuntu since 16.04 defaults to Python 3:
> >
> >> Python2 is not installed anymore by default on the server, cloud and
> the touch images, long live Python3! Python3 itself has been upgraded to
> the 3.5 series. -
> https://wiki.ubuntu.com/XenialXerus/ReleaseNotes#Python_3
> > RHEL 7.5 deprecates Python 2 (
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.5_release_notes/chap-red_hat_enterprise_linux-7.5_release_notes-deprecated_functionality
> ).
> >
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
> --
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Planning to port cqlsh to Python 3 (CASSANDRA-10190)

[Jonathan Haddad]

Both can work.  I did a lot of the work on the port of the Python
driver's object mapper (formerly cqlengine) to Python 3.  It's
reasonably straightforward if you use the six library.

Both pandas and numpy are dropping support for Python 2 at the end of
this year.  I'm fine with riding on their coattails.
On Fri, Jun 1, 2018 at 9:21 AM Russell Bateman  wrote:
>
> Support for, but not the very script, right? Because, as gently pointed
> out by several realists here, Python 2 is far from dead and arguably
> still the majority usage. That's only just now beginning to change. I
> think it will be more than 2 years before people begin asking what
> Python 2 was.
>
>
> On 06/01/2018 10:10 AM, Jonathan Haddad wrote:
> > Supporting both as a next step is logical, removing support for 2 in the
> > next year or two seems reasonable enough. Gotta rip the band aid off at
> > some point.
> >
> > On Fri, Jun 1, 2018 at 2:34 AM Michael Burman  wrote:
> >
> >> Hi,
> >>
> >> Deprecating in this context does not mean removing it or it being
> >> replaced by 3 (RHEL 7.x will remain with Python 2.x as default). It
> >> refers to future versions (>7), but there are none at this point. It
> >> appears Ubuntu has deviated from Debian in this sense, but Debian has
> >> not changed yet (likely Debian 10 will, but that's not out yet and has
> >> no announced release date).
> >>
> >> Thus, 2.x still remains the most used version for servers. And servers
> >> deployed at this point of time will use these versions for years.
> >>
> >> - Micke
> >>
> >>
> >> On 06/01/2018 10:52 AM, Murukesh Mohanan wrote:
> >>> On 2018/06/01 07:40:04, Michael Burman  wrote:
> >>>> IIRC, there's no major distribution yet that defaults to Python 3 (I
> >>>> think Ubuntu & Debian are still defaulting to Python 2 also). This will
> >>>> happen eventually (maybe), but not yet. Discarding Python 2 support
> >>>> would mean more base-OS work for most people wanting to run Cassandra
> >>>> and that's not a positive thing.
> >>>>
> >>> Ubuntu since 16.04 defaults to Python 3:
> >>>
> >>>> Python2 is not installed anymore by default on the server, cloud and
> >> the touch images, long live Python3! Python3 itself has been upgraded to
> >> the 3.5 series. -
> >> https://wiki.ubuntu.com/XenialXerus/ReleaseNotes#Python_3
> >>> RHEL 7.5 deprecates Python 2 (
> >> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.5_release_notes/chap-red_hat_enterprise_linux-7.5_release_notes-deprecated_functionality
> >> ).
> >>>
> >>>
> >>> -
> >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>>
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>
> >> --
> > Jon Haddad
> > http://www.rustyrazorblade.com
> > twitter: rustyrazorblade
> >
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Planning to port cqlsh to Python 3 (CASSANDRA-10190)

[Jonathan Haddad]

And that's why I said supporting both with six is the right path
forward, later dropping support for 2.  I'm not advocating we drop 2
support now, and I'm not asking for any sort of commitment.  I didn't
think adding support for 3 would be so controversial.
On Fri, Jun 1, 2018 at 9:40 AM Jeremiah D Jordan
 wrote:
>
> The community of people doing python development and the community of people 
> running Cassandra servers are not the same.  I am not fine riding the coat 
> tails of libraries used in python development.  As others have stated we need 
> to be following the lead of the OS vendors that people will be deploying 
> Cassandra on top of.  And those will not be dropping Python 2 at the end of 
> the year.
>
> -Jeremiah
>
> > On Jun 1, 2018, at 12:37 PM, Jonathan Haddad  wrote:
> >
> > Both can work.  I did a lot of the work on the port of the Python
> > driver's object mapper (formerly cqlengine) to Python 3.  It's
> > reasonably straightforward if you use the six library.
> >
> > Both pandas and numpy are dropping support for Python 2 at the end of
> > this year.  I'm fine with riding on their coattails.
> > On Fri, Jun 1, 2018 at 9:21 AM Russell Bateman  
> > wrote:
> >>
> >> Support for, but not the very script, right? Because, as gently pointed
> >> out by several realists here, Python 2 is far from dead and arguably
> >> still the majority usage. That's only just now beginning to change. I
> >> think it will be more than 2 years before people begin asking what
> >> Python 2 was.
> >>
> >>
> >> On 06/01/2018 10:10 AM, Jonathan Haddad wrote:
> >>> Supporting both as a next step is logical, removing support for 2 in the
> >>> next year or two seems reasonable enough. Gotta rip the band aid off at
> >>> some point.
> >>>
> >>> On Fri, Jun 1, 2018 at 2:34 AM Michael Burman  wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> Deprecating in this context does not mean removing it or it being
> >>>> replaced by 3 (RHEL 7.x will remain with Python 2.x as default). It
> >>>> refers to future versions (>7), but there are none at this point. It
> >>>> appears Ubuntu has deviated from Debian in this sense, but Debian has
> >>>> not changed yet (likely Debian 10 will, but that's not out yet and has
> >>>> no announced release date).
> >>>>
> >>>> Thus, 2.x still remains the most used version for servers. And servers
> >>>> deployed at this point of time will use these versions for years.
> >>>>
> >>>>- Micke
> >>>>
> >>>>
> >>>> On 06/01/2018 10:52 AM, Murukesh Mohanan wrote:
> >>>>> On 2018/06/01 07:40:04, Michael Burman  wrote:
> >>>>>> IIRC, there's no major distribution yet that defaults to Python 3 (I
> >>>>>> think Ubuntu & Debian are still defaulting to Python 2 also). This will
> >>>>>> happen eventually (maybe), but not yet. Discarding Python 2 support
> >>>>>> would mean more base-OS work for most people wanting to run Cassandra
> >>>>>> and that's not a positive thing.
> >>>>>>
> >>>>> Ubuntu since 16.04 defaults to Python 3:
> >>>>>
> >>>>>> Python2 is not installed anymore by default on the server, cloud and
> >>>> the touch images, long live Python3! Python3 itself has been upgraded to
> >>>> the 3.5 series. -
> >>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.ubuntu.com_XenialXerus_ReleaseNotes-23Python-5F3=DwIBaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=J5Su6wvm91QrOBcici7HyIiFiyzjrg8UnamYu8qtSRA=9OWAbO26grwiI2ly_-gAGBqJP9Mv6KPAKJyQu_OEDPc=
> >>>>> RHEL 7.5 deprecates Python 2 (
> >>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__access.redhat.com_documentation_en-2Dus_red-5Fhat-5Fenterprise-5Flinux_7_html_7.5-5Frelease-5Fnotes_chap-2Dred-5Fhat-5Fenterprise-5Flinux-2D7.5-5Frelease-5Fnotes-2Ddeprecated-5Ffunctionality=DwIBaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=J5Su6wvm91QrOBcici7HyIiFiyzjrg8UnamYu8qtSRA=CDFufWbcvq6VpoLJQVbCQP9rpvIv3ssNtKMQce-1vwU=
> >>>> ).
> >>>>>
> >>>>>
> >>>>> -
> >>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>>>> For additional commands, e-mail: dev-h...

Re: Testing 4.0 Post-Freeze

[Jonathan Haddad]

I agree with Josh. I don’t see how changing the convention around trunk
will improve the process, seems like it’ll only introduce a handful of
rollback commits when people forget.

Other than that, it all makes sense to me.

I’ve been working on a workload centric stress tool on and off for a little
bit in an effort to create something that will help with wider adoption in
stress testing. It differs from the stress we ship by including fully
functional stress workloads as well as a validation process. The idea being
to be flexible enough to test both performance and correctness in LWT and
MVs as well as other arbitrary workloads.

https://github.com/thelastpickle/tlp-stress

Jon


On Tue, Jul 3, 2018 at 12:28 PM Josh McKenzie  wrote:

> Why not just branch a 4.0-rel and bugfix there and merge up while still
> accepting new features or improvements on trunk?
>
> I don't think the potential extra engagement in testing will balance out
> the atrophy and discouraging contributions / community engagement we'd get
> by deferring all improvements and new features in an open-ended way.
>
> On Tue, Jul 3, 2018 at 1:33 PM sankalp kohli 
> wrote:
>
> > Hi cassandra-dev@,
> >
> > With the goal of making Cassandra's 4.0 the most stable major release to
> > date, we would like all committers of the project to consider joining us
> in
> > dedicating their time and attention to testing, running, and fixing
> issues
> > in 4.0 between the September freeze and the 4.0 beta release. This would
> > result in a freeze of new feature development on trunk or branches during
> > this period, instead focusing on writing, improving, and running tests or
> > fixing and reviewing bugs or performance regressions found in 4.0 or
> > earlier.
> >
> > How would this work?
> >
> > We propose that between the September freeze date and beta, a new branch
> > would not be created and trunk would only have bug fixes and performance
> > improvements committed to it. At the same time we do not want to
> discourage
> > community contributions. Not all contributors can be expected to be aware
> > of such a decision or may be new to the project. In cases where new
> > features are contributed during this time, the contributor can be
> informed
> > of the current status of the release process, be encouraged to contribute
> > to testing or bug fixing, and have their feature reviewed after the beta
> is
> > reached.
> >
> >
> > What happens when beta is reached?
> >
> > Ideally, contributors who have made significant contributions to the
> > release will stick around to continue testing between beta and final
> > release. Any additional folks who continue this focus would also be
> greatly
> > appreciated.
> >
> > What about before the freeze?
> >
> > Testing new features is of course important. This isn't meant to
> discourage
> > development – only to enable us to focus on testing and hardening 4.0 to
> > deliver Cassandra's most stable major release. We would like to see
> > adoption of 4.0 happen much more quickly than its predecessor.
> >
> > Thanks for considering this proposal,
> > Sankalp Kohli
>
-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: [VOTE] (Take 2) Release Apache Cassandra 3.11.2

[Jonathan Haddad]

+1
On Mon, Feb 12, 2018 at 9:51 PM mck  wrote:

>
>
> > I propose the following artifacts for release as 3.11.2.
>
>
> Thanks Michael for the recut, very much appreciated.
>
> +1
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Expensive metrics?

[Jonathan Haddad]

Hey Micke, very cool you're looking to improve C*'s performance, we would
absolutely benefit from it.

Have you done any other benchmarks beside the micro one to determine the
total effect of these metrics on the system overall?  Microbenchmarks are a
great way to tune small sections of code but they aren't a great starting
point.  It would be good if we could put some context around the idea by
benchmarking a tuned, single node (so there's less network overhead)
running on fast disks with compaction disabled so we can see what kind of
impact these metrics are adding.  Ideally we'd look at GC promotion and CPU
time using something like YourKit to identify the overall effect of the
metrics, so we can set our expectations and goals in a reasonable manner.
Happy to coordinate with you on this!

On Thu, Feb 22, 2018 at 8:08 AM Jeremiah D Jordan 
wrote:

> re: nanoTime vs currentTimeMillis there is a good blog post here about the
> timing of both and how your choice of Linux clock source can drastically
> effect the speed of the calls, and also showing that in general on linux
> there is no perf improvement for one over the other.
> http://pzemtsov.github.io/2017/07/23/the-slow-currenttimemillis.html
>
> > On Feb 22, 2018, at 11:01 AM, Blake Eggleston 
> wrote:
> >
> > Hi Micke,
> >
> > This is really cool, thanks for taking the time to investigate this. I
> believe the metrics around memtable insert time come in handy in
> identifying high partition contention in the memtable. I know I've been
> involved in a situation over the past year where we got actionable info
> from this metric. Reducing resolution to milliseconds is probably a no go
> since most things in this path should complete in less than a millisecond.
> >
> > Revisiting the use of the codahale metrics in the hot path like this
> definitely seems like a good idea though. I don't think it's been something
> we've talked about a lot, and it definitely looks like we could benefit
> from using something more specialized here. I think it's worth doing,
> especially since there won't be any major changes to how we do threading in
> 4.0. It's probably also worth opening a JIRA and investigating the calls to
> nano time. We at least need microsecond resolution here, and there could be
> something we haven't thought of? It's worth a look at least.
> >
> > Thanks,
> >
> > Blake
> >
> > On 2/22/18, 6:10 AM, "Michael Burman"  wrote:
> >
> >Hi,
> >
> >I wanted to get some input from the mailing list before making a JIRA
> >and potential fixes. I'll touch the performance more on latter part,
> but
> >there's one important question regarding the write latency metric
> >recording place. Currently we measure the writeLatency (and metric
> write
> >sampler..) in ColumnFamilyStore.apply() and this is also the metric we
> >then replicate to Keyspace metrics etc.
> >
> >This is an odd place for writeLatency. Not to mention it is in a
> >hot-path of Memtable-modifications, but it also does not measure the
> >real write latency, since it completely ignores the CommitLog latency
> in
> >that same process. Is the intention really to measure
> >Memtable-modification latency only or the actual write latencies?
> >
> >Then the real issue.. this single metric is a cause of huge overhead
> in
> >Memtable processing. There are several metrics / events in the CFS
> apply
> >method, including metric sampler, storageHook reportWrite,
> >colUpdateTimeDeltaHistogram and metric.writeLatency. These are not
> free
> >at all when it comes to the processing. I made a small JMH benchmark
> >here:
> https://gist.github.com/burmanm/b5b284bc9f1d410b1d635f6d3dac3ade
> >that I'll be referring to.
> >
> >The most offending of all these metrics is the writeLatency metric.
> What
> >it does is update the latency in codahale's timer, doing a histogram
> >update and then going through all the parent metrics also which update
> >the keyspace writeLatency and globalWriteLatency. When measuring the
> >performance of Memtable.put with parameter of 1 partition (to reduce
> the
> >ConcurrentSkipListMap search speed impact - that's separate issue and
> >takes a little bit longer to solve although I've started to prototype
> >something..) on my machine I see 1.3M/s performance with the metric
> and
> >when it is disabled the performance climbs to 4M/s. So the overhead
> for
> >this single metric is ~2/3 of total performance. That's insane. My
> perf
> >stats indicate that the CPU is starved as it can't get enough data in.
> >
> >Removing the replication from TableMetrics to the Keyspace & global
> >latencies in the write time (and doing this when metrics are requested
> >instead) improves the performance to 2.1M/s on my machine. It's an
> >improvement, but it's still huge amount. Even when we pressure 

Re: Why isn't there a separate JVM per table?

[Jonathan Haddad]

There's an incredible amount of work that would need to be done in order to
make any of this happen.  Basically a full rewrite of the entire codebase.
Years of effort.

The codebase would have to move to a shared-nothing actor & message based
communication mechanism before any of this is possible.  Fun in theory, but
considering removing singletons has been a multi-year, many failure effort,
I suspect we might need 10 years to refactor Cassandra to use multiple
JVMs.  By then maybe we'll have a pauseless / low pause collector and it
won't matter.

On Thu, Feb 22, 2018 at 3:59 PM kurt greaves  wrote:

> >
> >  ... compaction on its own jvm was also something I was thinking about,
> but
> > then I realized even more JVM sharding could be done at the table level.
>
>
> Compaction in it's own JVM makes sense. At the table level I'm not so sure
> about. Gotta be some serious overheads from running that many JVM's.
> Keyspace might be reasonable purely to isolate bad tables, but for the most
> part I'd think isolating every table isn't that beneficial and pretty
> complicated. In most cases people just fix their modelling so that they
> don't generate large amounts of GC, and hopefully test enough so they know
> how it will behave in production.
>
> If we did at the table level we would inevitable have to make each
> individual table incredibly tune-able which would be a bit tedious IMO.
> There's no way for us to smartly decide how much heap/memtable space/etc
> each table should use (not without some decent AI, anyway).
> ​
>

Re: NGCC 2018?

[Jonathan Haddad]

My interpretation of Nate's statement was that since there would be a bunch
of us at Lynn's event, we might as well do NGCC at the same time.

On Thu, Jul 26, 2018 at 9:03 PM Ben Bromhead  wrote:

> It sounds like there may be an appetite for something, but the NGCC in its
> current format is likely to not be that useful?
>
> Is a bay area event focused on C* developers something that is interesting
> for the broader dev community? In whatever format that may be?
>
> On Tue, Jul 24, 2018 at 5:02 PM Nate McCall  wrote:
>
> > This was discussed amongst the PMC recently. We did not come to a
> > conclusion and there were not terribly strong feelings either way.
> >
> > I don't feel like we need to hustle to get "NGCC" in place,
> > particularly given our decided focus on 4.0. However, that should not
> > stop us from doing an additional 'c* developer' event in sept. to
> > coincide with distributed data summit.
> >
> > On Wed, Jul 25, 2018 at 5:03 AM, Patrick McFadin 
> > wrote:
> > > Ben,
> > >
> > > Lynn Bender had offered a space the day before Distributed Data Summit
> in
> > > September (http://distributeddatasummit.com/) since we are both
> platinum
> > > sponsors. I thought he and Nate had talked about that being a good
> place
> > > for NGCC since many of us will be in town already.
> > >
> > > Nate, now that I've spoken for you, you can clarify, :D
> > >
> > > Patrick
> > >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> > --
> Ben Bromhead
> CTO | Instaclustr 
> +1 650 284 9692
> Reliability at Scale
> Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: [VOTE] Release Apache Cassandra 3.11.3 (Take 2)

[Jonathan Haddad]

+1

On Wed, Jul 25, 2018 at 11:35 AM Jeff Jirsa  wrote:

> +1
>
> On Wed, Jul 25, 2018 at 12:16 AM, Michael Shuler 
> wrote:
>
> > I propose the following artifacts for release as 3.11.3.
> >
> > sha1: 31d5d870f9f5b56391db46ba6cdf9e0882d8a5c0
> > Git:
> > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> > shortlog;h=refs/tags/3.11.3-tentative
> > Artifacts:
> > https://repository.apache.org/content/repositories/
> > orgapachecassandra-1164/org/apache/cassandra/apache-cassandra/3.11.3/
> > Staging repository:
> > https://repository.apache.org/content/repositories/
> > orgapachecassandra-1164/
> >
> > The Debian and RPM packages are available here:
> > http://people.apache.org/~mshuler
> >
> > The vote will be open for 72 hours (longer if needed).
> >
> > [1]: CHANGES.txt:
> > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> > blob_plain;f=CHANGES.txt;hb=refs/tags/3.11.3-tentative
> > [2]: NEWS.txt:
> > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> > blob_plain;f=CHANGES.txt;hb=refs/tags/3.11.3-tentative
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: [VOTE] Release Apache Cassandra 3.0.17 (Take 2)

[Jonathan Haddad]

+1

On Wed, Jul 25, 2018 at 11:35 AM Jeff Jirsa  wrote:

> +1
>
> On Wed, Jul 25, 2018 at 12:17 AM, Michael Shuler 
> wrote:
>
> > I propose the following artifacts for release as 3.0.17.
> >
> > sha1: d52c7b8c595cc0d06fc3607bf16e3f595f016bb6
> > Git:
> > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> > shortlog;h=refs/tags/3.0.17-tentative
> > Artifacts:
> > https://repository.apache.org/content/repositories/
> > orgapachecassandra-1165/org/apache/cassandra/apache-cassandra/3.0.17/
> > Staging repository:
> > https://repository.apache.org/content/repositories/
> > orgapachecassandra-1165/
> >
> > The Debian and RPM packages are available here:
> > http://people.apache.org/~mshuler
> >
> > The vote will be open for 72 hours (longer if needed).
> >
> > [1]: CHANGES.txt:
> > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> > blob_plain;f=CHANGES.txt;hb=refs/tags/3.0.17-tentative
> > [2]: NEWS.txt:
> > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> > blob_plain;f=CHANGES.txt;hb=refs/tags/3.0.17-tentative
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: [VOTE] Release Apache Cassandra 2.2.13

[Jonathan Haddad]

+1

On Wed, Jul 25, 2018 at 11:35 AM Jeff Jirsa  wrote:

> +1
>
> On Wed, Jul 25, 2018 at 12:17 AM, Michael Shuler 
> wrote:
>
> > I propose the following artifacts for release as 2.2.13.
> >
> > sha1: 3482370df5672c9337a16a8a52baba53b70a4fe8
> > Git:
> > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> > shortlog;h=refs/tags/2.2.13-tentative
> > Artifacts:
> > https://repository.apache.org/content/repositories/
> > orgapachecassandra-1167/org/apache/cassandra/apache-cassandra/2.2.13/
> > Staging repository:
> > https://repository.apache.org/content/repositories/
> > orgapachecassandra-1167/
> >
> > The Debian and RPM packages are available here:
> > http://people.apache.org/~mshuler
> >
> > The vote will be open for 72 hours (longer if needed).
> >
> > [1]: CHANGES.txt:
> > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> > blob_plain;f=CHANGES.txt;hb=refs/tags/2.2.13-tentative
> > [2]: NEWS.txt:
> > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=
> > blob_plain;f=CHANGES.txt;hb=refs/tags/2.2.13-tentative
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

G1GC is now default

[Jonathan Haddad]

I fired up trunk to check something, and noticed this:

INFO  [Service Thread] 2018-08-08 15:01:36,723 GCInspector.java:290 - G1
Young Generation GC in 339ms.  G1 Eden Space: 4634705920 -> 0; G1 Old Gen:
1190138352 -> 1435504616; G1 Survivor Space: 406847488 -> 301989888;

which I thought was a bit weird, since I was using trunk without making any
changes and didn't remember seeing a JIRA where we agreed to make that
change.  I looked back and saw it made it in as a side effect of
CASSANDRA-9608, but wasn't explicitly discussed in the ticket, and there's
no note of it in CHANGES.

I'm personally OK with this change as G1 is a safer bet for anyone who uses
the defaults, but we should be explicit about the change.  Can anyone think
of a reason why we'd need to revert this back to ParNew / CMS?

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: upgrade guava on trunk before 9/1?

[Jonathan Haddad]

Pushing it back means it’s a bigger risk later on. I’m +.5 on upgrading now

On Wed, Aug 15, 2018 at 11:46 PM dinesh.jo...@yahoo.com.INVALID
 wrote:

> Jason,
> Given that we're so close to the 9/1 date, I would err on the side of
> caution especially given the low value prop. If someone does run into Guava
> compatibility issues (and someone somewhere will), we can revisit this
> question then.
> Dinesh
>
> On Wednesday, August 15, 2018, 11:42:31 PM PDT, Dinesh A. Joshi <
> dinesh.jo...@gatech.edu> wrote:
>
>  Jason,
> Given that we're so close to the 9/1 date, I would err on the side of
> caution especially given the low value prop. If someone does run into Guava
> compatibility issues (and someone somewhere will), we can revisit this
> question then.
> Dinesh
>
> On Wednesday, August 15, 2018, 8:22:28 AM PDT, Salih Gedik <
> m...@salih.xyz> wrote:
>
>  Hi,
>
> Change logs are on Github releases page. It seems like only hash flooding
> protection which is added to ImmutableMap is relevant to Cassandra code. I
> haven’t checked whether we use deprecated APIs. But there isn’t much on
> table from what I see.
>
> Salih
> On 15 Aug 2018 17:55 +0300, Ariel Weisberg , wrote:
> > Hi,
> >
> > They don't even do release notes after 23. Also no API diffs. I mean I'm
> fine with it, but it's mostly just changing to another arbitrary version
> that won't match what is in apps.
> >
> > Ariel
> >
> > On Wed, Aug 15, 2018, at 10:48 AM, Jason Brown wrote:
> > > Hey Ariel,
> > >
> > > Tbqh, not that much. I was mostly thinking from the "I have conflicts
> on
> > > guava versions in my app because I pull in cassandra and XYZ
> libraries, and
> > > the transitive dependencies on guava use different versions" POV.
> Further,
> > > we'll be on this version of guava for 4.0 for at least two years from
> now.
> > >
> > > As I asked, "does anybody feeling strongly?". Personally, I'm sorta +0
> to
> > > +0.5, but I was just throwing this out there in case someone does
> really
> > > think it best we upgrade (and wants to make a contribution).
> > >
> > > -Jason
> > >
> > >
> > >
> > >
> > > On Wed, Aug 15, 2018 at 7:25 AM, Ariel Weisberg 
> wrote:
> > >
> > > > Hi,
> > > >
> > > > What do we get from Guava in exchange for upgrading?
> > > >
> > > > Ariel
> > > >
> > > > On Wed, Aug 15, 2018, at 10:19 AM, Jason Brown wrote:
> > > > > Hey all,
> > > > >
> > > > > Does anyone feel strongly about upgrading guava on trunk before
> the 9/1
> > > > > feature freeze for 4.0? We are currently at 23.3 (thanks to
> > > > > CASSANDRA-13997), and the current is 26.0.
> > > > >
> > > > > I took a quick look, and there's about 17 compilation errors. They
> fall
> > > > > into two categories, both of which appear not too difficult to
> resolve (I
> > > > > didn't look too closely, tbh).
> > > > >
> > > > > If anyone wants to tackle this LHF I can rustle up some review
> time.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > -Jason
> > > >
> > > > -
> > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > > >
> > > >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: GitHub PR ticket spam

[Jonathan Haddad]

+1 to worklog

On Mon, Aug 6, 2018 at 9:44 AM Ariel Weisberg  wrote:

> Hi,
>
> Great idea. +1 to moving it to the work log.
>
> Thanks,
> Ariel
>
> On Mon, Aug 6, 2018, at 12:40 PM, Aleksey Yeshchenko wrote:
> > Nice indeed. I assume it also doesn’t spam commits@ when done this way,
> > in which case double +1 from me.
> >
> > —
> > AY
> >
> > On 6 August 2018 at 17:18:36, Jeremiah D Jordan
> > (jeremiah.jor...@gmail.com) wrote:
> >
> > Oh nice. I like the idea of keeping it but moving it to the worklog tab.
> > +1 on that from me.
> >
> > > On Aug 6, 2018, at 5:34 AM, Stefan Podkowinski 
> wrote:
> > >
> > > +1 for worklog option
> > >
> > > Here's an example ticket from Arrow, where they seem to be using the
> > > same approach:
> > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_ARROW-2D2583=DwICaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=wYZwHSze6YITTXgzOrEvfr_onojahtjeJRzGAt8ByzM=KWt0xsOv9ESaieg432edGvPhktGkWHxVuLAdNyORiYY=
> <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_ARROW-2D2583=DwICaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=wYZwHSze6YITTXgzOrEvfr_onojahtjeJRzGAt8ByzM=KWt0xsOv9ESaieg432edGvPhktGkWHxVuLAdNyORiYY=>
>
> > >
> > >
> > > On 05.08.2018 09:56, Mick Semb Wever wrote:
> > >>> I find this a bit annoying while subscribed to commits@,
> > >>> especially since we created pr@ for these kind of messages. Also I
> don't
> > >>> really see any value in mirroring all github comments to the
> ticket.
> > >>
> > >>
> > >> I agree with you Stefan. It makes the jira tickets quite painful to
> read. And I tend to make comments on the commits rather than the PRs so to
> avoid spamming back to the jira ticket.
> > >>
> > >> But the linking to the PR is invaluable. And I can see Ariel's point
> about a chronological historical archive.
> > >>
> > >>
> > >>> Ponies would be for this to be mirrored to a tab
> > >>> separate from comments in JIRA.
> > >>
> > >>
> > >> Ariel, that would be the the "worklog" option.
> > >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__reference.apache.org_pmc_github=DwICaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=wYZwHSze6YITTXgzOrEvfr_onojahtjeJRzGAt8ByzM=1lWQawAO9fITzakpnmdzERuCbZs6IGQsUH_EEIMCMqs=
> <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__reference.apache.org_pmc_github=DwICaQ=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=wYZwHSze6YITTXgzOrEvfr_onojahtjeJRzGAt8ByzM=1lWQawAO9fITzakpnmdzERuCbZs6IGQsUH_EEIMCMqs=>
>
> > >>
> > >> If this works for you, and others, I can open a INFRA to switch to
> worklog.
> > >> wdyt?
> > >>
> > >>
> > >> Mick.
> > >>
> > >>
> -
> > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org  dev-unsubscr...@cassandra.apache.org>
> > >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 
> > >>
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org  dev-unsubscr...@cassandra.apache.org>
> > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> 
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Proposing an Apache Cassandra Management process

[Jonathan Haddad]

Speaking from experience (Reaper), I can say that developing a sidecar
admin / repair tool out of tree that's compatible with multiple versions
really isn't that big of a problem.  We've been doing it for a while now.

https://github.com/thelastpickle/cassandra-reaper/blob/master/.travis.yml

On Fri, Aug 17, 2018 at 9:39 AM Joseph Lynch  wrote:

> While I would love to use a different build system (e.g. gradle) for the
> sidecar, I agree with Dinesh that a separate repo would make sidecar
> development much harder to verify, especially on the testing and
> compatibility front. As Jeremiah mentioned we can always choose later to
> release the sidecar artifact separately or more frequently than the main
> server regardless of repo choice and as per Roopa's point having a separate
> release artifact (jar or deb/rpm) is probably a good idea until the default
> Cassandra packages don't automatically stop and start Cassandra on install.
>
> While we were developing the repair scheduler in a separate repo we had a
> number of annoying issues that only started surfacing once we started
> merging it directly into the trunk tree:
> 1. It is hard to compile/test against unreleased versions of Cassandra
> (e.g. the JMX interfaces changed a lot with 4.x, and it was pretty tricky
> to compile against that as the main project doesn't release nightly
> snapshots or anything like that, so we had to build local trunk jars and
> then manually dep them).
> 2. Integration testing and cross version compatibility is extremely hard.
> The sidecar is going to be involved in multi node coordination (e.g.
> monitoring, metrics, maintenance) and will be tightly coupled to JMX
> interface choices in the server and trying to make sure that it all works
> with multiple versions of Cassandra is much harder if it's a separate repo
> that has to have a mirroring release cycle to Cassandra. It seems much
> easier to have it in tree and just be like "the in tree sidecar is tested
> against that version of Cassandra". Every time we cut a Cassandra server
> branch the sidecar branches with it.
>
> We experience these pains all the time with Priam being in a separate repo,
> where every time we support a new Cassandra version a bunch of JMX
> endpoints break and we have to refactor the code to either call JMX methods
> by string or cut a new Priam branch. A separate artifact is pretty
> important, but a separate repo just allows drifts. Furthermore from the
> Priam experience I also don't think it's realistic to say one version of a
> sidecar artifact can actually support multiple versions.
>
> -Joey
>
> On Fri, Aug 17, 2018 at 12:00 PM Jeremiah D Jordan 
> wrote:
>
> > Not sure why the two things being in the same repo means they need the
> > same release process.  You can always do interim releases of the
> management
> > artifact between server releases, or even have completely decoupled
> > releases.
> >
> > -Jeremiah
> >
> > > On Aug 17, 2018, at 10:52 AM, Blake Eggleston 
> > wrote:
> > >
> > > I'd be more in favor of making it a separate project, basically for all
> > the reasons listed below. I'm assuming we'd want a management process to
> > work across different versions, which will be more awkward if it's in
> tree.
> > Even if that's not the case, keeping it in a different repo at this point
> > will make iteration easier than if it were in tree. I'd imagine (or at
> > least hope) that validating the management process for release would be
> > less difficult than the main project, so tying them to the Cassandra
> > release cycle seems unnecessarily restrictive.
> > >
> > >
> > > On August 17, 2018 at 12:07:18 AM, Dinesh Joshi (
> dinesh.jo...@yahoo.com.invalid)
> > wrote:
> > >
> > >> On Aug 16, 2018, at 9:27 PM, Sankalp Kohli 
> > wrote:
> > >>
> > >> I am bumping this thread because patch has landed for this with repair
> > functionality.
> > >>
> > >> I have a following proposal for this which I can put in the JIRA or
> doc
> > >>
> > >> 1. We should see if we can keep this in a separate repo like Dtest.
> > >
> > > This would imply a looser coupling between the two. Keeping things
> > in-tree is my preferred approach. It makes testing, dependency management
> > and code sharing easier.
> > >
> > >> 2. It should have its own release process.
> > >
> > > This means now there would be two releases that need to be managed and
> > coordinated.
> > >
> > >> 3. It should have integration tests for different versions of
> Cassandra
> > it will support.
> > >
> > > Given the lack of test infrastructure - this will be hard especially if
> > you have to qualify a matrix of builds.
> > >
> > > Dinesh
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >
> >
> >
> > -
> > To unsubscribe, e-mail: 

Re: Yet another repair solution

[Jonathan Haddad]

I'm also very interested.


On Tue, Aug 28, 2018 at 8:47 AM Dinesh Joshi 
wrote:

> > On Aug 28, 2018, at 6:33 AM, Marcus Olsson 
> wrote:
> >
> > Hi,
> >
> > With the risk of stirring the repair/side-car topic  even further I'd
> just like to mention that we have recently gotten approval to contribute
> our repair management side-car solution.
> > It's based on the proposal in
> https://issues.apache.org/jira/browse/CASSANDRA-10070 as a standalone
> application sitting next to each instance.
> > With the recent discussions in mind I'd just like to hear the thoughts
> from the community on this before we put in the effort of bringing our
> solution into open source.
> >
> > Would there be an interest of having yet another repair solution in the
> discussion?
>
> I personally think looking at multiple options is important. So yes, it
> would be interesting to see your solution.
>
> Dinesh
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Reaper as cassandra-admin

[Jonathan Haddad]

I don't believe #1 should be an issue, Mick has been reaching out.

Alex and Mick are putting together some architecture documentation, I won't
step on their toes.  Currently you can run Reaper as a single instance that
connects to your entire cluster, multiple instances in HA mode, and we're
finishing up the rework of the code to run it as a sidecar.

On Mon, Aug 27, 2018 at 6:02 PM Jeff Jirsa  wrote:

> Can you get all of the contributors cleared?
> What’s the architecture? Is it centralized? Is there a sidecar?
>
>
> > On Aug 27, 2018, at 5:36 PM, Jonathan Haddad  wrote:
> >
> > Hey folks,
> >
> > Mick brought this up in the sidecar thread, but I wanted to have a clear
> /
> > separate discussion about what we're thinking with regard to contributing
> > Reaper to the C* project.  In my mind, starting with Reaper is a great
> way
> > of having an admin right now, that we know works well at the kind of
> scale
> > we need.  We've worked with a lot of companies putting Reaper in prod (at
> > least 50), running on several hundred clusters.  The codebase has evolved
> > as a direct result of production usage, and we feel it would be great to
> > pair it with the 4.0 release.  There was a LOT of work done on the repair
> > logic to make things work across every supported version of Cassandra,
> with
> > a great deal of documentation as well.
> >
> > In case folks aren't aware, in addition to one off and scheduled repairs,
> > Reaper also does cluster wide snapshots, exposes thread pool stats, and
> > visualizes streaming (in trunk).
> >
> > We're hoping to get some feedback on our side if that's something people
> > are interested in.  We've gone back and forth privately on our own
> > preferences, hopes, dreams, etc, but I feel like a public discussion
> would
> > be healthy at this point.  Does anyone share the view of using Reaper as
> a
> > starting point?  What concerns to people have?
> > --
> > Jon Haddad
> > http://www.rustyrazorblade.com
> > twitter: rustyrazorblade
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Reaper as cassandra-admin

[Jonathan Haddad]

Hey folks,

Mick brought this up in the sidecar thread, but I wanted to have a clear /
separate discussion about what we're thinking with regard to contributing
Reaper to the C* project.  In my mind, starting with Reaper is a great way
of having an admin right now, that we know works well at the kind of scale
we need.  We've worked with a lot of companies putting Reaper in prod (at
least 50), running on several hundred clusters.  The codebase has evolved
as a direct result of production usage, and we feel it would be great to
pair it with the 4.0 release.  There was a LOT of work done on the repair
logic to make things work across every supported version of Cassandra, with
a great deal of documentation as well.

In case folks aren't aware, in addition to one off and scheduled repairs,
Reaper also does cluster wide snapshots, exposes thread pool stats, and
visualizes streaming (in trunk).

We're hoping to get some feedback on our side if that's something people
are interested in.  We've gone back and forth privately on our own
preferences, hopes, dreams, etc, but I feel like a public discussion would
be healthy at this point.  Does anyone share the view of using Reaper as a
starting point?  What concerns to people have?
-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Side Car New Repo vs not

[Jonathan Haddad]

Strongly agree with Blake.  In my mind supporting multiple versions is
mandatory.  As I've stated before, we already do it with Reaper, I'd
consider it a major misstep if we couldn't support multiple with the
project - provided admin tool.  It's the same reason dtests are separate -
they work with multiple versions.

The number of repos does not affect distribution - if we want to ship
Cassandra with the admin / repair tool (we should, imo), that can be part
of the build process.




On Mon, Aug 20, 2018 at 9:21 PM Blake Eggleston 
wrote:

> If the sidecar is going to be on a different release cadence, or support
> interacting with mixed mode clusters, then it should definitely be in a
> separate repo. I don’t even know how branching and merging would work in a
> repo that supports 2 separate release targets and/or mixed mode
> compatibility, but I’m pretty sure it would be a mess.
>
> As a cluster management tool, mixed mode is probably going to be a goal at
> some point. As a new project, it will benefit from not being tied to the C*
> release cycle (which would probably delay any sidecar release until
> whenever 4.1 is cut).
>
>
> On August 20, 2018 at 3:22:54 PM, Joseph Lynch (joe.e.ly...@gmail.com)
> wrote:
>
> I think that the pros of incubating the sidecar in tree as a tool first
> outweigh the alternatives at this point of time. Rough tradeoffs that I
> see:
>
> Unique pros of in tree sidecar:
> * Faster iteration speed in general. For example when we need to add a
> new
> JMX endpoint that the sidecar needs, or change something from JMX to a
> virtual table (e.g. for repair, or monitoring) we can do all changes
> including tests as one commit within the main repository and don't have
> to
> commit to main repo, sidecar repo, and dtest repo (juggling version
> compatibility along the way).
> * We can in the future more easily move serious background functionality
> like compaction or repair itself (not repair scheduling, actual
> repairing)
> into the sidecar with a single atomic commit, we don't have to do two
> phase
> commits where we add some IPC mechanism to allow us to support it in
> both,
> then turn it on in the sidecar, then turn it off in the server, etc...
> * I think that the verification is much easier (sounds like Jonathan
> disagreed on the other thread, I could certainly be wrong), and we don't
> have to worry about testing matrices to assure that the sidecar works
> with
> various versions as the version of the sidecar that is released with that
> version of Cassandra is the only one we have to certify works. If people
> want to pull in new versions or maintain backports they can do that at
> their discretion/testing.
> * We can iterate and prove value before committing to a choice. Since it
> will be a separate artifact from the start we can always move the
> artifact
> to a separate repo later (but moving the other way is harder).
> * Users will get the sidecar "for free" when they install the daemon,
> they
> don't need to take affirmative action to e.g. be able to restart their
> cluster, run repair, or back their data up; it just comes out of the box
> for free.
>
> Unique pros of a separate repository sidecar:
> * We can use a more modern build system like gradle instead of ant
> * Merging changes is less "scary" I guess (I feel like if you're not
> touching the daemon this is already true but I could see this being less
> worrisome for some).
> * Releasing a separate artifact is somewhat easier from a separate repo
> (especially if we have gradle which makes e.g. building debs and rpms
> trivial).
> * We could backport to previous versions without getting into arguments
> about bug fixes vs features.
> * Committers could be different from the main repo, which ... may be a
> useful thing
>
> Non unique pros of a sidecar (could be achieved in the main repo or in a
> separate repo):
> * A separate build artifact .jar/.deb/.rpm that can be installed
> separately. It's slightly easier with a separate repo but certainly not
> out
> of reach within a single repo (indeed the current patch already creates a
> separate jar, and we could create a separate .deb reasonably easily).
> Personally I think having a separate .deb/.rpm is premature at this point
> (for companies that really want it they can build their own packages
> using
> the .jars), but I think it really is a distracting issue from where the
> patch should go as we can always choose to remove experimental .jar files
> that the main daemon doesn't touch.
> * A separate process lifecycle. No matter where the sidecar goes, we get
> the benefit of restarting it being less dangerous for availability than
> restarting the main daemon.
>
> That all being said, these are strong opinions weakly held and I would
> rather get something actually committed so that we can prove value one
> way
> or the other and am therefore, of course, happy to put sidecar patches
> wherever someone can review and commit it.
>
> -Joey
>
> On Mon, Aug 

Re: Java 11 Z garbage collector

[Jonathan Haddad]

Advertised, yes, but so far I haven't found it to be any better than
ParNew + CMS or G1 in the performance tests I did when writing
http://thelastpickle.com/blog/2018/08/16/java11.html.

That said, I didn't try it with a huge heap (i think it was 16 or 24GB), so
maybe it'll do better if I throw 50 GB RAM at it.



On Thu, Aug 30, 2018 at 8:42 AM Carl Mueller
 wrote:

> https://www.opsian.com/blog/javas-new-zgc-is-very-exciting/
>
> .. max of 4ms for stop the world, large terabyte heaps, seems promising.
>
> Will this be a major boon to cassandra p99 times? Anyone know the aspects
> of cassandra that cause the most churn and lead to StopTheWorld GC? I was
> under the impression that bloom filters, caches, etc are statically
> allocated at startup.
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: NGCC 2018?

[Jonathan Haddad]

I'm thinking a month or two after 4.0 would give us time to unwind after
the release and start to give real thought to big changes coming in the
next release.  Let's focus on one thing at a time.

On Wed, Sep 5, 2018 at 12:29 PM sankalp kohli 
wrote:

> A good time for NGCC will be closer to 4.0 release where we can plan what
> we can put it on 4.0-next. I am not sure doing it now is going to help when
> we are months away from 4.0 release.
>
> On Fri, Aug 31, 2018 at 7:42 AM Jay Zhuang  wrote:
>
> > Are we going to have a dev event next month? Or anything this year? We
> may
> > also be able to provide space in bay area and help to organize it.
> (Please
> > let us know, so we could get final approval for that).
> >
> > On Fri, Jul 27, 2018 at 10:05 AM Jonathan Haddad 
> > wrote:
> >
> > > My interpretation of Nate's statement was that since there would be a
> > bunch
> > > of us at Lynn's event, we might as well do NGCC at the same time.
> > >
> > > On Thu, Jul 26, 2018 at 9:03 PM Ben Bromhead 
> > wrote:
> > >
> > > > It sounds like there may be an appetite for something, but the NGCC
> in
> > > its
> > > > current format is likely to not be that useful?
> > > >
> > > > Is a bay area event focused on C* developers something that is
> > > interesting
> > > > for the broader dev community? In whatever format that may be?
> > > >
> > > > On Tue, Jul 24, 2018 at 5:02 PM Nate McCall 
> > wrote:
> > > >
> > > > > This was discussed amongst the PMC recently. We did not come to a
> > > > > conclusion and there were not terribly strong feelings either way.
> > > > >
> > > > > I don't feel like we need to hustle to get "NGCC" in place,
> > > > > particularly given our decided focus on 4.0. However, that should
> not
> > > > > stop us from doing an additional 'c* developer' event in sept. to
> > > > > coincide with distributed data summit.
> > > > >
> > > > > On Wed, Jul 25, 2018 at 5:03 AM, Patrick McFadin <
> pmcfa...@gmail.com
> > >
> > > > > wrote:
> > > > > > Ben,
> > > > > >
> > > > > > Lynn Bender had offered a space the day before Distributed Data
> > > Summit
> > > > in
> > > > > > September (http://distributeddatasummit.com/) since we are both
> > > > platinum
> > > > > > sponsors. I thought he and Nate had talked about that being a
> good
> > > > place
> > > > > > for NGCC since many of us will be in town already.
> > > > > >
> > > > > > Nate, now that I've spoken for you, you can clarify, :D
> > > > > >
> > > > > > Patrick
> > > > > >
> > > > >
> > > > >
> -
> > > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > > > >
> > > > > --
> > > > Ben Bromhead
> > > > CTO | Instaclustr <https://www.instaclustr.com/>
> > > > +1 650 284 9692
> > > > Reliability at Scale
> > > > Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
> > > >
> > >
> > >
> > > --
> > > Jon Haddad
> > > http://www.rustyrazorblade.com
> > > twitter: rustyrazorblade
> > >
> >
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: QA signup

[Jonathan Haddad]

Really good idea JD. Keeping all the tests under an umbrella ticket for the
feature with everything linked back makes a lot of sense.

On Thu, Sep 6, 2018 at 11:09 PM J. D. Jordan 
wrote:

> I would suggest that JIRA’s tagged as 4.0 blockers be created for the list
> once it is fleshed out.  Test plans and results could be posted to said
> JIRAs, to be closed once a given test passes. Any bugs found can also then
> be related back to such a ticket for tracking them as well.
>
> -Jeremiah
>
> > On Sep 6, 2018, at 12:27 PM, Jonathan Haddad  wrote:
> >
> > I completely agree with you, Sankalp.  I didn't want to dig too deep into
> > the underlying testing methodology (and I still think we shouldn't just
> > yet) but if the goal is to have confidence in the release, our QA process
> > needs to be comprehensive.
> >
> > I believe that having focused teams for each component with a team leader
> > with support from committers & contributors gives us the best shot at
> > defining large scale functional tests that can be used to form both
> > progress and bug reports.  (A person could / hopefully will be on more
> than
> > one team).  Coming up with those comprehensive tests will be the jobs of
> > the teams, getting frequent bidirectional feedback on the dev ML.  Bugs
> go
> > in JIRA as per usual.
> >
> > Hopefully we can continue this process after the release, giving the
> > project more structure, and folding more people in over time as
> > contributors and ideally committers / PMC.
> >
> > Jon
> >
> >
> >> On Thu, Sep 6, 2018 at 1:15 PM sankalp kohli 
> wrote:
> >>
> >> Thanks for starting this Jon.
> >> Instead of saying "I tested streaming", we should define what all was
> >> tested like was all data transferred, what happened when stream failed,
> >> etc.
> >> Based on talking to a few users, looks like most testing is done by
> doing
> >> an operation or running a load and seeing if it "worked" and no errors
> in
> >> logs.
> >>
> >> Another important thing will be to fix bugs asap ahead of testing,  as
> >> fixes can lead to more bugs :)
> >>
> >>>> On Thu, Sep 6, 2018 at 7:52 AM Jonathan Haddad 
> wrote:
> >>>
> >>> I was thinking along the same lines.  For this to be successful I think
> >>> either weekly or bi-weekly summary reports back to the mailing list by
> >> the
> >>> team lead for each subsection on what's been tested and how it's been
> >>> tested will help keep things moving along.
> >>>
> >>> In my opinion the lead for each team should *not* be the contributor
> that
> >>> wrote the feature, but someone who's very interested in it and can use
> >> the
> >>> contributor as a resource.  I think it would be difficult for the
> >>> contributor to poke holes in their own work - if they could do that it
> >>> would have been done already.  This should be a verification process
> >> that's
> >>> independent as possible from the original work.
> >>>
> >>> In addition to the QA process, it would be great if we could get a docs
> >>> team together.  We've got quite a bit of undocumented features and
> nuance
> >>> still, I think hammering that out would be a good idea.  Mick brought
> up
> >>> updating the website docs in the thread on testing different JDK's [1],
> >> if
> >>> we could figure that out in the process we'd be in a really great
> >> position
> >>> from the user perspective.
> >>>
> >>> Jon
> >>>
> >>> [1]
> >>
> https://lists.apache.org/thread.html/5645178efb57939b96e73ab9c298e80ad8e76f11a563b4d250c1ae38@%3Cdev.cassandra.apache.org%3E
> >>>
> >>>>> On Thu, Sep 6, 2018 at 10:35 AM Jordan West 
> wrote:
> >>>>
> >>>> Thanks for staring this thread Jon!
> >>>>
> >>>>> On Thu, Sep 6, 2018 at 5:51 AM Jonathan Haddad 
> >>>> wrote:
> >>>>
> >>>>> For 4.0, I'm thinking it would be a good idea to put together a list
> >> of
> >>>> the
> >>>>> things that need testing and see if people are willing to help test /
> >>>> break
> >>>>> those things.  My goal here is to get as much coverage as possible,
> >> and
> >>>> let
> >>>>> folks focus on really hammering on specific things ra

Re: Proposing an Apache Cassandra Management process

[Jonathan Haddad]

It's interesting to me that someone would consider features of Reaper as
"technical debt"... an odd way to word it.  When we had proposed donating
Reaper to the project, I had made the assumption people would realize the
benefit of folding in a successful project.  A working codebase with a
large user base gives us an immediate tool, and it would let us migrate
everyone from using our tool to using an official tool.

>From the discussions we've had so far, nobody except Blake seems to care
about those users.  To me, the highest priority of this process is
considering what's best for the Cassandra community.  Providing an easy,
clear migration path forward for the thousands of Reaper users, supporting
all 4 backends and cassandra versions is the logical the way of
transitioning everyone to using the official tool.

If supporting everyone using 2.x with a postgres backend for storing
schedules isn't something anyone cares about, there's not much benefit from
using Reaper.  Gutting a bunch of the project won't help people migrate
over, and if we're not migrating everyone over, then we (TLP) will still
have to maintain Reaper.  That means we'll be indefinitely maintaining a
fork of the official admin, and probably not contributing to the main one,
our time is limited.  That's exactly what's going to happen if we start
with anything other than Reaper.  We've got a lot of teams asking for
features, and if people pay us to keep working on Reaper, we'll be doing
that.

So, with that in mind, if we're putting this to a vote, I'd like to be
clear - taking Reaper as-is about community - not it's technical prowess.
If we're not enabling users to move off our version and directly to the
official tool, I don't see a point in using Reaper at all except as a
reference to correctly run subrange repair and have a nice UI.  There's
plenty of things I'd do differently if I built it from scratch, I'm not
putting the codebase up on a pedestal.

This entire discussion has been incredibly frustrating for me, considering
we're talking about building an alternative to a well tested and widely
deployed codebase that likely won't produce anything of value for at least
a year (Cassandra 5?).  I'm also pretty sure most of the folks that have
cried out "tech debt" have spent less than 5 minutes looking at the
codebase.  I hope your enthusiasm at the moment would carry over if you
build a tech-debt free admin tool from the ground up.

TL;DR: If nobody cares about the Reaper community (which is a very large
subset of the cassandra community) there's no reason to start with Reaper,
it's not about the tech it's about the people.

Jon


On Sat, Sep 8, 2018 at 4:57 PM sankalp kohli  wrote:

> Most of the discussions have been around whether we take some side car or
> do a cherry pick approach where we add a feature at a time to side car and
> use the best implementation.
> We have been discussing this first in April and now for several days. I
> think we all want some progress here. We will start a vote soon..may be
> next week to decide which approach we will take. I don't see any other
> option to make progress besides a vote!!
>
> PS: I think these discussions are very good for the community and it shows
> people care about Apache Cassandra and it is a sign of healthy community.
> Several people offering to donate the side car or help build is showing the
> interest everyone has in it.
>
>
> On Sat, Sep 8, 2018 at 11:17 AM Joseph Lynch 
> wrote:
>
> > On Fri, Sep 7, 2018 at 10:00 PM Blake Eggleston 
> > wrote:
> > >
> > > Right, I understand the arguments for starting a new project. I’m not
> > saying reaper is, technically speaking, the best place to start. The
> point
> > I’m trying to make is that the non-technical advantages of using an
> > existing project as a starting point may outweigh the technical benefits
> of
> > a clean slate. Whether that’s the case or not, it’s not a strictly
> > technical decision, and the non-technical advantages of starting with
> > reaper need to be weighed.
> > >
> >
> > Technical debt doesn't just refer to the technical solutions, having
> > an existing user base means that a project has made promises in the
> > past (in the case of Priam 5+ years ago) that the new project would
> > have to keep if we make keeping users of those sidecars a goal (which
> > for the record I don't think should be a goal, I think the goal is to
> > make Cassandra the database work out of the box in the 4.x series).
> >
> > For example, Priam has to continue supporting the following as users
> > actively use them (including Netflix):
> > * Parallel token assignment and creation allowing parallel bootstrap
> > and parallel doubling of hundred node clusters at once (as long as you
> > use single tokens and run in AWS with asgs).
> > * 3+ backup solutions, as well as assumptions about where in e.g. S3
> > backups are present and what format they're present in.
> > * Numerous configuration options and UI elements (mostly 5 

Re: QA signup

[Jonathan Haddad]

I was thinking along the same lines.  For this to be successful I think
either weekly or bi-weekly summary reports back to the mailing list by the
team lead for each subsection on what's been tested and how it's been
tested will help keep things moving along.

In my opinion the lead for each team should *not* be the contributor that
wrote the feature, but someone who's very interested in it and can use the
contributor as a resource.  I think it would be difficult for the
contributor to poke holes in their own work - if they could do that it
would have been done already.  This should be a verification process that's
independent as possible from the original work.

In addition to the QA process, it would be great if we could get a docs
team together.  We've got quite a bit of undocumented features and nuance
still, I think hammering that out would be a good idea.  Mick brought up
updating the website docs in the thread on testing different JDK's [1], if
we could figure that out in the process we'd be in a really great position
from the user perspective.

Jon

[1]
https://lists.apache.org/thread.html/5645178efb57939b96e73ab9c298e80ad8e76f11a563b4d250c1ae38@%3Cdev.cassandra.apache.org%3E

On Thu, Sep 6, 2018 at 10:35 AM Jordan West  wrote:

> Thanks for staring this thread Jon!
>
> On Thu, Sep 6, 2018 at 5:51 AM Jonathan Haddad  wrote:
>
> > For 4.0, I'm thinking it would be a good idea to put together a list of
> the
> > things that need testing and see if people are willing to help test /
> break
> > those things.  My goal here is to get as much coverage as possible, and
> let
> > folks focus on really hammering on specific things rather than just
> firing
> > up a cluster and rubber stamping it.  If we're going to be able to
> > confidently deploy 4.0 quickly after it's release we're going to need a
> > high attention to detail.
> >
> >
> +1 to a more coordinated effort. I think we could use the Confluence that
> was set up a little bit ago since it was setup for this purpose, at least
> for finalized plans and results:
> https://cwiki.apache.org/confluence/display/CASSANDRA.
>
>
> > In addition to a signup sheet, I think providing some guidance on how to
> QA
> > each thing that's being tested would go a long way.  Throwing "hey please
> > test sstable streaming" over the wall will only get quality feedback from
> > folks that are already heavily involved in the development process.  It
> > would be nice to bring some new faces into the project by providing a
> > little guidance.
> >
>
> > We could help facilitate this even further by considering the people
> > signing up to test a particular feature as a team, with seasoned
> Cassandra
> > veterans acting as team leads.
> >
>
> +1 to this as well. I am always a fan of folks learning about a
> subsystem/project through testing. It can be challenging to get folks new
> to a project excited about testing first but for those that do, or for
> committers who want to learn another part of the db, its a great way to
> learn.
>
> Another thing we can do here is make sure teams are writing about the
> testing they are doing and their results. This will help share knowledge
> about techniques and approaches that others can then apply. This knowledge
> can be shared on the mailing list, a blog post, or in JIRA.
>
>  Jordan
>
>
> > Any thoughts?  I'm happy to take the lead on this.
> > --
> > Jon Haddad
> > http://www.rustyrazorblade.com
> > twitter: rustyrazorblade
> >
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

QA signup

[Jonathan Haddad]

For 4.0, I'm thinking it would be a good idea to put together a list of the
things that need testing and see if people are willing to help test / break
those things.  My goal here is to get as much coverage as possible, and let
folks focus on really hammering on specific things rather than just firing
up a cluster and rubber stamping it.  If we're going to be able to
confidently deploy 4.0 quickly after it's release we're going to need a
high attention to detail.

In addition to a signup sheet, I think providing some guidance on how to QA
each thing that's being tested would go a long way.  Throwing "hey please
test sstable streaming" over the wall will only get quality feedback from
folks that are already heavily involved in the development process.  It
would be nice to bring some new faces into the project by providing a
little guidance.

We could help facilitate this even further by considering the people
signing up to test a particular feature as a team, with seasoned Cassandra
veterans acting as team leads.

Any thoughts?  I'm happy to take the lead on this.
-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: QA signup

[Jonathan Haddad]

I completely agree with you, Sankalp.  I didn't want to dig too deep into
the underlying testing methodology (and I still think we shouldn't just
yet) but if the goal is to have confidence in the release, our QA process
needs to be comprehensive.

I believe that having focused teams for each component with a team leader
with support from committers & contributors gives us the best shot at
defining large scale functional tests that can be used to form both
progress and bug reports.  (A person could / hopefully will be on more than
one team).  Coming up with those comprehensive tests will be the jobs of
the teams, getting frequent bidirectional feedback on the dev ML.  Bugs go
in JIRA as per usual.

Hopefully we can continue this process after the release, giving the
project more structure, and folding more people in over time as
contributors and ideally committers / PMC.

Jon


On Thu, Sep 6, 2018 at 1:15 PM sankalp kohli  wrote:

> Thanks for starting this Jon.
> Instead of saying "I tested streaming", we should define what all was
> tested like was all data transferred, what happened when stream failed,
> etc.
> Based on talking to a few users, looks like most testing is done by doing
> an operation or running a load and seeing if it "worked" and no errors in
> logs.
>
> Another important thing will be to fix bugs asap ahead of testing,  as
> fixes can lead to more bugs :)
>
> On Thu, Sep 6, 2018 at 7:52 AM Jonathan Haddad  wrote:
>
> > I was thinking along the same lines.  For this to be successful I think
> > either weekly or bi-weekly summary reports back to the mailing list by
> the
> > team lead for each subsection on what's been tested and how it's been
> > tested will help keep things moving along.
> >
> > In my opinion the lead for each team should *not* be the contributor that
> > wrote the feature, but someone who's very interested in it and can use
> the
> > contributor as a resource.  I think it would be difficult for the
> > contributor to poke holes in their own work - if they could do that it
> > would have been done already.  This should be a verification process
> that's
> > independent as possible from the original work.
> >
> > In addition to the QA process, it would be great if we could get a docs
> > team together.  We've got quite a bit of undocumented features and nuance
> > still, I think hammering that out would be a good idea.  Mick brought up
> > updating the website docs in the thread on testing different JDK's [1],
> if
> > we could figure that out in the process we'd be in a really great
> position
> > from the user perspective.
> >
> > Jon
> >
> > [1]
> >
> >
> https://lists.apache.org/thread.html/5645178efb57939b96e73ab9c298e80ad8e76f11a563b4d250c1ae38@%3Cdev.cassandra.apache.org%3E
> >
> > On Thu, Sep 6, 2018 at 10:35 AM Jordan West  wrote:
> >
> > > Thanks for staring this thread Jon!
> > >
> > > On Thu, Sep 6, 2018 at 5:51 AM Jonathan Haddad 
> > wrote:
> > >
> > > > For 4.0, I'm thinking it would be a good idea to put together a list
> of
> > > the
> > > > things that need testing and see if people are willing to help test /
> > > break
> > > > those things.  My goal here is to get as much coverage as possible,
> and
> > > let
> > > > folks focus on really hammering on specific things rather than just
> > > firing
> > > > up a cluster and rubber stamping it.  If we're going to be able to
> > > > confidently deploy 4.0 quickly after it's release we're going to
> need a
> > > > high attention to detail.
> > > >
> > > >
> > > +1 to a more coordinated effort. I think we could use the Confluence
> that
> > > was set up a little bit ago since it was setup for this purpose, at
> least
> > > for finalized plans and results:
> > > https://cwiki.apache.org/confluence/display/CASSANDRA.
> > >
> > >
> > > > In addition to a signup sheet, I think providing some guidance on how
> > to
> > > QA
> > > > each thing that's being tested would go a long way.  Throwing "hey
> > please
> > > > test sstable streaming" over the wall will only get quality feedback
> > from
> > > > folks that are already heavily involved in the development process.
> It
> > > > would be nice to bring some new faces into the project by providing a
> > > > little guidance.
> > > >
> > >
> > > > We could help facilitate this even further by considering the people
> > > >

Re: Proposing an Apache Cassandra Management process

[Jonathan Haddad]

We haven’t even defined any requirements for an admin tool. It’s hard to
make a case for anything without agreement on what we’re trying to build.

On Fri, Sep 7, 2018 at 7:17 PM Jeff Jirsa  wrote:

> How can we continue moving this forward?
>
> Mick/Jon/TLP folks, is there a path here where we commit the
> Netflix-provided management process, and you augment Reaper to work with
> it?
> Is there a way we can make a larger umbrella that's modular that can
> support either/both?
> Does anyone believe there's a clear, objective argument that one is
> strictly better than the other? I haven't seen one.
>
>
>
> On Mon, Aug 20, 2018 at 4:14 PM Roopa Tangirala
>  wrote:
>
> > +1 to everything that Joey articulated with emphasis on the fact that
> > contributions should be evaluated based on the merit of code and their
> > value add to the whole offering. I  hope it does not matter whether that
> > contribution comes from PMC member or a person who is not a committer. I
> > would like the process to be such that it encourages the new members to
> be
> > a part of the community and not shy away from contributing to the code
> > assuming their contributions are valued differently than committers or
> PMC
> > members. It would be sad to see the contributions decrease if we go down
> > that path.
> >
> > *Regards,*
> >
> > *Roopa Tangirala*
> >
> > Engineering Manager CDE
> >
> > *(408) 438-3156 - mobile*
> >
> >
> >
> >
> >
> >
> > On Mon, Aug 20, 2018 at 2:58 PM Joseph Lynch 
> > wrote:
> >
> > > > We are looking to contribute Reaper to the Cassandra project.
> > > >
> > > Just to clarify are you proposing contributing Reaper as a project via
> > > donation or you are planning on contributing the features of Reaper as
> > > patches to Cassandra? If the former how far along are you on the
> donation
> > > process? If the latter, when do you think you would have patches ready
> > for
> > > consideration / review?
> > >
> > >
> > > > Looking at the patch it's very similar in its base design already,
> but
> > > > Reaper does has a lot more to offer. We have all been working hard to
> > > move
> > > > it to also being a side-car so it can be contributed. This raises a
> > > number
> > > > of relevant questions to this thread: would we then accept both works
> > in
> > > > the Cassandra project, and what burden would it put on the current
> PMC
> > to
> > > > maintain both works.
> > > >
> > > I would hope that we would collaborate on merging the best parts of all
> > > into the official Cassandra sidecar, taking the always on, shared
> > nothing,
> > > highly available system that we've contributed a patchset for and
> adding
> > in
> > > many of the repair features (e.g. schedules, a nice web UI) that Reaper
> > > has.
> > >
> > >
> > > > I share Stefan's concern that consensus had not been met around a
> > > > side-car, and that it was somehow default accepted before a patch
> > landed.
> > >
> > >
> > > I feel this is not correct or fair. The sidecar and repair discussions
> > have
> > > been anything _but_ "default accepted". The timeline of consensus
> > building
> > > involving the management sidecar and repair scheduling plans:
> > >
> > > Dec 2016: Vinay worked with Jon and Alex to try to collaborate on
> Reaper
> > to
> > > come up with design goals for a repair scheduler that could work at
> > Netflix
> > > scale.
> > >
> > > ~Feb 2017: Netflix believes that the fundamental design gaps prevented
> us
> > > from using Reaper as it relies heavily on remote JMX connections and
> > > central coordination.
> > >
> > > Sep. 2017: Vinay gives a lightning talk at NGCC about a highly
> available
> > > and distributed repair scheduling sidecar/tool. He is encouraged by
> > > multiple committers to build repair scheduling into the daemon itself
> and
> > > not as a sidecar so the database is truly eventually consistent.
> > >
> > > ~Jun. 2017 - Feb. 2018: Based on internal need and the positive
> feedback
> > at
> > > NGCC, Vinay and myself prototype the distributed repair scheduler
> within
> > > Priam and roll it out at Netflix scale.
> > >
> > > Mar. 2018: I open a Jira (CASSANDRA-14346) along with a detailed 20
> page
> > > design document for adding repair scheduling to the daemon itself and
> > open
> > > the design up for feedback from the community. We get feedback from
> Alex,
> > > Blake, Nate, Stefan, and Mick. As far as I know there were zero
> proposals
> > > to contribute Reaper at this point. We hear the consensus that the
> > > community would prefer repair scheduling in a separate distributed
> > sidecar
> > > rather than in the daemon itself and we re-work the design to match
> this
> > > consensus, re-aligning with our original proposal at NGCC.
> > >
> > > Apr 2018: Blake brings the discussion of repair scheduling to the dev
> > list
> > > (
> > >
> > >
> >
> https://lists.apache.org/thread.html/760fbef677f27aa5c2ab4c375c7efeb81304fea428deff986ba1c2eb@%3Cdev.cassandra.apache.org%3E
> > > ).
> > > 

Re: Testing 4.0 Post-Freeze

[Jonathan Haddad]

The most impactful change that I can see is the people that are
involved with development who are willing to -1 a release.  Those of
us with votes are TLP have a big incentive for 4.0 to be stable - we
want people to upgrade.  I assume folks at Netflix, Apple are also
very incentivized to see a very stable 4.0. That's a lot of committer
head count already.

I don't remember much community call to action in the past with regard
to getting folks testing releases with real workloads.  If we want
help, it's on us to ask.  Making it as easy as possible to test stuff
out and get feedback is also on us.  Banning folks from committing to
trunk isn't solving the main problem: it's still pretty difficult to
get involved.  We need to lower the barrier to entry for setting &
tearing down test clusters.  We also need to recruit community members
to be part of a QA feedback cycle.  Once folks are in, keeping them
involved for multiple iterations will improve their ability to give
feedback.

The nice part is, if we do it right, eventually maybe those folks will
commit some code and become committers down the line.

On Mon, Jul 9, 2018 at 11:31 AM sankalp kohli  wrote:
>
> We have done all this for previous releases and we know it has not worked
> well. So how giving it one more try is going to help here. Can someone
> outline what will change for 4.0 which will make it more successful?
>
> Not branching is one idea but we should discuss what will change now than
> iterating what has already happened.
>
> On Mon, Jul 9, 2018 at 11:25 AM Jeremy Hanna 
> wrote:
>
> > I wholeheartedly agree with efforts to make releases more stable and the
> > more contributors that add tests or test within the context of their own
> > use cases, the better.  +1 non-binding to the idea of better, more complete
> > testing for releases.
> >
> > When it comes to how to do this, I think that’s where there is
> > disagreement.  I personally don’t think that branching detracts from the
> > goal of stabilization.  There are a couple of personas involved here:
> >
> > * Longtime contributors/committers: I think it’s sufficient to generally
> > ask that whatever time/effort they can contribute be towards stabilizing,
> > testing, and especially testing their production scenarios to cover more
> > surface area when it comes to usage edge cases.  That along with testing
> > longer running things in those scenarios for other types of edge cases.
> >
> > * New contributors: They likely won’t know about the strategy.  They’re
> > just poking around and looking at lhf tickets or tickets that they need
> > done.  Those are typically low risk but at the same time we don’t want to
> > compromise stability by merging those in.  Reviewing low risk stuff for
> > inclusion doesn’t take a ton of time and gives them a sense that they can
> > be of help and empowers them.  After they have that first experience, then
> > a longer term contributor could share with them a blog post or tldr thread
> > summary about the 4.0 stabilization efforts (would be nice to have one to
> > point people too once we're done).  We could also start creating lhf
> > tickets with stabilization and testing in mind.
> >
> > Unless we want to make a fundamental change to the project’s process
> > (making trunk stable at all times going forward), then I don’t think
> > there’s a reason why branching would detract from these efforts.  In other
> > words if we’re just trying to say that we want to focus on stabilization
> > for the alpha/beta/rc of 4.0, then I would prefer branching along with
> > clear messaging.
> >
> > Jeremy
> >
> > > On Jul 9, 2018, at 12:43 PM, sankalp kohli 
> > wrote:
> > >
> > > How is this preventing someone from working and collaborating on an
> > Apache
> > > Project? You can still collaborate on making 4.0 more stable.
> > >
> > > Why will these committers not work on making 4.0 more stable and fix
> > broken
> > > tests? Considering  we cannot release without test passing, how will
> > these
> > > features be used?
> > >
> > > On Mon, Jul 9, 2018 at 10:03 AM Jonathan Haddad 
> > wrote:
> > >
> > >> I don't see how changing the process and banning feature commits is
> > >> going to be any help to the project. There may be a couple committers
> > >> who are interested in committing new features.
> > >>
> > >> I'm a -1 on changing the branching strategy in a way that prevents
> > >> people from working and collaborating on an Apache project.
> > >>
> > >> On Mon, Jul 9, 2018 at 9:56 AM sankalp ko

Re: Testing 4.0 Post-Freeze

[Jonathan Haddad]

I guess I look at the idea of changing the branching strategy as a
means of blocking work as a very odd way of solving a human problem.
Having a majority of votes temporarily block potentially good work
might be a good thing, and it might not matter at all.  It might also
frustrate some folks who have been around for a really long time.

I'm also making the assumption that I don't know every plausible
reason someone might need / want to merge into trunk and considering
that there's valid cases for it that we'll be blocking.

With regard to "the process has been broken for years" I've already
said my bit on what I considered to different now, nobody has
responded to that yet.  I think I've said all I need to on this, it's
probably just noise now, so I won't respond any further on this
thread.  I don't anticipate having a personal need to commit to a
future 5.0 release before 4.0 is out, so it won't impact me
personally.

On Tue, Jul 10, 2018 at 10:32 AM Benedict Elliott Smith
 wrote:
>
> That’s a peculiar way of looking at it.
>
> Committer status is not an absolute right to autonomy over the codebase.  
> It’s an embodiment of trust that you will follow the community's prevailing 
> rules around commit, and that you’re competent to do so.
>
> If the community wants to change those rules you’re trusted to follow, how 
> does this modify your right, or the trust emplaced in you?
>
>
>
>
>
> > On 10 Jul 2018, at 10:18, Jonathan Haddad  wrote:
> >
> > I guess I look at the initial voting in of committers as the process
> > by which people are trusted to merge things in.  This proposed process
> > revokes that trust. If Jonathan Ellis or Dave Brosius (arbitrarily
> > picked) wants to merge a new feature into trunk during the freeze, now
> > they're not allowed?  That's absurd.  People have already met the bar
> > and have been voted in by merit, they should not have their privilege
> > revoked.
> > On Tue, Jul 10, 2018 at 10:14 AM Ben Bromhead  wrote:
> >>
> >> Well put Mick
> >>
> >> +1
> >>
> >> On Tue, Jul 10, 2018 at 1:06 PM Aleksey Yeshchenko 
> >> wrote:
> >>
> >>> +1 from me too.
> >>>
> >>> —
> >>> AY
> >>>
> >>> On 10 July 2018 at 04:17:26, Mick Semb Wever (m...@apache.org) wrote:
> >>>
> >>>
> >>>> We have done all this for previous releases and we know it has not
> >>> worked
> >>>> well. So how giving it one more try is going to help here. Can someone
> >>>> outline what will change for 4.0 which will make it more successful?
> >>>
> >>>
> >>> I (again) agree with you Sankalp :-)
> >>>
> >>> Why not try something new?
> >>> It's easier to discuss these things more genuinely after trying it out.
> >>>
> >>> One of the differences in the branching approaches: to feature-freeze on a
> >>> 4.0 branch or on trunk; is who it is that has to then merge and work with
> >>> multiple branches.
> >>>
> >>> Where that small but additional effort is placed I think becomes a signal
> >>> to what the community values most: new features or stability.
> >>>
> >>> I think most folk would vote for stability, so why not give this approach
> >>> a go and to learn from it.
> >>> It also creates an incentive to make the feature-freeze period as short as
> >>> possible, moving us towards an eventual goal of not needing to
> >>> feature-freeze at all.
> >>>
> >>> regards,
> >>> Mick
> >>>
> >>> -
> >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>>
> >>> --
> >> Ben Bromhead
> >> CTO | Instaclustr <https://www.instaclustr.com/>
> >> +1 650 284 9692
> >> Reliability at Scale
> >> Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
> >
> >
> >
> > --
> > Jon Haddad
> > http://www.rustyrazorblade.com
> > twitter: rustyrazorblade
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>


-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Testing 4.0 Post-Freeze

[Jonathan Haddad]

I guess I look at the initial voting in of committers as the process
by which people are trusted to merge things in.  This proposed process
revokes that trust. If Jonathan Ellis or Dave Brosius (arbitrarily
picked) wants to merge a new feature into trunk during the freeze, now
they're not allowed?  That's absurd.  People have already met the bar
and have been voted in by merit, they should not have their privilege
revoked.
On Tue, Jul 10, 2018 at 10:14 AM Ben Bromhead  wrote:
>
> Well put Mick
>
> +1
>
> On Tue, Jul 10, 2018 at 1:06 PM Aleksey Yeshchenko 
> wrote:
>
> > +1 from me too.
> >
> > —
> > AY
> >
> > On 10 July 2018 at 04:17:26, Mick Semb Wever (m...@apache.org) wrote:
> >
> >
> > > We have done all this for previous releases and we know it has not
> > worked
> > > well. So how giving it one more try is going to help here. Can someone
> > > outline what will change for 4.0 which will make it more successful?
> >
> >
> > I (again) agree with you Sankalp :-)
> >
> > Why not try something new?
> > It's easier to discuss these things more genuinely after trying it out.
> >
> > One of the differences in the branching approaches: to feature-freeze on a
> > 4.0 branch or on trunk; is who it is that has to then merge and work with
> > multiple branches.
> >
> > Where that small but additional effort is placed I think becomes a signal
> > to what the community values most: new features or stability.
> >
> > I think most folk would vote for stability, so why not give this approach
> > a go and to learn from it.
> > It also creates an incentive to make the feature-freeze period as short as
> > possible, moving us towards an eventual goal of not needing to
> > feature-freeze at all.
> >
> > regards,
> > Mick
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> > --
> Ben Bromhead
> CTO | Instaclustr 
> +1 650 284 9692
> Reliability at Scale
> Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer



-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Scratch an itch

[Jonathan Haddad]

>
> Deferring huge amount of commits gives rebase/redo hell. That's the
> biggest impact and the order in which these deferred commits are then
> actually committed can make it more painful or less painful depending on
> the commit. And that in turn will have to then wait for each contributor
> to rebase/redo their commit and those timings might make more rebase
> issues. If those committers will want to rebase something after n-months
> or have time at that point.


This was a concern I had initially as well, but the more I thought about
it, the less concerned I became.  If trunk drifts far away from a 4.0
branch, we would *still* have to deal with the pain of merging everything
forward.  It would just change who is responsible for doing the work.


On Thu, Jul 12, 2018 at 10:54 AM Michael Burman  wrote:

> On 07/12/2018 07:38 PM, Stefan Podkowinski wrote:
> > this point? Also, if we tell someone that their contribution will be
> > reviewed and committed later after 4.0-beta, how is that actually making
> > a difference for that person, compared to committing it now for a 4.x
> > version. It may be satisfying to get a patch committed, but what matters
> > more is when the code will actually be released and deferring committing
> > contributions after 4.0-beta doesn't necessarily mean that there's any
> > disadvantage when it comes to that.
> >
> Deferring huge amount of commits gives rebase/redo hell. That's the
> biggest impact and the order in which these deferred commits are then
> actually committed can make it more painful or less painful depending on
> the commit. And that in turn will have to then wait for each contributor
> to rebase/redo their commit and those timings might make more rebase
> issues. If those committers will want to rebase something after n-months
> or have time at that point.
>
> That's a problem for all Cassandra patches that take huge time to commit
> and if this block takes a lot of time, then that will for sure be even
> more painful. I know products such as Kubernetes does the same (I guess
> that's where this idea might have come from) "trunk patches only", but
> their block is quite short.
>
> My wish is that this freeze does not last too long to kill enthusiasm
> towards committing to Cassandra. There are (I assume) many hobbyist who
> do this as a side-project instead of their daily work and might not have
> the capabilities to test 4.0 in a way that will trigger bugs (easy bugs
> are fixed quite quickly I hope). And if they feel like it's not worth
> the time at this point to invest time to Cassandra (because nothing they
> do will get merged) - they might move to another project. And there's no
> guarantee they will return. Getting stuff to the product is part of the
> satisfaction and without satisfaction there's no interest in continuing.
>
>- Micke
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re: Testing 4.0 Post-Freeze

[Jonathan Haddad]

I don't see how changing the process and banning feature commits is
going to be any help to the project. There may be a couple committers
who are interested in committing new features.

I'm a -1 on changing the branching strategy in a way that prevents
people from working and collaborating on an Apache project.

On Mon, Jul 9, 2018 at 9:56 AM sankalp kohli  wrote:
>
> I did not see a -1 but all +0s and a few +1s.
>
> On Mon, Jul 9, 2018 at 5:49 AM Josh McKenzie  wrote:
>
> > What did we land on? Sounds like we're pretty split without consensus on
> > "change the old branching strategy and reject new things on trunk during
> > stabilization" vs. "keep doing things the way we did but message strongly
> > that we won't be reviewing new things until 4.0 is stable".
> >
> > On Sat, Jul 7, 2018 at 5:01 PM Sankalp Kohli 
> > wrote:
> >
> > > Does anyone has any more feedback on this?
> > >
> > > > On Jul 4, 2018, at 05:36, Aleksey Yeshchenko 
> > wrote:
> > > >
> > > > For what it’s worth, I’m fine with both formal branching-level freeze
> > > and informal ‘let people commit to trunk if they can find a reviewer’ one
> > > and will support either.
> > > >
> > > > So long as either/both are communicated to the contributors, the only
> > > difference is in where new feature work gets accumulated: will stay a bit
> > > longer in personal branches in the latter way.
> > > >
> > > > —
> > > > AY
> > > >
> > > > On 4 July 2018 at 01:30:40, sankalp kohli (kohlisank...@gmail.com)
> > > wrote:
> > > >
> > > > Having an explicit way to tell the community that we all will work on
> > > > testing is better than writing a patch which will sit without review
> > > for
> > > > months. I think not having your patch reviewed for months is more
> > > > discouraging than following the community and helping with stability of
> > > > 4.0.
> > > >
> > > >
> > > >
> > > > On Tue, Jul 3, 2018 at 3:02 PM Josh McKenzie 
> > > wrote:
> > > >
> > > >>>
> > > >>> We propose that between the September freeze date and beta, a new
> > > branch
> > > >>> would not be created and trunk would only have bug fixes and
> > > performance
> > > >>> improvements committed to it.
> > > >>
> > > >>
> > > >> This is more of a call to action and announcement of intent than an
> > > attempt
> > > >>> to
> > > >>> enforce policy; we can and probably will branch off 4.0, and keep
> > > trunk
> > > >>> technically active.
> > > >>
> > > >> These are two very different statements. :) Which is it?
> > > >>
> > > >> On Tue, Jul 3, 2018 at 5:57 PM Aleksey Yeshchenko 
> > > >> wrote:
> > > >>
> > > >>> If we want to have a stable, usable 4.0.0 release out in the next
> > > 6-12
> > > >>> months, there needs to be a focused effort on getting it out - or
> > > else
> > > >>> it’ll just never happen.
> > > >>>
> > > >>> This is more of a call to action and announcement of intent than an
> > > >>> attempt to enforce policy; we can and probably will branch off 4.0,
> > > and
> > > >>> keep trunk technically active. But so long as there is a critical
> > mass
> > > of
> > > >>> active contributors who are on board with only/mostly working on
> > > >> stability,
> > > >>> bug fixes, and validation - both as assignees and reviewers - we’ll
> > > have
> > > >> a
> > > >>> de-facto freeze.
> > > >>>
> > > >>> And I have a feeling that there is such a critical mass.
> > > >>>
> > > >>> —
> > > >>> AY
> > > >>>
> > > >>> On 3 July 2018 at 22:23:38, Jeff Jirsa (jji...@gmail.com) wrote:
> > > >>>
> > > >>> I think there's value in the psychological commitment that if someone
> > > has
> > > >>> time to contribute, their contributions should be focused on
> > > validating a
> > > >>> release, not pushing future features.
> > > >>>
> > > >>>
&

Re: Testing 4.0 Post-Freeze

[Jonathan Haddad]

Not everyone wants to work and collaborate the way you do.  It seems
absurd to me to force everyone to hold off on merging into a single
branch just because a lot of us want to prioritize testing 4.0.

I think most committers are going to want to contribute to the 4.0
testing process more than they want to commit new features to trunk,
but I also think people shouldn't be banned from new bringing in new
features if that's what they want to do.

You're essentially giving a blanket -1 to all new features for a 3-6
month period.
On Mon, Jul 9, 2018 at 10:44 AM sankalp kohli  wrote:
>
> How is this preventing someone from working and collaborating on an Apache
> Project? You can still collaborate on making 4.0 more stable.
>
> Why will these committers not work on making 4.0 more stable and fix broken
> tests? Considering  we cannot release without test passing, how will these
> features be used?
>
> On Mon, Jul 9, 2018 at 10:03 AM Jonathan Haddad  wrote:
>
> > I don't see how changing the process and banning feature commits is
> > going to be any help to the project. There may be a couple committers
> > who are interested in committing new features.
> >
> > I'm a -1 on changing the branching strategy in a way that prevents
> > people from working and collaborating on an Apache project.
> >
> > On Mon, Jul 9, 2018 at 9:56 AM sankalp kohli 
> > wrote:
> > >
> > > I did not see a -1 but all +0s and a few +1s.
> > >
> > > On Mon, Jul 9, 2018 at 5:49 AM Josh McKenzie 
> > wrote:
> > >
> > > > What did we land on? Sounds like we're pretty split without consensus
> > on
> > > > "change the old branching strategy and reject new things on trunk
> > during
> > > > stabilization" vs. "keep doing things the way we did but message
> > strongly
> > > > that we won't be reviewing new things until 4.0 is stable".
> > > >
> > > > On Sat, Jul 7, 2018 at 5:01 PM Sankalp Kohli 
> > > > wrote:
> > > >
> > > > > Does anyone has any more feedback on this?
> > > > >
> > > > > > On Jul 4, 2018, at 05:36, Aleksey Yeshchenko 
> > > > wrote:
> > > > > >
> > > > > > For what it’s worth, I’m fine with both formal branching-level
> > freeze
> > > > > and informal ‘let people commit to trunk if they can find a
> > reviewer’ one
> > > > > and will support either.
> > > > > >
> > > > > > So long as either/both are communicated to the contributors, the
> > only
> > > > > difference is in where new feature work gets accumulated: will stay
> > a bit
> > > > > longer in personal branches in the latter way.
> > > > > >
> > > > > > —
> > > > > > AY
> > > > > >
> > > > > > On 4 July 2018 at 01:30:40, sankalp kohli (kohlisank...@gmail.com)
> > > > > wrote:
> > > > > >
> > > > > > Having an explicit way to tell the community that we all will work
> > on
> > > > > > testing is better than writing a patch which will sit without
> > review
> > > > > for
> > > > > > months. I think not having your patch reviewed for months is more
> > > > > > discouraging than following the community and helping with
> > stability of
> > > > > > 4.0.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Jul 3, 2018 at 3:02 PM Josh McKenzie  > >
> > > > > wrote:
> > > > > >
> > > > > >>>
> > > > > >>> We propose that between the September freeze date and beta, a new
> > > > > branch
> > > > > >>> would not be created and trunk would only have bug fixes and
> > > > > performance
> > > > > >>> improvements committed to it.
> > > > > >>
> > > > > >>
> > > > > >> This is more of a call to action and announcement of intent than
> > an
> > > > > attempt
> > > > > >>> to
> > > > > >>> enforce policy; we can and probably will branch off 4.0, and keep
> > > > > trunk
> > > > > >>> technically active.
> > > > > >>
> > > > > >> These are two very different statements. :) Which is it?
> > > > > >>

Re: Testing 4.0 Post-Freeze

[Jonathan Haddad]

My proposal is that we implement everything you're suggesting, except
we branch off as we have in the past rather than locking down trunk.
I'd encourage everyone to work to improve 4.0 in some way or another,
whether it be fixing bugs, testing in a staging environment
(feedback), writing tooling (data loaders, stress testing, cluster
deployment tools), improving unit / dtests tests and writing docs.
But outright blocking folks from committing a feature they may have
been working on for months makes me very uncomfortable.  I don't think
there's going to be much of this, but it seems a little authoritarian
to me to mandate that it's not allowed.

My personal preference is that everyone commit to making 4.0 stable on
day 1, and we work towards that goal as a community, but not in a
manner that's so heavy handed.

On Mon, Jul 9, 2018 at 11:06 AM sankalp kohli  wrote:
>
> If some committers want to bring in features without a path forward for
> releasing(as tests are broken), is that an acceptable state for you? How do
> we get out of this state.
>
> Like I said in my original email, this is a "proposal" and I am trying to
> see how we can improve things here. So if you are -1 on this, please help
> us propose something else to get these tests pass?
>
> And thanks for giving your feedback.
>
> On Mon, Jul 9, 2018 at 10:50 AM Jonathan Haddad  wrote:
>
> > Not everyone wants to work and collaborate the way you do.  It seems
> > absurd to me to force everyone to hold off on merging into a single
> > branch just because a lot of us want to prioritize testing 4.0.
> >
> > I think most committers are going to want to contribute to the 4.0
> > testing process more than they want to commit new features to trunk,
> > but I also think people shouldn't be banned from new bringing in new
> > features if that's what they want to do.
> >
> > You're essentially giving a blanket -1 to all new features for a 3-6
> > month period.
> > On Mon, Jul 9, 2018 at 10:44 AM sankalp kohli 
> > wrote:
> > >
> > > How is this preventing someone from working and collaborating on an
> > Apache
> > > Project? You can still collaborate on making 4.0 more stable.
> > >
> > > Why will these committers not work on making 4.0 more stable and fix
> > broken
> > > tests? Considering  we cannot release without test passing, how will
> > these
> > > features be used?
> > >
> > > On Mon, Jul 9, 2018 at 10:03 AM Jonathan Haddad 
> > wrote:
> > >
> > > > I don't see how changing the process and banning feature commits is
> > > > going to be any help to the project. There may be a couple committers
> > > > who are interested in committing new features.
> > > >
> > > > I'm a -1 on changing the branching strategy in a way that prevents
> > > > people from working and collaborating on an Apache project.
> > > >
> > > > On Mon, Jul 9, 2018 at 9:56 AM sankalp kohli 
> > > > wrote:
> > > > >
> > > > > I did not see a -1 but all +0s and a few +1s.
> > > > >
> > > > > On Mon, Jul 9, 2018 at 5:49 AM Josh McKenzie 
> > > > wrote:
> > > > >
> > > > > > What did we land on? Sounds like we're pretty split without
> > consensus
> > > > on
> > > > > > "change the old branching strategy and reject new things on trunk
> > > > during
> > > > > > stabilization" vs. "keep doing things the way we did but message
> > > > strongly
> > > > > > that we won't be reviewing new things until 4.0 is stable".
> > > > > >
> > > > > > On Sat, Jul 7, 2018 at 5:01 PM Sankalp Kohli <
> > kohlisank...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Does anyone has any more feedback on this?
> > > > > > >
> > > > > > > > On Jul 4, 2018, at 05:36, Aleksey Yeshchenko <
> > alek...@apple.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > For what it’s worth, I’m fine with both formal branching-level
> > > > freeze
> > > > > > > and informal ‘let people commit to trunk if they can find a
> > > > reviewer’ one
> > > > > > > and will support either.
> > > > > > > >
> > > > > > > > So long as either/both are communicated to the contributors,
> > the
> > > > only
&g

Re: [VOTE] Branching Change for 4.0 Freeze

[Jonathan Haddad]

+1,

I've come around on this, I think the long and short term benefits
will be worth it.
On Fri, Jul 13, 2018 at 11:17 AM Vinay Chella
 wrote:
>
> +1 nb
>
> Regards,
> Vinay Chella
>
>
> On Fri, Jul 13, 2018 at 11:15 AM Michael Shuler 
> wrote:
>
> > +0
> >
> > There are pros and cons. I do hope the pros work out and the cons aren't
> > too impactful. I thought about just abstaining, but figured a "meh,
> > whatever" vote was at least worth voicing.
> >
> > Michael
> >
> > On 07/11/2018 04:46 PM, sankalp kohli wrote:
> > > Hi,
> > > As discussed in the thread[1], we are proposing that we will not
> > branch
> > > on 1st September but will only allow following merges into trunk.
> > >
> > > a. Bug and Perf fixes to 4.0.
> > > b. Critical bugs in any version of C*.
> > > c. Testing changes to help test 4.0
> > >
> > > If someone has a change which does not fall under these three, we can
> > > always discuss it and have an exception.
> > >
> > > Vote will be open for 72 hours.
> > >
> > > Thanks,
> > > Sankalp
> > >
> > > [1]
> > >
> > https://lists.apache.org/thread.html/494c3ced9e83ceeb53fa127e44eec6e2588a01b769896b25867fd59f@%3Cdev.cassandra.apache.org%3E
> > >
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >



-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org