Re: A proposal to move away from Jira-centric development

[Mick Semb Wever]

On 17 August 2016 at 03:47, Benedict Elliott Smith 
wrote:

> What this project really needs, and the board is chomping at the bit about,
> is diversity.  The fact is, right now DataStax does 95% of the substantive
> development on the project, and so they make all the decisions.  As such,
> their internal community outweighs the Apache community.  I will emphasise
> clearly for my ex-colleagues, I'm not making any value judgement about
> this, just clarifying the crux of the discussion that everyone seems to be
> dancing around.
>


Thanks for that Benedict, it was well written and is an important issue.
And straight off the bat I interpret it without any negative implications
to DataStax.

Of course it is going to be difficult with so many of the community are
employees within one company.

Back to the issue I've always been a big fan of how discussions were kept
in JIRA, and always seen it as a significant addition to the Apache Way
(specifically CS50 in the Maturity Model). But it seems the times have
moved on and one specific "dev" mailing list is now being seen as
under-utilised.

http://community.apache.org/apache-way/apache-project-
maturity-model.html#consensus-building

There has been a number of tickets and decisions made along the way that
could have come out (early) onto the dev ML. For example sometimes
ideas/tickets come out appearing that they have already been
rubber-stamped. If outsiders can't find that early discussion around ideas
they can too easily presume the sinister, that DataStax is running the show
behind the scenes. That's no good for DataStax and it's no good for the
Cassandra. Personally I'm tired of having to defend both DataStax and the
Cassandra community when it boils down to silly misunderstandings like
this.

Efforts like those numbered by Jeremiah would be greatly appreciated.
It makes sense because if all we do is simply shuffling and duplicating
information around between two persistent transparent searchable channels
then we don't achieve much beyond breaking context and threads. When I read
CS50 it's not about the dev ML being the Apache Way while discussions on
tickets are not. It's that the community needs to identify all the really
important decisions early on and: deciding that the dev ML is the
'project's main communications channel'; raise them there. Just generally
increasing traffic to the dev ML is going to be a distraction to that goal.

~mck

Re: A proposal to move away from Jira-centric development

[Jeremy Hanna]

I think a separate mailing list for just ticket creation would be nice as well. 
 I think that’s what many of us filter down the commits@ list to.  That doesn’t 
have to happen in place of the proposed change but would make it easier for 
people to follow new issue creation.  From there I go to and follow/comment 
on/etc. issues I’m interested in.

> On Aug 16, 2016, at 4:06 PM, Eric Evans  wrote:
> 
> On Mon, Aug 15, 2016 at 9:22 AM, Jonathan Ellis  wrote:
> 
> [ ... ]
> 
>> I propose that we take advantage of the dev list to perform that
>> separation.  Major new features and architectural improvements should be
>> discussed first here, then when consensus on design is achieved, moved to
>> Jira for implementation and review.
>> 
>> I think this will also help with the problem when the initial idea proves
>> to be unworkable and gets revised substantially later after much
>> discussion.  It can be difficult to figure out what the conclusion was, as
>> review comments start to pile up afterwards.  Having that discussion on the
>> list, and summarizing on Jira, would mitigate this.
> 
> TL;DR +1
> 
> I think there are actually a couple of related, but disjoint issues here.
> 
> IMO, a JIRA should be the source of truth for an issue, a way to track
> any on-going efforts, and a historical account after-the-fact.
> Regardless of where you think discussions should take place, I would
> argue there is room for improvement here; Many of our JIRAs (I would
> argue the most interesting ones!), are very difficult to make use of
> for either of these cases (current status, or after-the-fact).  Some
> curation (as someone else pointed out in this thread), could go a long
> way.  Retitling and/or revising the description as the scope of a
> ticket evolves, or posting a summary or current status in the
> description body would be ways for people who are up to speed on an
> issue, to spend a few minutes making it valuable to others.  So would
> summarizing discussions that take place elsewhere.
> 
> The other issue is discoverability and/or inclusivity.  If the only
> way to keep abreast of what is happening is to follow the fire-hose of
> all JIRA updates, then contribution is going to be limited to those
> with the bandwidth.  If you work on Cassandra full-time, this probably
> doesn't seem like a big deal, but if your time is limited, then it can
> create quite a barrier (and I've been on both sides of this with
> Cassandra).  So moving serious discussions to the mailing list is also
> a sort of curation, since it creates a venue free of all the
> minutiae/noise.
> 
> My personal opinion is also that it's far easier to manage a given
> volume with email, and that the discussions are easier to follow (the
> interface is better at representing the ontology, for example), but
> from what I can gather, not everyone agrees so YMMV.
> 
> Cheers,
> 
> -- 
> Eric Evans
> john.eric.ev...@gmail.com

Re: A proposal to move away from Jira-centric development

[Eric Evans]

On Mon, Aug 15, 2016 at 9:22 AM, Jonathan Ellis  wrote:

[ ... ]

> I propose that we take advantage of the dev list to perform that
> separation.  Major new features and architectural improvements should be
> discussed first here, then when consensus on design is achieved, moved to
> Jira for implementation and review.
>
> I think this will also help with the problem when the initial idea proves
> to be unworkable and gets revised substantially later after much
> discussion.  It can be difficult to figure out what the conclusion was, as
> review comments start to pile up afterwards.  Having that discussion on the
> list, and summarizing on Jira, would mitigate this.

TL;DR +1

I think there are actually a couple of related, but disjoint issues here.

IMO, a JIRA should be the source of truth for an issue, a way to track
any on-going efforts, and a historical account after-the-fact.
Regardless of where you think discussions should take place, I would
argue there is room for improvement here; Many of our JIRAs (I would
argue the most interesting ones!), are very difficult to make use of
for either of these cases (current status, or after-the-fact).  Some
curation (as someone else pointed out in this thread), could go a long
way.  Retitling and/or revising the description as the scope of a
ticket evolves, or posting a summary or current status in the
description body would be ways for people who are up to speed on an
issue, to spend a few minutes making it valuable to others.  So would
summarizing discussions that take place elsewhere.

The other issue is discoverability and/or inclusivity.  If the only
way to keep abreast of what is happening is to follow the fire-hose of
all JIRA updates, then contribution is going to be limited to those
with the bandwidth.  If you work on Cassandra full-time, this probably
doesn't seem like a big deal, but if your time is limited, then it can
create quite a barrier (and I've been on both sides of this with
Cassandra).  So moving serious discussions to the mailing list is also
a sort of curation, since it creates a venue free of all the
minutiae/noise.

My personal opinion is also that it's far easier to manage a given
volume with email, and that the discussions are easier to follow (the
interface is better at representing the ontology, for example), but
from what I can gather, not everyone agrees so YMMV.

Cheers,

-- 
Eric Evans
john.eric.ev...@gmail.com

Re: A proposal to move away from Jira-centric development

[Eric Evans]

On Tue, Aug 16, 2016 at 3:09 PM, Benedict Elliott Smith
 wrote:
> This is a great example of email's inadequacies, as this innocuous (to me) 
> little textual
> act resulted instead in *different* quagmire, while the first potential 
> quagmire is still in
> play!
>
> Email is a minefield, and textual interactions can be exhausting.  So
> people tap out without fully expressing themselves, to retain their life
> and sanity.

My apologies, it wasn't my intention to drag you into a quagmire.  You
had indicated a reluctance to discuss a particular topic, and
attributed it to email.  Personally, I think that this is a) a
conversation worth having, and b) one that others have been reluctant
to engage in.  My intention was to understand the reluctance, and if
possible, encourage you (and others) to try.

Cheers,

-- 
Eric Evans
john.eric.ev...@gmail.com

Re: A proposal to move away from Jira-centric development

[Benedict Elliott Smith]

Like many difficult problems, it is easier to point them out than to
suggest improvements.  Anyway, I wasn't proposing we change the mechanisms
of communication, just excusing my simplification of (my view of) the
problem to avoid ending up in a quagmire on that topic.  This is a great
example of email's inadequacies, as this innocuous (to me) little textual
act resulted instead in *different* quagmire, while the first potential
quagmire is still in play!

Email is a minefield, and textual interactions can be exhausting.  So
people tap out without fully expressing themselves, to retain their life
and sanity.



On 16 August 2016 at 20:49, Eric Evans  wrote:

> On Tue, Aug 16, 2016 at 2:38 PM, Benedict Elliott Smith
>  wrote:
> > I think all complex, nuanced and especially emotive topics are
> challenging
> > to discuss over textual media, due to things like the attention span of
> > your readers, the difficulties in structuring your text, and especially
> the
> > hoops that have to be jumped through to minimise the potential for
> > misinterpretation, as well as correcting the inevitable
> misinterpretations
> > that happen anyway.
>
> Fair enough, I suppose, but some of these things are also difficult
> face to face.  Most people who collaborate over the Internet with
> people from different backgrounds in different timezones, etc, learn
> to adjust accordingly.  And, the asynchronicity of email is often a
> feature in this regard, giving people the opportunity to more
> carefully consider what they've read, and to be more deliberate in
> their response.
>
> I guess what I should have asked is, if not email, then how?
>
>
> --
> Eric Evans
> john.eric.ev...@gmail.com
>

Re: A proposal to move away from Jira-centric development

[Eric Evans]

On Tue, Aug 16, 2016 at 2:38 PM, Benedict Elliott Smith
 wrote:
> I think all complex, nuanced and especially emotive topics are challenging
> to discuss over textual media, due to things like the attention span of
> your readers, the difficulties in structuring your text, and especially the
> hoops that have to be jumped through to minimise the potential for
> misinterpretation, as well as correcting the inevitable misinterpretations
> that happen anyway.

Fair enough, I suppose, but some of these things are also difficult
face to face.  Most people who collaborate over the Internet with
people from different backgrounds in different timezones, etc, learn
to adjust accordingly.  And, the asynchronicity of email is often a
feature in this regard, giving people the opportunity to more
carefully consider what they've read, and to be more deliberate in
their response.

I guess what I should have asked is, if not email, then how?


-- 
Eric Evans
john.eric.ev...@gmail.com

Re: A proposal to move away from Jira-centric development

[Benedict Elliott Smith]

I think all complex, nuanced and especially emotive topics are challenging
to discuss over textual media, due to things like the attention span of
your readers, the difficulties in structuring your text, and especially the
hoops that have to be jumped through to minimise the potential for
misinterpretation, as well as correcting the inevitable misinterpretations
that happen anyway.  But that's a major side track we shouldn't deviate
down.

On 16 August 2016 at 20:28, Eric Evans  wrote:

> On Tue, Aug 16, 2016 at 1:34 PM, Benedict Elliott Smith
>  wrote:
> > This topic is complex, and fully exploring all of the detail would be
> onerous over email.
>
> Out of curiosity, why; What makes this topic so difficult to discuss over
> email?
>
> > DataStax, in my opinion, consciously tries to be a good citizen.
> However there
> > are emergent properties of a system with this imbalance that are not
> conscious,
> > and are suboptimal, and it is not unreasonable for the Apache apparatus
> to
> > try to "rectify" the imbalance.  I personally support that *in
> principle*, but I think
> > they're not going about it brilliantly right now.  I also doubt the
> success of any
> > such endeavour, given how difficult the problem is.
>
> This.  A good first step in my opinion would be for us all to simply
> recognize this.  An imbalance of this nature is not good for the
> project, full stop.  No malice needs to be attributed, no effigies
> burned, and it shouldn't be viewed as squaring up against those we
> know and respect who are employed by Datastax.
>
>
> --
> Eric Evans
> john.eric.ev...@gmail.com
>

Re: A proposal to move away from Jira-centric development

[Nate McCall]

+1 (non-binding)

Thanks Jeremiah. This is moving us in the right direction.

On Wed, Aug 17, 2016 at 5:31 AM, Jeremiah D Jordan <
jeremiah.jor...@gmail.com> wrote:

> Back to the topic at hand.  First, let us establish that all of this stuff
> will be happening “on the mailing lists”, all JIRA updates are sent to
> commits@ with the reply-to set to dev@, so “JIRA” is still “on the list".
>
> Now we just need to decide how we would like to best make use of these
> lists.  I propose that we keep dev@ fairly low volume so that people
> don’t feel the need to filter it out of their inbox, and thus possibly miss
> important discussions.
> If someone cares so much about the name of the list where stuff happens,
> then I propose we make dev-announce@ and if that happens we can replace
> commits@ with dev@ below and dev@ with dev-announce@ and start forwarding
> some JIRA stuff to dev@…
>
> In order to keep dev@ low volume (but higher than it currently is, as it
> has mostly been “no volume” lately) I propose the following:
>
> Someone has a major feature that they would like to discuss.  (Again this
> is just for major features, not every day bug fixes etc)
> 1. Make a JIRA for the thing you want to discuss (aka post the thing to
> commits@)
> 2. Post link to JIRA with a short description to dev@
> 3. Have a discussion on the JIRA (aka commits@) about the new thing.
> 4. If there is some major change/question on the JIRA that people feel
> needs some extra discussion/involvement email dev@ with question and link
> back to the JIRA
> 5. Have more discussions on the JIRA (aka commits@) about the new thing.
> 6. If something else comes up go back too step 4.
> 7. During this process of decision making keep the “Title” and
> “Description” fields of the JIRA (aka commits@) up to date with what is
> actually happening in the ticket.
> 8. Once things settle down make sub tasks or follow on tickets for
> actually implementing things linked to the initial ticket.
>
> That would keep the dev@ list informed of what is going on in new feature
> proposals, and it will keep discussions on JIRA tickets where they are
> easily referenced and kept in one place, so it is easy to get to, and easy
> for.
>
> -Jeremiah
>
> > On Aug 15, 2016, at 9:22 AM, Jonathan Ellis  wrote:
> >
> > A long time ago, I was a proponent of keeping most development
> discussions
> > on Jira, where tickets can be self contained and the threadless nature
> > helps keep discussions from getting sidetracked.
> >
> > But Cassandra was a lot smaller then, and as we've grown it has become
> > necessary to separate out the signal (discussions of new features and
> major
> > changes) from the noise of routine bug reports.
> >
> > I propose that we take advantage of the dev list to perform that
> > separation.  Major new features and architectural improvements should be
> > discussed first here, then when consensus on design is achieved, moved to
> > Jira for implementation and review.
> >
> > I think this will also help with the problem when the initial idea proves
> > to be unworkable and gets revised substantially later after much
> > discussion.  It can be difficult to figure out what the conclusion was,
> as
> > review comments start to pile up afterwards.  Having that discussion on
> the
> > list, and summarizing on Jira, would mitigate this.
> >
> > --
> > Jonathan Ellis
> > Project Chair, Apache Cassandra
> > co-founder, http://www.datastax.com
> > @spyced
>
>

Re: A proposal to move away from Jira-centric development

[Benedict Elliott Smith]

This was very much not my intention to imply.  I thought I had crafted the
email carefully to not imply that at all.  This topic is complex, and fully
exploring all of the detail would be onerous over email.  DataStax, in my
opinion, consciously tries to be a good citizen.  However there are
emergent properties of a system with this imbalance that are not conscious,
and are suboptimal, and it is not unreasonable for the Apache apparatus to
try to "rectify" the imbalance.  I personally support that *in principle*,
but I think they're not going about it brilliantly right now.  I also doubt
the success of any such endeavour, given how difficult the problem is.

I do, however, think the project could improve how welcoming it is.  Both
in the areas Jon mentions, but also in how much effort is put into
mentoring newcomers and responding to technical questions.  The project is
far from *unwelcoming, *but mentoring is (very) costly, and when success at
your dayjob is measured in the code you contribute, this clearly takes
priority.

I don't know how to change that - again, as far as conscious actions are
concerned, I have personally witnessed DataStax try to put more effort into
this, as well as trying to drum up new external contributors through
bootcamps.  But these efforts have had limited success.

On 16 August 2016 at 19:04, Dave Brosius  wrote:

> While i agree with this generally, it's misleading.
>
> It comes across like Datastax is dictating and excluding others from
> participating, or perhaps discouraging others or whatever.
>
> The truth is, whenever someone comes along who is independent, and
> interested in developing Apache Cassandra, they are welcomed, and do
> participate, and do develop, and soon after become Datastax employees.
> Not always of course, but a common pattern. It only makes sense for
> Datastax to hire people who are interested in and capable of developing
> Apache Cassandra. But the truth is a whole lot less sinister than the
> inference.
>
> --dave
> [not associated with Datastax]
>
>
>
> On 2016-08-16 13:47, Benedict Elliott Smith wrote:
>
>> This is a much more useful focusing of the discussion, in my opinion.  It
>> seemed to me that city hall was focusing on a very narrow definition of
>> project health.
>>
>> I would be the first to say the project need to improve here, but doing so
>> will be challenging;  I'm not sure anyone really knows how to go about it.
>> Which is why we end up in these local minima of discussions about the
>> minutiae of JIRA replication.
>>
>> What this project really needs, and the board is chomping at the bit
>> about,
>> is diversity.  The fact is, right now DataStax does 95% of the substantive
>> development on the project, and so they make all the decisions.  As such,
>> their internal community outweighs the Apache community.  I will emphasise
>> clearly for my ex-colleagues, I'm not making any value judgement about
>> this, just clarifying the crux of the discussion that everyone seems to be
>> dancing around.
>>
>> The question is, what can be done about it?  The project needs a lot of
>> new
>> highly productive and independent contributors who are capable of
>> meaningfully shaping project direction.  The problem is we don't know how
>> to achieve that.
>>
>>
>>
>> On 16 August 2016 at 17:24, Dennis E. Hamilton 
>> wrote:
>>
>>
>>>
>>> > -Original Message-
>>> > From: Eric Stevens [mailto:migh...@gmail.com]
>>> > Sent: Tuesday, August 16, 2016 06:10
>>> > To: dev@cassandra.apache.org
>>> > Subject: Re: A proposal to move away from Jira-centric development
>>> >
>>> > I agree with Benedict that we really shouldn't be getting into a
>>> > legalese
>>> > debate on this subject, however "it didn't happen" has been brought up
>>> > as a
>>> > hammer in this conversation multiple times, and I think it's important
>>> > that
>>> > we put it to rest.  It's pretty clear cut that projects are free to
>>> > disregard this advice.  "It didn't happen" is a motto, not a rule.
>>> [orcmid]
>>>
>>> >> ity-model.html
>>> >
>>>
>>> Please read them all, but especially the sections on Community, Consensus
>>> Building, and Independence.
>>>
>>> Apache projects are expected to govern themselves, PMCs are responsible
>>> for it, and PMC Chairs (officers of the foundation) are accountable to
>>> the
>>> Board on how the project is striving toward maturity.
>>>
>>> On occasion, deviations are so notable that there is objection.  It is
>>> not
>>> that folks run around policing the projects.  But there are occasions
>>> where
>>> there are concerns that a project has gone astray.
>>>
>>> One maturity factor that might not be emphasized enough is
>>> Sustainability.  It is about the transparency of project conduct, the
>>> inclusiveness of operation and visibility, and the ways that growth and
>>> turnover are accommodated.  Since we are 

Re: A proposal to move away from Jira-centric development

[Jonathan Haddad]

I don't know about everyone else, but a big deterrent in contributing code
to Cassandra for me (over the last 4 years or so) is the massive amount of
ramp up that needs to happen in order to get started working on something
meaningful.  This comes in a variety of forms - understanding how test
failures aren't actually failures (being corrected now), lack of comments
(for example:
https://github.com/PaytmLabs/cassandra/blob/master/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java),
out of date wiki (now semi retired but still a source of misinformation),
and class names that don't make sense.  Historically there has been a
massive amount of tribal knowledge required to contribute in a significant
way.  Going through JIRA history and asking people who wrote the feature is
the only way to find out what is supposed to be going on.  Let's not forget
the fact that it's a distributed database, so add to all the above issues
the fact that getting this thing right at all is a miraculous achievement.
In order to get more people contributing to the project there needs to be a
significant effort in making it more approachable.

I suspect that as the project continues to move faster, it'll become ever
harder for new contributors to join unless they are paid to work on
Cassandra full time.  Very few people are deploying Tick Tock releases on
purpose - 1 bug fix release doesn't exactly scream reassurance, sorry - so
the folks that are going to scratch their own itch by fixing bugs and
eventually contributing features they need is likely going to drop over
time.  This leaves full time paid positions such as those employed by
DataStax as the logical place to go if you are interested in working on
Cassandra.


On Tue, Aug 16, 2016 at 10:47 AM Benedict Elliott Smith 
wrote:

> This is a much more useful focusing of the discussion, in my opinion.  It
> seemed to me that city hall was focusing on a very narrow definition of
> project health.
>
> I would be the first to say the project need to improve here, but doing so
> will be challenging;  I'm not sure anyone really knows how to go about it.
> Which is why we end up in these local minima of discussions about the
> minutiae of JIRA replication.
>
> What this project really needs, and the board is chomping at the bit about,
> is diversity.  The fact is, right now DataStax does 95% of the substantive
> development on the project, and so they make all the decisions.  As such,
> their internal community outweighs the Apache community.  I will emphasise
> clearly for my ex-colleagues, I'm not making any value judgement about
> this, just clarifying the crux of the discussion that everyone seems to be
> dancing around.
>
> The question is, what can be done about it?  The project needs a lot of new
> highly productive and independent contributors who are capable of
> meaningfully shaping project direction.  The problem is we don't know how
> to achieve that.
>
>
>
> On 16 August 2016 at 17:24, Dennis E. Hamilton 
> wrote:
>
> >
> >
> > > -Original Message-
> > > From: Eric Stevens [mailto:migh...@gmail.com]
> > > Sent: Tuesday, August 16, 2016 06:10
> > > To: dev@cassandra.apache.org
> > > Subject: Re: A proposal to move away from Jira-centric development
> > >
> > > I agree with Benedict that we really shouldn't be getting into a
> > > legalese
> > > debate on this subject, however "it didn't happen" has been brought up
> > > as a
> > > hammer in this conversation multiple times, and I think it's important
> > > that
> > > we put it to rest.  It's pretty clear cut that projects are free to
> > > disregard this advice.  "It didn't happen" is a motto, not a rule.
> > [orcmid]
> >
> > <
> http://community.apache.org/apache-way/apache-project-maturity-model.html
> > >
> >
> > Please read them all, but especially the sections on Community, Consensus
> > Building, and Independence.
> >
> > Apache projects are expected to govern themselves, PMCs are responsible
> > for it, and PMC Chairs (officers of the foundation) are accountable to
> the
> > Board on how the project is striving toward maturity.
> >
> > On occasion, deviations are so notable that there is objection.  It is
> not
> > that folks run around policing the projects.  But there are occasions
> where
> > there are concerns that a project has gone astray.
> >
> > One maturity factor that might not be emphasized enough is
> > Sustainability.  It is about the transparency of project conduct, the
> > inclusiveness of operation and visibility, and the ways that growth and
> > turnover are accommodated.  Since we are looking at mottos, "community
> over
> > code" comes to mind.
> >
> > Project freedom is a bit like the freedom to drive at 100mph on an
> > arterial highway.  Occassionally, the infraction becomes worthy of
> > attention and even a road block and spike strips.
> >
> > While individual preferences are being discussed here, and I agree it is

Re: A proposal to move away from Jira-centric development

[Dave Brosius]

While i agree with this generally, it's misleading.

It comes across like Datastax is dictating and excluding others from 
participating, or perhaps discouraging others or whatever.


The truth is, whenever someone comes along who is independent, and 
interested in developing Apache Cassandra, they are welcomed, and do 
participate, and do develop, and soon after become Datastax 
employees. Not always of course, but a common pattern. It only makes 
sense for Datastax to hire people who are interested in and capable of 
developing Apache Cassandra. But the truth is a whole lot less sinister 
than the inference.


--dave
[not associated with Datastax]


On 2016-08-16 13:47, Benedict Elliott Smith wrote:
This is a much more useful focusing of the discussion, in my opinion.  
It

seemed to me that city hall was focusing on a very narrow definition of
project health.

I would be the first to say the project need to improve here, but doing 
so
will be challenging;  I'm not sure anyone really knows how to go about 
it.

Which is why we end up in these local minima of discussions about the
minutiae of JIRA replication.

What this project really needs, and the board is chomping at the bit 
about,
is diversity.  The fact is, right now DataStax does 95% of the 
substantive
development on the project, and so they make all the decisions.  As 
such,
their internal community outweighs the Apache community.  I will 
emphasise

clearly for my ex-colleagues, I'm not making any value judgement about
this, just clarifying the crux of the discussion that everyone seems to 
be

dancing around.

The question is, what can be done about it?  The project needs a lot of 
new

highly productive and independent contributors who are capable of
meaningfully shaping project direction.  The problem is we don't know 
how

to achieve that.



On 16 August 2016 at 17:24, Dennis E. Hamilton 


wrote:




> -Original Message-
> From: Eric Stevens [mailto:migh...@gmail.com]
> Sent: Tuesday, August 16, 2016 06:10
> To: dev@cassandra.apache.org
> Subject: Re: A proposal to move away from Jira-centric development
>
> I agree with Benedict that we really shouldn't be getting into a
> legalese
> debate on this subject, however "it didn't happen" has been brought up
> as a
> hammer in this conversation multiple times, and I think it's important
> that
> we put it to rest.  It's pretty clear cut that projects are free to
> disregard this advice.  "It didn't happen" is a motto, not a rule.
[orcmid]



Please read them all, but especially the sections on Community, 
Consensus

Building, and Independence.

Apache projects are expected to govern themselves, PMCs are 
responsible
for it, and PMC Chairs (officers of the foundation) are accountable to 
the

Board on how the project is striving toward maturity.

On occasion, deviations are so notable that there is objection.  It is 
not
that folks run around policing the projects.  But there are occasions 
where

there are concerns that a project has gone astray.

One maturity factor that might not be emphasized enough is
Sustainability.  It is about the transparency of project conduct, the
inclusiveness of operation and visibility, and the ways that growth 
and
turnover are accommodated.  Since we are looking at mottos, "community 
over

code" comes to mind.

Project freedom is a bit like the freedom to drive at 100mph on an
arterial highway.  Occassionally, the infraction becomes worthy of
attention and even a road block and spike strips.

While individual preferences are being discussed here, and I agree it 
is
more pertinent than top-posting versus bottom-posting, what is lacking 
is a
broad discussion on community.  Not incumbents and the 
karma-privileged,

but the overall community and how one sustains a thriving project that
strives for maturity.

Folks who are concerned about managing the mail stream and choosing 
what
matters to them might want to discuss ways of operating lists in 
support of
those concerns.  There are positions here and not enough questions 
about
what might be workable inside of the practices and policies that are 
the

waters Apache projects swim in.

 - Dennis

>
> Per ASF newbie FAQ referenced by someone else earlier [1]:
>
> > The section on ASF Mottos is especially useful as a reminder of the
> way
> things are in most ASF projects. This section includes such gems as:
> > * Put community before code.
> > * Let they that do the work make the decisions.
> > * If it didn't happen on a mailing list, it didn't happen.
> > * Don't feed the trolls.
>
> This is presented as a general guideline and not a hard rule, and as
> Benedict points out even this is preceded by a guideline suggesting that
> developers are free to seek alternatives.
[orcmid]

The alternatives must fit within the overall principles, however.  Not
deviate from or weaken them.  This is not an opening 

Re: A proposal to move away from Jira-centric development

[Benedict Elliott Smith]

This is a much more useful focusing of the discussion, in my opinion.  It
seemed to me that city hall was focusing on a very narrow definition of
project health.

I would be the first to say the project need to improve here, but doing so
will be challenging;  I'm not sure anyone really knows how to go about it.
Which is why we end up in these local minima of discussions about the
minutiae of JIRA replication.

What this project really needs, and the board is chomping at the bit about,
is diversity.  The fact is, right now DataStax does 95% of the substantive
development on the project, and so they make all the decisions.  As such,
their internal community outweighs the Apache community.  I will emphasise
clearly for my ex-colleagues, I'm not making any value judgement about
this, just clarifying the crux of the discussion that everyone seems to be
dancing around.

The question is, what can be done about it?  The project needs a lot of new
highly productive and independent contributors who are capable of
meaningfully shaping project direction.  The problem is we don't know how
to achieve that.



On 16 August 2016 at 17:24, Dennis E. Hamilton 
wrote:

>
>
> > -Original Message-
> > From: Eric Stevens [mailto:migh...@gmail.com]
> > Sent: Tuesday, August 16, 2016 06:10
> > To: dev@cassandra.apache.org
> > Subject: Re: A proposal to move away from Jira-centric development
> >
> > I agree with Benedict that we really shouldn't be getting into a
> > legalese
> > debate on this subject, however "it didn't happen" has been brought up
> > as a
> > hammer in this conversation multiple times, and I think it's important
> > that
> > we put it to rest.  It's pretty clear cut that projects are free to
> > disregard this advice.  "It didn't happen" is a motto, not a rule.
> [orcmid]
>
>  >
>
> Please read them all, but especially the sections on Community, Consensus
> Building, and Independence.
>
> Apache projects are expected to govern themselves, PMCs are responsible
> for it, and PMC Chairs (officers of the foundation) are accountable to the
> Board on how the project is striving toward maturity.
>
> On occasion, deviations are so notable that there is objection.  It is not
> that folks run around policing the projects.  But there are occasions where
> there are concerns that a project has gone astray.
>
> One maturity factor that might not be emphasized enough is
> Sustainability.  It is about the transparency of project conduct, the
> inclusiveness of operation and visibility, and the ways that growth and
> turnover are accommodated.  Since we are looking at mottos, "community over
> code" comes to mind.
>
> Project freedom is a bit like the freedom to drive at 100mph on an
> arterial highway.  Occassionally, the infraction becomes worthy of
> attention and even a road block and spike strips.
>
> While individual preferences are being discussed here, and I agree it is
> more pertinent than top-posting versus bottom-posting, what is lacking is a
> broad discussion on community.  Not incumbents and the karma-privileged,
> but the overall community and how one sustains a thriving project that
> strives for maturity.
>
> Folks who are concerned about managing the mail stream and choosing what
> matters to them might want to discuss ways of operating lists in support of
> those concerns.  There are positions here and not enough questions about
> what might be workable inside of the practices and policies that are the
> waters Apache projects swim in.
>
>  - Dennis
>
> >
> > Per ASF newbie FAQ referenced by someone else earlier [1]:
> >
> > > The section on ASF Mottos is especially useful as a reminder of the
> > way
> > things are in most ASF projects. This section includes such gems as:
> > > * Put community before code.
> > > * Let they that do the work make the decisions.
> > > * If it didn't happen on a mailing list, it didn't happen.
> > > * Don't feed the trolls.
> >
> > This is presented as a general guideline and not a hard rule, and as
> > Benedict points out even this is preceded by a guideline suggesting that
> > developers are free to seek alternatives.
> [orcmid]
>
> The alternatives must fit within the overall principles, however.  Not
> deviate from or weaken them.  This is not an opening for arbitrary conduct.
>
> If a major exception is required, it is up to the project to deliberate on
> the matter, agree on the desired exception and its justification, and take
> it to an appropriate venue for ratification.
>
> (It is useful to keep in mind that exceptions are not precedents for
> others to cherry-pick.)
>
> It is also the case that the PMC and, indeed the Chair (although consensus
> is always better), can set policies for the project.  They must be explicit
> and documented and available to all.
>
> It would be really great to stop fighting city hall and, instead, start an
> inquiry into how 

Re: 3.8, 3.9 release plan

[Michael Shuler]

On 08/16/2016 10:52 AM, Aleksey Yeschenko wrote:
> No objections, the plan sounds good to me.
> 
> In addition to that, prep for pushing 3.0.9 out with 3.9.

Thanks. Yes, 3.0.9 is also up for release, without any branch song and
dance :)

-- 
Michael

Re: 3.8, 3.9 release plan

[Aleksey Yeschenko]

No objections, the plan sounds good to me.

In addition to that, prep for pushing 3.0.9 out with 3.9.

-- 
AY

On 16 August 2016 at 16:51:24, Michael Shuler (mich...@pbandjelly.org) wrote:

Yesterday, it was suggested on #cassandra-dev that when 3.9 is ready for  
release, we release 3.8 with the same code base. My plan is to force  
push the contents of cassandra-3.9 branch to the cassandra-3.8 branch,  
updating the version appropriately, so we can build/test from the 3.8  
branch, as usual.  

Any objections to doing this push to 3.8 today, then again when we have  
a green light on 3.9 release?  

Thanks! Michael  

Re: A proposal to move away from Jira-centric development

[Eric Stevens]

I agree with Benedict that we really shouldn't be getting into a legalese
debate on this subject, however "it didn't happen" has been brought up as a
hammer in this conversation multiple times, and I think it's important that
we put it to rest.  It's pretty clear cut that projects are free to
disregard this advice.  "It didn't happen" is a motto, not a rule.

Per ASF newbie FAQ referenced by someone else earlier [1]:

> The section on ASF Mottos is especially useful as a reminder of the way
things are in most ASF projects. This section includes such gems as:
> * Put community before code.
> * Let they that do the work make the decisions.
> * If it didn't happen on a mailing list, it didn't happen.
> * Don't feed the trolls.

This is presented as a general guideline and not a hard rule, and as
Benedict points out even this is preceded by a guideline suggesting that
developers are free to seek alternatives.

Now since this is just a reference to the Incubator code of conduct's list
of mottos (again, not ASF policy), which best source I could find [2],
mirrors the newbie FAQ, but provides the additional insight that the
objective of the motto is transparency.  The spirit of this motto is not
meant to dictate a technology choice, but merely to indicate that
discussions should happen in open spaces where all are able to
participate.  The motto was authored in a time when "the lists" was the
only real option.

Jira absolutely meets the design goal of that motto, if that's the
direction the community chooses, and it's clear from both sources that
individual communities (they that do the work) are free to find the path
here that's best for them.

[1]
https://community.apache.org/newbiefaq.html#NewbieFAQ-IsthereaCodeofConductforApacheprojects
?
[2] *https://wiki.apache.org/incubator/CodeOfConduct#ASF_Mottos
*

On Tue, Aug 16, 2016 at 5:57 AM James Carman 
wrote:

> On Mon, Aug 15, 2016 at 10:23 AM Jonathan Ellis  wrote:
>
> > A long time ago, I was a proponent of keeping most development
> discussions
> > on Jira, where tickets can be self contained and the threadless nature
> > helps keep discussions from getting sidetracked.
> >
> > But Cassandra was a lot smaller then, and as we've grown it has become
> > necessary to separate out the signal (discussions of new features and
> major
> > changes) from the noise of routine bug reports.
> >
> > I propose that we take advantage of the dev list to perform that
> > separation.  Major new features and architectural improvements should be
> > discussed first here, then when consensus on design is achieved, moved to
> > Jira for implementation and review.
> >
> >
> +1!  I think it's important to point out here that nobody is proposing that
> folks have to send an email like:
>
> "I was thinking of naming my variable 'foo' here, what do you guys think?"
>
> However, discussions and decisions that have an impact on Cassandra and its
> direction/architecture (not an all-inclusive list here and we should use
> reason to decide) should happen on the mailing list.  The idea here is
> inclusiveness.  We want everyone in the community to have a chance to
> contribute to these discussions.
>

Re: Failing tests 2016-08-15 [cassandra-3.9]

[J. D. Jordan]

+1 for one email.

> On Aug 16, 2016, at 7:45 AM, Josh McKenzie  wrote:
> 
> Assuming we're single digit failures combined between the two, I think a
> single test failure email would be manageable.
> 
> On Tue, Aug 16, 2016 at 2:46 AM, Joel Knighton 
> wrote:
> 
>> ===
>> testall: 1 failure
>>  org.apache.cassandra.io.compress
>>  .CompressedRandomAccessReaderTest.testDataCorruptionDetection
>>New flaky failure. I've opened CASSANDRA-12465 and assigned
>>myself.
>> 
>> ===
>> dtest: All passed!
>> 
>> ===
>> novnode: All passed!
>> 
>> ===
>> upgrade: 3 failures
>>  upgrade_tests.cql_tests
>>  .TestCQLNodes2RF1_Upgrade_current_3_0_x_To_indev_3_x
>>  .map_keys_indexing_test
>>CASSANDRA-12192. Tyler Hobbs as assignee. They have identified
>>the cause and proposed a test fix. They are also investigating a C*
>>change here to improve robustness.
>>  upgrade_tests.cql_tests
>>  .TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x
>>  .map_keys_indexing_test
>>Same as above.
>>  upgrade_tests.paging_test
>>  .TestPagingDataNodes2RF1_Upgrade_current_2_2_x_To_indev_3_x
>>  .static_columns_paging_test
>>Potentially CASSANDRA-11195, which is open with no clear
>>progress.  I'll follow up with those on that issue tomorrow and see if
>>they agree that this is the same problem.
>> 
>> ===
>> Overall, the testing situation continues to look better. The massive
>> upgrade failures seem to have subsided, so we can continue to target
>> individual failures.
>> 
>> Since the 3.9 tests are getting to a manageable level, we should focus
>> on managing test failures on trunk as well. I will soon start tracking
>> these failures, as well as failures on the large dtest runs, which consist
>> of tests that have been segmented off due to increased cluster size.
>> 
>> On months that we're maintaining a 3.x bugfix branch as well as trunk,
>> is there any preference toward a single email or a separate email for
>> each branch?  Any other feedback is welcome, as always.
>> 

Re: Failing tests 2016-08-15 [cassandra-3.9]

[Josh McKenzie]

Assuming we're single digit failures combined between the two, I think a
single test failure email would be manageable.

On Tue, Aug 16, 2016 at 2:46 AM, Joel Knighton 
wrote:

> ===
> testall: 1 failure
>   org.apache.cassandra.io.compress
>   .CompressedRandomAccessReaderTest.testDataCorruptionDetection
> New flaky failure. I've opened CASSANDRA-12465 and assigned
> myself.
>
> ===
> dtest: All passed!
>
> ===
> novnode: All passed!
>
> ===
> upgrade: 3 failures
>   upgrade_tests.cql_tests
>   .TestCQLNodes2RF1_Upgrade_current_3_0_x_To_indev_3_x
>   .map_keys_indexing_test
> CASSANDRA-12192. Tyler Hobbs as assignee. They have identified
> the cause and proposed a test fix. They are also investigating a C*
> change here to improve robustness.
>   upgrade_tests.cql_tests
>   .TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x
>   .map_keys_indexing_test
> Same as above.
>   upgrade_tests.paging_test
>   .TestPagingDataNodes2RF1_Upgrade_current_2_2_x_To_indev_3_x
>   .static_columns_paging_test
> Potentially CASSANDRA-11195, which is open with no clear
> progress.  I'll follow up with those on that issue tomorrow and see if
> they agree that this is the same problem.
>
> ===
> Overall, the testing situation continues to look better. The massive
> upgrade failures seem to have subsided, so we can continue to target
> individual failures.
>
> Since the 3.9 tests are getting to a manageable level, we should focus
> on managing test failures on trunk as well. I will soon start tracking
> these failures, as well as failures on the large dtest runs, which consist
> of tests that have been segmented off due to increased cluster size.
>
> On months that we're maintaining a 3.x bugfix branch as well as trunk,
> is there any preference toward a single email or a separate email for
> each branch?  Any other feedback is welcome, as always.
>

Re: A proposal to move away from Jira-centric development

[James Carman]

On Mon, Aug 15, 2016 at 10:23 AM Jonathan Ellis  wrote:

> A long time ago, I was a proponent of keeping most development discussions
> on Jira, where tickets can be self contained and the threadless nature
> helps keep discussions from getting sidetracked.
>
> But Cassandra was a lot smaller then, and as we've grown it has become
> necessary to separate out the signal (discussions of new features and major
> changes) from the noise of routine bug reports.
>
> I propose that we take advantage of the dev list to perform that
> separation.  Major new features and architectural improvements should be
> discussed first here, then when consensus on design is achieved, moved to
> Jira for implementation and review.
>
>
+1!  I think it's important to point out here that nobody is proposing that
folks have to send an email like:

"I was thinking of naming my variable 'foo' here, what do you guys think?"

However, discussions and decisions that have an impact on Cassandra and its
direction/architecture (not an all-inclusive list here and we should use
reason to decide) should happen on the mailing list.  The idea here is
inclusiveness.  We want everyone in the community to have a chance to
contribute to these discussions.

Re: A proposal to move away from Jira-centric development

[Benedict Elliott Smith]

Unfortunately when rulebooks are consulted to shape this kind of
discussion, their ambiguity begins to show.  What does it mean for
something "to happen" on a mailing list?  It must be a loose
interpretation, because clearly many things do not "happen" on the mailing
list, such as all of the code development and commits to the codebase, as
well as an infinitude of micro decisions made by the implementor.  These
things clearly happen though.

It's also worth pointing out the *prior* rule, which presumably takes
precedence: "Let they that do the work make the decisions."  By this rule
perhaps we shouldn't even discuss on the mailing list as we may be
encroaching on their right to decide.

Now, this is all further clouded by the fact that we're quoting the Newbie
FAQs.  In other places different snappy phrases are used:

"Everything -- but *everything*-- inside the Apache world occurs *or is
reflected* in email"  (emphasis mine)
"If it isn't in my email, it didn't happen."

These are from the more official sounding "committer guide" and both
indicate commits@ receiving all JIRA comments means those comments "happen"
- although I don't know who the speaker in the second quote is, so perhaps
it has to end up in a very specific inbox.

Anyway, the point is: *let's not get into legalistic discussions when we
don't even have legalese.*

These rules are referred to as "mottos," "codes," "FAQs" - they are
guidelines, so should be interpreted with generosity.

But even if they are not, it seems the suggestion of noncompliance is a
stretch.  So let's just try to agree what the best policy is.


On 16 August 2016 at 11:44, James Carman  wrote:

> While all of these things are true, it's irrelevant.  The ASF has a clear
> policy on this (the "it didn't happen" policy).  Discussions and decisions
> about the project must be done on the mailing lists.  You may disagree with
> the policy (as many have before you) and feel free to take it up with the
> powers that be, but until that policy changes, it's what we have to adhere
> to.  The reason they chose mailing lists (IIUC) is that they are somewhat
> of a "least common denominator."
>
> I would suggest, instead of sending an email to the dev@ list saying "hey
> folks, go to JIRA and look at stuff", that we do the opposite.  Let's have
> the discussion on the mailing lists and in JIRA, we would link to the email
> threads for any supporting documentation about the ticket.
>
> On Mon, Aug 15, 2016 at 9:04 PM Eric Stevens  wrote:
>
> > There are a few strengths of discussion on the ticketing system over
> > mailing lists.  Mailing lists were fundamentally designed in the 1970's
> and
> > early 1980's, and the state of the art from a user experience perspective
> > has barely advanced since then.
> >
> > * Mailing lists tend to end up with fragmented threads for large
> > discussions, subject changes, conversation restarts, topic forks, and
> > simple etiquette errors - all of which can make it very difficult to
> locate
> > the entire discussion related to a feature.   There is no single source
> > that an interested party can study thoroughly to understand the entire
> > conversation, rather it's more of a scavenger hunt with no way to be
> > certain you've covered all the territory.  8844 for example would have
> > ended up being numerous parallel threads as people forked the
> conversation
> > to have side discussions or offer alternatives, there's no way such a
> > ticket would ever have simply been a single massive email thread with no
> > forks.
> >
> > * Mailing lists don't allow for selective subscription.  If I find a
> ticket
> > interesting, I can watch the ticket and follow along. Conversely and more
> > importantly if I find it uninteresting I don't have to wade through that
> > discussion as it progresses.  If I think I want to follow all tickets,
> that
> > should be possible too.  Likewise if I want to watch tickets that involve
> > certain components, certain milestones, certain labels, or even certain
> > contributors, I can create a subscription for such, and get emails
> > accordingly.  I can also subscribe to RSS feeds and add them to my news
> > reader if I prefer that approach better.  A tremendous amount of control
> is
> > given to the user over what they want to see, and how they want to see
> it.
> >
> > * The concern that Chris voiced about having to open a web browser to
> > participate is actually not true unless Apache's Jira install is not well
> > configured.  If you reply to an email notification from Jira it should
> > appear as a comment on the ticket.  It shouldn't exclude anyone (even
> those
> > who want to participate but somehow can't be motivated to create an
> account
> > in the ticketing system, but who _could_ be bothered to figure out the
> > arcane mailing list subscription incantation).
> >
> > * Permalinking conversations is an important capability.  It's possible
> > with a 

Re: A proposal to move away from Jira-centric development

[James Carman]

While all of these things are true, it's irrelevant.  The ASF has a clear
policy on this (the "it didn't happen" policy).  Discussions and decisions
about the project must be done on the mailing lists.  You may disagree with
the policy (as many have before you) and feel free to take it up with the
powers that be, but until that policy changes, it's what we have to adhere
to.  The reason they chose mailing lists (IIUC) is that they are somewhat
of a "least common denominator."

I would suggest, instead of sending an email to the dev@ list saying "hey
folks, go to JIRA and look at stuff", that we do the opposite.  Let's have
the discussion on the mailing lists and in JIRA, we would link to the email
threads for any supporting documentation about the ticket.

On Mon, Aug 15, 2016 at 9:04 PM Eric Stevens  wrote:

> There are a few strengths of discussion on the ticketing system over
> mailing lists.  Mailing lists were fundamentally designed in the 1970's and
> early 1980's, and the state of the art from a user experience perspective
> has barely advanced since then.
>
> * Mailing lists tend to end up with fragmented threads for large
> discussions, subject changes, conversation restarts, topic forks, and
> simple etiquette errors - all of which can make it very difficult to locate
> the entire discussion related to a feature.   There is no single source
> that an interested party can study thoroughly to understand the entire
> conversation, rather it's more of a scavenger hunt with no way to be
> certain you've covered all the territory.  8844 for example would have
> ended up being numerous parallel threads as people forked the conversation
> to have side discussions or offer alternatives, there's no way such a
> ticket would ever have simply been a single massive email thread with no
> forks.
>
> * Mailing lists don't allow for selective subscription.  If I find a ticket
> interesting, I can watch the ticket and follow along. Conversely and more
> importantly if I find it uninteresting I don't have to wade through that
> discussion as it progresses.  If I think I want to follow all tickets, that
> should be possible too.  Likewise if I want to watch tickets that involve
> certain components, certain milestones, certain labels, or even certain
> contributors, I can create a subscription for such, and get emails
> accordingly.  I can also subscribe to RSS feeds and add them to my news
> reader if I prefer that approach better.  A tremendous amount of control is
> given to the user over what they want to see, and how they want to see it.
>
> * The concern that Chris voiced about having to open a web browser to
> participate is actually not true unless Apache's Jira install is not well
> configured.  If you reply to an email notification from Jira it should
> appear as a comment on the ticket.  It shouldn't exclude anyone (even those
> who want to participate but somehow can't be motivated to create an account
> in the ticketing system, but who _could_ be bothered to figure out the
> arcane mailing list subscription incantation).
>
> * Permalinking conversations is an important capability.  It's possible
> with a mailing list, but it's nontrivial, when you want to create that
> permalink, you must first locate the discussion in the nonprimary interface
> (the online archives), which involves a lot more effort.  Historically
> we've also seen existing "permalinks" become invalidated with mailing list
> archive software is switched or upgraded.  This leads to the next point:
>
> * One of the simple but hugely valuable features of Jira is the short
> memorable ticket numbers.  Several people in this thread have mentioned
> 8844.  Those who care about that conversation know that ID by heart.  And
> in casual conversation if you want to bring someone's attention to an
> issue, you can mention it by ID without having to try to remember what the
> original thread subject was so the other participant can also hopefully
> remember and maybe locate it later.  Write the number down on a napkin and
> you _will_ find the issue, and know it's the right one, and not some
> similar but unrelated conversation.
>
> * Ticketing systems can maintain a summarized version of the conversation
> in the ticket's description as a shortcut for those who want to know the
> current state without having to read potentially months of back history to
> catch up (the event log model).  Event logs are a great way to capture
> changing state, but they're horridly inefficient if your only option is to
> start from 0 and replay the entire log, particularly when a lot of the
> contributors are as long winded as I am.
>
> On Mon, Aug 15, 2016 at 5:29 PM Jonathan Ellis  wrote:
>
> > ... but it's important to note that if we take this approach, we need to
> be
> > careful not to just summarize the conclusion of the discussion, but also
> > approaches that were examined and found to be unviable, and why.
> 

Failing tests 2016-08-15 [cassandra-3.9]

[Joel Knighton]

===
testall: 1 failure
  org.apache.cassandra.io.compress
  .CompressedRandomAccessReaderTest.testDataCorruptionDetection
New flaky failure. I've opened CASSANDRA-12465 and assigned
myself.

===
dtest: All passed!

===
novnode: All passed!

===
upgrade: 3 failures
  upgrade_tests.cql_tests
  .TestCQLNodes2RF1_Upgrade_current_3_0_x_To_indev_3_x
  .map_keys_indexing_test
CASSANDRA-12192. Tyler Hobbs as assignee. They have identified
the cause and proposed a test fix. They are also investigating a C*
change here to improve robustness.
  upgrade_tests.cql_tests
  .TestCQLNodes3RF3_Upgrade_current_3_x_To_indev_3_x
  .map_keys_indexing_test
Same as above.
  upgrade_tests.paging_test
  .TestPagingDataNodes2RF1_Upgrade_current_2_2_x_To_indev_3_x
  .static_columns_paging_test
Potentially CASSANDRA-11195, which is open with no clear
progress.  I'll follow up with those on that issue tomorrow and see if
they agree that this is the same problem.

===
Overall, the testing situation continues to look better. The massive
upgrade failures seem to have subsided, so we can continue to target
individual failures.

Since the 3.9 tests are getting to a manageable level, we should focus
on managing test failures on trunk as well. I will soon start tracking
these failures, as well as failures on the large dtest runs, which consist
of tests that have been segmented off due to increased cluster size.

On months that we're maintaining a 3.x bugfix branch as well as trunk,
is there any preference toward a single email or a separate email for
each branch?  Any other feedback is welcome, as always.