Re: Per blockng release on dtest

[Benjamin Lerer]

Regarding CASSANDRA-12620, it has been committed in the 3.0 branch at
c612cd8d7dbd24888c216ad53f974686b88dd601 and merged into 3.11. As, if I am
not mistaken, 3.11 should become the new 3.10 release, I do not think that
there is a problem.

Did I miss something Ariel?

On Tue, Jan 10, 2017 at 6:45 PM, Jeff Jirsa  wrote:

> +1
>
>
> On Tue, Jan 10, 2017 at 9:23 AM, Aleksey Yeschenko 
> wrote:
>
> > That’s a good point.
> >
> > So 3.11 after 3.10, then move on to 3.11.x further bug fix releases?
> >
> > +1 to that.
> >
> > --
> > AY
> >
> > On 10 January 2017 at 17:22:09, Michael Shuler (mich...@pbandjelly.org)
> > wrote:
> >
> > I had the same thought. 3.10 is the tick, so a 3.11 bugfix tock follows
> > the intended final fix release for closing out tick-tock. Throwing a
> > 3.10.1 out there would add more user confusion and would be the exact
> > same contents as a 3.11 release versioned package set anyway.
> >
> > --
> > Michael
> >
> > On 01/10/2017 11:18 AM, Josh McKenzie wrote:
> > > | If someone tries to upgrade 3.10 to whatever 4.0 ends up being I
> > > think they will hit the wrong answer bug. So I would advocate for
> > > having the fix brought
> > > into 3.10, but it was broken in 3.9 as well.
> > >
> > > Seems like we'd just release that as 3.10.1 (instead of 3.11) and just
> > > tell people "you can upgrade to 4.0 w/latest version of 3.10". This
> > > does violate the "even releases features, odd releases bugfix", so
> > > maybe a 3.11 as final 3.X line would help keep that consistent?
> > >
> > > I'd rather not open the can of worms of back-porting this to 3.9 as
> > > well to hold to our claim of "any 3.X can go to 4.0".
> > >
> > > On Tue, Jan 10, 2017 at 12:13 PM, Ariel Weisberg 
> > wrote:
> > >> Hi,
> > >>
> > >>
> > >>
> > >> The upgrade tests are tricky because they upgrade from an existing
> > >> release to a current release. The bug is in 3.9 and won't be fixed
> until
> > >> 3.11 because the test checks out and builds 3.9 right now. 3.10
> doesn't
> > >> include the commit that fixes the issue so it will fail after 3.10 is
> > >> released and the test is updated to check out 3.10.
> > >>
> > >>
> > >> We claim to support upgrade from any 3.x version to 4.0. If someone
> > >> tries to upgrade 3.10 to whatever 4.0 ends up being I think they will
> > >> hit the wrong answer bug. So I would advocate for having the fix
> brought
> > >> into 3.10, but it was broken in 3.9 as well.
> > >>
> > >>
> > >> Some of the tests fail because trunk complains of unreadable stables
> and
> > >> I suspect that isn't a bug it's just something that is no longer
> > >> supported due to thrift removal, but I haven't fixed those yet. Those
> > >> are probably issues with trunk or the tests.
> > >>
> > >>
> > >> Others fail for reasons I haven't triaged yet. I'm struggling with my
> > >> own issues getting the tests to run locally.
> > >>
> > >>
> > >> Ariel
> > >>
> > >>
> > >>
> > >> On Tue, Jan 10, 2017, at 11:49 AM, Nate McCall wrote:
> > >>
> > 
> > >>
> >  I concede it would be fine to do it gradually. Once the pace of
> >  issues
> >  introduced by new development is beaten by the pace at which
> >  they are
> >  addressed I think things will go well.
> > >>
> > >>>
> > >>
> > >>> So from Michael's JIRA query:
> > >>
> > >>> https://issues.apache.org/jira/browse/CASSANDRA-12617?
> > jql=project%20%3D%20CASSANDRA%20AND%20fixVersion%20%3D%203.
> > 10%20AND%20resolution%20%3D%20Unresolved
> > >>>
> > >>
> > >>> Are we good for 3.10 after we get those cleaned up?
> > >>
> > >>>
> > >>
> > >>> Ariel, you made reference to:
> > >>
> > >>> https://github.com/apache/cassandra/commit/
> > c612cd8d7dbd24888c216ad53f974686b88dd601
> > >>>
> > >>
> > >>> Do we need to re-open an issue to have this applied to 3.10 and add
> it
> > >>> to the above list?
> > >>
> > >>>
> > >>
> > 
> > >>
> >  On Tue, Jan 10, 2017, at 11:17 AM, Josh McKenzie wrote:
> > >>
> > >
> > >>
> > > Sankalp's proposal of us progressively tightening up our standards
> > > allows
> > > us to get code out the door and regain some lost momentum on
> > > the 3.10
> > > release failures and blocking, and gives us time as a community to
> > > adjust
> > > our behavior without the burden of an ever-later slipped release
> > > hanging
> > > over our heads. There's plenty of bugfixes in the 3.X line; the
> > > more time
> > > people can have to kick the tires on that code, the more things
> > > we can
> > > find
> > >>
> > > and the better future releases will be.
> > >>
> > >>>
> > >>
> > >>>
> > >>
> > >>> +1 On gradually moving to this. Dropping releases with huge change
> > >>
> > >>> lists has never gone well for us in the past.
> > >>
> > >>
> >
> >
>

Re: Per blockng release on dtest

[Jeff Jirsa]

+1


On Tue, Jan 10, 2017 at 9:23 AM, Aleksey Yeschenko 
wrote:

> That’s a good point.
>
> So 3.11 after 3.10, then move on to 3.11.x further bug fix releases?
>
> +1 to that.
>
> --
> AY
>
> On 10 January 2017 at 17:22:09, Michael Shuler (mich...@pbandjelly.org)
> wrote:
>
> I had the same thought. 3.10 is the tick, so a 3.11 bugfix tock follows
> the intended final fix release for closing out tick-tock. Throwing a
> 3.10.1 out there would add more user confusion and would be the exact
> same contents as a 3.11 release versioned package set anyway.
>
> --
> Michael
>
> On 01/10/2017 11:18 AM, Josh McKenzie wrote:
> > | If someone tries to upgrade 3.10 to whatever 4.0 ends up being I
> > think they will hit the wrong answer bug. So I would advocate for
> > having the fix brought
> > into 3.10, but it was broken in 3.9 as well.
> >
> > Seems like we'd just release that as 3.10.1 (instead of 3.11) and just
> > tell people "you can upgrade to 4.0 w/latest version of 3.10". This
> > does violate the "even releases features, odd releases bugfix", so
> > maybe a 3.11 as final 3.X line would help keep that consistent?
> >
> > I'd rather not open the can of worms of back-porting this to 3.9 as
> > well to hold to our claim of "any 3.X can go to 4.0".
> >
> > On Tue, Jan 10, 2017 at 12:13 PM, Ariel Weisberg 
> wrote:
> >> Hi,
> >>
> >>
> >>
> >> The upgrade tests are tricky because they upgrade from an existing
> >> release to a current release. The bug is in 3.9 and won't be fixed until
> >> 3.11 because the test checks out and builds 3.9 right now. 3.10 doesn't
> >> include the commit that fixes the issue so it will fail after 3.10 is
> >> released and the test is updated to check out 3.10.
> >>
> >>
> >> We claim to support upgrade from any 3.x version to 4.0. If someone
> >> tries to upgrade 3.10 to whatever 4.0 ends up being I think they will
> >> hit the wrong answer bug. So I would advocate for having the fix brought
> >> into 3.10, but it was broken in 3.9 as well.
> >>
> >>
> >> Some of the tests fail because trunk complains of unreadable stables and
> >> I suspect that isn't a bug it's just something that is no longer
> >> supported due to thrift removal, but I haven't fixed those yet. Those
> >> are probably issues with trunk or the tests.
> >>
> >>
> >> Others fail for reasons I haven't triaged yet. I'm struggling with my
> >> own issues getting the tests to run locally.
> >>
> >>
> >> Ariel
> >>
> >>
> >>
> >> On Tue, Jan 10, 2017, at 11:49 AM, Nate McCall wrote:
> >>
> 
> >>
>  I concede it would be fine to do it gradually. Once the pace of
>  issues
>  introduced by new development is beaten by the pace at which
>  they are
>  addressed I think things will go well.
> >>
> >>>
> >>
> >>> So from Michael's JIRA query:
> >>
> >>> https://issues.apache.org/jira/browse/CASSANDRA-12617?
> jql=project%20%3D%20CASSANDRA%20AND%20fixVersion%20%3D%203.
> 10%20AND%20resolution%20%3D%20Unresolved
> >>>
> >>
> >>> Are we good for 3.10 after we get those cleaned up?
> >>
> >>>
> >>
> >>> Ariel, you made reference to:
> >>
> >>> https://github.com/apache/cassandra/commit/
> c612cd8d7dbd24888c216ad53f974686b88dd601
> >>>
> >>
> >>> Do we need to re-open an issue to have this applied to 3.10 and add it
> >>> to the above list?
> >>
> >>>
> >>
> 
> >>
>  On Tue, Jan 10, 2017, at 11:17 AM, Josh McKenzie wrote:
> >>
> >
> >>
> > Sankalp's proposal of us progressively tightening up our standards
> > allows
> > us to get code out the door and regain some lost momentum on
> > the 3.10
> > release failures and blocking, and gives us time as a community to
> > adjust
> > our behavior without the burden of an ever-later slipped release
> > hanging
> > over our heads. There's plenty of bugfixes in the 3.X line; the
> > more time
> > people can have to kick the tires on that code, the more things
> > we can
> > find
> >>
> > and the better future releases will be.
> >>
> >>>
> >>
> >>>
> >>
> >>> +1 On gradually moving to this. Dropping releases with huge change
> >>
> >>> lists has never gone well for us in the past.
> >>
> >>
>
>

Re: Per blockng release on dtest

[Nate McCall]

>
> So 3.11 after 3.10, then move on to 3.11.x further bug fix releases?
>

+1

Re: Per blockng release on dtest

[Jonathan Ellis]

+1

On Tue, Jan 10, 2017 at 11:23 AM, Aleksey Yeschenko 
wrote:

> That’s a good point.
>
> So 3.11 after 3.10, then move on to 3.11.x further bug fix releases?
>
> +1 to that.
>
> --
> AY
>
> On 10 January 2017 at 17:22:09, Michael Shuler (mich...@pbandjelly.org)
> wrote:
>
> I had the same thought. 3.10 is the tick, so a 3.11 bugfix tock follows
> the intended final fix release for closing out tick-tock. Throwing a
> 3.10.1 out there would add more user confusion and would be the exact
> same contents as a 3.11 release versioned package set anyway.
>
> --
> Michael
>
> On 01/10/2017 11:18 AM, Josh McKenzie wrote:
> > | If someone tries to upgrade 3.10 to whatever 4.0 ends up being I
> > think they will hit the wrong answer bug. So I would advocate for
> > having the fix brought
> > into 3.10, but it was broken in 3.9 as well.
> >
> > Seems like we'd just release that as 3.10.1 (instead of 3.11) and just
> > tell people "you can upgrade to 4.0 w/latest version of 3.10". This
> > does violate the "even releases features, odd releases bugfix", so
> > maybe a 3.11 as final 3.X line would help keep that consistent?
> >
> > I'd rather not open the can of worms of back-porting this to 3.9 as
> > well to hold to our claim of "any 3.X can go to 4.0".
> >
> > On Tue, Jan 10, 2017 at 12:13 PM, Ariel Weisberg 
> wrote:
> >> Hi,
> >>
> >>
> >>
> >> The upgrade tests are tricky because they upgrade from an existing
> >> release to a current release. The bug is in 3.9 and won't be fixed until
> >> 3.11 because the test checks out and builds 3.9 right now. 3.10 doesn't
> >> include the commit that fixes the issue so it will fail after 3.10 is
> >> released and the test is updated to check out 3.10.
> >>
> >>
> >> We claim to support upgrade from any 3.x version to 4.0. If someone
> >> tries to upgrade 3.10 to whatever 4.0 ends up being I think they will
> >> hit the wrong answer bug. So I would advocate for having the fix brought
> >> into 3.10, but it was broken in 3.9 as well.
> >>
> >>
> >> Some of the tests fail because trunk complains of unreadable stables and
> >> I suspect that isn't a bug it's just something that is no longer
> >> supported due to thrift removal, but I haven't fixed those yet. Those
> >> are probably issues with trunk or the tests.
> >>
> >>
> >> Others fail for reasons I haven't triaged yet. I'm struggling with my
> >> own issues getting the tests to run locally.
> >>
> >>
> >> Ariel
> >>
> >>
> >>
> >> On Tue, Jan 10, 2017, at 11:49 AM, Nate McCall wrote:
> >>
> 
> >>
>  I concede it would be fine to do it gradually. Once the pace of
>  issues
>  introduced by new development is beaten by the pace at which
>  they are
>  addressed I think things will go well.
> >>
> >>>
> >>
> >>> So from Michael's JIRA query:
> >>
> >>> https://issues.apache.org/jira/browse/CASSANDRA-12617?
> jql=project%20%3D%20CASSANDRA%20AND%20fixVersion%20%3D%203.
> 10%20AND%20resolution%20%3D%20Unresolved
> >>>
> >>
> >>> Are we good for 3.10 after we get those cleaned up?
> >>
> >>>
> >>
> >>> Ariel, you made reference to:
> >>
> >>> https://github.com/apache/cassandra/commit/
> c612cd8d7dbd24888c216ad53f974686b88dd601
> >>>
> >>
> >>> Do we need to re-open an issue to have this applied to 3.10 and add it
> >>> to the above list?
> >>
> >>>
> >>
> 
> >>
>  On Tue, Jan 10, 2017, at 11:17 AM, Josh McKenzie wrote:
> >>
> >
> >>
> > Sankalp's proposal of us progressively tightening up our standards
> > allows
> > us to get code out the door and regain some lost momentum on
> > the 3.10
> > release failures and blocking, and gives us time as a community to
> > adjust
> > our behavior without the burden of an ever-later slipped release
> > hanging
> > over our heads. There's plenty of bugfixes in the 3.X line; the
> > more time
> > people can have to kick the tires on that code, the more things
> > we can
> > find
> >>
> > and the better future releases will be.
> >>
> >>>
> >>
> >>>
> >>
> >>> +1 On gradually moving to this. Dropping releases with huge change
> >>
> >>> lists has never gone well for us in the past.
> >>
> >>
>
>


-- 
Jonathan Ellis
co-founder, http://www.datastax.com
@spyced

Re: Per blockng release on dtest

[Josh McKenzie]

> So 3.11 after 3.10, then move on to 3.11.x further bug fix releases?
+1

On Tue, Jan 10, 2017 at 12:23 PM, Aleksey Yeschenko  wrote:
> That’s a good point.
>
> So 3.11 after 3.10, then move on to 3.11.x further bug fix releases?
>
> +1 to that.
>
> --
> AY
>
> On 10 January 2017 at 17:22:09, Michael Shuler (mich...@pbandjelly.org) wrote:
>
> I had the same thought. 3.10 is the tick, so a 3.11 bugfix tock follows
> the intended final fix release for closing out tick-tock. Throwing a
> 3.10.1 out there would add more user confusion and would be the exact
> same contents as a 3.11 release versioned package set anyway.
>
> --
> Michael
>
> On 01/10/2017 11:18 AM, Josh McKenzie wrote:
>> | If someone tries to upgrade 3.10 to whatever 4.0 ends up being I
>> think they will hit the wrong answer bug. So I would advocate for
>> having the fix brought
>> into 3.10, but it was broken in 3.9 as well.
>>
>> Seems like we'd just release that as 3.10.1 (instead of 3.11) and just
>> tell people "you can upgrade to 4.0 w/latest version of 3.10". This
>> does violate the "even releases features, odd releases bugfix", so
>> maybe a 3.11 as final 3.X line would help keep that consistent?
>>
>> I'd rather not open the can of worms of back-porting this to 3.9 as
>> well to hold to our claim of "any 3.X can go to 4.0".
>>
>> On Tue, Jan 10, 2017 at 12:13 PM, Ariel Weisberg  wrote:
>>> Hi,
>>>
>>>
>>>
>>> The upgrade tests are tricky because they upgrade from an existing
>>> release to a current release. The bug is in 3.9 and won't be fixed until
>>> 3.11 because the test checks out and builds 3.9 right now. 3.10 doesn't
>>> include the commit that fixes the issue so it will fail after 3.10 is
>>> released and the test is updated to check out 3.10.
>>>
>>>
>>> We claim to support upgrade from any 3.x version to 4.0. If someone
>>> tries to upgrade 3.10 to whatever 4.0 ends up being I think they will
>>> hit the wrong answer bug. So I would advocate for having the fix brought
>>> into 3.10, but it was broken in 3.9 as well.
>>>
>>>
>>> Some of the tests fail because trunk complains of unreadable stables and
>>> I suspect that isn't a bug it's just something that is no longer
>>> supported due to thrift removal, but I haven't fixed those yet. Those
>>> are probably issues with trunk or the tests.
>>>
>>>
>>> Others fail for reasons I haven't triaged yet. I'm struggling with my
>>> own issues getting the tests to run locally.
>>>
>>>
>>> Ariel
>>>
>>>
>>>
>>> On Tue, Jan 10, 2017, at 11:49 AM, Nate McCall wrote:
>>>
>
>>>
> I concede it would be fine to do it gradually. Once the pace of
> issues
> introduced by new development is beaten by the pace at which
> they are
> addressed I think things will go well.
>>>

>>>
 So from Michael's JIRA query:
>>>
 https://issues.apache.org/jira/browse/CASSANDRA-12617?jql=project%20%3D%20CASSANDRA%20AND%20fixVersion%20%3D%203.10%20AND%20resolution%20%3D%20Unresolved

>>>
 Are we good for 3.10 after we get those cleaned up?
>>>

>>>
 Ariel, you made reference to:
>>>
 https://github.com/apache/cassandra/commit/c612cd8d7dbd24888c216ad53f974686b88dd601

>>>
 Do we need to re-open an issue to have this applied to 3.10 and add it
 to the above list?
>>>

>>>
>
>>>
> On Tue, Jan 10, 2017, at 11:17 AM, Josh McKenzie wrote:
>>>
>>
>>>
>> Sankalp's proposal of us progressively tightening up our standards
>> allows
>> us to get code out the door and regain some lost momentum on
>> the 3.10
>> release failures and blocking, and gives us time as a community to
>> adjust
>> our behavior without the burden of an ever-later slipped release
>> hanging
>> over our heads. There's plenty of bugfixes in the 3.X line; the
>> more time
>> people can have to kick the tires on that code, the more things
>> we can
>> find
>>>
>> and the better future releases will be.
>>>

>>>

>>>
 +1 On gradually moving to this. Dropping releases with huge change
>>>
 lists has never gone well for us in the past.
>>>
>>>
>

Re: Per blockng release on dtest

[Aleksey Yeschenko]

That’s a good point.

So 3.11 after 3.10, then move on to 3.11.x further bug fix releases?

+1 to that.

-- 
AY

On 10 January 2017 at 17:22:09, Michael Shuler (mich...@pbandjelly.org) wrote:

I had the same thought. 3.10 is the tick, so a 3.11 bugfix tock follows  
the intended final fix release for closing out tick-tock. Throwing a  
3.10.1 out there would add more user confusion and would be the exact  
same contents as a 3.11 release versioned package set anyway.  

--  
Michael  

On 01/10/2017 11:18 AM, Josh McKenzie wrote:  
> | If someone tries to upgrade 3.10 to whatever 4.0 ends up being I  
> think they will hit the wrong answer bug. So I would advocate for  
> having the fix brought  
> into 3.10, but it was broken in 3.9 as well.  
>  
> Seems like we'd just release that as 3.10.1 (instead of 3.11) and just  
> tell people "you can upgrade to 4.0 w/latest version of 3.10". This  
> does violate the "even releases features, odd releases bugfix", so  
> maybe a 3.11 as final 3.X line would help keep that consistent?  
>  
> I'd rather not open the can of worms of back-porting this to 3.9 as  
> well to hold to our claim of "any 3.X can go to 4.0".  
>  
> On Tue, Jan 10, 2017 at 12:13 PM, Ariel Weisberg  wrote:  
>> Hi,  
>>  
>>  
>>  
>> The upgrade tests are tricky because they upgrade from an existing  
>> release to a current release. The bug is in 3.9 and won't be fixed until  
>> 3.11 because the test checks out and builds 3.9 right now. 3.10 doesn't  
>> include the commit that fixes the issue so it will fail after 3.10 is  
>> released and the test is updated to check out 3.10.  
>>  
>>  
>> We claim to support upgrade from any 3.x version to 4.0. If someone  
>> tries to upgrade 3.10 to whatever 4.0 ends up being I think they will  
>> hit the wrong answer bug. So I would advocate for having the fix brought  
>> into 3.10, but it was broken in 3.9 as well.  
>>  
>>  
>> Some of the tests fail because trunk complains of unreadable stables and  
>> I suspect that isn't a bug it's just something that is no longer  
>> supported due to thrift removal, but I haven't fixed those yet. Those  
>> are probably issues with trunk or the tests.  
>>  
>>  
>> Others fail for reasons I haven't triaged yet. I'm struggling with my  
>> own issues getting the tests to run locally.  
>>  
>>  
>> Ariel  
>>  
>>  
>>  
>> On Tue, Jan 10, 2017, at 11:49 AM, Nate McCall wrote:  
>>  
  
>>  
 I concede it would be fine to do it gradually. Once the pace of  
 issues  
 introduced by new development is beaten by the pace at which  
 they are  
 addressed I think things will go well.  
>>  
>>>  
>>  
>>> So from Michael's JIRA query:  
>>  
>>> https://issues.apache.org/jira/browse/CASSANDRA-12617?jql=project%20%3D%20CASSANDRA%20AND%20fixVersion%20%3D%203.10%20AND%20resolution%20%3D%20Unresolved
>>>   
>>>  
>>  
>>> Are we good for 3.10 after we get those cleaned up?  
>>  
>>>  
>>  
>>> Ariel, you made reference to:  
>>  
>>> https://github.com/apache/cassandra/commit/c612cd8d7dbd24888c216ad53f974686b88dd601
>>>   
>>>  
>>  
>>> Do we need to re-open an issue to have this applied to 3.10 and add it  
>>> to the above list?  
>>  
>>>  
>>  
  
>>  
 On Tue, Jan 10, 2017, at 11:17 AM, Josh McKenzie wrote:  
>>  
>  
>>  
> Sankalp's proposal of us progressively tightening up our standards  
> allows  
> us to get code out the door and regain some lost momentum on  
> the 3.10  
> release failures and blocking, and gives us time as a community to  
> adjust  
> our behavior without the burden of an ever-later slipped release  
> hanging  
> over our heads. There's plenty of bugfixes in the 3.X line; the  
> more time  
> people can have to kick the tires on that code, the more things  
> we can  
> find  
>>  
> and the better future releases will be.  
>>  
>>>  
>>  
>>>  
>>  
>>> +1 On gradually moving to this. Dropping releases with huge change  
>>  
>>> lists has never gone well for us in the past.  
>>  
>>  

Re: Per blockng release on dtest

[Michael Shuler]

I had the same thought. 3.10 is the tick, so a 3.11 bugfix tock follows
the intended final fix release for closing out tick-tock. Throwing a
3.10.1 out there would add more user confusion and would be the exact
same contents as a 3.11 release versioned package set anyway.

-- 
Michael

On 01/10/2017 11:18 AM, Josh McKenzie wrote:
> | If someone tries to upgrade 3.10 to whatever 4.0 ends up being I
> think they will hit the wrong answer bug. So I would advocate for
> having the fix brought
> into 3.10, but it was broken in 3.9 as well.
> 
> Seems like we'd just release that as 3.10.1 (instead of 3.11) and just
> tell people "you can upgrade to 4.0 w/latest version of 3.10". This
> does violate the "even releases features, odd releases bugfix", so
> maybe a 3.11 as final 3.X line would help keep that consistent?
> 
> I'd rather not open the can of worms of back-porting this to 3.9 as
> well to hold to our claim of "any 3.X can go to 4.0".
> 
> On Tue, Jan 10, 2017 at 12:13 PM, Ariel Weisberg  wrote:
>> Hi,
>>
>>
>>
>> The upgrade tests are tricky because they upgrade from an existing
>> release to a current release. The bug is in 3.9 and won't be fixed until
>> 3.11 because the test  checks out and builds 3.9 right now. 3.10 doesn't
>> include the commit that fixes the issue so it will fail after 3.10 is
>> released and the test is updated to check out 3.10.
>>
>>
>> We claim to support upgrade from any 3.x version to 4.0. If someone
>> tries to upgrade 3.10 to whatever 4.0 ends up being I think they will
>> hit the wrong answer bug. So I would advocate for having the fix brought
>> into 3.10, but it was broken in 3.9 as well.
>>
>>
>> Some of the tests fail because trunk complains of unreadable stables and
>> I suspect that isn't a bug it's just something that is no longer
>> supported due to thrift removal, but I haven't fixed those yet. Those
>> are probably issues with trunk or the tests.
>>
>>
>> Others fail for reasons I haven't triaged yet. I'm struggling with my
>> own issues getting the tests to run locally.
>>
>>
>> Ariel
>>
>>
>>
>> On Tue, Jan 10, 2017, at 11:49 AM, Nate McCall wrote:
>>

>>
 I concede it would be fine to do it gradually. Once the pace of
 issues
 introduced by new development is beaten by the pace at which
 they are
 addressed I think things will go well.
>>
>>>
>>
>>> So from Michael's JIRA query:
>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-12617?jql=project%20%3D%20CASSANDRA%20AND%20fixVersion%20%3D%203.10%20AND%20resolution%20%3D%20Unresolved
>>>
>>
>>> Are we good for 3.10 after we get those cleaned up?
>>
>>>
>>
>>> Ariel, you made reference to:
>>
>>> https://github.com/apache/cassandra/commit/c612cd8d7dbd24888c216ad53f974686b88dd601
>>>
>>
>>> Do we need to re-open an issue to have this applied to 3.10 and add it
>>> to the above list?
>>
>>>
>>

>>
 On Tue, Jan 10, 2017, at 11:17 AM, Josh McKenzie wrote:
>>
>
>>
> Sankalp's proposal of us progressively tightening up our standards
> allows
> us to get code out the door and regain some lost momentum on
> the 3.10
> release failures and blocking, and gives us time as a community to
> adjust
> our behavior without the burden of an ever-later slipped release
> hanging
> over our heads. There's plenty of bugfixes in the 3.X line; the
> more time
> people can have to kick the tires on that code, the more things
> we can
> find
>>
> and the better future releases will be.
>>
>>>
>>
>>>
>>
>>> +1 On gradually moving to this. Dropping releases with huge change
>>
>>> lists has never gone well for us in the past.
>>
>>

Re: Per blockng release on dtest

[Nate McCall]

> Seems like we'd just release that as 3.10.1 (instead of 3.11) and just
> tell people "you can upgrade to 4.0 w/latest version of 3.10". This
> does violate the "even releases features, odd releases bugfix", so
> maybe a 3.11 as final 3.X line would help keep that consistent?

This feels like a decent compromise to me.

Re: Per blockng release on dtest

[Josh McKenzie]

| If someone tries to upgrade 3.10 to whatever 4.0 ends up being I
think they will hit the wrong answer bug. So I would advocate for
having the fix brought
into 3.10, but it was broken in 3.9 as well.

Seems like we'd just release that as 3.10.1 (instead of 3.11) and just
tell people "you can upgrade to 4.0 w/latest version of 3.10". This
does violate the "even releases features, odd releases bugfix", so
maybe a 3.11 as final 3.X line would help keep that consistent?

I'd rather not open the can of worms of back-porting this to 3.9 as
well to hold to our claim of "any 3.X can go to 4.0".

On Tue, Jan 10, 2017 at 12:13 PM, Ariel Weisberg  wrote:
> Hi,
>
>
>
> The upgrade tests are tricky because they upgrade from an existing
> release to a current release. The bug is in 3.9 and won't be fixed until
> 3.11 because the test  checks out and builds 3.9 right now. 3.10 doesn't
> include the commit that fixes the issue so it will fail after 3.10 is
> released and the test is updated to check out 3.10.
>
>
> We claim to support upgrade from any 3.x version to 4.0. If someone
> tries to upgrade 3.10 to whatever 4.0 ends up being I think they will
> hit the wrong answer bug. So I would advocate for having the fix brought
> into 3.10, but it was broken in 3.9 as well.
>
>
> Some of the tests fail because trunk complains of unreadable stables and
> I suspect that isn't a bug it's just something that is no longer
> supported due to thrift removal, but I haven't fixed those yet. Those
> are probably issues with trunk or the tests.
>
>
> Others fail for reasons I haven't triaged yet. I'm struggling with my
> own issues getting the tests to run locally.
>
>
> Ariel
>
>
>
> On Tue, Jan 10, 2017, at 11:49 AM, Nate McCall wrote:
>
>> >
>
>> > I concede it would be fine to do it gradually. Once the pace of
>> > issues
>> > introduced by new development is beaten by the pace at which
>> > they are
>> > addressed I think things will go well.
>
>>
>
>> So from Michael's JIRA query:
>
>> https://issues.apache.org/jira/browse/CASSANDRA-12617?jql=project%20%3D%20CASSANDRA%20AND%20fixVersion%20%3D%203.10%20AND%20resolution%20%3D%20Unresolved
>>
>
>> Are we good for 3.10 after we get those cleaned up?
>
>>
>
>> Ariel, you made reference to:
>
>> https://github.com/apache/cassandra/commit/c612cd8d7dbd24888c216ad53f974686b88dd601
>>
>
>> Do we need to re-open an issue to have this applied to 3.10 and add it
>> to the above list?
>
>>
>
>> >
>
>> > On Tue, Jan 10, 2017, at 11:17 AM, Josh McKenzie wrote:
>
>> >>
>
>> >> Sankalp's proposal of us progressively tightening up our standards
>> >> allows
>> >> us to get code out the door and regain some lost momentum on
>> >> the 3.10
>> >> release failures and blocking, and gives us time as a community to
>> >> adjust
>> >> our behavior without the burden of an ever-later slipped release
>> >> hanging
>> >> over our heads. There's plenty of bugfixes in the 3.X line; the
>> >> more time
>> >> people can have to kick the tires on that code, the more things
>> >> we can
>> >> find
>
>> >> and the better future releases will be.
>
>>
>
>>
>
>> +1 On gradually moving to this. Dropping releases with huge change
>
>> lists has never gone well for us in the past.
>
>

Re: Per blockng release on dtest

[Ariel Weisberg]

Hi,



The upgrade tests are tricky because they upgrade from an existing
release to a current release. The bug is in 3.9 and won't be fixed until
3.11 because the test  checks out and builds 3.9 right now. 3.10 doesn't
include the commit that fixes the issue so it will fail after 3.10 is
released and the test is updated to check out 3.10.


We claim to support upgrade from any 3.x version to 4.0. If someone
tries to upgrade 3.10 to whatever 4.0 ends up being I think they will
hit the wrong answer bug. So I would advocate for having the fix brought
into 3.10, but it was broken in 3.9 as well.


Some of the tests fail because trunk complains of unreadable stables and
I suspect that isn't a bug it's just something that is no longer
supported due to thrift removal, but I haven't fixed those yet. Those
are probably issues with trunk or the tests.


Others fail for reasons I haven't triaged yet. I'm struggling with my
own issues getting the tests to run locally.


Ariel



On Tue, Jan 10, 2017, at 11:49 AM, Nate McCall wrote:

> >

> > I concede it would be fine to do it gradually. Once the pace of
> > issues
> > introduced by new development is beaten by the pace at which
> > they are
> > addressed I think things will go well.

>

> So from Michael's JIRA query:

> https://issues.apache.org/jira/browse/CASSANDRA-12617?jql=project%20%3D%20CASSANDRA%20AND%20fixVersion%20%3D%203.10%20AND%20resolution%20%3D%20Unresolved
>

> Are we good for 3.10 after we get those cleaned up?

>

> Ariel, you made reference to:

> https://github.com/apache/cassandra/commit/c612cd8d7dbd24888c216ad53f974686b88dd601
>

> Do we need to re-open an issue to have this applied to 3.10 and add it
> to the above list?

>

> >

> > On Tue, Jan 10, 2017, at 11:17 AM, Josh McKenzie wrote:

> >>

> >> Sankalp's proposal of us progressively tightening up our standards
> >> allows
> >> us to get code out the door and regain some lost momentum on
> >> the 3.10
> >> release failures and blocking, and gives us time as a community to
> >> adjust
> >> our behavior without the burden of an ever-later slipped release
> >> hanging
> >> over our heads. There's plenty of bugfixes in the 3.X line; the
> >> more time
> >> people can have to kick the tires on that code, the more things
> >> we can
> >> find

> >> and the better future releases will be.

>

>

> +1 On gradually moving to this. Dropping releases with huge change

> lists has never gone well for us in the past.

Re: Per blockng release on dtest

[Aleksey Yeschenko]

I would personally favour pushing 3.10 out without waiting for the pretty 
innocent
#13113 resolution.

With the amount of bug fixes accumulated in the 3.X branch it’s borderline
irresponsible to not release them out to the users.

-- 
AY

On 10 January 2017 at 17:05:57, Michael Shuler (mich...@pbandjelly.org) wrote:

Generally, fixver has only been set during commits - I only marked 3.10  
and blocker status to highlight the few that failed votes, in order to  
sort of cheerlead "fix me so we can release!" JIRA tickets. The full  
test-failure list is probably the more "realistic" view, since any of  
those may occur. As I also just replied, an auth_test method is the  
current failure on c-3.11 branch. Mark it as a blocker? Re-run the job  
and hope for green? Unmark the current 3.10 fixver blockers, since they  
didn't fail? (Likely to get some other failure or maybe a full pass)  

--  
Michael  

On 01/10/2017 10:56 AM, Josh McKenzie wrote:  
> I assume you meant the query w/out 12617 embedded?  
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20CASSANDRA%20AND%20fixVersion%20%3D%203.10%20AND%20resolution%20%3D%20Unresolved
>   
>  
> Do we have confidence that all test failures have fixVersion attached  
> correctly? The list of test failures w/out fixVersion is pretty daunting:  
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20CASSANDRA%20AND%20labels%20in%20(test%2C%20test-failure%2C%20dtest%2C%20unittest)%20AND%20resolution%20%3D%20Unresolved%20AND%20fixversion%20%3D%20null%20and%20labels%20!%3D%20windows%20ORDER%20BY%20created%20asc
>   
>  
> On Tue, Jan 10, 2017 at 11:49 AM, Nate McCall  wrote:  
>  
>>>  
>>> I concede it would be fine to do it gradually. Once the pace of issues  
>>> introduced by new development is beaten by the pace at which they are  
>>> addressed I think things will go well.  
>>  
>> So from Michael's JIRA query:  
>> https://issues.apache.org/jira/browse/CASSANDRA-12617?  
>> jql=project%20%3D%20CASSANDRA%20AND%20fixVersion%20%3D%203.  
>> 10%20AND%20resolution%20%3D%20Unresolved  
>>  
>> Are we good for 3.10 after we get those cleaned up?  
>>  
>> Ariel, you made reference to:  
>> https://github.com/apache/cassandra/commit/c612cd8d7dbd24888c216ad53f9746  
>> 86b88dd601  
>>  
>> Do we need to re-open an issue to have this applied to 3.10 and add it  
>> to the above list?  
>>  
>>>  
>>> On Tue, Jan 10, 2017, at 11:17 AM, Josh McKenzie wrote:  
  
 Sankalp's proposal of us progressively tightening up our standards  
>> allows  
 us to get code out the door and regain some lost momentum on the 3.10  
 release failures and blocking, and gives us time as a community to  
>> adjust  
 our behavior without the burden of an ever-later slipped release hanging  
 over our heads. There's plenty of bugfixes in the 3.X line; the more  
>> time  
 people can have to kick the tires on that code, the more things we can  
 find  
 and the better future releases will be.  
>>  
>>  
>> +1 On gradually moving to this. Dropping releases with huge change  
>> lists has never gone well for us in the past.  
>>  
>  

Re: Per blockng release on dtest

[Michael Shuler]

Generally, fixver has only been set during commits - I only marked 3.10
and blocker status to highlight the few that failed votes, in order to
sort of cheerlead "fix me so we can release!" JIRA tickets. The full
test-failure list is probably the more "realistic" view, since any of
those may occur. As I also just replied, an auth_test method is the
current failure on c-3.11 branch. Mark it as a blocker? Re-run the job
and hope for green? Unmark the current 3.10 fixver blockers, since they
didn't fail? (Likely to get some other failure or maybe a full pass)

-- 
Michael

On 01/10/2017 10:56 AM, Josh McKenzie wrote:
> I assume you meant the query w/out 12617 embedded?
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20CASSANDRA%20AND%20fixVersion%20%3D%203.10%20AND%20resolution%20%3D%20Unresolved
> 
> Do we have confidence that all test failures have fixVersion attached
> correctly? The list of test failures w/out fixVersion is pretty daunting:
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20CASSANDRA%20AND%20labels%20in%20(test%2C%20test-failure%2C%20dtest%2C%20unittest)%20AND%20resolution%20%3D%20Unresolved%20AND%20fixversion%20%3D%20null%20and%20labels%20!%3D%20windows%20ORDER%20BY%20created%20asc
> 
> On Tue, Jan 10, 2017 at 11:49 AM, Nate McCall  wrote:
> 
>>>
>>> I concede it would be fine to do it gradually. Once the pace of issues
>>> introduced by new development is beaten by the pace at which they are
>>> addressed I think things will go well.
>>
>> So from Michael's JIRA query:
>> https://issues.apache.org/jira/browse/CASSANDRA-12617?
>> jql=project%20%3D%20CASSANDRA%20AND%20fixVersion%20%3D%203.
>> 10%20AND%20resolution%20%3D%20Unresolved
>>
>> Are we good for 3.10 after we get those cleaned up?
>>
>> Ariel, you made reference to:
>> https://github.com/apache/cassandra/commit/c612cd8d7dbd24888c216ad53f9746
>> 86b88dd601
>>
>> Do we need to re-open an issue to have this applied to 3.10 and add it
>> to the above list?
>>
>>>
>>> On Tue, Jan 10, 2017, at 11:17 AM, Josh McKenzie wrote:

 Sankalp's proposal of us progressively tightening up our standards
>> allows
 us to get code out the door and regain some lost momentum on the 3.10
 release failures and blocking, and gives us time as a community to
>> adjust
 our behavior without the burden of an ever-later slipped release hanging
 over our heads. There's plenty of bugfixes in the 3.X line; the more
>> time
 people can have to kick the tires on that code, the more things we can
 find
 and the better future releases will be.
>>
>>
>> +1 On gradually moving to this. Dropping releases with huge change
>> lists has never gone well for us in the past.
>>
> 

Re: Per blockng release on dtest

[Michael Shuler]

Latest cassandra-3.11_dtest run failed on one test,
system_auth_ks_is_alterable_test:

https://issues.apache.org/jira/browse/CASSANDRA-13113

The dtest variations (novnode, offheap, upgrade, large) have other
failures, but if the green light for release is unit tests and the
default dtest, we're close.

http://cassci.datastax.com/view/cassandra-3.11/

-- 
Michael

On 01/10/2017 10:49 AM, Nate McCall wrote:
>>
>> I concede it would be fine to do it gradually. Once the pace of issues
>> introduced by new development is beaten by the pace at which they are
>> addressed I think things will go well.
> 
> So from Michael's JIRA query:
> https://issues.apache.org/jira/browse/CASSANDRA-12617?jql=project%20%3D%20CASSANDRA%20AND%20fixVersion%20%3D%203.10%20AND%20resolution%20%3D%20Unresolved
> 
> Are we good for 3.10 after we get those cleaned up?
> 
> Ariel, you made reference to:
> https://github.com/apache/cassandra/commit/c612cd8d7dbd24888c216ad53f974686b88dd601
> 
> Do we need to re-open an issue to have this applied to 3.10 and add it
> to the above list?
> 
>>
>> On Tue, Jan 10, 2017, at 11:17 AM, Josh McKenzie wrote:
>>>
>>> Sankalp's proposal of us progressively tightening up our standards allows
>>> us to get code out the door and regain some lost momentum on the 3.10
>>> release failures and blocking, and gives us time as a community to adjust
>>> our behavior without the burden of an ever-later slipped release hanging
>>> over our heads. There's plenty of bugfixes in the 3.X line; the more time
>>> people can have to kick the tires on that code, the more things we can
>>> find
>>> and the better future releases will be.
> 
> 
> +1 On gradually moving to this. Dropping releases with huge change
> lists has never gone well for us in the past.
> 

Re: Per blockng release on dtest

[Josh McKenzie]

I assume you meant the query w/out 12617 embedded?
https://issues.apache.org/jira/issues/?jql=project%20%3D%20CASSANDRA%20AND%20fixVersion%20%3D%203.10%20AND%20resolution%20%3D%20Unresolved

Do we have confidence that all test failures have fixVersion attached
correctly? The list of test failures w/out fixVersion is pretty daunting:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20CASSANDRA%20AND%20labels%20in%20(test%2C%20test-failure%2C%20dtest%2C%20unittest)%20AND%20resolution%20%3D%20Unresolved%20AND%20fixversion%20%3D%20null%20and%20labels%20!%3D%20windows%20ORDER%20BY%20created%20asc

On Tue, Jan 10, 2017 at 11:49 AM, Nate McCall  wrote:

> >
> > I concede it would be fine to do it gradually. Once the pace of issues
> > introduced by new development is beaten by the pace at which they are
> > addressed I think things will go well.
>
> So from Michael's JIRA query:
> https://issues.apache.org/jira/browse/CASSANDRA-12617?
> jql=project%20%3D%20CASSANDRA%20AND%20fixVersion%20%3D%203.
> 10%20AND%20resolution%20%3D%20Unresolved
>
> Are we good for 3.10 after we get those cleaned up?
>
> Ariel, you made reference to:
> https://github.com/apache/cassandra/commit/c612cd8d7dbd24888c216ad53f9746
> 86b88dd601
>
> Do we need to re-open an issue to have this applied to 3.10 and add it
> to the above list?
>
> >
> > On Tue, Jan 10, 2017, at 11:17 AM, Josh McKenzie wrote:
> >>
> >> Sankalp's proposal of us progressively tightening up our standards
> allows
> >> us to get code out the door and regain some lost momentum on the 3.10
> >> release failures and blocking, and gives us time as a community to
> adjust
> >> our behavior without the burden of an ever-later slipped release hanging
> >> over our heads. There's plenty of bugfixes in the 3.X line; the more
> time
> >> people can have to kick the tires on that code, the more things we can
> >> find
> >> and the better future releases will be.
>
>
> +1 On gradually moving to this. Dropping releases with huge change
> lists has never gone well for us in the past.
>

Re: Per blockng release on dtest

[Nate McCall]

>
> I concede it would be fine to do it gradually. Once the pace of issues
> introduced by new development is beaten by the pace at which they are
> addressed I think things will go well.

So from Michael's JIRA query:
https://issues.apache.org/jira/browse/CASSANDRA-12617?jql=project%20%3D%20CASSANDRA%20AND%20fixVersion%20%3D%203.10%20AND%20resolution%20%3D%20Unresolved

Are we good for 3.10 after we get those cleaned up?

Ariel, you made reference to:
https://github.com/apache/cassandra/commit/c612cd8d7dbd24888c216ad53f974686b88dd601

Do we need to re-open an issue to have this applied to 3.10 and add it
to the above list?

>
> On Tue, Jan 10, 2017, at 11:17 AM, Josh McKenzie wrote:
>>
>> Sankalp's proposal of us progressively tightening up our standards allows
>> us to get code out the door and regain some lost momentum on the 3.10
>> release failures and blocking, and gives us time as a community to adjust
>> our behavior without the burden of an ever-later slipped release hanging
>> over our heads. There's plenty of bugfixes in the 3.X line; the more time
>> people can have to kick the tires on that code, the more things we can
>> find
>> and the better future releases will be.


+1 On gradually moving to this. Dropping releases with huge change
lists has never gone well for us in the past.

Re: Per blockng release on dtest

[Ariel Weisberg]

Hi,

I concede it would be fine to do it gradually. Once the pace of issues
introduced by new development is beaten by the pace at which they are
addressed I think things will go well.

Ariel

On Tue, Jan 10, 2017, at 11:17 AM, Josh McKenzie wrote:
> @ariel: you're letting the perfect be the enemy of the good here. We (as
> a
> project) have been releasing with a smattering of test failures and
> upgrade
> edge-cases back into perpetuity. While that doesn't make it ideal or
> justify continuing the behavior, getting a green testall + dtest for 3.10
> is a strong incremental improvement. Integrating other tests in the
> "block
> if not green" on subsequent releases is likewise an improvement.
> 
> I strongly advocate for incremental change in expectations of the
> community's behavior rather than a black-and-white, "this has to be
> perfect
> or we block" mentality.
> 
> Sankalp's proposal of us progressively tightening up our standards allows
> us to get code out the door and regain some lost momentum on the 3.10
> release failures and blocking, and gives us time as a community to adjust
> our behavior without the burden of an ever-later slipped release hanging
> over our heads. There's plenty of bugfixes in the 3.X line; the more time
> people can have to kick the tires on that code, the more things we can
> find
> and the better future releases will be.
> 
> 
> 
> 
> 
> On Tue, Jan 10, 2017 at 10:33 AM, Ariel Weisberg 
> wrote:
> 
> > Hi,
> >
> > At least some of those failures are real. I don't think we should
> > release 3.10 until the real failures are addressed. As I said earlier
> > one of them is a wrong answer bug that is not going to be fixed in 3.10.
> >
> > Can we just ignore failures because we think they don't mean anything?
> > Who is going to check which of the 60 failures is real?
> >
> > These tests were passing just fine at the beginning of December and then
> > commits happened and now the tests are failing. That is exactly what
> > their for. They are good tests. I don't think it matters if the failures
> > are "real" today because those are valid tests and they don't test
> > anything if they fail for spurious reasons. They are a critical part of
> > the Cassandra infrastructure as much as the storage engine or network
> > code.
> >
> > In my opinion the tests need to be fixed and people need to fix them as
> > they break them and we need to figure out how to get from people
> > breaking them and it going unnoticed to they break it and then fix it in
> > a time frame that fits the release schedule.
> >
> > My personal opinion is that releases are a reward for finishing the job.
> > Releasing without finishing the job creates the wrong incentive
> > structure for the community. If you break something you are no longer
> > the person that blocked the release you are just one of several people
> > breaking things without consequence.
> >
> > I think that rapid feedback and triaging combined with releases blocked
> > by the stuff individual contributors have broken is the way to more
> > consistent releases both schedule wise and quality wise.
> >
> > Regarding delaying 3.10? Who exactly is the consumer that is chomping at
> > the bit to get another release? One that doesn't reliably upgrade from a
> > previous version?
> >
> > Ariel
> >
> > On Tue, Jan 10, 2017, at 08:13 AM, Josh McKenzie wrote:
> > > First, I think we need to clarify if we're blocking on just testall +
> > > dtest
> > > or blocking on *all test jobs*.
> > >
> > > If the latter, upgrade tests are the elephant in the room:
> > > http://cassci.datastax.com/view/cassandra-3.11/job/
> > cassandra-3.11_dtest_upgrade/lastCompletedBuild/testReport/
> > >
> > > Do we have confidence that the reported failures are all test problems
> > > and
> > > not w/Cassandra itself? If so, is that documented somewhere?
> > >
> > > On Mon, Jan 9, 2017 at 7:33 PM, Nate McCall  wrote:
> > >
> > > > I'm not sure I understand the culmination of the past couple of
> > threads on
> > > > this.
> > > >
> > > > With a situation like:
> > > > http://cassci.datastax.com/view/cassandra-3.11/job/
> > cassandra-3.11_dtest/
> > > > lastCompletedBuild/testReport/
> > > >
> > > > We have some sense of stability on what might be flaky tests(?).
> > > > Again, I'm not sure what our criteria is specifically.
> > > >
> > > > Basically, it feels like we are in a stalemate right now. How do we
> > > > move forward?
> > > >
> > > > -Nate
> > > >
> >

Re: Per blockng release on dtest

[Aleksey Yeschenko]

If they aren’t regressions from 3.9, we should still push 3.10 out.

The branch has accumulated a lot of fixes, for problems that *are* real.
Just have a look at CHANGES.txt.

By holding 3.10 you are denying those (arguably few, but still) users fixes for 
bugs that we
know are in.

It’s been more than 3 months now, delaying it further is unreasonable. The 
branch needs to be uncorked.

I would also prefer that people who -1, in particular bindingly, were prepared 
to go and fix the offending tests,
if they are blocking the vote on the ground of tests. Can’t expect test 
failures to magically go away all
by themselves.

-- 
AY

On 10 January 2017 at 15:33:45, Ariel Weisberg (ar...@weisberg.ws) wrote:

Hi,  

At least some of those failures are real. I don't think we should  
release 3.10 until the real failures are addressed. As I said earlier  
one of them is a wrong answer bug that is not going to be fixed in 3.10.  

Can we just ignore failures because we think they don't mean anything?  
Who is going to check which of the 60 failures is real?  

These tests were passing just fine at the beginning of December and then  
commits happened and now the tests are failing. That is exactly what  
their for. They are good tests. I don't think it matters if the failures  
are "real" today because those are valid tests and they don't test  
anything if they fail for spurious reasons. They are a critical part of  
the Cassandra infrastructure as much as the storage engine or network  
code.  

In my opinion the tests need to be fixed and people need to fix them as  
they break them and we need to figure out how to get from people  
breaking them and it going unnoticed to they break it and then fix it in  
a time frame that fits the release schedule.  

My personal opinion is that releases are a reward for finishing the job.  
Releasing without finishing the job creates the wrong incentive  
structure for the community. If you break something you are no longer  
the person that blocked the release you are just one of several people  
breaking things without consequence.  

I think that rapid feedback and triaging combined with releases blocked  
by the stuff individual contributors have broken is the way to more  
consistent releases both schedule wise and quality wise.  

Regarding delaying 3.10? Who exactly is the consumer that is chomping at  
the bit to get another release? One that doesn't reliably upgrade from a  
previous version?  

Ariel  

On Tue, Jan 10, 2017, at 08:13 AM, Josh McKenzie wrote:  
> First, I think we need to clarify if we're blocking on just testall +  
> dtest  
> or blocking on *all test jobs*.  
>  
> If the latter, upgrade tests are the elephant in the room:  
> http://cassci.datastax.com/view/cassandra-3.11/job/cassandra-3.11_dtest_upgrade/lastCompletedBuild/testReport/
>   
>  
> Do we have confidence that the reported failures are all test problems  
> and  
> not w/Cassandra itself? If so, is that documented somewhere?  
>  
> On Mon, Jan 9, 2017 at 7:33 PM, Nate McCall  wrote:  
>  
> > I'm not sure I understand the culmination of the past couple of threads on  
> > this.  
> >  
> > With a situation like:  
> > http://cassci.datastax.com/view/cassandra-3.11/job/cassandra-3.11_dtest/  
> > lastCompletedBuild/testReport/  
> >  
> > We have some sense of stability on what might be flaky tests(?).  
> > Again, I'm not sure what our criteria is specifically.  
> >  
> > Basically, it feels like we are in a stalemate right now. How do we  
> > move forward?  
> >  
> > -Nate  
> >  

Re: Per blockng release on dtest

[Josh McKenzie]

@ariel: you're letting the perfect be the enemy of the good here. We (as a
project) have been releasing with a smattering of test failures and upgrade
edge-cases back into perpetuity. While that doesn't make it ideal or
justify continuing the behavior, getting a green testall + dtest for 3.10
is a strong incremental improvement. Integrating other tests in the "block
if not green" on subsequent releases is likewise an improvement.

I strongly advocate for incremental change in expectations of the
community's behavior rather than a black-and-white, "this has to be perfect
or we block" mentality.

Sankalp's proposal of us progressively tightening up our standards allows
us to get code out the door and regain some lost momentum on the 3.10
release failures and blocking, and gives us time as a community to adjust
our behavior without the burden of an ever-later slipped release hanging
over our heads. There's plenty of bugfixes in the 3.X line; the more time
people can have to kick the tires on that code, the more things we can find
and the better future releases will be.





On Tue, Jan 10, 2017 at 10:33 AM, Ariel Weisberg  wrote:

> Hi,
>
> At least some of those failures are real. I don't think we should
> release 3.10 until the real failures are addressed. As I said earlier
> one of them is a wrong answer bug that is not going to be fixed in 3.10.
>
> Can we just ignore failures because we think they don't mean anything?
> Who is going to check which of the 60 failures is real?
>
> These tests were passing just fine at the beginning of December and then
> commits happened and now the tests are failing. That is exactly what
> their for. They are good tests. I don't think it matters if the failures
> are "real" today because those are valid tests and they don't test
> anything if they fail for spurious reasons. They are a critical part of
> the Cassandra infrastructure as much as the storage engine or network
> code.
>
> In my opinion the tests need to be fixed and people need to fix them as
> they break them and we need to figure out how to get from people
> breaking them and it going unnoticed to they break it and then fix it in
> a time frame that fits the release schedule.
>
> My personal opinion is that releases are a reward for finishing the job.
> Releasing without finishing the job creates the wrong incentive
> structure for the community. If you break something you are no longer
> the person that blocked the release you are just one of several people
> breaking things without consequence.
>
> I think that rapid feedback and triaging combined with releases blocked
> by the stuff individual contributors have broken is the way to more
> consistent releases both schedule wise and quality wise.
>
> Regarding delaying 3.10? Who exactly is the consumer that is chomping at
> the bit to get another release? One that doesn't reliably upgrade from a
> previous version?
>
> Ariel
>
> On Tue, Jan 10, 2017, at 08:13 AM, Josh McKenzie wrote:
> > First, I think we need to clarify if we're blocking on just testall +
> > dtest
> > or blocking on *all test jobs*.
> >
> > If the latter, upgrade tests are the elephant in the room:
> > http://cassci.datastax.com/view/cassandra-3.11/job/
> cassandra-3.11_dtest_upgrade/lastCompletedBuild/testReport/
> >
> > Do we have confidence that the reported failures are all test problems
> > and
> > not w/Cassandra itself? If so, is that documented somewhere?
> >
> > On Mon, Jan 9, 2017 at 7:33 PM, Nate McCall  wrote:
> >
> > > I'm not sure I understand the culmination of the past couple of
> threads on
> > > this.
> > >
> > > With a situation like:
> > > http://cassci.datastax.com/view/cassandra-3.11/job/
> cassandra-3.11_dtest/
> > > lastCompletedBuild/testReport/
> > >
> > > We have some sense of stability on what might be flaky tests(?).
> > > Again, I'm not sure what our criteria is specifically.
> > >
> > > Basically, it feels like we are in a stalemate right now. How do we
> > > move forward?
> > >
> > > -Nate
> > >
>

Re: Per blockng release on dtest

[Ariel Weisberg]

Hi,

At least some of those failures are real. I don't think we should
release 3.10 until the real failures are addressed. As I said earlier
one of them is a wrong answer bug that is not going to be fixed in 3.10.

Can we just ignore failures because we think they don't mean anything?
Who is going to check which of the 60 failures is real?

These tests were passing just fine at the beginning of December and then
commits happened and now the tests are failing. That is exactly what
their for. They are good tests. I don't think it matters if the failures
are "real" today because those are valid tests and they don't test
anything if they fail for spurious reasons. They are a critical part of
the Cassandra infrastructure as much as the storage engine or network
code.

In my opinion the tests need to be fixed and people need to fix them as
they break them and we need to figure out how to get from people
breaking them and it going unnoticed to they break it and then fix it in
a time frame that fits the release schedule.

My personal opinion is that releases are a reward for finishing the job.
Releasing without finishing the job creates the wrong incentive
structure for the community. If you break something you are no longer
the person that blocked the release you are just one of several people
breaking things without consequence.

I think that rapid feedback and triaging combined with releases blocked
by the stuff individual contributors have broken is the way to more
consistent releases both schedule wise and quality wise.

Regarding delaying 3.10? Who exactly is the consumer that is chomping at
the bit to get another release? One that doesn't reliably upgrade from a
previous version?
 
Ariel

On Tue, Jan 10, 2017, at 08:13 AM, Josh McKenzie wrote:
> First, I think we need to clarify if we're blocking on just testall +
> dtest
> or blocking on *all test jobs*.
> 
> If the latter, upgrade tests are the elephant in the room:
> http://cassci.datastax.com/view/cassandra-3.11/job/cassandra-3.11_dtest_upgrade/lastCompletedBuild/testReport/
> 
> Do we have confidence that the reported failures are all test problems
> and
> not w/Cassandra itself? If so, is that documented somewhere?
> 
> On Mon, Jan 9, 2017 at 7:33 PM, Nate McCall  wrote:
> 
> > I'm not sure I understand the culmination of the past couple of threads on
> > this.
> >
> > With a situation like:
> > http://cassci.datastax.com/view/cassandra-3.11/job/cassandra-3.11_dtest/
> > lastCompletedBuild/testReport/
> >
> > We have some sense of stability on what might be flaky tests(?).
> > Again, I'm not sure what our criteria is specifically.
> >
> > Basically, it feels like we are in a stalemate right now. How do we
> > move forward?
> >
> > -Nate
> >

Re: Per blockng release on dtest

[sankalp kohli]

I think we should start with blocking 3.10 releases on testall + Dtest.
After 3.10, we can start blocking it on other jobs for each release after
that. This will make sure we make progress and dont cause 3.10 to sit for a
long time. Thoughts?

On Tue, Jan 10, 2017 at 5:13 AM, Josh McKenzie  wrote:

> First, I think we need to clarify if we're blocking on just testall + dtest
> or blocking on *all test jobs*.
>
> If the latter, upgrade tests are the elephant in the room:
> http://cassci.datastax.com/view/cassandra-3.11/job/
> cassandra-3.11_dtest_upgrade/lastCompletedBuild/testReport/
>
> Do we have confidence that the reported failures are all test problems and
> not w/Cassandra itself? If so, is that documented somewhere?
>
> On Mon, Jan 9, 2017 at 7:33 PM, Nate McCall  wrote:
>
> > I'm not sure I understand the culmination of the past couple of threads
> on
> > this.
> >
> > With a situation like:
> > http://cassci.datastax.com/view/cassandra-3.11/job/cassandra-3.11_dtest/
> > lastCompletedBuild/testReport/
> >
> > We have some sense of stability on what might be flaky tests(?).
> > Again, I'm not sure what our criteria is specifically.
> >
> > Basically, it feels like we are in a stalemate right now. How do we
> > move forward?
> >
> > -Nate
> >
>

Re: Per blockng release on dtest

[Josh McKenzie]

First, I think we need to clarify if we're blocking on just testall + dtest
or blocking on *all test jobs*.

If the latter, upgrade tests are the elephant in the room:
http://cassci.datastax.com/view/cassandra-3.11/job/cassandra-3.11_dtest_upgrade/lastCompletedBuild/testReport/

Do we have confidence that the reported failures are all test problems and
not w/Cassandra itself? If so, is that documented somewhere?

On Mon, Jan 9, 2017 at 7:33 PM, Nate McCall  wrote:

> I'm not sure I understand the culmination of the past couple of threads on
> this.
>
> With a situation like:
> http://cassci.datastax.com/view/cassandra-3.11/job/cassandra-3.11_dtest/
> lastCompletedBuild/testReport/
>
> We have some sense of stability on what might be flaky tests(?).
> Again, I'm not sure what our criteria is specifically.
>
> Basically, it feels like we are in a stalemate right now. How do we
> move forward?
>
> -Nate
>