Re: [Twisted-Python] List of required builds before a merge

[Jean-Paul Calderone]

On Tue, Mar 13, 2018 at 9:36 AM, Adi Roiban  wrote:

> We can try Circle-CI.
> They don't offer any free versions. Even for open source.
> When I did the initial work for Twisted with Travis and Appveyor I
> contacted Circle-CI to see if we can get a discount.
>
> They offered OSX - Seed plan which comes with 500 minutes/month - free.
>

They advertise this on their site now:
https://circleci.com/pricing/#faq-section-os-x

So I would expect Twisted could get it.

Your point about the minutes limit is a good one though.  What is 500
minutes to Twisted?  Around 30 builds?  It seems likely Twisted would run
out most months.  Certainly, maintaining separate coverage and non-coverage
builds would totally blow this limit.  And one of the big advantages of
hosted CI over buildbot - letting third-party PRs trigger builds - would
seem to be out of the question.


> I stopped as the people on IRC told me that Circle-CI is not better than
> Travis.
>

Presumably at some point this was true.  These days, in my experience,
CircleCI definitely seems competitive.  It has near feature parity (and
more features in some areas), a less crufty configuration system, and seems
better resourced (but this may just be better resource management - eg, the
500 minute macOS limit for free open source users).

Of course, there's also the fact that the macOS Buildbot slaves are
*not* offline
right now.  I know they've gone offline but as far as I know, their
maintainer is fairly responsive to reports of this.  So, is macOS really
part of this problem?

The immediate problem is all of the Rackspace-hosted slaves that are gone,
right?

So maybe what's more interesting is that CircleCI offers a way to replace
those - with either VM or container (including custom-image) builders.

Jean-Paul


>
> I am happy to try again with Circle-CI
>
> --
>
> We might go over 500 minutes. I suggest running the tests in stages.
>
> Run twistedchecker/pyflakes/newsfragment/Ubuntu tests first.
> Only when they all pass we should trigger Windows and OSX tests.
>
> I am also running the tests on stage... For example, Debian/RHEL/SUSE
> pass 99.99% if Ubuntu pass... so those tests are executed only later
> in the stage.
>
> --
>
>
> I don't have much time to contribute to Twisted infrastructure, and I
> would like to spend the available time doing reviews and helping
> people contribute to Twisted.
>
> If there is a better plan, I am happy to go with that.
>
> Thanks for your time :)
> Adi
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] List of required builds before a merge

[Glyph]

On Mar 12, 2018, at 10:27 PM, Amber Brown  wrote:
> 
> The buildbots went after someone said that the RAX hosting was going away, 
> and I (and a few others) didn't get the (annoyingly quiet) correction that it 
> was only for new projects (which was not how the original was written). All 
> the twisted list got was the original letter from the SFC without a "false 
> alarm" followup. By the time people pointed it out, it was too late.
> 
> I have the ansible configs to rebuild them all, but unfortunately, Life has 
> not stopped since January and hasn't got worse. If anyone wants to take a 
> stab, the ansible configs are in the twisted-infra repo.

Thanks for the explanation; I was tearing my hair out (what little remains, 
anyway) trying to figure out what the heck had happened! :)  I do remember that 
ill-fated email.  Normally I'd say we should never deprovision infrastructure 
until it's torn, bloody, from our lifeless hands, but the way the free 
Rackspace account is set up means overages go to bill the SFC and deplete their 
general fund, so I can see not wanting to have anything unnecessary hanging 
around there if the discount were to end.

(For those of you that may not have been informed at the time: 
https://www.theregister.co.uk/2017/10/20/rackspace_ends_discount_hosting_for_open_source_projects/
 

 )

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] List of required builds before a merge

[Amber Brown]

The buildbots went after someone said that the RAX hosting was going away,
and I (and a few others) didn't get the (annoyingly quiet) correction that
it was only for new projects (which was not how the original was written).
All the twisted list got was the original letter from the SFC without a
"false alarm" followup. By the time people pointed it out, it was too late.

I have the ansible configs to rebuild them all, but unfortunately, Life has
not stopped since January and hasn't got worse. If anyone wants to take a
stab, the ansible configs are in the twisted-infra repo.

- Amber

On 13 Mar. 2018 16:08, "Glyph"  wrote:

>
>
> On Mar 12, 2018, at 4:59 AM, Adi Roiban  wrote:
>
> Hi,
>
> It is not clear to me what builders need to pass before we can merge
> something.
>
> I expect that all supported "platforms" need to pass, but it is not
> clear what are the currently supported platforms.
>
> We have this info in the wiki but it does not help.
> https://twistedmatrix.com/trac/wiki/ReviewProcess#Authors:
> Howtomergethechangetotrunk
>
> In GitHub I can see Travis / Appveyor and OSX from Buildot as "Required"
>
>
> These are marked as "required" because they're necessary, but not
> sufficient.
>
> Is that all?
>
>
> Ideally all the supported buildbots should be passing.  It's a real shame
> that the offline buildbots do not report any status, because it makes it
> very easy to miss them.  (I personally did not realize the ramifications of
> the way buildbot repots status until I looked at https://buildbot.
> twistedmatrix.com/boxes-all?branch=trunk_builds=10 just now).
>
> If I check the "supported" group in Buildbot, I see many more builders.
> The problem is that a significant number of slaves are down and those
> builders are not available.
>
>
> So, normally I'd say, like Jean-Paul did, that we should just get in touch
> with the maintainers of the buildbots in question.
>
> But it seems the buildbots in question were the ones we had running on our
> donated Rackspace Cloud account.
>
> Logging into the control panel for that account, literally all the servers
> except for the buildmaster (i.e. buildbot.twistedmatrix.com) have been
> deleted.  Not just shut down, but, completely gone.  This is baffling to
> me.  I do not know who could have done this or why.  There does not appear
> to be an audit log I can consult.  Based on billing data, and consistent
> with the buildbot logs, it appears that this occurred some time in early
> January.
>
> Is Fedora still supported and required?
>
>
> That's the hope.  Those buildbots appear to be online.
>
> I suggest to use GitHub "Required" marker to document what platforms are
> supported.
>
>
> I want to agree with you.  However, our tests are not reliable or
> performant enough for this.
>
> The "required" marker makes it *impossible *to merge changes without a
> passing status or an administrator override.  This has an unfortunate set
> of corollaries.  Assuming a non-administrator reviewer:
>
>
>1. If a single builder has a temporary configuration issue and you're
>not an administrator, you can't merge any code.
>2. Let's say the probably of an intermittent test failing is 50 to 1.
>A 2% chance.  The probability of a test suite passing is 98%.  We have 36
>supported builders.  The probability of all the builders passing for a
>successful run is then just *13%*; roughly 1 in 10 valid branches will
>be able to land. (I think our probability is actually quite a bit better
>than this these days, but you get my drift.)
>3. Even if a contributor can force all the builds to re-run (which
>requires special permissions, and thus needs to wait for a project member)
>getting a successful run on every builder could require 2 or 3 tries, which
>could be 2 or 3 *hours* of waiting just to get one successful run on a
>platform that you know is not relevant to the change you're testing.
>
>
> Therefore keeping a small core set of "most pass" statuses and allowing
> for some human judgement about the rest is a practical necessity given the
> level of compute resources available to us.
>
> We don't have time to maintain the infrastructure, so I suggest to
> drop support for anything that is not supported by Travis and
> Appveyor.
>
>
> My preference would be to simply drop all the buildbots which have been
> (for some reason) destroyed from the supported build matrix, since the
> buildbots are still covering a multiplicity of kernels and environments
> that travis and appveyor aren't.  But, I don't have the time to do much
> more than write this email, so if we have no other volunteers for
> maintenance, I will support your decision to tear down the buildbots for
> now.
>
> Jean-Paul recently pointed out that CircleCI has much more performant
> macOS builds than Travis, so if someone were motivated to make that change
> but didn't want to keep maintaining hardware, that might be one 

Re: [Twisted-Python] List of required builds before a merge

[Glyph]

> On Mar 12, 2018, at 4:59 AM, Adi Roiban  wrote:
> 
> Hi,
> 
> It is not clear to me what builders need to pass before we can merge 
> something.
> 
> I expect that all supported "platforms" need to pass, but it is not
> clear what are the currently supported platforms.
> 
> We have this info in the wiki but it does not help.
> https://twistedmatrix.com/trac/wiki/ReviewProcess#Authors:Howtomergethechangetotrunk
> 
> In GitHub I can see Travis / Appveyor and OSX from Buildot as "Required"

These are marked as "required" because they're necessary, but not sufficient.

> Is that all?

Ideally all the supported buildbots should be passing.  It's a real shame that 
the offline buildbots do not report any status, because it makes it very easy 
to miss them.  (I personally did not realize the ramifications of the way 
buildbot repots status until I looked at 
https://buildbot.twistedmatrix.com/boxes-all?branch=trunk_builds=10 
 just 
now).

> If I check the "supported" group in Buildbot, I see many more builders.
> The problem is that a significant number of slaves are down and those
> builders are not available.

So, normally I'd say, like Jean-Paul did, that we should just get in touch with 
the maintainers of the buildbots in question.

But it seems the buildbots in question were the ones we had running on our 
donated Rackspace Cloud account.

Logging into the control panel for that account, literally all the servers 
except for the buildmaster (i.e. buildbot.twistedmatrix.com 
) have been deleted.  Not just shut down, 
but, completely gone.  This is baffling to me.  I do not know who could have 
done this or why.  There does not appear to be an audit log I can consult.  
Based on billing data, and consistent with the buildbot logs, it appears that 
this occurred some time in early January.

> Is Fedora still supported and required?

That's the hope.  Those buildbots appear to be online.

> I suggest to use GitHub "Required" marker to document what platforms are 
> supported.

I want to agree with you.  However, our tests are not reliable or performant 
enough for this.

The "required" marker makes it impossible to merge changes without a passing 
status or an administrator override.  This has an unfortunate set of 
corollaries.  Assuming a non-administrator reviewer:

If a single builder has a temporary configuration issue and you're not an 
administrator, you can't merge any code.
Let's say the probably of an intermittent test failing is 50 to 1.  A 2% 
chance.  The probability of a test suite passing is 98%.  We have 36 supported 
builders.  The probability of all the builders passing for a successful run is 
then just 13%; roughly 1 in 10 valid branches will be able to land. (I think 
our probability is actually quite a bit better than this these days, but you 
get my drift.)
Even if a contributor can force all the builds to re-run (which requires 
special permissions, and thus needs to wait for a project member) getting a 
successful run on every builder could require 2 or 3 tries, which could be 2 or 
3 hours of waiting just to get one successful run on a platform that you know 
is not relevant to the change you're testing.

Therefore keeping a small core set of "most pass" statuses and allowing for 
some human judgement about the rest is a practical necessity given the level of 
compute resources available to us.

> We don't have time to maintain the infrastructure, so I suggest to
> drop support for anything that is not supported by Travis and
> Appveyor.

My preference would be to simply drop all the buildbots which have been (for 
some reason) destroyed from the supported build matrix, since the buildbots are 
still covering a multiplicity of kernels and environments that travis and 
appveyor aren't.  But, I don't have the time to do much more than write this 
email, so if we have no other volunteers for maintenance, I will support your 
decision to tear down the buildbots for now.

Jean-Paul recently pointed out that CircleCI has much more performant macOS 
builds than Travis, so if someone were motivated to make that change but didn't 
want to keep maintaining hardware, that might be one way to go.

> I know that this might be disruptive.
> I think that we need it in order to raise awareness that supporting a
> platform is not easy.

I do hope that this will provoke some potential volunteers to come forward to 
help maintain our failing infrastructure.

> If someone (including me) cares about a platform they should find a way to 
> help to project supporting that platform.

> What do you think?

I do hope that if you're going to make a change, you'll consider something 
slightly less drastic than blowing up the buildbots entirely :).  But with a 
dozen servers having just disappeared with no explanation, it's a course of 
action which at least makes sense.

-g

Re: [Twisted-Python] List of required builds before a merge

[Jean-Paul Calderone]

On Mon, Mar 12, 2018 at 7:59 AM, Adi Roiban  wrote:

> Hi,
>
> It is not clear to me what builders need to pass before we can merge
> something.
>
> I expect that all supported "platforms" need to pass, but it is not
> clear what are the currently supported platforms.
>
> We have this info in the wiki but it does not help.
> https://twistedmatrix.com/trac/wiki/ReviewProcess#Authors:
> Howtomergethechangetotrunk
>
> In GitHub I can see Travis / Appveyor and OSX from Buildot as "Required"
>
> Is that all?
>
> --
>
> If I check the "supported" group in Buildbot, I see many more builders.
> The problem is that a significant number of slaves are down and those
> builders are not available.
>
> 
>
> Is Fedora still supported and required?
>
> ---
>
> I suggest to use GitHub "Required" marker to document what platforms
> are supported.
>
> We don't have time to maintain the infrastructure, so I suggest to
> drop support for anything that is not supported by Travis and
> Appveyor.
>

It would help to have a list of what coverage this would remove.  What
platforms are only covered by Travis and Appveyor?   What tests are only
run there?  What platforms are only covered by Buildbot?  What tests are
only run there?

Without this information, it's not really possible to make an informed
decision.  No user cares about whether we drop buildbot.  Some user might
care if we, for example, drop HTTP support.


>
> I know that this might be disruptive.
> I think that we need it in order to raise awareness that supporting a
> platform is not easy.
> If someone (including me) cares about a platform they should find a
> way to help to project supporting that platform.
>
>
Note that some people cared about some platforms and they found a way to
help in donating a buildslave.  Do the operators of the offline slaves
*know* that the slaves are offline?  Maybe all that's missing is some
notification to the operators when their slave goes away.  If that's all,
jumping straight to "throw away all of buildbot" seems like an overreaction.

Jean-Paul



> What do you think?
> --
> Adi Roiban
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] List of required builds before a merge

[Adi Roiban]

Hi,

It is not clear to me what builders need to pass before we can merge something.

I expect that all supported "platforms" need to pass, but it is not
clear what are the currently supported platforms.

We have this info in the wiki but it does not help.
https://twistedmatrix.com/trac/wiki/ReviewProcess#Authors:Howtomergethechangetotrunk

In GitHub I can see Travis / Appveyor and OSX from Buildot as "Required"

Is that all?

--

If I check the "supported" group in Buildbot, I see many more builders.
The problem is that a significant number of slaves are down and those
builders are not available.



Is Fedora still supported and required?

---

I suggest to use GitHub "Required" marker to document what platforms
are supported.

We don't have time to maintain the infrastructure, so I suggest to
drop support for anything that is not supported by Travis and
Appveyor.

I know that this might be disruptive.
I think that we need it in order to raise awareness that supporting a
platform is not easy.
If someone (including me) cares about a platform they should find a
way to help to project supporting that platform.

What do you think?
-- 
Adi Roiban

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python