> On Feb 19, 2017, at 7:01 PM, Steve Dower <steve.do...@python.org> wrote: > > On 19Feb2017 1554, Donald Stufft wrote: >> We don’t actually have Windows tests being run at all on pull requests >> (largely because AppVeyor took too long to run), though we obviously >> still have the buildbot fleet running post commit. This is something we >> should ideally rectify though! >> >> On the Travis builders, we skip running the unit tests all together if >> the PR was for something that couldn’t possibly change the outcome. I’m >> not familiar enough with the capabilities of the Python test runner to >> know if it can mark tests somehow to be excluded by default and only run >> if a certain flag is ran, if so it shouldn’t be very hard to make >> something that only runs those tests on changes to PCBuilder/* or >> tools/msi/*. Otherwise it might need a separate set of tests for it that >> can be independently ran. > > I can easily get some VSTS (*.visualstudio.com) test runners going with VMs > on Azure. I'm just not quite sure how to feed the results back into the > github status checks right now, or how to make them publicly maintainable > (that is, they'd be very much owned and only accessible by me, and likely > only as long as I'm employed by Microsoft, which isn't great for > sustainability - though of course anyone else can also set up the build > definition and their own pool of VMs). > > No promises on dates, but this is something that I'll likely do regardless. I > want the Windows tests as badly as anyone :) > > Cheers, > Steve
Getting compute power is part of the problem, the other problem is something to control those compute nodes and interact with GitHub. The main piece of software I know in this space is buildbot and Jenkins. We obviously have a buildbot fleet already (and re-using that would mean less maintenance in general) I don’t know what the pre-merge workflow for buildbot looks like though. I do know that Jenkins is an option, they have a plugin for handling it (although it’s not as nice as travis). It requires whitelisting people whose PRs are allowed to be tested or for a committer to come along and say that a particular PR is OK to test. It does this because the Jenkins model does not have any isolation in the builders so you have to have someone review the code PR to Jenkins running it or a malicious PR person gets the ability to do some nasty stuff. Another alternative is https://concourse.ci/ <https://concourse.ci/>, which doesn’t have a defined story for isolating pull request builds, but there are some sketches for how it should be done (https://github.com/concourse/concourse/issues/366 <https://github.com/concourse/concourse/issues/366>) but I don’t know how well that works on Windows since It uses containers to run builds in (I think it has a no-op-ish thing on Windows, but that might mean that the security properties don’t exist then, I dunno). Securely running untrusted code is hard :| — Donald Stufft
_______________________________________________ core-workflow mailing list core-workflow@python.org https://mail.python.org/mailman/listinfo/core-workflow This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct
