Ben Reser wrote: > 1) Decrease the amount of testing done by the every commit buildbots on the > branches. This bots should be a quick sanity check, does it build and does it > pass basic tests. > > 2) Create a buildbot setup that generates a tarball on each branch nightly (if > there were any changes). It would do this in the same process that a RM would > use (and eventually would become the process for producing the tarball). > > 3) This tarball would then have various build bots run tests off it. These > tests would be much more extensive. In particular it would run cross version > (SERVER_MINOR_VERSION), packing, sharding and other variations of our test > suite that frankly probably don't get as much attention now as they should. > > 4) Eventually make it so that the RM asks the buildbot to produce the release > tarball, including running the nightly buildbots against it. RM then > downloads > the tarball, compares it to the branch@revision for differences looking for > any > unexpected diffs, and then signs it. It gets put up for everyone to approve. > > 5) Humans would no longer do much testing. We'd have buildbots for the major > platforms, which would have been included by our manual testing from the past. > The build bots would be running far more extensive tests than we ever would be > likely to run on our own. So the only testing we'd need to do is verifying > that the builds match the source we expect and whatever sanity testing we felt > like we wanted to do.
One more thing we'd need to to before signing, of course: verify that the buildbot test runs have completed to our satisfaction (right tarball, sufficient platforms, etc.). > This should lower the bar dramatically for getting tests/signatures out. > Humans don't need to do this boring and repetitive work. It should let > everyone spend more time on more important issues than manual testing. > > Infra had offered us some more buildbot resources here recently. This seems > like the logical way to use them. +1 to all you said. - Julian
