One thing that I've seen in successful other communities is that when folks run the long running tests they also document the specifics of the configuration that they used. Things like your java version, operating system, EC2 instance type. Then this all gets fed into a table published with the release notes.
I think it does a lot to give users confidence in the releases when they can see that there has been testing on a variety of platforms with a variety of settings. It's also a super rough way to catch performance regressions since the release notes will contain that ITBLL ran for 24 hours with X entries on Y machines and that can be compared across versions. Mike On Mon, Sep 18, 2017 at 3:10 PM, Andrew Purtell <apurt...@apache.org> wrote: > I also think it's fair to make allowances for folks who want to test the RC > but might be short on personal time. So there should be a list of essential > items (like sigs, sums, shas/tags) and then a list of optional items (ltt, > pe, itbll, etc.) > > On Mon, Sep 18, 2017 at 12:52 PM, Stack <st...@duboce.net> wrote: > > > On Mon, Sep 18, 2017 at 11:49 AM, Misty Stanley-Jones <mi...@apache.org> > > wrote: > > > > > I keep feeling like we need a checklist for all the different things > that > > > should be tested when people vote on a release candidate. Not everyone > > > needs to do all the checks, but probably at least someone should be > doing > > > each of the following (and maybe things I am forgetting): > > > > > > - Binary .tar.gz and .zip extract > > > - Source .tar.gz and .zip extract > > > - All md5sums match > > > - The source .tar.gz matches the hash on the Git repo > > > - Everything is signed correctly > > > - The source compiles into exactly the binary > > > - The source compiles with both OpenJDK and Oracle JDK (?) > > > - The binaries from the binary .tar.gz and .zip run as expected > > > - The binaries built from the source .tar.gz and .zip run as expected > > > - The standard tests all run and pass, with the exception of known > > flakies > > > - LTT and PE tools run and look good > > > > > > > > > Would it be helpful to have some kind of dashboard where people could > > "sign > > > up" to test different things on that checklist, and maybe to require > that > > > all the checks have been done by at least someone before the RC can be > > > released? Maybe it could be something silly like a Google Form, even. > Or > > a > > > spreadsheet. > > > > > > I think it might also be less intimidating for newbies who have never > > voted > > > on an RC before (and we really really want as many people as possible > to > > > participate in the voting process). > > > > > > What do y'all think? > > > > > > Misty > > > > > > > > > Sounds great. > > > > St.Ack > > > > P.S.Here is a link to an old email that points at a LarsH doc. which > cribs > > from an old Dave Wang wiki page that tried to do similar: > > http://search-hadoop.com/m/HBase/YGbb22RVtnt76g?subj=Re+Releases+tests+ > > > > > > -- > Best regards, > Andrew > > Words like orphans lost among the crosstalk, meaning torn from truth's > decrepit hands > - A23, Crosstalk >
