Ralph and all, My 0.02US$
We are kind of limited by the github API https://developer.github.com/v3/repos/statuses/ Basically, a status is pending, success, error or failure plus a string. A possible work around is to have Jenkins set labels on the PR. If only valgrind fails, the status could be succes, and the valgrind failure cold be reported via the status string (that no one might bother reading, but this is an other story) or via a label (Should the label be for success or failure ?) I agree it is not obvious (not to say impossible) to fully understand what Jenkins is doing under the hood. That could/should be documented, or at least, the Jenkins plugin could be made published (Public repository like the bot used to set labels/milestones/assignees, or private in the ompi-tests repository) I will give some more thoughts to the testing part. Cheers, Gilles On Wednesday, November 25, 2015, Ralph Castain <r...@open-mpi.org> wrote: > Hi folks > > I wanted to pull this conversation out from the specific issue where it > was being conducted as I think it merits a broader discussion. I understand > and appreciate the role of the Jenkins testing - what I am trying to find > is a way to make that testing more usable. > > There are two things that I think would help: > > 1. separating the tests being conducted into different “buckets”. We now > have several types of testing being conducted: > > * simple build tests. These don’t involve any execution. If something > fails a build test, it would be > very helpful to clearly state exactly what configure options were > being used, and what compiler. > Ideally, such failures would be labeled as “build”. > > * valgrind tests. These are problematic in that they are not > necessarily PR-specific - if anything > causes a leak or valgrind issue, every PR is marked as “failed” and > can lead to wasted time > chasing non-existent problems with a specific PR. Unfortunately, I > can’t think of a way to get > Jenkins to properly deal with the issue other than to mark such > test results as “valgrind” so > they are clearly called out as being in that category > > * distribution tests that build tarballs, run “make distcheck”, etc. > These usually fail due to something > not being included in the tarball, or some directory not being > completely cleaned. This is > another case where it is really important to know, for example, that > someone used a platform > file when building the tarball, so it would really help to know > exactly how this test was conducted. > Ideally, any distribution test failure would be marked as > “distribution” so we know what happened. > > * run tests that execute various programs. Lots of things can go wrong > here, many of them > dependent on exactly how the code was built (so we know which > components were > around) and how it was run (e.g., default MCA param file). Ideally, > these failures would > be marked as “run”. > > Please note: when I ask for a clear statement of configuration options > etc, what I’m saying is that it is very hard to sift thru hundreds of lines > of output to find, for example, the cmd line that failed. A more concise > test output would make debugging much faster and easier, and therefore make > Jenkins testing much more usable. > > > 2. having the Jenkins testers clearly tell us what tests they are > expecting us to pass. Perhaps a list of these could be posted somewhere, > and some notification given as to when those lists are being changed? It > would help avoid surprises and allow developers a chance to test things > themselves before posting PRs. > > I know I’m asking for some effort on behalf of those running these > servers. However, I think it would make those efforts much more useful. > Ralph > > _______________________________________________ > devel mailing list > de...@open-mpi.org <javascript:;> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/11/18388.php
