Thanks for the idea. For IBM's internal CI that's similar to what we do (but I don't think we take advantage of the XML report - I'll mention that to our CI folks) - we run MTT for every internal PR. It takes a few hours per PR to complete though. I don't know how acceptable that is for the community, but we should talk about it. Maybe there is a subset of the ompi-tests repo that we can run in each PR then let full MTT runs test more throughly.
On Wed, Jul 12, 2017 at 12:53 AM, Gilles Gouaillardet < gilles.gouaillar...@gmail.com> wrote: > Josh and all, > > an other or complementary idea is to use MTT from jenkins and with the > junit plugin. > iirc, the python client can generate a junit xml report, and fwiw, i > made a simplistic proof of concept in the perl client. > the idea is the junit jenkins plugins display a summary of all tests > (ok, ko, skip) but also provides some tendencies (e.g. things are > going better or worst) > > if we only want to use the junit plugin, an option is to have a > "dummy" jenkins job that query results or download xml reports from > the mtt server, so the results and tendencies can be displayed by > jenkins > > Cheers, > > Gilles > > On Wed, Jul 12, 2017 at 5:20 AM, Josh Hursey <jjhur...@open-mpi.org> > wrote: > > In the Open MPI face-to-face meeting we had a long discussion about how > to > > better harness MTT such that new failures are identified and the > community > > alerted to their existence. The current manual way is not working. Our > > ultimate goal here is to know if we are making forward progress, and if > > something new fails the community knows about it immediately and > > automatically. > > > > We had a bunch of discussion and decided to think it over some more then > > come back to a teleconf in the next week or two to make a plan. The MTT > > group is scheduled to meet on Thursday, July 20 at 4 pm EST. We already > know > > that doesn't work for everyone interested so I setup a doodle poll to > pick a > > time for this specific discussion. > > > > https://doodle.com/poll/8zdetnnv4iaaawg4 > > > > Please fill out the doodle poll by Thursday, July 13, 2017 at 5 pm EST. > I'll > > pick a time then. > > > > > > > > Three topics were identified to make progress. > > 1. Move MTT (Perl) Client to new server submission interface > > 2. Adding an "expected fail" list to MTT Client. > > 3. Drive the number of MTT reported failures to 0 > > > > (1) This will start the movement to the new RESTful MTT server. > Eventually > > we want to disable the old PHP submission mechanism. This is a first > step. > > Josh needs to re-test this interface to confirm it is still working as > > expected with the existing 'v3' database. > > > > (2) For each branch/test-suite/runtime-configuration identify a test > run as > > "known to fail". These will be flagged in the MTT Database and MTT > > Viewer/Reporter as separate from other failures ("failed, but we > expected to > > pass"). Is this part of the INI file or a separate 'thing'? > > > > (3) The idea is that the "failed, but we expected to pass" number should > be > > 0 for all sites. The individual site is responsible for maintaining their > > "known to fail" list. If the "failed, but we expected to pass" number is > >0 > > then this is a 'new failure' and an email to the community is generated. > > > > > > -- > > Josh Hursey > > IBM Spectrum MPI Developer > > > > _______________________________________________ > > devel mailing list > > devel@lists.open-mpi.org > > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel > _______________________________________________ > devel mailing list > devel@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel > -- Josh Hursey IBM Spectrum MPI Developer
_______________________________________________ devel mailing list devel@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
