If OpenHPC folks want to cultivate such a repo I think that would be great.
If such a repo existed then I think we should work with them on making sure that it integrates nicely into MTT so we can add it to the nightly regressions. Getting a release of MTT is something I would like to see happen sooner rather than later. Now that we have more folks working in the code, we might be able to do that this year. But that's a discussion for the MTT list. The things I would want in such a repo would be: (1) Ability to submit new individual unit tests and test suites (2) Ability to discuss existing and proposed tests in a similar way to what we have on GitHub (3) Ability to create some 'best practice' notes on running a test suite with a particular MPI implementation. This will become important for any performance regression tests that might get pushed there. If such a repo existed, then we (Open MPI) could come up with a nice FAQ/wikipage describing how 'ompi-tests' is private, but if one wanted a broader range of tests then look at the public repo from OpenHPC. Then give some instructions on the MTT side about how to drop it in to those configurations. On Thu, May 19, 2016 at 5:24 PM, Ralph Castain <r...@open-mpi.org> wrote: > Assume that we are able to “package” MTT so it can be upstreamed to the > OpenHPC folks (as they have requested). It then becomes a little more > “salable” as a generalized testing tool, which means that the tests in our > repo could be used by others without having to deal with the MTT-specific > harness calls. > > At that point, it seems to me one could make a case for a general MPI test > repo that supported MTT, and OpenHPC could even be a reasonable home for > it. I can certainly raise the question there, if you like. > > > On May 19, 2016, at 8:24 AM, Josh Hursey <jjhur...@open-mpi.org> wrote: > > I think talking to the MPICH folks about creating a common test pool might > be useful. More useful would be to get the MPI Forum to 'bless' it and take > input from all of the MPI venders. Maybe you all can talk about that in a > sidebar at the MPI Forum - depending on what we figure out here. > > The MPICH tests [1] contains a few test suites that are freely available > as tarball downloads on their site (though they seem a bit stale - last > updated 2012). They also have the 'tests' subdirectory in their repo [2]. > > > From my perspective, the benefits of creating a new public test repo on > GitHub would be (in no particular order, numbers provided to aid in > reference during followup discussion): > > (1) As we create new MPI test cases for issues, we file them publicly. > Currently, we drop them in ompi-tests for convenience. As a result, the > public cannot see/use them. Same for new test suites that we might create > as part of a new MPI standard interface proposal (e.g., FT, Sessions, or > even back to Nonblocking collectives). > > (2) GitHub allows us to foster a discussion about the tests in a way that > is easily traceable. More so than using a mailing list and discussing a > tarball drop of a test suite. > > (3) We can associate a MTT snippet of code for Test Build/Run that can be > used to run that test suite. Instead of putting this in with the MTT repo. > This would make these snippets slightly easier to find. > > (4) We can contain references to other test suites, even if we don't copy > them into our public test suite. This would allow us to say "hey, there are > these great MPICH/Netpipe/IMB/... tests at this URL" then give them > instructions on how we run them in MTT (or outside of MTT). If we have any > patches for that test suite we could collect them in the sub-directory > until (if) they get incorporated upstream. We could also associate notes on > runtime parameters that help tune Open MPI for a particular performance > test at various scales. > > (5) Generally, it would make it easier for the community to pick up some > unit tests and run them to do a 'make deep-check' on their MPI install. I > think there is a hesitation to use the test suite from another MPI > implementation with Open MPI as it might not be as consistent as one would > like. Any failures would have to be investigated as a difference of > implementation (do their tests use our CLI correctly?), difference of MPI > Standard interpretation (as often happens as the MPI Forum rolls out > errata), or actual bug in Open MPI. For an end user that is tough to > determine. For MPI implementers it is easier since we live those > differences. > > > Let me play a little devils advocate here too on a couple points [not that > I need to really, I'm sure I'll get some help here]. (#) reference the > points above. > > (1) Shouldn't the MPI forum be doing this? Maybe. That is for the MPI > Forum to decide, and might be difficult to achieve given the diversity of > stakeholders at that level. However, if there is a public repo for MPI > tests that the MPI Forum can all agree on then that might be a good place > to start. I still think (4) would apply, but maybe we can solve that > differently. > > (2) How much do we really debate the correctness of our tests? Early on in > the project, quite often. Lately it still happens (NULL datatype issue, for > example). That discussion can only happen internally since the test cases > we reference are not public unless we expose them (which is toe'ing the > line of redistribution). There is value in broadening the discussion > outside of the Open MPI developers on some of those test cases. At the very > least for educational benefit of the community about some of the more > subtle edges of the MPI standard. > > (3) Shouldn't that sample code live with MTT? Maybe. I can see both sides > of this one. I do think there is some value in separating out the > functional code of MTT from the test suite configurations. > > > -- Josh > > [1] http://www.mcs.anl.gov/research/projects/mpi/mpi-test/tsuite.html > [2] http://trac.mpich.org/projects/mpich/browser/test > > > On Thu, May 19, 2016 at 6:50 AM, Jeff Squyres (jsquyres) < > jsquy...@cisco.com> wrote: > >> I was initially for this idea, but now I find myself conflicted. >> >> Specifically: what's the value in yet another MPI test suite? >> >> Sure, we've got a bunch of tests that no one else has (i.e., the things >> we've home-grown over the years). Some are great tests. Some will need to >> be cleaned up and generalized. Some are user-contributed. Some are >> ompi-specific. >> >> Should these tests be contributed to some other test suite? >> >> Specifically: I'm wondering about the MPICH test suite - that seems to be >> the only remaining big MPI test suite these days. Is it worth having a >> discussion w the MPICH folks to see if a) their test suite is general >> enough for all MPI implementations, and B) if they would accept a bunch of >> random tests from us? >> >> And if not, I think I'd like to understand better the value add that we >> can provide by making another MPI test suite. >> >> Sent from my phone. No type good. >> >> > On May 18, 2016, at 11:54 PM, Josh Hursey <jjhur...@open-mpi.org> >> wrote: >> > >> > WHAT: Create a public test repo (ompi-tests-public) to collect >> > >> > WHY: ompi-tests is private, and difficult/impossible to open up. There >> is a demand for a public collection of unit tests. This repo would allow us >> to cultivate such a collection of unit tests. >> > >> > WHERE: open-mpi GitHub project >> > >> > TIMEOUT: Teleconf, Tue., May 24, 2016 >> > >> > MORE DETAIL: >> > >> > Over the years we have had periodic public requests for access to our >> ompi-tests repo. Opening up ompi-tests to the public is nearly impossible >> since, given the age of some of these tests, determining if we can >> redistribute these tests has become complicated. >> > >> > Recently we had two different requests on the MTT users and Open MPI >> devel list about access to the ompi-tests repo for testing. This got me >> thinking that we could try to cultivate a public set of tests with a clear >> lineage and license. >> > >> > >> > Below are my current thoughts for structure and maintenance of the >> repo. Certainly up for discussion. >> > >> > Similar to the existing ompi-tests repo structure. >> > - Call the repository "ompi-tests-public" >> > - The repo will contain at least one test suite ('misc' - see below). >> > - Each test suite can have its own build system >> > - Each test suite should contain a README-MTT.md which will contain a >> sample Test Build and Test Run section for use in MTT. >> > >> > All tests contributed will be covered under the Open MPI license >> agreement unless a specific test suite has a different, but compatible >> license. To contribute a test to the repo a developer must sign the >> contributor agreement. Contributions must go through a PR + Review process >> (similar to how we maintain ompi-release). This is meant to provide an >> opportunity to review the tests for correctness before acceptance into the >> repo. >> > >> > We will seed the repo with a 'misc' test suite. This test suite is >> meant to collect all of the miscellaneous tests created by Open MPI >> developers including those regression tests that emerge as part of Issues >> or MPI Forum discussions, for example. It will contain the unit tests >> currently in the Open MPI's examples directory - what we have been calling >> the 'trivial' test suite. I was thinking about breaking it down into >> roughly MPI Standard chapters. >> > >> > If someone wants to contribute a whole suite of tests they can do so in >> a separate directory in the repo with it's own build system. The license >> must be compatible with the Open MPI license, and, in particular, allow the >> code to be freely distributed. >> > >> > >> > Let me know what you think. Certainly everything here is open for >> discussion, and we will likely need to refine aspects as we go. >> > >> > -- Josh >> > >> > >> > >> > _______________________________________________ >> > devel mailing list >> > de...@open-mpi.org >> > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel >> > Link to this post: >> http://www.open-mpi.org/community/lists/devel/2016/05/18997.php >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: >> http://www.open-mpi.org/community/lists/devel/2016/05/19000.php >> > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2016/05/19002.php > > > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2016/05/19005.php >
