I think talking to the MPICH folks about creating a common test pool might be useful. More useful would be to get the MPI Forum to 'bless' it and take input from all of the MPI venders. Maybe you all can talk about that in a sidebar at the MPI Forum - depending on what we figure out here.
The MPICH tests [1] contains a few test suites that are freely available as tarball downloads on their site (though they seem a bit stale - last updated 2012). They also have the 'tests' subdirectory in their repo [2]. >From my perspective, the benefits of creating a new public test repo on GitHub would be (in no particular order, numbers provided to aid in reference during followup discussion): (1) As we create new MPI test cases for issues, we file them publicly. Currently, we drop them in ompi-tests for convenience. As a result, the public cannot see/use them. Same for new test suites that we might create as part of a new MPI standard interface proposal (e.g., FT, Sessions, or even back to Nonblocking collectives). (2) GitHub allows us to foster a discussion about the tests in a way that is easily traceable. More so than using a mailing list and discussing a tarball drop of a test suite. (3) We can associate a MTT snippet of code for Test Build/Run that can be used to run that test suite. Instead of putting this in with the MTT repo. This would make these snippets slightly easier to find. (4) We can contain references to other test suites, even if we don't copy them into our public test suite. This would allow us to say "hey, there are these great MPICH/Netpipe/IMB/... tests at this URL" then give them instructions on how we run them in MTT (or outside of MTT). If we have any patches for that test suite we could collect them in the sub-directory until (if) they get incorporated upstream. We could also associate notes on runtime parameters that help tune Open MPI for a particular performance test at various scales. (5) Generally, it would make it easier for the community to pick up some unit tests and run them to do a 'make deep-check' on their MPI install. I think there is a hesitation to use the test suite from another MPI implementation with Open MPI as it might not be as consistent as one would like. Any failures would have to be investigated as a difference of implementation (do their tests use our CLI correctly?), difference of MPI Standard interpretation (as often happens as the MPI Forum rolls out errata), or actual bug in Open MPI. For an end user that is tough to determine. For MPI implementers it is easier since we live those differences. Let me play a little devils advocate here too on a couple points [not that I need to really, I'm sure I'll get some help here]. (#) reference the points above. (1) Shouldn't the MPI forum be doing this? Maybe. That is for the MPI Forum to decide, and might be difficult to achieve given the diversity of stakeholders at that level. However, if there is a public repo for MPI tests that the MPI Forum can all agree on then that might be a good place to start. I still think (4) would apply, but maybe we can solve that differently. (2) How much do we really debate the correctness of our tests? Early on in the project, quite often. Lately it still happens (NULL datatype issue, for example). That discussion can only happen internally since the test cases we reference are not public unless we expose them (which is toe'ing the line of redistribution). There is value in broadening the discussion outside of the Open MPI developers on some of those test cases. At the very least for educational benefit of the community about some of the more subtle edges of the MPI standard. (3) Shouldn't that sample code live with MTT? Maybe. I can see both sides of this one. I do think there is some value in separating out the functional code of MTT from the test suite configurations. -- Josh [1] http://www.mcs.anl.gov/research/projects/mpi/mpi-test/tsuite.html [2] http://trac.mpich.org/projects/mpich/browser/test On Thu, May 19, 2016 at 6:50 AM, Jeff Squyres (jsquyres) <jsquy...@cisco.com > wrote: > I was initially for this idea, but now I find myself conflicted. > > Specifically: what's the value in yet another MPI test suite? > > Sure, we've got a bunch of tests that no one else has (i.e., the things > we've home-grown over the years). Some are great tests. Some will need to > be cleaned up and generalized. Some are user-contributed. Some are > ompi-specific. > > Should these tests be contributed to some other test suite? > > Specifically: I'm wondering about the MPICH test suite - that seems to be > the only remaining big MPI test suite these days. Is it worth having a > discussion w the MPICH folks to see if a) their test suite is general > enough for all MPI implementations, and B) if they would accept a bunch of > random tests from us? > > And if not, I think I'd like to understand better the value add that we > can provide by making another MPI test suite. > > Sent from my phone. No type good. > > > On May 18, 2016, at 11:54 PM, Josh Hursey <jjhur...@open-mpi.org> wrote: > > > > WHAT: Create a public test repo (ompi-tests-public) to collect > > > > WHY: ompi-tests is private, and difficult/impossible to open up. There > is a demand for a public collection of unit tests. This repo would allow us > to cultivate such a collection of unit tests. > > > > WHERE: open-mpi GitHub project > > > > TIMEOUT: Teleconf, Tue., May 24, 2016 > > > > MORE DETAIL: > > > > Over the years we have had periodic public requests for access to our > ompi-tests repo. Opening up ompi-tests to the public is nearly impossible > since, given the age of some of these tests, determining if we can > redistribute these tests has become complicated. > > > > Recently we had two different requests on the MTT users and Open MPI > devel list about access to the ompi-tests repo for testing. This got me > thinking that we could try to cultivate a public set of tests with a clear > lineage and license. > > > > > > Below are my current thoughts for structure and maintenance of the repo. > Certainly up for discussion. > > > > Similar to the existing ompi-tests repo structure. > > - Call the repository "ompi-tests-public" > > - The repo will contain at least one test suite ('misc' - see below). > > - Each test suite can have its own build system > > - Each test suite should contain a README-MTT.md which will contain a > sample Test Build and Test Run section for use in MTT. > > > > All tests contributed will be covered under the Open MPI license > agreement unless a specific test suite has a different, but compatible > license. To contribute a test to the repo a developer must sign the > contributor agreement. Contributions must go through a PR + Review process > (similar to how we maintain ompi-release). This is meant to provide an > opportunity to review the tests for correctness before acceptance into the > repo. > > > > We will seed the repo with a 'misc' test suite. This test suite is meant > to collect all of the miscellaneous tests created by Open MPI developers > including those regression tests that emerge as part of Issues or MPI Forum > discussions, for example. It will contain the unit tests currently in the > Open MPI's examples directory - what we have been calling the 'trivial' > test suite. I was thinking about breaking it down into roughly MPI Standard > chapters. > > > > If someone wants to contribute a whole suite of tests they can do so in > a separate directory in the repo with it's own build system. The license > must be compatible with the Open MPI license, and, in particular, allow the > code to be freely distributed. > > > > > > Let me know what you think. Certainly everything here is open for > discussion, and we will likely need to refine aspects as we go. > > > > -- Josh > > > > > > > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org > > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel > > Link to this post: > http://www.open-mpi.org/community/lists/devel/2016/05/18997.php > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2016/05/19000.php >
