On Thu, Apr 12, 2012 at 12:17:44PM -0300, Lucas Meneghel Rodrigues wrote: > So, we started this thread at github: > > https://github.com/autotest/autotest/issues/273 > > But we feel like it's a better idea to move it here. > > Bottom line, we'd like to separate the test modules (both client and > server side) in separate repositories. We have obviously the pros and > cons. Here are some of the points previously raised by Ademar:
Thanks for moving the discussion to the list Lucas! I have more comments in the end. > > """ > Pros: > - Scalability: we want autotest to grow orders of magnitude, and the > current model simply doesn't scale. Right now Lucas spends a good amount > of his time reviewing tests and maintaining the good quality of the > autotest codebase. Once we move the tests out of autotest.git, we get a > clear separation and different patch queues with different priorities > (and lots of different maintainers). > - If we want autotest to be used by 3rd parties (and we do), we have to > consider tests written outside of autotest.git as first class citizens. > One of the best ways to do that is by leveraging the playing field and > eating our own dogfood: tests written by us should work under the same > scenario and cirscunstances of the 3rd parties tests and that means > outside of the autotest repository. And this should be the default. It > will help us keep the API stable, enforce a clear separation between > tests and the framework/library and give us a clear understanding of any > usability problems or breakage. Think of multiple releases of autotest: > you'll need to declare compatibility between releases and some tests > will require Autotest >= X. One of the best ways to keep track of these > changes is by having tests separated from the framework. > - 3rd party developers, most of the time, are not interested in > contributing to autotest itself. They want to use autotest to write > their tests, but get scared of the complexity of the codebase. KISS is > king here: we need a clean, intuitive and self descriptive directory > strucutre on autotest.git to attract new contributors, and having tests > there, with control files, output dirs, reference files and a changelog > with test changes just doesn't help. Ditto for people writting new > tests. > - I've heard this request multiple times during meetings with managers > or developers from other teams: they want to have their own tests inside > their own repository. Submitting and maintaining their tests to > autotest.git is a burden. > > Cons: > - It's a change and it's natural for us to resist it. > - For people already familiar with the codebase, writing new tests or > fixing current tests will require more work, as we may have to submit > patches to multiple repositories. But this is a good thing: it'll help > us keep in mind the API stability and better document and understand the > changes that get into autotest.git. > """ A summary of a few other points raised in the github discussion: - About the risk of splitting the tests (we'll have to keep tests and the library in sync): We definitely want 3rd parties to write their own tests and keep them running on their side. And we want that without the need of forking autotest. This is even supported right now. So the risk already exists for these users and we would be irresponsible in breaking the API without proper versioning of the library and a proper release-notes. That's why in my mind we should have the autotest library properly versioned and tests completely decoupled from it. - The current tests validate the framework (working as kind of "unit-tests" for autotest itself) The lack of unit-tests in autotest is a separate problem, which should not be a requirement for the split, as there will be possible to keep validating the framework with the external tests anyway (the same way applications using a library find bugs in the library, external tests will find bugs in the autotest framework). - About the scalability and 3rd party use: "These are more people/project/communication dynamic issues, not code issues. Splitting code will just fork Lucas as opposed to cloning him." Please think about the future, not the current set of tests. I envision autotest growing in orders of magnitude, with projects implementing their own tests and using the power of the autotest framework to run and instrument these tests. The current tests are just a small portion of the potential tests that can be "users" of the autotest framework/library, each its own maintainers. And in that scenario, concentrating tests inside autotest.git will bring a considerable maintenance burden and increase the bar for new contributors (not to mention it's not desirable: many users of autotest don't intend to contribute to the framework, they just want to use the framework for their own tests, and nowadays they usually fork autotest). Besides, splitting the tests will immediately free developers allocated to the core framework from even paying attention to the evolution of the tests and vice-versa. That should be it by now. :) Cheers, Ademar -- Ademar de Souza Reis Jr. Red Hat ^[:wq! _______________________________________________ Autotest mailing list Autotest@test.kernel.org http://test.kernel.org/cgi-bin/mailman/listinfo/autotest
