On 04/16/2012 11:37 PM, Lucas Meneghel Rodrigues wrote: > On Mon, 2012-04-16 at 14:08 -0400, Chris Evich wrote: >> On 04/13/2012 05:34 PM, Ademar de Souza Reis Jr. wrote: >>> On Thu, Apr 12, 2012 at 12:17:44PM -0300, Lucas Meneghel Rodrigues wrote: >>>> So, we started this thread at github: >>>> >>>> https://github.com/autotest/autotest/issues/273 >>>> >>>> But we feel like it's a better idea to move it here. >>>> >>>> Bottom line, we'd like to separate the test modules (both client and >>>> server side) in separate repositories. We have obviously the pros and >>>> cons. Here are some of the points previously raised by Ademar: >>> >>> Thanks for moving the discussion to the list Lucas! >>> >>> I have more comments in the end. >>> >>>> >>>> """ >>>> Pros: >>>> - Scalability: we want autotest to grow orders of magnitude, and the >>>> current model simply doesn't scale. Right now Lucas spends a good amount >>>> of his time reviewing tests and maintaining the good quality of the >>>> autotest codebase. Once we move the tests out of autotest.git, we get a >>>> clear separation and different patch queues with different priorities >>>> (and lots of different maintainers). >>>> - If we want autotest to be used by 3rd parties (and we do), we have to >>>> consider tests written outside of autotest.git as first class citizens. >>>> One of the best ways to do that is by leveraging the playing field and >>>> eating our own dogfood: tests written by us should work under the same >>>> scenario and cirscunstances of the 3rd parties tests and that means >>>> outside of the autotest repository. And this should be the default. It >>>> will help us keep the API stable, enforce a clear separation between >>>> tests and the framework/library and give us a clear understanding of any >>>> usability problems or breakage. Think of multiple releases of autotest: >>>> you'll need to declare compatibility between releases and some tests >>>> will require Autotest>= X. One of the best ways to keep track of these >>>> changes is by having tests separated from the framework. >>>> - 3rd party developers, most of the time, are not interested in >>>> contributing to autotest itself. They want to use autotest to write >>>> their tests, but get scared of the complexity of the codebase. KISS is >>>> king here: we need a clean, intuitive and self descriptive directory >>>> strucutre on autotest.git to attract new contributors, and having tests >>>> there, with control files, output dirs, reference files and a changelog >>>> with test changes just doesn't help. Ditto for people writting new >>>> tests. >>>> - I've heard this request multiple times during meetings with managers >>>> or developers from other teams: they want to have their own tests inside >>>> their own repository. Submitting and maintaining their tests to >>>> autotest.git is a burden. >>>> >>>> Cons: >>>> - It's a change and it's natural for us to resist it. >>>> - For people already familiar with the codebase, writing new tests or >>>> fixing current tests will require more work, as we may have to submit >>>> patches to multiple repositories. But this is a good thing: it'll help >>>> us keep in mind the API stability and better document and understand the >>>> changes that get into autotest.git. >>>> """ >>> >>> A summary of a few other points raised in the github discussion: >>> >>> - About the risk of splitting the tests (we'll have to keep tests >>> and the library in sync): >>> >>> We definitely want 3rd parties to write their own tests and >>> keep them running on their side. And we want that without the >>> need of forking autotest. This is even supported right now. >>> >>> So the risk already exists for these users and we would be >>> irresponsible in breaking the API without proper versioning of >>> the library and a proper release-notes. >>> >>> That's why in my mind we should have the autotest library >>> properly versioned and tests completely decoupled from it. >>> >>> >>> - The current tests validate the framework (working as kind of >>> "unit-tests" for autotest itself) >>> >>> The lack of unit-tests in autotest is a separate problem, which >>> should not be a requirement for the split, as there will be >>> possible to keep validating the framework with the external >>> tests anyway (the same way applications using a library find >>> bugs in the library, external tests will find bugs in the >>> autotest framework). >>> >>> >>> - About the scalability and 3rd party use: "These are more >>> people/project/communication dynamic issues, not code issues. >>> Splitting code will just fork Lucas as opposed to cloning him." >>> >>> Please think about the future, not the current set of tests. I >>> envision autotest growing in orders of magnitude, with projects >>> implementing their own tests and using the power of the >>> autotest framework to run and instrument these tests. >>> >>> The current tests are just a small portion of the potential >>> tests that can be "users" of the autotest framework/library, >>> each its own maintainers. And in that scenario, concentrating >>> tests inside autotest.git will bring a considerable maintenance >>> burden and increase the bar for new contributors (not to >>> mention it's not desirable: many users of autotest don't intend >>> to contribute to the framework, they just want to use the >>> framework for their own tests, and nowadays they usually fork >>> autotest). >>> >>> Besides, splitting the tests will immediately free developers >>> allocated to the core framework from even paying attention to >>> the evolution of the tests and vice-versa. >>> >>> That should be it by now. :) >>> >>> Cheers, >>> Ademar >>> >> >> Well argued, and thanks for the more explicit reasoning. I agree, it >> will be key to have autotest library internal and repo. version as part >> of API, and hooking onto this from tests as much as possible. Not an >> impossible task, but certainly some amount of additional effort would be >> required. >> >> With respect to scaleability, I may need more convincing (and presumably >> others may too). It would help the argument greatly if we were planning >> on increasing the number and/or complexity of tests. Then I could see >> how we'd have problems with scale. > > I think Ademar is more concerned about declaring the 'core' framework > effectively independent from the other tests, to make it easier to > follow (one won't see test related patches anymore). The current tests > inside autotest will be kept by the same team, only the commits won't be > mangled in a single repo. > >> Another thing to consider if we split, but include some tests: What's >> the criteria by which a test is bundled, or not? i.e. I can envision >> future scaleability problems if it's too inclusive of one test class >> over another. Perhaps, it would be best to just draw the line and >> declare NO tests are included (besides unittests maybe)? > > Unittests definitely have to stay, plus the self-test tests, as Ademar > mentioned.
Tests can be bundled as sub projects and still be hosted by the autotest git repo. Once such tests gain traction from some upstream development project we can consider of moving them outside. > > > _______________________________________________ > Autotest mailing list > Autotest@test.kernel.org > http://test.kernel.org/cgi-bin/mailman/listinfo/autotest _______________________________________________ Autotest mailing list Autotest@test.kernel.org http://test.kernel.org/cgi-bin/mailman/listinfo/autotest
