On 04/17/2012 03:04 AM, Dor Laor wrote: > On 04/16/2012 11:37 PM, Lucas Meneghel Rodrigues wrote: >> On Mon, 2012-04-16 at 14:08 -0400, Chris Evich wrote: >>> On 04/13/2012 05:34 PM, Ademar de Souza Reis Jr. wrote: >>>> On Thu, Apr 12, 2012 at 12:17:44PM -0300, Lucas Meneghel Rodrigues >>>> wrote: >>>>> So, we started this thread at github: >>>>> >>>>> https://github.com/autotest/autotest/issues/273 >>>>> >>>>> But we feel like it's a better idea to move it here. >>>>> >>>>> Bottom line, we'd like to separate the test modules (both client and >>>>> server side) in separate repositories. We have obviously the pros and >>>>> cons. Here are some of the points previously raised by Ademar: >>>> >>>> Thanks for moving the discussion to the list Lucas! >>>> >>>> I have more comments in the end. >>>> >>>>> >>>>> """ >>>>> Pros: >>>>> - Scalability: we want autotest to grow orders of magnitude, and the >>>>> current model simply doesn't scale. Right now Lucas spends a good >>>>> amount >>>>> of his time reviewing tests and maintaining the good quality of the >>>>> autotest codebase. Once we move the tests out of autotest.git, we >>>>> get a >>>>> clear separation and different patch queues with different priorities >>>>> (and lots of different maintainers). >>>>> - If we want autotest to be used by 3rd parties (and we do), we >>>>> have to >>>>> consider tests written outside of autotest.git as first class >>>>> citizens. >>>>> One of the best ways to do that is by leveraging the playing field and >>>>> eating our own dogfood: tests written by us should work under the same >>>>> scenario and cirscunstances of the 3rd parties tests and that means >>>>> outside of the autotest repository. And this should be the default. It >>>>> will help us keep the API stable, enforce a clear separation between >>>>> tests and the framework/library and give us a clear understanding >>>>> of any >>>>> usability problems or breakage. Think of multiple releases of >>>>> autotest: >>>>> you'll need to declare compatibility between releases and some tests >>>>> will require Autotest>= X. One of the best ways to keep track of these >>>>> changes is by having tests separated from the framework. >>>>> - 3rd party developers, most of the time, are not interested in >>>>> contributing to autotest itself. They want to use autotest to write >>>>> their tests, but get scared of the complexity of the codebase. KISS is >>>>> king here: we need a clean, intuitive and self descriptive directory >>>>> strucutre on autotest.git to attract new contributors, and having >>>>> tests >>>>> there, with control files, output dirs, reference files and a >>>>> changelog >>>>> with test changes just doesn't help. Ditto for people writting new >>>>> tests. >>>>> - I've heard this request multiple times during meetings with managers >>>>> or developers from other teams: they want to have their own tests >>>>> inside >>>>> their own repository. Submitting and maintaining their tests to >>>>> autotest.git is a burden. >>>>> >>>>> Cons: >>>>> - It's a change and it's natural for us to resist it. >>>>> - For people already familiar with the codebase, writing new tests or >>>>> fixing current tests will require more work, as we may have to submit >>>>> patches to multiple repositories. But this is a good thing: it'll help >>>>> us keep in mind the API stability and better document and >>>>> understand the >>>>> changes that get into autotest.git. >>>>> """ >>>> >>>> A summary of a few other points raised in the github discussion: >>>> >>>> - About the risk of splitting the tests (we'll have to keep tests >>>> and the library in sync): >>>> >>>> We definitely want 3rd parties to write their own tests and >>>> keep them running on their side. And we want that without the >>>> need of forking autotest. This is even supported right now. >>>> >>>> So the risk already exists for these users and we would be >>>> irresponsible in breaking the API without proper versioning of >>>> the library and a proper release-notes. >>>> >>>> That's why in my mind we should have the autotest library >>>> properly versioned and tests completely decoupled from it. >>>> >>>> >>>> - The current tests validate the framework (working as kind of >>>> "unit-tests" for autotest itself) >>>> >>>> The lack of unit-tests in autotest is a separate problem, which >>>> should not be a requirement for the split, as there will be >>>> possible to keep validating the framework with the external >>>> tests anyway (the same way applications using a library find >>>> bugs in the library, external tests will find bugs in the >>>> autotest framework). >>>> >>>> >>>> - About the scalability and 3rd party use: "These are more >>>> people/project/communication dynamic issues, not code issues. >>>> Splitting code will just fork Lucas as opposed to cloning him." >>>> >>>> Please think about the future, not the current set of tests. I >>>> envision autotest growing in orders of magnitude, with projects >>>> implementing their own tests and using the power of the >>>> autotest framework to run and instrument these tests. >>>> >>>> The current tests are just a small portion of the potential >>>> tests that can be "users" of the autotest framework/library, >>>> each its own maintainers. And in that scenario, concentrating >>>> tests inside autotest.git will bring a considerable maintenance >>>> burden and increase the bar for new contributors (not to >>>> mention it's not desirable: many users of autotest don't intend >>>> to contribute to the framework, they just want to use the >>>> framework for their own tests, and nowadays they usually fork >>>> autotest). >>>> >>>> Besides, splitting the tests will immediately free developers >>>> allocated to the core framework from even paying attention to >>>> the evolution of the tests and vice-versa. >>>> >>>> That should be it by now. :) >>>> >>>> Cheers, >>>> Ademar >>>> >>> >>> Well argued, and thanks for the more explicit reasoning. I agree, it >>> will be key to have autotest library internal and repo. version as part >>> of API, and hooking onto this from tests as much as possible. Not an >>> impossible task, but certainly some amount of additional effort would be >>> required. >>> >>> With respect to scaleability, I may need more convincing (and presumably >>> others may too). It would help the argument greatly if we were planning >>> on increasing the number and/or complexity of tests. Then I could see >>> how we'd have problems with scale. >> >> I think Ademar is more concerned about declaring the 'core' framework >> effectively independent from the other tests, to make it easier to >> follow (one won't see test related patches anymore). The current tests >> inside autotest will be kept by the same team, only the commits won't be >> mangled in a single repo. >> >>> Another thing to consider if we split, but include some tests: What's >>> the criteria by which a test is bundled, or not? i.e. I can envision >>> future scaleability problems if it's too inclusive of one test class >>> over another. Perhaps, it would be best to just draw the line and >>> declare NO tests are included (besides unittests maybe)? >> >> Unittests definitely have to stay, plus the self-test tests, as Ademar >> mentioned. > > Tests can be bundled as sub projects and still be hosted by the autotest > git repo. Once such tests gain traction from some upstream development > project we can consider of moving them outside. >
Similarly (to lmr/Ademar's 'make it easier to follow' point), mailing lists can be filtered based on keywords. --- I agree in principal with the majority of points made on this topic so-far. I also think it's good for us to work toward a stable/elegant API with more version-aware 'smarts' built-in to tests. i.e. enabling a split in the future. It's also good that we want to foster community use and growth of the project. However, my perception is that splitting would mostly be a one-way undertaking. At this point in time, I'm not sure the extra burden and risk is greater than the current level of suffering. I also wonder if it might be possible to achieve the goals of splitting, w/o actually splitting (i.e. repo., workflow, and ML keywords, whatever else). Maybe an enhanced development workflow and repository management stuffs. Today, the 5-20 mailing list and github mails per day doesn't seem difficult to manage. Esp., given the slow, methodical and careful moves in updates to the core client/server code. If at some point in the future, it's clearly a PITA to have everything blobbed together, then by all means let's split the test content. Until then, I believe we shouldn't 'put the horse in front of the carrot', and 'if it ain't broke, don't fix it'. However, if the group decides in favor of a split, then I'll still be on-board and support the effort as best as I can. -- Chris Evich, RHCA, RHCE, RHCDS, RHCSS Quality Assurance Engineer e-mail: cevich + `@' + redhat.com o: 1-888-RED-HAT1 x44214 _______________________________________________ Autotest mailing list Autotest@test.kernel.org http://test.kernel.org/cgi-bin/mailman/listinfo/autotest
