On Tue, Apr 17, 2012 at 12:35:48PM -0400, Chris Evich wrote:
> On 04/17/2012 03:04 AM, Dor Laor wrote:
> > On 04/16/2012 11:37 PM, Lucas Meneghel Rodrigues wrote:
> >> On Mon, 2012-04-16 at 14:08 -0400, Chris Evich wrote:
> >>> On 04/13/2012 05:34 PM, Ademar de Souza Reis Jr. wrote:
> >>>> On Thu, Apr 12, 2012 at 12:17:44PM -0300, Lucas Meneghel Rodrigues
> >>>> wrote:
> >>>>> So, we started this thread at github:
> >>>>>
> >>>>> https://github.com/autotest/autotest/issues/273
> >>>>>
> >>>>> But we feel like it's a better idea to move it here.
> >>>>>
> >>>>> Bottom line, we'd like to separate the test modules (both client and
> >>>>> server side) in separate repositories. We have obviously the pros and
> >>>>> cons. Here are some of the points previously raised by Ademar:
> >>>>
> >>>> Thanks for moving the discussion to the list Lucas!
> >>>>
> >>>> I have more comments in the end.
> >>>>
> >>>>>
> >>>>> """
> >>>>> Pros:
> >>>>> - Scalability: we want autotest to grow orders of magnitude, and the
> >>>>> current model simply doesn't scale. Right now Lucas spends a good
> >>>>> amount
> >>>>> of his time reviewing tests and maintaining the good quality of the
> >>>>> autotest codebase. Once we move the tests out of autotest.git, we
> >>>>> get a
> >>>>> clear separation and different patch queues with different priorities
> >>>>> (and lots of different maintainers).
> >>>>> - If we want autotest to be used by 3rd parties (and we do), we
> >>>>> have to
> >>>>> consider tests written outside of autotest.git as first class
> >>>>> citizens.
> >>>>> One of the best ways to do that is by leveraging the playing field and
> >>>>> eating our own dogfood: tests written by us should work under the same
> >>>>> scenario and cirscunstances of the 3rd parties tests and that means
> >>>>> outside of the autotest repository. And this should be the default. It
> >>>>> will help us keep the API stable, enforce a clear separation between
> >>>>> tests and the framework/library and give us a clear understanding
> >>>>> of any
> >>>>> usability problems or breakage. Think of multiple releases of
> >>>>> autotest:
> >>>>> you'll need to declare compatibility between releases and some tests
> >>>>> will require Autotest>= X. One of the best ways to keep track of these
> >>>>> changes is by having tests separated from the framework.
> >>>>> - 3rd party developers, most of the time, are not interested in
> >>>>> contributing to autotest itself. They want to use autotest to write
> >>>>> their tests, but get scared of the complexity of the codebase. KISS is
> >>>>> king here: we need a clean, intuitive and self descriptive directory
> >>>>> strucutre on autotest.git to attract new contributors, and having
> >>>>> tests
> >>>>> there, with control files, output dirs, reference files and a
> >>>>> changelog
> >>>>> with test changes just doesn't help. Ditto for people writting new
> >>>>> tests.
> >>>>> - I've heard this request multiple times during meetings with managers
> >>>>> or developers from other teams: they want to have their own tests
> >>>>> inside
> >>>>> their own repository. Submitting and maintaining their tests to
> >>>>> autotest.git is a burden.
> >>>>>
> >>>>> Cons:
> >>>>> - It's a change and it's natural for us to resist it.
> >>>>> - For people already familiar with the codebase, writing new tests or
> >>>>> fixing current tests will require more work, as we may have to submit
> >>>>> patches to multiple repositories. But this is a good thing: it'll help
> >>>>> us keep in mind the API stability and better document and
> >>>>> understand the
> >>>>> changes that get into autotest.git.
> >>>>> """
> >>>>
> >>>> A summary of a few other points raised in the github discussion:
> >>>>
> >>>> - About the risk of splitting the tests (we'll have to keep tests
> >>>> and the library in sync):
> >>>>
> >>>> We definitely want 3rd parties to write their own tests and
> >>>> keep them running on their side. And we want that without the
> >>>> need of forking autotest. This is even supported right now.
> >>>>
> >>>> So the risk already exists for these users and we would be
> >>>> irresponsible in breaking the API without proper versioning of
> >>>> the library and a proper release-notes.
> >>>>
> >>>> That's why in my mind we should have the autotest library
> >>>> properly versioned and tests completely decoupled from it.
> >>>>
> >>>>
> >>>> - The current tests validate the framework (working as kind of
> >>>> "unit-tests" for autotest itself)
> >>>>
> >>>> The lack of unit-tests in autotest is a separate problem, which
> >>>> should not be a requirement for the split, as there will be
> >>>> possible to keep validating the framework with the external
> >>>> tests anyway (the same way applications using a library find
> >>>> bugs in the library, external tests will find bugs in the
> >>>> autotest framework).
> >>>>
> >>>>
> >>>> - About the scalability and 3rd party use: "These are more
> >>>> people/project/communication dynamic issues, not code issues.
> >>>> Splitting code will just fork Lucas as opposed to cloning him."
> >>>>
> >>>> Please think about the future, not the current set of tests. I
> >>>> envision autotest growing in orders of magnitude, with projects
> >>>> implementing their own tests and using the power of the
> >>>> autotest framework to run and instrument these tests.
> >>>>
> >>>> The current tests are just a small portion of the potential
> >>>> tests that can be "users" of the autotest framework/library,
> >>>> each its own maintainers. And in that scenario, concentrating
> >>>> tests inside autotest.git will bring a considerable maintenance
> >>>> burden and increase the bar for new contributors (not to
> >>>> mention it's not desirable: many users of autotest don't intend
> >>>> to contribute to the framework, they just want to use the
> >>>> framework for their own tests, and nowadays they usually fork
> >>>> autotest).
> >>>>
> >>>> Besides, splitting the tests will immediately free developers
> >>>> allocated to the core framework from even paying attention to
> >>>> the evolution of the tests and vice-versa.
> >>>>
> >>>> That should be it by now. :)
> >>>>
> >>>> Cheers,
> >>>> Ademar
> >>>>
> >>>
> >>> Well argued, and thanks for the more explicit reasoning. I agree, it
> >>> will be key to have autotest library internal and repo. version as part
> >>> of API, and hooking onto this from tests as much as possible. Not an
> >>> impossible task, but certainly some amount of additional effort would be
> >>> required.
> >>>
> >>> With respect to scaleability, I may need more convincing (and presumably
> >>> others may too). It would help the argument greatly if we were planning
> >>> on increasing the number and/or complexity of tests. Then I could see
> >>> how we'd have problems with scale.
> >>
First of all, I believe we already have a real scalability issue
with the current workflow.
About the plan on increasing the number of tests: we have forks
of autotest spread around for no good reason. We have several
other teams wanting or planning to adopt autotest (and they don't
want to submit their tests to autotest). Autotest is even being
packaged in Fedora.
So tests living outside of the autotest repo is a fact. The
question is why should we keep a specific set of tests inside our
own repository when there are (many) advantages in keeping them
out.
> >> I think Ademar is more concerned about declaring the 'core' framework
> >> effectively independent from the other tests, to make it easier to
> >> follow (one won't see test related patches anymore). The current tests
> >> inside autotest will be kept by the same team, only the commits won't be
> >> mangled in a single repo.
> >>
> >>> Another thing to consider if we split, but include some tests: What's
> >>> the criteria by which a test is bundled, or not? i.e. I can envision
> >>> future scaleability problems if it's too inclusive of one test class
> >>> over another. Perhaps, it would be best to just draw the line and
> >>> declare NO tests are included (besides unittests maybe)?
> >>
> >> Unittests definitely have to stay, plus the self-test tests, as Ademar
> >> mentioned.
+1: I don't think tests belong to the autotest (core) repository
at all (as Dor mentioned, no problem in hosting them inside the
same github umbrella).
See the attached picture: it's part of the new "autotest vision"
we've been discussing for a while.
> >
> > Tests can be bundled as sub projects and still be hosted by the autotest
> > git repo. Once such tests gain traction from some upstream development
> > project we can consider of moving them outside.
> >
>
> Similarly (to lmr/Ademar's 'make it easier to follow' point), mailing
> lists can be filtered based on keywords.
I never saw this work in practice and it's far from a complete
solution. Besides, the ml traffic is not the biggest problem
here.
>
> ---
>
> I agree in principal with the majority of points made on this topic
> so-far. I also think it's good for us to work toward a stable/elegant
> API with more version-aware 'smarts' built-in to tests. i.e. enabling a
> split in the future. It's also good that we want to foster community
> use and growth of the project.
>
> However, my perception is that splitting would mostly be a one-way
> undertaking. At this point in time, I'm not sure the extra burden and
> risk is greater than the current level of suffering. I also wonder if
> it might be possible to achieve the goals of splitting, w/o actually
> splitting (i.e. repo., workflow, and ML keywords, whatever else). Maybe
> an enhanced development workflow and repository management stuffs.
>
> Today, the 5-20 mailing list and github mails per day doesn't seem
> difficult to manage. Esp., given the slow, methodical and careful moves
> in updates to the core client/server code. If at some point in the
> future, it's clearly a PITA to have everything blobbed together, then by
> all means let's split the test content.
The fact that you're managing them easily doesn't mean it's not a
problem for others. :)
>
> Until then, I believe we shouldn't 'put the horse in front of the
> carrot', and 'if it ain't broke, don't fix it'. However, if the group
> decides in favor of a split, then I'll still be on-board and support the
> effort as best as I can.
Risking getting into an infinite discussion (we may have to agree
in disagreeing here): I do believe the current model is "broken"
and needs to be fixed ("broken may be too strong here, but
anyway).
(I started writting a new paragraph listing the pros/cons here,
but I would just repeat myself. I don't see any other cons
besides "increase the burden on current developers who are used
to the current workflow (questionable)", and "It's a change and
it's natural for us to resist it", which are both included in the
original message).
If for some reason we figure out it was a bad decision, we can
merge the tests back. The exercise itself will be useful to
validate the usecase "support tests running outside of our
framework" and "keep stable, versioned APIs".
BTW, I recommend you use reply-to-all instead of just reply (I
would have seen your message way sooner if Cc'ed).
Cheers,
- Ademar
--
Ademar de Souza Reis Jr.
Red Hat
^[:wq!
_______________________________________________
Autotest mailing list
Autotest@test.kernel.org
http://test.kernel.org/cgi-bin/mailman/listinfo/autotest
