On Mon, Nov 18, 2013 at 03:16:45PM +0800, Feng Yang wrote: > On 11/15/2013 12:21 AM, Ademar de Souza Reis Jr. wrote: > >Hi there. > > > >During the KVM Forum I discussed the future of autotest/virt-test > >with the KVM folks there (Lucas, Cleber and Rudá) and we worked > >on a list of ideas and features for the mid and long term. > > > >This RFC summarizes our discussions and the features we have in > >our backlog. Your feedback and requirements are very welcome. > > > > > >Motivation / Goals > >------------------ > > > > The primary goals of the test automation team working on the > > KVM stack are: > > > > 1. Offer very good and high-quality tools and APIs to the QE > > teams and the KVM stack developers (KVM, QEMU, libvirt, etc) > > who want to implement automated tests. > > > > 2. Maintain a stable test grid running close to (and with > > participation from) KVM developers. With this test grid, we > > want to be able to report and pinpoint issues still during > > the development phases, thus increasing collaboration and > > test coverage. > > > > In order to accomplish these goals, we are planning some > > major changes to the way autotest and virt-test work. We > > believe these changes will bring the autotest projects to > > "the next level" and bring more cooperation between KVM > > developers and QE. We also believe these changes will make > > the projects useful far beyond the scope of virtualization. > > > > Below you'll find the list of features we have in our > > backlog. Some of them are work-in-progress, but they're all > > open to feedback, requirements and ideas. > > > >
<snip> > > > >Out-of-tree tests > >----------------- > > > >Users should be free to implement their tests outside of the > >autotest repositories if they want (either in their own > >repository, or in the same repository where the code being tested > >resides). > > > >This will require autotest API stability and versioning, so that > >users can run their tests using a known stable version of the API > >and tools. > > > > > >Virt testing API > >---------------- > > > >We have a lot of code in the virt-test APIs which is shared > >between several virt-test tests. Given the amount of new tests > >and changes we have in virt-tests, we suffer with API instability > >and some code churn. > > > >We want to make the virt testing API stable (or at least part of > >it), with predictable behavior, documentation and proper > >versioning. > > > >----- x ----- > > > >As you all can see, this is an ambitious list that will require > >many changes to the autotest framework and virt-tests codebases. > > > >We've been working on some of these features already, but the > >main, disruptive changes will take place in the next months and > >may take a year or more to be in good shape, ready to be > >released. > > > >To handle the transition with minimal disruption to current > >autotest users and test writers, our plan is to: > > > > - Make a stable release of autotest and freeze both the API and > > the behavior of the tools, while we start working on the more > > disruptive changes in an unstable branch. > > > - Start the separation of virt-tests into 1) a stable API and > > 2) upstream tests that make use of it (together with parts of > > the API which may require constant changes). We expect to > > spend the next 2 or 3 months on this split and we don't know > > where the line will be drawn yet. Once done, we'll release a > > stable virt-test API and start working on the more intrusive > > changes in a new branch. > > Hi, Ademar Hi Feng Yang. > > Thanks for your introduction. Could you give more information > about the tree split? I'll try. The truth is we don't have all answers yet. We know there'll be some changes ahead, but we don't know exactly where the line will be drawn and what the impact will be. We'll need to start touching the code to be sure: that's why we're proposing to spend the next 2-3 months in this split, to make everything go smoothly. > > What is the purpose of this split? Make the API tree could > works without tests tree, then people could have a Lightweight > framework to run their own code? Yes, that's the main motivation. The second motivation is that there will be many changes to the APIs and tools to implement the use-cases above and during development we won't be able to keep up with both test-code and the API/Framework code. We'll need to work on a smaller code-base, where there might plenty of API and code churn until we get to the final version. At the same time, we don't want to slow down the users of the current virt-test tools and APIs, so we'll have to keep the current code stable and working, and merge the changes later, once the new API and tools are in good shape. > > Which part code put into API tree? We don't have the answer yet. The motivation is to find parts of the API which are stable (stable as in "stable API" -- small rate of changes) and move it to the framework. Not everything will be there. > > Does these two tree have any duplicate code? If yes, How to > keep them sync? How many code should duplicated in two tree? The plan is to minimize it, but it may be necessary to keep some duplication for now, during development. Once the new API and tools are stable, we'll have a "merge window" where we'll resolve the conflicts, move code between the two code-bases and make sure everything is running smoothly. > > How many resource we need to make these two tree sync? We don't expect a need for extra resources. The sync will be made at the end of the development process, when the new tree is stable. The new code will be frozen and developers will focus on the sync between the two code-bases before a final release of the new API and tools. > > How about new case design during this tree update? Still > design test script under current way? > Yes. The current code will be frozen and available as a stable branch, which should be used by current users until the new code is considered stable, which will take a while (initial expectation is of 10 months before a new release with the foundation for the new use-cases in place). > > Will we still put all the test script (kvm, libvirt, v2v) in one tree? We don't want to force anything on anybody. This will be up to the repository maintainers and test developers. Personally, I don't see any reason to split it right now. > > As you know we always need update some framework API under > virttest folder during new case script designing. How to deal > with this situation? As it's done today, in the stable code-base. The developers working on the new code-base will take care of merging the code and drawing the line between the different APIs later, at a merge window, as specified in the paragraph below: > > > - Before we make the first release of the autotest framework > > with the new features and the new virt-test API, we'll spend > > the necessary amount of time making sure all upstream tests > > (virt-tests) are compatible with the new features and changes > > being introduced by the framework. Again, we don't have all answers, but as you can see from the use-cases we mentioned above, there will be several changes to the API and tools. To keep things working smoothly, we'll support the current (stable) code-base and start working on a new one with the new features (everything will be upstream and open). We'll sync both at the end of the development phase, before a stable release. Remember that the cooperation will stay in place: we'll still host weekly hangouts and keep everybody up-to-date with the new developments. Major changes will be announced and discussed in the open, just like we're doing right now with this RFC. Thanks. - Ademar -- Ademar de Souza Reis Jr. Red Hat ^[:wq! _______________________________________________ Virt-test-devel mailing list [email protected] https://www.redhat.com/mailman/listinfo/virt-test-devel
