On Mon, Oct 20, 2008 at 05:41:21PM +0530, Subrata Modak wrote: > I would apolozise that i did not bring these set of discussions in their > logical conclusion. I would hence do this in some couple of days to > come. > > ======================================================================== > Issue # 1 > ======================================================================== > As i can infer the logic majority of you have proposed, to keep LTP away > from the realm of Kernel Build and Test, and concentrate more on real > testing of the Kernel and other improvement areas; i would agree with > you. It is not that i did not understand what and where LTP should > really focus into, but, i wanted to bring to everybody´s notice some > aspirations from some quarters. Now, having strongly convinced the > actual need for this, we would not think of LTP needing to do Auto > Kernel Build/install, Distro fetch/install. We would rather let some > other project do that and keep LTP focussed on the real need to test the > kernel better. > > > On Wed, Aug 13, 2008 at 02:02:55PM +0530, Subrata Modak wrote: > > > Hi, > > > > > > Recent OLS 2008 was a critical point in LTP´s evolution, as i got the > > > opportunity to meet several people across the Linux ecosystem, and > > > listened to their opinion about LTP. Here i would start a mail chain > > > with the above Subject line, discuss each and every issue in this > > > mailing list, collate everybody´s opinion on those issue(s) and take > > > action accordingly. These are the people i encountered: > > > > > > 1) People, who uses LTP heavily. And they suggested lots of improvement > > > to it. We will discuss those issues in mails from now, > > > > > > 2) People, who have heard about LTP and not used it till now. They > > > promised that they will give a try, > > > > > > 3) People, who has never heard about it. So, it was an opportunity to > > > convey them what LTP is all about. I hope people in Category 2 & 3 will > > > start using LTP soon, and we will get an enlarged user base and hence > > > bringing more contribution in future. > > > > > > ================= > > > ISSUE # 1 > > > ================= > > > The heavy users made a point of LTP having the capability to automate > > > testing completely. What they meant was LTP to have capability to do: > > > 1) Kernel Build, > > > 2) Kernel Install/Distro install, > > > 3) Then do specific/all tests, > > > > > > They said that this feature will simplify the way they work. I would > > > like to know what you all think about this. > > > > > > What i feel is, every project should evolve and should be flexible > > > enough to meet their users requirement dynamically, and should not be > > > tied down with the limitations of it´s initial design constraints. If > > > automating kernel build, install and tests is a requirement coming from > > > the user community, then we need to give a hard look at it. I would like > > > to know what you think about this. > > > > [ Sorry for arriving late in the discussion (hollydays) ] > > > > I definitely agree with others that 1) and 2) should be done in separate > > projects. > > > > However, 3) (doing selected tests) has to be done in LTP, IMHO. > > > > AFAICS, what LTP currently provides for this is (please correct me where I > > am > > wrong): > > - run all tests, > > - run a (whole) suite of tests (eg "syscalls"), > > -? run a single test. > > > > What I would like: > > i) run a custom subset of tests, out of existing suites (eg not all > > "syscalls"), > > ii) disable some tests in a suite (eg disable fork07 in "syscalls"), but > > still > > report that the test was disabled. > > > > Yes, this can be done. What i can think of presently, the best way to do > so is to: > 1) Provide a custom command file under ltp/runtest/*, which contains a > set of entries (similar way you enter in ltp/runtest/* files) you would > not like to run. > 2) Let runltp do an internal grep from it´s -f <files> and exclude those > tests you mentioned. We may need to introduce another option, say -F > <file2>. Where file2 basically contains tests that you want to exclude > from running.
I like this idea. This is less intrusive than what I suggested. > > > For i), why not writing my own file under runtest/? Because some tests need > > a > > few setup before being run, and I don't want to copy these setup in all > > custom > > runlists. > > To address this, I suggest to move those setup commands in > > dedicated executables located together with the test sources (say > > abort01.setup > > under testcases/kernel/syscalls/abort/). Those *.setup could be simple shell > > scripts (abort01.setup would contain "ulimit -c 1024") or binaries as well. > > Yes, this is absolutely possible. You can install your setup script > during make install, and invoke the script during test run. Let the > script execute the original test case binary if it feels that all system > requirements are made, and report results. We have already started doing > it for lots of scripts, the latest i remember is this: > > http://ltp.cvs.sourceforge.net/viewvc/ltp/ltp/testcases/kernel/syscalls/ioctl/test_ioctl?revision=1.1&view=markup Yes something like that. This is however not absolutely required if we implement selective test disabling the way you suggested it. > > > For ii) a usecase is the following: when implementing a new feature, often > > the > > feature is buggy and breaks several tests for different reasons. "Break" > > means > > the test result is "FAIL", or the machine was crashed as well. As long as > > fixing those bugs is fast, one can simply comment out the failing tests in > > the > > related file under runtest/. However, when it takes a long time to fix those > > bugs, having a report giving the list of successful tests as well as the > > ones > > that were (intentionally) disabled gives some bugfix progress bar. > > To address this, I suggest to, first, do as I suggested for i), and > > second, replace the setup code in the runlists by wrapping code when desired > > Example to disable fork07: > > fork07 fork07 > > becomes > > fork07 disabled fork07 > > > > Where "disabled" is a script living in the same directory as test binaries, > > that > > contains: > > > > <begin script> > > #!/bin/sh > > > > echo $1 disabled > > <end script> > > > > If what I suggest looks ok, I can write patches. > > Sure. Please let us know the way you would like to do this. Following your suggestion, using a -F <disabled_tests> option for runltp: 1) put a list of test names, one per line, in <disabled_tests>; 2) in runltp, build a sed script from the list of test names in <disabled_tests>, which purpose is to replace every line of its input beginning with a disabled test name with 'disabled <test_name>'; 3) in runltp, filter the generated 'alltests' file with the generated sed script. That's it. I have no time to do this right now, unfortunately. However, your feedback is welcome :) Thanks, Louis -- Dr Louis Rilling Kerlabs Skype: louis.rilling Batiment Germanium Phone: (+33|0) 6 80 89 08 23 80 avenue des Buttes de Coesmes http://www.kerlabs.com/ 35700 Rennes
signature.asc
Description: Digital signature
------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________ Ltp-list mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/ltp-list
