I would apolozise that i did not bring these set of discussions in their logical conclusion. I would hence do this in some couple of days to come.
======================================================================== Issue # 1 ======================================================================== As i can infer the logic majority of you have proposed, to keep LTP away from the realm of Kernel Build and Test, and concentrate more on real testing of the Kernel and other improvement areas; i would agree with you. It is not that i did not understand what and where LTP should really focus into, but, i wanted to bring to everybody´s notice some aspirations from some quarters. Now, having strongly convinced the actual need for this, we would not think of LTP needing to do Auto Kernel Build/install, Distro fetch/install. We would rather let some other project do that and keep LTP focussed on the real need to test the kernel better. > On Wed, Aug 13, 2008 at 02:02:55PM +0530, Subrata Modak wrote: > > Hi, > > > > Recent OLS 2008 was a critical point in LTP´s evolution, as i got the > > opportunity to meet several people across the Linux ecosystem, and > > listened to their opinion about LTP. Here i would start a mail chain > > with the above Subject line, discuss each and every issue in this > > mailing list, collate everybody´s opinion on those issue(s) and take > > action accordingly. These are the people i encountered: > > > > 1) People, who uses LTP heavily. And they suggested lots of improvement > > to it. We will discuss those issues in mails from now, > > > > 2) People, who have heard about LTP and not used it till now. They > > promised that they will give a try, > > > > 3) People, who has never heard about it. So, it was an opportunity to > > convey them what LTP is all about. I hope people in Category 2 & 3 will > > start using LTP soon, and we will get an enlarged user base and hence > > bringing more contribution in future. > > > > ================= > > ISSUE # 1 > > ================= > > The heavy users made a point of LTP having the capability to automate > > testing completely. What they meant was LTP to have capability to do: > > 1) Kernel Build, > > 2) Kernel Install/Distro install, > > 3) Then do specific/all tests, > > > > They said that this feature will simplify the way they work. I would > > like to know what you all think about this. > > > > What i feel is, every project should evolve and should be flexible > > enough to meet their users requirement dynamically, and should not be > > tied down with the limitations of it´s initial design constraints. If > > automating kernel build, install and tests is a requirement coming from > > the user community, then we need to give a hard look at it. I would like > > to know what you think about this. > > [ Sorry for arriving late in the discussion (hollydays) ] > > I definitely agree with others that 1) and 2) should be done in separate > projects. > > However, 3) (doing selected tests) has to be done in LTP, IMHO. > > AFAICS, what LTP currently provides for this is (please correct me where I am > wrong): > - run all tests, > - run a (whole) suite of tests (eg "syscalls"), > -? run a single test. > > What I would like: > i) run a custom subset of tests, out of existing suites (eg not all > "syscalls"), > ii) disable some tests in a suite (eg disable fork07 in "syscalls"), but still > report that the test was disabled. > Yes, this can be done. What i can think of presently, the best way to do so is to: 1) Provide a custom command file under ltp/runtest/*, which contains a set of entries (similar way you enter in ltp/runtest/* files) you would not like to run. 2) Let runltp do an internal grep from it´s -f <files> and exclude those tests you mentioned. We may need to introduce another option, say -F <file2>. Where file2 basically contains tests that you want to exclude from running. > For i), why not writing my own file under runtest/? Because some tests need a > few setup before being run, and I don't want to copy these setup in all custom > runlists. > To address this, I suggest to move those setup commands in > dedicated executables located together with the test sources (say > abort01.setup > under testcases/kernel/syscalls/abort/). Those *.setup could be simple shell > scripts (abort01.setup would contain "ulimit -c 1024") or binaries as well. Yes, this is absolutely possible. You can install your setup script during make install, and invoke the script during test run. Let the script execute the original test case binary if it feels that all system requirements are made, and report results. We have already started doing it for lots of scripts, the latest i remember is this: http://ltp.cvs.sourceforge.net/viewvc/ltp/ltp/testcases/kernel/syscalls/ioctl/test_ioctl?revision=1.1&view=markup > For ii) a usecase is the following: when implementing a new feature, often the > feature is buggy and breaks several tests for different reasons. "Break" means > the test result is "FAIL", or the machine was crashed as well. As long as > fixing those bugs is fast, one can simply comment out the failing tests in the > related file under runtest/. However, when it takes a long time to fix those > bugs, having a report giving the list of successful tests as well as the ones > that were (intentionally) disabled gives some bugfix progress bar. > To address this, I suggest to, first, do as I suggested for i), and > second, replace the setup code in the runlists by wrapping code when desired > Example to disable fork07: > fork07 fork07 > becomes > fork07 disabled fork07 > > Where "disabled" is a script living in the same directory as test binaries, > that > contains: > > <begin script> > #!/bin/sh > > echo $1 disabled > <end script> > > If what I suggest looks ok, I can write patches. Sure. Please let us know the way you would like to do this. Regards-- Subrata > > Thanks, > > Louis > ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Ltp-list mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/ltp-list
