Sorry, I wanted to respond to this yesterday, but, somehow did not happen.
On Wed, 2009-02-04 at 12:44 +1100, David Gibson wrote: > On Tue, Feb 03, 2009 at 11:29:32AM -0600, Adam Litke wrote: > > Adding Mel Gorman and Eric Munson to this discussion since they are the > > team working on libhugetlbfs now. > > Content-Description: Forwarded message - Re: [Fwd: [RFC] [HUGETLB tests] > Should we scrap them for libhugetlbfs ??] > > Date: Tue, 3 Feb 2009 16:58:18 +0000 > > From: Mel Gorman <[email protected]> > > To: Adam Litke <[email protected]> > > Subject: Re: [Fwd: [RFC] [HUGETLB tests] Should we scrap them for > > libhugetlbfs ??] > > > > Howdy, you didn't actually cc me to them so I didn't respond back to them > > directly. However, feel free to forward this mail to them cc'd on my my, > > Eric's and Larry's IBM internal addresses. > > > > On Tue, Feb 03, 2009 at 08:20:45AM -0600, Adam Litke wrote: > > > > > > > > Hi, > > > > > > I had long been hearing about the LTP hugetlb test cases being broken > > > and no more able to conform to recent changes to hugetlbfs. > > > > AFAIK, the libhugetlbfs developers have not been using the LTP-based > > tests. For distribution testing, we always use the regression tests that > > are part of the library on the assumption their coverage of code and > > corner cases was higher. > > > > > So, every > > > time an issue crops up, we need to do some surgery to restore those test > > > cases. And the present test cases need a major overhaul from some expert > > > in hugetlbfs, who can find time to fix them. > > > > > > > Well, essentially you would be reproducing the expertise that is > > codified in the libhugetlbfs regression tests. They have been built up > > over time where a kernel bug was found, a unit test added, fixed and the > > unit test left in place. Yes. And we want to take advantage of the same. Since, libhugetlbfs will evolve continuously, so do the test cases inside them. And LTP will get updated in that process. > > > However, we have a good testsuite on hugetlbfs from: > > > http://sourceforge.net/projects/libhugetlbfs, > > > which gets updated continuously and improved upon. I did a comparative > > > code coverage (on 2.6.28) for: > > > 1) Hugetlb tests in LTP, and, > > > (http://ltp.cvs.sourceforge.net/viewvc/ltp/ltp/testcases/kernel/mem/hugetlb/), > > > 2) Hugetlb tests in LIBHUGETLBFS > > > (http://sourceforge.net/projects/libhugetlbfs), > > > > > > LIBHUGETLBFS tests has a better code coverage (15.9%) over LTP-Hugetlb > > > (12.4%). > > > > This is very interesting. 15.9% of what? The hugetlbfs code only or the > > kernel overall? > 15.9% of overall kernel code. I just attached the snapshot of the first page, as the complete tarball where you can drill down and see to each and every individual file´s code coverage, essentially runs to few MBs, which cannot be attached in this mail. If someone wants i can send him/them personally the entire set of files. > Indeed. It would also be interesting to get the combined coverage of > both test sets to see if the libhugetlbfs tests cover strictly more > area than the LTP ones, or if there's something that the LTP tests > cover which the library ones don't (in which case we should extend the > libhugetlbfs suite to include it). Good point. > > > > If it does make sense, then i can go ahead and see how i can > > > remove the present tests and integrate the new ones, and how to keep > > > them updating from the latest libhugetlb releases. > > > > I would suggest you check exactly what LTP is testing and make sure the > > the equivalent is being covered by the libhugetlbfs tests in case we are > > missing something. > > > > There are three large problems with a straight port of the tests to LTP. > > > > 1. The tests were not written with LTP in mind so there will be API > > issues around things like reporting a test failure. > > > > 2. Some of the tests are known to fail if the kernel is old or the > > pagesize being used is 16GB. It's a planned workitem to identify where > > expected failures occur and report EXPECTED_FAIL instead of FAIL > > but it's not done at the moment. > > > > 3. Some tests depend on libhugetlbfs functionality. Some of the tests call > > APIs > > that are available in the library to make sure the library is functioning > > as expected. Some are the public API like calling gethugepagesize() > > (manual page in library) but others are parts of an internal API that > > the tests link to directly to make sure the library internals are solid. > > These will be less straight-forward to port. You could just restrict > > yourself to the tests that test kernel interfaces though and that > > would be useful. Yes, there will be a huge effort required if we are looking at porting the tests. At the moment i am not looking at porting the libhugetlbfs tests to port according to LTP APIs. None of us has the time to do that i know. I ran the tests of libhugetlbfs and is satisfied the way it reports results clearly. So, the main point is for the user to understand the outcome of tests. If that objective is meet, then there is no problem in any tests being integrated to LTP. So, i am looking more for integration of existing ones rather than porting to LTP. There are other tests inside LTP, which has been added over time and do not strictly use the LTP APIs. But they are/can-be used/run by the LTP engine. I would like to see it work this way where we make an additional entry: LIBHUGETLBFS_TESTS (cd testcases/kernel/mem/hugetlb/libhugetlbfs-latest/tests; ./run_tests.sh) in http://ltp.cvs.sourceforge.net/viewvc/ltp/ltp/runtest/hugetlb, And then people can simply run using: ./runltp -f hugetlb, I may not go into the exact way to integrating right now, but still this may be the way to go. > > > > One potential way around this is for the LTP tests to run the libhugetlbfs > > tests instead of writing their own. That way the one set of tests are being > > used. It would report FAIL if any one of the libhugetlbfs tests reported > > failure. This would provided added incentive for the libhugetlbfs people to > > people to get their "expected failure" detection right. > > > > > Can you kindly reply with a Signoff to this mail, if you do not have > > > objection in including the tests to LTP from libhugetlbfs ? > > > > I can't speak for Adam obviously but I can't see any reason to object to the > > tests being ported to LTP but check out any licensing issues. libhugetlbfs > > is LGPL and oddly that would apply to the test cases as well. If LTP is GPL > > or something else, that might be a problem - don't ask me, I'm not a lawyer. > Yes, i would also try to contact some legal person inside IBM and see how this can be achieved. > LGPL code into a GPL project is fine (by design, the LGPL is strictly > less restrictive than the GPL). > > > If you are looking for a libhugetlbfs developer to actually port the tests > > to LTP, that's a different issue. There are only two active libhugetlbfs > > developers within IBM and porting tests to LTP is not on their list of > > planned items at the moment. It would be difficult to justify why a port > > to LTP would take a higher priority over other proposed work items and what > > the benefits would be but the final call is not mine of course. > > > > Some sort of wrapper that downloaded a stable version of libhugetlbfs > > and then whatever the latest version is and testing both of those is an > > interesting possibility though. It could even be done in addition to the > > existing LTP tests instead of a replacement. Yes. And if somebody can point out that certain/all LTP hugetlb tests are covered under libhugetlbfs tests, then i can remove those/all corresponding LTP tests. The remaining can be retained inside LTP. And when you incorporate some additional cases in LTP - which may not be there in libhugetlbfs currently - into libhugetlbfs, slowly those can also be removed from LTP as well. So, basically my idea is to get the things like this: When you do a new release of libhugetlbfs, you can send an announcement mail with a CC to ltp-list. I can go ahead and pick up the diff and apply it to LTP. Regards-- Subrata > ------------------------------------------------------------------------------ Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com _______________________________________________ Ltp-list mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/ltp-list
