Will, Thanks for posting this message. I think we do need to come up with a strategy as to how we validate the kernel interface and the library. Keep in mind that the library is optional, you do not need the library to use the kernel interface. Tools can replace it with whatever they like, they may even have everything hardcoded.
On Tue, Aug 01, 2006 at 03:03:56PM -0400, William Cohen wrote: > Having confidience that the code works as expect would be a really > good thing for perfmon2. Currently, there has been a significant > developement on perform2 in the kernel, the user-space libraries, and > example programs demonstrating how to use it. Software packages such > as gdb and gcc have dejagnu test suites that check for previous > problems and check that the code functions as expected. There should > be similar testing available for perfmon2. These tests should be be > able to run unattended. > I have never used dejagnu myself. So I don't know how easy this is. > The perfmon2 kernel additions, shared libraries, and differences in > performance monitoring hardware make it a bit more difficult to use > some of the existing tools. Testing of user-space programs can catch > cases where the program crashes, but testing of the kernel may cause a > crash of the system and require reboot. Valgrind intercepts the > systemcalls to model the kernels actions on userspace and the > additional systemcalls for perfmon2 are not recognized by valgrind > (some patches developed to address this Valgrind issue). Some > programming tools such as cgov are not designed to work with shared > libraries. Finally, each processor implementation has its own > implentation of performance monitoring hardware, the testsuite need to > be designed to allow for the variations between processor performance > monitoring hardware. > > > Testing the Kernel code > > Want to check the following in kernel: > -Exercise the various sections of perfmon kernel code > -Invalid arguments to system calls handled gracefully > -No problems with locks > Functional testing is something we can do in a portable way. I have written a long long time ago some simple tests. They do not require knowledge about a particular PMU. It gets more difficult when you are trying to verify that measurement results are correct. You cannot simply count cycles and expect the same value over and over. You have to use an event for which you know the answer in advance. In the past, on IA-64, I have used the event that counts misaligned loads retired. This is very convenient because those do not happen frequently and you can fairly easily create a test where you know the answer. I suspect on x86 and others we may be able to construct such test cases, maybe using MMX events. But there are lots of things that can be tested without looking at the counts, just looking at permissions for instance. > Much of the kernel testing could probably be done as part of the > libpfm testing. > Well, it depends on what you are trying to test. Functional testing does not require the library. > > Testing the shared libraries, examples, and pfmon > > Want to check the following in shared libraries: > -Exercise the various sections of library and example programs > -Invalid arguments to library handled gracefully > -Check for use of uninitialized memory access past array bounds > The library interface is divded into two sections: generic and model-specific. The geenric interface is published in pfmlib.h and works for all PMU models. Then in each PMU model-specific header file you may have function specific to a PMU model. For instance, on IA-64, pfm_ita2_get_event_group() is specific to Itanium2. Those would have to be tested as well, yet they are not as critical as the generic ones. The key generic function is pfm_dispatch_events(), and unfortunately, it is difficult to use it in a generic manner, i.e., you need to know the answer which is always model specific. For instance to verify that is does the correct job for LOADS_RETIRED on Itanium2, you need to know this event can only be measured by PMC5. That means that some of the tests for this function would have to be model specific even though this is a generic function. > At OLS2006 the use of GCC's mudflap option on libpfm example code was > demonstrated and confirmed an array bounds problems found earlier in > libpfm. Patches for valgrind have also been created to handle the new > perfmon2 systemcalls this should allow checking for problems with > memory allocation. The main limitations of these tools is the code > cannot be statically linked because they replace some of the memory > allocation routines. Currently, the libpfm examples are statically > linked. > Yes, mudflap is an interesting option to gcc. Yet when I used it after OLS I had some difficulty decoding the output. I could not find a good documentation explaining how to interepret the messages. > It would be nice if valgrind had stripped down instrumentation to do > code coverage. It would be easy to get code coverage of shared > libraries then. Some people have used cachegrind to do this, but this > has significant slowdown due to the additional overhead to track > memory references and simulate the cache. The following URL outlines > what is required to make a stripped down code coverage skin, but > doesn't address mapping the data back to source code: > > wwwse.inf.tu-dresden.de/data/courses/ss06/SFT/exercise/13_slides.pdf > > Probably want to have several additional rules in the libpfm and pfmon > make file to run the tests, e.g. "check", "check_syscall", > "check_library", and "check_examples". Yes, we could certainly integrate the testing into the source tree and make empire for libpfm. Will, do you intend to take a stab at some of the tests you are proposing here? And if so, which one would you do first? Thanks. -- -Stephane _______________________________________________ perfmon mailing list [email protected] http://www.hpl.hp.com/hosted/linux/mail-archives/perfmon/
