Hi, All,
We are working on a test suite to test the Linux kernel MCE handling
logic named mce-test, which can be find at:
git://git.kernel.org/pub/scm/utils/cpu/mce/mce-test.git. There is a
HOWTO for it, which is attached with the mail.
We think the testing target of mce-test is part of that of LTP, so it is
better to integrate mce-test into LTP. Do you think so?
As for integration scheme, maybe we can first add mce-test into LTP as a
separate test case, just as kdump test case. What's your opinions?
Any comment is welcome!
Best Regards,
Huang Ying
MCE test suite HOWTO
====================
11 November 2008
Huang Ying
Section 4.2 (Test with kdump test driver) is based on the README of
LTP kdump test case.
Abstract
--------
This document explains the structure and design of MCE test suite, the
kernel patch and user space tools needed for automatic tests, usage
guide and how to add new test cases into test suite.
0. Quick shortcut
------------------
- Get mcelog git version from
git://git.kernel.org/pub/scm/utils/cpu/mce/mcelog.git.
and install in /usr/sbin (or rather first in your $PATH)
git clone git://git.kernel.org/pub/scm/utils/cpu/mce/mcelog.git
cd mcelog
make
sudo make install
- Get mce-inject git version from
git://git.kernel.org/pub/scm/utils/cpu/mce/mce-inject.git.
git clone git://git.kernel.org/pub/scm/utils/cpu/mce/mce-inject.git
cd mce-inject
make
sudo make install
- Run make test-simple
This will do the basic tests, but not the more complicated kdump ones.
For more information on those read below.
1. Introduction
---------------
The MCE test suite is a collection of tools and test scripts for
testing the Linux kernel MCE processing features. The goal is to cover
most Linux kernel MCE processing code paths and features with
automation tests.
If you just want to start testing as quickly as possible, you can skip
section 2 and section 3, and go section 4.1 directly.
2. Structure
------------
The main intention behind the design is to re-use test cases amongst
various test methods (represented as test drivers), such as kdump
based, kernel MCE panic log (tolerant=3) based, etc.
2.1 Test cases
Test cases are grouped into test case classes. The test cases in one
class share the similar triggering, result collecting and result
verifying methods. They can be used in same set of test drivers. The
interface of a test case class is a shell script, usually named as
cases.sh under a sub-directory of cases/. The following command line
option should be supported by the test case class shell script:
cases.sh enumerate enumerate test cases in class, print test
case names to stdout
cases.sh trigger trigger the test case
cases.sh get_result get the result of test case
cases.sh verify verify the result of test case, and print
the verify result to stdout
When execute cases.sh [trigger|get_result|verify], the test case is
specified via environment variable this_case, which must be one of the
test case names returned by "cases.sh enumerate".
Other environment variables are also used to pass some information
from driver to test cases, such as:
this_case name current test case
driver name of test driver
klog file name which holds kernel log during test
KSRC_DIR (for gcov) kernel source code directory
GCOV (for gcov) gcov collection method
vmcore (for kdump) vmcore file name
reboot (for kdump) indicate there is a reboot between test
case trigger and test case verify, some
context has been gone.
Several test case classes are provided with the test suite. They are
all based on mce-inject MCE software injection tool, and follows same
structure.
cases/soft-inj/<class name>/cases.sh Interface of the test case class
cases/soft-inj/<class name>/data/ Directory contains data file for
mce-inject tool.
cases/soft-inj/<class name>/refer/ Directory contains data file for
reference MCE records if necessary.
For document of various test cases, please refer to doc/cases/*.
2.2 Test drivers
Test drivers drive the test procedure, its main structure is a loop
over test case classes specified in configuration file. For each test
case class, test driver loops over test cases returned by "cases.sh
enumerate". And, for each test case, it calls "cases.sh" to trigger,
get_result and verify the test case. Test driver also do some common
work for test cases, such as kdump driver collects vmcore file, and
invoking gcovdump command to get gcov data file.
The interface of test driver is driver.sh, which is usually put in
drivers/<driver_name>/ directory. The test configuration file should
be used as the only command line parameter for driver.sh. Test case
classes should be specified in test configuration file as CASES
variable, details below.
2.3 Test configuration file
Test configuration file is a shell script to specify parameters for
test drivers and test cases. It must be put in config/ directory. The
parameters are represented as shell variables as follow:
CASES Name of test case classes, separate by
white space.
START_BACKGROUND Shell command to start a background process
during testing, used for random testing.*
STOP_BACKGROUND Shell command to stop the background process
during testing.
COREDIR (for kdump) directory contains Linux kernel crash core
dump after kdump.
VMLINUX (for kdump, gcov) vmlinux of Linux kernel
GCOV (for gcov) Enable GCOV if set none zero.
KSRC_DIR (for gcov) Kernel source code directory
* To test MCE processing under random environment, a background
process can be automatically run simultaneously during MCE
testing. The start/stop command is specified via START_BACKGROUND
and STOP_BACKGROUND.
2.4 Test result
After test, the general test result will go results/<driver_name>/result.
The format of general test result is as follow:
<test case name>:
Passed: item 1 description
Failed: item 2 description
...
Passed: item n description
One blank line is used to separate test cases.
Additional test result for various test cases will go
"results/<driver_name>/<case_name>/<xxx>. For in-package test case
class, additional test results include:
results/<driver_name>/soft-inject/<case_name>/klog
Kernel log during testing
results/<driver_name>/soft-inject/<case_name>/mcelog
mcelog output during testing
results/<driver_name>/soft-inject/<case_name>/mcelog_refer
mcelog output reference
results/<driver_name>/soft-inject/<case_name>/mce_64.c.gcov (for gcov)
gcov output file
3. Tools
--------
3.1 mce-inject
mce-inject is a software MCE injection tool, which is based on Linux
kernel software MCE injection mechanism. To inject a MCE into Linux
kernel via mce-inject, a data file should be provided. The syntax is
similar to the logging output by mcelog with some extensions.
Please refer to the documentation of mce-inject for more information.
The mce-inject program must be executable in $PATH.
3.2 mcelog
mcelog read /dev/mcelog and prints the stored machine check records to
stdout. It is used by MCE test suite to verify MCE records generated
by kernel is same as reference records, at most time, same as input
records. The current git mcelog version is needed for MCE test suite to
work properly. Please refer to document of mcelog for more
information. The latest mcelog can be gotten via git snapshot from
git://git.kernel.org/pub/scm/utils/cpu/mce/mcelog.git.
Note you need the git version of mcelog available in $PATH.
3.3 readcore
readcore read crash dump core file and output memory contents at
specified virtual address with specified length or zero
terminated. readcore is used to read some kernel information from
/proc/kcore and kdump core file. The source code of readcore is put
in tools/readcore. For more information about readcore please refer to
README of readcore, which is in tools/readcore/README.
3.4 gcovdump
gcov is a test coverage tool, the original implementation is used for
user space program only. LTP (Linux Test Project) provides the kernel
gcov support. But MCE test involves panic or kdump, so gcovdump is
developed to dump gcov data from kdump crash dump core. gcovdump has
been merged by LTP cvs. For more information please refer to gcovdump
document. The latest gcovdump can be gotten from cvs:
http://ltp.cvs.sourceforge.net/viewvc/ltp/utils/analysis/gcov-kdump/.
4. Usage Guide
--------------
4.1 Test with simple test driver
4.1.1 Simple test driver
The simple test driver just call cases.sh of test cases one by one in
a loop. So it is not permitted for test cases to trigger real panic or
reboot during test. For MCE testing, a special processing mode to just
log everything in case of MCE is used for the simple test driver, it
is enabled via set MCE parameter "tolerant=3" during
testing. "tolerant" can be set via writing:
/sys/devices/system/machinecheck/machinecheck0/tolerant
4.1.2 test instruction
The following is the basic test instruction, for some additional
features such as gcov support, please refer to corresponding
instructions.
a. Linux kernel and user space tools as follow should be installed
- A Linux kernel with MCE injection support. Latest Linux kernel
MCE injection patches can be found at:
git://git.kernel.org/pub/scm/linux/kernel/git/ak/linux-mce-2.6.git
- mce-inject tool (see 3.1)
- mcelog with proper version (see 3.2)
b. Modify config/simple.conf or create a new test configuration
file. Refer to section 2.3 for more instruction about test
configuration file.
c. Run "make". Carefully check for any errors.
d. To be root and invoke simple test driver on test configuration file
as follow
drivers/simple/driver.sh config/simple.conf
e. General test result will go results/simple/result. Test log will go
work/simple/log. Additional test results for various test cases
will go results/simple/<test case>/<xxx>. For more details about
in-package test case class, please refer to section 2.1.
4.2 Test with kdump test driver
4.2.1 kdump test driver
The kdump test driver is based on the kdump test case in Linux Test
Project, thank LTP for their excellent work!
The kdump driver helps run tests which trigger crash/panic and
generate result and report via kdump. The test scripts cycle through a
series of crash/panic scenarios. Each test cycle does the following:
a. Triggers a test case which triggers crash/panic (MCE with tolerant=1).
b. Kdump kernel boots and saves a vmcore.
c. System reboots to 1st kernel.
d. Verifies test case, generate result and report.
e. After a 1 to 2 minute delay, the next test case is run.
4.2.2 test instruction
Follow the steps to setup kdump test driver.
The test driver is written for SuSE Linux Enterprise Server 10 (and
onward releases), OpenSUSE, Fedora, Debian, as well as RedHat
Enterprise Linux 5. Since KDUMP is supported by the above mentioned
distro's the test driver was written and tested on them. Contribution
towards supporting more distributions are welcome.
a. Install Linux kernel with MCE injection support. Latest Linux
kernel MCE injection patches can be found at: FIXME url.
b. Install these additional packages:
For SLES10 or OpenSUSE Distro:
* kernel-kdump
* kernel-source
* kexec-tools
For RHEL5 or Fedora distro:
* kexec-tools
* kernel-devel
c. In addition to bzImage and modules of Linux kernel should be
installed on test machine, the vmlinux of Linux kernel should be
put on test machine and specified via VMLINUX in test configuration
file.
d. Make sure the partition where the test driver is running has space
for the tests results and one vmcore file (size of physical
memory).
e. Create a new test configuration file or use a existing one in
config/, such as kdump.conf. Note: not all test case classes can be
used with kdump test driver, see "important points" below.
f. Run "make". Carefully check for any errors.
g. To be root and run "drivers/kdump/driver.sh <conf>"
h. After test is done, the test log of the last run of kdump driver will
be displayed on main console.
Few Important points to remember:
- kdump test driver request that a real panic should be triggered when
test case is triggered. So not all test case classes can be used
with kdump test driver, for example, all test case classes for
corrected MCE can not be used with kdump test driver.
- If you need to stop the tests before all test cases have run, run
"crontab -r" and "killall driver.sh" within 1 minute after the 1st
kernel reboots. Then, if you'd like to carry on tests from that point
on, run:
rm work/kdump/stamps/setupped
drivers/kdump/driver.sh <conf>
If you'd like to start tests from the beginning, run:
make reset
drivers/kdump/driver.sh <conf>
- If a failure occurs when booting the kdump kernel, you'll need to
manually reset the system so it reboots back to the 1st kernel and
continues on to the next test. For this reason, it's best to monitor
the tests from a console. If possible, setup a serial console (not a
must, any type of console setup will do). If using minicom, enable
saving of kernel messages displayed on minicom into a file, by
pressing ctrl+a+l on the console. Else, when it is observed that the
kdump kernel has failed to boot, manually copy the boot message into
a file to enable the debugging the cause of the hang.
- The results are saved in results/kdump/result, which also shows
where you are in the test run. When the "Test run complete" entry
appears in that file, you're done. Verbose log can be found at
work/log.
- The test machine would be unavailable for any other work during the
period of the test run.
4.3 Gcov support
Gcov is a test coverage tool. It can be used to discover untested
parts of program, collect branch taken statistics to optimize program,
etc. In MCE test suite, it is used to get test coverage, that is,
which C statements are covered by each test case.
Gcov support is optional, if you don't care about test coverage
information, just skip this section.
a. Make sure your kernel has gcov support. You can find lasted kernel
gcov patches from:
http://ltp.sourceforge.net/coverage/gcov.php
A README for kernel gcov can be found from:
http://ltp.sourceforge.net/coverage/gcov/readme.php
Notes: CONFIG_GCOV_ALL does not work for me. Add the line
EXTRA_CFLAGS += $(KBUILD_GCOV_FLAGS)
to the respective Makefiles are more stable. For example, this line
can be added into "linux/arch/x86/kernel/cpu/mcheck/Makefile"
b. If you want to use gcov with kdump test driver, please install
gcovdump tool(see section 3.4). The latest gcovdump can be gotten
from cvs:
http://ltp.cvs.sourceforge.net/viewvc/ltp/utils/analysis/gcov-kdump/.
c. Linux kernel source source code should be put on the test
machine. Its root directory should be specified in test
configuration file via KSRC_DIR.
d. In addition to bzImage and modules of Linux kernel should be
installed on test machine, the vmlinux of Linux kernel should be
put on test machine and specified via VMLINUX of test configuration
file.
e. Make sure gcov is available in your test system. It comes with gcc
package normally. If kdump test driver is used, a tool named
gcovdump is also needed to dump *.gcda from crash dump image.
f. In test configuration file, make sure the following setting is
available:
# enable GCOV support
GCOV=1
# kernel source is needed to get gcov graph
KSRC_DIR=<kernel source directory>
VMLINUX=<vmlinux>
g. After testing, *.c.gcov will be generated in test case result
directory, such as
results/kdump/soft-inj/non-panic/corrected/mce_64.c.gcov.
h. To merge gcov graph data from several test cases, a tool named
gcov_merge.py in tools sub-directory can be used. For example,
tools/gcov_merge results/kdump/soft-inj/*/*/mce_64.c.gcov
Will output merged gcov graph from all test cases under
soft-inj. This can be used to check coverage of several test cases.
4.4 tool
Some tools are provided to help analyze test result.
- tools/grep_result.sh
Grep from general test result (results/<driver_name>/result) in
terms of test case instead of line, because the result of one test
case may span several line.
Usage:
cat results/<driver_name>/result | tools/grep_result.sh <grep options>
Where <grep options> are same as options available to /bin/grep.
5. Add test cases
-----------------
5.1 Add test case to in-package test class
All in-package test classes use mce-inject software injection tool and
follows same structure. The steps to add a new test case is as follow:
a. Find an appropriate test case class to add your test case.
b. Add a new mce-inject data file into to cases/soft-inj/<class name>/data/.
c. If the reference mcelog is different from mce-inject input data
file, put that reference file into cases/soft-inj/<class_name>/refer/.
d. In cases/soft-inj/<class name>/cases.sh, there are shell commands
"case" in shell functions get_result() and verify(). Add a branch
in each shell command "case" for your test case.
5.2 Add test class
To add a new test class, add a cases.sh under a sub-directory of
cases/, and follow the test case class interface definition in section
2.1. The general result output format should follow that in section
2.4.
------------------------------------------------------------------------------
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list
