Re: [OE-core] [PATCH] boost: add ptest support
On 08/29/2018 01:22 AM, openembedded-core-boun...@lists.openembedded.org wrote: > On 08/28/2018 10:25 AM, Yang Wang wrote: >> On 08/27/2018 08:57 PM, Randy MacLeod wrote: >>> On 08/27/2018 06:17 PM, Yang Wang wrote: Not sure if it's worth to run the Ptest on QEMU though, I also run Ptest on SIMICS simulators, thousands of tests didn't get run, looks like the result was not good as well. Now my nightly Ptest runs on x86 device and gets consistent result every day: 2018-08-27T06:26 -- 2018-08-27T09:52 Passed: 40518 Failed: 289 Skipped: 1876 >>> >>> Consistent results are good and > 90% pass rate is very good. >>> What are the stats using qemux86-64 and/or simics? >>> >>> I don't expect that qemu results would be as close to real hardware >>> as Simics but it is quite good and freely available. >>> >> Actually, Ptest has 37 test suites as far as I know, different test >> suites spent different time on QEMU and hardware, here is a list of >> Ptest suites and their case number and spent time for running: >> >> > Just want to make it look better: > == > # Suite Name Case # Time to Run Case # Time to Run > qemu-x86-64 intel-xeon-core2 > 1 acl 2 1m 380 1m > 2 attr 143 1m 143 1m > 3 bash 79 8m 79 4m > 4 bluez5 7 6m 7 6m > 5 bzip2 6 1m 6 1m > 6 dbus-test 15 3m 15 1m > 7 diffutils 20 1m 20 2m > 8 e2fsprogs 147 9m 335 10m > 9 ethtool 2 1m 2 1m > 10 flex 114 3m 114 1m > 11 gawk 300 3m 298 2m > 12 gdbm 30 2m 25 2m > 13 glib-2.0 62 14m 220 6m > 14 gzip 51 4m 18 1m > 15 kbd 15 1m 7 1m > 16 libevent 22 6m 1 3m > 17 libpcre 34 3m 3 1m > 18 libxml2 1 1m 0 1m > 19 lzo 75 8m 5 3m > 20 mdadm 6m 6m > 21 nettle 90 3m 90 3m > 22 numactl 8 3m > 23 openssh 13 27m 47 52m > 24 openssl 87 47m 56 15m > 25 parted 64 5m 38 10m > 26 perl 101 20m 2440 47m > 27 perl5 110 17m 2406 29m > 28 python 10961 1h 5m 32323 20m > 29 rsyslog 2200 3h 37m 25 8m > 30 sed 147 1m 86 3m > 31 slang 96 1m > 32 strace 1557 1h 13m 431 6m > 33 systemd 305 9m 155 3m > 34 tcl 869 53m 206 6m > 35 tcpdump 451 7m 413 3m > 36 util-linux 516 25m 514 13m > 37 zlib 2 1m 2 1m > Overall 18080 10h 30m 40415 4h 15m > >> >> As you can see, running subset of them on QEMU could be a solution if >> people do not want to spend too much time on it or simulator is the >> preferred test device. Would it make sense using qemu to run the `make check` tests (when available) and together with ptests and auto upgrade bot (which is run every two weeks?), automate upgrading of recipes to some extent? If auto upgrade helper manages to use devtool to upgrade successfully, run these tests on image/qemu and if the tests pass (as compared against last good baseline which can be stored as metadata in AUH or the test result log repository), just send the upgraded recipe patch to list instead of sending to the recipe maintainer listed, reducing manual intervention to some extent and perhaps tested (to some extent), frequent and smaller upgrades ... Thanks, Anuj -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] boost: add ptest support
On 08/28/2018 10:25 AM, Yang Wang wrote: On 08/27/2018 08:57 PM, Randy MacLeod wrote: On 08/27/2018 06:17 PM, Yang Wang wrote: Not sure if it's worth to run the Ptest on QEMU though, I also run Ptest on SIMICS simulators, thousands of tests didn't get run, looks like the result was not good as well. Now my nightly Ptest runs on x86 device and gets consistent result every day: 2018-08-27T06:26 -- 2018-08-27T09:52 Passed: 40518 Failed: 289 Skipped: 1876 Consistent results are good and > 90% pass rate is very good. What are the stats using qemux86-64 and/or simics? I don't expect that qemu results would be as close to real hardware as Simics but it is quite good and freely available. Actually, Ptest has 37 test suites as far as I know, different test suites spent different time on QEMU and hardware, here is a list of Ptest suites and their case number and spent time for running: Just want to make it look better: == # Suite Name Case # Time to Run Case # Time to Run qemu-x86-64 intel-xeon-core2 1 acl 2 1m 380 1m 2 attr 143 1m 143 1m 3 bash 79 8m 79 4m 4 bluez5 7 6m 7 6m 5 bzip2 6 1m 6 1m 6 dbus-test 15 3m 15 1m 7 diffutils 20 1m 20 2m 8 e2fsprogs 147 9m 335 10m 9 ethtool 2 1m 2 1m 10 flex 114 3m 114 1m 11 gawk 300 3m 298 2m 12 gdbm 30 2m 25 2m 13 glib-2.0 62 14m 220 6m 14 gzip 51 4m 18 1m 15 kbd 15 1m 7 1m 16 libevent 22 6m 1 3m 17 libpcre 34 3m 3 1m 18 libxml2 1 1m 0 1m 19 lzo 75 8m 5 3m 20 mdadm 6m 6m 21 nettle 90 3m 90 3m 22 numactl 8 3m 23 openssh 13 27m 47 52m 24 openssl 87 47m 56 15m 25 parted 64 5m 38 10m 26 perl 101 20m 2440 47m 27 perl5 110 17m 2406 29m 28 python 10961 1h 5m 32323 20m 29 rsyslog 2200 3h 37m 25 8m 30 sed 147 1m 86 3m 31 slang 96 1m 32 strace 1557 1h 13m 431 6m 33 systemd 305 9m 155 3m 34 tcl 869 53m 206 6m 35 tcpdump 451 7m 413 3m 36 util-linux 516 25m 514 13m 37 zlib 2 1m 2 1m Overall 18080 10h 30m 40415 4h 15m As you can see, running subset of them on QEMU could be a solution if people do not want to spend too much time on it or simulator is the preferred test device. Thanks, -Yang Thanks, Yang Wang -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] boost: add ptest support
On 08/27/2018 08:57 PM, Randy MacLeod wrote: On 08/27/2018 06:17 PM, Yang Wang wrote: Not sure if it's worth to run the Ptest on QEMU though, I also run Ptest on SIMICS simulators, thousands of tests didn't get run, looks like the result was not good as well. Now my nightly Ptest runs on x86 device and gets consistent result every day: 2018-08-27T06:26 -- 2018-08-27T09:52 Passed: 40518 Failed: 289 Skipped: 1876 Consistent results are good and > 90% pass rate is very good. What are the stats using qemux86-64 and/or simics? I don't expect that qemu results would be as close to real hardware as Simics but it is quite good and freely available. Actually, Ptest has 37 test suites as far as I know, different test suites spent different time on QEMU and hardware, here is a list of Ptest suites and their case number and spent time for running: # Suite Name Suite Location Case # Time to Run Case # Time to Run qemu-x86-64 intel-xeon-core2 1 acl /lib64/acl/ptest/run-ptest 2 1m 380 1m 2 attr /lib64/attr/ptest/run-ptest 143 1m 143 1m 3 bash /lib64/bash/ptest/run-ptest 79 8m 79 4m 4 bluez5 /lib64/bluez5/ptest/run-ptest 7 6m 7 6m 5 bzip2 /lib64/bzip2/ptest/run-ptest 6 1m 6 1m 6 dbus-test /lib64/dbus-test/ptest/run-ptest 15 3m 15 1m 7 diffutils /lib64/diffutils/ptest/run-ptest 20 1m 20 2m 8 e2fsprogs /lib64/e2fsprogs/ptest/run-ptest 147 9m 335 10m 9 ethtool /lib64/ethtool/ptest/run-ptest 2 1m 2 1m 10 flex /lib64/flex/ptest/run-ptest 114 3m 114 1m 11 gawk /lib64/gawk/ptest/run-ptest 300 3m 298 2m 12 gdbm /lib64/gdbm/ptest/run-ptest 30 2m 25 2m 13 glib-2.0 /lib64/glib-2.0/ptest/run-ptest 62 14m 220 6m 14 gzip /lib64/gzip/ptest/run-ptest 51 4m 18 1m 15 kbd /lib64/kbd/ptest/run-ptest 15 1m 7 1m 16 libevent /lib64/libevent/ptest/run-ptest 22 6m 1 3m 17 libpcre /lib64/libpcre/ptest/run-ptest 34 3m 3 1m 18 libxml2 /lib64/libxml2/ptest/run-ptest 1 1m 0 1m 19 lzo /lib64/lzo/ptest/run-ptest 75 8m 5 3m 20 mdadm /lib64/mdadm/ptest/run-ptest 6m 6m 21 nettle /lib64/nettle/ptest/run-ptest 90 3m 90 3m 22 numactl /lib64/numactl/ptest/run-ptest 8 3m 23 openssh /lib64/openssh/ptest/run-ptest 13 27m 47 52m 24 openssl /lib64/openssl/ptest/run-ptest 87 47m 56 15m 25 parted /lib64/parted/ptest/run-ptest 64 5m 38 10m 26 perl /lib64/perl/ptest/run-ptest 101 20m 2440 47m 27 perl5 /lib64/perl5/ptest/run-ptest 110 17m 2406 29m 28 python /lib64/python/ptest/run-ptest 10961 1h 5m 32323 20m 29 rsyslog /lib64/rsyslog/ptest/run-ptest 2200 3h 37m 25 8m 30 sed /lib64/sed/ptest/run-ptest 147 1m 86 3m 31 slang /lib64/slang/ptest/run-ptest 96 1m 32 strace /lib64/strace/ptest/run-ptest 1557 1h 13m 431 6m 33 systemd /lib64/systemd/ptest/run-ptest 305 9m 155 3m 34 tcl /lib64/tcl/ptest/run-ptest 869 53m 206 6m 35 tcpdump /lib64/tcpdump/ptest/run-ptest 451 7m 413 3m 36 util-linux /lib64/util-linux/ptest/run-ptest 516 25m 514 13m 37 zlib /lib64/zlib/ptest/run-ptest 2 1m 2 1m Overall 18080 10h 30m 40415 4h 15m As you can see, running subset of them on QEMU could be a solution if people do not want to spend too much time on it or simulator is the preferred test device. Thanks, -Yang Thanks, Yang Wang -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] boost: add ptest support
On Mon, 2018-08-27 at 20:57 -0400, Randy MacLeod wrote: > On 08/27/2018 06:17 PM, Yang Wang wrote: > > Not sure if it's worth to run the Ptest on QEMU though, I also run > > Ptest > > on SIMICS simulators, thousands of tests didn't get run, looks like > > the > > result was not good as well. > > > > Now my nightly Ptest runs on x86 device and gets consistent result > > every > > day: > > > > 2018-08-27T06:26 -- 2018-08-27T09:52 > > Passed: 40518 > > Failed: 289 > > Skipped: 1876 > > Consistent results are good and > 90% pass rate is very good. > What are the stats using qemux86-64 and/or simics? > > I don't expect that qemu results would be as close to real hardware > as Simics but it is quite good and freely available. I think this is an example of where we may need to start adding "stage 2" testing to the autobuilder. The first stage tests would be enough to get patches merged and be the faster ones, the second stage would be the longer running things which we'd only trigger when stage one had passed, maybe on milestone releases. We'd put off doing this due to the old creaking codebase. With the new autobuilder codebase, this should be much more straightforward to do... boost ptest would be something I'd put in the second stage. Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] boost: add ptest support
On 08/27/2018 06:17 PM, Yang Wang wrote: Not sure if it's worth to run the Ptest on QEMU though, I also run Ptest on SIMICS simulators, thousands of tests didn't get run, looks like the result was not good as well. Now my nightly Ptest runs on x86 device and gets consistent result every day: 2018-08-27T06:26 -- 2018-08-27T09:52 Passed: 40518 Failed: 289 Skipped: 1876 Consistent results are good and > 90% pass rate is very good. What are the stats using qemux86-64 and/or simics? I don't expect that qemu results would be as close to real hardware as Simics but it is quite good and freely available. Thanks, Yang Wang -- # Randy MacLeod # Wind River Linux -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH] boost: add ptest support
Ping.
Thanks,
On 2018年05月18日 16:55, mingli...@windriver.com wrote:
From: Mingli Yu
* Add a patch to remove rpath when compiling tests.
* The rules for tests don't work in cross-compiling,
they try to run the executable after compiling and
linking the test program, so add a patch to hack
the rules, and add the running logic in run-ptest
* There are 2000+ test cases for 26 added libs, I'm
not able to make all of them work for now and I think
ptest is not necessary to cover all of them, so add
a patch to remove some of the tests, which include:
- those require extra dependencies
- those require specific target hardware
- those require input data or config files
- those compile fail with unknown reason
- those require command parameters/options when running
Signed-off-by: Jackie Huang
Signed-off-by: Mingli Yu
---
meta/recipes-support/boost/boost.inc | 66 +
.../boost/boost/0001-test-remove-some-tests.patch | 1728
.../boost/boost/boost-remove-rpath.patch | 28 +
.../boost-testing-hack-the-rules-for-ptest.patch | 53 +
meta/recipes-support/boost/boost/run-ptest | 41 +
5 files changed, 1916 insertions(+)
create mode 100644
meta/recipes-support/boost/boost/0001-test-remove-some-tests.patch
create mode 100644 meta/recipes-support/boost/boost/boost-remove-rpath.patch
create mode 100644
meta/recipes-support/boost/boost/boost-testing-hack-the-rules-for-ptest.patch
create mode 100644 meta/recipes-support/boost/boost/run-ptest
diff --git a/meta/recipes-support/boost/boost.inc
b/meta/recipes-support/boost/boost.inc
index 6984bee777..cf9a5d41e2 100644
--- a/meta/recipes-support/boost/boost.inc
+++ b/meta/recipes-support/boost/boost.inc
@@ -69,6 +69,27 @@ python __anonymous () {
d.setVar("BJAM_EXTRA", " ".join(extras))
}
+inherit ptest
+
+SRC_URI_append = " \
+file://run-ptest \
+file://boost-remove-rpath.patch \
+file://boost-testing-hack-the-rules-for-ptest.patch \
+file://0001-test-remove-some-tests.patch \
+"
+# Required by some of the tests
+FILES_${PN}-ptest += " \
+ ${libdir}/libboost_regex_extra.so.* \
+ ${libdir}/libboost_regex_recursive.so.* \
+ ${libdir}/liba.so.* \
+ ${libdir}/libsingle_instance_lib2.so.* \
+ ${libdir}/libsingle_instance_lib1.so.* \
+ ${libdir}/libdisable-lib_*.so.* \
+ ${libdir}/libdll_*.so.* \
+ ${libdir}/libthrow_test.so.* \
+"
+RDEPENDS_${PN}-ptest = "coreutils"
+
# Override the contents of specific packages
FILES_${PN}-graph_parallel = "${libdir}/libboost_graph_parallel.so.*"
FILES_${PN}-locale = "${libdir}/libboost_locale.so.*"
@@ -181,6 +202,51 @@ do_compile() {
--debug-configuration
}
+do_compile_ptest() {
+ for lib in ${BOOST_LIBS}; do
+ # ignore python3 since there is no such
+ # sub-dir in libs
+ [ ${lib} = "python3" ] && continue
+
+ cd ${S}/libs/${lib}/test
+ [ -f build/Jamfile.v2 ] && cd build
+ bjam ${BJAM_OPTS} --prefix=${prefix} \
+ --exec-prefix=${exec_prefix} \
+ --libdir=${libdir} \
+ --includedir=${includedir}
+ done
+}
+
+do_install_ptest() {
+ build_bin="${S}/${TARGET_SYS}/boost/bin.v2"
+
+ for lib in ${BOOST_LIBS}; do
+ # ignore python3 since there is no such
+ # sub-dir in libs
+ [ "${lib}" = "python3" ] && continue
+
+ install -d ${D}/${PTEST_PATH}/${lib}
+ lib_test_dir="${build_bin}/libs/${lib}/test"
+ for testdir in `ls -d ${lib_test_dir}/*.test
${lib_test_dir}/build/*.test 2>/dev/null`; do
+ testname=`basename ${testdir%.test}`
+ testfile=`find ${testdir} -name ${testname}`
+ if [ -n "${testfile}" -a -f "${testfile}" ]; then
+ install -m 0755 ${testfile}
${D}${PTEST_PATH}/${lib}
+ fi
+ done
+ done
+
+ extra_ptest_libs="libboost_regex_extra.so* libboost_regex_recursive.so.*
liba.so.* libsingle_instance_lib2.so.* libsingle_instance_lib1.so.* libdisable-lib_*.so.*
libdll_*.so* libthrow_test.so*"
+ for lib in ${extra_ptest_libs}; do
+ f=`find ${build_bin}/libs -name ${lib}`
+ if [ -n "${f}" ]; then
+ install -m 0755 ${f} ${D}${libdir}
+ fi
+ done
+
+ sed -i 's/@BOOST_LIBS@/${BOOST_LIBS}/' ${D}${PTEST_PATH}/run-ptest
+}
+
do_install() {
bjam ${BJAM_OPTS} \
--libdir=${D}${libdir} \
diff --git a/meta/recipes-support/boost/boost/0001-test-remove-some-tests.patch
b/meta/recipes-support/boost/boost/0001-test-remove-some-tests.patch
new file mode 100644
index 00..7d2f916822
--- /dev/null
+++
