Hello everyone, I have been briefed about the need to add an infrastructure to be able to retrieve some test files (test output, logs, etc) in Yocto/CI automation to ease debugging of some hard-to-reproduce issues. Before starting to implement a solution, I would like to get some feedback about what I have understood from the original issue and what I propose to implement to improve this. Please find below my reflexions, feel free to comment, correct, enrich and/or approve.
# Motivation Some test failures appearing in automatic testing can be hard to fix, based on too few informations from failing tests (see https://bugzilla.yoctoproject.org/show_bug.cgi?id=14901 ). Sometimes packages/binaries under test are able to generate more detailed output in a dedicated file, but there is actually no standardized way of retrieving such output files. There has been some custom efforts done to solve this issue for specific recipes, for example for lttng-tools tests : https://git.yoctoproject.org/poky/commit/?id=2ea27691aa57951aaba3cc1714a080a112d15408 The goal of this RFC is to propose an architecture for a generic system which allows to retrieve an arbitrary list of files from a test run. # Use case example lttng-tools recipe provides LTTNG tools tests as ptests (see meta/recipes-kernel/lttng/lttng-tools/run-ptest) and generate some log files (error.log, test-suite.log), it would be convenient to retrieve those files. # Proposal - Introduce an "artifacts.conf" file - this file contains a "files of interest" list to be retrieved onto host after running ptests (exact format to be defined) - artifacts.conf is specific to a ptest package and is stored next to run-ptest script in package - artifacts.conf contains, for each ptest package, the following informations: - name of file to retrieve - artifacts.conf will be installed in target image (as for package-specific run-ptest script) - each ptest package MAY have an artifacts.conf file - When running ptests, for each available ptest, ptest-runner (on the target) will read artifacts.conf if it exists, and retrieve corresponding artifacts file in an artifacts directory, with one subdirectory per ptest. - Artifacts will be aggregated only if at least one test has a status different from "PASS" - Once tests are finished, tests manager on host (ptest.py) will fetch artifacts directory. It will put this directory alongside testresults file on host - This mechanism will work for both real targets and emulated targets (Qemu), and will based on SSH to transfer artifacts back to host # Additional questions - once retrieved from target, aside from default test results directory (build/tmp/log/oeqa/<...>), where should we store artifacts ? For automated testing, should we publish them alongside other test and regression reports (in web server directory) ? - current proposal is focused on ptests: should I broaden the scope and include all kind of runtime tests (as found in meta/lib/oeqa/runtime/cases ?) - are there examples of tests producing files to retrieve with non-static names (which will then require the artifacts.conf to be smarter) ?
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#181596): https://lists.openembedded.org/g/openembedded-core/message/181596 Mute This Topic: https://lists.openembedded.org/mt/99066378/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
