I think having to write a custom artifacts.conf for each ptest is not a good direction, as it's both too much work, and requires good understanding what is important in the output and what isn't :) When a ptest fails, it would be enough to simply make a tarball of /usr/lib/component/ptest/ with everything in it. Maybe /tmp and /var/log as well.
Alex On Mon, 22 May 2023 at 16:58, Alexis Lothoré via lists.openembedded.org <[email protected]> wrote: > > On 5/22/23 16:52, Alexis Lothoré via lists.openembedded.org wrote: > > Hello everyone, > > I have been briefed about the need to add an infrastructure to be able to > > retrieve some test files (test output, logs, etc) in Yocto/CI > > automation to ease debugging of some hard-to-reproduce issues. Before > > starting to implement a solution, I would like to get some feedback about > > what I have understood from the original issue and what I propose to > > implement to improve this. Please find below my reflexions, feel free > > to comment, correct, enrich and/or approve. > > > > # Motivation > > > > Some test failures appearing in automatic testing can be hard to fix, based > > on too few informations from failing tests (see > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=14901 ). Sometimes > > packages/binaries under test are able to generate more detailed output in a > > dedicated file, but there is actually no standardized way of retrieving > > such output files. There has been some custom efforts done to solve this > > issue > > for specific recipes, for example for lttng-tools tests : > > https://git.yoctoproject.org/poky/commit/?id=2ea27691aa57951aaba3cc1714a080a112d15408 > > > > The goal of this RFC is to propose an architecture for a generic system > > which > > allows to retrieve an arbitrary list of files from a test run. > > > > # Use case example > > > > lttng-tools recipe provides LTTNG tools tests as ptests (see > > meta/recipes-kernel/lttng/lttng-tools/run-ptest) and generate some log > > files (error.log, test-suite.log), it would be convenient to retrieve those > > files. > > > > # Proposal > > > > - Introduce an "artifacts.conf" file > > - this file contains a "files of interest" list to be retrieved onto > > host after running ptests (exact format to be defined) > > - artifacts.conf is specific to a ptest package and is stored next to > > run-ptest script in package > > - artifacts.conf contains, for each ptest package, the following > > informations: > > - name of file to retrieve > > - artifacts.conf will be installed in target image (as for > > package-specific run-ptest script) > > - each ptest package MAY have an artifacts.conf file > > - When running ptests, for each available ptest, ptest-runner (on the > > target) will read artifacts.conf if it exists, and retrieve > > corresponding artifacts file in an artifacts directory, with one > > subdirectory per ptest. > > - Artifacts will be aggregated only if at least one test has a status > > different from "PASS" > > - Once tests are finished, tests manager on host (ptest.py) will fetch > > artifacts directory. It will put this directory alongside testresults > > file on host > > - This mechanism will work for both real targets and emulated targets > > (Qemu), and will based on SSH to transfer artifacts back to host > > > > # Additional questions > > > > - once retrieved from target, aside from default test results directory > > (build/tmp/log/oeqa/<...>), where should we store artifacts ? For > > automated testing, should we publish them alongside other test and > > regression reports (in web server directory) ? > > - current proposal is focused on ptests: should I broaden the scope and > > include all kind of runtime tests (as found in > > meta/lib/oeqa/runtime/cases ?) > > - are there examples of tests producing files to retrieve with non-static > > names (which will then require the artifacts.conf to be smarter) ? > > > > Fixing Alexandre Belloni's address in CC, sorry for the noise > > -- > Alexis Lothoré, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#181633): https://lists.openembedded.org/g/openembedded-core/message/181633 Mute This Topic: https://lists.openembedded.org/mt/99066378/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
