[OE-Core][PATCH] oeqa/lib/utils/postactions: fix host disk usage stats retrieval

From: Alexis Lothoré The recently introduced postactions module can raise, on failing ptests, the following warning: WARNING: core-image-ptest-glib-2.0-1.0-r0 do_testimage: Can not get host disk usage: [Errno 2] No such file or directory: '/usr/bin/df' The issue is likely not happening

[OE-Core][PATCH v4 0/5] testimage: add failed test post actions and fetch more data

Hello, this small series is related to some disk space issue observed by the SWAT team (see [1]) which eventually leads to tests failure. In order to help diagnose those issues, I propose to enrich the retrieved artifacts with some additional data, like disk usage on target and on host (in case

[OE-Core][PATCH v4 5/5] oeqa/utils/postactions: testimage: add host disk usage stat as post action

From: Alexis Lothoré Since the target under test can be a virtualized guest, when some tests fail because of disk usage (see [1]), also fetch disk usage statistics from host to allow checking whether a host disk space saturation could affect running tests. [1]

[OE-Core][PATCH v4 1/5] lib/oeqa: share get_json_result_dir helper

From: Alexis Lothoré Multiple places in oeqa need to get the log output path, and redefine a small helper to accomplish this Define this helper in lib/oeqa/utils/__init__.py and import it wherever needed to allow using it. Signed-off-by: Alexis Lothoré --- There is one additional place

[OE-Core][PATCH v4 4/5] oeqa/utils/postactions: add target disk usage stat as post action

From: Alexis Lothoré In order to debug issues related to disk space (see [1]), add a failed tests post action to retrieve disk usage on the target. Rely on the test context object to run the corresponding command onto the target [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=15220

[OE-Core][PATCH v4 3/5] oeqa/utils/postactions: isolate directory creation in dedicated action

From: Alexis Lothoré In order to be able to create actions that could store new files during failed test post actions, we need to split artifacts directory creation from artifacts retrieval. Create a new dedicated action to create artifacts main directory so we can add actions creating files in

[OE-Core][PATCH v4 2/5] testimage: create a list of failed test post actions

From: Alexis Lothoré testimage is able to detect whenever a test run leads to some tests failing, and execute some actions in this case. The only action currently defined in such case is to retrieve artifacts from the target under test, as listed in TESTIMAGE_FAILED_QA_ARTIFACTS In order to be

Re: [OE-Core][PATCH v3 1/5] lib/oeqa: share get_json_result_dir helper

On 2/23/24 18:27, Richard Purdie wrote: > On Fri, 2024-02-23 at 16:16 +0100, Alexis Lothoré via > lists.openembedded.org wrote: >> From: Alexis Lothoré >> >> Multiple places in oeqa need to get the log output path, and redefine >> a >> small helper to accomplish this >> >> Define this helper in

[OE-Core][PATCH v3 0/5] testimage: add failed test post actions and fetch more data

Hello, this small series is related to some disk space issue observed by the SWAT team (see [1]) which eventually leads to tests failure. In order to help diagnose those issues, I propose to enrich the retrieved artifacts with some additional data, like disk usage on target and on host (in case

[OE-Core][PATCH v3 3/5] oeqa/utils/postactions: isolate directory creation in dedicated action

From: Alexis Lothoré In order to be able to create actions that could store new files during failed test post actions, we need to split artifacts directory creation from artifacts retrieval. Create a new dedicated action to create artifacts main directory so we can add actions creating files in

[OE-Core][PATCH v3 4/5] oeqa/utils/postactions: add target disk usage stat as post action

From: Alexis Lothoré In order to debug issues related to disk space (see [1]), add a failed tests post action to retrieve disk usage on the target. Rely on the test context object to run the corresponding command onto the target [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=15220

[OE-Core][PATCH v3 5/5] oeqa/utils/postactions: testimage: add host disk usage stat as post action

From: Alexis Lothoré Since the target under test can be a virtualized guest, when some tests fail because of disk usage (see [1]), also fetch disk usage statistics from host to allow checking whether a host disk space saturation could affect running tests. [1]

[OE-Core][PATCH v3 2/5] testimage: create a list of failed test post actions

From: Alexis Lothoré testimage is able to detect whenever a test run leads to some tests failing, and execute some actions in this case. The only action currently defined in such case is to retrieve artifacts from the target under test, as listed in TESTIMAGE_FAILED_QA_ARTIFACTS In order to be

[OE-Core][PATCH v3 1/5] lib/oeqa: share get_json_result_dir helper

From: Alexis Lothoré Multiple places in oeqa need to get the log output path, and redefine a small helper to accomplish this Define this helper in lib/oeqa/utils/__init__.py and import it wherever needed to allow using it. Signed-off-by: Alexis Lothoré --- There is one additional place

Re: [OE-Core][PATCH v2 2/5] testimage: create a list of failed test post actions

On 2/23/24 15:42, Richard Purdie wrote: > On Fri, 2024-02-23 at 15:03 +0100, Alexis Lothoré via lists.openembedded.org > wrote: >> From: Alexis Lothoré >> >> testimage is able to detect whenever a test run leads to some tests >> failing, and execute some actions in this case. The only action

[OE-Core][PATCH v2 0/5] testimage: add failed test post actions and fetch more data

Hello, this small series is related to some disk space issue observed by the SWAT team (see [1]) which eventually leads to tests failure. In order to help diagnose those issues, I propose to enrich the retrieved artifacts with some additional data, like disk usage on target and on host (in case

[OE-Core][PATCH v2 5/5] oeqa/utils/postactions: testimage: add host disk usage stat as post action

From: Alexis Lothoré Since the target under test can be a virtualized guest, when some tests fail because of disk usage (see [1]), also fetch disk usage statistics from host to allow checking whether a host disk space saturation could affect running tests. [1]

[OE-Core][PATCH v2 4/5] oeqa/utils/postactions: add target disk usage stat as post action

From: Alexis Lothoré In order to debug issues related to disk space (see [1]), add a failed tests post action to retrieve disk usage on the target. Rely on the test context object to run the corresponding command onto the target [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=15220

[OE-Core][PATCH v2 2/5] testimage: create a list of failed test post actions

From: Alexis Lothoré testimage is able to detect whenever a test run leads to some tests failing, and execute some actions in this case. The only action currently defined in such case is to retrieve artifacts from the target under test, as listed in TESTIMAGE_FAILED_QA_ARTIFACTS In order to be

[OE-Core][PATCH v2 3/5] oeqa/utils/postactions: isolate directory creation in dedicated action

From: Alexis Lothoré In order to be able to create actions that could store new files during failed test post actions, we need to split artifacts directory creation from artifacts retrieval. Create a new dedicated action to create artifacts main directory so we can add actions creating files in

[OE-Core][PATCH v2 1/5] lib/oeqa: share get_json_result_dir helper

From: Alexis Lothoré Multiple places in oeqa need to get the log output path, and redefine a small helper to accomplish this Define this helper in lib/oeqa/utils/__init__.py and import it wherever needed to allow using it. Signed-off-by: Alexis Lothoré --- There is one additional place

Re: [OE-Core][PATCH 1/4] testimage: create a list of failed test post actions

On 2/21/24 08:56, Richard Purdie wrote: > On Wed, 2024-02-21 at 08:53 +0100, Alexis Lothoré wrote: >> On 2/21/24 08:34, Richard Purdie wrote: >>> On Tue, 2024-02-20 at 21:01 +0100, Alexis Lothoré via >>> lists.openembedded.org wrote: From: Alexis Lothoré >> >> [...] >>

Re: [OE-Core][PATCH 1/4] testimage: create a list of failed test post actions

Hello Richard, On 2/21/24 08:34, Richard Purdie wrote: > On Tue, 2024-02-20 at 21:01 +0100, Alexis Lothoré via > lists.openembedded.org wrote: >> From: Alexis Lothoré [...] >> +## >> +# General post actions runner >>

[OE-Core][PATCH 3/4] testimage: add target disk usage stat as post action

From: Alexis Lothoré In order to debug issues related to disk space (see [1]), add a failed tests post action to retrieve disk usage on the target. Rely on the test context object to run the corresponding command onto the target [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=15220

[OE-Core][PATCH 4/4] testimage: add host disk usage stat as post action

From: Alexis Lothoré Since the target under test can be a virtualized guest, when some tests fail because of disk usage (see [1]), also fetch disk usage statistics from host to allow checking whether a host disk space saturation could affect running tests. [1]

[OE-Core][PATCH 0/4] testimage: add failed test post actions and fetch more data

Hello, this small series is related to some disk space issue observed by the SWAT team (see [1]) which eventually leads to tests failure. In order to help diagnose those issues, I propose to enrich the retrieved artifacts with some additional data, like disk usage on target and on host (in case

[OE-Core][PATCH 2/4] testimage: isolate artifacts directory creation in dedicated post action

From: Alexis Lothoré In order to be able to create actions that could store new files during failed test post actions, we need to split artifacts directory creation from artifacts retrieval. Create a new dedicated action to create artifacts main directory so we can add actions creating files in

[OE-Core][PATCH 1/4] testimage: create a list of failed test post actions

From: Alexis Lothoré testimage is able to detect whenever a test run leads to some tests failing, and execute some actions in this case. The only action currently defined in such case is to retrieve artifacts from the target under test, as listed in TESTIMAGE_FAILED_QA_ARTIFACTS In order to be

[OE-Core][PATCH] testimage: log exception when failing to retrieve artifacts

From: Alexis Lothoré Despite managing to retrieve the failed ptests artifacts, testimage seems to dump some retrieval errors like the following one: WARNING: core-image-ptest-valgrind-1.0-r0 do_testimage: Can not retrieve /usr/lib/valgrind/ptest from test target Log the corresponding exception

Re: [OE-Core][PATCH 1/2] testimage: move TESTIMAGE_FAILED_QA_ARTIFACTS default value to core-image-minimal

On 1/24/24 17:20, Richard Purdie wrote: > On Wed, 2024-01-24 at 15:29 +0100, Alexis Lothoré via > lists.openembedded.org wrote: >> From: Alexis Lothoré [...] >> + >> +# When any test fails, TESTIMAGE_FAILED_QA ARTIFACTS will be parsed and for >> +# each entry in it, if artifact pointed by path

[OE-Core][PATCH 0/2] testimage: enable artifacts retrieval for failed tests

Hello, this small series is a very late follow-up to my initial artifacts retrieval series (sorry for the delay). The initial series can be found in [1]. The goal is to retrieve some specific files from target whenever some tests fail, to ease debugging. The feature relies on a new

[OE-Core][PATCH 2/2] core-image-ptest: retrieve ptests directory when ptests fail

From: Alexis Lothoré TESTIMAGE_FAILED_QA_ARTIFACTS is defined in core-image-minimal with a minimal list of files to retrieve whenever a runtime test fails. By appending the ptest directory only in core-image-ptest.bb, thanks to multiconfig feature used in the recipe, only failing ptests will

[OE-Core][PATCH 1/2] testimage: move TESTIMAGE_FAILED_QA_ARTIFACTS default value to core-image-minimal

From: Alexis Lothoré TESTIMAGE_FAILED_QA_ARTIFACTS currently sets a default list of files to be saved whenever some tests fail. Unfortunately, this default value is very easily discarded, because TESTIMAGE_FAILED_QA_ARTIFACTS is expected to be enriched by some core recipes (example: ptest

Re: [OE-Core][PATCH] scripts/resulttool: group all regressions in regression report

Hi Richard, sorry for the late reply. I see that you have decided to apply the patch in the mean time, but here are my comments On 11/4/23 04:13, Richard Purdie wrote: > On Fri, 2023-11-03 at 13:50 -0700, Alexis Lothoré via > lists.openembedded.org wrote: >> From: Alexis Lothoré >> 5

[OE-Core][PATCH] scripts/resulttool: group all regressions in regression report

From: Alexis Lothoré Commit c304fcbe0588b1078373558c2ddf36064bcdf214 introduced a grouping when listing regressions. This grouping has been added only for ptests. It has been observed that any other kind of tests could benefit from it. For example, current regression reports can show the

[OE-Core][PATCH v2 2/4] scripts/resulttool: rearrange regressions report order

From: Alexis Lothoré Regressions reports currently reports matching pairs and improved pairs first, then regressions. Change order to print regressions first, which is the most valuable info in the report, and then print improvements and matches at the bottom. Signed-off-by: Alexis Lothoré

[OE-Core][PATCH v2 4/4] scripts/yocto_testresults_query: add option to change display limit

From: Alexis Lothoré Add a "-l"/"--limit" option to allow changing the display limit in resulttool. - If no value is passed, resulttool uses its default value. - If 0 is passed, the display limit is removed and every regression will be displayed - If a custom value is passed, this value

[OE-Core][PATCH v2 3/4] scripts/resulttool: make additional info more compact

From: Alexis Lothoré Since "matched" and "improved" tests are not as important as regressions, reduce the place they take in the regression report: - merge "matched" and "improved" tests, while removing the label - add a single line of additional info per pair Those changes make the "Matches

[OE-Core][PATCH v2 1/4] scripts/resulttool: limit the number of changes displayed per test

From: Alexis Lothoré Most of the changes list generated in regression reports fall in one of the two following categories: - there is only a few (<10) changes listed and the info is valuable/relevant - the list is huge (> 100 ? 1000 ?) and basically tells us that the whole tests category

[OE-Core][PATCH v2 0/4] Add a display limit for regression report generation

It has been observed that useful information in regression report can be drowned in huge regression lists which are often false-positives (for example, a whole set of tests has been temporarily disabled). Additionally, the regression report usually starts with not-so-useful information when

Re: [OE-Core][PATCH 0/2] Add a display limit for regression report generation

Hello Alexandre, On 10/20/23 08:05, Alexandre Belloni via lists.openembedded.org wrote: > Hello Alexis,> > https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5886/steps/14/logs/stdio > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5935/steps/14/logs/stdio >

[OE-Core][PATCH 0/2] Add a display limit for regression report generation

It has been observed that useful information in regression report can be drowned in huge regression lists which are often false-positives (for example, a whole set of tests has been temporarily disabled). This series brings a default limit to how many changes are displayed per base/target

[OE-Core][PATCH 2/2] scripts/yocto_testresults_query: add option to change display limit

From: Alexis Lothoré Add a "-l"/"--limit" option to allow changing the display limit in resulttool. - If no value is passed, resulttool uses its default value. - If 0 is passed, the display limit is removed and every regression will be displayed - If a custom value is passed, this value

[OE-Core][PATCH 1/2] scripts/resulttool: limit the number of changes displayed per test

From: Alexis Lothoré Most of the changes list generated in regression reports fall in one of the two following categories: - there is only a few (<10) changes listed and the info is valuable/relevant - the list is huge (> 100 ? 1000 ?) and basically tells us that the whole tests category

Re: [OE-Core][PATCH 0/2] Fix regression reporting for master-next

On 10/10/23 11:30, Alexis Lothoré via lists.openembedded.org wrote: > With those two patches, I have been able to properly generate the > regression report from [1] with the following command: It looks like I forgot to paste the relevant command. Here it is, for documentation purpose:

[OE-Core][PATCH 2/2] oeqa/utils/gitarchive: ensure tag matches regex before getting its fields

From: Alexis Lothoré Whenever we ask gitarchive to retrieve test results for specific revisions, we first do a "large" search in get_tags, which uses glob patterns with git ls-remote, and then we filter received tags with a regex to parse the tags fields. Currently gitarchive assumes that all

[OE-Core][PATCH 0/2] Fix regression reporting for master-next

There are still issues in Autobuilder when trying to generate regression reports on master-next branch, as visible in [1]. This issue makes build status as failed After being finally able to replicate the issue locally (which is quite difficult since master-next is a force-pushed branch), I

[OE-Core][PATCH 1/2] oeqa/utils/gitarchive: fix tag pattern searching

From: Alexis Lothoré Whenever we ask gitarchive to search for tags, we can provide it with a pattern (containing glob patterns). However, when searching for example for tags matching branch master-next, it can find more tags which does not correspond exactly to branch master-next (e.g.

[OE-core] [PATCH] scripts/resulttool: do not try to parse metadata as tests results

From: Alexis Lothoré When regression report is computed during a CI build, a lot of errors often appears regarding missing test status: ERROR: Failed to retrieved base test case status: ptestresult.sections ERROR: Failed to retrieved base test case status: ptestresult.sections ERROR: Failed to

Re: [OE-Core][PATCH] oeqa/utils/gitarchive: replace warning with info when reading local tags

Hi Richard, On 8/25/23 08:41, Richard Purdie wrote: > On Thu, 2023-08-24 at 11:18 +0200, Alexis Lothoré via > lists.openembedded.org wrote: >> From: Alexis Lothoré >> >> Whenever a script needs to list tags, if it falls back to the third method >> (reading local tags only), it emits a warning.

[OE-Core][PATCH] oeqa/utils/gitarchive: replace warning with info when reading local tags

From: Alexis Lothoré Whenever a script needs to list tags, if it falls back to the third method (reading local tags only), it emits a warning. While this warning is useful for future diagnostic if some tagging issues re-appear, it makes buildperf autobuilds status as "Passed with warnings",

[OE-Core][PATCH 0/2] oeqa/utils/gitarchive: put back local tags listing as third tag listing method

Hello, this is another attempt to properly fix tag computation in Autobuilder. First attempt ([1]) completely replaced "git tag" with "git ls-remote" to make sure to know about all tags when computing new tag name. It raised some issue because oe-git-archive works with local repositories with no

[OE-Core][PATCH 1/2] oeqa/utils/gitarchive: allow to pass a logger to get_tags

From: Alexis Lothoré Propagate a "log" parameter to get_tags in order to know what method is used to retrieve existing tags Signed-off-by: Alexis Lothoré --- meta/lib/oeqa/selftest/cases/gitarchivetests.py | 13 ++--- meta/lib/oeqa/utils/gitarchive.py | 11 ++- 2

[OE-Core][PATCH 2/2] oeqa/utils/gitarchive: fall back to local tags when listing existing tags

From: Alexis Lothoré e9cff55e73cc has switched tag listing from bare "git tag" to "git ls-remote" to make sure not to miss remote tags which are not fetched locally. This mechanism first checks for configured remote repository, next for possibly passed url, and then fails if none worked.

Re: [OE-Core][PATCH v2 3/3] oeqa/selftest/gitarchive: add tests about tags lisiting when no remote is configured

Hello Richard, On 8/18/23 16:17, Alexis Lothoré via lists.openembedded.org wrote: > From: Alexis Lothoré > > Add specific tests on gitarchive for when tag listing is required but no > remote is configured in target directory: it should either succeed if valid > url is provided, or fail is url

[OE-Core][PATCH v2 0/3] oeqa/utils/gitarchive: fix tag name computation

Hello, this series brings a fix to a sporadic tag push issue observed in autobuilder. The bug is documented in bugzilla #15140 ([1]). Basically, whenever the autobuilder creates a new tag on test results, it is only aware of "local" tags, which is kind of faulty since used repository is a shallow

[OE-Core][PATCH v2 1/3] oeqa/selftest: introduce gitarchive tests

From: Alexis Lothoré Add a test suite for gitarchive.py. For now, only introduce tests on methods which needs to read existing tags The tests rely on tmpdirs to create local, "fake" results repository in order to allow basic git commands Signed-off-by: Alexis Lothoré --- V2: no change ---

[OE-Core][PATCH v2 3/3] oeqa/selftest/gitarchive: add tests about tags lisiting when no remote is configured

From: Alexis Lothoré Add specific tests on gitarchive for when tag listing is required but no remote is configured in target directory: it should either succeed if valid url is provided, or fail is url is not provided or wrong Signed-off-by: Alexis Lothoré ---

[OE-Core][PATCH v2 2/3] oeqa/utils/gitarchive: fix tag computation when creating archive

From: Alexis Lothoré Sporadic errors have been observed in autobuilder when trying to store new tests results: error: failed to push some refs to 'push.yoctoproject.org:yocto-testresults' hint: Updates were rejected because the tag already exists in the remote. The new tag name is generated by

Re: [OE-Core][PATCH 2/2] oeqa/utils/gitarchive: fix tag computation when creating archive

Hello Richard, On 8/15/23 23:13, Richard Purdie wrote: > On Fri, 2023-08-11 at 14:55 +0200, Alexis Lothoré via > lists.openembedded.org wrote: >> From: Alexis Lothoré >> >> Sporadic errors have been observed in autobuilder when trying to store new >> tests results: >> >> error: failed to push

[OE-Core][PATCH 0/2] oeqa/utils/gitarchive: fix tag name computation

Hello, this series brings a fix to a sporadic tag push issue observed in autobuilder. The bug is documented in bugzilla #15140 ([1]). Basically, whenever the autobuilder creates a new tag on test results, it is only aware of "local" tags, which is kind of faulty since used repository is a shallow

[OE-Core][PATCH 2/2] oeqa/utils/gitarchive: fix tag computation when creating archive

From: Alexis Lothoré Sporadic errors have been observed in autobuilder when trying to store new tests results: error: failed to push some refs to 'push.yoctoproject.org:yocto-testresults' hint: Updates were rejected because the tag already exists in the remote. The new tag name is generated by

[OE-Core][PATCH 1/2] oeqa/selftest: introduce gitarchive tests

From: Alexis Lothoré Add a test suite for gitarchive.py. For now, only introduce tests on methods which needs to read existing tags The tests rely on tmpdirs to create local, "fake" results repository in order to allow basic git commands Signed-off-by: Alexis Lothoré ---

[OE-Core][PATCH 0/2] add custom string when regression is about missing test result

From: Alexis Lothoré Hello, this very small series intend to eliminate confusion about regressions reported by resulttool whenever a test result is present in "base test result" but absent from "target test result". This scenario leads to many "XXX -> None" lines. The series allows to define

[OE-Core][PATCH 1/2] scripts/resulttool: allow to replace test raw status with custom string

From: Alexis Lothoré Add a STATUS_STRINGS dictionnary matching raw statuses to custom strings. Whenever a regression must be reported, raw status is searched in the custom statuses dict (key search is case insensitive). If no custom string is found, raw status is kept and used in regression

[OE-Core][PATCH 2/2] scripts/resulttool: define custom string for "not found" test results

From: Alexis Lothoré Whenever a test result is present in base test result but absent from target test results, we have an entry in regression report looking like the following one: ptestresult.apr.testfile: PASSED -> None The "None" status may be a bit confusing, so replace it with a

[OE-Core][PATCH] scripts/resulttool: add mention about new detected tests

From: Alexis Lothoré Some regression reports show a lot of "PASSED->None" transitions. When such big lot of identical transitions are observed, it could be that tests are now failing, but it could also be that some tests has been renamed. To detect such case, add a log in regression report to

Re: [OE-Core][PATCH v3 4/4] core-image-ptest: append ptest directory to artifacts list

Hi Mikko, On 6/9/23 08:52, Mikko Rapeli wrote: > Hi, > > On Fri, Jun 09, 2023 at 08:48:02AM +0200, Alexis Lothoré via > lists.openembedded.org wrote: >> From: Alexis Lothoré >> >> TESTIMAGE_FAILED_QA_ARTIFACTS is defined in testimage.bbclass with a >> minimal list of files to retrieve when a

[OE-Core][PATCH v3 4/4] core-image-ptest: append ptest directory to artifacts list

From: Alexis Lothoré TESTIMAGE_FAILED_QA_ARTIFACTS is defined in testimage.bbclass with a minimal list of files to retrieve when a test fail. By appending the ptest directory only in core-image-ptest.bb, thanks to multiconfig feature used in the recipe, only failing ptests will lead to

[OE-Core][PATCH v3 1/4] oeqa/core/runner: add helper to know about expected failures

From: Alexis Lothoré Testing framework currently uses the unittest.expectedFailure decorator for tests that can have intermittent failures (see PTEST_EXPECT_FAILURE = "1") in core-image-ptest.bb. While it allows upper layers to run tests without failing on "fragile" tests, it prevents those from

[OE-Core][PATCH v3 2/4] oeqa/target/ssh: update options for SCP

From: Alexis Lothoré By default scp expects files. Passing -r option allows to copy directories too Signed-off-by: Alexis Lothoré --- Changes since v1: - drop legacy scp protocol option --- meta/lib/oeqa/core/target/ssh.py | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git

[OE-Core][PATCH v3 3/4] testimage: implement test artifacts retriever for failing tests

From: Alexis Lothoré Add a basic artifacts retrievers in testimage class which: - triggers when at least one runtime test fails but tests execution encountered no major issue - reads a list of paths to retrieve from TESTIMAGE_FAILED_QA_ARTIFACTS - checks for artifacts presence on target -

[OE-Core][PATCH v3 0/4] add failed tests artifacts retriever

From: Alexis Lothoré This series is a proposal to bring in an "artifact retriever" to ease debugging when some runtime tests fails. This is a follow-up to the v2 ([1]) and its corresponding RFC ([2]), which in turn is a proposal to address general debugging issues like [3] The main change is

Re: [OE-Core][PATCH v2 2/2] testimage: implement test artifacts retriever for failing tests

Hello Alexander, Richard, Thanks for the feedback. Indeed my testing setup is quite minimal, so it may not reflect how many files may be pulled in real cases. On 6/7/23 11:20, Alexander Kanavin wrote: > What might work better without code complications is testimage.bbclass > setting only the

[OE-Core][PATCH v2 0/2] add failed tests artifacts retriever

This series is a proposal to bring in an "artifact retriever" to ease debugging when some runtime tests fails. This is a follow-up to the initial version ([1]) and its corresponding RFC ([2]), which in turn is a proposal to address general debugging issues like [3] In the proposed form the

[OE-Core][PATCH v2 1/2] oeqa/target/ssh: update options for SCP

By default scp expects files. Passing -r option allows to copy directories too Signed-off-by: Alexis Lothoré --- Changes since v1: - drop legacy scp protocol option --- meta/lib/oeqa/core/target/ssh.py | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git

[OE-Core][PATCH v2 2/2] testimage: implement test artifacts retriever for failing tests

Add a basic artifacts retrievers in testimage class which: - triggers when at least one runtime test fails but tests execution encountered no major issue - reads a list of paths to retrieve from TESTIMAGE_FAILED_QA_ARTIFACTS - checks for artifacts presence on target - retrieve those files over

Re: [OE-Core][RFC][PATCH 3/3] testimage: implement test artifacts retriever for failing tests

Hi Mikko, sorry for late reply, and thanks for the additional feedback On 6/2/23 15:07, Mikko Rapeli wrote: > Hi, > > These changes are an improvement, but based on my experience in product test > automation, > instead of collecting logs after testing is completed, it is better to > capture

Re: [OE-Core][RFC][PATCH 1/3] oeqa/target/ssh: update options for SCP

On 6/2/23 14:02, Richard Purdie wrote: > On Fri, 2023-06-02 at 11:50 +0200, Alexis Lothoré via > lists.openembedded.org wrote: >> Add two options for SCP: >> - by default scp expects sftp server to be available on target, which is >> not always true (e.g: core-image-minimal image). Pass -O to

Re: [OE-Core][RFC][PATCH 2/3] testimage: shut down DUT later

Hello Richard, On 6/2/23 13:55, Richard Purdie wrote: > On Fri, 2023-06-02 at 11:50 +0200, Alexis Lothoré via > lists.openembedded.org wrote: >> Move target stop (for Qemu targets: qemu image stop) later in testimage >> main function, especially _after_ having access to general test results, to >>

Re: [OE-Core][RFC][PATCH 3/3] testimage: implement test artifacts retriever for failing tests

Hello Alexander, Mikko, On 6/2/23 13:52, Mikko Rapeli wrote: > Hi, > > On Fri, Jun 02, 2023 at 01:46:03PM +0200, Alexander Kanavin wrote: >> On Fri, 2 Jun 2023 at 11:50, Alexis Lothoré via lists.openembedded.org >> wrote: >>> +# When any test fails, if path to an artifacts configuration

[OE-Core][RFC][PATCH 3/3] testimage: implement test artifacts retriever for failing tests

Add a basic artifacts retrievers in testimage class which: - triggers when at least one runtime test fails - reads a list of files to retrieve from ARTIFACTS_LIST_PATH - retrieve those files over scp thanks to existing ssh class - store those files in an "artifacts" directory in "tmp/log/oeqa/"

[OE-Core][RFC][PATCH 2/3] testimage: shut down DUT later

Move target stop (for Qemu targets: qemu image stop) later in testimage main function, especially _after_ having access to general test results, to allow executing some other actions (like retrieving some files) on DUT depending on test results (e.g. : retrieve some log files) Signed-off-by:

[OE-Core][RFC][PATCH 0/3] add failed test artifacts retriever

This series is a proposal to bring in an "artifact retriever" to ease debugging when some runtime tests fails. This is a follow-up to corresponding RFC ([1]), which in turn is a proposal to address general debugging issues like [2] In the proposed form the retriever is pretty simple/dumb: it

[OE-Core][RFC][PATCH 1/3] oeqa/target/ssh: update options for SCP

Add two options for SCP: - by default scp expects sftp server to be available on target, which is not always true (e.g: core-image-minimal image). Pass -O to force SCP to fallback to legacy SCP protocol - by default scp expects files. Passing -r option allows to copy directories too

Re: [OE-core][RFC] tests artifacts retriever

Hello Alexander, thanks for the feedback On 5/23/23 11:50, Alexander Kanavin wrote: > I think having to write a custom artifacts.conf for each ptest is not > a good direction, as it's both too much work, and requires good > understanding what is important in the output and what isn't :) When a >

Re: [OE-core][RFC] tests artifacts retriever

On 5/22/23 16:52, Alexis Lothoré via lists.openembedded.org wrote: > Hello everyone, > I have been briefed about the need to add an infrastructure to be able to > retrieve some test files (test output, logs, etc) in Yocto/CI > automation to ease debugging of some hard-to-reproduce issues. Before >

[OE-core][RFC] tests artifacts retriever

Hello everyone, I have been briefed about the need to add an infrastructure to be able to retrieve some test files (test output, logs, etc) in Yocto/CI automation to ease debugging of some hard-to-reproduce issues. Before starting to implement a solution, I would like to get some feedback about

[OE-core] [PATCH] scripts/yocto_testresults_query.py: fix regression reports for branches with slashes

From: Alexis Lothoré Regression reports are not generated on some integration branches because yocto_testresults_query.py truncates branches names with slashes when it passes it to resulttool. For example, "abelloni/master-next" is truncated to "abelloni" Fix this unwanted branch truncation by

[OE-core] [PATCH] scripts/yocto_testresults_query.py: set proper branches when using resulttool

From: Alexis Lothoré The script currently only works if base and target can be found on default branches. It breaks if we try to generate a regression report between revisions that live on different branches (as needed on integration and testing branches). For example, the following command:

Re: [OE-core] [PATCH 0/6] keep reducing regression reports noise

Hello Tim, On 3/1/23 07:24, Tim Orling wrote: > On Tue, Feb 28, 2023 at 10:10 AM Alexis Lothoré via lists.openembedded.org > > > wrote: > > From: Alexis Lothoré > > >

[OE-core] [PATCH 1/6] scripts/resulttool: call fixup_ptest_names in regression_common

From: Alexis Lothoré ptests names not only need to be fixed for regression based on git testresults but also for testsresults provided "manually" Move ptests naming fixup in regression_common to share the fixup between both regression use cases Signed-off-by: Alexis Lothoré ---

[OE-core] [PATCH 3/6] oeqa/selftest/resulttool: fix fake data

From: Alexis Lothoré resulttool searches for "status" field, not "STATUS", in results. This fix is more to avoid confusion than fixing anything, since the updated tests are about regression.can_be_compared, which does not check for "status" Signed-off-by: Alexis Lothoré ---

[OE-core] [PATCH 3/6] oeqa/selftest/resulttool: fix fake data used for testing

From: Alexis Lothoré resulttool searches for "status" field, not "STATUS", in results. This fix is more to avoid confusion than fixing anything, since the updated tests are about regression.can_be_compared, which does not check for "status" Signed-off-by: Alexis Lothoré ---

[OE-core] [PATCH 6/6] scripts/resulttool: do not count newly passing tests as regressions

From: Alexis Lothoré resulttool regression module simply compare a base test status to a target test result status. This approach raises many false positives since all XXX -> PASS transitions (XXX being any status different from PASS) are flagged as regression. - Do not list XXX -> PASS

[OE-core] [PATCH 2/6] oeqa/selftest/resulttool: fix ptest tests

From: Alexis Lothoré There are multiple issues with the tests for the ptest fixup mechanism introduced in c45d58f003e8d8b323169ca9d479dc49c43a9974: - the feature does not impact regression.can_be_compared but directly the comparison step in regression.regression_common - the "status" field was

[OE-core] [PATCH 4/6] scripts/resulttool: fix ptests results containing a non reproducible path

From: Alexis Lothoré Some ptests results may be wrongly sampled, resulting in some part of the error being propagated in test name. For example: "ptestresult.binutils-ld.in testcase

[OE-core] [PATCH 5/6] oeqa/selftest/resulttool: add test for error propagation in test name filtering

From: Alexis Lothoré Signed-off-by: Alexis Lothoré --- .../oeqa/selftest/cases/resulttooltests.py| 24 +++ 1 file changed, 24 insertions(+) diff --git a/meta/lib/oeqa/selftest/cases/resulttooltests.py b/meta/lib/oeqa/selftest/cases/resulttooltests.py index

[OE-core] [PATCH 0/6] keep reducing regression reports noise

From: Alexis Lothoré Hello, here is another batch of fixes to reduce noise in regression reports. Fixes are directly linked to main noise sources seen in 4.2_M3 regression report ([1]). - fix some existing selftests for resulttool - add more filters for ptests incorrectly logging failures. The

[OE-core] [PATCH 2/6] oeqa/selftest/resulttool: fix ptest filtering tests

From: Alexis Lothoré There are multiple issues with the tests for the ptest fixup mechanism introduced in c45d58f003e8d8b323169ca9d479dc49c43a9974: - the feature does not impact regression.can_be_compared but directly the comparison step in regression.regression_common - the "status" field was

  1   2   >