Tim, > > Typo: pytest.ini file? Non, pytest.init is a config file for pytest, next step is to merge this file with tox.init. tox.ini file are the configuration files of the tox project, and can also be used to hold pytest configuration if they have a [pytest] section.
----- Mail original ----- De: "Richard Purdie" <[email protected]> À: "Tim Orling" <[email protected]>, "Alassane Yattara" <[email protected]> Cc: "bitbake-devel" <[email protected]>, "toaster" <[email protected]> Envoyé: Mardi 31 Octobre 2023 16:36:42 Objet: Re: [Toaster] [bitbake-devel] [PATCH 8/8] Toaster: Added pytest.init file On Tue, 2023-10-31 at 08:16 -0700, Tim Orling wrote: > > > On Mon, Oct 30, 2023 at 12:51 PM Alassane Yattara > <[email protected]> wrote: > > Added pytest init file to configure pytest and generate test > > reports. > > > > > Typo: pytest.ini file? > > > > > Objective > > - We propose to integrate the plugin Pytest to generate test > > reports for Toaster application. > > - To keep configurations to a minimal, we propose to integrate > > Pytest with Tox(tox.ini). > > - Tox will automate the execution of the Pytest test cases. > > > > Pros > > - Generate test reports > > - Create a historical record of test results over time > > - Track Toasters stability and quality > > - Documentation and Transparency > > - Debugging, troubleshooting : identify regressions (input, > > output,) > > > > Cons > > - Limited predefined report options. Although Pytest provides > > different report > > formats, its default reports are rather straightforward, We might > > have to put extra effort into crafting them ourselves. > > > > > One of the benefits of pytest is in fact that report plugins are > available and fairly straightforward to implement if we need our own > custom report format (although we should avoid custom plugins and use > upstream maintained ones wherever possible). > > We do need to ensure that plain text output is generated for > consumption by the AutoBuilder. Fancy html output is going to clutter > the testresults.json that will be where these test results will > ultimately be stored: > https://git.yoctoproject.org/poky/plain/scripts/resulttool > https://git.yoctoproject.org/yocto-testresults/ > https://docs.yoctoproject.org/test-manual/understand-autobuilder.html?highlight=resulttool#resulttool > > The reason is that the testresults.json is consumed by tooling to > look for regressions. It must be machine readable and easily > processed. Tim is correct that we do need text/machine readable results for tesresults.json but there is also a case for generating graphical/html reports alongside the results. An example would be our build performance tests: https://autobuilder.yocto.io/pub/non-release/20231030-22/testresults/buildperf-alma8/perf-alma8_master_20231030150044_28324600cb.html where we've saved text data but also show it more visually. Cheers, Richard
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#5931): https://lists.yoctoproject.org/g/toaster/message/5931 Mute This Topic: https://lists.yoctoproject.org/mt/102299113/21656 Group Owner: [email protected] Unsubscribe: https://lists.yoctoproject.org/g/toaster/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
