On Tue, 2023-10-31 at 08:16 -0700, Tim Orling wrote: > > > On Mon, Oct 30, 2023 at 12:51 PM Alassane Yattara > <[email protected]> wrote: > > Added pytest init file to configure pytest and generate test > > reports. > > > > > Typo: pytest.ini file? > > > > > Objective > > - We propose to integrate the plugin Pytest to generate test > > reports for Toaster application. > > - To keep configurations to a minimal, we propose to integrate > > Pytest with Tox(tox.ini). > > - Tox will automate the execution of the Pytest test cases. > > > > Pros > > - Generate test reports > > - Create a historical record of test results over time > > - Track Toasters stability and quality > > - Documentation and Transparency > > - Debugging, troubleshooting : identify regressions (input, > > output,) > > > > Cons > > - Limited predefined report options. Although Pytest provides > > different report > > formats, its default reports are rather straightforward, We might > > have to put extra effort into crafting them ourselves. > > > > > One of the benefits of pytest is in fact that report plugins are > available and fairly straightforward to implement if we need our own > custom report format (although we should avoid custom plugins and use > upstream maintained ones wherever possible). > > We do need to ensure that plain text output is generated for > consumption by the AutoBuilder. Fancy html output is going to clutter > the testresults.json that will be where these test results will > ultimately be stored: > https://git.yoctoproject.org/poky/plain/scripts/resulttool > https://git.yoctoproject.org/yocto-testresults/ > https://docs.yoctoproject.org/test-manual/understand-autobuilder.html?highlight=resulttool#resulttool > > The reason is that the testresults.json is consumed by tooling to > look for regressions. It must be machine readable and easily > processed.
Tim is correct that we do need text/machine readable results for tesresults.json but there is also a case for generating graphical/html reports alongside the results. An example would be our build performance tests: https://autobuilder.yocto.io/pub/non-release/20231030-22/testresults/buildperf-alma8/perf-alma8_master_20231030150044_28324600cb.html where we've saved text data but also show it more visually. Cheers, Richard
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#5925): https://lists.yoctoproject.org/g/toaster/message/5925 Mute This Topic: https://lists.yoctoproject.org/mt/102299113/21656 Group Owner: [email protected] Unsubscribe: https://lists.yoctoproject.org/g/toaster/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
