On Sat, 2023-02-25 at 16:59 +0100, Alexis Lothoré wrote: > Hello Richard, > as usual, thanks for the prompt feedback ! > > On 2/25/23 13:32, Richard Purdie wrote: > > On Sat, 2023-02-25 at 09:15 +0000, Richard Purdie via > > lists.openembedded.org wrote: > > > On Fri, 2023-02-24 at 18:06 +0000, Richard Purdie via > > > lists.openembedded.org wrote: > > > > Hi Alexis, > > > > > > > > Firstly, this looks very much improved, thanks. It is great to start to > > > > see some meaningful data from this. > > > > > > > > On Fri, 2023-02-24 at 17:45 +0100, Alexis Lothoré via > > > > lists.openembedded.org wrote: > > > > > After manual inspection on some entries, the remaining oeselftest > > > > > regression > > > > > raised in the report seems valid. There are still some issues to > > > > > tackle: > > > > > - it seems that now one major remaining source of noise is on the > > > > > "runtime" > > > > > tests (comparison to tests not run on "target" results) > > > > > - when a ptest managed by oe-selftest fails, I guess the remaining > > > > > tests are not > > > > > run, so when 1 failure is logged, we have many "PASSED->None" > > > > > transitions in > > > > > regression report, we should probably silence it. > > > > > - some transitions appear as regression while those are in fact > > > > > improvements > > > > > (e.g: "UNRESOLVED->PASSED") > > > > > > > > I had quick play. Firstly, if I try "yocto_testresults_query.py > > > > regression-report 4.2_M1 4.2_M2" in an openembedded-core repository > > > > instead of poky, it breaks. That isn't surprising but we should either > > > > make it work or show a sensible error. > > Oh right, I am working in a Poky build configuration, so I have assumed that > this > would be the unique use case. > Since the test results commits are tightly coupled to revisions in poky (so > not > oecore), I plan to merely log an error about not found revision (and > suggesting > the user to check that the repository is poky and not oecore). > But please let me know if I miss a major use case here and that a smarter > fallback plan (shallow-clone poky if we are running in oecore ?) is needed
I'm happy to for it just to give an human readable error, someone can add this functionality if they need/want it. > > > I think I might be tempted to merge this series and then we can change > > > the code to improve from here as this is clearly a vast improvement on > > > where we were! Improvements can be incremental on top of these changes. > > I am in favor of this :) If it is OK for you, I will just re-submit a series > with > the fix for the proper error logging when running the tool from oecore and > not poky. > > Next we could introduce all the suggestions you have suggested, but I feel > that > with the quick increase of "hotfixes" count to support issues with older test > results, and for the sake of maintainability of resulttool and its submodules, > those specific hotfixes need to be properly isolated (and documented), like > in a > "regression_quirks.py" or something like that. What do you think ? I'm hoping we don't have many of these quirks. We have a huge history at this point so it would be sad if the tool can't work with it. From what I've seen so far, we can manage with the code in the regression module itself. I've tried to add some comments. I wondered what to do with this series since I needed to get M3 built. Since this series was available and mostly usable, it would be better to have a nicer report this time, it is a good test of the code. In the end I've merged most of it, along with my two tweaks to handle LTP and the bigger ptest results issue. I couldn't take one set of the selftests since they simply don't work. This will give us a useful realworld test of the M3 report. I'm working on the assumption you'll send a follow up series with the tests, the oe-core check and some of the other issues I've mentioned in other emails? Cheers, Richard
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#177733): https://lists.openembedded.org/g/openembedded-core/message/177733 Mute This Topic: https://lists.openembedded.org/mt/97209732/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
