Hi Ludwig, On Mon, 18 May 2026 at 10:07, Ludwig Nussel <[email protected]> wrote: > > On 5/15/26 15:41, Simon Glass wrote: > > Hi Ludwig, > > > > On 2026-05-13T14:08:10, Ludwig Nussel <[email protected]> wrote: > > > >> The tests were created with the help of copilot and manual fixups. I > >> have to admit that the log output of those tests is not something I can > >> grok. I've left the tags copilot added in the commits even though > >> checkpatch complains. Not sure what the policy is. > > > > Thanks for being upfront. For my work I have adjusted patman to drop these > > tags. > > > > The "log output I can't grok" part worries me more than the tag. If > > you can't follow what the test is exercising, neither can the next > > person debugging a failure :-) > > Meanwhile I got it. The AI helped me to get started. > Anyway, here is what confused me. Suppose we introduce a small typo: > - 'No signature node found', False) > + 'No signature node fond', False) > > Then run > $ test/py/test.py --bd sandbox "$@" -k 'test_vboot -vvx > > You'll get a dump of all kinds of irrelevant things like openssl key > generation or u-boot startup. Then finally: > > FAILED > test/py/tests/test_vboot.py::test_vboot[sha1-basic-sha1--None-False-True-False-False] > - assert 'No signature node fond' in "1769 bytes read in 0 msWorking FDT > set to 100image_pre_load_sig_setup() INFO: no info for image pre-load > sig check\r\r\n## Loading kernel (any) from FIT Image at 00000100 > ...\r\r\n Using 'conf-1' configuration\r\r\n Verifying Hash > Integrity ... fit_config_verify_required_keys() No signature node found: > FDT_ERR_NOTFOUND\r\r\nBad Data Hash\r\r\nERROR -2: No such file or > directory: can't get kernel image!" > + where "1769 bytes read in 0 msWorking FDT set to > 100image_pre_load_sig_setup() INFO: no info for image pre-load sig > check\r\r\n## Loading kernel (any) from FIT Image at 00000100 ...\r\r\n > Using 'conf-1' configuration\r\r\n Verifying Hash Integrity ... > fit_config_verify_required_keys() No signature node found: > FDT_ERR_NOTFOUND\r\r\nBad Data Hash\r\r\nERROR -2: No such file or > directory: can't get kernel image!" = <built-in method join of str > object at 0xa46de8>(['1769 bytes read in 0 ms', 'Working FDT set to > 100', "image_pre_load_sig_setup() INFO: no info for image pre-load sig > check\r\r\n## Loading kernel (any) from FIT Image at 00000100 ...\r\r\n > Using 'conf-1' configuration\r\r\n Verifying Hash Integrity ... > fit_config_verify_required_keys() No signature node found: > FDT_ERR_NOTFOUND\r\r\nBad Data Hash\r\r\nERROR -2: No such file or > directory: can't get kernel image!"]) > + where <built-in method join of str object at 0xa46de8> = ''.join > > To my untrained eyes that just looks like a lot of garbage :-) > To find the failing line of code you have to scroll back to the actual > backtrace. Now in this example it's kind of easy as we know what we > broke. If u-boot produces different output for some reason (like my > patches) I found it really hard to spot what's wrong in all the noise.
Yes I agree this is quite tricky. Part of this is pytest trying to be helpful, but I'm sure we can improve it. > > The parametrized test_vboot function doesn't make it easier either, esp > since some of the parameters select an entirely different set of tests > (test_global_sign/test_required_key/test_with_algo). > > The ubman.log.* calls would explain some context and progress but are > not in the console, only in the html. The HTML could be more useful due > to the structuring but isn't as the distracting stuff is on the same > level as the actual tests. All output leading up to the failure gets > displayed by the js but the relevant output that is actually looked at > by assert() is in just one line. > > Last but not least even if the presentation of all this information was > improved, there's still some information about the good case missing. Ie > what did the output look like when the test was written? The split between console and html bugs me, too. Thanks for looking at this. Regards, Simon
