On 5/15/26 15:41, Simon Glass wrote:
Hi Ludwig,
On 2026-05-13T14:08:10, Ludwig Nussel <[email protected]> wrote:
The tests were created with the help of copilot and manual fixups. I
have to admit that the log output of those tests is not something I can
grok. I've left the tags copilot added in the commits even though
checkpatch complains. Not sure what the policy is.
Thanks for being upfront. For my work I have adjusted patman to drop these tags.
The "log output I can't grok" part worries me more than the tag. If
you can't follow what the test is exercising, neither can the next
person debugging a failure :-)
Meanwhile I got it. The AI helped me to get started.
Anyway, here is what confused me. Suppose we introduce a small typo:
- 'No signature node found', False)
+ 'No signature node fond', False)
Then run
$ test/py/test.py --bd sandbox "$@" -k 'test_vboot -vvx
You'll get a dump of all kinds of irrelevant things like openssl key
generation or u-boot startup. Then finally:
FAILED
test/py/tests/test_vboot.py::test_vboot[sha1-basic-sha1--None-False-True-False-False]
- assert 'No signature node fond' in "1769 bytes read in 0 msWorking FDT
set to 100image_pre_load_sig_setup() INFO: no info for image pre-load
sig check\r\r\n## Loading kernel (any) from FIT Image at 00000100
...\r\r\n Using 'conf-1' configuration\r\r\n Verifying Hash
Integrity ... fit_config_verify_required_keys() No signature node found:
FDT_ERR_NOTFOUND\r\r\nBad Data Hash\r\r\nERROR -2: No such file or
directory: can't get kernel image!"
+ where "1769 bytes read in 0 msWorking FDT set to
100image_pre_load_sig_setup() INFO: no info for image pre-load sig
check\r\r\n## Loading kernel (any) from FIT Image at 00000100 ...\r\r\n
Using 'conf-1' configuration\r\r\n Verifying Hash Integrity ...
fit_config_verify_required_keys() No signature node found:
FDT_ERR_NOTFOUND\r\r\nBad Data Hash\r\r\nERROR -2: No such file or
directory: can't get kernel image!" = <built-in method join of str
object at 0xa46de8>(['1769 bytes read in 0 ms', 'Working FDT set to
100', "image_pre_load_sig_setup() INFO: no info for image pre-load sig
check\r\r\n## Loading kernel (any) from FIT Image at 00000100 ...\r\r\n
Using 'conf-1' configuration\r\r\n Verifying Hash Integrity ...
fit_config_verify_required_keys() No signature node found:
FDT_ERR_NOTFOUND\r\r\nBad Data Hash\r\r\nERROR -2: No such file or
directory: can't get kernel image!"])
+ where <built-in method join of str object at 0xa46de8> = ''.join
To my untrained eyes that just looks like a lot of garbage :-)
To find the failing line of code you have to scroll back to the actual
backtrace. Now in this example it's kind of easy as we know what we
broke. If u-boot produces different output for some reason (like my
patches) I found it really hard to spot what's wrong in all the noise.
The parametrized test_vboot function doesn't make it easier either, esp
since some of the parameters select an entirely different set of tests
(test_global_sign/test_required_key/test_with_algo).
The ubman.log.* calls would explain some context and progress but are
not in the console, only in the html. The HTML could be more useful due
to the structuring but isn't as the distracting stuff is on the same
level as the actual tests. All output leading up to the failure gets
displayed by the js but the relevant output that is actually looked at
by assert() is in just one line.
Last but not least even if the presentation of all this information was
improved, there's still some information about the good case missing. Ie
what did the output look like when the test was written?
cu
Ludwig
--
Ludwig Nussel
Siemens AG
www.siemens.com
