On 4/20/20 8:45 PM, Simon Glass wrote: > Hi Heinrich, > > On Mon, 20 Apr 2020 at 12:23, Heinrich Schuchardt <[email protected]> wrote: >> >> On 4/20/20 1:38 AM, Simon Glass wrote: >>> On Sat, 18 Apr 2020 at 06:09, Heinrich Schuchardt <[email protected]> >>> wrote: >>>> >>>> By passing -ra to pytest we get a summary indicating which tests failed >>>> and why tests were skipped. >>>> >>>> Here is an example output: >>>> >>>> ======================== short test summary info ========================= >>>> SKIPPED [1] test/py/tests/test_efi_loader.py:81: No DHCP server available >>>> SKIPPED [1] test/py/tests/test_efi_loader.py:100: No static network >>>> configuration is defined >>>> SKIPPED [2] test/py/tests/test_efi_loader.py:115: Network not initialized >>>> >>>> Signed-off-by: Heinrich Schuchardt <[email protected]> >>>> --- >>>> test/run | 6 +++--- >>>> 1 file changed, 3 insertions(+), 3 deletions(-) >>>> >>> >>> This is really noisy - I get lots of extra output. Can we make this an >>> option? >> >> When I run 'make tests' I get 41 out of 199 lines explaining skipped >> and failed tests. >> >> Lines like these are noise because there is no actionable information: >> >> test/py/tests/test_fs/test_basic.py >> sssssssssssssssssssssssssssssssssssssss [ 0%] >> test/py/tests/test_fs/test_ext.py ssssssssssssssssssssss [ 0%] >> test/py/tests/test_fs/test_mkdir.py ssssssssssss [ 0%] >> test/py/tests/test_fs/test_symlink.py ssss [ 0%] >> test/py/tests/test_fs/test_unlink.py ssssssssssssss [ 0%] >> >> This new line has actionable information: >> >> SKIPPED [13] test/py/tests/test_fs/conftest.py:340: Setup failed for >> filesystem: fat16 >> >> Next step is to change this line to provide a more useful output, e.g. >> >> - except CalledProcessError: >> - pytest.skip('Setup failed for filesystem: ' + fs_type) >> + except CalledProcessError as err: >> + pytest.skip('Setup failed for filesystem: ' + fs_type + \ >> + ', {}'.format(err)) >> >> SKIPPED [13] test/py/tests/test_fs/conftest.py:340: Setup failed for >> filesystem: fat16, Command 'mkfs.vfat -F 16 >> build-sandbox/persistent-data/3GB.fat16.img' returned non-zero exit >> status 127. >> >> Now we know that that the test is wrong by assuming that mkfs.vfat is in >> the path instead of using /usr/sbin/mkfs.vfat or /sbin/mkfs.vfat and we >> can fix it. >> >> We should get rid of all skipped tests especially on Travis CI and >> Gitlab CI. Further we should provide instructions to set up a local >> system to avoid skipping tests. >> >> Simon, why do you want to remove the actionable information? > > I don't want to remove it. It isn't there now! > > Let's fix it before we enable it. Otherwise it is just noise. The > device tree fiasco is a real pain. > > BTW the fs tests are flaky for me so I seldom run them.
What do you mean by flaky? Do you mean unreliable (cf. https://www.urbandictionary.com/define.php?term=flaky)? What is unreliable about the tests? Best regards Heinrich > > Regards, > Simon >
