On 4/20/20 8:32 PM, Tom Rini wrote: > On Mon, Apr 20, 2020 at 08:23:08PM +0200, Heinrich Schuchardt wrote: >> On 4/20/20 1:38 AM, Simon Glass wrote: >>> On Sat, 18 Apr 2020 at 06:09, Heinrich Schuchardt <[email protected]> >>> wrote: >>>> >>>> By passing -ra to pytest we get a summary indicating which tests failed >>>> and why tests were skipped. >>>> >>>> Here is an example output: >>>> >>>> ======================== short test summary info ========================= >>>> SKIPPED [1] test/py/tests/test_efi_loader.py:81: No DHCP server available >>>> SKIPPED [1] test/py/tests/test_efi_loader.py:100: No static network >>>> configuration is defined >>>> SKIPPED [2] test/py/tests/test_efi_loader.py:115: Network not initialized >>>> >>>> Signed-off-by: Heinrich Schuchardt <[email protected]> >>>> --- >>>> test/run | 6 +++--- >>>> 1 file changed, 3 insertions(+), 3 deletions(-) >>>> >>> >>> This is really noisy - I get lots of extra output. Can we make this an >>> option? >> >> When I run 'make tests' I get 41 out of 199 lines explaining skipped >> and failed tests. >> >> Lines like these are noise because there is no actionable information: >> >> test/py/tests/test_fs/test_basic.py >> sssssssssssssssssssssssssssssssssssssss [ 0%] >> test/py/tests/test_fs/test_ext.py ssssssssssssssssssssss [ 0%] >> test/py/tests/test_fs/test_mkdir.py ssssssssssss [ 0%] >> test/py/tests/test_fs/test_symlink.py ssss [ 0%] >> test/py/tests/test_fs/test_unlink.py ssssssssssssss [ 0%] >> >> This new line has actionable information: >> >> SKIPPED [13] test/py/tests/test_fs/conftest.py:340: Setup failed for >> filesystem: fat16 >> >> Next step is to change this line to provide a more useful output, e.g. >> >> - except CalledProcessError: >> - pytest.skip('Setup failed for filesystem: ' + fs_type) >> + except CalledProcessError as err: >> + pytest.skip('Setup failed for filesystem: ' + fs_type + \ >> + ', {}'.format(err)) >> >> SKIPPED [13] test/py/tests/test_fs/conftest.py:340: Setup failed for >> filesystem: fat16, Command 'mkfs.vfat -F 16 >> build-sandbox/persistent-data/3GB.fat16.img' returned non-zero exit >> status 127. >> >> Now we know that that the test is wrong by assuming that mkfs.vfat is in >> the path instead of using /usr/sbin/mkfs.vfat or /sbin/mkfs.vfat and we >> can fix it. >> >> We should get rid of all skipped tests especially on Travis CI and >> Gitlab CI. Further we should provide instructions to set up a local >> system to avoid skipping tests. >> >> Simon, why do you want to remove the actionable information? > > OK, so I ran this through "qcheck" and I got a bit more output, but some > of which indicates we should figure out how to enable these tests for > sandbox out of the box. > > The follow-up patch I really want to see is passing -ra in the > travis/gitlab/azure file as I very much agree we should get everything > running in CI or have a good reason why not (ie the MMC tests might be > hard to enable on all platforms via qemu but we should make a good try > at it). > > And giving better error messages too would be very helpful so it's > clearer why we aren't (and why some people would not want to) run the FS > tests for example as we really haven't figured out a good 100% > non-root-user method to do those tests. >
The tests run fine on my Debian system with https://github.com/xypron2/u-boot/commit/f465d52a5a539761d9e5602331280ce07a1bcbca where I assume mkfs.* and fsck.ext4 are in /sbin. The same should work on Ubuntu. On current distributions there are also symlinks in /usr/sbin/ but https://askubuntu.com/questions/674847/sudo-usr-sbin-mkfs-ext4-command-not-found indicates that this does not hold true for elder distributions. I will submit the test once tested in Travis CI and Gitlab CI. Best regards Heinrich
