Hi Heinrich, On Mon, 20 Apr 2020 at 12:23, Heinrich Schuchardt <[email protected]> wrote: > > On 4/20/20 1:38 AM, Simon Glass wrote: > > On Sat, 18 Apr 2020 at 06:09, Heinrich Schuchardt <[email protected]> > > wrote: > >> > >> By passing -ra to pytest we get a summary indicating which tests failed > >> and why tests were skipped. > >> > >> Here is an example output: > >> > >> ======================== short test summary info ========================= > >> SKIPPED [1] test/py/tests/test_efi_loader.py:81: No DHCP server available > >> SKIPPED [1] test/py/tests/test_efi_loader.py:100: No static network > >> configuration is defined > >> SKIPPED [2] test/py/tests/test_efi_loader.py:115: Network not initialized > >> > >> Signed-off-by: Heinrich Schuchardt <[email protected]> > >> --- > >> test/run | 6 +++--- > >> 1 file changed, 3 insertions(+), 3 deletions(-) > >> > > > > This is really noisy - I get lots of extra output. Can we make this an > > option? > > When I run 'make tests' I get 41 out of 199 lines explaining skipped > and failed tests. > > Lines like these are noise because there is no actionable information: > > test/py/tests/test_fs/test_basic.py > sssssssssssssssssssssssssssssssssssssss [ 0%] > test/py/tests/test_fs/test_ext.py ssssssssssssssssssssss [ 0%] > test/py/tests/test_fs/test_mkdir.py ssssssssssss [ 0%] > test/py/tests/test_fs/test_symlink.py ssss [ 0%] > test/py/tests/test_fs/test_unlink.py ssssssssssssss [ 0%] > > This new line has actionable information: > > SKIPPED [13] test/py/tests/test_fs/conftest.py:340: Setup failed for > filesystem: fat16 > > Next step is to change this line to provide a more useful output, e.g. > > - except CalledProcessError: > - pytest.skip('Setup failed for filesystem: ' + fs_type) > + except CalledProcessError as err: > + pytest.skip('Setup failed for filesystem: ' + fs_type + \ > + ', {}'.format(err)) > > SKIPPED [13] test/py/tests/test_fs/conftest.py:340: Setup failed for > filesystem: fat16, Command 'mkfs.vfat -F 16 > build-sandbox/persistent-data/3GB.fat16.img' returned non-zero exit > status 127. > > Now we know that that the test is wrong by assuming that mkfs.vfat is in > the path instead of using /usr/sbin/mkfs.vfat or /sbin/mkfs.vfat and we > can fix it. > > We should get rid of all skipped tests especially on Travis CI and > Gitlab CI. Further we should provide instructions to set up a local > system to avoid skipping tests. > > Simon, why do you want to remove the actionable information?
I don't want to remove it. It isn't there now! Let's fix it before we enable it. Otherwise it is just noise. The device tree fiasco is a real pain. BTW the fs tests are flaky for me so I seldom run them. Regards, Simon
