Am 15.08.2023 um 14:17 hat Denis V. Lunev geschrieben: > Hi! > > Small side note. > > I am 100% sure that I have run this set of tests and > there was no fault. I have re-run them and once > again has not get the fault :-) > > The reason for that is quite interesting: > * the test does not start due to the absence of the > 'certool' utility from gnutls > > This brings the very important question. > > Should we *FAIL* when important utility is missed > or skip? I believe that our goal is to fail to avoid such > cases. > > What do you think?
In general I think it makes sense that FAIL means that the test could run as expected, but we got an unexpected result (i.e. this is likely a QEMU bug), and SKIP means that the test couldn't meaningfully be performed on the host system. Making more things hard dependencies for the test would mean that it's harder to miss an instance like this, but it would also make it harder to run the test suite on a system that doesn't have the dependencies available (and you might not even have root privileges to install them). I think I'd leave things as they are now, but recommend that you occasionally check the tests reported as "not run" to see if you could easily provide the thing they would need. Kevin
