Bruno Haible <br...@clisp.org> writes: > Simon Josefsson wrote: >> In my experience, adding gnulib self-tests to projects lead to a >> maintainance cost to debug gnulib self-tests failures, which are rarely >> of importance to the core project. > > It depends on the project. > > For some projects, the gnulib self-tests are useful for quickly dispatching > responsibility to gnulib. For example, in coreutils, if some 'mkfifo' test > fails on some platform, the first thing a maintainer will look at is whether > the gnulib tests of the 'mkfifo' and 'mknod' modules fail as well. If yes, > he can quickly delegate to gnulib. Similarly, in GNU libunistring on macOS, > if I see not only the GNU libunistring tests fail but also the gnulib > tests regarding iconv(), I know that the macOS iconv() is broken and that > I need to start adding workarounds in gnulib. > > In a project that uses only more-or-less "conventional" gnulib modules, > on the other hand, the gnulib self-tests probably don't help.
Agreed. >> Btw, if enable it would be nice to add a simple instruction to the >> README saying how to run 'make check' with gnulib self-tests disabled. >> Is there any agreed on practice to achieve that? > > Not that I know of. The only common practice that I know of, so far, is > to put the gnulib-tests directory last in the SUBDIRS variable, so that > gnulib-tests failures don't mask failures of the package's core tests. I've gone back and forward on exactly that choice. One problem with putting the gnulib tests last is that for most normal cases, after a successful run, the user will see the output of the gnulib self tests in the terminal, not the actual project's self tests. IMHO, it looks nicer to see the result of the project's self-tests at the end of the run, and have the gnulib self tests higher up because they are internal. This is also relevant for CI/CD systems: sometimes you only see the tail of the log output, and displaying earlier outputs requires extra work. Scrolling to the bottom of an e-mail with log output is also easier than scrolling to the middle. I think the ideal state I would prefer is if the gnulib self-tests are run first (with a simple way to disable them, for speed) and then displayed, but a failure won't cause the projects' core self tests to not be run. If the core self tests pass, then any failure in the gnulib self tests can be shown. If the core self tests fail, the gnulib self tests should be shown _before_ the failing core self tests. This results in the most useful order of display for me, and hopefully others have similar preferences. I don't know how to achieve that easily though. I think the decision to not include gnulib self-tests was done before my involvement in inetutils. Unless there is more substantial pushback here, I suggest to just re-enable them and see how things goes. The display and execution order is mostly irrelevant if the self-checks all pass anyway, as they should :) /Simon
signature.asc
Description: PGP signature
