Re: [PATCH] t6018-rev-list-glob: fix 'empty stdin' test

[SZEDER Gábor]

On Wed, Aug 22, 2018 at 7:53 PM Eric Sunshine <sunsh...@sunshineco.com> wrote:

> Can you say a word or two (here in the email thread) about how you're
> finding these failures (across the various test fixes you've posted
> recently)? Are you instrumenting the code in some fashion? Or, finding
> them by visual inspection?

Errors from system commands in our tests look like these:

  grep: file3: No such file or directory
  sed: -e expression #1, char 2: extra characters after command
  diff: sub1/.git: No such file or directory
  tar: rmtlseek not stopped at a record boundary
  tar: Error is not recoverable: exiting now

while errors from the shell running the test like these:

  t0020-crlf.sh: 8: eval: cannot open two: No such file
  t6018-rev-list-glob.sh: 4: eval: cannot open expect: No such file
  t7408-submodule-reference.sh: 615: test: =: unexpected operator

i.e. lines starting with various system commands' or test scripts'
names, followed by ': '.

So I've modified t/Makefile to not remove the 'test-results' directory
after a successful 'make test':

diff --git a/t/Makefile b/t/Makefile
index ea36cf7ac7..c7b1655593 100644
--- a/t/Makefile
+++ b/t/Makefile
@@ -54,10 +54,11 @@ pre-clean:
        $(RM) -r '$(TEST_RESULTS_DIRECTORY_SQ)'

 clean-except-prove-cache:
-       $(RM) -r 'trash directory'.* '$(TEST_RESULTS_DIRECTORY_SQ)'
+       $(RM) -r 'trash directory'.*
        $(RM) -r valgrind/bin

 clean: clean-except-prove-cache
+       $(RM) '$(TEST_RESULTS_DIRECTORY_SQ)'

 distclean: clean
        $(RM) .prove

And then scanned the results of a '--verbose-log -x' test run with:

  grep -E 
'^(awk|basename|cat|cd|chmod|cmp|cp|cut|diff|dirname|egrep|find|fgrep|grep|gunzip|gzip|ln|mkdir|mkfifo|mktemp|mv|readlink|rmdir|sed|sort|tar|touch|tr|ulimit|umask|uniq|unzip|wc|zipinfo|t[0-9][0-9][0-9][0-9]-[^:]*\.sh):
' test-results/*.out

and then, for lack of something better to do ;), I started looking at
the simpler looking errors.

I've though about how a check like this could be automated, but
haven't had any workable idea yet.  There are commands that can
legitimately print errors, e.g. when checking for a prereq which the
system doesn't have (e.g. the 'tar' errors above, I think).  And the
list of system commands in the grep pattern above is surely incomplete
and will likely change in the future...