On 03/13/2016 02:13 PM, Samuel Holland wrote: > On 03/13/2016 01:18 PM, Rob Landley wrote: >> On 03/13/2016 03:34 AM, Andy Chu wrote: >>> FWIW I think the test harness is missing a few concepts: >>> >>> - exit code >> >> blah; echo $? >> >>> - stderr >> >> 2>&1 > > I think the idea here was the importance of differentiating between > stdout and stderr, and between text output and return code.
You can do this now. "2>&1 > /dev/null" gives you only stdout, or 2>file... There are many options. > simple as having a separate output variable for each type of output. > > Granted, it will usually be unambiguous as to the correctness of the > program, Yes, adding complexity to every test that isn't usually needed. > but having the return code in the output string can be > confusing to the human looking at the test case. Plus, why would you not > want to verify the exit code for every test? Because science is about reducing variables and isolating to test specific things? Because "we can so clearly we should" is the gnu approach to things? Because you're arguing for adding complexity to the test suite to do things it can already do, and in many cases is already doing? Because we've already got 5 required arguments for each test and you're proposing adding a couple more, and clearly that's going to make the test suite easier to maintain and encourage additional contributions? I'm not saying it's _not_ a good extension, but you did ask. Complexity is a cost, spend it wisely. You're saying each test should test more things, which means potential false positives and more investigation about why a test failed (and/or more complex reporting format). Also, "the return code" implies none of the tests are pipelines, or multi-stage "do thing && examine thing" (which _already_ fails if do thing returned failure, and with the error_msg() stuff would have said why to stderr already). Yesterday I was poking at mv tests which have a lot of "mv one two && [ -e two ] && [ ! -e one ] && echo yes" sort of constructs. What is "the exit code" from that? > It's a lot of duplication > to write "echo $?" in all of the test cases. I don't. Sometimes I go "blah && yes" or "blah || yes", when the return code is specifically what I'm testing, and sometimes checking the return code and checking the output are two separate tests. Keep in mind that error_msg() and friends produce output, and the tests don't catch stderr by default but pass it through. If we catch stderr by default and a test DOESN'T check it, then it's ignored instead of visibile to the caller. Also, keep in mind I want the host version to pass most of these tests too, and if there are gratuitous differences in behavior I don't WANT the test to fail based on something I don't care about and wasn't trying to test. You're arguing for a tighter sieve with smaller holes when I've already received a bunch of failing tests that were written against gnu and never _tried_ against the toybox version, and failed for reasons that aren't real failures. > As for stdout/stderr, it helps make sure diagnostic messages are going > to the right stream when not using the helper functions. Right now diagnostic messages are visible in the output when running the test. There shouldn't be any by default, when there is it's pretty obvious because those lines aren't colored. I'm all for improving the test suite, but "what I think the test suite should be trying to do differs from what you think the test suite should be trying to do, therefore I am right" is missing some steps. Rob - All syllogisms have three parts, therefore this is not a syllogism. _______________________________________________ Toybox mailing list [email protected] http://lists.landley.net/listinfo.cgi/toybox-landley.net
