What is a 'program' in Section 15.3.3.1?
15.3.3.1 Command-line arguments for test drivers The first non-option argument passed to the test driver is the program to be run, and all the following ones are command-line options and arguments for this program. A few minor questions: 1: What is the program to be run? 2: What are the arguments for this program? 3: What is the format of the command line executing a custom driver? I am looking at the tap-driver.sh and guessing what some of this means. I have absolutely no idea that my guesses make sense, but: 1: Are the programs check/recheck from make check, make recheck respectively? 2: Are the arguments the [AM_]_LOG_DRIVER_FLAGS driver options? 3: Is the format of the command line: driver custom_driver_options [AM_]_LOG_DRIVER_FLAGS check|recheck And, based on these guesses, there is no program to be run and there are no arguments for this program. Could someone clarify this? Maintenance turns design into chaos
Re: What is a 'program' in Section 15.3.3.1?
In looking at tap-driver.sh there doesn't appear to be a place where a 'program' is accepted on the input command line. It appears that after all options are read if the input command line '$#' is not zero then an error is declared. So, is the TAP interface different from other Custom Test Driver interfaces? The failure of the past is the challenge of the present and the success of the future.
How are data files input for to custom drivers?
15.2.1 Scripts-based Testsuites If the special variable TESTS is defined, its value is taken to be a list of programs or scripts to run in order to do the testing. Under the appropriate circumstances, it's possible for TESTS to list also data files to be passed to one or more test scripts defined by different means (the so-called log compilers, see Parallel Test Harness). If I define: LOG_DRIVER = myscript.sh TESTS = data What happens? If I define: (the tap-driver.sh is a custom driver by definition) TEST_EXTENSIONS = .sh LOG_DRIVER = tap-driver.sh SH_LOG_DRIVER = tap-driver.sh TESTS = data script.sh Then is seems that the test harness would have to do: cat data | tap-driver.sh ./script.sh | tap-driver.sh Because the awk loop in tap-driver.sh does a getline which reads from an input pipe (stdin). Have I read the code and the document correctly? This is really confusing to me. In 15.3.3.1 Command-line arguments for test drivers the manual says The first non-option argument passed to the test driver is the program to be run, and all the following ones are command-line options and arguments for this program. Which seems to mean that for custom drivers we get for: TEST_EXTENSIONS = SH SH_LOG_DRIVER = custom_driver LOG_DRIVER = custom_driver TESTS = program test.sh Which should execute as: custom_driver test_harness_options program custom_driver test_harness_options test.sh Which is certainly not done with the tap-driver. So what in tarnation is going on? The manual make very clear how to input a data file. For example If I define: TESTS = script.sh Then according to: 15.2.1 Scripts-based Testsuites Test programs that need data files should look for them in srcdir (which is both a make variable and an environment variable made available to the tests), so that they work when building in a separate directory (see Build Directories in The Autoconf Manual), and in particular for the distcheck rule (see Checking the Distribution). So it seems clear what to do if script.sh needs a data file.
RE: What is a 'program' in Section 15.3.3.1?
Thanks. (Also see lists.gnu.org/archive/html/automake/2015-05/msg00051.html How are data files input for to custom drivers?) The TAP interface is very different. TAP is supposed to be a custom test driver. Is the interface and manner of calling different from other custom drivers, and the API and other comments describing custom drivers in the manual? It depends on subordinate scripts printing I'm very confused here. It looks to me like tap-driver.sh is a standalone script and doesn't need any help scripts. The input data is processed in awk and all the needed functions are defined in this context. For GraphicsMagick, I created a script which is sourced by actual test scripts (.tap scripts), which provides generic functions used by those scripts. The functions support pass/fail detection based on program exit status, but also support 'xfail' by being passed a set of feature id strings which are matched which the actual available features. If the test fails, but a necessary feature is missing, then an extra diagnosis is added which Automake's TAP script will turn into an informative XFAIL message: I am confused. Using Automake the Developer can generate a reference to a class of test cases defined in the TESTS variable. Each one of the scripts is required to output the results of one of more tests that they run in quasi-TAP format. The TAP script, tap-driver.sh, takes the output and generates a trs file. Included with the tap-driver.sh there is a means to generate XFAIL and XPASS, however this seems to be global to all subtests executed in a TESTS defined test. Each ok or not ok returned will be translated (as required) to XPASS or XFAIL depending on --expect-failure=yes. As an example: TEST_EXTENSIONS =.abc ABC_LOG_DRIVER = tap-driver.sh TESTS = test1.abc test2.abc Means that tap-driver.sh is called twice, once for test1.abc, once for test2.abc. Each of these tests can return one or more of ok, not ok, skip, bail, or a YAML-like text string defined by --diagnostic-string and --comments. Another point of confusion is that the options defined for the custom test drivers have the format a=b if 'a' has a value, but tap-driver.sh does not allow a=b, it requires a b. Anyhow, thanks for the heads up, I will be looking at your files. And, as always, I am confused.
RE: What is a 'program' in Section 15.3.3.1?
On Wed, 27 May 2015, Arthur Schwarz wrote: TAP is supposed to be a custom test driver. Is the interface and manner of calling different from other custom drivers, and the API and other comments describing custom drivers in the manual? I think that the method of calling is similar to other type of tests except that the list of tests 'TESTS' is a list of scripts which produce TAP output strings. It depends on subordinate scripts printing I'm very confused here. It looks to me like tap-driver.sh is a standalone script and doesn't need any help scripts. The input data is processed in awk and all the needed functions are defined in this context. If the test program was a C program so it was a binary and printed 'ok' and 'not ok' outputs, and is able to find any input files without assistance, then no extra shim script should be required. In my case I was replacing perhaps 1000 individual classic Automake test scripts and wanted to replace them with far fewer TAP scripts. I am confused. Using Automake the Developer can generate a reference to a class of test cases defined in the TESTS variable. Each one of the scripts is required to output the results of one of more tests that they run in quasi-TAP format. The TAP script, tap-driver.sh, takes the output and generates a trs file. Included with the tap-driver.sh there is a means to generate XFAIL and XPASS, however this seems to be global to all subtests executed in a TESTS defined test. Each ok or not ok returned will be translated (as required) to XPASS or XFAIL depending on --expect-failure=yes. A TAP test program/script may contain many tests and it may output multiple test lines. The script indicates how many tests it plans to run by first printing a line like 1..532 to indicate that it plans to run 532 tests. Each test gets is own PASS, FAIL, XFAIL indication. As an example: TEST_EXTENSIONS =.abc ABC_LOG_DRIVER = tap-driver.sh TESTS = test1.abc test2.abc Means that tap-driver.sh is called twice, once for test1.abc, once for test2.abc. Each of these tests can return one or more of ok, not ok, skip, bail, or a YAML-like text string defined by --diagnostic-string and --comments. Yes, but tap-driver.sh interprets the output while the test program/scripts runs and it can handle an arbitary number of tests/results in the same test script. Anyhow, thanks for the heads up, I will be looking at your files. And, as always, I am confused. I am not surprised. :-) Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: What is a 'program' in Section 15.3.3.1?
On Wed, 27 May 2015, Arthur Schwarz wrote: In looking at tap-driver.sh there doesn't appear to be a place where a 'program' is accepted on the input command line. It appears that after all options are read if the input command line '$#' is not zero then an error is declared. So, is the TAP interface different from other Custom Test Driver interfaces? The TAP interface is very different. It depends on subordinate scripts printing 'ok' or 'not ok' (plus some optional extra diagnostic info) rather than return codes. The subordinate scripts take full responsibility for the set of tests to be performed, what programs are invoked, and the options supplied to those programs. For GraphicsMagick, I created a script which is sourced by actual test scripts (.tap scripts), which provides generic functions used by those scripts. The functions support pass/fail detection based on program exit status, but also support 'xfail' by being passed a set of feature id strings which are matched which the actual available features. If the test fails, but a necessary feature is missing, then an extra diagnosis is added which Automake's TAP script will turn into an informative XFAIL message: not ok # SKIP requires foo support See the implementation at http://hg.code.sf.net/p/graphicsmagick/code/file/b84df92f710a/scripts/tap-functions.shi and http://hg.code.sf.net/p/graphicsmagick/code/file/b84df92f710a/tests/common.shi and http://hg.code.sf.net/p/graphicsmagick/code/file/b84df92f710a/tests/rwfile.tap Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: What is a 'program' in Section 15.3.3.1?
On 2015-05-27 13:55 -0700, Arthur Schwarz wrote: In looking at tap-driver.sh there doesn't appear to be a place where a 'program' is accepted on the input command line. It appears that after all options are read if the input command line '$#' is not zero then an error is declared. So, is the TAP interface different from other Custom Test Driver interfaces? I am guessing that you are referring to this line in tap-driver.sh: test $# -gt 0 || usage_error missing test command The error is only printed if $# is *equal* to zero (i.e., additional arguments including the program name are mandatory). Regards, -- Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)
