Re: [PATCH] testsuite: Fix rtems-test-check not excluding tests.

[Chris Johns]

On 16/04/2017 11:54, Gedare Bloom wrote:

Thanks for adding some documentation. I spotted a couple minor
typographic issues.


Thank you for the review and the comments. I have fixed the issues and 
push the change.


Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [PATCH] testsuite: Fix rtems-test-check not excluding tests.

[Gedare Bloom]

Thanks for adding some documentation. I spotted a couple minor
typographic issues.

On Sat, Apr 15, 2017 at 7:57 PM, Chris Johns  wrote:
> The include file handling was broken.
>
> Add a test configuration data README.
> ---
>  testsuites/README.testdata  | 72 
> +
>  testsuites/automake/subdirs.am  |  7 ++--
>  tools/build/rtems-test-check-py | 60 +++---
>  3 files changed, 125 insertions(+), 14 deletions(-)
>  create mode 100644 testsuites/README.testdata
>
> diff --git a/testsuites/README.testdata b/testsuites/README.testdata
> new file mode 100644
> index 00..7210feab93
> --- /dev/null
> +++ b/testsuites/README.testdata
> @@ -0,0 +1,72 @@
> +RTEMS Testsuite Configuration
> +=
> +
> +The RTEMS Testsuite lets you configure the tests for a BSP. Every BSP can 
> have
> +a test configuration data file and this file is read when building the
> +tests. The test configuration data can control what tests are build and now 
> the
> +tests are built.
> +
> +The test configuration data files have a `.tcfg` file extension. You can
> +include other test configuration data files reducing repeated test
> +sequences. There is also a global test configuration data file for global
> +configurations.
> +
> +Command
> +---
> +
> +The biuld system invokes the test check tool to determine what it does.
build

> +
> + rtems-test-check mode bsp testconfig includepaths test[s]
> +
> +Mode (mode)
> +~~~
> +
> +The check command modes are:
> +
> + 1. `exclude`: The input list of tests is checked against the exluded tests 
> and
excluded

> +the tests that are not excluded are printed.
> +
> + 2. `flags': The test provide is check for any specical build flags. These
provided is checked
special

> +flags can enumerate a test in a specific way.
> +
> +BSP (bsp)
> +~
> +
> +The name of the BSP.
> +
> +Test Configuration (testconfig)
> +~~~
> +
> +The path to the BSP's test configration file. The the file does not exist the
configuration
If the file?

> +input list of tests is returned and not flags a reported.
I'm not sure what "not flags a reported" means.

> +
> +Include Paths (inputpaths)
> +~~
> +
> +A colon (`:`) separated list of paths test configuration files are search
> +for.
> +
> +Tests (test[s])
> +~~~
> +
> +If the mode is `exclude` this argument is a list of tests to be checked. If 
> the
> +mode is `flags` a single test is required and the flags for the test are 
> returned.
> +
> +File Format
> +---
> +
> +The file is an ASCII text file of lines. And text after and including the `#`
And -> Any?
Also, it probably should say "Any text on a line after ..."? Or just
say "# starts a comment that ends at the new line."

> +character is removed. Empty lines are ignored.
> +
> +A line is either the `include` directive or a test state. The states are:
> +
> + exclude   : Exclude the test from being built.
> + expected-fail : The test is built but expected to fail.
> + user-input: The test requires user input and may be aborted when running
> + the tests
Add a period after tests for consistency.

> + indeterminate : The test result is indeterminate. This means the test may 
> pass
> + or may fail therefore it cannot be included in the 
> requression
regression

> + results.
> + benchmark : The test is a benchmark and maybe is aborted when running 
> the
> + tests because of the load and time the test may take to run.
> + Performance changes are currently not viewed as a 
> regression.
> diff --git a/testsuites/automake/subdirs.am b/testsuites/automake/subdirs.am
> index 70a7ecb78c..63b1da1049 100644
> --- a/testsuites/automake/subdirs.am
> +++ b/testsuites/automake/subdirs.am
> @@ -15,7 +15,8 @@ all-local:
>   target="all"; \
> fi; \
> tcheck="$(top_srcdir)/../../tools/build/rtems-test-check-py"; \
> -   
> tdata="$(top_srcdir)/../../c/src/lib/libbsp/$(RTEMS_CPU)/$(RTEMS_BSP_FAMILY)/make/custom/$(RTEMS_BSP)-testsuite.tcfg";
>  \
> +   tdata="$(RTEMS_BSP)-testsuite.tcfg"; \
> +   
> tincludes="$(top_srcdir)/../../c/src/lib/libbsp/$(RTEMS_CPU)/$(RTEMS_BSP_FAMILY)/make/custom:$(top_srcdir)/..";
>  \
> if test -f "$$tdata"; then \
>   
> vtdata="$(RTEMS_CPU)/$(RTEMS_BSP_FAMILY)/make/custom/$(RTEMS_BSP)-testsuite.tcfg";
>  \
> else \
> @@ -23,7 +24,7 @@ all-local:
> fi; \
> echo "BSP Testsuite Data: $$vtdata"; \
> if test -f $$tcheck; then \
> - list=`$$tcheck exclude $$tdata $(top_srcdir)/.. $(RTEMS_BSP) 
> $(_SUBDIRS)`; \
> + list=`$$tcheck exclude $(RTEMS_BSP) $$tdata $$tincludes 
> $(_SUBDIRS)`; \
> else \
>   list=$(_SUBDIRS); \
> fi; \
> @@ -31,7 +32,7 @@ all-local:
>   echo "Making