Re: [Intel-gfx] [PATCH i-g-t v2] lib/igt_core.c: Expand --run-subtest functionality.

[Morton, Derek J]

Hi,

I have spoken to Dave. I do not want to implement multiple --run-subtest as it 
is not something I require and I do not think it is worth adding unless someone 
has a specific need for it. However I understand that whatever is done should 
not make that harder in the future to add multiple --run-subtest and so the 
--run-subtest should always be additive. The issue seems to be how to specify 
"add all subtests except ..." in a 'standard' way. The fnmatch() used currently 
only supports *?[] so there is no way of specifying 'not'. What would be the 
feeling if I was to replace fnmatch() with support for --run-subtest supporting 
proper regex expressions by including regex.h and using regexec()?

As far as I can see the impact of that would be a change of behaviour of * and 
? in the expression which might not be desirable if the wildcard expressions 
are in use. As an alternative I could add a new parameter --run-subtest-regex 
which allows the list of subtests to be run to be specified with a regex.

Would that be acceptable?

//Derek

>
>
>-Original Message-
>From: Gordon, David S 
>Sent: Thursday, January 28, 2016 12:58 PM
>To: Morton, Derek J; intel-gfx@lists.freedesktop.org
>Cc: daniel.vet...@ffwll.ch
>Subject: Re: [Intel-gfx] [PATCH i-g-t v2] lib/igt_core.c: Expand --run-subtest 
>functionality.
>
>On 28/01/16 11:05, Derek Morton wrote:
>> Added support for specifying arbitary lists of subtests to run, or to 
>> exclude from being run if prefixed by ^ or !.
>>
>> subtest1,subtest2 Will run subtest1 and subtest2
>> ^subtest1,subtest2 or !subtest1,subtest2 will run all subtests except
>> subtest1 and subtest2.
>>
>> Any subtest string not starting ! or ^ and not containing a comma is 
>> treated as a normal wildcard expression.
>>
>> This is required mainly on android to exclude subtests that test 
>> features that do not exist in the android driver while still being 
>> able to run other subtests in the binary when a wildcard expression is 
>> insufficient.
>>
>> v2: Use comma as list separator (Ville Syrjala) support both ^ and ! 
>> as not operators (Dave Gordon)
>>
>> Signed-off-by: Derek Morton <derek.j.mor...@intel.com>
>> ---
>>   lib/igt_core.c | 48 ++--
>>   1 file changed, 46 insertions(+), 2 deletions(-)
>>
>> diff --git a/lib/igt_core.c b/lib/igt_core.c index 6b69bb7..5d243af 
>> 100644
>> --- a/lib/igt_core.c
>> +++ b/lib/igt_core.c
>> @@ -207,7 +207,15 @@
>>* To do that obtain the lists of subtests with "--list-subtests", which 
>> can be
>>* run as non-root and doesn't require the i915 driver to be loaded (or any
>>* intel gpu to be present). Then individual subtests can be run 
>> with
>> - * "--run-subtest". Usage help for tests with subtests can be 
>> obtained with the
>> + * "--run-subtest". --run-subtest accepts wildcard characters. A list 
>> + of
>> + * subtests to run may be specified by using ',' as a separator.
>> + * A prefix of ^ or ! may be added to invert the logic, e.g. run all tests 
>> except...
>> + *
>> + * - --run-subtest basic* will run all subtests starting basic.
>> + * - --run-subtest subtest1,subtest2 will run only subtest1 and 
>> + subtest2
>> + * - --run-subtest ^subtest1,subtest2 will run all those except 
>> + subtest1 and subtest2
>> + *
>> + * Usage help for tests with subtests can be obtained with the
>>* "--help" command line option.
>>*/
>
>This is the syntax option 1 from my previous mail (but not allowing multiple 
>--run-subtest options?), which I think is a bit less powerful and a bit less 
>desirable because of the perhaps unexpected precedence of OR above NOT 
>(^sub1,sub2 => NOT(sub1 OR sub2).
>
>Do you think the advantages of option 2 (full generality, can express any 
>boolean, expected precedence of operators) would make it worth the extra 
>effort?
>
>Whichever we pick now, it will be set in stone as it will be very inconvenient 
>to edit every script using the feature once it's been in use for a while!
>
>Another idea: have you considered bash(1) extglob syntax? See the manpage for 
>bash, section "Pattern Matching". It's quite powerful, familiar to least to 
>script writers, and easy to try out in bash:
>
>$ : enable extglob mode
>$ shopt -s extglob
>$ : create an array of test names
>$ declare -a tests=( starting phase1 phase2 wibble phase3 wobble done ) $ : 
>see what matches a pattern:
>$ function match() { for w in "${tests[@]}" ; do [[ "$w" == $1 ]] && echo "$w&quo

Re: [Intel-gfx] [PATCH i-g-t v2] lib/igt_core.c: Expand --run-subtest functionality.

[Dave Gordon]

On 28/01/16 12:57, Dave Gordon wrote:

On 28/01/16 11:05, Derek Morton wrote:

Added support for specifying arbitary lists of subtests to run, or
to exclude from being run if prefixed by ^ or !.

subtest1,subtest2 Will run subtest1 and subtest2
^subtest1,subtest2 or !subtest1,subtest2 will run all subtests except
subtest1 and subtest2.

Any subtest string not starting ! or ^ and not containing a comma is
treated as a normal wildcard expression.

This is required mainly on android to exclude subtests that test
features that do not exist in the android driver while still being able
to run other subtests in the binary when a wildcard expression is
insufficient.

v2: Use comma as list separator (Ville Syrjala)
support both ^ and ! as not operators (Dave Gordon)

Signed-off-by: Derek Morton 
---
  lib/igt_core.c | 48 ++--
  1 file changed, 46 insertions(+), 2 deletions(-)

diff --git a/lib/igt_core.c b/lib/igt_core.c
index 6b69bb7..5d243af 100644
--- a/lib/igt_core.c
+++ b/lib/igt_core.c
@@ -207,7 +207,15 @@
   * To do that obtain the lists of subtests with "--list-subtests",
which can be
   * run as non-root and doesn't require the i915 driver to be loaded
(or any
   * intel gpu to be present). Then individual subtests can be run with
- * "--run-subtest". Usage help for tests with subtests can be
obtained with the
+ * "--run-subtest". --run-subtest accepts wildcard characters. A list of
+ * subtests to run may be specified by using ',' as a separator.
+ * A prefix of ^ or ! may be added to invert the logic, e.g. run all
tests except...
+ *
+ * - --run-subtest basic* will run all subtests starting basic.
+ * - --run-subtest subtest1,subtest2 will run only subtest1 and subtest2
+ * - --run-subtest ^subtest1,subtest2 will run all those except
subtest1 and subtest2
+ *
+ * Usage help for tests with subtests can be obtained with the
   * "--help" command line option.
   */


This is the syntax option 1 from my previous mail (but not allowing
multiple --run-subtest options?), which I think is a bit less powerful
and a bit less desirable because of the perhaps unexpected precedence of
OR above NOT (^sub1,sub2 => NOT(sub1 OR sub2).

Do you think the advantages of option 2 (full generality, can express
any boolean, expected precedence of operators) would make it worth the
extra effort?

Whichever we pick now, it will be set in stone as it will be very
inconvenient to edit every script using the feature once it's been in
use for a while!

Another idea: have you considered bash(1) extglob syntax? See the
manpage for bash, section "Pattern Matching". It's quite powerful,
familiar to least to script writers, and easy to try out in bash:


GNU libc has a globbing function that lets you override readdir() and 
friends so you can use it on a string or list of strings.


http://www.gnu.org/software/libc/manual/html_mono/libc.html#Globbing

Perl's Text::Glob implements glob(3) style matching that can be used to 
match against text, rather than fetching names from a filesystem.


Or have a look at wildmat(3) as standardised by RFC3977, or better still 
uwildmat(3).


https://tools.ietf.org/html/rfc3977#section-4
http://linux.die.net/man/3/uwildmat

That seems to give about the right tradeoff of complexity vs flexibility!

.Dave.
___
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [Intel-gfx] [PATCH i-g-t v2] lib/igt_core.c: Expand --run-subtest functionality.

[Dave Gordon]

On 28/01/16 11:05, Derek Morton wrote:

Added support for specifying arbitary lists of subtests to run, or
to exclude from being run if prefixed by ^ or !.

subtest1,subtest2 Will run subtest1 and subtest2
^subtest1,subtest2 or !subtest1,subtest2 will run all subtests except
subtest1 and subtest2.

Any subtest string not starting ! or ^ and not containing a comma is
treated as a normal wildcard expression.

This is required mainly on android to exclude subtests that test
features that do not exist in the android driver while still being able
to run other subtests in the binary when a wildcard expression is
insufficient.

v2: Use comma as list separator (Ville Syrjala)
support both ^ and ! as not operators (Dave Gordon)

Signed-off-by: Derek Morton 
---
  lib/igt_core.c | 48 ++--
  1 file changed, 46 insertions(+), 2 deletions(-)

diff --git a/lib/igt_core.c b/lib/igt_core.c
index 6b69bb7..5d243af 100644
--- a/lib/igt_core.c
+++ b/lib/igt_core.c
@@ -207,7 +207,15 @@
   * To do that obtain the lists of subtests with "--list-subtests", which can 
be
   * run as non-root and doesn't require the i915 driver to be loaded (or any
   * intel gpu to be present). Then individual subtests can be run with
- * "--run-subtest". Usage help for tests with subtests can be obtained with the
+ * "--run-subtest". --run-subtest accepts wildcard characters. A list of
+ * subtests to run may be specified by using ',' as a separator.
+ * A prefix of ^ or ! may be added to invert the logic, e.g. run all tests 
except...
+ *
+ * - --run-subtest basic* will run all subtests starting basic.
+ * - --run-subtest subtest1,subtest2 will run only subtest1 and subtest2
+ * - --run-subtest ^subtest1,subtest2 will run all those except subtest1 and 
subtest2
+ *
+ * Usage help for tests with subtests can be obtained with the
   * "--help" command line option.
   */


This is the syntax option 1 from my previous mail (but not allowing 
multiple --run-subtest options?), which I think is a bit less powerful 
and a bit less desirable because of the perhaps unexpected precedence of 
OR above NOT (^sub1,sub2 => NOT(sub1 OR sub2).


Do you think the advantages of option 2 (full generality, can express 
any boolean, expected precedence of operators) would make it worth the 
extra effort?


Whichever we pick now, it will be set in stone as it will be very 
inconvenient to edit every script using the feature once it's been in 
use for a while!


Another idea: have you considered bash(1) extglob syntax? See the 
manpage for bash, section "Pattern Matching". It's quite powerful, 
familiar to least to script writers, and easy to try out in bash:


$ : enable extglob mode
$ shopt -s extglob
$ : create an array of test names
$ declare -a tests=( starting phase1 phase2 wibble phase3 wobble done )
$ : see what matches a pattern:
$ function match() { for w in "${tests[@]}" ; do [[ "$w" == $1 ]] && 
echo "$w" ; done ; }

$ match '*'
starting
phase1
phase2
wibble
phase3
wobble
done
$ match 'phase*'
phase1
phase2
phase3
$ match 'w?bble'
wibble
wobble
$ match '[!pw]*'
starting
done
$ match '@(starting|done)'
starting
done
$ match '!(starting|done)'
phase1
phase2
wibble
phase3
wobble
$ match '!(phase*)'
starting
wibble
wobble
done
$ match '!(phase*|w?bble)'
starting
done
$ : etc, etc, ad lib

.Dave.

___
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

[Intel-gfx] [PATCH i-g-t v2] lib/igt_core.c: Expand --run-subtest functionality.

[Derek Morton]

Added support for specifying arbitary lists of subtests to run, or
to exclude from being run if prefixed by ^ or !.

subtest1,subtest2 Will run subtest1 and subtest2
^subtest1,subtest2 or !subtest1,subtest2 will run all subtests except
subtest1 and subtest2.

Any subtest string not starting ! or ^ and not containing a comma is
treated as a normal wildcard expression.

This is required mainly on android to exclude subtests that test
features that do not exist in the android driver while still being able
to run other subtests in the binary when a wildcard expression is
insufficient.

v2: Use comma as list separator (Ville Syrjala)
support both ^ and ! as not operators (Dave Gordon)

Signed-off-by: Derek Morton 
---
 lib/igt_core.c | 48 ++--
 1 file changed, 46 insertions(+), 2 deletions(-)

diff --git a/lib/igt_core.c b/lib/igt_core.c
index 6b69bb7..5d243af 100644
--- a/lib/igt_core.c
+++ b/lib/igt_core.c
@@ -207,7 +207,15 @@
  * To do that obtain the lists of subtests with "--list-subtests", which can be
  * run as non-root and doesn't require the i915 driver to be loaded (or any
  * intel gpu to be present). Then individual subtests can be run with
- * "--run-subtest". Usage help for tests with subtests can be obtained with the
+ * "--run-subtest". --run-subtest accepts wildcard characters. A list of
+ * subtests to run may be specified by using ',' as a separator.
+ * A prefix of ^ or ! may be added to invert the logic, e.g. run all tests 
except...
+ *
+ * - --run-subtest basic* will run all subtests starting basic.
+ * - --run-subtest subtest1,subtest2 will run only subtest1 and subtest2
+ * - --run-subtest ^subtest1,subtest2 will run all those except subtest1 and 
subtest2
+ *
+ * Usage help for tests with subtests can be obtained with the
  * "--help" command line option.
  */
 
@@ -786,6 +794,41 @@ void igt_simple_init_parse_opts(int *argc, char **argv,
extra_opt_handler, handler_data);
 }
 
+static bool check_testlist(const char *subtest_name)
+{
+   char *p;
+
+   /* Run subtests not in list
+* Look for subtest_name in list of form ^subtest1,subtest2,subtest3
+* or !test1,subtest2,subtest3
+* return true if not found.
+*/
+   if ((run_single_subtest[0] == '^') || (run_single_subtest[0] == '!')) {
+   char prefix = run_single_subtest[0];
+   p = strstr(run_single_subtest, subtest_name);
+   if (!((p) &&
+((*(p-1) == prefix) || (*(p-1) == ',')) &&
+((*(p+strlen(subtest_name)) == ',') || 
(*(p+strlen(subtest_name)) == '\0'
+   return true;
+   }
+   /* Run subtests in list
+* Look for subtest_name in list of form subtest1,subtest2,subtest3
+* return true if found.
+*/
+   else if (strstr(run_single_subtest, ",")) {
+   p = strstr(run_single_subtest, subtest_name);
+   if ((p) &&
+   ((p == run_single_subtest) || (*(p-1) == ',')) &&
+   ((*(p+strlen(subtest_name)) == ',') || 
(*(p+strlen(subtest_name)) == '\0')))
+   return true;
+   }
+   /* Run subtests that match shell wildcard */
+   else if (fnmatch(run_single_subtest, subtest_name, 0) == 0)
+   return true;
+
+   return false;
+}
+
 /*
  * Note: Testcases which use these helpers MUST NOT output anything to stdout
  * outside of places protected by igt_run_subtest checks - the piglit
@@ -814,7 +857,8 @@ bool __igt_run_subtest(const char *subtest_name)
}
 
if (run_single_subtest) {
-   if (fnmatch(run_single_subtest, subtest_name, 0) != 0)
+
+   if (check_testlist(subtest_name) == false)
return false;
else
run_single_subtest_found = true;
-- 
1.9.1

___
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/intel-gfx