Fabian Keil <[email protected]> wrote: > Daniel Stenberg <[email protected]> wrote: > > > On Wed, 21 May 2014, Fabian Keil wrote: > > > > > I'm still doing that and would like to get the required runtests.pl > > > changes > > > upstream after making them more generic. > > > > Thanks for keeping it up Fabian. I'm in favor of making sure that the curl > > test suite can be re-used for more purposes as I believe it can help > > improving > > more code and it will help the test suite and thus it also improves curl > > itself. > > Great. I'll look into making the patches more generic in the near feature.
Attached are generic versions of the patches. Of course I wouldn't mind if all of them got committed, but the last two could be rendered obsolete by changing the scope of a couple of variables from "my" to "our" and maybe that would be preferable from your point of view? > > > Patch 0003 isn't absolutely required and the chosen syntax turned out to > > > be > > > flawed > > > > ... but you're still proposing we merge it? > > Yes, but not in its current form. As there don't seem to be any objections > against the idea in general, I propose I change the patch to use a more > convenient syntax. I think it would be sufficient to allow to specify > multiple test number on the same line. For example I currently need: > > test:0030: Expected to fail with Privoxy. In case of missing server headers > Privoxy generates an error message the test doesn't expect. > test:0037: Expected to fail with Privoxy. In case of missing server headers > Privoxy generates an error message the test doesn't expect. > test:0066: Expected to fail with Privoxy. In case of missing server headers > Privoxy generates an error message the test doesn't expect. > test:1079: Expected to fail with Privoxy. In case of missing server headers > Privoxy generates an error message the test doesn't expect. > > Which could be expressed as: > test:30, 37, 66, 1079: Expected to fail with Privoxy. In case of missing > server headers Privoxy generates an error message the test doesn't expect. This is done in patch 3, one of the patches I now consider non-essential due to -R option added in patch 2. Fabian
From 7c6cc6bf5e8e3d09f85a6c9245abf584b74888ac Mon Sep 17 00:00:00 2001 From: Fabian Keil <[email protected]> Date: Sun, 29 Aug 2010 14:12:30 +0200 Subject: [PATCH 1/4] runtests.pl: Add a -P option to specify an external proxy ... that should be used when executing the tests. This doesn't work for all test types yet. The assumption is that the proxy is an HTTP proxy and the mode sets a User-Agent that allows the proxy to detect that the request is part of a curl test (only works for tool=curl). This option should be used together with -R to provide a customized compareparts() version that knows which proxy-specific header differences should be ignored. --- tests/runtests.pl | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/tests/runtests.pl b/tests/runtests.pl index 99be57d..b75831e 100755 --- a/tests/runtests.pl +++ b/tests/runtests.pl @@ -114,6 +114,9 @@ my $HOST6IP="[::1]"; # address on which the test server listens my $CLIENTIP="127.0.0.1"; # address which curl uses for incoming connections my $CLIENT6IP="[::1]"; # address which curl uses for incoming connections +my $use_external_proxy = 0; +my $proxy_address; + my $base = 8990; # base port number my $HTTPPORT; # HTTP server port @@ -695,6 +698,13 @@ sub stopserver { } ####################################################################### +# Return flags to let curl use an external HTTP proxy +# +sub getexternalproxyflags { + return " --proxy $proxy_address -H 'User-Agent: curl regression tests' "; +} + +####################################################################### # Verify that the server that runs on $ip, $port is our server. This also # implies that we can speak with it, as there might be occasions when the # server runs fine but we cannot talk to it ("Failed to connect to ::1: Can't @@ -726,6 +736,9 @@ sub verifyhttp { $flags .= "--globoff "; $flags .= "-1 " if($has_axtls); $flags .= "--insecure " if($proto eq 'https'); + if($use_external_proxy) { + $flags .= getexternalproxyflags(); + } $flags .= "\"$proto://$ip:$port/${bonus}verifiedserver\""; my $cmd = "$VCURL $flags 2>$verifylog"; @@ -800,6 +813,9 @@ sub verifyftp { $flags .= "--verbose "; $flags .= "--globoff "; $flags .= $extra; + if($use_external_proxy) { + $flags .= getexternalproxyflags(); + } $flags .= "\"$proto://$ip:$port/verifiedserver\""; my $cmd = "$VCURL $flags 2>$verifylog"; @@ -862,6 +878,9 @@ sub verifyrtsp { $flags .= "--silent "; $flags .= "--verbose "; $flags .= "--globoff "; + if($use_external_proxy) { + $flags .= getexternalproxyflags(); + } # currently verification is done using http $flags .= "\"http://$ip:$port/verifiedserver\"";; @@ -1003,6 +1022,9 @@ sub verifyhttptls { $flags .= "--tlsauthtype SRP "; $flags .= "--tlsuser jsmith "; $flags .= "--tlspassword abc "; + if($use_external_proxy) { + $flags .= getexternalproxyflags(); + } $flags .= "\"https://$ip:$port/verifiedserver\"";; my $cmd = "$VCURL $flags 2>$verifylog"; @@ -3090,6 +3112,10 @@ sub singletest { } } } + if($use_external_proxy) { + $ENV{http_proxy} = $proxy_address; + $ENV{HTTPS_PROXY} = $proxy_address; + } if(!$why) { # TODO: @@ -3317,6 +3343,9 @@ sub singletest { $fail_due_event_based--; } $cmdargs .= $cmd; + if ($use_external_proxy) { + $cmdargs .= " --proxy $proxy_address -H 'User-Agent: curl regression tests'"; + } } else { $cmdargs = " $cmd"; # $cmd is the command line for the test file @@ -4659,6 +4688,11 @@ while(@ARGV) { elsif($ARGV[0] eq "-p") { $postmortem=1; } + elsif($ARGV[0] eq "-P") { + shift @ARGV; + $use_external_proxy=1; + $proxy_address=$ARGV[0]; + } elsif($ARGV[0] eq "-l") { # lists the test case names only $listonly=1; @@ -4710,6 +4744,7 @@ Usage: runtests.pl [options] [test selection(s)] -l list all test case names/descriptions -n no valgrind -p print log file contents when a test fails + -P proxy use the specified proxy -r run time statistics -rf full run time statistics -s short output -- 1.9.0
From 97c0184000ab7859c68d62f27f8912414c55f208 Mon Sep 17 00:00:00 2001 From: Fabian Keil <[email protected]> Date: Thu, 5 Jun 2014 12:56:36 +0200 Subject: [PATCH 2/4] runtests.pl: Add -R to require additional perl libraries This is useful to change the behaviour of the script without having to modify the file itself, for example to use a custom compareparts() function that ignores header differences that are expected to occur when an external proxy is being used. Such differences are proxy-specific and thus the modifications should be maintained together with the proxy. --- tests/runtests.pl | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/tests/runtests.pl b/tests/runtests.pl index b75831e..af21140 100755 --- a/tests/runtests.pl +++ b/tests/runtests.pl @@ -4657,6 +4657,11 @@ while(@ARGV) { # short output $short=1; } + elsif($ARGV[0] eq "-R") { + # require additional library file + shift @ARGV; + require $ARGV[0]; + } elsif($ARGV[0] eq "-am") { # automake-style output $short=1; @@ -4747,6 +4752,7 @@ Usage: runtests.pl [options] [test selection(s)] -P proxy use the specified proxy -r run time statistics -rf full run time statistics + -R path require an additional perl library file to replace certain functions -s short output -am automake style output PASS/FAIL: [number] [name] -t[N] torture (simulate memory alloc failures); N means fail Nth alloc -- 1.9.0
From 62ffa42b7e4c402bea6621c2a4c30a38540d3107 Mon Sep 17 00:00:00 2001 From: Fabian Keil <[email protected]> Date: Sat, 17 Nov 2012 12:12:42 +0100 Subject: [PATCH 3/4] runtests.pl: Add an -m option to specify a "test manifest" file It's supposed to change runtests.pl's behaviour in various ways but at the moment it can only contain restraints for test numbers, keywords and tools. The idea is to let third parties like the Privoxy project distribute a manifest file with their tarballs that specifies which curl tests are expected to work, without having to fork the whole curl test suite. It might make sense to let the manifest specify additional (project-specific) tests directories and provide additional command line arguments for runtests.pl. The syntax should be changed to be extendable and maybe more closely reflect the "curl test" syntax. Currently it's a bunch of lines like these: test:$TESTNUMBER:Reason why this test with number $TESTNUMBER should be skipped keyword:$KEYWORD:Reason why tests whose keywords contain the $KEYWORD should be skipped tool:$TOOL:Reason why tests with tools that contain $TOOL should be skipped To specify multiple $TESTNUMBERs, $KEYWORDs and $TOOLs on a single line, split them with commas. --- tests/runtests.pl | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/tests/runtests.pl b/tests/runtests.pl index af21140..f24da4f 100755 --- a/tests/runtests.pl +++ b/tests/runtests.pl @@ -116,6 +116,7 @@ my $CLIENT6IP="[::1]"; # address which curl uses for incoming connections my $use_external_proxy = 0; my $proxy_address; +my %custom_skip_reasons; my $base = 8990; # base port number @@ -3055,6 +3056,31 @@ sub singletest { } } + if (!$why && defined $custom_skip_reasons{test}{$testnum}) { + $why = $custom_skip_reasons{test}{$testnum}; + } + + if (!$why && defined $custom_skip_reasons{tool}) { + foreach my $tool (getpart("client", "tool")) { + foreach my $tool_skip_pattern (keys $custom_skip_reasons{tool}) { + if ($tool =~ /$tool_skip_pattern/i) { + $why = $custom_skip_reasons{tool}{$tool_skip_pattern}; + } + } + } + } + + if (!$why && defined $custom_skip_reasons{keyword}) { + foreach my $keyword (getpart("info", "keywords")) { + foreach my $keyword_skip_pattern (keys $custom_skip_reasons{keyword}) { + if ($keyword =~ /$keyword_skip_pattern/i) { + $why = $custom_skip_reasons{keyword}{$keyword_skip_pattern}; + } + } + } + } + + # test definition may instruct to (un)set environment vars # this is done this early, so that the precheck can use environment # variables and still bail out fine on errors @@ -4698,6 +4724,27 @@ while(@ARGV) { $use_external_proxy=1; $proxy_address=$ARGV[0]; } + elsif($ARGV[0] eq "-m") { + shift @ARGV; + my $manifest = $ARGV[0]; + open(my $fd, "<", $manifest) or die "Couldn't open '$manifest': $!"; + while (my $line = <$fd>) { + next if ($line =~ /^#/); + chomp $line; + my ($type, $patterns, $skip_reason) = split(/\s*:\s*/, $line, 3); + + die "Unsupported type: $type\n" if($type !~ /^keyword|test|tool$/); + + foreach my $pattern (split(/,/, $patterns)) { + if ($type =~ /^test$/) { + # Strip leading zeros in the test number + $pattern = int($pattern); + } + $custom_skip_reasons{$type}{$pattern} = $skip_reason; + } + } + close($fd); + } elsif($ARGV[0] eq "-l") { # lists the test case names only $listonly=1; @@ -4747,6 +4794,7 @@ Usage: runtests.pl [options] [test selection(s)] -h this help text -k keep stdout and stderr files present after tests -l list all test case names/descriptions + -m file load the specifed test manifest to decide which tests get executed -n no valgrind -p print log file contents when a test fails -P proxy use the specified proxy -- 1.9.0
From 7ff8c44d1e51058c397e19884d101ba1895c5b0c Mon Sep 17 00:00:00 2001 From: Fabian Keil <[email protected]> Date: Sat, 17 Nov 2012 11:42:52 +0100 Subject: [PATCH 4/4] runtests.pl: Add the -H option to specify a non-default IPv4 host IP address This is useful when testing external proxies with the -P option. --- tests/runtests.pl | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tests/runtests.pl b/tests/runtests.pl index f24da4f..3a3f44d 100755 --- a/tests/runtests.pl +++ b/tests/runtests.pl @@ -4679,6 +4679,10 @@ while(@ARGV) { $gdbthis=1; $gdbxwin=1; } + elsif ($ARGV[0] eq "-H") { + shift @ARGV; + $HOSTIP=$ARGV[0]; + } elsif($ARGV[0] eq "-s") { # short output $short=1; @@ -4792,6 +4796,7 @@ Usage: runtests.pl [options] [test selection(s)] -g run the test case with gdb -gw run the test case with gdb as a windowed application -h this help text + -H addr use the specified IPv4 host IP address instead of the default -k keep stdout and stderr files present after tests -l list all test case names/descriptions -m file load the specifed test manifest to decide which tests get executed -- 1.9.0
signature.asc
Description: PGP signature
------------------------------------------------------------------- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
