Fabian Keil <[email protected]> wrote: > Daniel Stenberg <[email protected]> wrote:
> > Can elaborate on how you test with "external proxies" and what was the > > problem > > you fixed here? > > I'm using a couple of runtest.pl modifications that haven't all been > finished yet. I added them to the tarball as a "preview" (0010-0017). > > With the patches applied, I test Privoxy like this: > ./runtests.pl -n -P http://10.0.0.1:8118/ -H 10.0.0.1 -m > curl-test-manifest-for-privoxy HTTP I'm still doing that and would like to get the required runtests.pl changes upstream after making them more generic. Making it more convenient to run (parts of) curl's regression tests with external proxies doesn't require a lot of changes and as it should increase the number of people who frequently test (parts of) curl, I think curl itself would benefit as well. I attached some of the patches I'm currently using as examples. The only Privoxy-specific patch is 0001, to make it generic it should be sufficient to add another option to specify an external script to compare() the test results with proxy-specific knowledge (currently done inline). Patch 0003 isn't absolutely required and the chosen syntax turned out to be flawed (it's painful to read and write, I rely on another script to regenerate the file ...), but the following part of the commit message is still relevant: | The idea is to let third parties like the Privoxy project | distribute a manifest file with their tarballs that specifies | which curl tests are expected to work, without having to | fork the whole curl test suite. | | It would make sense to let the manifest specify additional | (project-specific) tests directories and provide additional | command line arguments for runtests.pl. Feedback welcome. Fabian
From 3f9bcc4a0dc2e37b4af0fb533e95b1687ae457f6 Mon Sep 17 00:00:00 2001 From: Fabian Keil <[email protected]> Date: Sun, 29 Aug 2010 14:12:30 +0200 Subject: [PATCH 1/4] runtests.pl: Add a -P option to specify an external proxy ... that should be used when executing the tests. This doesn't work for all test types yet. The assumption is that the proxy is an HTTP proxy and the mode enables a couple of "tricks" to reduce false positives: - Set a User-Agent that allows the proxy to detect that the request is part of a curl test (only works for tool=curl). - Filter out "Proxy-Connection:" headers when checking for test success. - Deal with tests that don't expect CRLF header endings as long as the test uses it consistently. - Reduce spaces in headers with a too-simplistic heuristic that happens to work for the existing tests. - Strip connection-established headers if they are unexpected by the test. --- tests/runtests.pl | 110 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) diff --git a/tests/runtests.pl b/tests/runtests.pl index 99be57d..7d90aa1 100755 --- a/tests/runtests.pl +++ b/tests/runtests.pl @@ -114,6 +114,9 @@ my $HOST6IP="[::1]"; # address on which the test server listens my $CLIENTIP="127.0.0.1"; # address which curl uses for incoming connections my $CLIENT6IP="[::1]"; # address which curl uses for incoming connections +my $use_external_proxy = 0; +my $proxy_address; + my $base = 8990; # base port number my $HTTPPORT; # HTTP server port @@ -726,6 +729,9 @@ sub verifyhttp { $flags .= "--globoff "; $flags .= "-1 " if($has_axtls); $flags .= "--insecure " if($proto eq 'https'); + if($use_external_proxy) { + $flags .= " --proxy $proxy_address -H 'User-Agent: curl regression tests' "; + } $flags .= "\"$proto://$ip:$port/${bonus}verifiedserver\""; my $cmd = "$VCURL $flags 2>$verifylog"; @@ -800,6 +806,9 @@ sub verifyftp { $flags .= "--verbose "; $flags .= "--globoff "; $flags .= $extra; + if($use_external_proxy) { + $flags .= " --proxy $proxy_address -H 'User-Agent: curl regression tests' "; + } $flags .= "\"$proto://$ip:$port/verifiedserver\""; my $cmd = "$VCURL $flags 2>$verifylog"; @@ -862,6 +871,9 @@ sub verifyrtsp { $flags .= "--silent "; $flags .= "--verbose "; $flags .= "--globoff "; + if($use_external_proxy) { + $flags .= " --proxy $proxy_address -H 'User-Agent: curl regression tests' "; + } # currently verification is done using http $flags .= "\"http://$ip:$port/verifiedserver\"";; @@ -1003,6 +1015,9 @@ sub verifyhttptls { $flags .= "--tlsauthtype SRP "; $flags .= "--tlsuser jsmith "; $flags .= "--tlspassword abc "; + if($use_external_proxy) { + $flags .= " --proxy $proxy_address -H 'User-Agent: curl regression tests' "; + } $flags .= "\"https://$ip:$port/verifiedserver\"";; my $cmd = "$VCURL $flags 2>$verifylog"; @@ -2230,6 +2245,12 @@ sub filteroff { return 0; } +sub print_skipped_header ($) { + my $skipped_header = shift; + $skipped_header =~ s@\r?\n$@@; + print "Skipping '$skipped_header'\n"; +} + ####################################################################### # compare test results with the expected output, we might filter off # some pattern that is allowed to differ, output test results @@ -2238,6 +2259,82 @@ sub compare { # filter off patterns _before_ this comparison! my ($testnum, $testname, $subject, $firstref, $secondref)=@_; + if ($use_external_proxy) { + my @tmp; + my $crlf_expected = 0; + my $connection_header_expected = 0; + my $parsing_headers = 0; + my $connection_established_expected = 0; + + foreach (@$secondref) { + if (/^HTTP/) { + $parsing_headers = 1; + # This check will likely need modifications for other proxies + if (/Connection established/) { + $connection_established_expected = 1; + } + } + if (/^\r?\n$/) { + $parsing_headers = 0; + } + if (/\r\n$/) { + $crlf_expected = 1; # XXX: assume the expectancy is consistant. + } + if (/^Connection:/) { + $connection_header_expected = 1; + } + if (/^Connection:/) { + $connection_header_expected = 1; + } + if ($parsing_headers) { + # This is probably a header, normalize spaces + # similar to the way Privoxy does. This is required + # for tests 29, 40, 42 and 54. + # XXX: this is a hack + next if /"/; + s@ +@ @g; + } + } + + if($verbose) { + print "Expecting " . ($crlf_expected ? "" : "no ") . "crlf\n"; + print "Expecting " . ($connection_header_expected ? "a" : "no") . " Connection: header\n"; + print "Expecting " . ($connection_established_expected ? "" : "no ") . "Connection established\n"; + } + + $parsing_headers = 0; + my $remove_headers = 0; + foreach (@$firstref) { + if (/^HTTP/) { + $parsing_headers = 1; + # This check will likely need modifications for other proxies + if (!$connection_established_expected && /Connection established/) { + $remove_headers = 1; + } + } + if ($remove_headers) { + print_skipped_header($_) if($verbose); + $remove_headers = 0 if (/^\r?\n$/); + next; + } + if (/^\r?\n$/) { + $parsing_headers = 0; + } + + s@\r\n$@\n@ unless ($crlf_expected); + if (m/^Proxy-Connection: keep-alive/ or m/^User-Agent: curl regression tests/) { + print_skipped_header($_) if($verbose); + next; + } + if (m/^Connection:/ and not $connection_header_expected) { + print_skipped_header($_) if($verbose); + next; + } + push @tmp, $_; + } + $firstref = \@tmp; + } + my $result = compareparts($firstref, $secondref); if($result) { @@ -3090,6 +3187,10 @@ sub singletest { } } } + if($use_external_proxy) { + $ENV{http_proxy} = $proxy_address; + $ENV{HTTPS_PROXY} = $proxy_address; + } if(!$why) { # TODO: @@ -3317,6 +3418,9 @@ sub singletest { $fail_due_event_based--; } $cmdargs .= $cmd; + if ($use_external_proxy) { + $cmdargs .= " --proxy $proxy_address -H 'User-Agent: curl regression tests'"; + } } else { $cmdargs = " $cmd"; # $cmd is the command line for the test file @@ -4659,6 +4763,11 @@ while(@ARGV) { elsif($ARGV[0] eq "-p") { $postmortem=1; } + elsif($ARGV[0] eq "-P") { + shift @ARGV; + $use_external_proxy=1; + $proxy_address=$ARGV[0]; + } elsif($ARGV[0] eq "-l") { # lists the test case names only $listonly=1; @@ -4710,6 +4819,7 @@ Usage: runtests.pl [options] [test selection(s)] -l list all test case names/descriptions -n no valgrind -p print log file contents when a test fails + -P proxy use the specified proxy -r run time statistics -rf full run time statistics -s short output -- 1.9.0
From 40a9839aa6fd8cd3fbc0e8d180e78fd4ff067fea Mon Sep 17 00:00:00 2001 From: Fabian Keil <[email protected]> Date: Sat, 17 Nov 2012 11:42:52 +0100 Subject: [PATCH 2/4] runtests.pl: Add the -H option to specify a non-default IPv4 host IP address This is useful when testing external proxies with the -P option. --- tests/runtests.pl | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tests/runtests.pl b/tests/runtests.pl index 7d90aa1..b8ec7f9 100755 --- a/tests/runtests.pl +++ b/tests/runtests.pl @@ -4728,6 +4728,10 @@ while(@ARGV) { $gdbthis=1; $gdbxwin=1; } + elsif ($ARGV[0] eq "-H") { + shift @ARGV; + $HOSTIP=$ARGV[0]; + } elsif($ARGV[0] eq "-s") { # short output $short=1; @@ -4815,6 +4819,7 @@ Usage: runtests.pl [options] [test selection(s)] -g run the test case with gdb -gw run the test case with gdb as a windowed application -h this help text + -H addr use the specified IPv4 host IP address instead of the default -k keep stdout and stderr files present after tests -l list all test case names/descriptions -n no valgrind -- 1.9.0
From f414884978fe4f6697e62fffe3d314925d8d46ab Mon Sep 17 00:00:00 2001 From: Fabian Keil <[email protected]> Date: Sat, 17 Nov 2012 12:12:42 +0100 Subject: [PATCH 3/4] runtests.pl: Add an -m option to specify a "test manifest" file It's supposed to change runtests.pl's behaviour in various ways but at the moment it can only contain restraints for test numbers, keywords and tools. The syntax should be changed to be extendable and maybe more closely reflect the "curl test" syntax. Currently it's a bunch of lines like these: test:$TESTNUMBER:Reason why this test with number $TESTNUMBER should be skipped keyword:$KEYWORD:Reason why tests whose keywords contain the $KEYWORD should be skipped tool:$TOOL:Reason why tests with tools that contain $TOOL should be skipped The idea is to let third parties like the Privoxy project distribute a manifest file with their tarballs that specifies which curl tests are expected to work, without having to fork the whole curl test suite. It would make sense to let the manifest specify additional (project-specific) tests directories and provide additional command line arguments for runtests.pl. --- tests/runtests.pl | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/tests/runtests.pl b/tests/runtests.pl index b8ec7f9..ab66987 100755 --- a/tests/runtests.pl +++ b/tests/runtests.pl @@ -116,6 +116,7 @@ my $CLIENT6IP="[::1]"; # address which curl uses for incoming connections my $use_external_proxy = 0; my $proxy_address; +my %custom_skip_reasons; my $base = 8990; # base port number @@ -3130,6 +3131,31 @@ sub singletest { } } + if (!$why && defined $custom_skip_reasons{test}{$testnum}) { + $why = $custom_skip_reasons{test}{$testnum}; + } + + if (!$why && defined $custom_skip_reasons{tool}) { + foreach my $tool (getpart("client", "tool")) { + foreach my $tool_skip_pattern (keys $custom_skip_reasons{tool}) { + if ($tool =~ /$tool_skip_pattern/i) { + $why = $custom_skip_reasons{tool}{$tool_skip_pattern}; + } + } + } + } + + if (!$why && defined $custom_skip_reasons{keyword}) { + foreach my $keyword (getpart("info", "keywords")) { + foreach my $keyword_skip_pattern (keys $custom_skip_reasons{keyword}) { + if ($keyword =~ /$keyword_skip_pattern/i) { + $why = $custom_skip_reasons{keyword}{$keyword_skip_pattern}; + } + } + } + } + + # test definition may instruct to (un)set environment vars # this is done this early, so that the precheck can use environment # variables and still bail out fine on errors @@ -4772,6 +4798,24 @@ while(@ARGV) { $use_external_proxy=1; $proxy_address=$ARGV[0]; } + elsif($ARGV[0] eq "-m") { + shift @ARGV; + my $manifest = $ARGV[0]; + open(my $fd, "<", $manifest) or die "Couldn't open '$manifest': $!"; + while (my $line = <$fd>) { + next if ($line =~ /^#/); + chomp $line; + my ($type, $pattern, $skip_reason) = split(/\s*:\s*/, $line, 3); + + die "Unsupported type: $type\n" if($type !~ /^keyword|test|tool$/); + if ($type =~ /^test$/) { + # Strip leading zeros in the test number + $pattern = int($pattern); + } + $custom_skip_reasons{$type}{$pattern} = $skip_reason; + } + close($fd); + } elsif($ARGV[0] eq "-l") { # lists the test case names only $listonly=1; @@ -4822,6 +4866,7 @@ Usage: runtests.pl [options] [test selection(s)] -H addr use the specified IPv4 host IP address instead of the default -k keep stdout and stderr files present after tests -l list all test case names/descriptions + -m file load the specifed test manifest to decide which tests get executed -n no valgrind -p print log file contents when a test fails -P proxy use the specified proxy -- 1.9.0
signature.asc
Description: PGP signature
------------------------------------------------------------------- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
