Hi Frederic,
Op 7-1-2019 om 10:13 schreef Frederic Lecaille:
On 12/23/18 11:38 PM, PiBa-NL wrote:
As requested hereby the regtest send for inclusion into the git
repository.
It is OK like that.
Note that you patch do not add reg-test/filters/common.pem which could
be a symlink to ../ssl/common.pem.
Also note that since 8f16148Christopher's commit, we add such a line
where possible:
${no-htx} option http-use-htx
We should also rename your test files to reg-test/filters/h00000.*
Thank you.
Fred.
Together with these changes you have supplied me already off-list, i've
also added a " --max-time 15" for the curl request, that should be
sufficient for most systems to complete the 3 second testcase, and
allows the shell command to complete without varnishtest killing it
after a timeout and not showing any of the curl output..
One last question, currently its being added to a new folder:
reg-test/filters/ , perhaps it should be in reg-test/compression/ ?
If you agree that needs changing i guess that can be done upon
committing it?
Note that the test fails on my FreeBSD system when using HTX when using
'2.0-dev0-251a6b7 2019/01/08', i'm not aware it ever worked (i didn't
test it with HTX before..).
**** top 15.2 shell_out|curl: (28) Operation timed out after 15036
milliseconds with 187718 bytes received
Log attached.. Would it help to log it with the complete "filter trace
name BEFORE / filter compression / filter trace name AFTER" ? Or are
there other details i could try and gather?
Regards,
PiBa-NL (Pieter)
From 793e770b399157a1549a2655612a29845b165dd6 Mon Sep 17 00:00:00 2001
From: PiBa-NL <piba.nl....@gmail.com>
Date: Sun, 23 Dec 2018 21:21:51 +0100
Subject: [PATCH] REGTEST: filters: add compression test
This test checks that data transferred with compression is correctly received
at different download speeds
---
reg-tests/filters/common.pem | 1 +
reg-tests/filters/s00000.lua | 19 ++++++++++++++
reg-tests/filters/s00000.vtc | 59 ++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 79 insertions(+)
create mode 120000 reg-tests/filters/common.pem
create mode 100644 reg-tests/filters/s00000.lua
create mode 100644 reg-tests/filters/s00000.vtc
diff --git a/reg-tests/filters/common.pem b/reg-tests/filters/common.pem
new file mode 120000
index 00000000..a4433d56
--- /dev/null
+++ b/reg-tests/filters/common.pem
@@ -0,0 +1 @@
+../ssl/common.pem
\ No newline at end of file
diff --git a/reg-tests/filters/s00000.lua b/reg-tests/filters/s00000.lua
new file mode 100644
index 00000000..2cc874b9
--- /dev/null
+++ b/reg-tests/filters/s00000.lua
@@ -0,0 +1,19 @@
+
+local data = "abcdefghijklmnopqrstuvwxyz"
+local responseblob = ""
+for i = 1,10000 do
+ responseblob = responseblob .. "\r\n" .. i .. data:sub(1, math.floor(i % 27))
+end
+
+http01applet = function(applet)
+ local response = responseblob
+ applet:set_status(200)
+ applet:add_header("Content-Type", "application/javascript")
+ applet:add_header("Content-Length", string.len(response)*10)
+ applet:start_response()
+ for i = 1,10 do
+ applet:send(response)
+ end
+end
+
+core.register_service("fileloader-http01", "http", http01applet)
diff --git a/reg-tests/filters/s00000.vtc b/reg-tests/filters/s00000.vtc
new file mode 100644
index 00000000..231344a6
--- /dev/null
+++ b/reg-tests/filters/s00000.vtc
@@ -0,0 +1,59 @@
+# Checks that compression doesnt cause corruption..
+
+varnishtest "Compression validation"
+#REQUIRE_VERSION=1.6
+
+feature ignore_unknown_macro
+
+haproxy h1 -conf {
+global
+# log stdout format short daemon
+ lua-load ${testdir}/s00000.lua
+
+defaults
+ mode http
+ log global
+ ${no-htx} option http-use-htx
+ option httplog
+
+frontend main-https
+ bind "fd@${fe1}" ssl crt ${testdir}/common.pem
+ compression algo gzip
+ compression type text/html text/plain application/json
application/javascript
+ compression offload
+ use_backend TestBack if TRUE
+
+backend TestBack
+ server LocalSrv ${h1_fe2_addr}:${h1_fe2_port}
+
+listen fileloader
+ mode http
+ bind "fd@${fe2}"
+ http-request use-service lua.fileloader-http01
+} -start
+
+shell {
+ HOST=${h1_fe1_addr}
+ if [ "${h1_fe1_addr}" = "::1" ] ; then
+ HOST="\[::1\]"
+ fi
+
+ md5=$(which md5 || which md5sum)
+
+ if [ -z $md5 ] ; then
+ echo "MD5 checksum utility not found"
+ exit 1
+ fi
+
+ expectchecksum="4d9c62aa5370b8d5f84f17ec2e78f483"
+
+ for opt in "" "--limit-rate 300K" "--limit-rate 500K" ; do
+ checksum=$(curl --max-time 15 --compressed -k
"https://$HOST:${h1_fe1_port}"; $opt | $md5 | cut -d ' ' -f1)
+ if [ "$checksum" != "$expectchecksum" ] ; then
+ echo "Expecting checksum $expectchecksum"
+ echo "Received checksum: $checksum"
+ exit 1;
+ fi
+ done
+
+} -run
--
2.11.0
root@freebsd11:/usr/ports/net/haproxy-devel # varnishtest -v -t 100 -Dno-htx=""
./work/haproxy-251a6b7/reg-tests/filters/s00000.vtc
* top 0.0 TEST ./work/haproxy-251a6b7/reg-tests/filters/s00000.vtc starting
**** top 0.0 extmacro def pwd=/usr/ports/net/haproxy-devel
**** top 0.0 extmacro def no-htx=
**** top 0.0 extmacro def localhost=127.0.0.1
**** top 0.0 extmacro def bad_backend=127.0.0.1 18802
**** top 0.0 extmacro def bad_ip=192.0.2.255
**** top 0.0 macro def
testdir=/usr/ports/net/haproxy-devel/./work/haproxy-251a6b7/reg-tests/filters
**** top 0.0 macro def tmpdir=/tmp/vtc.96685.14be13d7
** top 0.0 === varnishtest "Compression validation"
* top 0.0 VTEST Compression validation
** top 0.0 === feature ignore_unknown_macro
** top 0.0 === haproxy h1 -conf {
**** h1 0.0 macro def h1_cli_sock=::1 23496
**** h1 0.0 macro def h1_cli_addr=::1
**** h1 0.0 macro def h1_cli_port=23496
**** h1 0.0 setenv(cli, 3)
**** h1 0.0 macro def h1_fe1_sock=::1 51434
**** h1 0.0 macro def h1_fe1_addr=::1
**** h1 0.0 macro def h1_fe1_port=51434
**** h1 0.0 setenv(fe1, 4)
**** h1 0.0 macro def h1_fe2_sock=::1 20633
**** h1 0.0 macro def h1_fe2_addr=::1
**** h1 0.0 macro def h1_fe2_port=20633
**** h1 0.0 setenv(fe2, 5)
**** h1 0.0 conf| global
**** h1 0.0 conf|\tstats socket /tmp/vtc.96685.14be13d7/h1/stats.sock level
admin mode 600
**** h1 0.0 conf| stats socket "fd@${cli}" level admin
**** h1 0.0 conf|
**** h1 0.0 conf|global
**** h1 0.0 conf|#\tlog stdout format short daemon
**** h1 0.0
conf|\tlua-load\t\t/usr/ports/net/haproxy-devel/./work/haproxy-251a6b7/reg-tests/filters/s00000.lua
**** h1 0.0 conf|
**** h1 0.0 conf|defaults
**** h1 0.0 conf|\tmode\t\t\thttp
**** h1 0.0 conf|\tlog\t\t\tglobal
**** h1 0.0 conf|\t option http-use-htx
**** h1 0.0 conf|\toption\t\t\thttplog
**** h1 0.0 conf|
**** h1 0.0 conf|frontend main-https
**** h1 0.0 conf|\tbind\t\t\t"fd@${fe1}" ssl crt
/usr/ports/net/haproxy-devel/PB-TEST/common.pem
**** h1 0.0 conf|\tcompression algo gzip
**** h1 0.0 conf|\tcompression type text/html text/plain application/json
application/javascript
**** h1 0.0 conf|\tcompression offload
**** h1 0.0 conf|\tuse_backend TestBack if TRUE
**** h1 0.0 conf|
**** h1 0.0 conf|backend TestBack
**** h1 0.0 conf|\tserver\tLocalSrv ::1:20633
**** h1 0.0 conf|
**** h1 0.0 conf|listen fileloader
**** h1 0.0 conf|\tmode http
**** h1 0.0 conf|\tbind "fd@${fe2}"
**** h1 0.0 conf|\thttp-request use-service lua.fileloader-http01
** h1 0.0 haproxy_start
**** h1 0.0 opt_worker 0 opt_daemon 0 opt_check_mode 0
**** h1 0.0 argv|exec haproxy -d -f /tmp/vtc.96685.14be13d7/h1/cfg
**** h1 0.0 XXX 7 @586
*** h1 0.0 PID: 96689
**** h1 0.0 macro def h1_pid=96689
**** h1 0.0 macro def h1_name=/tmp/vtc.96685.14be13d7/h1
** top 0.0 === shell {
**** top 0.0 shell_cmd|exec 2>&1 ;
**** top 0.0 shell_cmd| HOST=::1
**** top 0.0 shell_cmd| if [ "::1" = "::1" ] ; then
**** top 0.0 shell_cmd| HOST="\[::1\]"
**** top 0.0 shell_cmd| fi
**** top 0.0 shell_cmd|
**** top 0.0 shell_cmd| md5=$(which md5 || which md5sum)
**** top 0.0 shell_cmd|
**** top 0.0 shell_cmd| if [ -z $md5 ] ; then
**** top 0.0 shell_cmd| echo "MD5 checksum utility not found"
**** top 0.0 shell_cmd| exit 1
**** top 0.0 shell_cmd| fi
**** top 0.0 shell_cmd|
**** top 0.0 shell_cmd| expectchecksum="4d9c62aa5370b8d5f84f17ec2e78f483"
**** top 0.0 shell_cmd|
**** top 0.0 shell_cmd| for opt in "" "--limit-rate 300K" "--limit-rate
500K" ; do
**** top 0.0 shell_cmd| checksum=$(curl --max-time 15 --compressed -k
"https://$HOST:51434"; $opt | $md5 | cut -d ' ' -f1)
**** top 0.0 shell_cmd| if [ "$checksum" != "$expectchecksum" ] ; then
**** top 0.0 shell_cmd| echo "Expecting checksum $expectchecksum"
**** top 0.0 shell_cmd| echo "Received checksum: $checksum"
**** top 0.0 shell_cmd| exit 1;
**** top 0.0 shell_cmd| fi
**** top 0.0 shell_cmd| done
**** top 0.0 shell_cmd|
*** h1 0.1 debug|[WARNING] 007/221855 (96689) : config : missing timeouts
for frontend 'main-https'.
*** h1 0.1 debug| | While not properly invalid, you will certainly
encounter various problems
*** h1 0.1 debug| | with such a configuration. To fix this, please ensure
that all following
*** h1 0.1 debug| | timeouts are set to a non-zero value: 'client',
'connect', 'server'.
*** h1 0.1 debug|[WARNING] 007/221855 (96689) : config : log format ignored
for frontend 'main-https' since it has no log address.
*** h1 0.1 debug|[WARNING] 007/221855 (96689) : config : missing timeouts
for backend 'TestBack'.
*** h1 0.1 debug| | While not properly invalid, you will certainly
encounter various problems
*** h1 0.1 debug| | with such a configuration. To fix this, please ensure
that all following
*** h1 0.1 debug| | timeouts are set to a non-zero value: 'client',
'connect', 'server'.
*** h1 0.1 debug|[WARNING] 007/221855 (96689) : config : missing timeouts
for proxy 'fileloader'.
*** h1 0.1 debug| | While not properly invalid, you will certainly
encounter various problems
*** h1 0.1 debug| | with such a configuration. To fix this, please ensure
that all following
*** h1 0.1 debug| | timeouts are set to a non-zero value: 'client',
'connect', 'server'.
*** h1 0.1 debug|[WARNING] 007/221855 (96689) : config : log format ignored
for proxy 'fileloader' since it has no log address.
*** h1 0.1 debug|[WARNING] 007/221855 (96689) : Setting
tune.ssl.default-dh-param to 1024 by default, if your workload permits it you
should set it to at least 2048. Please set a value >= 1024 to make this warning
disappear.
*** h1 0.1 debug|Note: setting global.maxconn to 2000.
*** h1 0.1 debug|Available polling systems :
*** h1 0.1 debug| kqueue : pref=300,
*** h1 0.1 debug| test result OK
*** h1 0.1 debug| poll : pref=200, test result OK
*** h1 0.1 debug| select : pref=150, test result FAILED
*** h1 0.1 debug|Total: 3 (2 usable), will use kqueue.
*** h1 0.1 debug|
*** h1 0.1 debug|Available filters :
*** h1 0.1 debug|\t[SPOE] spoe
*** h1 0.1 debug|\t[COMP] compression
*** h1 0.1 debug|\t[CACHE] cache
*** h1 0.1 debug|\t[TRACE] trace
*** h1 0.1 debug|Using kqueue() as the polling mechanism.
*** h1 0.2 debug|00000000:main-https.accept(0004)=000a from [::1:11266]
ALPN=<none>
*** h1 0.2 debug|00000000:main-https.clireq[000a:ffffffff]: GET / HTTP/1.1
*** h1 0.2 debug|00000000:main-https.clihdr[000a:ffffffff]: host:
[::1]:51434
*** h1 0.2 debug|00000000:main-https.clihdr[000a:ffffffff]: user-agent:
curl/7.60.0
*** h1 0.2 debug|00000000:main-https.clihdr[000a:ffffffff]: accept: */*
*** h1 0.2 debug|00000000:main-https.clihdr[000a:ffffffff]:
accept-encoding: deflate, gzip
*** h1 0.2 debug|00000001:fileloader.accept(0005)=000c from [::1:10697]
ALPN=<none>
*** h1 0.2 debug|00000001:fileloader.clireq[000c:ffffffff]: GET / HTTP/1.1
*** h1 0.2 debug|00000001:fileloader.clihdr[000c:ffffffff]: host:
[::1]:51434
*** h1 0.2 debug|00000001:fileloader.clihdr[000c:ffffffff]: user-agent:
curl/7.60.0
*** h1 0.2 debug|00000001:fileloader.clihdr[000c:ffffffff]: accept: */*
*** h1 0.2 debug|00000001:fileloader.srvrep[000c:ffffffff]: HTTP/1.1 200 OK
*** h1 0.2 debug|00000001:fileloader.srvhdr[000c:ffffffff]: content-type:
application/javascript
*** h1 0.2 debug|00000001:fileloader.srvhdr[000c:ffffffff]: content-length:
1888190
*** h1 0.2 debug|00000000:TestBack.srvrep[000a:000b]: HTTP/1.1 200 OK
*** h1 0.2 debug|00000000:TestBack.srvhdr[000a:000b]: content-type:
application/javascript
*** h1 0.2 debug|00000000:TestBack.srvhdr[000a:000b]: content-length:
1888190
*** h1 0.2 debug|00000001:fileloader.srvcls[000c:ffffffff]
*** h1 0.2 debug|00000001:fileloader.clicls[000c:ffffffff]
*** h1 0.2 debug|00000001:fileloader.closed[000c:ffffffff]
*** h1 0.2 debug|00000002:fileloader.accept(0005)=000c from [::1:10697]
ALPN=<none>
*** h1 0.2 debug|00000000:TestBack.srvcls[000a:000b]
*** h1 0.2 debug|00000000:TestBack.clicls[000a:000b]
*** h1 0.2 debug|00000000:TestBack.closed[000a:000b]
*** h1 0.2 debug|00000003:main-https.accept(0004)=000a from [::1:11266]
ALPN=<none>
*** h1 0.2 debug|00000003:main-https.clicls[000a:ffffffff]
*** h1 0.2 debug|00000003:main-https.closed[000a:ffffffff]
*** h1 0.2 debug|00000004:main-https.accept(0004)=000a from [::1:60862]
ALPN=<none>
*** h1 0.2 debug|00000004:main-https.clireq[000a:ffffffff]: GET / HTTP/1.1
*** h1 0.2 debug|00000004:main-https.clihdr[000a:ffffffff]: host:
[::1]:51434
*** h1 0.2 debug|00000004:main-https.clihdr[000a:ffffffff]: user-agent:
curl/7.60.0
*** h1 0.2 debug|00000004:main-https.clihdr[000a:ffffffff]: accept: */*
*** h1 0.2 debug|00000004:main-https.clihdr[000a:ffffffff]:
accept-encoding: deflate, gzip
*** h1 0.2 debug|00000005:fileloader.accept(0005)=000e from [::1:54389]
ALPN=<none>
*** h1 0.2 debug|00000005:fileloader.clireq[000e:ffffffff]: GET / HTTP/1.1
*** h1 0.2 debug|00000005:fileloader.clihdr[000e:ffffffff]: host:
[::1]:51434
*** h1 0.2 debug|00000005:fileloader.clihdr[000e:ffffffff]: user-agent:
curl/7.60.0
*** h1 0.2 debug|00000005:fileloader.clihdr[000e:ffffffff]: accept: */*
*** h1 0.2 debug|00000005:fileloader.srvrep[000e:ffffffff]: HTTP/1.1 200 OK
*** h1 0.2 debug|00000005:fileloader.srvhdr[000e:ffffffff]: content-type:
application/javascript
*** h1 0.2 debug|00000005:fileloader.srvhdr[000e:ffffffff]: content-length:
1888190
*** h1 0.2 debug|00000004:TestBack.srvrep[000a:000d]: HTTP/1.1 200 OK
*** h1 0.2 debug|00000004:TestBack.srvhdr[000a:000d]: content-type:
application/javascript
*** h1 0.2 debug|00000004:TestBack.srvhdr[000a:000d]: content-length:
1888190
*** h1 0.2 debug|00000005:fileloader.srvcls[000e:ffffffff]
*** h1 0.3 debug|00000005:fileloader.clicls[000e:ffffffff]
*** h1 0.3 debug|00000005:fileloader.closed[000e:ffffffff]
*** h1 0.3 debug|00000006:fileloader.accept(0005)=000e from [::1:54389]
ALPN=<none>
*** h1 1.2 debug|00000002:fileloader.clicls[000c:ffffffff]
*** h1 1.2 debug|00000002:fileloader.closed[000c:ffffffff]
**** top 15.2 shell_out| % Total % Received % Xferd Average Speed Time
Time Time Current
**** top 15.2 shell_out| Dload Upload Total
Spent Left Speed
**** top 15.2 shell_out|\r 0 0 0 0 0 0 0 0
--:--:-- --:--:-- --:--:-- 0\r100 417k 0 417k 0 0 2414k 0
--:--:-- --:--:-- --:--:-- 2414k
**** top 15.2 shell_out| % Total % Received % Xferd Average Speed Time
Time Time Current
**** top 15.2 shell_out| Dload Upload Total
Spent Left Speed
**** top 15.2 shell_out|\r 0 0 0 0 0 0 0 0
--:--:-- --:--:-- --:--:-- 0\r100 183k 0 183k 0 0 109k 0
--:--:-- 0:00:01 --:--:-- 109k\r100 183k 0 183k 0 0 68887 0
--:--:-- 0:00:02 --:--:-- 68862\r100 183k 0 183k 0 0 49555 0
--:--:-- 0:00:03 --:--:-- 49542\r100 183k 0 183k 0 0 38688 0
--:--:-- 0:00:04 --:--:-- 38680\r100 183k 0 183k 0 0 31735 0
--:--:-- 0:00:05 --:--:-- 31730\r100 183k 0 183k 0 0 26897 0
--:--:-- 0:00:06 --:--:-- 0\r100 183k 0 183k 0 0 23345 0
--:--:-- 0:00:08 --:--:-- 0\r100 183k 0 183k 0 0 20619 0
--:--:-- 0:00:09 --:--:-- 0\r100 183k 0 183k 0 0 18463 0
--:--:-- 0:00:10 --:--:-- 0\r100 183k 0 183k 0 0 16714 0
--:--:-- 0:00:11 --:--:-- 0\r100 183k 0 183k 0 0 15269 0
--:--:-- 0:00:12 --:--:-- 0\r100 183k 0 183k 0 0 14052 0
--:--:-- 0:00:13 --:--:-- 0\r100 183k 0 183k 0 0 13017 0
--:--:-- 0:00:14 --:--:-- 0\r100 183k 0 183k 0 0 12484 0
--:--:-- 0:00:15 --:--:-- 0
**** top 15.2 shell_out|curl: (28) Operation timed out after 15036
milliseconds with 187718 bytes received
**** top 15.2 shell_out|Expecting checksum 4d9c62aa5370b8d5f84f17ec2e78f483
**** top 15.2 shell_out|Received checksum: f8ea6a8f971541896cbdb625309f7d13
**** top 15.2 shell_status = 0x0001
---- top 15.2 shell_exit not as expected: got 0x0001 wanted 0x0000
* top 15.2 RESETTING after
./work/haproxy-251a6b7/reg-tests/filters/s00000.vtc
** h1 15.2 Reset and free h1 haproxy 96689
** h1 15.2 Wait
** h1 15.2 Stop HAproxy pid=96689
**** h1 15.2 STDOUT poll 0x11
**** h1 15.2 Kill(2)=0: No error: 0
** h1 15.3 WAIT4 pid=96689 status=0x0002 (user 0.172273 sys 0.039152)
* top 15.3 TEST ./work/haproxy-251a6b7/reg-tests/filters/s00000.vtc FAILED
# top TEST ./work/haproxy-251a6b7/reg-tests/filters/s00000.vtc FAILED
(15.345) exit=2
root@freebsd11:/usr/ports/net/haproxy-devel # varnishtest -t 100 -Dno-htx="#"
./work/haproxy-251a6b7/reg-tests/filters/s00000.vtc
# top TEST ./work/haproxy-251a6b7/reg-tests/filters/s00000.vtc passed
(2.526)
