Re: freebsd ci is broken - commit 08fa16e - curl download stalls in reg-tests/compression/lua_validation.vtc

[Willy Tarreau]

On Wed, Jan 15, 2020 at 08:38:49PM +0100, PiBa-NL wrote:
> Hi Olivier, Willy, Ilya,
> 
> Thanks! I confirm 2.2-dev0-ac81474 fixes this issue for me. And cirrus-ci
> also shows 'all green' again :).
> 
> Running the same test with 16vCPU and kqueue enabled its 'all okay': 0 tests
> failed, 0 tests skipped, 200 tests passed.

Excellent, thanks for the quick feedback! At least this showed up
another area that definitely requires a final lifting for 2.2!

Willy

Re: freebsd ci is broken - commit 08fa16e - curl download stalls in reg-tests/compression/lua_validation.vtc

[PiBa-NL]

Hi Olivier, Willy, Ilya,

Thanks! I confirm 2.2-dev0-ac81474 fixes this issue for me. And 
cirrus-ci also shows 'all green' again :).


Running the same test with 16vCPU and kqueue enabled its 'all okay': 0 
tests failed, 0 tests skipped, 200 tests passed.


Op 15-1-2020 om 19:20 schreef Olivier Houchard:

Hi guys,

On Tue, Jan 14, 2020 at 09:45:34PM +0100, Willy Tarreau wrote:

Hi guys,

On Tue, Jan 14, 2020 at 08:02:51PM +0100, PiBa-NL wrote:

Below a part of the output that the test generates for me. The first curl
request seems to succeed, but the second one runs into a timeout..
When compiled with the commit before 08fa16e 


Ah, and unsurprizingly I'm the author :-/

I'm wondering why it only affects FreeBSD (very likely kqueue in fact, I
suppose it works if you start with -dk). Maybe something subtle escaped
me in the poller after the previous changes.


Should i update to a newer FreeBSD version, or is it likely unrelated, and
in need of some developer attention.. Do you (Willy or anyone), need more
information from my side? Or is there a patch i can try to validate?

I don't think I need more info for now and your version has nothing to do
with this (until proven otherwise). I apparently really broke something
there. I think I have a FreeBSD VM somewhere, in the worst case I'll ask
Olivier for some help :-)


To give you a quick update, we investigating that, and I'm still not really
sure why it only affects FreeBSD, but we fully understood the problem, and
it should be fixed by now.

Regards,

Olivier


Regards,
PiBa-NL (Pieter)

Re: freebsd ci is broken - commit 08fa16e - curl download stalls in reg-tests/compression/lua_validation.vtc

[Olivier Houchard]

Hi guys,

On Tue, Jan 14, 2020 at 09:45:34PM +0100, Willy Tarreau wrote:
> Hi guys,
> 
> On Tue, Jan 14, 2020 at 08:02:51PM +0100, PiBa-NL wrote:
> > Below a part of the output that the test generates for me. The first curl
> > request seems to succeed, but the second one runs into a timeout..
> > When compiled with the commit before 08fa16e 
> > 
> 
> Ah, and unsurprizingly I'm the author :-/
> 
> I'm wondering why it only affects FreeBSD (very likely kqueue in fact, I
> suppose it works if you start with -dk). Maybe something subtle escaped
> me in the poller after the previous changes.
> 
> > Should i update to a newer FreeBSD version, or is it likely unrelated, and
> > in need of some developer attention.. Do you (Willy or anyone), need more
> > information from my side? Or is there a patch i can try to validate?
> 
> I don't think I need more info for now and your version has nothing to do
> with this (until proven otherwise). I apparently really broke something
> there. I think I have a FreeBSD VM somewhere, in the worst case I'll ask
> Olivier for some help :-)
> 

To give you a quick update, we investigating that, and I'm still not really
sure why it only affects FreeBSD, but we fully understood the problem, and
it should be fixed by now.

Regards,

Olivier

Re: freebsd ci is broken - commit 08fa16e - curl download stalls in reg-tests/compression/lua_validation.vtc

[Илья Шипицин]

ср, 15 янв. 2020 г. в 04:18, PiBa-NL :

> Hi Ilya, Willy,
>
> Op 14-1-2020 om 21:40 schreef Илья Шипицин:
> > PiBa, how many CPU cores are you running ?
> >
> > it turned out that I run tests on very low vm, which only has 1 core.
> > and tests pass.
> > cirrus-ci as far as I remember do have many cores.
> I was running with 16 cores..
> >
> > can you find single core vm ?
>
> Well, i reconfigured the VM to have 1 core, but same issue seems to show
> up, though not on every time the test is run, and actually a bit less
> often..
> Below some additional testresults with different kqueue / vCPU settings..
>

I run tests on MS Azure B1s vm, it is cheapest size. Single "shared" core.
Tests always pass.
looks like we've found race condition :)


>
>
> *VM with 1 vCPU*
>
> Running: ./vtest/VTest-master/vtest -Dno-htx=no -l -k -b 50M -t 5 -n 20
> ./work/haproxy-08fa16e/reg-tests/compression/lua_validation.vtc
>Results in: 4 tests failed, 0 tests skipped, 16 tests passed
>
> Adding "nokqueue" in the vtc file i get:
>8 tests failed, 0 tests skipped, 12 tests passed
>4 tests failed, 0 tests skipped, 16 tests passed
>
> So its a bit random, but the 'nokqueue' directive does not seem to
> affect results much..
>
>
> *With 16 vCPU*
> Without nokqueue: 16 tests failed, 0 tests skipped, 4 tests passed
> With nokqueue (using poll): 17 tests failed, 0 tests skipped, 3 tests
> passed
>
> The failure rate seems certainly higher with many cores..
>
>
> * Using commit 0eae632 it works OK*
> Just to be sure i re-tested on 16 cores with 2.2-dev0-0eae632 but that
> does nicely pass: 0 tests failed, 0 tests skipped, 20 tests passed
>
> Regards,
> PiBa-NL (Pieter)
>
>

Re: freebsd ci is broken - commit 08fa16e - curl download stalls in reg-tests/compression/lua_validation.vtc

[PiBa-NL]

Hi Ilya, Willy,

Op 14-1-2020 om 21:40 schreef Илья Шипицин:

PiBa, how many CPU cores are you running ?

it turned out that I run tests on very low vm, which only has 1 core. 
and tests pass.

cirrus-ci as far as I remember do have many cores.

I was running with 16 cores..


can you find single core vm ?


Well, i reconfigured the VM to have 1 core, but same issue seems to show 
up, though not on every time the test is run, and actually a bit less 
often..

Below some additional testresults with different kqueue / vCPU settings..


*VM with 1 vCPU*

Running: ./vtest/VTest-master/vtest -Dno-htx=no -l -k -b 50M -t 5 -n 20 
./work/haproxy-08fa16e/reg-tests/compression/lua_validation.vtc

  Results in: 4 tests failed, 0 tests skipped, 16 tests passed

Adding "nokqueue" in the vtc file i get:
  8 tests failed, 0 tests skipped, 12 tests passed
  4 tests failed, 0 tests skipped, 16 tests passed

So its a bit random, but the 'nokqueue' directive does not seem to 
affect results much..



*With 16 vCPU*
Without nokqueue: 16 tests failed, 0 tests skipped, 4 tests passed
With nokqueue (using poll): 17 tests failed, 0 tests skipped, 3 tests passed

The failure rate seems certainly higher with many cores..


* Using commit 0eae632 it works OK*
Just to be sure i re-tested on 16 cores with 2.2-dev0-0eae632 but that 
does nicely pass: 0 tests failed, 0 tests skipped, 20 tests passed


Regards,
PiBa-NL (Pieter)

Re: freebsd ci is broken - commit 08fa16e - curl download stalls in reg-tests/compression/lua_validation.vtc

[Willy Tarreau]

Hi guys,

On Tue, Jan 14, 2020 at 08:02:51PM +0100, PiBa-NL wrote:
> Below a part of the output that the test generates for me. The first curl
> request seems to succeed, but the second one runs into a timeout..
> When compiled with the commit before 08fa16e 
> 

Ah, and unsurprizingly I'm the author :-/

I'm wondering why it only affects FreeBSD (very likely kqueue in fact, I
suppose it works if you start with -dk). Maybe something subtle escaped
me in the poller after the previous changes.

> Should i update to a newer FreeBSD version, or is it likely unrelated, and
> in need of some developer attention.. Do you (Willy or anyone), need more
> information from my side? Or is there a patch i can try to validate?

I don't think I need more info for now and your version has nothing to do
with this (until proven otherwise). I apparently really broke something
there. I think I have a FreeBSD VM somewhere, in the worst case I'll ask
Olivier for some help :-)

Thanks for the report!
Willy

Re: freebsd ci is broken - commit 08fa16e - curl download stalls in reg-tests/compression/lua_validation.vtc

[Илья Шипицин]

PiBa, how many CPU cores are you running ?

it turned out that I run tests on very low vm, which only has 1 core. and
tests pass.
cirrus-ci as far as I remember do have many cores.

can you find single core vm ?

ср, 15 янв. 2020 г. в 00:02, PiBa-NL :

> Hi Ilya,
>
> Thanks!
>
> Op 14-1-2020 om 07:48 schreef Илья Шипицин:
>
> Hello,
>
> since
>
> https://github.com/haproxy/haproxy/commit/08fa16e397ffb1c6511b98ade2a3bfff9435e521
>
> freebsd CI is red: https://cirrus-ci.com/task/5960933184897024
>
> I'd say "it is something with CI itself",  when I run the same tests
> locally on freebsd, it is green.
>
> Sadly i do get the same problem on my test server (version info below its
> version 11.1 is a bit outdated, but hasn't failed my before...).
>
>
> PiBa ?
>
>
> thanks,
> Ilya Shipitcin
>
> Below a part of the output that the test generates for me. The first curl
> request seems to succeed, but the second one runs into a timeout..
> When compiled with the commit before 08fa16e
> 
> it does not show that behaviour.. Current latest(24c928c) commit is still
> affected..
>
>  top  shell_out|  % Total% Received % Xferd  Average Speed
> TimeTime Time  Current
>  top  shell_out| Dload  Upload
> Total   SpentLeft  Speed
>  top  shell_out|\r  0 00 00 0  0  0
> --:--:-- --:--:-- --:--:-- 0\r100  418k0  418k0 0
> 1908k  0 --:--:-- --:--:-- --:--:-- 1908k
>  top  shell_out|  % Total% Received % Xferd  Average Speed
> TimeTime Time  Current
>  top  shell_out| Dload  Upload
> Total   SpentLeft  Speed
>  top  shell_out|\r  0 00 00 0  0  0
> --:--:-- --:--:-- --:--:-- 0\r100  141k0  141k0 0
> 284k  0 --:--:-- --:--:-- --:--:--  284k\r100  343k0  343k0
> 0   156k  0 --:--:--  0:00:02 --:--:--  156k\r100  343k0  343k
> 0 0   105k  0 --:--:--  0:00:03 --:--:--  105k\r100  343k0
> 343k0 0  81274  0 --:--:--  0:00:04 --:--:-- 81274\r100
> 343k0  343k0 0  65228  0 --:--:--  0:00:05 --:--:--
> 65240\r100  343k0  343k0 0  54481  0 --:--:--  0:00:06
> --:--:-- 34743\r100  343k0  343k0 0  46768  0 --:--:--
> 0:00:07 --:--:-- 0\r100  343k0  343k0 0  40968  0
> --:--:--  0:00:08 --:--:-- 0\r100  343k0  343k0 0
> 36452  0 --:--:--  0:00:09 --:--:-- 0\r100  343k0  343k
> 0 0  32830  0 --:--:--  0:00:10 --:--:-- 0\r100  343k0
> 343k0 0  29865  0 --:--:--  0:00:11 --:--:-- 0\r100
> 343k0  343k0 0  27395  0 --:--:--  0:00:12 --:--:--
> 0\r100  343k0  343k0 0  25297  0 --:--:--  0:00:13
> --:--:-- 0\r100  343k0  343k0 0  23500  0 --:--:--
> 0:00:14 --:--:-- 0\r100  343k0  343k0 0  23431  0
> --:--:--  0:00:15 --:--:-- 0
>  top  shell_out|curl: (28) Operation timed out after 15002
> milliseconds with 351514 bytes received
>  top  shell_out|Expecting checksum 4d9c62aa5370b8d5f84f17ec2e78f483
>  top  shell_out|Received checksum: da2d120aedfd693eeba9cf1e578897a8
>  top  shell_status = 0x0001
>  top  shell_exit not as expected: got 0x0001 wanted 0x
> *top  RESETTING after
> ./work/haproxy-08fa16e/reg-tests/compression/lua_validation.vtc
>
> Should i update to a newer FreeBSD version, or is it likely unrelated, and
> in need of some developer attention.. Do you (Willy or anyone), need more
> information from my side? Or is there a patch i can try to validate?
>
> Regards,
> PiBa-NL (Pieter)
>
>
> Yes im running a somewhat outdated OS here:
>   FreeBSD freebsd11 11.1-RELEASE FreeBSD 11.1-RELEASE #0 r321309: Fri Jul
> 21 02:08:28 UTC 2017
> r...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
>
> Version used:
>   haproxy -vv
> HA-Proxy version 2.2-dev0-08fa16e 2020/01/08 - https://haproxy.org/
> Status: development branch - not safe for use in production.
> Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
> Build options :
>   TARGET  = freebsd
>   CPU = generic
>   CC  = cc
>   CFLAGS  = -pipe -g -fstack-protector -fno-strict-aliasing
> -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv
> -fno-strict-overflow -Wno-null-dereference -Wno-unused-label
> -Wno-unused-parameter -Wno-sign-compare -Wno-ignored-qualifiers
> -Wno-unused-command-line-argument -Wno-missing-field-initializers
> -Wno-address-of-packed-member -DFREEBSD_PORTS -DFREEBSD_PORTS
>   OPTIONS = USE_PCRE=1 USE_PCRE_JIT=1 USE_REGPARM=1 USE_STATIC_PCRE=1
> USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_ACCEPT4=1 USE_ZLIB=1
>
> Feature list : -EPOLL +KQUEUE -MY_EPOLL -MY_SPLICE -NETFILTER +PCRE
> +PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED
> 

Re: freebsd ci is broken - commit 08fa16e - curl download stalls in reg-tests/compression/lua_validation.vtc

[PiBa-NL]

Hi Ilya,

Thanks!

Op 14-1-2020 om 07:48 schreef Илья Шипицин:

Hello,

since
https://github.com/haproxy/haproxy/commit/08fa16e397ffb1c6511b98ade2a3bfff9435e521

freebsd CI is red: https://cirrus-ci.com/task/5960933184897024

I'd say "it is something with CI itself",  when I run the same tests 
locally on freebsd, it is green.
Sadly i do get the same problem on my test server (version info below 
its version 11.1 is a bit outdated, but hasn't failed my before...).


PiBa ?


thanks,
Ilya Shipitcin


Below a part of the output that the test generates for me. The first 
curl request seems to succeed, but the second one runs into a timeout..
When compiled with the commit before 08fa16e 
 
it does not show that behaviour.. Current latest(24c928c) commit is 
still affected..


 top  shell_out|  % Total    % Received % Xferd  Average Speed   
Time    Time Time  Current
 top  shell_out| Dload Upload   
Total   Spent    Left  Speed
 top  shell_out|\r  0 0    0 0    0 0  0 0 --:--:-- 
--:--:-- --:--:-- 0\r100  418k    0  418k    0 0  1908k  0 
--:--:-- --:--:-- --:--:-- 1908k
 top  shell_out|  % Total    % Received % Xferd  Average Speed   
Time    Time Time  Current
 top  shell_out| Dload Upload   
Total   Spent    Left  Speed
 top  shell_out|\r  0 0    0 0    0 0  0 0 --:--:-- 
--:--:-- --:--:-- 0\r100  141k    0  141k    0 0   284k  0 
--:--:-- --:--:-- --:--:--  284k\r100  343k    0 343k    0 0   
156k  0 --:--:--  0:00:02 --:--:-- 156k\r100  343k    0  343k    
0 0   105k  0 --:--:-- 0:00:03 --:--:--  105k\r100  343k    0  
343k    0 0 81274  0 --:--:--  0:00:04 --:--:-- 81274\r100  
343k    0 343k    0 0  65228  0 --:--:--  0:00:05 --:--:-- 
65240\r100  343k    0  343k    0 0  54481  0 --:--:-- 0:00:06 
--:--:-- 34743\r100  343k    0  343k    0 0 46768  0 --:--:--  
0:00:07 --:--:-- 0\r100  343k    0 343k    0 0  40968  0 
--:--:--  0:00:08 --:--:-- 0\r100  343k    0  343k    0 0  
36452  0 --:--:--  0:00:09 --:--:-- 0\r100  343k    0  343k    
0 0  32830  0 --:--:--  0:00:10 --:--:-- 0\r100  343k    0  
343k    0 0 29865  0 --:--:--  0:00:11 --:--:-- 0\r100  
343k    0 343k    0 0  27395  0 --:--:--  0:00:12 --:--:-- 
0\r100  343k    0  343k    0 0  25297  0 --:--:--  0:00:13 
--:--:-- 0\r100  343k    0  343k    0 0  23500  0 --:--:--  
0:00:14 --:--:-- 0\r100  343k    0  343k    0 0 23431  0 
--:--:--  0:00:15 --:--:-- 0
 top  shell_out|curl: (28) Operation timed out after 15002 
milliseconds with 351514 bytes received

 top  shell_out|Expecting checksum 4d9c62aa5370b8d5f84f17ec2e78f483
 top  shell_out|Received checksum: da2d120aedfd693eeba9cf1e578897a8
 top  shell_status = 0x0001
 top  shell_exit not as expected: got 0x0001 wanted 0x
*    top  RESETTING after 
./work/haproxy-08fa16e/reg-tests/compression/lua_validation.vtc


Should i update to a newer FreeBSD version, or is it likely unrelated, 
and in need of some developer attention.. Do you (Willy or anyone), need 
more information from my side? Or is there a patch i can try to validate?


Regards,
PiBa-NL (Pieter)


Yes im running a somewhat outdated OS here:
  FreeBSD freebsd11 11.1-RELEASE FreeBSD 11.1-RELEASE #0 r321309: Fri 
Jul 21 02:08:28 UTC 2017 
r...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64


Version used:
  haproxy -vv
HA-Proxy version 2.2-dev0-08fa16e 2020/01/08 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Build options :
  TARGET  = freebsd
  CPU = generic
  CC  = cc
  CFLAGS  = -pipe -g -fstack-protector -fno-strict-aliasing 
-fno-strict-aliasing -Wdeclaration-after-statement -fwrapv 
-fno-strict-overflow -Wno-null-dereference -Wno-unused-label 
-Wno-unused-parameter -Wno-sign-compare -Wno-ignored-qualifiers 
-Wno-unused-command-line-argument -Wno-missing-field-initializers 
-Wno-address-of-packed-member -DFREEBSD_PORTS -DFREEBSD_PORTS
  OPTIONS = USE_PCRE=1 USE_PCRE_JIT=1 USE_REGPARM=1 USE_STATIC_PCRE=1 
USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_ACCEPT4=1 USE_ZLIB=1


Feature list : -EPOLL +KQUEUE -MY_EPOLL -MY_SPLICE -NETFILTER +PCRE 
+PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD 
-PTHREAD_PSHARED +REGPARM +STATIC_PCRE -STATIC_PCRE2 +TPROXY 
-LINUX_TPROXY -LINUX_SPLICE +LIBCRYPT -CRYPT_H -VSYSCALL +GETADDRINFO 
+OPENSSL +LUA -FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY -TFO 
-NS -DL -RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER 
-PRCTL -THREAD_DUMP -EVPORTS


Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support