Re: commit 493d9dc makes a SVN-checkout stall..
Hi Olivier, Willy, Just to confirm, as expected it (c3500c3) indeed works for me :). Thanks for the quick fix! Regards, PiBa-NL (Pieter) Op 25-3-2020 om 17:16 schreef Willy Tarreau: On Wed, Mar 25, 2020 at 05:08:03PM +0100, Olivier Houchard wrote: That is... interesting, not sure I reached such an outstanding result. Oh I stopped trying to guess long ago :-) This is now fixed, sorry about that ! Confirmed, much better now, thanks! Willy
Re: commit 493d9dc makes a SVN-checkout stall..
On Wed, Mar 25, 2020 at 05:08:03PM +0100, Olivier Houchard wrote: > That is... interesting, not sure I reached such an outstanding result. Oh I stopped trying to guess long ago :-) > This is now fixed, sorry about that ! Confirmed, much better now, thanks! Willy
Re: commit 493d9dc makes a SVN-checkout stall..
Hi Willy, On Wed, Mar 25, 2020 at 02:03:56PM +0100, Willy Tarreau wrote: > On Wed, Mar 25, 2020 at 12:40:33PM +0100, Olivier Houchard wrote: > > I think I figured it out, and commit > > 69664419d209d2f64dbcd90f991e7ec2efff2ee2 > > should fix it, at least now I can do a full svn checkout. > > > > Can you confirm it does the trick for you ? > > Olivier, I think you made a last minute change after your fix because > it broke the build: > > https://travis-ci.com/github/haproxy/haproxy/jobs/302517378 > > src/mux_h1.c:2495:23: error: incompatible pointer types passing 'struct > connection **' to parameter of type 'const struct buffer *' > [-Werror,-Wincompatible-pointer-types] > if (unlikely(b_data(>conn))) > ^~ > include/common/compiler.h:98:40: note: expanded from macro 'unlikely' > #define unlikely(x) (__builtin_expect((x) != 0, 0)) >^ > include/common/buf.h:100:50: note: passing argument to parameter 'b' here > static inline size_t b_data(const struct buffer *b) > ^ > I think it was h1c->dbuf not conn. > > Willy That is... interesting, not sure I reached such an outstanding result. This is now fixed, sorry about that ! Olivier
Re: commit 493d9dc makes a SVN-checkout stall..
On Wed, Mar 25, 2020 at 12:40:33PM +0100, Olivier Houchard wrote: > I think I figured it out, and commit 69664419d209d2f64dbcd90f991e7ec2efff2ee2 > should fix it, at least now I can do a full svn checkout. > > Can you confirm it does the trick for you ? Olivier, I think you made a last minute change after your fix because it broke the build: https://travis-ci.com/github/haproxy/haproxy/jobs/302517378 src/mux_h1.c:2495:23: error: incompatible pointer types passing 'struct connection **' to parameter of type 'const struct buffer *' [-Werror,-Wincompatible-pointer-types] if (unlikely(b_data(>conn))) ^~ include/common/compiler.h:98:40: note: expanded from macro 'unlikely' #define unlikely(x) (__builtin_expect((x) != 0, 0)) ^ include/common/buf.h:100:50: note: passing argument to parameter 'b' here static inline size_t b_data(const struct buffer *b) ^ I think it was h1c->dbuf not conn. Willy
Re: commit 493d9dc makes a SVN-checkout stall..
On Wed, Mar 25, 2020 at 04:06:27PM +0500, ??? wrote: > btw, is there some tool to test at least RFC 7230 compliance ? Not that I'm aware of, especially given the complexity of the HTTP/1 specs which try to cover all known use cases and implementations. > also, there are various specific implementation like MAPI / MS RPC which > might require some testing tool as well > > inspired by h2spec actually. can/should we add some HTTP/1.1 tests as well ? I don't think so. H2 is extremely simple and strict in its definition, this is not the case for H1 and complying is more a matter of how strict you are with the SHOULD or MAY than anything else. I'd say that in practice most of the VTC files already rely on some basic compliance and are already OK for this. We can always improve, of course, but this shouldn't deserve unreasonable efforts. Willy
Re: commit 493d9dc makes a SVN-checkout stall..
Hi Pieter, On Wed, Mar 25, 2020 at 01:42:14AM +0100, Olivier Houchard wrote: > Hi Pieter, > > On Wed, Mar 25, 2020 at 01:14:46AM +0100, PiBa-NL wrote: > > Hi List, Willy, > > > > Today i thought lets give v2.2-dev5 a try for my production environment ;). > > Soon it turned out to cause SVN-Checkout to stall/disconnect for a > > repository we run locally in a Collab-SVN server. > > > > I tracked it down to this commit: 493d9dc (MEDIUM: mux-h1: do not blindly > > wake up the tasklet at end of request anymore) causing the problem for the > > first time. Is there something tricky there that can be suspected to cause > > the issue.? Perhaps a patch i can try? > > > > While 'dissecting' the issue i deleted the whole directory each time and > > performed a new svn-checkout several times. It doesn't always stall at the > > exact same point but usually after checking out around +- 20 files something > > between 0.5 and 2 MB. , the commit before that one allows me to checkout > > 500+MB through haproxy without issue.. A wireshark seems to show that > > haproxy is sending several of RST,ACK packets for a 4 different connections > > to the svn-server at the same milisecond after it was quiet for 2 seconds.. > > The whole issue happens in a timeframe of start of checkout till when it > > stalls within 15 seconds. > > > > The 'nokqueue' i usually try on my FreeBSD machine doesn't change anything. > > > > Hope you have an idea where to look. Providing captures/logs is a bit > > difficult without some careful scrubbing.. > > > > Regards, > > PiBa-NL (Pieter) > > > > No answer yet, but I just tried to do a checkout of the FreeBSD svn tree > via haproxy, and it indeed doesn't work, it's a bit late right now, but > I'll have a look tomorrow :) > I think I figured it out, and commit 69664419d209d2f64dbcd90f991e7ec2efff2ee2 should fix it, at least now I can do a full svn checkout. Can you confirm it does the trick for you ? Thanks ! Olivier
Re: commit 493d9dc makes a SVN-checkout stall..
btw, is there some tool to test at least RFC 7230 compliance ? also, there are various specific implementation like MAPI / MS RPC which might require some testing tool as well inspired by h2spec actually. can/should we add some HTTP/1.1 tests as well ? ср, 25 мар. 2020 г. в 11:02, Willy Tarreau : > Hi guys, > > On Wed, Mar 25, 2020 at 01:42:14AM +0100, Olivier Houchard wrote: > (...) > > > Hope you have an idea where to look. Providing captures/logs is a bit > > > difficult without some careful scrubbing.. > > I think a good test would be to just revert that patch, or selectively > each one of the 3 lines it contains. > > > No answer yet, but I just tried to do a checkout of the FreeBSD svn tree > > via haproxy, and it indeed doesn't work, it's a bit late right now, but > > I'll have a look tomorrow :) > > OK thanks. I remember that you had some opinions on some of them at some > point. However I'm tempted to think that if any of them needs to be > changed, > we're probably missing something somewhere which could indicate that other > subscribes might randomly fail, because each time it's in h1_detach() and > I don't see why we should always wake up if already subscribed. > > Thanks, > Willy > >
Re: commit 493d9dc makes a SVN-checkout stall..
Hi guys, On Wed, Mar 25, 2020 at 01:42:14AM +0100, Olivier Houchard wrote: (...) > > Hope you have an idea where to look. Providing captures/logs is a bit > > difficult without some careful scrubbing.. I think a good test would be to just revert that patch, or selectively each one of the 3 lines it contains. > No answer yet, but I just tried to do a checkout of the FreeBSD svn tree > via haproxy, and it indeed doesn't work, it's a bit late right now, but > I'll have a look tomorrow :) OK thanks. I remember that you had some opinions on some of them at some point. However I'm tempted to think that if any of them needs to be changed, we're probably missing something somewhere which could indicate that other subscribes might randomly fail, because each time it's in h1_detach() and I don't see why we should always wake up if already subscribed. Thanks, Willy
Re: commit 493d9dc makes a SVN-checkout stall..
Hi Pieter, On Wed, Mar 25, 2020 at 01:14:46AM +0100, PiBa-NL wrote: > Hi List, Willy, > > Today i thought lets give v2.2-dev5 a try for my production environment ;). > Soon it turned out to cause SVN-Checkout to stall/disconnect for a > repository we run locally in a Collab-SVN server. > > I tracked it down to this commit: 493d9dc (MEDIUM: mux-h1: do not blindly > wake up the tasklet at end of request anymore) causing the problem for the > first time. Is there something tricky there that can be suspected to cause > the issue.? Perhaps a patch i can try? > > While 'dissecting' the issue i deleted the whole directory each time and > performed a new svn-checkout several times. It doesn't always stall at the > exact same point but usually after checking out around +- 20 files something > between 0.5 and 2 MB. , the commit before that one allows me to checkout > 500+MB through haproxy without issue.. A wireshark seems to show that > haproxy is sending several of RST,ACK packets for a 4 different connections > to the svn-server at the same milisecond after it was quiet for 2 seconds.. > The whole issue happens in a timeframe of start of checkout till when it > stalls within 15 seconds. > > The 'nokqueue' i usually try on my FreeBSD machine doesn't change anything. > > Hope you have an idea where to look. Providing captures/logs is a bit > difficult without some careful scrubbing.. > > Regards, > PiBa-NL (Pieter) > No answer yet, but I just tried to do a checkout of the FreeBSD svn tree via haproxy, and it indeed doesn't work, it's a bit late right now, but I'll have a look tomorrow :) Regards, Olivier
commit 493d9dc makes a SVN-checkout stall..
Hi List, Willy, Today i thought lets give v2.2-dev5 a try for my production environment ;). Soon it turned out to cause SVN-Checkout to stall/disconnect for a repository we run locally in a Collab-SVN server. I tracked it down to this commit: 493d9dc (MEDIUM: mux-h1: do not blindly wake up the tasklet at end of request anymore) causing the problem for the first time. Is there something tricky there that can be suspected to cause the issue.? Perhaps a patch i can try? While 'dissecting' the issue i deleted the whole directory each time and performed a new svn-checkout several times. It doesn't always stall at the exact same point but usually after checking out around +- 20 files something between 0.5 and 2 MB. , the commit before that one allows me to checkout 500+MB through haproxy without issue.. A wireshark seems to show that haproxy is sending several of RST,ACK packets for a 4 different connections to the svn-server at the same milisecond after it was quiet for 2 seconds.. The whole issue happens in a timeframe of start of checkout till when it stalls within 15 seconds. The 'nokqueue' i usually try on my FreeBSD machine doesn't change anything. Hope you have an idea where to look. Providing captures/logs is a bit difficult without some careful scrubbing.. Regards, PiBa-NL (Pieter) ### Complete config (that still reproduces the issue.. things cant get much simpler than this..): frontend InternalSites.8.6-merged bind 192.168.8.67:80 mode http use_backend APP01-JIRA-SVN_ipvANY backend APP01-JIRA-SVN_ipvANY mode http server svn 192.168.104.20:8080 ### uname -a FreeBSD freebsd11 11.1-RELEASE FreeBSD 11.1-RELEASE #0 r321309: Fri Jul 21 02:08:28 UTC 2017 r...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 ### haproxy -vv HA-Proxy version 2.2-dev5-3e128fe 2020/03/24 - https://haproxy.org/ Status: development branch - not safe for use in production. Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open Build options : TARGET = freebsd CPU = generic CC = cc CFLAGS = -pipe -g -fstack-protector -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-null-dereference -Wno-unused-label -Wno-unused-parameter -Wno-sign-compare -Wno-ignored-qualifiers -Wno-unused-command-line-argument -Wno-missing-field-initializers -Wno-address-of-packed-member -DFREEBSD_PORTS -DFREEBSD_PORTS OPTIONS = USE_PCRE=1 USE_PCRE_JIT=1 USE_STATIC_PCRE=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_ACCEPT4=1 USE_ZLIB=1 Feature list : -EPOLL +KQUEUE -NETFILTER +PCRE +PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -BACKTRACE +STATIC_PCRE -STATIC_PCRE2 +TPROXY -LINUX_TPROXY -LINUX_SPLICE +LIBCRYPT -CRYPT_H +GETADDRINFO +OPENSSL +LUA -FUTEX +ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY -TFO -NS -DL -RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER -PRCTL -THREAD_DUMP -EVPORTS Default settings : bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with multi-threading support (MAX_THREADS=64, default=16). Built with OpenSSL version : OpenSSL 1.0.2k-freebsd 26 Jan 2017 Running on OpenSSL version : OpenSSL 1.0.2k-freebsd 26 Jan 2017 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2 Built with Lua version : Lua 5.3.4 Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY Built with PCRE version : 8.40 2017-01-11 Running on PCRE version : 8.40 2017-01-11 PCRE library supports JIT : yes Encrypted password support via crypt(3): yes Built with zlib version : 1.2.11 Running on zlib version : 1.2.11 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Available polling systems : kqueue : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use kqueue. Available multiplexer protocols : (protocols marked as cannot be specified using 'proto' keyword) h2 : mode=HTTP side=FE|BE mux=H2 fcgi : mode=HTTP side=BE mux=FCGI : mode=HTTP side=FE|BE mux=H1 : mode=TCP side=FE|BE mux=PASS Available services : none Available filters : [SPOE] spoe [CACHE] cache [FCGI] fcgi-app [TRACE] trace [COMP] compression