Re: [PATCH] BUILD: makefile: Update feature flags for FreeBSD

[Brad Smith]

ping.

On 9/15/2020 3:10 AM, Brad Smith wrote:

This updates the feature flags for FreeBSD.

FreeBSD 10 adds support for accept4().

Enable getaddrinfo().

 From the FreeBSD port / package.



diff --git a/Makefile b/Makefile
index 934ca1666..e69870595 100644
--- a/Makefile
+++ b/Makefile
@@ -363,11 +363,11 @@ ifeq ($(TARGET),solaris)
TARGET_LDFLAGS = -lnsl -lsocket
  endif
  
-# FreeBSD 5 and above

+# FreeBSD 10 and above
  ifeq ($(TARGET),freebsd)
set_target_defaults = $(call default_opts, \
  USE_POLL USE_TPROXY USE_LIBCRYPT USE_THREAD USE_CPU_AFFINITY USE_KQUEUE   
\
-USE_CLOSEFROM)
+USE_ACCEPT4 USE_CLOSEFROM USE_GETADDRINFO)
  endif
  
  # Mac OS/X

stable-bot: Bugfixes waiting for a release 2.2 (11), 2.1 (34), 2.0 (26), 1.8 (11)

[stable-bot]

Hi,

This is a friendly bot that watches fixes pending for the next haproxy-stable 
release!  One such e-mail is sent periodically once patches are waiting in the 
last maintenance branch, and an ideal release date is computed based on the 
severity of these fixes and their merge date.  Responses to this mail must be 
sent to the mailing list.


Last release 2.2.3 was issued on 2020-09-08.  There are currently 11 patches in 
the queue cut down this way:
- 3 MEDIUM, first one merged on 2020-09-11
- 8 MINOR, first one merged on 2020-09-22

Thus the computed ideal release date for 2.2.4 would be 2020-10-11, which is in 
three weeks or less.

Last release 2.1.8 was issued on 2020-07-31.  There are currently 34 patches in 
the queue cut down this way:
- 1 MAJOR, first one merged on 2020-09-07
- 11 MEDIUM, first one merged on 2020-08-05
- 22 MINOR, first one merged on 2020-08-11

Thus the computed ideal release date for 2.1.9 would be 2020-09-04, which was 
two weeks ago.

Last release 2.0.17 was issued on 2020-07-31.  There are currently 26 patches 
in the queue cut down this way:
- 1 MAJOR, first one merged on 2020-09-07
- 11 MEDIUM, first one merged on 2020-08-05
- 14 MINOR, first one merged on 2020-08-11

Thus the computed ideal release date for 2.0.18 would be 2020-10-04, which is 
in two weeks or less.

Last release 1.8.26 was issued on 2020-08-03.  There are currently 11 patches 
in the queue cut down this way:
- 4 MEDIUM, first one merged on 2020-08-05
- 7 MINOR, first one merged on 2020-08-03

Thus the computed ideal release date for 1.8.27 would be 2020-10-26, which is 
in five weeks or less.

The current list of patches in the queue is:
 - 2.0, 2.1  - MAJOR   : contrib/spoa-server: Fix unhandled 
python call leading to memory leak
 - 2.0, 2.1  - MEDIUM  : doc: Fix replace-path action 
description
 - 2.0, 2.1  - MEDIUM  : mux-h1: always apply the timeout on 
half-closed connections
 - 2.1   - MEDIUM  : ssl: memory leak of ocsp data at 
SSL_CTX_free()
 - 1.8, 2.0, 2.1, 2.2- MEDIUM  : pattern: Renew the pattern 
expression revision when it is pruned
 - 2.0, 2.1  - MEDIUM  : http-ana: Don't wait to send 1xx 
responses received from servers
 - 2.0, 2.1  - MEDIUM  : mux-h1: Refresh H1 connection timeout 
after a synchronous send
 - 2.0, 2.1  - MEDIUM  : htx: smp_prefetch_htx() must always 
validate the direction
 - 2.0, 2.1  - MEDIUM  : ssl: does not look for all SNIs before 
chosing a certificate
 - 1.8, 2.0  - MEDIUM  : mux-h2: Don't fail if nothing is 
parsed for a legacy chunk response
 - 2.2   - MEDIUM  : ssl: Don't call ssl_sock_io_cb() 
directly.
 - 1.8, 2.0, 2.1 - MEDIUM  : ssl: check OCSP calloc in 
ssl_sock_load_ocsp()
 - 2.0, 2.1  - MEDIUM  : contrib/spoa-server: Fix ipv4_address 
used instead of ipv6_address
 - 2.2   - MEDIUM  : h2: report frame bits only for handled 
types
 - 1.8, 2.0, 2.1 - MEDIUM  : map/lua: Return an error if a map is 
loaded during runtime
 - 2.1   - MINOR   : converters: Store the sink in an arg 
pointer for debug() converter
 - 2.0, 2.1  - MINOR   : contrib/spoa-server: Updating 
references to free in case of failure
 - 2.2   - MINOR   : ssl/crt-list: crt-list could end 
without a \n
 - 2.1   - MINOR   : lua: Duplicate map name to load it 
when a new Map object is created
 - 2.1   - MINOR   : arg: Fix leaks during arguments 
validation for fetches/converters
 - 2.0, 2.1, 2.2 - MINOR   : http-fetch: Don't set the sample type 
during the htx prefetch
 - 1.8, 2.0, 2.1 - MINOR   : lua: Check argument type to convert it 
to IPv4/IPv6 arg validation
 - 2.0, 2.1  - MINOR   : contrib/spoa-server: Ensure ip address 
references are freed
 - 1.8, 2.0, 2.1 - MINOR   : lua: Check argument type to convert it 
to IP mask in arg validation
 - 2.1   - MINOR   : http-rules: Replace path and 
query-string in "replace-path" action"
 - 2.0, 2.1  - MINOR   : auth: report valid crypto(3) support 
depending on build options
 - 2.2   - MINOR   : config: Fix memory leak on config 
parse listen
 - 1.8, 2.0, 2.1 - MINOR   : reload: do not fail when no socket is 
sent
 - 2.1, 2.2  - MINOR   : h2/trace: do not display "stream 
error" after a frame ACK
 - 2.1   - MINOR   : lua: Duplicate lua strings in sample 
fetches/converters arg array
 - 2.2   - MINOR   : Fix memory leaks cfg_parse_peers
 - 1.8, 2.0, 2.1 - MINOR   : startup: haproxy -s cause 100% cpu
 - 2.1   - MINOR   : http-rules: Replace path and 
query-string in 

Re: Is spoa thread safe?

[Christopher Faulet]

Le 22/09/2020 à 08:26, Reinhard Vicinus a écrit :

Reproduction is pretty simple:

use the example configuration from haproxy/contrib/spoa_server
start up a haproxy version with default nbthread value greater then 1 like this: 
"haproxy -f spoa-server.conf -d -Ws"

start the example spoa script: "./spoa -f ps_python.py -d"
run some curl requests: "curl localhost:10001/"
the variable sess.iprep.ip_score is only set on the first and n-th request



It is an issue with the spoa server. By default, 5 workers are started except in 
debug mode. In this case, there is only one worker. And I discovered that the 
spoa server has a design flaw. A worker is only able to handle one connection at 
a time. I don't know if this is expected or not. But it is a problem if you have 
more threads on HAProxy side than workers on spoa server. Because, in HAProxy, 
the spoe applets are sticky on threads. This means you will have at least one 
applet per thread, each one owning a connection to the spoa server. Thus, with 
only one worker, only one applet on one thread will be able to send messages to 
the spoa server. All others will be blocked on the connection establishment, 
leading to a timeout.


Worst, the spoa server does not support the pipelining or the async mode. So a 
worker is only able to process one message at a time, synchronously. On HAProxy 
side, this means an applet will be busy during the message processing. So new 
applets will be created to process other messages. And even in pipelining/async 
mode, more applets may be created to handle the load. Thus, it is really hard to 
use it on a production environment.


But to solve your issue, don't start the spoa server in debug mode and set the 
right number of worker (>= nbthread) with the -n argument. This should work with 
a very lightweight load.


--
Christopher Faulet

Re: Is spoa thread safe?

[Reinhard Vicinus]

Reproduction is pretty simple:

use the example configuration from haproxy/contrib/spoa_server
start up a haproxy version with default nbthread value greater then 1
like this: "haproxy -f spoa-server.conf -d -Ws"
start the example spoa script: "./spoa -f ps_python.py -d"
run some curl requests: "curl localhost:10001/"
the variable sess.iprep.ip_score is only set on the first and n-th request

On 9/22/20 8:10 AM, Илья Шипицин wrote:
> Can try to rebuild haproxy with thread sanitizer?
>
> Or describe repro steps
>
> On Mon, Sep 21, 2020, 9:49 PM Reinhard Vicinus  > wrote:
>
> Hi,
>
> if I try to connect to a spoa server with haproxy 2.2.3 and nbthread
> greater then 1, then if n is the nbthread value, only the first and
> thereafter n-th requests are answered correct and requests in between
> are not correctly forwarded to the spoa server. Is this a known
> limitation or is this a bug that will hopefully get fixed in the
> future?
> Or am I missing something?
>
> Kind regards
> Reinhard Vicinus
>


-- 
Reinhard Vicinus
Metaways Infosystems GmbH
Pickhuben 2, D-20457 Hamburg

E-Mail: r.vici...@metaways.de
Web:http://www.metaways.de
Tel:+49 (0)40 317031-524
Fax:+49 (0)40 317031-10

Metaways Infosystems GmbH - Sitz: D-22967 Tremsbüttel
Handelsregister: Amtsgericht Lübeck HRB 4508 AH
Geschäftsführung: Hermann Thaele, Lüder-H. Thaele

Re: Is spoa thread safe?

[Илья Шипицин]

Can try to rebuild haproxy with thread sanitizer?

Or describe repro steps

On Mon, Sep 21, 2020, 9:49 PM Reinhard Vicinus  wrote:

> Hi,
>
> if I try to connect to a spoa server with haproxy 2.2.3 and nbthread
> greater then 1, then if n is the nbthread value, only the first and
> thereafter n-th requests are answered correct and requests in between
> are not correctly forwarded to the spoa server. Is this a known
> limitation or is this a bug that will hopefully get fixed in the future?
> Or am I missing something?
>
> Kind regards
> Reinhard Vicinus
>
>