Re: 1.9.6: SIGFPE in fwrr_update_position

[Willy Tarreau]

Hi Maksim,

On Tue, Apr 16, 2019 at 08:15:42AM +0300, ?? ? wrote:
> Hi Willy!
> 
> Actually I don't think this is a CPU fault. The reason is that I have same
> cores with non-zero dividers on 4 more hardware servers with different CPU
> models.

OK that's very useful info, thank you.

> So I agree upon another thread activity. The unique thing about
> these servers - all of them use haproxy-agent to set up weights of their
> backends. Other instances with no haproxy-agent in their configs don't
> produce cores.

Great, this will definitely help me validate my hypothesis. I'm not sure
the fix will be easy but I'm back to this.

Thanks!
Willy

Re: 1.9.6: SIGFPE in fwrr_update_position

[Максим Куприянов]

Hi Willy!

Actually I don't think this is a CPU fault. The reason is that I have same
cores with non-zero dividers on 4 more hardware servers with different CPU
models. So I agree upon another thread activity. The unique thing about
these servers – all of them use haproxy-agent to set up weights of their
backends. Other instances with no haproxy-agent in their configs don't
produce cores.

пн, 15 апр. 2019 г. в 23:48, Willy Tarreau :

> Hi Maksim,
>
> On Thu, Apr 11, 2019 at 02:03:43PM +0200, Willy Tarreau wrote:
> > I tried to follow all paths that lead to a zero cur_eweight that I could
> > find and none of them leave the server in the tree. Then I tried to find
> > all cases where this entry is updated or used and all are under the
> server
> > lock, meaning that I don't see how another thread could have changed the
> > value between the check and the use. I must obviously be wrong on at
> least
> > one of them but I really can't figure which one.
>
> Actually I think I found one way to get there with a lock missing. The
> impossible case in your trace made me think that since it's very unlikely
> that the CPU is faulty (never impossible but extremely rare), another
> thread was possibly still doing something in our back before the crash
> happened, and fixed the value again before the dump was done. These are
> thus two very quick changes. I don't see what sequence of actions can do
> this but I think I want to study one code path that looks suspicious to
> me. I need to double-check this tomorrow after some sleep, I'll keep you
> informed.
>
> Cheers,
> Willy
>

Re: Infinite loop after 39cc020af BUG/MEDIUM: streams: Don't remove the SI_FL_ERR flag in si_update_both()

[Willy Tarreau]

On Mon, Apr 15, 2019 at 12:46:05PM -0400, Richard Russo wrote:
> After the weekend, the test machine looks fine. Thanks!

Thank you for this positive feedback, Richard, much appreciated!

Willy

Re: 1.9.6: SIGFPE in fwrr_update_position

[Willy Tarreau]

Hi Maksim,

On Thu, Apr 11, 2019 at 02:03:43PM +0200, Willy Tarreau wrote:
> I tried to follow all paths that lead to a zero cur_eweight that I could
> find and none of them leave the server in the tree. Then I tried to find
> all cases where this entry is updated or used and all are under the server
> lock, meaning that I don't see how another thread could have changed the
> value between the check and the use. I must obviously be wrong on at least
> one of them but I really can't figure which one.

Actually I think I found one way to get there with a lock missing. The
impossible case in your trace made me think that since it's very unlikely
that the CPU is faulty (never impossible but extremely rare), another
thread was possibly still doing something in our back before the crash
happened, and fixed the value again before the dump was done. These are
thus two very quick changes. I don't see what sequence of actions can do
this but I think I want to study one code path that looks suspicious to
me. I need to double-check this tomorrow after some sleep, I'll keep you
informed.

Cheers,
Willy

Re: Infinite loop after 39cc020af BUG/MEDIUM: streams: Don't remove the SI_FL_ERR flag in si_update_both()

[Richard Russo]

After the weekend, the test machine looks fine. Thanks!

-- 
  Richard Russo
  to...@enslaves.us

On Fri, Apr 12, 2019, at 10:33 AM, Richard Russo wrote:
> Thank you; I had missed the context from 1.9.6.  I've updated my test 
> machine and will report back on Monday (or earlier, if it runs into 
> trouble)
> 
> -- 
>   Richard Russo
>   to...@enslaves.us
> 
> On Fri, Apr 12, 2019, at 4:17 AM, Olivier Houchard wrote:
> > Hi,
> > 
> > On Fri, Apr 12, 2019 at 08:37:10AM +0200, Maciej Zdeb wrote:
> > > Hi Richard,
> > > 
> > > Those patches from Olivier (in streams) are related to my report from
> > > thread "[1.9.6] One of haproxy processes using 100% CPU", but as it turned
> > > out it wasn't a single bug and issue is not entirely fixed yet.
> > > 
> > > Currently I'm testing some additional patches from Olivier which hopefully
> > > fix the issue definitely.
> > > 
> > 
> > Indeed, the rmoeval of SI_FL_ERR in si_update_both() was bogus, and covered
> > misuses of it.
> > With the great help of Maciej, we investigated this, and I just pushed what
> > we fixed so far. Richard, I'd be really interested in knowing if you still
> > have issues with the latest master.
> > 
> > Thanks !
> > 
> > Olivier
> > 
> > > pt., 12 kwi 2019 o 00:01 Richard Russo  napisał(a):
> > > 
> > > > It seems that after applying 39cc020af, if a stream gets the SI_FL_ERR
> > > > flag, process_stream can keep going back to redo around stream.c:line 
> > > > 2503:
> > > >
> > > > if (si_f->state == SI_ST_DIS || si_f->state != si_f_prev_state ||
> > > > si_b->state == SI_ST_DIS || si_b->state != si_b_prev_state ||
> > > > ((si_f->flags | si_b->flags) & SI_FL_ERR) ||
> > > > (((req->flags ^ rqf_last) | (res->flags ^ rpf_last)) &
> > > > CF_MASK_ANALYSER))
> > > >  goto redo;
> > > >
> > > > Now that si_update_both no longer clears the SI_FL_ERR flag, and nothing
> > > > else does, the goto will get called forever. I don't understand this
> > > > section enough to try to reproduce this, but I found several processes
> > > > stuck here on a machine testing from yesterday's HEAD.
> > > >
> > > > Richard
> > > >
> > > > --
> > > >   Richard Russo
> > > >   to...@enslaves.us
> > > >
> > > >
> >
> 
>

Re: SSL termination with HA proxy

[Aleksandar Lazic]

Hi.

Am 15.04.2019 um 18:06 schrieb bhanu chandra suman:
> Hi,
> 
> As per your mail i can understand again my create certificates in that server
> (.pemkey). is it right

Yes.

For a TLS/SSL server is at least a Key and a Certificate required.

Do you have already a Key and a Certificate?

Maybe this post helps you to create certificates.
https://serversforhackers.com/c/using-ssl-certificates-with-haproxy

Regards
Aleks

> On Mon, Apr 15, 2019 at 9:27 PM Aleksandar Lazic  > wrote:
> 
> Hi.
> 
> Am 15.04.2019 um 17:55 schrieb bhanu chandra suman:
> >
> > root@ip-172-31-80-163:~# uname -a
> > Linux ip-172-31-80-163 4.15.0-1035-aws #37-Ubuntu SMP Mon Mar 18 
> 16:15:14 UTC
> > 20                                                                      
>  
>      
> >           19 x86_64 x86_64 x86_64 GNU/Linux
> > root@ip-172-31-80-163:~# haproxy -v
> > HA-Proxy version 1.8.8-1ubuntu0.4 2019/01/24
> > Copyright 2000-2018 Willy Tarreau    >>
> 
> Well I assume this version have TLS/SSL enabled as you haven't used `-vv`!
> 
> Please take a look into this blog post which describes how to add TLS/SSL
> termination into haproxy.
> 
> 
> https://www.haproxy.com/blog/how-to-get-ssl-with-haproxy-getting-rid-of-stunnel-stud-nginx-or-pound/
> 
> Regards
> Aleks
> 
> > On Mon, Apr 15, 2019 at 8:58 PM Aleksandar Lazic  
> > >> wrote:
> >
> >     Hi.
> >
> >     Please keep the Mailinglist in the loop.
> >
> >     Am 15.04.2019 um 17:27 schrieb bhanu chandra suman:
> >     > image.png
> >
> >     It's not easy to copy text from Screenshot's so please copy text 
> into
> the mail.
> >
> >     Please use 2 v.
> >
> >     haproxy -vv
> >
> >     Thanks.
> >
> >     > On Mon, Apr 15, 2019 at 8:53 PM Aleksandar Lazic 
>  
> >     >
> >     > 
>  >     >
> >     >     Hi.
> >     >
> >     >     Am 15.04.2019 um 17:19 schrieb bhanu chandra suman:
> >     >     > Hi Team,
> >     >     >
> >     >     > I installed haproxy in ubuntu machine. and after that i 
> edited the
> >     >     haproxy.cfg file.
> >     >
> >     >     Please can you tell us more about this.
> >     >
> >     >     haproxy -vv
> >     >     uname -a
> >     >
> >     >     > bind *:18083
> >     >     > mode http
> >     >     > default_backend backendnodes
> >     >     > backend backendnodes
> >     >     > balance roundrobin
> >     >     > option forwardfor
> >     >     > server node1 x.x.x.x:18083 check
> >     >     > server node2 x.x.x.x:18083 check
> >     >     > listen stats
> >     >     > bind :32700
> >     >     > stats enable
> >     >     > stats uri /
> >     >     > stats hide-version
> >     >     > stats auth user:password
> >     >     > Its working fine.but i need SSL termination with HA proxy.
> >     >     > could you please help me this issue.
> >     >
> >     >     Please take a look into this blog post which describes how 
> TLS/SSL
> >     Termination
> >     >     works in haproxy.
> >     >
> >     >   
> >   
>   
> https://www.haproxy.com/blog/how-to-get-ssl-with-haproxy-getting-rid-of-stunnel-stud-nginx-or-pound/
> >     >
> >     >     > --
> >     >     > S.B.C.Suman
> >     >
> >     >     Regards
> >     >     Aleks
> >     >
> >     >
> >     >
> >     > --
> >     > S.B.C.Suman
> >     > +91 9989894950.
> >
> >
> >
> > --
> > S.B.C.Suman
> > +91 9989894950.
> 
> 
> 
> -- 
> S.B.C.Suman
> +91 9989894950.

RE: SSL termination with HA proxy

[Gibson, Brian (IMS)]

You need to run haproxy –vv not hparoxy –v.  Your output should look something 
like this:
haproxy -vv
HA-Proxy version 1.8.19 2019/02/11
Copyright 2000-2019 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv 
-Wno-unused-label
  OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 
USE_SYSTEMD=1 USE_PCRE2=1 USE_PCRE2_JIT=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.1a  20 Nov 2018
Running on OpenSSL version : OpenSSL 1.1.1b  26 Feb 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE2 version : 10.31 2018-02-12
PCRE2 library supports JIT : yes
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), 
raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace

From: bhanu chandra suman [mailto:bhanuchandra.su...@gmail.com]
Sent: Monday, April 15, 2019 11:56 AM
To: Aleksandar Lazic 
Cc: haproxy 
Subject: Re: SSL termination with HA proxy

[https://mailtrack.io/trace/mail/fbd2a0eab7e2e5568c9b88276b6623f9505f8adb.png?u=3280423]

root@ip-172-31-80-163:~# uname -a
Linux ip-172-31-80-163 4.15.0-1035-aws #37-Ubuntu SMP Mon Mar 18 16:15:14 UTC 
20  
  19 x86_64 x86_64 x86_64 GNU/Linux
root@ip-172-31-80-163:~# haproxy -v
HA-Proxy version 1.8.8-1ubuntu0.4 2019/01/24
Copyright 2000-2018 Willy Tarreau mailto:wi...@haproxy.org>>



On Mon, Apr 15, 2019 at 8:58 PM Aleksandar Lazic 
mailto:al-hapr...@none.at>> wrote:
Hi.

Please keep the Mailinglist in the loop.

Am 15.04.2019 um 17:27 schrieb bhanu chandra suman:
> image.png

It's not easy to copy text from Screenshot's so please copy text into the mail.

Please use 2 v.

haproxy -vv

Thanks.

> On Mon, Apr 15, 2019 at 8:53 PM Aleksandar Lazic 
> mailto:al-hapr...@none.at>
> >> wrote:
>
> Hi.
>
> Am 15.04.2019 um 17:19 schrieb bhanu chandra suman:
> > Hi Team,
> >
> > I installed haproxy in ubuntu machine. and after that i edited the
> haproxy.cfg file.
>
> Please can you tell us more about this.
>
> haproxy -vv
> uname -a
>
> > bind *:18083
> > mode http
> > default_backend backendnodes
> > backend backendnodes
> > balance roundrobin
> > option forwardfor
> > server node1 x.x.x.x:18083 check
> > server node2 x.x.x.x:18083 check
> > listen stats
> > bind :32700
> > stats enable
> > stats uri /
> > stats hide-version
> > stats auth user:password
> > Its working fine.but i need SSL termination with HA proxy.
> > could you please help me this issue.
>
> Please take a look into this blog post which describes how TLS/SSL 
> Termination
> works in haproxy.
>
> 
> https://www.haproxy.com/blog/how-to-get-ssl-with-haproxy-getting-rid-of-stunnel-stud-nginx-or-pound/
>
> > --
> > S.B.C.Suman
>
> Regards
> Aleks
>
>
>
> --
> S.B.C.Suman
> +91 9989894950.


--
S.B.C.Suman
+91 9989894950.



Information in this e-mail may be confidential. It is intended only for the 
addressee(s) identified above. If you are not the addressee(s), or an employee 
or agent of the addressee(s), please note that any dissemination, distribution, 
or copying of this communication is strictly prohibited. If you have received 
this e-mail in error, please notify the sender of the error.

Re: SSL termination with HA proxy

[Aleksandar Lazic]

Hi.

Am 15.04.2019 um 17:55 schrieb bhanu chandra suman:
> 
> root@ip-172-31-80-163:~# uname -a
> Linux ip-172-31-80-163 4.15.0-1035-aws #37-Ubuntu SMP Mon Mar 18 16:15:14 UTC
> 20                                                                            
>  
>           19 x86_64 x86_64 x86_64 GNU/Linux
> root@ip-172-31-80-163:~# haproxy -v
> HA-Proxy version 1.8.8-1ubuntu0.4 2019/01/24
> Copyright 2000-2018 Willy Tarreau  >

Well I assume this version have TLS/SSL enabled as you haven't used `-vv`!

Please take a look into this blog post which describes how to add TLS/SSL
termination into haproxy.

https://www.haproxy.com/blog/how-to-get-ssl-with-haproxy-getting-rid-of-stunnel-stud-nginx-or-pound/

Regards
Aleks

> On Mon, Apr 15, 2019 at 8:58 PM Aleksandar Lazic  > wrote:
> 
> Hi.
> 
> Please keep the Mailinglist in the loop.
> 
> Am 15.04.2019 um 17:27 schrieb bhanu chandra suman:
> > image.png
> 
> It's not easy to copy text from Screenshot's so please copy text into the 
> mail.
> 
> Please use 2 v.
> 
> haproxy -vv
> 
> Thanks.
> 
> > On Mon, Apr 15, 2019 at 8:53 PM Aleksandar Lazic  
> > >> wrote:
> >
> >     Hi.
> >
> >     Am 15.04.2019 um 17:19 schrieb bhanu chandra suman:
> >     > Hi Team,
> >     >
> >     > I installed haproxy in ubuntu machine. and after that i edited the
> >     haproxy.cfg file.
> >
> >     Please can you tell us more about this.
> >
> >     haproxy -vv
> >     uname -a
> >
> >     > bind *:18083
> >     > mode http
> >     > default_backend backendnodes
> >     > backend backendnodes
> >     > balance roundrobin
> >     > option forwardfor
> >     > server node1 x.x.x.x:18083 check
> >     > server node2 x.x.x.x:18083 check
> >     > listen stats
> >     > bind :32700
> >     > stats enable
> >     > stats uri /
> >     > stats hide-version
> >     > stats auth user:password
> >     > Its working fine.but i need SSL termination with HA proxy.
> >     > could you please help me this issue.
> >
> >     Please take a look into this blog post which describes how TLS/SSL
> Termination
> >     works in haproxy.
> >
> >   
>  
> https://www.haproxy.com/blog/how-to-get-ssl-with-haproxy-getting-rid-of-stunnel-stud-nginx-or-pound/
> >
> >     > --
> >     > S.B.C.Suman
> >
> >     Regards
> >     Aleks
> >
> >
> >
> > --
> > S.B.C.Suman
> > +91 9989894950.
> 
> 
> 
> -- 
> S.B.C.Suman
> +91 9989894950.

Re: SSL termination with HA proxy

[bhanu chandra suman]

root@ip-172-31-80-163:~# uname -a
Linux ip-172-31-80-163 4.15.0-1035-aws #37-Ubuntu SMP Mon Mar 18 16:15:14
UTC 20
  19 x86_64 x86_64 x86_64 GNU/Linux
root@ip-172-31-80-163:~# haproxy -v
HA-Proxy version 1.8.8-1ubuntu0.4 2019/01/24
Copyright 2000-2018 Willy Tarreau 



On Mon, Apr 15, 2019 at 8:58 PM Aleksandar Lazic  wrote:

> Hi.
>
> Please keep the Mailinglist in the loop.
>
> Am 15.04.2019 um 17:27 schrieb bhanu chandra suman:
> > image.png
>
> It's not easy to copy text from Screenshot's so please copy text into the
> mail.
>
> Please use 2 v.
>
> haproxy -vv
>
> Thanks.
>
> > On Mon, Apr 15, 2019 at 8:53 PM Aleksandar Lazic  > > wrote:
> >
> > Hi.
> >
> > Am 15.04.2019 um 17:19 schrieb bhanu chandra suman:
> > > Hi Team,
> > >
> > > I installed haproxy in ubuntu machine. and after that i edited the
> > haproxy.cfg file.
> >
> > Please can you tell us more about this.
> >
> > haproxy -vv
> > uname -a
> >
> > > bind *:18083
> > > mode http
> > > default_backend backendnodes
> > > backend backendnodes
> > > balance roundrobin
> > > option forwardfor
> > > server node1 x.x.x.x:18083 check
> > > server node2 x.x.x.x:18083 check
> > > listen stats
> > > bind :32700
> > > stats enable
> > > stats uri /
> > > stats hide-version
> > > stats auth user:password
> > > Its working fine.but i need SSL termination with HA proxy.
> > > could you please help me this issue.
> >
> > Please take a look into this blog post which describes how TLS/SSL
> Termination
> > works in haproxy.
> >
> >
> https://www.haproxy.com/blog/how-to-get-ssl-with-haproxy-getting-rid-of-stunnel-stud-nginx-or-pound/
> >
> > > --
> > > S.B.C.Suman
> >
> > Regards
> > Aleks
> >
> >
> >
> > --
> > S.B.C.Suman
> > +91 9989894950.
>
>

-- 
S.B.C.Suman
+91 9989894950.

Re: SSL termination with HA proxy

[Aleksandar Lazic]

Hi.

Please keep the Mailinglist in the loop.

Am 15.04.2019 um 17:27 schrieb bhanu chandra suman:
> image.png

It's not easy to copy text from Screenshot's so please copy text into the mail.

Please use 2 v.

haproxy -vv

Thanks.

> On Mon, Apr 15, 2019 at 8:53 PM Aleksandar Lazic  > wrote:
> 
> Hi.
> 
> Am 15.04.2019 um 17:19 schrieb bhanu chandra suman:
> > Hi Team,
> >
> > I installed haproxy in ubuntu machine. and after that i edited the
> haproxy.cfg file.
> 
> Please can you tell us more about this.
> 
> haproxy -vv
> uname -a
> 
> > bind *:18083
> > mode http
> > default_backend backendnodes
> > backend backendnodes
> > balance roundrobin
> > option forwardfor
> > server node1 x.x.x.x:18083 check
> > server node2 x.x.x.x:18083 check
> > listen stats
> > bind :32700
> > stats enable
> > stats uri /
> > stats hide-version
> > stats auth user:password
> > Its working fine.but i need SSL termination with HA proxy.
> > could you please help me this issue.
> 
> Please take a look into this blog post which describes how TLS/SSL 
> Termination
> works in haproxy.
> 
> 
> https://www.haproxy.com/blog/how-to-get-ssl-with-haproxy-getting-rid-of-stunnel-stud-nginx-or-pound/
> 
> > --
> > S.B.C.Suman
> 
> Regards
> Aleks
> 
> 
> 
> -- 
> S.B.C.Suman
> +91 9989894950.

Re: SSL termination with HA proxy

[Aleksandar Lazic]

Hi.

Am 15.04.2019 um 17:19 schrieb bhanu chandra suman:
> Hi Team,
> 
> I installed haproxy in ubuntu machine. and after that i edited the 
> haproxy.cfg file.

Please can you tell us more about this.

haproxy -vv
uname -a

> bind *:18083
> mode http
> default_backend backendnodes
> backend backendnodes
> balance roundrobin
> option forwardfor
> server node1 x.x.x.x:18083 check
> server node2 x.x.x.x:18083 check
> listen stats
> bind :32700
> stats enable
> stats uri /
> stats hide-version
> stats auth user:password
> Its working fine.but i need SSL termination with HA proxy.
> could you please help me this issue.

Please take a look into this blog post which describes how TLS/SSL Termination
works in haproxy.

https://www.haproxy.com/blog/how-to-get-ssl-with-haproxy-getting-rid-of-stunnel-stud-nginx-or-pound/

> -- 
> S.B.C.Suman

Regards
Aleks

SSL termination with HA proxy

[bhanu chandra suman]

Hi Team,

I installed haproxy in ubuntu machine. and after that i edited the
haproxy.cfg file.
bind *:18083
mode http
default_backend backendnodes
backend backendnodes
balance roundrobin
option forwardfor
server node1 x.x.x.x:18083 check
server node2 x.x.x.x:18083 check
listen stats
bind :32700
stats enable
stats uri /
stats hide-version
stats auth user:password
Its working fine.but i need SSL termination with HA proxy.
could you please help me this issue.


-- 
S.B.C.Suman
+91 9989894950.

Re: [PATCH] BUG/MEDIUM: map: Fix memory leak in the map converter

[Willy Tarreau]

Hi Nenad,

On Fri, Apr 12, 2019 at 10:54:28PM +0200, Nenad Merdanovic wrote:
> The allocated trash chunk is not freed properly and causes a memory leak
> exhibited as the growth in the trash pool allocations. Bug was introduced
> in commit 271022 (BUG/MINOR: map: fix map_regm with backref).

Good catch! Now merged. However be extremely careful with map_regm().
As the age of this bug indicates it, almost nobody's using it, and for a
reason which is that just like map_reg() it can be extremely expensive
if users start to update the map with lots of entries without understanding
the impacts of iterating over hundreds of regex. However it's true that it
can be extremely convenient to perform some small transformations a-la sed.

Cheers,
Willy