HAProxy in front of Docker Enterprise problem

[Norman Branitsky]

I have an HAProxy 1.7 server sitting in front of a number of Docker Enterprise 
Manager nodes and Worker nodes.
The Worker nodes don't appear to have any problem with HAProxy terminating the 
SSL and connecting to them via HTTP.
The Manager nodes are the problem.
They insist on installing their own certificates (either self-signed or CA 
signed).
They will only listen to HTTPS traffic.

So my generic frontend_main-ssl says:
bind :443  ssl crt /etc/CONFIG/haproxy-1.7/certs/cert.pem

The backend has the following server statement:
server xxx 10.240.12.248:443 ssl verify none

But apparently this doesn't work - the client gets the SSL certificate provided 
by the HAProxy server
instead of the certificate provided by the Manager node. This causes the 
Manager node to barf.

Do I have to make HAProxy listen on 8443 and just do a tcp frontend/backend for 
the Manager nodes?

Norman Branitsky

Re: haproxy segfault

[Vincent Bernat]

 ❦ 12 février 2019 21:44 +01, Mildis :

> I'm struggling with Stretch/systemd to generate the coredump on crash.
> Even running haproxy by hand with ulimit -c unlimited does not
> generate a coredump.

Also install haproxy-dbgsym. You need to comment the chroot directive in
your HAProxy configuration if it's enabled. Also, you need to set the
core pattern to a fixed directory where haproxy user can write to, like:

sysctl -w kernel.core_pattern=/tmp/core.%p

Then, on next segfault, you should get your coredump.
-- 
Let me take you a button-hole lower.
-- William Shakespeare, "Love's Labour's Lost"

Re: haproxy segfault

[Lukas Tribus]

On Tue, 12 Feb 2019 at 21:46, Mildis  wrote:
>
>
> Le 12 févr. 2019 à 21:26, Christopher Faulet  a écrit :
>
>
> Hi,
>
> A recent fix about a double free has been merge in HAProxy 1.9:
>
>  http://git.haproxy.org/?p=haproxy-1.9.git;a=commit;h=451c5a88
>
> Maybe you've hit this bug.
>
>
> Did this bug has been introduced in 1.9.4 ?
> I haven't notice this behavior before.

Yes. Also see the 1.8.19 release notes.

Lukas

Re: haproxy segfault

[Mildis]

> Le 12 févr. 2019 à 21:26, Christopher Faulet  a écrit :
>> 
> Hi,
> 
> A recent fix about a double free has been merge in HAProxy 1.9:
> 
>  http://git.haproxy.org/?p=haproxy-1.9.git;a=commit;h=451c5a88 
> 
> 
> Maybe you've hit this bug.

Did this bug has been introduced in 1.9.4 ?
I haven't notice this behavior before.

> 
> -- 
> Christopher Faulet

Re: haproxy segfault

[Mildis]

> Le 12 févr. 2019 à 18:58, Aleksandar Lazic  a écrit :
> 
> Hi.
> 
> Can you activate coredumps
> 
> ulimit -c unlimited
> 
> you should find the core in
> 
> /tmp
> 
> or just search for core on the filesystem
> 

I'm struggling with Stretch/systemd to generate the coredump on crash.
Even running haproxy by hand with ulimit -c unlimited does not generate a 
coredump.

Even after a reboot, the +1e2000 offset is the same.

> You can get a backtrace with the following command as soon as you have a 
> coredump
> 
> gdb /usr/sbin/haproxy #YOUR_CORE_FILE#
> bt full
> 
>> Thanks,
>> mildis
> 
> Regards
> aleks

Re: haproxy segfault

[Christopher Faulet]

Le 12/02/2019 à 18:36, Mildis a écrit :

Hi list,

haproxy is segfaulting multiple times these days for no apparent reason.
At first i thought is was a load issue but even few RPS made it crash.

Symptoms are always the same : segfault of a worker then spawn of a new.
If load is very high, spawned worker segfault immediatly.

In the messages log, the offset is always the same (+1e2000).

I'm running 1.9.4 (from vincent bernat package) in Debian stretch.

In haproxy logs :
Feb 12 11:36:54 ns3089939 haproxy[32688]: [ALERT] 042/113654 (32688) : Current 
worker #1 (32689) exited with code 139 (Segmentation fault)
Feb 12 11:36:54 ns3089939 haproxy[32688]: [ALERT] 042/113654 (32688) : 
exit-on-failure: killing every workers with SIGTERM
Feb 12 11:36:54 ns3089939 haproxy[32688]: [WARNING] 042/113654 (32688) : All 
workers exited. Exiting... (139)

In /var/log/messages
kernel: traps: haproxy[32689] general protection ip:561e5b799375 
sp:7ffe6fd3f2f0 error:0 in haproxy[561e5b72d000+1e2000]

"show errors" is empty.

How could I diagnose further without impacting production too much ?


Hi,

A recent fix about a double free has been merge in HAProxy 1.9:

  http://git.haproxy.org/?p=haproxy-1.9.git;a=commit;h=451c5a88

Maybe you've hit this bug.

--
Christopher Faulet

Re: haproxy segfault

[Aleksandar Lazic]

Hi.

Am 12.02.2019 um 18:36 schrieb Mildis:
> Hi list,
> 
> haproxy is segfaulting multiple times these days for no apparent reason.
> At first i thought is was a load issue but even few RPS made it crash.
> 
> Symptoms are always the same : segfault of a worker then spawn of a new.
> If load is very high, spawned worker segfault immediatly.
> 
> In the messages log, the offset is always the same (+1e2000).
> 
> I'm running 1.9.4 (from vincent bernat package) in Debian stretch.
> 
> In haproxy logs :
> Feb 12 11:36:54 ns3089939 haproxy[32688]: [ALERT] 042/113654 (32688) : 
> Current worker #1 (32689) exited with code 139 (Segmentation fault)
> Feb 12 11:36:54 ns3089939 haproxy[32688]: [ALERT] 042/113654 (32688) : 
> exit-on-failure: killing every workers with SIGTERM
> Feb 12 11:36:54 ns3089939 haproxy[32688]: [WARNING] 042/113654 (32688) : All 
> workers exited. Exiting... (139)
> 
> In /var/log/messages
> kernel: traps: haproxy[32689] general protection ip:561e5b799375 
> sp:7ffe6fd3f2f0 error:0 in haproxy[561e5b72d000+1e2000]
> 
> "show errors" is empty.
> 
> How could I diagnose further without impacting production too much ?

Can you activate coredumps

ulimit -c unlimited

you should find the core in

/tmp

or just search for core on the filesystem

You can get a backtrace with the following command as soon as you have a 
coredump

gdb /usr/sbin/haproxy #YOUR_CORE_FILE#
bt full

> Thanks,
> mildis

Regards
aleks

haproxy segfault

[Mildis]

Hi list,

haproxy is segfaulting multiple times these days for no apparent reason.
At first i thought is was a load issue but even few RPS made it crash.

Symptoms are always the same : segfault of a worker then spawn of a new.
If load is very high, spawned worker segfault immediatly.

In the messages log, the offset is always the same (+1e2000).

I'm running 1.9.4 (from vincent bernat package) in Debian stretch.

In haproxy logs :
Feb 12 11:36:54 ns3089939 haproxy[32688]: [ALERT] 042/113654 (32688) : Current 
worker #1 (32689) exited with code 139 (Segmentation fault)
Feb 12 11:36:54 ns3089939 haproxy[32688]: [ALERT] 042/113654 (32688) : 
exit-on-failure: killing every workers with SIGTERM
Feb 12 11:36:54 ns3089939 haproxy[32688]: [WARNING] 042/113654 (32688) : All 
workers exited. Exiting... (139)

In /var/log/messages
kernel: traps: haproxy[32689] general protection ip:561e5b799375 
sp:7ffe6fd3f2f0 error:0 in haproxy[561e5b72d000+1e2000]

"show errors" is empty.

How could I diagnose further without impacting production too much ?

Thanks,
mildis

Re: Anyone heard about DPDK?

[Aleksandar Lazic]

Hi all.

Wow so much feedback, thanks all for the answers ;-)

Am 12.02.2019 um 15:23 schrieb Alexandre Cassen:
> There has been a lot of applications/stack built around DPDK last few years.
> Mostly because people found it easy to code stuff around DPDK and are so happy
> to display perf graph about their DPDK application vs plain Linux Kernel 
> stack.

Would you like to share such a comparison?

> My intention here would be to warn a little bit about this collective 
> enthusiasm
> around DPDK. Integrating DPDK is easy and mostly fun (even if you have to 
> learn
> and dig into their rte lib and mbuf related), but most of people are 
> completely
> blind about security ! Ok Linux kernel and netdev is slow in respect of NIC
> available nowadays (10G, 40G and multiple 100G on core-networks), but using
> Linux TCP/IP stack you will benefit the hardcore hacking task done during last
> 30years by Linux netdev core guys ! this long process mostly fix and solve
> hardcore issues and for some : security issues. And you will certainly not be
> protected by a 'super fast' self proclaimed performance soft. Mostly because
> these applications are mostly features oriented than security or protocol
> full-picture, and are using this 'super fast, best of ever' argument to 
> enforce
> people mind to adopt.

When I take a look into the doc then I see some security informations.

https://doc.dpdk.org/guides/prog_guide/rte_security.html

How does such a application handle the security topic?

> The way DPDK is working in polling mode is certainly not the best at all. DPDK
> is PCI 'stealing' NIC from kernel to handle/manage itself in userspace by
> forcing active loop (100% CPU polling) to handle descriptors and convert to
> mbuf. latter you can 'forward' mbuf to Linux kernel by using KNI netdevice to
> use Linux Kernel machinery as a slow-path for complicated/not_focused
> packet-flow (most application are using KNI for ARP,DHCP,...). But most of the
> time application are implementing 'minimal' adjacent network features to make 
> it
> work in its networking environment : and here is the problem: you are focused 
> on
> perf and because of it you are making shortcut about considering potential
> threats... a prediction could be to see large number of network security holes
> opened, and specially an old bunch of security holes making a fun revival (a 
> lot
> of fun with TCP)

So this means that a application can be used with DPDK when it uses the
KNI (=Kernel NIC Interface) right?

https://doc.dpdk.org/guides/prog_guide/kernel_nic_interface.html

How much "slower" is the way via KNI?

> In contrast recent Linux Kernel introduced XDP and eBPF machinery that are
> certainly much more future proof than DPDK. First consideration in XDP design 
> is
> : you only TAP in data/packet you are interested in and not making an hold-up 
> on
> whole traffic. So XDP is for fast path but only for protocol or workflow
> identified. You program and attach an eBPF program to a specific NIC, if there
> is no match then packet simply continue its journey into Linux Kernel stack.
> 
> XDP is a response from kernel netdev community to address DPDK users. The fact
> that DPDK introduced and extended PMP to support AF_XDP is certainly a sign 
> that
> XDP is going/doing into the right direction.

Sounds a interesting future for the linux kernel.

When we take a look into the container and cloud world, does this DPDK makes any
sense? I mean when I run a container on AWS/Google/Azure I'm normally so far
from any Hardware that this high traffic possibility isn't available for the
container, right?

To the list members:
Maybe it's offtopic from the HAProxy list so please apologize for all the noise.

> regs,
> Alexandre

Regards
Aleks

> On 12/02/2019 14:04, Federico Iezzi wrote:
>> Nowadays most VNF (virtual network function) in the telco operators are built
>> around DPDK. Not demos, most 5G will be like that. 4G is migrating as we 
>> speak
>> on this new architecture.
>> There isn't any TCP stack built-it but the libraries can be used to build 
>> one.
>> VPP has integrated DPDK in this way.
>>
>> Linux network stack is not designed to managed millions of packets per 
>> second,
>> DPDK bypass it completely offloading everything in userspace. The beauty is
>> that also the physical nic drivers are in userspace using specific DPDK
>> drivers. Linux networking stack works in interrupt mode, DPDK is in polling
>> mode, basically with a while true.
>>
>>  From F5 at the dpdk summit as a relevant reference to what HAProxy does.
>> https://dpdksummitnorthamerica2018.sched.com/event/IhiF/dpdk-on-f5-big-ip-virtual-adcs-brent-blood-f5-networks
>>
>> https://www.youtube.com/watch?v=6zu81p3oTeo
>>
>> Regards,
>> Federico
>>
>> On Tue, 12 Feb 2019 at 11:08, Julien Laffaye > > wrote:
>>
>>     Something like http://seastar.io/ or https://fd.io/ ? :)
>>
>>     On Mon, Feb 11, 2019 at 11:25 AM Baptiste >     

Re: Anyone heard about DPDK?

[Alexandre Cassen]

There has been a lot of applications/stack built around DPDK last few 
years. Mostly because people found it easy to code stuff around DPDK and 
are so happy to display perf graph about their DPDK application vs plain 
Linux Kernel stack.


My intention here would be to warn a little bit about this collective 
enthusiasm around DPDK. Integrating DPDK is easy and mostly fun (even if 
you have to learn and dig into their rte lib and mbuf related), but most 
of people are completely blind about security ! Ok Linux kernel and 
netdev is slow in respect of NIC available nowadays (10G, 40G and 
multiple 100G on core-networks), but using Linux TCP/IP stack you will 
benefit the hardcore hacking task done during last 30years by Linux 
netdev core guys ! this long process mostly fix and solve hardcore 
issues and for some : security issues. And you will certainly not be 
protected by a 'super fast' self proclaimed performance soft. Mostly 
because these applications are mostly features oriented than security or 
protocol full-picture, and are using this 'super fast, best of ever' 
argument to enforce people mind to adopt.


The way DPDK is working in polling mode is certainly not the best at 
all. DPDK is PCI 'stealing' NIC from kernel to handle/manage itself in 
userspace by forcing active loop (100% CPU polling) to handle 
descriptors and convert to mbuf. latter you can 'forward' mbuf to Linux 
kernel by using KNI netdevice to use Linux Kernel machinery as a 
slow-path for complicated/not_focused packet-flow (most application are 
using KNI for ARP,DHCP,...). But most of the time application are 
implementing 'minimal' adjacent network features to make it work in its 
networking environment : and here is the problem: you are focused on 
perf and because of it you are making shortcut about considering 
potential threats... a prediction could be to see large number of 
network security holes opened, and specially an old bunch of security 
holes making a fun revival (a lot of fun with TCP)


In contrast recent Linux Kernel introduced XDP and eBPF machinery that 
are certainly much more future proof than DPDK. First consideration in 
XDP design is : you only TAP in data/packet you are interested in and 
not making an hold-up on whole traffic. So XDP is for fast path but only 
for protocol or workflow identified. You program and attach an eBPF 
program to a specific NIC, if there is no match then packet simply 
continue its journey into Linux Kernel stack.


XDP is a response from kernel netdev community to address DPDK users. 
The fact that DPDK introduced and extended PMP to support AF_XDP is 
certainly a sign that XDP is going/doing into the right direction.


regs,
Alexandre



On 12/02/2019 14:04, Federico Iezzi wrote:
Nowadays most VNF (virtual network function) in the telco operators are 
built around DPDK. Not demos, most 5G will be like that. 4G is migrating 
as we speak on this new architecture.
There isn't any TCP stack built-it but the libraries can be used to 
build one. VPP has integrated DPDK in this way.


Linux network stack is not designed to managed millions of packets per 
second, DPDK bypass it completely offloading everything in userspace. 
The beauty is that also the physical nic drivers are in userspace using 
specific DPDK drivers. Linux networking stack works in interrupt mode, 
DPDK is in polling mode, basically with a while true.


 From F5 at the dpdk summit as a relevant reference to what HAProxy does.
https://dpdksummitnorthamerica2018.sched.com/event/IhiF/dpdk-on-f5-big-ip-virtual-adcs-brent-blood-f5-networks
https://www.youtube.com/watch?v=6zu81p3oTeo

Regards,
Federico

On Tue, 12 Feb 2019 at 11:08, Julien Laffaye > wrote:


Something like http://seastar.io/ or https://fd.io/ ? :)

On Mon, Feb 11, 2019 at 11:25 AM Baptiste mailto:bed...@gmail.com>> wrote:

Hi,

HAProxy requires a TCP stack below it. DPDK itself is not enough.

Baptiste

Re: Anyone heard about DPDK?

[Federico Iezzi]

Nowadays most VNF (virtual network function) in the telco operators are
built around DPDK. Not demos, most 5G will be like that. 4G is migrating as
we speak on this new architecture.
There isn't any TCP stack built-it but the libraries can be used to build
one. VPP has integrated DPDK in this way.

Linux network stack is not designed to managed millions of packets per
second, DPDK bypass it completely offloading everything in userspace. The
beauty is that also the physical nic drivers are in userspace using
specific DPDK drivers. Linux networking stack works in interrupt mode, DPDK
is in polling mode, basically with a while true.

>From F5 at the dpdk summit as a relevant reference to what HAProxy does.
https://dpdksummitnorthamerica2018.sched.com/event/IhiF/dpdk-on-f5-big-ip-virtual-adcs-brent-blood-f5-networks
https://www.youtube.com/watch?v=6zu81p3oTeo

Regards,
Federico

On Tue, 12 Feb 2019 at 11:08, Julien Laffaye  wrote:

> Something like http://seastar.io/ or https://fd.io/ ? :)
>
> On Mon, Feb 11, 2019 at 11:25 AM Baptiste  wrote:
>
>> Hi,
>>
>> HAProxy requires a TCP stack below it. DPDK itself is not enough.
>>
>> Baptiste
>>
>>>

Re: forwarded https request missing

[Kevin Decherf]

Hello,

On Tue, Feb 12, 2019, at 11:15, Pan wrote:
> Hi,
> 
> Setting up a haproxy to just forward http requests to other server 
> works fine, but https requests are forwarded as 
> http://www.example.com:443, not https://www.example.com.

[snip]

> backend bk
>  server src www.example.com

You must provide the target port on the backend server, see documentation:

  If unset, the same port the client
  connected to will be used

-- 
Kevin Decherf - @Kdecherf
GPG 0x108ABD75A81E6E2F
https://kdecherf.com

forwarded https request missing

[Pan]

Hi,

Setting up a haproxy to just forward http requests to other server works fine, 
but https requests are forwarded as http://www.example.com:443 
, not https://www.example.com 
.

Then got 400 response with body saying something like 'The plain HTTP request 
was sent to HTTPS port’.

below is the haproxy conf. Is there anything that is missing to make this work?



global
  log /dev/log  local0
  log /dev/log  local1 notice
  chroot /var/lib/haproxy
  stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
  stats timeout 30s
  user haproxy
  group haproxy
  daemon

  # Default SSL material locations
  ca-base /etc/ssl/certs
  crt-base /etc/ssl/private
  ssl-default-bind-ciphers 
ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
  ssl-default-bind-options no-sslv3
  tune.ssl.default-dh-param 2048

defaults
  log global
  mode  http
  option  httplog
  option  dontlognull
  timeout connect 5000
  timeout client  5
  timeout server  5
  errorfile 400 /etc/haproxy/errors/400.http
  errorfile 403 /etc/haproxy/errors/403.http
  errorfile 408 /etc/haproxy/errors/408.http
  errorfile 500 /etc/haproxy/errors/500.http
  errorfile 502 /etc/haproxy/errors/502.http
  errorfile 503 /etc/haproxy/errors/503.http
  errorfile 504 /etc/haproxy/errors/504.http

frontend http-in
  bind :80
  bind :443 crt /etc/haproxy/cert.d ssl
  use_backend bk

backend bk
  server src www.example.com

Re: Anyone heard about DPDK?

[Julien Laffaye]

Something like http://seastar.io/ or https://fd.io/ ? :)

On Mon, Feb 11, 2019 at 11:25 AM Baptiste  wrote:

> Hi,
>
> HAProxy requires a TCP stack below it. DPDK itself is not enough.
>
> Baptiste
>
>>