Re: Transparent proxy issue on FreeBSD

[Rainer Duffner]

> Am 07.03.2023 um 18:26 schrieb Marc West :
> 
> On 2023-03-07 08:09:04, Rainer Duffner wrote:
>> I admit I only toyed with TP, so I really don???t know what I???m doing 
>> there, but:
>> 
>> Have you tried to just use pfSense for this? The developer of the package 
>> (https://github.com/PiBa-NL) seemed to be active here, but I haven???t seen 
>> anything from him since 2020, so I wonder if he has moved on.
>> 
>> My co-workers use OPNSense for this purpose - and on VMWare, they insist 
>> that only em(4) NICs work.
>> 
>> 
>> If you don???t find his email-address, I can mail it to you.
> 
> Thanks for the suggestion. I haven't tried HAProxy on pfSense but the
> working transparent config and related ipfw fwd rules we have did come
> from PiBa-NL [1].


Ah, ok.

Either ask on the freebsd-forum or the mailing-list - or try with 
OPNSense/pfSense and if the problem persists, you might get more response on 
the forums there.

pf and ipfw are very specialized parts of the kernel and very few developers 
want to touch it, AFAIK.


> Everything does function perfectly until a brief
> period with production traffic and something happens to cause the tproxy
> bind errors and request failures to start. I'm just not sure what is
> going wrong or how to debug further.
> 
> [1] https://www.mail-archive.com/haproxy@formilux.org/msg09923.html
> 

Re: Transparent proxy issue on FreeBSD

[Rainer Duffner]

> Am 07.03.2023 um 08:46 schrieb Marc West :
> 
> 
> 
> Any other thoughts to look at or data that would be helpful to collect?
> 


I admit I only toyed with TP, so I really don’t know what I’m doing there, but:

Have you tried to just use pfSense for this? The developer of the package 
(https://github.com/PiBa-NL) seemed to be active here, but I haven’t seen 
anything from him since 2020, so I wonder if he has moved on.

My co-workers use OPNSense for this purpose - and on VMWare, they insist that 
only em(4) NICs work.


If you don’t find his email-address, I can mail it to you.

Re: OT: About WebPageTest results (was Re: SSL Labs says my server isn't doing ssl session resumption)

[Rainer Duffner]

> Am 21.06.2021 um 18:25 schrieb Shawn Heisey :
> 
> On 2021-06-20 06:03, Shawn Heisey wrote:
>> Unrelated, and off topic because it's mostly about Apache, but strange:
>> I've been doing some tests with webpagetest.org, and seeing REALLY
>> long load times for some resources in their waterfall graph.  I see no
>> speed problems when I load the pages from my workstation at home.
> 
> Followup on this, information which others here might find useful:
> 
> By default WebPageTest defaults to traffic shaping of 5 Mbps down and 1 Mbps 
> up, which it thinks simulates a cable connection.  That's laughable -- I get 
> 460 Mbps down and 12 Mpbs up on my cable connection, and I'm not even paying 
> for the maximum bandwidth I COULD get.
> 
> Long story short, hitting a web page with about 25 megabytes of images takes 
> over 40 seconds for WebPageTest to render.  If I switch from that default 
> "5/1 Mbps Cable" traffic shaping to native (no traffic shaping at all) the 
> render takes 1.8 seconds, which is approximately what I see when I hit the 
> page myself.  Server in AWS.
> 
> When I do the math, 40 seconds is actually quite fast for downloading those 
> images on a 5 megabit connection.  So there was no actual problem.  WBT needs 
> to make the choice of traffic shaping a lot more prominent, and provide more 
> realistic options than what they have at the moment.  To even see bandwidth 
> options, you have to open advanced settings.  And the only option I could see 
> in their list that's faster than the default (aside from native) is FIOS, 
> which they've got at 20Mb down and 5Mb up.  They have forums, I'll make 
> suggestions there.
> 
> Thanks,
> Shawn
> 


It’s probably to make DDoSes more difficult (like basically everything these 
days)


I never got around to host my own WPT instance (for work).

I mainly use the public version to to get „a feeling“ for the speed and to weed 
out any caching effects of local browsers with pages too complex to use curl or 
httpie….

Re: NFS mounts freezing via Haproxy

[Rainer Duffner]

> Am 22.05.2018 um 06:46 schrieb TomK :
> 
> Trying to mount an NFS share vi an Haproxy / Keepalived configuration. When I 
> mount the NFS share directly from the host, bypassing Haproxy / Keepalived, 
> it works fine.  However, when I try via the Haproxy / Keepalived combination, 
> it freezes.



Maybe I’m a little slow - but what exactly is this config trying to achieve?

Re: HaProxy Hang

[Rainer Duffner]

> Am 03.03.2017 um 15:07 schrieb David King :
> 
> Hi All
> 
> Hoping someone will be able to help, we're running a bit of an interesting 
> setup
> 
> we have 3 HAProxy nodes running freebsd 11.0 , each host runs 4 jails, each 
> running haproxy, but only one of the jails is under any real load
> 
> 


Do you use ZFS?


We have an internal software (some sort of monitoring agent) that also hangs in 
jails, from time to time.

The guy who wrote it found out it’s because of mmap (I don’t know the 
specifics).

The processes end up as unkillable in „D“ state and we need to reboot the hosts 
to fix it.

As the purpose of the hosts is not to run the agent, we usually let it hang and 
restart when it’s convenient.


The systems are FreeBSD 10.3, though (running nginx and varnish in different 
jails).

Re: WAF in HAProxy

[Rainer Duffner]

> Am 06.05.2016 um 00:15 schrieb Thierry FOURNIER 
> :
> 
> Hi,
> 
> You can look here:
> 
>   http://discourse.haproxy.org/t/ironbee-in-haproxy/92
> 
> Thierry
> 
> 



Is that project actually alive?
The last (and what looks like only) commit this year was to adjust the year for 
the copyright.
That in general is not really the most assuring sign for a healthy open source 
project.

Re: Linux or FreeBSD ?

[Rainer Duffner]

> Am 30.09.2015 um 16:25 schrieb Jeff Palmer :
> 
> Arnall,
> 
> 
> This advice is less of an haproxy specific response, and more of
> general information.
> 
> As someone who's tried to manage mixed infrastructure, I would push
> back if possible, unles syour organization has decided to move to
> freebsd entirely.
> 


Very few do that.
FreeBSD fulfills its purposes, though.
Even if you try to standardize on one „flavor“ of Linux, you will still end up 
with other flavors - simply because not everything runs on your particular 
flavor.
And you’re not going to run all of your applications on all of your platforms 
anyway. So the QA-effort should be manageable.
But that doesn’t mean it’s wise to introduce a half dozen different platforms, 
either - unless you have enough people to handle all of it.

How many systems (with Debian) are we talking about anyway?
And how many HA-Proxies are supposed to be migrated?

What are the sysadmin’s technical points for moving?
Besides probably not wanting to deal with Debian’s head-ache-inducing idea of 
an OS - that’s a given ;-)

Unless OP is doing some *really fancy stuff*, there’s IMO no pure technical 
show-stopper for a switch.

Re: Linux or FreeBSD ?

[Rainer Duffner]

> Am 01.10.2015 um 01:22 schrieb Willy Tarreau :
> 
>> 
> 
> I'd be tempted to place my judgement between yours and Jeff's. I'd say
> that if the company is already using the target OS on any other place,
> the cost of switching is low. If the load balancer is the opportunity
> to introduce a new OS, it's a bad idea. By nature a load balancer is
> very OS-dependant, and has bugs. Sometimes it's not trivial to tell
> if a bug is in haproxy or the underlying OS until you get network
> traces and/or strace output (BTW as far as I know, strace still doesn't
> support amd64 on FreeBSD). Mixing the two can cast a bad image on the
> new OS just because admins will initially not know well how to tune it
> for the load and to ensure stability, will not easily troubleshoot
> tricky issues, and a lot of frustration will result from this.
> 



Probably.
But OP’s admin will have his reasons for wanting FreeBSD in the picture.
My guess would be that FreeBSD is the OS he’s more familiar with debugging.
FreeBSD has ktrace - and dtrace (if you know how to use it, that is…)

Here, most of our LBs run HAproxy on FreeBSD.
Sometimes, they’re not. Because…reasons ;-)

Why?
Well, historically, most LBs and reverse-proxies ran FreeBSD (with NGINX).
So it was more or less a „natural“ choice, with some pushing from my side 
(cough).

FreeBSD has CARP.
Linux has keepalived.
etc.

I don’t think we’ll ever get so much traffic that either one will be superior 
to the other. And I seriously doubt OP will.

FreeBSD 10.1 has most of the optimizations that Netflix uses turned-on out of 
the box - but they do file-serving with NGINX.
In their (extreme) case, it works better.
Proxying/load-balancing is a bit different.

I like FreeBSD because I can get a very stable, simple, low overhead, 
no-nonsense OS with a reasonable shelf-live and update-cycle while still being 
able to get up-to-date packages directly from upstream.


> You should expect roughly the same performance on both OS so that is
> not a consideration for switching or not switching. Really keep in
> mind the admin cost, the cost of it being the exception in all your
> system and possibly different debugging tools. It's very likely that
> it will not be a problem, but better be aware of this.
> 


That’s what you get by hiring a FreeBSD guy.
If OP had hired a CentOS guy, I bet he'd want to switch everything to CentOS 
(or even Atomic Server…)
;-)

Re: Is FTP through haproxy at all viable?

[Rainer Duffner]

 
 I consider openssh for sftp pretty much unusable for clients/customers.


I wouldn’t say that.
Certainly true if they don’t actually know what they’re doing.

As for the setup: yes, the first directory users can write to in a chroot-setup 
is a subdirectory of the home directory (because $HOME needs to be owned by 
root).

But everything else is pretty simple. You don’t need any special devices or 
other stuff in the chroot itself. 
It basically just works in my experience.

If you want to chroot a full, interactive shell, though, you’re jumping into a 
world of pain…

Doesn’t have much to do with haproxy, though.

Personally, I’m not sure if load-balancing FTP is worth the effort.
Also, it looks like it’s quite „fragile“ and as such the load-balancing might 
break more often than a single-server without load-balancing.

Re: tcp-check for IMAP SSL ?

[Rainer Duffner]

 Am 01.01.2015 um 14:37 schrieb PiBa-NL piba.nl@gmail.com:
 
 Yosef Amir schreef op 1-1-2015 om 13:57:
 
 listen IMAP_SSL
 mode tcp
 bind :443 name VVM_SSL
 balance roundrobin
 tcp-check connect port 443
 Maybe try the 'ssl' keyword as below. (i have not tested it at all..)
 tcp-check connect port 443 ssl
 
 option tcp-check
 tcp-check expect string  ?
 server MIPS3 3.3.3.3 check
 server MIPS4 4.4.4.4 check
  



Hi,
Port 143 will actually be inline-TLS (STARTTLS).
SSL is on port 993.


The above answer should be correct, according to this:

http://comments.gmane.org/gmane.comp.web.haproxy/19274 
http://comments.gmane.org/gmane.comp.web.haproxy/19274

But only for SSL. Don’t know about inline-TLS.


Rainer

Re: 1.5.9 crashes every 4 hours, like clockwork

[Rainer Duffner]

 Am 11.12.2014 um 15:26 schrieb David Adams dr...@yahoo.com 
 mailto:dr...@yahoo.com:
 
 We are running 1.5.9 on Centos 6.5.  It crashes 10 seconds (give or take a 
 few seconds) after 1am, 5am, 9am, 1pm, 5pm and 9pm, like clockwork; let's 
 call that CRASHTIME.  Previously we'd been using 1.5.3 on the same hardware 
 for some months without crashes.  Once the crashes started we moved to 1.5.9 
 but they continue.  If we manually restart it a minute or two before 
 CRASHTIME it stills crashes when CRASHTIME arrives a minute or two later.


Interesting.
I’ve got a (single) VM where haproxy also crashes rather regularly (almost 
daily) at around 22:30-ish.

I though it was because of 1.4.20-something, but it didn’t stop when I upgraded 
to 1.5.x
Then, I thought it was FreeBSD 9 and upgraded to FreeBSD 10.
It’s now on 10.1 and still crashes.

Almost all my haproxy-VMs are actually provisioned with chef and are pretty 
similar and I’ve got this issue nowhere else.
I build the package myself on my own poudriere-server and the same package 
works elsewhere on much busier servers without problems.


We’ve got an icinga event-handler that restarts it…



Rainer

HAPROXY for IMAP, SMTP

[Rainer Duffner]

Hi,

we use HAPROXY for incoming mail, outgoing mail (authenticated), POP3, IMAP.

With incoming mail, I can make use of HAProxy’s send-proxy feature to make the 
source-IP known to the backend SMTP-servers.
(Works in the lab, I just need to move a few hundred customers off port 25 for 
authenticated SMTP, as send-proxy is incompatible with authentication (right?))

But what about authenticated SMTP connections (which go on Port 587 or 465)?.

We get a fair amount of abuse from hijacked accounts.
I need to know the original IP from these connections, too, so I can quickly 
see if it connects from China, Pakistan or whatever (our customers are 99.99% 
only connecting from domestic fix and dynamic IPs and authenticated connections 
from multiple IPs from multiple countries to the same account are 100% 
hijacked).
Same in principle for POP3 and IMAP.

Is there no other way other than running TPROXY mode (which I want to avoid and 
is AFAIK also not recommended)?

I have about 15k individual users.

As traffic is going to be almost 100% encrypted in the near future, I can't 
even run something like SNORT on the LB and just process the logs from that….



Have the patches from this thread:
http://marc.info/?t=13662203193r=1w=2

been incoporated into the HAproxy 1.5 source tree since then?

haproxy sending RSTs to backend-servers

[Rainer Duffner]

Hi,

I’ve configured nginx+haproxy in front of a couple of IIS servers.
NGINX terminates SSL.

configuration is as following:

global
  log /var/run/log   local5
  log /var/run/log   local1 notice
  #log loghostlocal0 info
  maxconn 4096
  #debug
  #quiet
  user www
  group www
  daemon

defaults
  log global
  modehttp
  retries 2
  timeout client 50s
  timeout connect 5s
  timeout server 50s
  option dontlognull
  option forwardfor
  option httplog
  option redispatch
  balance  leastconn
  http-check expect string server_up
  http-check disable-on-404
  default-server minconn 50 maxconn 100 

# Set up application listeners here.

frontend app-main-prod
  mode http
  bind 0.0.0.0:8000
  maxconn 2000
  default_backend app-main-prod-back

frontend app-import
  mode http
  bind 0.0.0.0:8001
  maxconn 2000
  default_backend app-import-back

frontend app-images
  mode http
  bind 0.0.0.0:8002
  maxconn 2000
  default_backend app-images-back


backend app-main-prod-back
  balance leastconn
  fullconn 2000
  mode http
  option httpchk GET /healthcheck.aspx HTTP/1.1\r\nHost:\
www.app.ch\r\nConnection:\ close cookie SERVERID insert indirect nocache
  server appsrv-one  192.168.69.17:80 weight 1 maxconn 1000 check
cookie s1 server appsrv-two  192.168.69.18:80 weight 1 maxconn 1000
check cookie s2

backend app-import-back
  balance leastconn
  fullconn 2000
  mode http
  #option httpchk GET /healthcheck.aspx HTTP/1.1\r\nHost:\
import.app.ch\r\nConnection:\ close server appsrv-import-one
192.168.69.32:80 weight 1 maxconn 1000 check #server appsrv-import-two
192.168.69.33:80 weight 1 maxconn 1000 check

backend app-images-back
  balance leastconn
  fullconn 2000
  mode http
  option httpchk GET /healthcheck.aspx HTTP/1.1\r\nHost:\
images.app.ch\r\nConnection:\ close server appsrv-images-one
192.168.69.41:80 weight 1 maxconn 1000 check #server appsrv-images-two
192.168.69.42:80 weight 1 maxconn 1000 check


listen admin 0.0.0.0:22002
  mode http
  stats uri /



What happens is that it will mostly work, but in wireshark, I see a lot
of RST being sent from the haproxy-server to the backend IIS-servers.
This doesn’t make sense and is probably the reason I see so many 50x in
the logs and why occasionally gateway-errors are being shown to users
because nginx can’t find any live servers…

Can anyone see any obvious error in the config?

Is it possible to query the query the status of a server and use it in an ACL?

[Rainer Duffner]

Hi,

I want to take the status of a server of a given backend and use it in
another backend or in the frontend.
If that possible?
I though there might be something simular to
nbsrv() - but I haven't found anything.




Best Regards
Rainer

Can you balance-out service-checks better?

[Rainer Duffner]

Hi,


we will put haproxy in front of a Zimbra infrastructure (which we have 
split-up, so that there is a „front end“, with pop, imap, smtp and a „back 
end“, where the mail sits).

I have too haproxy-servers (active/standby via CARP) that are checking the 
front-ends.
I check:
 - smtp
 - smtps
 - submit
 - pop + pops
 - imap + imaps

from both haproxy-servers simultaneously.

If I use the default check frequency, it just bombards the servers with 
requests that often can’t even finish in the time it takes to launch the next 
check.
If I increase the check-frequency too much, it will take longer to take a 
server out of the pool in case of failure - and checks still don’t 
„balance-out“ (or do they?).
But they are all more or less connected: if one of them fails, it’s highly 
likely that all the others will fail, too.

So, ideally, I’d like to have something like this:
 - check service A (maybe POP3)
 - wait maybe 30s
 - than check the next service (e.g. POP3S).
 - if one fails, remove that backend-server from the pool for all services
 - alternatively, instead of doing the above, re-schedule the checks so the 
next check happens immediately 


Does that sound insane?
;-)

Re: Can you balance-out service-checks better?

[Rainer Duffner]

Am 28.08.2014 um 22:41 schrieb Baptiste bed...@gmail.com:
 
 
 Hi,
 
 maybe you could share your HAProxy configuration :)
 By default, HAProxy tests a service every 3s, which is fine. It just
 does a tcp connect, so nothing complicated for your server to handle.
 


Since we switched to haproxy-1.5, I changed the checks to do a more or less 
full layer7-check (except for the SSL-services).
Couldn’t get a match for the IMAP string it sends, so skipped that, too.



 Can you confirm that if POP fails on a server, it means that IMAP and
 SMTP will fail too?
 (this is what I'm understanding from your mail above).
 

It’s very likely.
All use the same backend-service in the end.
There’s an additional pair of SMTP-servers here (ep01+ep02) - they are 
independent of the other two servers (pm01+pm02).
But I’d also like to limit checking there, as of course all the checks for 
smtp, smtps+submit all go to the same postfix in the end….


Here’s the config.


global
  log 127.0.0.1   local0
  log 127.0.0.1   local1 notice
  #log loghostlocal0 info
  maxconn 4096
  #debug
  #quiet
  user www
  group www
  daemon

defaults
  log global
  modehttp
  retries 2
  timeout client 50s
  timeout connect 5s
  timeout server 50s
  option dontlognull
  option forwardfor
  option httplog
  option redispatch
  balance  roundrobin
  default-server minconn 50 maxconn 100 

# Set up application listeners here.

frontend pop3-pm
  mode tcp
  bind 192.168.185.254:110
  maxconn 2000
  default_backend pop3-pm-backend

frontend imap4-pm
  mode tcp
  bind 192.168.185.254:143
  maxconn 2000
  default_backend imap4-pm-backend

frontend pop3s-pm
  mode tcp
  bind 192.168.185.254:995
  maxconn 2000
  default_backend pop3s-pm-backend

frontend imap4s-pm
  mode tcp
  bind 192.168.185.254:993
  maxconn 2000
  default_backend imap4s-pm-backend

frontend smtp-ep
  mode tcp
  bind 192.168.185.254:25
  maxconn 2000
  default_backend smtp-ep-backend

frontend smtps-ep
  mode tcp
  bind 192.168.185.254:465
  maxconn 2000
  default_backend smtps-ep-backend

frontend submit-ep
  mode tcp
  bind 192.168.185.254:587
  maxconn 2000
  default_backend submit-ep-backend

frontend smtp-zimbra
  mode tcp
  bind 192.168.185.253:25
  maxconn 2000
  default_backend smtp-zimbra-backend

frontend http-webmail
  bind 192.168.185.254:5000
  maxconn 6000
  default_backend http-webmail-backend
#
#
#

backend pop3-pm-backend
  balance roundrobin
  mode tcp
  option tcp-check
  tcp-check expect string +OK\ POP3\ ready
  tcp-check send quit\r\n
  tcp-check expect string +OK
  server pm01  192.168.185.233:110 check inter 30s fastinter 2s downinter 2s
  server pm02  192.168.185.234:110 check inter 30s fastinter 2s downinter 2s

backend pop3s-pm-backend
  balance roundrobin
  mode tcp
#  this is ssl, so it does not work here
#  option tcp-check
#  tcp-check expect string +OK\ POP3\ ready
  server pm01  192.168.185.233:995 check inter 30s fastinter 2s downinter 2s
  server pm02  192.168.185.234:995 check inter 30s fastinter 2s downinter 2s

backend imap4-pm-backend
  balance roundrobin
  mode tcp
  option tcp-check
  tcp-check expect rstring OK\ IMAP4\ ready
  tcp-check send 001 logout\r\n
  #tcp-check expect string *\ BYE\ Zimbra\ IMAP\ server\ terminating\ 
connection\r\n001\ OK\ completed
  server pm01  192.168.185.233:143 check inter 30s fastinter 2s downinter 2s
  server pm02  192.168.185.234:143 check inter 30s fastinter 2s downinter 2s

backend imap4s-pm-backend
  balance roundrobin
  mode tcp
  server pm01  192.168.185.233:993 check inter 30s fastinter 2s downinter 2s
  server pm02  192.168.185.234:993 check inter 30s fastinter 2s downinter 2s

backend smtp-ep-backend
  balance roundrobin
  mode tcp
  option smtpchk HELO mail.this.here
  server ep01 192.168.185.198:25 check inter 45s fastinter 2s downinter 2s 
  server ep02 192.168.185.199:25 check inter 45s fastinter 2s downinter 2s

backend smtps-ep-backend
  balance roundrobin
  mode tcp
  #option smtpchk HELO mail.this.here
  server ep01 192.168.185.198:465 check inter 45s fastinter 2s downinter 2s
  server ep02 192.168.185.199:465 check inter 45s fastinter 2s downinter 2s

backend submit-ep-backend
  balance roundrobin
  mode tcp
  option smtpchk HELO mail.scalera.ch
  server ep01 192.168.185.198:587 check inter 45s fastinter 2s downinter 2s
  server ep02 192.168.185.199:587 check inter 45s fastinter 2s downinter 2s

backend smtp-zimbra-backend
  balance roundrobin
  mode tcp
  option smtpchk HELO mail.this.here
  server pm01 192.168.185.233:25 check inter 45s fastinter 2s downinter 2s
  server pm02 192.168.185.234:25 check inter 45s fastinter 2s downinter 2s

backend http-webmail-backend
  balance leastconn
  mode http
  option httpchk GET /
  #http-check expect string Webmail Login Page
  http-check expect string Web Client Login Page
  http-check disable-on-404
  cookie SERVERID insert indirect nocache
  server pm1 192.168.185.233:80 check  maxconn 3000 cookie s1 inter 10s 
fastinter 2s downinter 

Re: Can you balance-out service-checks better?

[Rainer Duffner]

Am 28.08.2014 um 23:21 schrieb Baptiste bed...@gmail.com:

 Ok,
 I would create a monitoring backend, such as below:


Hey, thanks a lot!

I will try this and report back.



Best Regards,
Rainer

Re: Load balancing FTP with HAProxy behind a firewall

[Rainer Duffner]

 hdr(host) ACL only applies to HTTP.
 Furthermore, I'm not sure there is a notion of Host header in FTP ;)


Last time I looked (admittedly with 1.4) into FTP+HAProxy, the
end-result was that it was just not possible.

AFAIK, you can use LVS for that on Linux.

Re: haproxy dumps core

[Rainer Duffner]

Am 30.07.2013 um 21:40 schrieb Lukas Tribus luky...@hotmail.com:

 Hi Rainer!
 
 
 I'm using haproxy on FreeBSD 9.1-amd64 inside a VMware VM.
 
 I realized that when I have a situation where all servers in a backend
 are down, haproxy crashes:
 Jul 30 08:03:52 px2-bla kernel: pid 58816 (haproxy), uid 80:
 exited on signal 11 (core dumped)
 
 pkg info|grep haproxy
 haproxy-1.4.24 The Reliable, High Performance
 
 can you post the output of haproxy -vv?
 
 
 
 After some tinkering, I got a core-dump out of it:
 
 The core-dump doesn't look very useful, seems like the debugging symbols
 where stripped.
 



Hi,

sorry, I haven't had time to look into this, but now I've been able to generate 
a core (and run it through gdb)


gdb /usr/local/sbin/haproxy haproxy.3272
   
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type show copying to see the conditions.
There is absolutely no warranty for GDB.  Type show warranty for details.
This GDB was configured as amd64-marcel-freebsd...
Core was generated by `haproxy'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/libcrypt.so.5...done.
Loaded symbols for /lib/libcrypt.so.5
Reading symbols from /usr/local/lib/libpcreposix.so.0...done.
Loaded symbols for /usr/local/lib/libpcreposix.so.0
Reading symbols from /usr/local/lib/libpcre.so.3...done.
Loaded symbols for /usr/local/lib/libpcre.so.3
Reading symbols from /lib/libc.so.7...done.
Loaded symbols for /lib/libc.so.7
Reading symbols from /lib/libthr.so.3...done.
Loaded symbols for /lib/libthr.so.3
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x0043d0e9 in process_session (t=0x801866f00) at src/session.c:1434
1434src/session.c: No such file or directory.
in src/session.c
[New Thread 801807400 (LWP 100105/unknown)]
[New LWP 100114]
(gdb) bt
#0  0x0043d0e9 in process_session (t=0x801866f00) at src/session.c:1434
#1  0x00408420 in process_runnable_tasks (next=0x7fffdafc) at 
src/task.c:234
#2  0x004028e3 in run_poll_loop () at src/haproxy.c:1002
#3  0x0040455d in main (argc=value optimized out, 
argv=0x7fffdba0) at src/haproxy.c:1288


Can you make something of this?


I found it may be a config-file problem.
Apart from comments, the only difference between a config-file that makes 
haproxy dump core and one that doesn't is:

   maxconn 500
   server server1  ip:80 weight 1 check
---
   maxconn 500 server server1  ip:80 weight 1 check



Best Regards
Rainer

haproxy dumps core

[Rainer Duffner]

Hi,

I'm using haproxy on FreeBSD 9.1-amd64 inside a VMware VM.

I realized that when I have a situation where all servers in a backend
are down, haproxy crashes:
Jul 30 08:03:52 px2-bla kernel: pid 58816 (haproxy), uid 80:
exited on signal 11 (core dumped)

pkg info|grep haproxy
haproxy-1.4.24 The Reliable, High Performance
TCP/HTTP Load Balancer 
# ldd /usr/local/sbin/haproxy
/usr/local/sbin/haproxy: libcrypt.so.5 = /lib/libcrypt.so.5
(0x8008c7000) libc.so.7 = /lib/libc.so.7 (0x800ae6000)

I've got the following options:

cat /usr/local/etc/poudriere.d/91amd64-options/net_haproxy/options 
# This file is auto-generated by 'make config'.
# Options for haproxy-1.4.24
_OPTIONS_READ=haproxy-1.4.24
_FILE_COMPLETE_OPTIONS_LIST=PCRE DPCRE SPCRE
OPTIONS_FILE_SET+=PCRE
OPTIONS_FILE_UNSET+=DPCRE
OPTIONS_FILE_SET+=SPCRE

After some tinkering, I got a core-dump out of it:

(px2-bla /root) 0 #
gdb /usr/local/sbin/haproxy /var/tmp/haproxy.58816
GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software
Foundation, Inc. GDB is free software, covered by the GNU General
Public License, and you are welcome to change it and/or distribute
copies of it under certain conditions. Type show copying to see the
conditions. There is absolutely no warranty for GDB.  Type show
warranty for details. This GDB was configured as
amd64-marcel-freebsd...(no debugging symbols found)... Core was
generated by `haproxy'. Program terminated with signal 11, Segmentation
fault. Reading symbols from /lib/libcrypt.so.5...(no debugging symbols
found)...done. Loaded symbols for /lib/libcrypt.so.5
Reading symbols from /lib/libc.so.7...(no debugging symbols
found)...done. Loaded symbols for /lib/libc.so.7
Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols
found)...done. Loaded symbols for /libexec/ld-elf.so.1
#0  0x0043be27 in ?? ()
(gdb) bt
#0  0x0043be27 in ?? ()
#1  0x004087e1 in ?? ()
#2  0x00402c01 in ?? ()
#3  0x00404607 in ?? ()
#4  0x00402ade in ?? ()
#5  0x0008006c9000 in ?? ()
#6  0x in ?? ()
#7  0x in ?? ()
#8  0x0006 in ?? ()
#9  0x7fffdde8 in ?? ()
#10 0x7fffde00 in ?? ()
#11 0x7fffde03 in ?? ()
#12 0x7fffde06 in ?? ()
#13 0x7fffde21 in ?? ()
#14 0x7fffde24 in ?? ()
#15 0x in ?? ()
#16 0x7fffde39 in ?? ()
#17 0x7fffde47 in ?? ()
#18 0x7fffde4f in ?? ()
#19 0x7fffde63 in ?? ()
#20 0x7fffdeba in ?? ()
#21 0x7fffdec7 in ?? ()
#22 0x7fffded1 in ?? ()
#23 0x7fffdeef in ?? ()
#24 0x7fffdefa in ?? ()
#25 0x7fffdf04 in ?? ()
#26 0x7fffdf0f in ?? ()
#27 0x7fffdf20 in ?? ()
#28 0x7fffdf39 in ?? ()
#29 0x7fffdf4c in ?? ()
#30 0x7fffdf59 in ?? ()
#31 0x7fffdf65 in ?? ()
#32 0x in ?? ()
#33 0x0003 in ?? ()
#34 0x00400040 in ?? ()
#35 0x0004 in ?? ()
#36 0x0038 in ?? ()
#37 0x0005 in ?? ()
#38 0x0008 in ?? ()
#39 0x0006 in ?? ()
#40 0x1000 in ?? ()
#41 0x0008 in ?? ()
#42 0x in ?? ()
#43 0x0009 in ?? ()
#44 0x00402a50 in ?? ()
#45 0x0007 in ?? ()
#46 0x0008006ae000 in ?? ()
#47 0x000f in ?? ()
#48 signal handler called
#49 0x in ?? ()
Previous frame inner to this frame (corrupt stack?)


I'd like to know what is causing this.


Config is like this:

global
  log 127.0.0.1   local0
  log 127.0.0.1   local1 notice
  #log loghostlocal0 info
  maxconn 4096
  #debug
  #quiet
  user www
  group www
  daemon

defaults
  log global
  modehttp
  retries 2
  timeout client 50s
  timeout connect 5s
  timeout server 50s
  option dontlognull
  option forwardfor
  option httplog
  option redispatch
  balance  source
  option httpchk GET /ipmon.txt HTTP/1.0\r\n\r\n
  http-check expect rstring OK
  http-check disable-on-404
  http-send-name-header X-Target-Server
  default-server minconn 50 maxconn 100 

# Set up application listeners here.

frontend s
  maxconn 8000
  bind 0.0.0.0:8000
  default_backend servers-old-s
  reqidel ^X-Forwarded-For:.*

frontend s-stage
  maxconn 8000
  bind 0.0.0.0:8002
  default_backend servers-old-s-stage
  reqidel ^X-Forwarded-For:.*

frontend p
  maxconn 8000
  bind 0.0.0.0:8004
  default_backend servers-old-p
  reqidel ^X-Forwarded-For:.*

frontend p-stage
  maxconn 8000
  bind 0.0.0.0:8006
  default_backend servers-old-p-stage
  reqidel ^X-Forwarded-For:.*

frontend d-old
  maxconn 8000
  bind 0.0.0.0:8008
  default_backend servers-old-d
  reqidel ^X-Forwarded-For:.*



backend servers-old-d
  fullconn 8000
  #option httpchk GET /ip_monitor_mysql.php HTTP/1.1\r\nHost:
www.d.domain\r\nConnection:\ close server app2   first.ip:80 weight 1
check server input1 second.ip:80 weight 1 check

backend servers-old-s
  fullconn 

Re: haproxy dumps core

[Rainer Duffner]

Am Tue, 30 Jul 2013 21:40:34 +0200
schrieb Lukas Tribus luky...@hotmail.com:

 Hi Rainer!
 
 
  I'm using haproxy on FreeBSD 9.1-amd64 inside a VMware VM.
 
  I realized that when I have a situation where all servers in a
  backend are down, haproxy crashes:
  Jul 30 08:03:52 px2-bla kernel: pid 58816 (haproxy), uid 80:
  exited on signal 11 (core dumped)
 
  pkg info|grep haproxy
  haproxy-1.4.24 The Reliable, High Performance
 
 can you post the output of haproxy -vv?


(px2-bla /root) 0 # haproxy -vv
HA-Proxy version 1.4.24 2013/06/17
Copyright 2000-2013 Willy Tarreau w...@1wt.eu

Build options :
  TARGET  = freebsd
  CPU = generic
  CC  = cc
  CFLAGS  = -O2 -pipe -fno-strict-aliasing -DFREEBSD_PORTS
  OPTIONS = USE_STATIC_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents =
200

Encrypted password support via crypt(3): yes

Available polling systems :
 kqueue : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use kqueue.

 

 
  After some tinkering, I got a core-dump out of it:
 
 The core-dump doesn't look very useful, seems like the debugging
 symbols where stripped.
 
 
 Could you recompile haproxy with the following CFLAGS:
  make CFLAGS=-g -O0 TARGET=[...]
 
 and regenerate the core-dump. The GDB output should be more
 informative then.
 
 If the executable comes from a packaging system (ports?), you may be
 able to use a debug-package instead of recompiling haproxy (although
 compiler optimization may obfuscate the backtrace).


I'll look into it. It's created by our poudriere package-building
system.



Regards,
Rainer