Re: socket bind error

2015-05-20 Thread Nathan Williams
arg. ok, it was SELinux... we recently re-worked how we prepare our base
image and the new method seems to leave SELinux enabled... turned that off
and everything's working peachy.

Thanks!

On Wed, May 20, 2015 at 4:16 PM Lukas Tribus luky...@hotmail.com wrote:

  hi all,
 
  I'm working on standing up a new haproxy instance to manage redis
  directly on our redis hosts since our main load-balancer does periodic
  reloads and restarts for things like OCSP stapling that good ol'
  amnesiac HTTP handles just fine, but longer-lived TCP connections like
  our redis clients don't care too much for.
 
  I managed to put together a configuration that works fine in local
  testing (vagrant configured by test-kitchen), but for some reason when
  I try to push this to staging, haproxy is refusing to start,
  complaining that it can't bind to the keepalived-managed VIP. For the
  life of me I can't figure out what the problem is, but hopefully
  someone here will be able to give me some pointers?

 Not sure, can you run haproxy directly (without systemd) through strace,
 to see what exactly the kernel returns?

 Whats the kernel release anyway?

 What happens if you add the transparent keyword on the bind
 configuration line (so that the sysctl setting is not needed)?



 Regards,

 Lukas




socket bind error

2015-05-20 Thread Nathan Williams
hi all,

I'm working on standing up a new haproxy instance to manage redis directly
on our redis hosts since our main load-balancer does periodic reloads and
restarts for things like OCSP stapling that good ol' amnesiac HTTP handles
just fine, but longer-lived TCP connections like our redis clients don't
care too much for.

I managed to put together a configuration that works fine in local testing
(vagrant configured by test-kitchen), but for some reason when I try to
push this to staging, haproxy is refusing to start, complaining that it
can't bind to the keepalived-managed VIP. For the life of me I can't figure
out what the problem is, but hopefully someone here will be able to give me
some pointers? Thanks in advance for your help :)

The error message:

```bash
[root@redis02.stage ~]# journalctl -ln5 -u haproxy.service --no-pager
-- Logs begin at Wed 2015-05-20 22:35:37 UTC, end at Wed 2015-05-20
22:45:55 UTC. --
May 20 22:35:47 redis02.stage.iad01.treehouse systemd[1]: Starting HAProxy
Load Balancer...
May 20 22:35:47 redis02.stage.iad01.treehouse systemd[1]: Started HAProxy
Load Balancer.
May 20 22:35:47 redis02.stage.iad01.treehouse haproxy-systemd-wrapper[794]:
haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
May 20 22:35:47 redis02.stage.iad01.treehouse haproxy-systemd-wrapper[794]:
[ALERT] 139/223547 (801) : Starting proxy redis: cannot bind socket [
10.240.36.71:6379]
May 20 22:35:47 redis02.stage.iad01.treehouse haproxy-systemd-wrapper[794]:
haproxy-systemd-wrapper: exit, haproxy RC=256
```

version info:

```bash
[root@redis02.stage ~]# haproxy -vvv
HA-Proxy version 1.5.4 2014/09/02
Copyright 2000-2014 Willy Tarreau w...@1wt.eu

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing
  OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1
USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.
```

the configuration:

```bash
[root@redis02.stage ~]# cat /etc/haproxy/haproxy.cfg
# Generated by Chef
# Changes will be overwritten!
global
  user haproxy
  group haproxy
  stats socket /var/lib/haproxy/stats.sock
  log /dev/log local0 info
  maxconn 5

defaults TCP
  mode tcp
  log global
  option tcplog
  option tcpka
  source 10.240.36.71

listen redis
  bind 10.240.36.71:6379
  default-server on-marked-down shutdown-sessions
  option tcp-check
  tcp-check send PING\r\n
  tcp-check expect string +PONG
  tcp-check send info\ replication\r\n
  tcp-check expect string role:master
  tcp-check send QUIT\r\n
  tcp-check expect string +OK
  server redis01.stage 10.240.36.27:6379 backup check inter 1000 rise 2
fall 5
  server redis02.stage 10.240.36.63:6379 backup check inter 1000 rise 2
fall 5
```

listening services:

```bash
[root@redis02.stage ~]# netstat -lptn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address   Foreign Address State
PID/Program name
tcp0  0 0.0.0.0:26379   0.0.0.0:*   LISTEN
 2449/redis-sentinel
tcp0  0 10.240.36.63:6379   0.0.0.0:*   LISTEN
 2388/redis-server 1
tcp0  0 127.0.0.1:3030  0.0.0.0:*   LISTEN
 930/ruby
tcp0  0 0.0.0.0:22  0.0.0.0:*   LISTEN
 782/sshd
tcp0  0 127.0.0.1:250.0.0.0:*   LISTEN
 919/master
tcp0  0 127.0.0.1:2812  0.0.0.0:*   LISTEN
 784/monit
tcp6   0  0 :::26379:::*LISTEN
 2449/redis-sentinel
tcp6   0  0 :::22   :::*LISTEN
 782/sshd
tcp6   0  0 ::1:25  :::*LISTEN
 919/master
```

local addresses:

```bash
[root@redis02.stage ~]# ip a
1: lo: LOOPBACK,UP,LOWER_UP mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
   valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
   valid_lft forever preferred_lft forever
2: eth0: 

RE: socket bind error

2015-05-20 Thread Lukas Tribus
 hi all, 
  
 I'm working on standing up a new haproxy instance to manage redis  
 directly on our redis hosts since our main load-balancer does periodic  
 reloads and restarts for things like OCSP stapling that good ol'  
 amnesiac HTTP handles just fine, but longer-lived TCP connections like  
 our redis clients don't care too much for. 
  
 I managed to put together a configuration that works fine in local  
 testing (vagrant configured by test-kitchen), but for some reason when  
 I try to push this to staging, haproxy is refusing to start,  
 complaining that it can't bind to the keepalived-managed VIP. For the  
 life of me I can't figure out what the problem is, but hopefully  
 someone here will be able to give me some pointers?

Not sure, can you run haproxy directly (without systemd) through strace,
to see what exactly the kernel returns?

Whats the kernel release anyway?

What happens if you add the transparent keyword on the bind
configuration line (so that the sysctl setting is not needed)?



Regards,

Lukas