Yuan,
maybe something here http://lnxmon.com/haproxy/
> Thanks,
> ; Yuan
I modified a config from your blog that you showed me and came up with this:
global
log 127.0.0.1 local0 notice
maxconn 2000
user haproxy
group haproxy
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
timeout connect 5000
timeout client 10000
timeout server 10000
listen jokefire 0.0.0.0:80
mode http
stats enable
stats uri /haproxy?stats
stats realm Strictly\ Private
stats auth admin:secret
balance roundrobin
option httpclose
option forwardfor
server varnish1 10.10.10.5:80 check
server varnish2 10.10.10.6:80 check
listen mysql-cluster
bind 0.0.0.0:3306
mode tcp
balance roundrobin
maxconn 5200
option mysql-check user haproxy_root
server mysql-1 10.10.10.7:3306 check
server mysql-2 10.10.10.8:3306 check
And that seemed to work. I can see that both ports are listening now:
[root@ha1:/etc/haproxy] #lsof -i :80 -i :3306
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
haproxy 27136 haproxy 4u IPv4 7563913 0t0 TCP *:http (LISTEN)
haproxy 27136 haproxy 6u IPv4 7563915 0t0 TCP *:mysql (LISTEN)
Although I am not aware of the real difference between this and my previous
config that allows this to work is.
Not a huge issue at this point since it's working. But if anyone wants to
take a stab at this, be my guest!
Thanks,
Tim
On Sat, Jul 25, 2015 at 12:15 AM, Gmail <[email protected]> wrote:
> maybe something here http://lnxmon.com/haproxy/
>
> Thanks,
> ; Yuan
>
>
> On 07/25/2015 12:10 PM, Igor Cicimov wrote:
>
>> You need to run haproxy as root to bind to ports lower than 1024
>> On 25/07/2015 1:36 PM, "Tim Dunphy" <[email protected]> wrote:
>>
>> Hi Yuan,
>>>
>>> Nice.
>>>
>>>> Do you use selinux in prod.
>>>> regards,
>>>> ; Yuan
>>>>
>>>
>>> Yep! Actually I use it every chance I get. Prod/stage/dev and my own
>>> hobby
>>> environments. And right now actually what I was discussing was a hobby
>>> environment.
>>>
>>> And actually if I could bother you guys one more time, I do have one more
>>> issue to solve. LOL
>>>
>>> And this time it's guaranteed not to be an SELinux issue. Because I tried
>>> running haproxy with SELInux on and off this time.
>>>
>>> But what's happening now, is that HA/Proxy is not creating the http port
>>> for the 'stats' interface. I've setup stats to listen on port 80. But for
>>> some reason that's not happening.
>>>
>>> Here's my config one more time, with the trouble part in bold:
>>>
>>> global
>>> log 127.0.0.1 local0 notice
>>> user haproxy
>>> group haproxy
>>>
>>> defaults
>>> log global
>>> retries 2
>>> timeout connect 3000
>>> timeout server 5000
>>> timeout client 5000
>>>
>>> listen mysql-cluster
>>> bind 0.0.0.0:3306
>>> mode tcp
>>> option mysql-check user haproxy_check
>>> balance roundrobin
>>> server mysql-1 52.3.28.48:3306 check
>>> server mysql-2 52.2.0.176:3306 check
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *listen 0.0.0.0:80 <http://0.0.0.0:80> mode http stats enable
>>> stats uri / stats realm Strictly\ Private stats auth admin:secret*
>>> Currently haproxy is listening on the first port specified* - 3306 - *but
>>> not listening on port 80.
>>>
>>> Observe:
>>>
>>> [root@ha1:/etc/haproxy] #lsof -i :3306
>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>>> *haproxy 11653 haproxy 4u IPv4 7145270 0t0 TCP *:mysql
>>> (LISTEN)*
>>>
>>>
>>> [root@ha1:/etc/haproxy] #lsof -i :80
>>> [root@ha1:/etc/haproxy] #
>>>
>>> [root@ha1:/etc/haproxy] #telnet localhost 80
>>> Trying 127.0.0.1...
>>> telnet: connect to address 127.0.0.1: Connection refused
>>>
>>> Port 80 simply isn't listening.
>>>
>>> And this time, I can't blame it on SELinux being on:
>>>
>>> [root@ha1:/etc/haproxy] #getenforce
>>> Permissive
>>>
>>> I've grepped thru /var/log/messages but not turned up any clues to this
>>> one.
>>>
>>> And I really would like to get the stats interface up and running.
>>>
>>> Any thoughts here? I'm wondering what I can do to get stats working.
>>>
>>> Thanks,
>>> Tim
>>>
>>>
>>>
>>> On Fri, Jul 24, 2015 at 10:52 PM, Gmail <[email protected]> wrote:
>>>
>>> Nice.
>>>> Do you use selinux in prod.
>>>> regards,
>>>> ; Yuan
>>>>
>>>> On 07/25/2015 09:17 AM, Tim Dunphy wrote:
>>>>
>>>> Bingo!!!
>>>>>
>>>>> The problem was with SELinux. Not sure what took me so long to think of
>>>>> it...!!!
>>>>>
>>>>> So set the mysql listener back to port 3306. Turned off SELinux with
>>>>> setenforce 0. Then it started right up!!! And port 3306 was listening.
>>>>>
>>>>> Then I consulted with audit2why and saw the following:
>>>>>
>>>>> type=AVC msg=audit(1437786617.963:28856863): avc: denied {
>>>>> name_connect }
>>>>> for pid=29175 comm="haproxy" dest=3306
>>>>> scontext=system_u:system_r:haproxy_t:s0
>>>>> tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket
>>>>>
>>>>> Was caused by:
>>>>> The boolean haproxy_connect_any was set incorrectly.
>>>>> Description:
>>>>> Allow haproxy to connect any
>>>>>
>>>>> Allow access by executing:
>>>>> # *setsebool -P haproxy_connect_any 1*
>>>>>
>>>>>
>>>>> I just ran that command you see above in bold, and then all was right
>>>>> with
>>>>> the world.
>>>>>
>>>>> [root@ha1:/etc/haproxy] #systemctl status haproxy
>>>>> haproxy.service - HAProxy Load Balancer
>>>>> Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)
>>>>> Active: active (running) since Sat 2015-07-25 01:14:53 UTC; 33s
>>>>> ago
>>>>> Main PID: 30618 (haproxy-systemd)
>>>>> CGroup: /system.slice/haproxy.service
>>>>> ├─30618 /usr/sbin/haproxy-systemd-wrapper -f
>>>>> /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
>>>>> ├─30619 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
>>>>> /run/haproxy.pid -Ds
>>>>> └─30620 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
>>>>> /run/haproxy.pid -Ds
>>>>>
>>>>> Jul 25 01:14:53 ha1 systemd[1]: Starting HAProxy Load Balancer...
>>>>> Jul 25 01:14:53 ha1 systemd[1]: Started HAProxy Load Balancer.
>>>>> Jul 25 01:14:53 ha1 haproxy-systemd-wrapper[30618]:
>>>>> haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f
>>>>> /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
>>>>>
>>>>> [root@ha1:/etc/haproxy] #lsof -i :3306
>>>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>>>>> haproxy 30620 haproxy 1u IPv4 7075172 0t0 TCP
>>>>> ha1.example.com:55499->ec2-52-2-0-xxx.compute-1.amazonaws.com:mysql
>>>>> (SYN_SENT)
>>>>> haproxy 30620 haproxy 4u IPv4 7074731 0t0 TCP *:mysql
>>>>> (LISTEN)
>>>>>
>>>>>
>>>>> Thanks for nudging me in the right direction. All I had to hear was the
>>>>> word 'selinux' and from there it all fell into place!
>>>>>
>>>>> Thanks!!
>>>>> Tim
>>>>>
>>>>> On Fri, Jul 24, 2015 at 8:20 PM, Gmail <[email protected]> wrote:
>>>>>
>>>>> I could be completely wrong here and I am curious to know the answer
>>>>>
>>>>>> myself. Please don't take this as a solution, just my thoughts.
>>>>>>
>>>>>> First, you can not use backend ip-address of 10.x.x.x subnet because
>>>>>> each
>>>>>> account's VPC is seggregated. If you do want to use 10.X.X.X ipadress
>>>>>> you
>>>>>> have to setup a inter VPC endpoint in AWS. I would just use EIP.
>>>>>>
>>>>>> For the port 3306, try to use nc to listen on that port or iperf. Do
>>>>>> yo
>>>>>> uhave iptables turned on.
>>>>>>
>>>>>> I would check "systemctl -l status haproxy.service"
>>>>>>
>>>>>> I would check lsof -i why can't bind to 3306 on loopback ipaddress.
>>>>>>
>>>>>> I would check iptables or selinux preventing the bind.
>>>>>>
>>>>>> It wil be interesting to know the source ipaddress of MySQL client ec2
>>>>>> instance.
>>>>>> Interesting if you can Copy/paste output of "telnet
>>>>>> <haproxynode_ipaddress> 3306" from mysql client ec2 instance , here.
>>>>>> Interesting if you can Copy/paste output of "telnet 10.10.10.10 3306"
>>>>>> from haproxy ec2 instances, here.
>>>>>> Interesting if you can Copy/paste output of "telnet 10.10.10.11 3306"
>>>>>> from haproxy ec2 instances, here.
>>>>>>
>>>>>> I I was doing this, maybe I would consider testing something like ;
>>>>>> ..
>>>>>> frontend mysql_lb_fe 0.0.0.0:3306
>>>>>> ....
>>>>>> acl host_myql_lb hdr(host) -i mysql-lb
>>>>>> ..
>>>>>> ..
>>>>>> use_backend mysql_lb_backend if host mysql_lb
>>>>>> ..
>>>>>> ..
>>>>>> backend mysql_lb_be
>>>>>> ..
>>>>>> ..
>>>>>>
>>>>>> option mysql-check user haproxy_check
>>>>>> balance roundrobin
>>>>>> server mysql-1 10.10.10.10:3306 check
>>>>>> server mysql-2 10.10.10.11:3306 check
>>>>>>
>>>>>> Thanks,
>>>>>> ; Yuan
>>>>>>
>>>>>>
>>>>>> On 07/25/2015 06:41 AM, Tim Dunphy wrote:
>>>>>>
>>>>>> Hello Nenad,
>>>>>>
>>>>>>> Jul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: [ALERT]
>>>>>>>
>>>>>>> 204/034418 (25035) : *Starting proxy mysql-cluster:
>>>>>>> cannot
>>>>>>>
>>>>>>>> bind
>>>>>>>> s...:3306]*
>>>>>>>>
>>>>>>>> Nothing listening on the port I'm trying to bind to: 3306
>>>>>>>>
>>>>>>> [root@ha1:~] #ss -lpt | fgrep 3306
>>>>>>> [root@ha1:~] #lsof -i :3306
>>>>>>> [root@ha1:~] #netstat -tulpn | grep -i listen | grep 3306
>>>>>>> [root@ha1:~] #
>>>>>>>
>>>>>>> While we're on the subject of listening ports, here's a list of all
>>>>>>> listening ports on the haproxy host:
>>>>>>>
>>>>>>> [root@ha1:~] #netstat -tulpn | grep -i listen
>>>>>>> tcp 0 0 0.0.0.0:35145 0.0.0.0:*
>>>>>>> LISTEN -
>>>>>>> tcp 0 0 0.0.0.0:56814 0.0.0.0:*
>>>>>>> LISTEN 16346/rpc.statd
>>>>>>> tcp 0 0 0.0.0.0:111 0.0.0.0:*
>>>>>>> LISTEN 16455/rpcbind
>>>>>>> tcp 0 0 0.0.0.0:22 0.0.0.0:*
>>>>>>> LISTEN 16396/sshd
>>>>>>> tcp6 0 0 :::49349 :::*
>>>>>>> LISTEN 16346/rpc.statd
>>>>>>> tcp6 0 0 :::111 :::*
>>>>>>> LISTEN 16455/rpcbind
>>>>>>> tcp6 0 0 :::47314 :::*
>>>>>>> LISTEN -
>>>>>>> tcp6 0 0 :::22 :::*
>>>>>>> LISTEN 16396/sshd
>>>>>>>
>>>>>>> I thought I was beginning to understand this problem. That haproxy
>>>>>>> was
>>>>>>> trying to bind on port 3306 from the mysql host on another machine.
>>>>>>> But
>>>>>>> come to think of it, that doesn't make a lot of sense.
>>>>>>>
>>>>>>> Because I already have haproxy setup for some web servers, and there
>>>>>>> it
>>>>>>> creates port 80 on the haproxy node. It's not trying to connect to a
>>>>>>> foreign source. Not sure where I got that idea!!
>>>>>>>
>>>>>>> I also tried binding the mysql section to another port that wasn't in
>>>>>>> use.
>>>>>>> I tried port 3307,3308. I even tried binding the mysql section of the
>>>>>>> config to a weird port I just grabbed off of the top of my head. I
>>>>>>> tried
>>>>>>> binding it to port 4444.
>>>>>>>
>>>>>>> And there I still got a bind error:
>>>>>>>
>>>>>>> [ALERT] 204/223303 (13081) : Starting proxy mysql-cluster: cannot
>>>>>>> bind
>>>>>>> socket [0.0.0.0:4444]
>>>>>>>
>>>>>>>
>>>>>>> Now watch this!! If I bind the mysql section to port 80 instead
>>>>>>> of
>>>>>>> any
>>>>>>> other port.. haproxy starts up without complaint!
>>>>>>>
>>>>>>> listen mysql-cluster
>>>>>>> bind 0.0.0.0:80
>>>>>>> mode tcp
>>>>>>> option mysql-check user haproxy_check
>>>>>>> balance roundrobin
>>>>>>> server mysql-1 10.0.0.xxx :3306 check
>>>>>>> server mysql-2 10.0.0.xxx:3306 check
>>>>>>>
>>>>>>> [root@ha1:/etc/haproxy] #systemctl status haproxy
>>>>>>> haproxy.service - HAProxy Load Balancer
>>>>>>> Loaded: loaded (/usr/lib/systemd/system/haproxy.service;
>>>>>>> enabled)
>>>>>>> Active: active (running) since Fri 2015-07-24 22:35:03 UTC; 4s
>>>>>>> ago
>>>>>>> Main PID: 13213 (haproxy-systemd)
>>>>>>> CGroup: /system.slice/haproxy.service
>>>>>>> ├─13213 /usr/sbin/haproxy-systemd-wrapper -f
>>>>>>> /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
>>>>>>> ├─13214 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg
>>>>>>> -p
>>>>>>> /run/haproxy.pid -Ds
>>>>>>> └─13215 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg
>>>>>>> -p
>>>>>>> /run/haproxy.pid -Ds
>>>>>>>
>>>>>>> Jul 24 22:35:03 ha1 systemd[1]: Starting HAProxy Load Balancer...
>>>>>>> *Jul 24 22:35:03 ha1 systemd[1]: Started HAProxy Load Balancer.*
>>>>>>>
>>>>>>> Jul 24 22:35:03 ha1 haproxy-systemd-wrapper[13213]:
>>>>>>> haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f
>>>>>>> /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
>>>>>>>
>>>>>>> Ok. What...the...heck!!
>>>>>>>
>>>>>>> So why do you think that haproxy is only happy starting up on port
>>>>>>> 80?
>>>>>>> I
>>>>>>> would think that I should be able to specify any arbitrary port for
>>>>>>> it
>>>>>>> to
>>>>>>> listen on in a 'listen' sub-block.
>>>>>>>
>>>>>>> I guess I could have my app contact the database using port 80. But
>>>>>>> that's
>>>>>>> a little... weird. I installed haproxy using yum from the 'updates'
>>>>>>> repository. Is there any reason anyone can think of as to why haproxy
>>>>>>> refuses to start on any port other than port 80??
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Tim
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jul 24, 2015 at 4:59 PM, Nenad Merdanovic <[email protected]>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hello Tim,
>>>>>>>
>>>>>>> On Fri, Jul 24, 2015 at 1:46 PM, Tim Dunphy <
>>>>>>>> [email protected]
>>>>>>>>
>>>>>>>> <mailto:[email protected]>> wrote:
>>>>>>>>> listen mysql-cluster
>>>>>>>>> bind 127.0.0.1:3306 <http://127.0.0.1:3306>
>>>>>>>>>
>>>>>>>>> mode tcp
>>>>>>>>> option mysql-check user haproxy_check
>>>>>>>>> balance roundrobin
>>>>>>>>> server mysql-1 10.10.10.10:3306 <
>>>>>>>>> http://10.10.10.10:3306>
>>>>>>>>>
>>>>>>>>> check
>>>>>>>>>
>>>>>>>> server mysql-2 10.10.10.11:3306 <
>>>>>>>>
>>>>>>>>> http://10.10.10.11:3306>
>>>>>>>>>
>>>>>>>>> check
>>>>>>>>>
>>>>>>>> Jul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]:
>>>>>>>> [ALERT]
>>>>>>>>
>>>>>>>>> 204/034418 (25035) : *Starting proxy mysql-cluster:
>>>>>>>>> cannot
>>>>>>>>> bind
>>>>>>>>> s...:3306]*
>>>>>>>>>
>>>>>>>>> Can you check if something is listening on 127.0.0.1:3306
>>>>>>>>>
>>>>>>>> (netstat, ss,
>>>>>>>> lsof)? For example:
>>>>>>>> ss -lpt | fgrep 3306
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Nenad
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>> --
>>> GPG me!!
>>>
>>> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>>>
>>>
>>>
>
--
GPG me!!
gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
