Re: haproxy can't bind to mysql port
By run I meant you have to start it as root user which you are doing
anyway. Can you run:
# nc -l -p 80
as root just to confirm you can bind to port 80?
If I stop haproxy and run that command this is what I get:
[root@ha1:~] #nc -l -p 80
POST
/index.php?title=Special%3ARunJobstasks=jobsmaxjobs=1sigexpiry=1437841644signature=2f2d2dbd28afbc4ecf7e1f59708ff018a30427a9
HTTP/1.1
Host: wiki.example.com
Connection: Close
Content-Length: 0
Odd, since haproxy isnt' running currently.
On Sat, Jul 25, 2015 at 12:07 PM, Igor Cicimov
ig...@encompasscorporation.com wrote:
By run I meant you have to start it as root user which you are doing
anyway. Can you run:
# nc -l -p 80
as root just to confirm you can bind to port 80?
On 25/07/2015 2:10 PM, Igor Cicimov ig...@encompasscorporation.com
wrote:
You need to run haproxy as root to bind to ports lower than 1024
On 25/07/2015 1:36 PM, Tim Dunphy bluethu...@gmail.com wrote:
Hi Yuan,
Nice.
Do you use selinux in prod.
regards,
; Yuan
Yep! Actually I use it every chance I get. Prod/stage/dev and my own
hobby environments. And right now actually what I was discussing was a
hobby environment.
And actually if I could bother you guys one more time, I do have one
more issue to solve. LOL
And this time it's guaranteed not to be an SELinux issue. Because I
tried running haproxy with SELInux on and off this time.
But what's happening now, is that HA/Proxy is not creating the http port
for the 'stats' interface. I've setup stats to listen on port 80. But for
some reason that's not happening.
Here's my config one more time, with the trouble part in bold:
global
log 127.0.0.1 local0 notice
user haproxy
group haproxy
defaults
log global
retries 2
timeout connect 3000
timeout server 5000
timeout client 5000
listen mysql-cluster
bind 0.0.0.0:3306
mode tcp
option mysql-check user haproxy_check
balance roundrobin
server mysql-1 52.3.28.48:3306 check
server mysql-2 52.2.0.176:3306 check
*listen 0.0.0.0:80 http://0.0.0.0:80mode httpstats enable
stats uri /stats realm Strictly\ Privatestats auth admin:secret*
Currently haproxy is listening on the first port specified* - 3306 - *but
not listening on port 80.
Observe:
[root@ha1:/etc/haproxy] #lsof -i :3306
COMMAND PIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME
*haproxy 11653 haproxy4u IPv4 7145270 0t0 TCP *:mysql
(LISTEN)*
[root@ha1:/etc/haproxy] #lsof -i :80
[root@ha1:/etc/haproxy] #
[root@ha1:/etc/haproxy] #telnet localhost 80
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
Port 80 simply isn't listening.
And this time, I can't blame it on SELinux being on:
[root@ha1:/etc/haproxy] #getenforce
Permissive
I've grepped thru /var/log/messages but not turned up any clues to this
one.
And I really would like to get the stats interface up and running.
Any thoughts here? I'm wondering what I can do to get stats working.
Thanks,
Tim
On Fri, Jul 24, 2015 at 10:52 PM, Gmail longwuy...@gmail.com wrote:
Nice.
Do you use selinux in prod.
regards,
; Yuan
On 07/25/2015 09:17 AM, Tim Dunphy wrote:
Bingo!!!
The problem was with SELinux. Not sure what took me so long to think of
it...!!!
So set the mysql listener back to port 3306. Turned off SELinux with
setenforce 0. Then it started right up!!! And port 3306 was listening.
Then I consulted with audit2why and saw the following:
type=AVC msg=audit(1437786617.963:28856863): avc: denied {
name_connect }
for pid=29175 comm=haproxy dest=3306
scontext=system_u:system_r:haproxy_t:s0
tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket
Was caused by:
The boolean haproxy_connect_any was set incorrectly.
Description:
Allow haproxy to connect any
Allow access by executing:
# *setsebool -P haproxy_connect_any 1*
I just ran that command you see above in bold, and then all was right
with
the world.
[root@ha1:/etc/haproxy] #systemctl status haproxy
haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)
Active: active (running) since Sat 2015-07-25 01:14:53 UTC; 33s ago
Main PID: 30618 (haproxy-systemd)
CGroup: /system.slice/haproxy.service
├─30618 /usr/sbin/haproxy-systemd-wrapper -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid
├─30619 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
└─30620 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
Jul 25 01:14:53 ha1 systemd[1]: Starting HAProxy Load Balancer...
Jul 25 01:14:53 ha1 systemd[1]: Started HAProxy Load Balancer.
Jul 25 01:14:53 ha1 haproxy-systemd-wrapper[30618]:
haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
Re: haproxy can't bind to mysql port
By run I meant you have to start it as root user which you are doing
anyway. Can you run:
# nc -l -p 80
as root just to confirm you can bind to port 80?
On 25/07/2015 2:10 PM, Igor Cicimov ig...@encompasscorporation.com
wrote:
You need to run haproxy as root to bind to ports lower than 1024
On 25/07/2015 1:36 PM, Tim Dunphy bluethu...@gmail.com wrote:
Hi Yuan,
Nice.
Do you use selinux in prod.
regards,
; Yuan
Yep! Actually I use it every chance I get. Prod/stage/dev and my own
hobby environments. And right now actually what I was discussing was a
hobby environment.
And actually if I could bother you guys one more time, I do have one more
issue to solve. LOL
And this time it's guaranteed not to be an SELinux issue. Because I tried
running haproxy with SELInux on and off this time.
But what's happening now, is that HA/Proxy is not creating the http port
for the 'stats' interface. I've setup stats to listen on port 80. But for
some reason that's not happening.
Here's my config one more time, with the trouble part in bold:
global
log 127.0.0.1 local0 notice
user haproxy
group haproxy
defaults
log global
retries 2
timeout connect 3000
timeout server 5000
timeout client 5000
listen mysql-cluster
bind 0.0.0.0:3306
mode tcp
option mysql-check user haproxy_check
balance roundrobin
server mysql-1 52.3.28.48:3306 check
server mysql-2 52.2.0.176:3306 check
*listen 0.0.0.0:80 http://0.0.0.0:80mode httpstats enable
stats uri /stats realm Strictly\ Privatestats auth admin:secret*
Currently haproxy is listening on the first port specified* - 3306 - *but
not listening on port 80.
Observe:
[root@ha1:/etc/haproxy] #lsof -i :3306
COMMAND PIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME
*haproxy 11653 haproxy4u IPv4 7145270 0t0 TCP *:mysql (LISTEN)*
[root@ha1:/etc/haproxy] #lsof -i :80
[root@ha1:/etc/haproxy] #
[root@ha1:/etc/haproxy] #telnet localhost 80
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
Port 80 simply isn't listening.
And this time, I can't blame it on SELinux being on:
[root@ha1:/etc/haproxy] #getenforce
Permissive
I've grepped thru /var/log/messages but not turned up any clues to this
one.
And I really would like to get the stats interface up and running.
Any thoughts here? I'm wondering what I can do to get stats working.
Thanks,
Tim
On Fri, Jul 24, 2015 at 10:52 PM, Gmail longwuy...@gmail.com wrote:
Nice.
Do you use selinux in prod.
regards,
; Yuan
On 07/25/2015 09:17 AM, Tim Dunphy wrote:
Bingo!!!
The problem was with SELinux. Not sure what took me so long to think of
it...!!!
So set the mysql listener back to port 3306. Turned off SELinux with
setenforce 0. Then it started right up!!! And port 3306 was listening.
Then I consulted with audit2why and saw the following:
type=AVC msg=audit(1437786617.963:28856863): avc: denied {
name_connect }
for pid=29175 comm=haproxy dest=3306
scontext=system_u:system_r:haproxy_t:s0
tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket
Was caused by:
The boolean haproxy_connect_any was set incorrectly.
Description:
Allow haproxy to connect any
Allow access by executing:
# *setsebool -P haproxy_connect_any 1*
I just ran that command you see above in bold, and then all was right
with
the world.
[root@ha1:/etc/haproxy] #systemctl status haproxy
haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)
Active: active (running) since Sat 2015-07-25 01:14:53 UTC; 33s ago
Main PID: 30618 (haproxy-systemd)
CGroup: /system.slice/haproxy.service
├─30618 /usr/sbin/haproxy-systemd-wrapper -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid
├─30619 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
└─30620 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
Jul 25 01:14:53 ha1 systemd[1]: Starting HAProxy Load Balancer...
Jul 25 01:14:53 ha1 systemd[1]: Started HAProxy Load Balancer.
Jul 25 01:14:53 ha1 haproxy-systemd-wrapper[30618]:
haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
[root@ha1:/etc/haproxy] #lsof -i :3306
COMMAND PIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME
haproxy 30620 haproxy1u IPv4 7075172 0t0 TCP
ha1.example.com:55499-ec2-52-2-0-xxx.compute-1.amazonaws.com:mysql
(SYN_SENT)
haproxy 30620 haproxy4u IPv4 7074731 0t0 TCP *:mysql (LISTEN)
Thanks for nudging me in the right direction. All I had to hear was the
word 'selinux' and from there it all fell into place!
Thanks!!
Tim
On Fri, Jul 24, 2015 at 8:20 PM, Gmail longwuy...@gmail.com wrote:
I could be completely wrong here and I am curious to know the answer
Re: haproxy can't bind to mysql port
Also, lsof output shows PID = 11653 and the user/owner/launcher of the process = haproxy. cheers, ; Yuan On 07/26/2015 12:13 AM, Gmail wrote: I am uncertain about syntax but the diff is appname of sorts for the port 80 listener. Someone may comment with more details ; Earlier = *listen 0.0.0.0:80 [ no app name string ] Now= listen jokefire 0.0.0.0:80 [ app name == jokefire and also no astrisk visible ] Maybe attempt restart without any appname and also with/without that asterisk. Deep dives need more awareness. Glad all settled. Cheers, ; Yuan On 07/25/2015 11:51 PM, Tim Dunphy wrote: Yuan, maybe something here http://lnxmon.com/haproxy/ Thanks, ; Yuan I modified a config from your blog that you showed me and came up with this: global log 127.0.0.1 local0 notice maxconn 2000 user haproxy group haproxy defaults log global modehttp option httplog option dontlognull retries 3 option redispatch timeout connect 5000 timeout client 1 timeout server 1 listen jokefire 0.0.0.0:80 mode http stats enable stats uri /haproxy?stats stats realm Strictly\ Private stats auth admin:secret balance roundrobin option httpclose option forwardfor server varnish1 10.10.10.5:80 check server varnish2 10.10.10.6:80 check listen mysql-cluster bind 0.0.0.0:3306 mode tcp balance roundrobin maxconn 5200 option mysql-check user haproxy_root server mysql-1 10.10.10.7:3306 check server mysql-2 10.10.10.8:3306 check And that seemed to work. I can see that both ports are listening now: [root@ha1:/etc/haproxy] #lsof -i :80 -i :3306 COMMAND PIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME haproxy 27136 haproxy4u IPv4 7563913 0t0 TCP *:http (LISTEN) haproxy 27136 haproxy6u IPv4 7563915 0t0 TCP *:mysql (LISTEN) Although I am not aware of the real difference between this and my previous config that allows this to work is. Not a huge issue at this point since it's working. But if anyone wants to take a stab at this, be my guest! Thanks, Tim On Sat, Jul 25, 2015 at 12:15 AM, Gmail longwuy...@gmail.com wrote: maybe something here http://lnxmon.com/haproxy/ Thanks, ; Yuan On 07/25/2015 12:10 PM, Igor Cicimov wrote: You need to run haproxy as root to bind to ports lower than 1024 On 25/07/2015 1:36 PM, Tim Dunphy bluethu...@gmail.com wrote: Hi Yuan, Nice. Do you use selinux in prod. regards, ; Yuan Yep! Actually I use it every chance I get. Prod/stage/dev and my own hobby environments. And right now actually what I was discussing was a hobby environment. And actually if I could bother you guys one more time, I do have one more issue to solve. LOL And this time it's guaranteed not to be an SELinux issue. Because I tried running haproxy with SELInux on and off this time. But what's happening now, is that HA/Proxy is not creating the http port for the 'stats' interface. I've setup stats to listen on port 80. But for some reason that's not happening. Here's my config one more time, with the trouble part in bold: global log 127.0.0.1 local0 notice user haproxy group haproxy defaults log global retries 2 timeout connect 3000 timeout server 5000 timeout client 5000 listen mysql-cluster bind 0.0.0.0:3306 mode tcp option mysql-check user haproxy_check balance roundrobin server mysql-1 52.3.28.48:3306 check server mysql-2 52.2.0.176:3306 check *listen 0.0.0.0:80 http://0.0.0.0:80mode httpstats enable stats uri /stats realm Strictly\ Privatestats auth admin:secret* Currently haproxy is listening on the first port specified* - 3306 - *but not listening on port 80. Observe: [root@ha1:/etc/haproxy] #lsof -i :3306 COMMAND PIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME *haproxy 11653 haproxy4u IPv4 7145270 0t0 TCP *:mysql (LISTEN)* [root@ha1:/etc/haproxy] #lsof -i :80 [root@ha1:/etc/haproxy] # [root@ha1:/etc/haproxy] #telnet localhost 80 Trying 127.0.0.1... telnet: connect to address 127.0.0.1: Connection refused Port 80 simply isn't listening. And this time, I can't blame it on SELinux being on: [root@ha1:/etc/haproxy] #getenforce Permissive I've grepped thru /var/log/messages but not turned up any clues to this one. And I really would like to get the stats interface up and running. Any thoughts here? I'm wondering what I can do to get stats working. Thanks, Tim On Fri, Jul 24, 2015 at 10:52 PM, Gmail longwuy...@gmail.com wrote: Nice. Do you use selinux in prod. regards, ; Yuan On 07/25/2015 09:17 AM, Tim Dunphy wrote: Bingo!!! The problem was with SELinux. Not sure what took me so long to think of it...!!! So set the mysql listener back to port 3306. Turned off SELinux with setenforce 0. Then it started right up!!! And port 3306
Re: haproxy can't bind to mysql port
You need to run haproxy as root to bind to ports lower than 1024
I tried running haproxy as root/root:
[root@ha1:/etc/haproxy] #egrep user|group haproxy.cfg| grep -v option
user root
group root
Then restarted the service. No difference!
[root@ha1:/etc/haproxy] #systemctl restart haproxy
[root@ha1:/etc/haproxy] #lsof -i :80
[root@ha1:/etc/haproxy] #getenforce
Permissive
Thanks for the suggestion anyway!
On Sat, Jul 25, 2015 at 12:10 AM, Igor Cicimov
ig...@encompasscorporation.com wrote:
You need to run haproxy as root to bind to ports lower than 1024
On 25/07/2015 1:36 PM, Tim Dunphy bluethu...@gmail.com wrote:
Hi Yuan,
Nice.
Do you use selinux in prod.
regards,
; Yuan
Yep! Actually I use it every chance I get. Prod/stage/dev and my own
hobby environments. And right now actually what I was discussing was a
hobby environment.
And actually if I could bother you guys one more time, I do have one more
issue to solve. LOL
And this time it's guaranteed not to be an SELinux issue. Because I tried
running haproxy with SELInux on and off this time.
But what's happening now, is that HA/Proxy is not creating the http port
for the 'stats' interface. I've setup stats to listen on port 80. But for
some reason that's not happening.
Here's my config one more time, with the trouble part in bold:
global
log 127.0.0.1 local0 notice
user haproxy
group haproxy
defaults
log global
retries 2
timeout connect 3000
timeout server 5000
timeout client 5000
listen mysql-cluster
bind 0.0.0.0:3306
mode tcp
option mysql-check user haproxy_check
balance roundrobin
server mysql-1 52.3.28.48:3306 check
server mysql-2 52.2.0.176:3306 check
*listen 0.0.0.0:80 http://0.0.0.0:80mode httpstats enable
stats uri /stats realm Strictly\ Privatestats auth admin:secret*
Currently haproxy is listening on the first port specified* - 3306 - *but
not listening on port 80.
Observe:
[root@ha1:/etc/haproxy] #lsof -i :3306
COMMAND PIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME
*haproxy 11653 haproxy4u IPv4 7145270 0t0 TCP *:mysql (LISTEN)*
[root@ha1:/etc/haproxy] #lsof -i :80
[root@ha1:/etc/haproxy] #
[root@ha1:/etc/haproxy] #telnet localhost 80
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
Port 80 simply isn't listening.
And this time, I can't blame it on SELinux being on:
[root@ha1:/etc/haproxy] #getenforce
Permissive
I've grepped thru /var/log/messages but not turned up any clues to this
one.
And I really would like to get the stats interface up and running.
Any thoughts here? I'm wondering what I can do to get stats working.
Thanks,
Tim
On Fri, Jul 24, 2015 at 10:52 PM, Gmail longwuy...@gmail.com wrote:
Nice.
Do you use selinux in prod.
regards,
; Yuan
On 07/25/2015 09:17 AM, Tim Dunphy wrote:
Bingo!!!
The problem was with SELinux. Not sure what took me so long to think of
it...!!!
So set the mysql listener back to port 3306. Turned off SELinux with
setenforce 0. Then it started right up!!! And port 3306 was listening.
Then I consulted with audit2why and saw the following:
type=AVC msg=audit(1437786617.963:28856863): avc: denied {
name_connect }
for pid=29175 comm=haproxy dest=3306
scontext=system_u:system_r:haproxy_t:s0
tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket
Was caused by:
The boolean haproxy_connect_any was set incorrectly.
Description:
Allow haproxy to connect any
Allow access by executing:
# *setsebool -P haproxy_connect_any 1*
I just ran that command you see above in bold, and then all was right
with
the world.
[root@ha1:/etc/haproxy] #systemctl status haproxy
haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)
Active: active (running) since Sat 2015-07-25 01:14:53 UTC; 33s ago
Main PID: 30618 (haproxy-systemd)
CGroup: /system.slice/haproxy.service
├─30618 /usr/sbin/haproxy-systemd-wrapper -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid
├─30619 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
└─30620 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
Jul 25 01:14:53 ha1 systemd[1]: Starting HAProxy Load Balancer...
Jul 25 01:14:53 ha1 systemd[1]: Started HAProxy Load Balancer.
Jul 25 01:14:53 ha1 haproxy-systemd-wrapper[30618]:
haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
[root@ha1:/etc/haproxy] #lsof -i :3306
COMMAND PIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME
haproxy 30620 haproxy1u IPv4 7075172 0t0 TCP
ha1.example.com:55499-ec2-52-2-0-xxx.compute-1.amazonaws.com:mysql
(SYN_SENT)
haproxy 30620 haproxy4u IPv4 7074731 0t0 TCP *:mysql (LISTEN)
Thanks
Re: haproxy can't bind to mysql port
Yuan,
maybe something here http://lnxmon.com/haproxy/
Thanks,
; Yuan
I modified a config from your blog that you showed me and came up with this:
global
log 127.0.0.1 local0 notice
maxconn 2000
user haproxy
group haproxy
defaults
log global
modehttp
option httplog
option dontlognull
retries 3
option redispatch
timeout connect 5000
timeout client 1
timeout server 1
listen jokefire 0.0.0.0:80
mode http
stats enable
stats uri /haproxy?stats
stats realm Strictly\ Private
stats auth admin:secret
balance roundrobin
option httpclose
option forwardfor
server varnish1 10.10.10.5:80 check
server varnish2 10.10.10.6:80 check
listen mysql-cluster
bind 0.0.0.0:3306
mode tcp
balance roundrobin
maxconn 5200
option mysql-check user haproxy_root
server mysql-1 10.10.10.7:3306 check
server mysql-2 10.10.10.8:3306 check
And that seemed to work. I can see that both ports are listening now:
[root@ha1:/etc/haproxy] #lsof -i :80 -i :3306
COMMAND PIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME
haproxy 27136 haproxy4u IPv4 7563913 0t0 TCP *:http (LISTEN)
haproxy 27136 haproxy6u IPv4 7563915 0t0 TCP *:mysql (LISTEN)
Although I am not aware of the real difference between this and my previous
config that allows this to work is.
Not a huge issue at this point since it's working. But if anyone wants to
take a stab at this, be my guest!
Thanks,
Tim
On Sat, Jul 25, 2015 at 12:15 AM, Gmail longwuy...@gmail.com wrote:
maybe something here http://lnxmon.com/haproxy/
Thanks,
; Yuan
On 07/25/2015 12:10 PM, Igor Cicimov wrote:
You need to run haproxy as root to bind to ports lower than 1024
On 25/07/2015 1:36 PM, Tim Dunphy bluethu...@gmail.com wrote:
Hi Yuan,
Nice.
Do you use selinux in prod.
regards,
; Yuan
Yep! Actually I use it every chance I get. Prod/stage/dev and my own
hobby
environments. And right now actually what I was discussing was a hobby
environment.
And actually if I could bother you guys one more time, I do have one more
issue to solve. LOL
And this time it's guaranteed not to be an SELinux issue. Because I tried
running haproxy with SELInux on and off this time.
But what's happening now, is that HA/Proxy is not creating the http port
for the 'stats' interface. I've setup stats to listen on port 80. But for
some reason that's not happening.
Here's my config one more time, with the trouble part in bold:
global
log 127.0.0.1 local0 notice
user haproxy
group haproxy
defaults
log global
retries 2
timeout connect 3000
timeout server 5000
timeout client 5000
listen mysql-cluster
bind 0.0.0.0:3306
mode tcp
option mysql-check user haproxy_check
balance roundrobin
server mysql-1 52.3.28.48:3306 check
server mysql-2 52.2.0.176:3306 check
*listen 0.0.0.0:80 http://0.0.0.0:80mode httpstats enable
stats uri /stats realm Strictly\ Privatestats auth admin:secret*
Currently haproxy is listening on the first port specified* - 3306 - *but
not listening on port 80.
Observe:
[root@ha1:/etc/haproxy] #lsof -i :3306
COMMAND PIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME
*haproxy 11653 haproxy4u IPv4 7145270 0t0 TCP *:mysql
(LISTEN)*
[root@ha1:/etc/haproxy] #lsof -i :80
[root@ha1:/etc/haproxy] #
[root@ha1:/etc/haproxy] #telnet localhost 80
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
Port 80 simply isn't listening.
And this time, I can't blame it on SELinux being on:
[root@ha1:/etc/haproxy] #getenforce
Permissive
I've grepped thru /var/log/messages but not turned up any clues to this
one.
And I really would like to get the stats interface up and running.
Any thoughts here? I'm wondering what I can do to get stats working.
Thanks,
Tim
On Fri, Jul 24, 2015 at 10:52 PM, Gmail longwuy...@gmail.com wrote:
Nice.
Do you use selinux in prod.
regards,
; Yuan
On 07/25/2015 09:17 AM, Tim Dunphy wrote:
Bingo!!!
The problem was with SELinux. Not sure what took me so long to think of
it...!!!
So set the mysql listener back to port 3306. Turned off SELinux with
setenforce 0. Then it started right up!!! And port 3306 was listening.
Then I consulted with audit2why and saw the following:
type=AVC msg=audit(1437786617.963:28856863): avc: denied {
name_connect }
for pid=29175 comm=haproxy dest=3306
scontext=system_u:system_r:haproxy_t:s0
tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket
Was caused by:
The boolean haproxy_connect_any was set incorrectly.
Description:
Allow haproxy to connect any
Allow access by executing:
# *setsebool -P haproxy_connect_any 1*
I just ran that command you see above in bold, and then all was right
with
the
Re: haproxy can't bind to mysql port
I am uncertain about syntax but the diff is appname of sorts for the
port 80 listener. Someone may comment with more details ;
Earlier = *listen 0.0.0.0:80 [ no app name string ]
Now= listen jokefire 0.0.0.0:80 [ app name == jokefire and also no
astrisk visible ]
Maybe attempt restart without any appname and also with/without that
asterisk.
Deep dives need more awareness.
Glad all settled.
Cheers,
; Yuan
On 07/25/2015 11:51 PM, Tim Dunphy wrote:
Yuan,
maybe something here http://lnxmon.com/haproxy/
Thanks,
; Yuan
I modified a config from your blog that you showed me and came up with this:
global
log 127.0.0.1 local0 notice
maxconn 2000
user haproxy
group haproxy
defaults
log global
modehttp
option httplog
option dontlognull
retries 3
option redispatch
timeout connect 5000
timeout client 1
timeout server 1
listen jokefire 0.0.0.0:80
mode http
stats enable
stats uri /haproxy?stats
stats realm Strictly\ Private
stats auth admin:secret
balance roundrobin
option httpclose
option forwardfor
server varnish1 10.10.10.5:80 check
server varnish2 10.10.10.6:80 check
listen mysql-cluster
bind 0.0.0.0:3306
mode tcp
balance roundrobin
maxconn 5200
option mysql-check user haproxy_root
server mysql-1 10.10.10.7:3306 check
server mysql-2 10.10.10.8:3306 check
And that seemed to work. I can see that both ports are listening now:
[root@ha1:/etc/haproxy] #lsof -i :80 -i :3306
COMMAND PIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME
haproxy 27136 haproxy4u IPv4 7563913 0t0 TCP *:http (LISTEN)
haproxy 27136 haproxy6u IPv4 7563915 0t0 TCP *:mysql (LISTEN)
Although I am not aware of the real difference between this and my previous
config that allows this to work is.
Not a huge issue at this point since it's working. But if anyone wants to
take a stab at this, be my guest!
Thanks,
Tim
On Sat, Jul 25, 2015 at 12:15 AM, Gmail longwuy...@gmail.com wrote:
maybe something here http://lnxmon.com/haproxy/
Thanks,
; Yuan
On 07/25/2015 12:10 PM, Igor Cicimov wrote:
You need to run haproxy as root to bind to ports lower than 1024
On 25/07/2015 1:36 PM, Tim Dunphy bluethu...@gmail.com wrote:
Hi Yuan,
Nice.
Do you use selinux in prod.
regards,
; Yuan
Yep! Actually I use it every chance I get. Prod/stage/dev and my own
hobby
environments. And right now actually what I was discussing was a hobby
environment.
And actually if I could bother you guys one more time, I do have one more
issue to solve. LOL
And this time it's guaranteed not to be an SELinux issue. Because I tried
running haproxy with SELInux on and off this time.
But what's happening now, is that HA/Proxy is not creating the http port
for the 'stats' interface. I've setup stats to listen on port 80. But for
some reason that's not happening.
Here's my config one more time, with the trouble part in bold:
global
log 127.0.0.1 local0 notice
user haproxy
group haproxy
defaults
log global
retries 2
timeout connect 3000
timeout server 5000
timeout client 5000
listen mysql-cluster
bind 0.0.0.0:3306
mode tcp
option mysql-check user haproxy_check
balance roundrobin
server mysql-1 52.3.28.48:3306 check
server mysql-2 52.2.0.176:3306 check
*listen 0.0.0.0:80 http://0.0.0.0:80mode httpstats enable
stats uri /stats realm Strictly\ Privatestats auth admin:secret*
Currently haproxy is listening on the first port specified* - 3306 - *but
not listening on port 80.
Observe:
[root@ha1:/etc/haproxy] #lsof -i :3306
COMMAND PIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME
*haproxy 11653 haproxy4u IPv4 7145270 0t0 TCP *:mysql
(LISTEN)*
[root@ha1:/etc/haproxy] #lsof -i :80
[root@ha1:/etc/haproxy] #
[root@ha1:/etc/haproxy] #telnet localhost 80
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
Port 80 simply isn't listening.
And this time, I can't blame it on SELinux being on:
[root@ha1:/etc/haproxy] #getenforce
Permissive
I've grepped thru /var/log/messages but not turned up any clues to this
one.
And I really would like to get the stats interface up and running.
Any thoughts here? I'm wondering what I can do to get stats working.
Thanks,
Tim
On Fri, Jul 24, 2015 at 10:52 PM, Gmail longwuy...@gmail.com wrote:
Nice.
Do you use selinux in prod.
regards,
; Yuan
On 07/25/2015 09:17 AM, Tim Dunphy wrote:
Bingo!!!
The problem was with SELinux. Not sure what took me so long to think of
it...!!!
So set the mysql listener back to port 3306. Turned off SELinux with
setenforce 0. Then it started right up!!! And port 3306 was listening.
Then I consulted with audit2why and saw the following:
type=AVC msg=audit(1437786617.963:28856863): avc: denied {
name_connect }
for pid=29175
Re: haproxy can't bind to mysql port
Hi there Igor, How did you do that? By putting the haproxy's security group or the haproxy's ip to the mysql group inbound rule? Sorry but which non local IP is that? I put the external IP of the haproxy server into the security group on the mysql server. In this case the external IP from the haproxy host is an elastic IP. And I saw that when I did that and performed a grant on the database I could log into the db from the haproxy machine. How many interfaces haproxy has? Is it connected to the 10.10.10.0/24 network at all? Looks to me you are trying to use VIP's or something which does not work in same way as in normal lan's. Don't forget that in AWS we are dealing with SDN's so giving l0 or any other interface a second IP address localy on the instance using ip tool lets say will simply not work. That IP is not visible to the SDN and the interface will never send or receive any traffic. You need that IP allocated to the haproxy interface (no option for l0 here) via EC2 console or aws cli tool. The IP's I was using in the example config I posted to the thread earlier was using fake IP's, not the real ones. In reality I had used the external IP's which are on a 52.3.22 subnet (for example on one host). The haproxy machines have just one interface, and they're both on the same VPC. This is needed for failover. Those IPs actually are 10-net addresses. So maybe I'll need to put those IP's into the SG's on the database nodes to get this working? So for example if I do a command like this: [root@ha1:/etc/haproxy] #ifconfig | grep inet | grep -v inet6 inet 10.0.0.XXX netmask 255.255.255.0 broadcast 10.0.0.255 inet 127.0.0.1 netmask 255.0.0.0 What I get back is a 10-net address and the loopback. Only thing is that the database hosts are on separate AWS accounts. So I don't know how the database nodes will be able to recognize the internal 10-net addresses from the haproxy hosts. Do you have any suggestions on how this can (or can't) work? Thanks, Tim On Fri, Jul 24, 2015 at 1:11 AM, Igor Cicimov ig...@encompasscorporation.com wrote: On Fri, Jul 24, 2015 at 1:46 PM, Tim Dunphy bluethu...@gmail.com wrote: Hi all, I'm attempting to setup mysql load balancing using HA/Proxy. Seemed pretty straight forward at first. I'm using Amazon ec2 for all nodes. First I made sure that the haproxy nodes could contact the mysql boxes by opening up the security group from the mysql boxes to the haproxy ones on port 3306. How did you do that? By putting the haproxy's security group or the haproxy's ip to the mysql group inbound rule? If IP which one is that? I setup the following config: global log 127.0.0.1 local0 notice user haproxy group haproxy defaults log global retries 2 timeout connect 3000 timeout server 5000 timeout client 5000 listen mysql-cluster bind 127.0.0.1:3306 mode tcp option mysql-check user haproxy_check balance roundrobin server mysql-1 10.10.10.10:3306 check server mysql-2 10.10.10.11:3306 check listen 0.0.0.0:80 mode http stats enable stats uri / stats realm Strictly\ Private stats auth admin:secret And ensured that haproxy could bind to non local IP's: Sorry but which non local IP is that? How many interfaces haproxy has? Is it connected to the 10.10.10.0/24 network at all? Looks to me you are trying to use VIP's or something which does not work in same way as in normal lan's. Don't forget that in AWS we are dealing with SDN's so giving l0 or any other interface a second IP address localy on the instance using ip tool lets say will simply not work. That IP is not visible to the SDN and the interface will never send or receive any traffic. You need that IP allocated to the haproxy interface (no option for l0 here) via EC2 console or aws cli tool. [root@ha1:/etc/haproxy] #grep ipv4 /etc/sysctl.conf net.ipv4.ip_nonlocal_bind=1 [root@ha1:/etc/haproxy] #sysctl -p net.ipv4.ip_nonlocal_bind = 1 Yet when I try to start up haproxy I get the following result: [root@ha1:/etc/haproxy] #systemctl status haproxy haproxy.service - HAProxy Load Balancer Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled) Active: inactive (dead) since Fri 2015-07-24 03:44:18 UTC; 9s ago Process: 25034 ExecStart=/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid (code=exited, status=0/SUCCESS) Main PID: 25034 (code=exited, status=0/SUCCESS) Jul 24 03:44:18 ha1 systemd[1]: Starting HAProxy Load Balancer... Jul 24 03:44:18 ha1 systemd[1]: Started HAProxy Load Balancer. Jul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/hap...id -Ds Jul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: [ALERT] 204/034418 (25035) : *Starting proxy mysql-cluster: cannot bind s...:3306]* Jul 24 03:44:18 ha1*
Re: haproxy can't bind to mysql port
Hello Tim, On Fri, Jul 24, 2015 at 1:46 PM, Tim Dunphy bluethu...@gmail.com mailto:bluethu...@gmail.com wrote: listen mysql-cluster bind 127.0.0.1:3306 http://127.0.0.1:3306 mode tcp option mysql-check user haproxy_check balance roundrobin server mysql-1 10.10.10.10:3306 http://10.10.10.10:3306 check server mysql-2 10.10.10.11:3306 http://10.10.10.11:3306 check Jul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: [ALERT] 204/034418 (25035) : *Starting proxy mysql-cluster: cannot bind s...:3306]* Can you check if something is listening on 127.0.0.1:3306 (netstat, ss, lsof)? For example: ss -lpt | fgrep 3306 Regards, Nenad
Re: haproxy can't bind to mysql port
Hello Nenad, Jul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: [ALERT] 204/034418 (25035) : *Starting proxy mysql-cluster: cannot bind s...:3306]* Nothing listening on the port I'm trying to bind to: 3306 [root@ha1:~] #ss -lpt | fgrep 3306 [root@ha1:~] #lsof -i :3306 [root@ha1:~] #netstat -tulpn | grep -i listen | grep 3306 [root@ha1:~] # While we're on the subject of listening ports, here's a list of all listening ports on the haproxy host: [root@ha1:~] #netstat -tulpn | grep -i listen tcp0 0 0.0.0.0:35145 0.0.0.0:* LISTEN - tcp0 0 0.0.0.0:56814 0.0.0.0:* LISTEN 16346/rpc.statd tcp0 0 0.0.0.0:111 0.0.0.0:* LISTEN 16455/rpcbind tcp0 0 0.0.0.0:22 0.0.0.0:* LISTEN 16396/sshd tcp6 0 0 :::49349:::* LISTEN 16346/rpc.statd tcp6 0 0 :::111 :::* LISTEN 16455/rpcbind tcp6 0 0 :::47314:::* LISTEN - tcp6 0 0 :::22 :::* LISTEN 16396/sshd I thought I was beginning to understand this problem. That haproxy was trying to bind on port 3306 from the mysql host on another machine. But come to think of it, that doesn't make a lot of sense. Because I already have haproxy setup for some web servers, and there it creates port 80 on the haproxy node. It's not trying to connect to a foreign source. Not sure where I got that idea!! I also tried binding the mysql section to another port that wasn't in use. I tried port 3307,3308. I even tried binding the mysql section of the config to a weird port I just grabbed off of the top of my head. I tried binding it to port . And there I still got a bind error: [ALERT] 204/223303 (13081) : Starting proxy mysql-cluster: cannot bind socket [0.0.0.0:] Now watch this!! If I bind the mysql section to port 80 instead of any other port.. haproxy starts up without complaint! listen mysql-cluster bind 0.0.0.0:80 mode tcp option mysql-check user haproxy_check balance roundrobin server mysql-1 10.0.0.xxx :3306 check server mysql-2 10.0.0.xxx:3306 check [root@ha1:/etc/haproxy] #systemctl status haproxy haproxy.service - HAProxy Load Balancer Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled) Active: active (running) since Fri 2015-07-24 22:35:03 UTC; 4s ago Main PID: 13213 (haproxy-systemd) CGroup: /system.slice/haproxy.service ├─13213 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid ├─13214 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds └─13215 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds Jul 24 22:35:03 ha1 systemd[1]: Starting HAProxy Load Balancer... *Jul 24 22:35:03 ha1 systemd[1]: Started HAProxy Load Balancer.* Jul 24 22:35:03 ha1 haproxy-systemd-wrapper[13213]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds Ok. What...the...heck!! So why do you think that haproxy is only happy starting up on port 80? I would think that I should be able to specify any arbitrary port for it to listen on in a 'listen' sub-block. I guess I could have my app contact the database using port 80. But that's a little... weird. I installed haproxy using yum from the 'updates' repository. Is there any reason anyone can think of as to why haproxy refuses to start on any port other than port 80?? Thanks, Tim On Fri, Jul 24, 2015 at 4:59 PM, Nenad Merdanovic ni...@nimzo.info wrote: Hello Tim, On Fri, Jul 24, 2015 at 1:46 PM, Tim Dunphy bluethu...@gmail.com mailto:bluethu...@gmail.com wrote: listen mysql-cluster bind 127.0.0.1:3306 http://127.0.0.1:3306 mode tcp option mysql-check user haproxy_check balance roundrobin server mysql-1 10.10.10.10:3306 http://10.10.10.10:3306 check server mysql-2 10.10.10.11:3306 http://10.10.10.11:3306 check Jul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: [ALERT] 204/034418 (25035) : *Starting proxy mysql-cluster: cannot bind s...:3306]* Can you check if something is listening on 127.0.0.1:3306 (netstat, ss, lsof)? For example: ss -lpt | fgrep 3306 Regards, Nenad -- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B On Fri, Jul 24, 2015 at 4:59 PM, Nenad Merdanovic ni...@nimzo.info wrote: Hello Tim, On Fri, Jul 24, 2015 at 1:46 PM, Tim Dunphy bluethu...@gmail.com mailto:bluethu...@gmail.com wrote: listen mysql-cluster bind 127.0.0.1:3306 http://127.0.0.1:3306 mode tcp option mysql-check user haproxy_check balance roundrobin server mysql-1 10.10.10.10:3306 http://10.10.10.10:3306 check
Re: haproxy can't bind to mysql port
Nice.
Do you use selinux in prod.
regards,
; Yuan
On 07/25/2015 09:17 AM, Tim Dunphy wrote:
Bingo!!!
The problem was with SELinux. Not sure what took me so long to think of
it...!!!
So set the mysql listener back to port 3306. Turned off SELinux with
setenforce 0. Then it started right up!!! And port 3306 was listening.
Then I consulted with audit2why and saw the following:
type=AVC msg=audit(1437786617.963:28856863): avc: denied { name_connect }
for pid=29175 comm=haproxy dest=3306
scontext=system_u:system_r:haproxy_t:s0
tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket
Was caused by:
The boolean haproxy_connect_any was set incorrectly.
Description:
Allow haproxy to connect any
Allow access by executing:
# *setsebool -P haproxy_connect_any 1*
I just ran that command you see above in bold, and then all was right with
the world.
[root@ha1:/etc/haproxy] #systemctl status haproxy
haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)
Active: active (running) since Sat 2015-07-25 01:14:53 UTC; 33s ago
Main PID: 30618 (haproxy-systemd)
CGroup: /system.slice/haproxy.service
├─30618 /usr/sbin/haproxy-systemd-wrapper -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid
├─30619 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
└─30620 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
Jul 25 01:14:53 ha1 systemd[1]: Starting HAProxy Load Balancer...
Jul 25 01:14:53 ha1 systemd[1]: Started HAProxy Load Balancer.
Jul 25 01:14:53 ha1 haproxy-systemd-wrapper[30618]:
haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
[root@ha1:/etc/haproxy] #lsof -i :3306
COMMAND PIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME
haproxy 30620 haproxy1u IPv4 7075172 0t0 TCP
ha1.example.com:55499-ec2-52-2-0-xxx.compute-1.amazonaws.com:mysql
(SYN_SENT)
haproxy 30620 haproxy4u IPv4 7074731 0t0 TCP *:mysql (LISTEN)
Thanks for nudging me in the right direction. All I had to hear was the
word 'selinux' and from there it all fell into place!
Thanks!!
Tim
On Fri, Jul 24, 2015 at 8:20 PM, Gmail longwuy...@gmail.com wrote:
I could be completely wrong here and I am curious to know the answer
myself. Please don't take this as a solution, just my thoughts.
First, you can not use backend ip-address of 10.x.x.x subnet because each
account's VPC is seggregated. If you do want to use 10.X.X.X ipadress you
have to setup a inter VPC endpoint in AWS. I would just use EIP.
For the port 3306, try to use nc to listen on that port or iperf. Do yo
uhave iptables turned on.
I would check systemctl -l status haproxy.service
I would check lsof -i why can't bind to 3306 on loopback ipaddress.
I would check iptables or selinux preventing the bind.
It wil be interesting to know the source ipaddress of MySQL client ec2
instance.
Interesting if you can Copy/paste output of telnet
haproxynode_ipaddress 3306 from mysql client ec2 instance , here.
Interesting if you can Copy/paste output of telnet 10.10.10.10 3306
from haproxy ec2 instances, here.
Interesting if you can Copy/paste output of telnet 10.10.10.11 3306
from haproxy ec2 instances, here.
I I was doing this, maybe I would consider testing something like ;
..
frontend mysql_lb_fe 0.0.0.0:3306
acl host_myql_lb hdr(host) -i mysql-lb
..
..
use_backend mysql_lb_backend if host mysql_lb
..
..
backend mysql_lb_be
..
..
option mysql-check user haproxy_check
balance roundrobin
server mysql-1 10.10.10.10:3306 check
server mysql-2 10.10.10.11:3306 check
Thanks,
; Yuan
On 07/25/2015 06:41 AM, Tim Dunphy wrote:
Hello Nenad,
Jul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: [ALERT]
204/034418 (25035) : *Starting proxy mysql-cluster: cannot bind
s...:3306]*
Nothing listening on the port I'm trying to bind to: 3306
[root@ha1:~] #ss -lpt | fgrep 3306
[root@ha1:~] #lsof -i :3306
[root@ha1:~] #netstat -tulpn | grep -i listen | grep 3306
[root@ha1:~] #
While we're on the subject of listening ports, here's a list of all
listening ports on the haproxy host:
[root@ha1:~] #netstat -tulpn | grep -i listen
tcp0 0 0.0.0.0:35145 0.0.0.0:*
LISTEN -
tcp0 0 0.0.0.0:56814 0.0.0.0:*
LISTEN 16346/rpc.statd
tcp0 0 0.0.0.0:111 0.0.0.0:*
LISTEN 16455/rpcbind
tcp0 0 0.0.0.0:22 0.0.0.0:*
LISTEN 16396/sshd
tcp6 0 0 :::49349:::*
LISTEN 16346/rpc.statd
tcp6 0 0 :::111 :::*
LISTEN 16455/rpcbind
tcp6 0 0 :::47314:::*
LISTEN -
tcp6 0 0 :::22 :::*
LISTEN 16396/sshd
I thought I was beginning to understand this problem. That haproxy was
trying
Re: haproxy can't bind to mysql port
Hi Yuan,
Nice.
Do you use selinux in prod.
regards,
; Yuan
Yep! Actually I use it every chance I get. Prod/stage/dev and my own hobby
environments. And right now actually what I was discussing was a hobby
environment.
And actually if I could bother you guys one more time, I do have one more
issue to solve. LOL
And this time it's guaranteed not to be an SELinux issue. Because I tried
running haproxy with SELInux on and off this time.
But what's happening now, is that HA/Proxy is not creating the http port
for the 'stats' interface. I've setup stats to listen on port 80. But for
some reason that's not happening.
Here's my config one more time, with the trouble part in bold:
global
log 127.0.0.1 local0 notice
user haproxy
group haproxy
defaults
log global
retries 2
timeout connect 3000
timeout server 5000
timeout client 5000
listen mysql-cluster
bind 0.0.0.0:3306
mode tcp
option mysql-check user haproxy_check
balance roundrobin
server mysql-1 52.3.28.48:3306 check
server mysql-2 52.2.0.176:3306 check
*listen 0.0.0.0:80 http://0.0.0.0:80mode httpstats enable
stats uri /stats realm Strictly\ Privatestats auth admin:secret*
Currently haproxy is listening on the first port specified* - 3306 - *but
not listening on port 80.
Observe:
[root@ha1:/etc/haproxy] #lsof -i :3306
COMMAND PIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME
*haproxy 11653 haproxy4u IPv4 7145270 0t0 TCP *:mysql (LISTEN)*
[root@ha1:/etc/haproxy] #lsof -i :80
[root@ha1:/etc/haproxy] #
[root@ha1:/etc/haproxy] #telnet localhost 80
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
Port 80 simply isn't listening.
And this time, I can't blame it on SELinux being on:
[root@ha1:/etc/haproxy] #getenforce
Permissive
I've grepped thru /var/log/messages but not turned up any clues to this one.
And I really would like to get the stats interface up and running.
Any thoughts here? I'm wondering what I can do to get stats working.
Thanks,
Tim
On Fri, Jul 24, 2015 at 10:52 PM, Gmail longwuy...@gmail.com wrote:
Nice.
Do you use selinux in prod.
regards,
; Yuan
On 07/25/2015 09:17 AM, Tim Dunphy wrote:
Bingo!!!
The problem was with SELinux. Not sure what took me so long to think of
it...!!!
So set the mysql listener back to port 3306. Turned off SELinux with
setenforce 0. Then it started right up!!! And port 3306 was listening.
Then I consulted with audit2why and saw the following:
type=AVC msg=audit(1437786617.963:28856863): avc: denied { name_connect
}
for pid=29175 comm=haproxy dest=3306
scontext=system_u:system_r:haproxy_t:s0
tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket
Was caused by:
The boolean haproxy_connect_any was set incorrectly.
Description:
Allow haproxy to connect any
Allow access by executing:
# *setsebool -P haproxy_connect_any 1*
I just ran that command you see above in bold, and then all was right with
the world.
[root@ha1:/etc/haproxy] #systemctl status haproxy
haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)
Active: active (running) since Sat 2015-07-25 01:14:53 UTC; 33s ago
Main PID: 30618 (haproxy-systemd)
CGroup: /system.slice/haproxy.service
├─30618 /usr/sbin/haproxy-systemd-wrapper -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid
├─30619 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
└─30620 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
Jul 25 01:14:53 ha1 systemd[1]: Starting HAProxy Load Balancer...
Jul 25 01:14:53 ha1 systemd[1]: Started HAProxy Load Balancer.
Jul 25 01:14:53 ha1 haproxy-systemd-wrapper[30618]:
haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
[root@ha1:/etc/haproxy] #lsof -i :3306
COMMAND PIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME
haproxy 30620 haproxy1u IPv4 7075172 0t0 TCP
ha1.example.com:55499-ec2-52-2-0-xxx.compute-1.amazonaws.com:mysql
(SYN_SENT)
haproxy 30620 haproxy4u IPv4 7074731 0t0 TCP *:mysql (LISTEN)
Thanks for nudging me in the right direction. All I had to hear was the
word 'selinux' and from there it all fell into place!
Thanks!!
Tim
On Fri, Jul 24, 2015 at 8:20 PM, Gmail longwuy...@gmail.com wrote:
I could be completely wrong here and I am curious to know the answer
myself. Please don't take this as a solution, just my thoughts.
First, you can not use backend ip-address of 10.x.x.x subnet because each
account's VPC is seggregated. If you do want to use 10.X.X.X ipadress you
have to setup a inter VPC endpoint in AWS. I would just use EIP.
For the port 3306, try to use nc to listen on that port or iperf. Do yo
uhave iptables turned on.
I would check systemctl -l status
Re: haproxy can't bind to mysql port
You need to run haproxy as root to bind to ports lower than 1024
On 25/07/2015 1:36 PM, Tim Dunphy bluethu...@gmail.com wrote:
Hi Yuan,
Nice.
Do you use selinux in prod.
regards,
; Yuan
Yep! Actually I use it every chance I get. Prod/stage/dev and my own hobby
environments. And right now actually what I was discussing was a hobby
environment.
And actually if I could bother you guys one more time, I do have one more
issue to solve. LOL
And this time it's guaranteed not to be an SELinux issue. Because I tried
running haproxy with SELInux on and off this time.
But what's happening now, is that HA/Proxy is not creating the http port
for the 'stats' interface. I've setup stats to listen on port 80. But for
some reason that's not happening.
Here's my config one more time, with the trouble part in bold:
global
log 127.0.0.1 local0 notice
user haproxy
group haproxy
defaults
log global
retries 2
timeout connect 3000
timeout server 5000
timeout client 5000
listen mysql-cluster
bind 0.0.0.0:3306
mode tcp
option mysql-check user haproxy_check
balance roundrobin
server mysql-1 52.3.28.48:3306 check
server mysql-2 52.2.0.176:3306 check
*listen 0.0.0.0:80 http://0.0.0.0:80mode httpstats enable
stats uri /stats realm Strictly\ Privatestats auth admin:secret*
Currently haproxy is listening on the first port specified* - 3306 - *but
not listening on port 80.
Observe:
[root@ha1:/etc/haproxy] #lsof -i :3306
COMMAND PIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME
*haproxy 11653 haproxy4u IPv4 7145270 0t0 TCP *:mysql (LISTEN)*
[root@ha1:/etc/haproxy] #lsof -i :80
[root@ha1:/etc/haproxy] #
[root@ha1:/etc/haproxy] #telnet localhost 80
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
Port 80 simply isn't listening.
And this time, I can't blame it on SELinux being on:
[root@ha1:/etc/haproxy] #getenforce
Permissive
I've grepped thru /var/log/messages but not turned up any clues to this
one.
And I really would like to get the stats interface up and running.
Any thoughts here? I'm wondering what I can do to get stats working.
Thanks,
Tim
On Fri, Jul 24, 2015 at 10:52 PM, Gmail longwuy...@gmail.com wrote:
Nice.
Do you use selinux in prod.
regards,
; Yuan
On 07/25/2015 09:17 AM, Tim Dunphy wrote:
Bingo!!!
The problem was with SELinux. Not sure what took me so long to think of
it...!!!
So set the mysql listener back to port 3306. Turned off SELinux with
setenforce 0. Then it started right up!!! And port 3306 was listening.
Then I consulted with audit2why and saw the following:
type=AVC msg=audit(1437786617.963:28856863): avc: denied {
name_connect }
for pid=29175 comm=haproxy dest=3306
scontext=system_u:system_r:haproxy_t:s0
tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket
Was caused by:
The boolean haproxy_connect_any was set incorrectly.
Description:
Allow haproxy to connect any
Allow access by executing:
# *setsebool -P haproxy_connect_any 1*
I just ran that command you see above in bold, and then all was right
with
the world.
[root@ha1:/etc/haproxy] #systemctl status haproxy
haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)
Active: active (running) since Sat 2015-07-25 01:14:53 UTC; 33s ago
Main PID: 30618 (haproxy-systemd)
CGroup: /system.slice/haproxy.service
├─30618 /usr/sbin/haproxy-systemd-wrapper -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid
├─30619 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
└─30620 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
Jul 25 01:14:53 ha1 systemd[1]: Starting HAProxy Load Balancer...
Jul 25 01:14:53 ha1 systemd[1]: Started HAProxy Load Balancer.
Jul 25 01:14:53 ha1 haproxy-systemd-wrapper[30618]:
haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
[root@ha1:/etc/haproxy] #lsof -i :3306
COMMAND PIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME
haproxy 30620 haproxy1u IPv4 7075172 0t0 TCP
ha1.example.com:55499-ec2-52-2-0-xxx.compute-1.amazonaws.com:mysql
(SYN_SENT)
haproxy 30620 haproxy4u IPv4 7074731 0t0 TCP *:mysql (LISTEN)
Thanks for nudging me in the right direction. All I had to hear was the
word 'selinux' and from there it all fell into place!
Thanks!!
Tim
On Fri, Jul 24, 2015 at 8:20 PM, Gmail longwuy...@gmail.com wrote:
I could be completely wrong here and I am curious to know the answer
myself. Please don't take this as a solution, just my thoughts.
First, you can not use backend ip-address of 10.x.x.x subnet because
each
account's VPC is seggregated. If you do want to use 10.X.X.X ipadress
you
have to setup a
Re: haproxy can't bind to mysql port
maybe something here http://lnxmon.com/haproxy/
Thanks,
; Yuan
On 07/25/2015 12:10 PM, Igor Cicimov wrote:
You need to run haproxy as root to bind to ports lower than 1024
On 25/07/2015 1:36 PM, Tim Dunphy bluethu...@gmail.com wrote:
Hi Yuan,
Nice.
Do you use selinux in prod.
regards,
; Yuan
Yep! Actually I use it every chance I get. Prod/stage/dev and my own hobby
environments. And right now actually what I was discussing was a hobby
environment.
And actually if I could bother you guys one more time, I do have one more
issue to solve. LOL
And this time it's guaranteed not to be an SELinux issue. Because I tried
running haproxy with SELInux on and off this time.
But what's happening now, is that HA/Proxy is not creating the http port
for the 'stats' interface. I've setup stats to listen on port 80. But for
some reason that's not happening.
Here's my config one more time, with the trouble part in bold:
global
log 127.0.0.1 local0 notice
user haproxy
group haproxy
defaults
log global
retries 2
timeout connect 3000
timeout server 5000
timeout client 5000
listen mysql-cluster
bind 0.0.0.0:3306
mode tcp
option mysql-check user haproxy_check
balance roundrobin
server mysql-1 52.3.28.48:3306 check
server mysql-2 52.2.0.176:3306 check
*listen 0.0.0.0:80 http://0.0.0.0:80mode httpstats enable
stats uri /stats realm Strictly\ Privatestats auth admin:secret*
Currently haproxy is listening on the first port specified* - 3306 - *but
not listening on port 80.
Observe:
[root@ha1:/etc/haproxy] #lsof -i :3306
COMMAND PIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME
*haproxy 11653 haproxy4u IPv4 7145270 0t0 TCP *:mysql (LISTEN)*
[root@ha1:/etc/haproxy] #lsof -i :80
[root@ha1:/etc/haproxy] #
[root@ha1:/etc/haproxy] #telnet localhost 80
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
Port 80 simply isn't listening.
And this time, I can't blame it on SELinux being on:
[root@ha1:/etc/haproxy] #getenforce
Permissive
I've grepped thru /var/log/messages but not turned up any clues to this
one.
And I really would like to get the stats interface up and running.
Any thoughts here? I'm wondering what I can do to get stats working.
Thanks,
Tim
On Fri, Jul 24, 2015 at 10:52 PM, Gmail longwuy...@gmail.com wrote:
Nice.
Do you use selinux in prod.
regards,
; Yuan
On 07/25/2015 09:17 AM, Tim Dunphy wrote:
Bingo!!!
The problem was with SELinux. Not sure what took me so long to think of
it...!!!
So set the mysql listener back to port 3306. Turned off SELinux with
setenforce 0. Then it started right up!!! And port 3306 was listening.
Then I consulted with audit2why and saw the following:
type=AVC msg=audit(1437786617.963:28856863): avc: denied {
name_connect }
for pid=29175 comm=haproxy dest=3306
scontext=system_u:system_r:haproxy_t:s0
tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket
Was caused by:
The boolean haproxy_connect_any was set incorrectly.
Description:
Allow haproxy to connect any
Allow access by executing:
# *setsebool -P haproxy_connect_any 1*
I just ran that command you see above in bold, and then all was right
with
the world.
[root@ha1:/etc/haproxy] #systemctl status haproxy
haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)
Active: active (running) since Sat 2015-07-25 01:14:53 UTC; 33s ago
Main PID: 30618 (haproxy-systemd)
CGroup: /system.slice/haproxy.service
├─30618 /usr/sbin/haproxy-systemd-wrapper -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid
├─30619 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
└─30620 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
Jul 25 01:14:53 ha1 systemd[1]: Starting HAProxy Load Balancer...
Jul 25 01:14:53 ha1 systemd[1]: Started HAProxy Load Balancer.
Jul 25 01:14:53 ha1 haproxy-systemd-wrapper[30618]:
haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
[root@ha1:/etc/haproxy] #lsof -i :3306
COMMAND PIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME
haproxy 30620 haproxy1u IPv4 7075172 0t0 TCP
ha1.example.com:55499-ec2-52-2-0-xxx.compute-1.amazonaws.com:mysql
(SYN_SENT)
haproxy 30620 haproxy4u IPv4 7074731 0t0 TCP *:mysql (LISTEN)
Thanks for nudging me in the right direction. All I had to hear was the
word 'selinux' and from there it all fell into place!
Thanks!!
Tim
On Fri, Jul 24, 2015 at 8:20 PM, Gmail longwuy...@gmail.com wrote:
I could be completely wrong here and I am curious to know the answer
myself. Please don't take this as a solution, just my thoughts.
First, you can not use backend ip-address of 10.x.x.x subnet because
each
account's VPC is seggregated. If you do want to use 10.X.X.X
Re: haproxy can't bind to mysql port
Bingo!!!
The problem was with SELinux. Not sure what took me so long to think of
it...!!!
So set the mysql listener back to port 3306. Turned off SELinux with
setenforce 0. Then it started right up!!! And port 3306 was listening.
Then I consulted with audit2why and saw the following:
type=AVC msg=audit(1437786617.963:28856863): avc: denied { name_connect }
for pid=29175 comm=haproxy dest=3306
scontext=system_u:system_r:haproxy_t:s0
tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket
Was caused by:
The boolean haproxy_connect_any was set incorrectly.
Description:
Allow haproxy to connect any
Allow access by executing:
# *setsebool -P haproxy_connect_any 1*
I just ran that command you see above in bold, and then all was right with
the world.
[root@ha1:/etc/haproxy] #systemctl status haproxy
haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)
Active: active (running) since Sat 2015-07-25 01:14:53 UTC; 33s ago
Main PID: 30618 (haproxy-systemd)
CGroup: /system.slice/haproxy.service
├─30618 /usr/sbin/haproxy-systemd-wrapper -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid
├─30619 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
└─30620 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
Jul 25 01:14:53 ha1 systemd[1]: Starting HAProxy Load Balancer...
Jul 25 01:14:53 ha1 systemd[1]: Started HAProxy Load Balancer.
Jul 25 01:14:53 ha1 haproxy-systemd-wrapper[30618]:
haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
[root@ha1:/etc/haproxy] #lsof -i :3306
COMMAND PIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME
haproxy 30620 haproxy1u IPv4 7075172 0t0 TCP
ha1.example.com:55499-ec2-52-2-0-xxx.compute-1.amazonaws.com:mysql
(SYN_SENT)
haproxy 30620 haproxy4u IPv4 7074731 0t0 TCP *:mysql (LISTEN)
Thanks for nudging me in the right direction. All I had to hear was the
word 'selinux' and from there it all fell into place!
Thanks!!
Tim
On Fri, Jul 24, 2015 at 8:20 PM, Gmail longwuy...@gmail.com wrote:
I could be completely wrong here and I am curious to know the answer
myself. Please don't take this as a solution, just my thoughts.
First, you can not use backend ip-address of 10.x.x.x subnet because each
account's VPC is seggregated. If you do want to use 10.X.X.X ipadress you
have to setup a inter VPC endpoint in AWS. I would just use EIP.
For the port 3306, try to use nc to listen on that port or iperf. Do yo
uhave iptables turned on.
I would check systemctl -l status haproxy.service
I would check lsof -i why can't bind to 3306 on loopback ipaddress.
I would check iptables or selinux preventing the bind.
It wil be interesting to know the source ipaddress of MySQL client ec2
instance.
Interesting if you can Copy/paste output of telnet
haproxynode_ipaddress 3306 from mysql client ec2 instance , here.
Interesting if you can Copy/paste output of telnet 10.10.10.10 3306
from haproxy ec2 instances, here.
Interesting if you can Copy/paste output of telnet 10.10.10.11 3306
from haproxy ec2 instances, here.
I I was doing this, maybe I would consider testing something like ;
..
frontend mysql_lb_fe 0.0.0.0:3306
acl host_myql_lb hdr(host) -i mysql-lb
..
..
use_backend mysql_lb_backend if host mysql_lb
..
..
backend mysql_lb_be
..
..
option mysql-check user haproxy_check
balance roundrobin
server mysql-1 10.10.10.10:3306 check
server mysql-2 10.10.10.11:3306 check
Thanks,
; Yuan
On 07/25/2015 06:41 AM, Tim Dunphy wrote:
Hello Nenad,
Jul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: [ALERT]
204/034418 (25035) : *Starting proxy mysql-cluster: cannot bind
s...:3306]*
Nothing listening on the port I'm trying to bind to: 3306
[root@ha1:~] #ss -lpt | fgrep 3306
[root@ha1:~] #lsof -i :3306
[root@ha1:~] #netstat -tulpn | grep -i listen | grep 3306
[root@ha1:~] #
While we're on the subject of listening ports, here's a list of all
listening ports on the haproxy host:
[root@ha1:~] #netstat -tulpn | grep -i listen
tcp0 0 0.0.0.0:35145 0.0.0.0:*
LISTEN -
tcp0 0 0.0.0.0:56814 0.0.0.0:*
LISTEN 16346/rpc.statd
tcp0 0 0.0.0.0:111 0.0.0.0:*
LISTEN 16455/rpcbind
tcp0 0 0.0.0.0:22 0.0.0.0:*
LISTEN 16396/sshd
tcp6 0 0 :::49349:::*
LISTEN 16346/rpc.statd
tcp6 0 0 :::111 :::*
LISTEN 16455/rpcbind
tcp6 0 0 :::47314:::*
LISTEN -
tcp6 0 0 :::22 :::*
LISTEN 16396/sshd
I thought I was beginning to understand this problem. That haproxy was
trying to bind on port 3306 from the mysql host on
Re: haproxy can't bind to mysql port
On Fri, Jul 24, 2015 at 1:46 PM, Tim Dunphy bluethu...@gmail.com wrote: Hi all, I'm attempting to setup mysql load balancing using HA/Proxy. Seemed pretty straight forward at first. I'm using Amazon ec2 for all nodes. First I made sure that the haproxy nodes could contact the mysql boxes by opening up the security group from the mysql boxes to the haproxy ones on port 3306. How did you do that? By putting the haproxy's security group or the haproxy's ip to the mysql group inbound rule? If IP which one is that? I setup the following config: global log 127.0.0.1 local0 notice user haproxy group haproxy defaults log global retries 2 timeout connect 3000 timeout server 5000 timeout client 5000 listen mysql-cluster bind 127.0.0.1:3306 mode tcp option mysql-check user haproxy_check balance roundrobin server mysql-1 10.10.10.10:3306 check server mysql-2 10.10.10.11:3306 check listen 0.0.0.0:80 mode http stats enable stats uri / stats realm Strictly\ Private stats auth admin:secret And ensured that haproxy could bind to non local IP's: Sorry but which non local IP is that? How many interfaces haproxy has? Is it connected to the 10.10.10.0/24 network at all? Looks to me you are trying to use VIP's or something which does not work in same way as in normal lan's. Don't forget that in AWS we are dealing with SDN's so giving l0 or any other interface a second IP address localy on the instance using ip tool lets say will simply not work. That IP is not visible to the SDN and the interface will never send or receive any traffic. You need that IP allocated to the haproxy interface (no option for l0 here) via EC2 console or aws cli tool. [root@ha1:/etc/haproxy] #grep ipv4 /etc/sysctl.conf net.ipv4.ip_nonlocal_bind=1 [root@ha1:/etc/haproxy] #sysctl -p net.ipv4.ip_nonlocal_bind = 1 Yet when I try to start up haproxy I get the following result: [root@ha1:/etc/haproxy] #systemctl status haproxy haproxy.service - HAProxy Load Balancer Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled) Active: inactive (dead) since Fri 2015-07-24 03:44:18 UTC; 9s ago Process: 25034 ExecStart=/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid (code=exited, status=0/SUCCESS) Main PID: 25034 (code=exited, status=0/SUCCESS) Jul 24 03:44:18 ha1 systemd[1]: Starting HAProxy Load Balancer... Jul 24 03:44:18 ha1 systemd[1]: Started HAProxy Load Balancer. Jul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/hap...id -Ds Jul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: [ALERT] 204/034418 (25035) : *Starting proxy mysql-cluster: cannot bind s...:3306]* Jul 24 03:44:18 ha1* h*aproxy-systemd-wrapper[25034]: haproxy-systemd-wrapper: exit, haproxy RC=256 Hint: Some lines were ellipsized, use -l to show in full. So it seems that haproxy is expecting to have mysql already listening on port 3306. But mysql is runnign on two external nodes with port 3306 open to the two haproxy machines. What am I doing wrong? And how can I get this to work? Thanks, TIm -- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B -- Igor Cicimov | DevOps p. +61 (0) 433 078 728 e. ig...@encompasscorporation.com http://encompasscorporation.com/ w*.* encompasscorporation.com a. Level 4, 65 York Street, Sydney 2000
