Tim Dunphy schreef op 25-7-2015 om 17:00:
You need to run haproxy as root to bind to ports lower than 1024
I tried running haproxy as root/root:
[root@ha1:/etc/haproxy] #egrep "user|group" haproxy.cfg| grep -v option
user root
group root
user and group dont affect what user haproxy starts under. binding
happens before these get applied.
Then restarted the service. No difference!
[root@ha1:/etc/haproxy] #systemctl restart haproxy
dont use systemctl for testing..
just run: haproxy -f haproxy.cfg
[root@ha1:/etc/haproxy] #lsof -i :80
[root@ha1:/etc/haproxy] #getenforce
Permissive
Thanks for the suggestion anyway!
On Sat, Jul 25, 2015 at 12:10 AM, Igor Cicimov
<ig...@encompasscorporation.com
<mailto:ig...@encompasscorporation.com>> wrote:
You need to run haproxy as root to bind to ports lower than 1024
On 25/07/2015 1:36 PM, "Tim Dunphy" <bluethu...@gmail.com
<mailto:bluethu...@gmail.com>> wrote:
Hi Yuan,
Nice.
Do you use selinux in prod.
regards,
; Yuan
Yep! Actually I use it every chance I get. Prod/stage/dev and
my own hobby environments. And right now actually what I was
discussing was a hobby environment.
And actually if I could bother you guys one more time, I do
have one more issue to solve. LOL
And this time it's guaranteed not to be an SELinux issue.
Because I tried running haproxy with SELInux on and off this time.
But what's happening now, is that HA/Proxy is not creating the
http port for the 'stats' interface. I've setup stats to
listen on port 80. But for some reason that's not happening.
Here's my config one more time, with the trouble part in bold:
global
log 127.0.0.1 local0 notice
user haproxy
group haproxy
defaults
log global
retries 2
timeout connect 3000
timeout server 5000
timeout client 5000
listen mysql-cluster
bind 0.0.0.0:3306 <http://0.0.0.0:3306>
mode tcp
option mysql-check user haproxy_check
balance roundrobin
server mysql-1 52.3.28.48:3306 <http://52.3.28.48:3306> check
server mysql-2 52.2.0.176:3306 <http://52.2.0.176:3306> check
*
listen 0.0.0.0:80 <http://0.0.0.0:80>
mode http
stats enable
stats uri /
stats realm Strictly\ Private
stats auth admin:secret
*
Currently haproxy is listening on the first port specified*-
3306 - *but not listening on port 80.
Observe:
[root@ha1:/etc/haproxy] #lsof -i :3306
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
*haproxy 11653 haproxy 4u IPv4 7145270 0t0 TCP
*:mysql (LISTEN)*
[root@ha1:/etc/haproxy] #lsof -i :80
[root@ha1:/etc/haproxy] #
[root@ha1:/etc/haproxy] #telnet localhost 80
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1 <http://127.0.0.1>:
Connection refused
Port 80 simply isn't listening.
And this time, I can't blame it on SELinux being on:
[root@ha1:/etc/haproxy] #getenforce
Permissive
I've grepped thru /var/log/messages but not turned up any
clues to this one.
And I really would like to get the stats interface up and
running.
Any thoughts here? I'm wondering what I can do to get stats
working.
Thanks,
Tim
On Fri, Jul 24, 2015 at 10:52 PM, Gmail <longwuy...@gmail.com
<mailto:longwuy...@gmail.com>> wrote:
Nice.
Do you use selinux in prod.
regards,
; Yuan
On 07/25/2015 09:17 AM, Tim Dunphy wrote:
Bingo!!!
The problem was with SELinux. Not sure what took me so
long to think of
it...!!!
So set the mysql listener back to port 3306. Turned
off SELinux with
setenforce 0. Then it started right up!!! And port
3306 was listening.
Then I consulted with audit2why and saw the following:
type=AVC msg=audit(1437786617.963:28856863): avc:
denied { name_connect }
for pid=29175 comm="haproxy" dest=3306
scontext=system_u:system_r:haproxy_t:s0
tcontext=system_u:object_r:mysqld_port_t:s0
tclass=tcp_socket
Was caused by:
The boolean haproxy_connect_any was set
incorrectly.
Description:
Allow haproxy to connect any
Allow access by executing:
# *setsebool -P haproxy_connect_any 1*
I just ran that command you see above in bold, and
then all was right with
the world.
[root@ha1:/etc/haproxy] #systemctl status haproxy
haproxy.service - HAProxy Load Balancer
Loaded: loaded
(/usr/lib/systemd/system/haproxy.service; disabled)
Active: active (running) since Sat 2015-07-25
01:14:53 UTC; 33s ago
Main PID: 30618 (haproxy-systemd)
CGroup: /system.slice/haproxy.service
├─30618 /usr/sbin/haproxy-systemd-wrapper -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid
├─30619 /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
└─30620 /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
Jul 25 01:14:53 ha1 systemd[1]: Starting HAProxy Load
Balancer...
Jul 25 01:14:53 ha1 systemd[1]: Started HAProxy Load
Balancer.
Jul 25 01:14:53 ha1 haproxy-systemd-wrapper[30618]:
haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
[root@ha1:/etc/haproxy] #lsof -i :3306
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE
NAME
haproxy 30620 haproxy 1u IPv4 7075172 0t0 TCP
ha1.example.com:55499->ec2-52-2-0-xxx.compute-1.amazonaws.com:mysql
(SYN_SENT)
haproxy 30620 haproxy 4u IPv4 7074731 0t0
TCP *:mysql (LISTEN)
Thanks for nudging me in the right direction. All I
had to hear was the
word 'selinux' and from there it all fell into place!
Thanks!!
Tim
On Fri, Jul 24, 2015 at 8:20 PM, Gmail
<longwuy...@gmail.com <mailto:longwuy...@gmail.com>>
wrote:
I could be completely wrong here and I am curious
to know the answer
myself. Please don't take this as a solution, just
my thoughts.
First, you can not use backend ip-address of
10.x.x.x subnet because each
account's VPC is seggregated. If you do want to
use 10.X.X.X ipadress you
have to setup a inter VPC endpoint in AWS. I would
just use EIP.
For the port 3306, try to use nc to listen on that
port or iperf. Do yo
uhave iptables turned on.
I would check "systemctl -l status haproxy.service"
I would check lsof -i why can't bind to 3306 on
loopback ipaddress.
I would check iptables or selinux preventing the bind.
It wil be interesting to know the source ipaddress
of MySQL client ec2
instance.
Interesting if you can Copy/paste output of "telnet
<haproxynode_ipaddress> 3306" from mysql client
ec2 instance , here.
Interesting if you can Copy/paste output of
"telnet 10.10.10.10 3306"
from haproxy ec2 instances, here.
Interesting if you can Copy/paste output of
"telnet 10.10.10.11 3306"
from haproxy ec2 instances, here.
I I was doing this, maybe I would consider testing
something like ;
..
frontend mysql_lb_fe 0.0.0.0:3306
<http://0.0.0.0:3306>
....
acl host_myql_lb hdr(host) -i mysql-lb
..
..
use_backend mysql_lb_backend if host mysql_lb
..
..
backend mysql_lb_be
..
..
option mysql-check user haproxy_check
balance roundrobin
server mysql-1 10.10.10.10:3306
<http://10.10.10.10:3306> check
server mysql-2 10.10.10.11:3306
<http://10.10.10.11:3306> check
Thanks,
; Yuan
On 07/25/2015 06:41 AM, Tim Dunphy wrote:
Hello Nenad,
Jul 24 03:44:18 ha1
haproxy-systemd-wrapper[25034]: [ALERT]
204/034418 (25035) : *Starting
proxy mysql-cluster: cannot bind
s...:3306]*
Nothing listening on the port I'm trying to
bind to: 3306
[root@ha1:~] #ss -lpt | fgrep 3306
[root@ha1:~] #lsof -i :3306
[root@ha1:~] #netstat -tulpn | grep -i listen
| grep 3306
[root@ha1:~] #
While we're on the subject of listening ports,
here's a list of all
listening ports on the haproxy host:
[root@ha1:~] #netstat -tulpn | grep -i listen
tcp 0 0 0.0.0.0:35145
<http://0.0.0.0:35145> 0.0.0.0:*
LISTEN -
tcp 0 0 0.0.0.0:56814
<http://0.0.0.0:56814> 0.0.0.0:*
LISTEN 16346/rpc.statd
tcp 0 0 0.0.0.0:111
<http://0.0.0.0:111> 0.0.0.0:*
LISTEN 16455/rpcbind
tcp 0 0 0.0.0.0:22
<http://0.0.0.0:22> 0.0.0.0:*
LISTEN 16396/sshd
tcp6 0 0 :::49349 :::*
LISTEN 16346/rpc.statd
tcp6 0 0 :::111 :::*
LISTEN 16455/rpcbind
tcp6 0 0 :::47314 :::*
LISTEN -
tcp6 0 0 :::22 :::*
LISTEN 16396/sshd
I thought I was beginning to understand this
problem. That haproxy was
trying to bind on port 3306 from the mysql
host on another machine. But
come to think of it, that doesn't make a lot
of sense.
Because I already have haproxy setup for some
web servers, and there it
creates port 80 on the haproxy node. It's not
trying to connect to a
foreign source. Not sure where I got that idea!!
I also tried binding the mysql section to
another port that wasn't in use.
I tried port 3307,3308. I even tried binding
the mysql section of the
config to a weird port I just grabbed off of
the top of my head. I tried
binding it to port 4444.
And there I still got a bind error:
[ALERT] 204/223303 (13081) : Starting proxy
mysql-cluster: cannot bind
socket [0.0.0.0:4444 <http://0.0.0.0:4444>]
Now watch this!! If I bind the mysql
section to port 80 instead of any
other port.. haproxy starts up without complaint!
listen mysql-cluster
bind 0.0.0.0:80 <http://0.0.0.0:80>
mode tcp
option mysql-check user haproxy_check
balance roundrobin
server mysql-1 10.0.0.xxx :3306 check
server mysql-2 10.0.0.xxx:3306 check
[root@ha1:/etc/haproxy] #systemctl status haproxy
haproxy.service - HAProxy Load Balancer
Loaded: loaded
(/usr/lib/systemd/system/haproxy.service; enabled)
Active: active (running) since Fri
2015-07-24 22:35:03 UTC; 4s ago
Main PID: 13213 (haproxy-systemd)
CGroup: /system.slice/haproxy.service
├─13213
/usr/sbin/haproxy-systemd-wrapper -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid
├─13214 /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
└─13215 /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p
/run/haproxy.pid -Ds
Jul 24 22:35:03 ha1 systemd[1]: Starting
HAProxy Load Balancer...
*Jul 24 22:35:03 ha1 systemd[1]: Started
HAProxy Load Balancer.*
Jul 24 22:35:03 ha1
haproxy-systemd-wrapper[13213]:
haproxy-systemd-wrapper: executing
/usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
Ok. What...the...heck!!
So why do you think that haproxy is only happy
starting up on port 80? I
would think that I should be able to specify
any arbitrary port for it to
listen on in a 'listen' sub-block.
I guess I could have my app contact the
database using port 80. But that's
a little... weird. I installed haproxy using
yum from the 'updates'
repository. Is there any reason anyone can
think of as to why haproxy
refuses to start on any port other than port 80??
Thanks,
Tim
On Fri, Jul 24, 2015 at 4:59 PM, Nenad
Merdanovic <ni...@nimzo.info
<mailto:ni...@nimzo.info>>
wrote:
Hello Tim,
On Fri, Jul 24, 2015 at 1:46 PM,
Tim Dunphy <bluethu...@gmail.com
<mailto:bluethu...@gmail.com>
<mailto:bluethu...@gmail.com
<mailto:bluethu...@gmail.com>>> wrote:
listen mysql-cluster
bind 127.0.0.1:3306
<http://127.0.0.1:3306>
<http://127.0.0.1:3306>
mode tcp
option mysql-check user
haproxy_check
balance roundrobin
server mysql-1
10.10.10.10:3306
<http://10.10.10.10:3306>
<http://10.10.10.10:3306>
check
server mysql-2
10.10.10.11:3306
<http://10.10.10.11:3306>
<http://10.10.10.11:3306>
check
Jul 24 03:44:18 ha1
haproxy-systemd-wrapper[25034]: [ALERT]
204/034418 (25035) :
*Starting proxy mysql-cluster: cannot bind
s...:3306]*
Can you check if something is listening on
127.0.0.1:3306 <http://127.0.0.1:3306>
(netstat, ss,
lsof)? For example:
ss -lpt | fgrep 3306
Regards,
Nenad
--
GPG me!!
gpg --keyserver pool.sks-keyservers.net
<http://pool.sks-keyservers.net> --recv-keys F186197B
--
GPG me!!
gpg --keyserver pool.sks-keyservers.net
<http://pool.sks-keyservers.net> --recv-keys F186197B
