Hi Yuan,

Nice.
> Do you use selinux in prod.
> regards,
> ; Yuan


Yep! Actually I use it every chance I get. Prod/stage/dev and my own hobby
environments. And right now actually what I was discussing was a hobby
environment.

And actually if I could bother you guys one more time, I do have one more
issue to solve. LOL

And this time it's guaranteed not to be an SELinux issue. Because I tried
running haproxy with SELInux on and off this time.

But what's happening now, is that HA/Proxy is not creating the http port
for the 'stats' interface. I've setup stats to listen on port 80. But for
some reason that's not happening.

Here's my config one more time, with the trouble part in bold:

global
    log 127.0.0.1 local0 notice
    user haproxy
    group haproxy

defaults
    log global
    retries 2
    timeout connect 3000
    timeout server 5000
    timeout client 5000

listen mysql-cluster
    bind 0.0.0.0:3306
    mode tcp
    option mysql-check user haproxy_check
    balance roundrobin
    server mysql-1 52.3.28.48:3306 check
    server mysql-2 52.2.0.176:3306 check








*listen 0.0.0.0:80 <http://0.0.0.0:80>    mode http    stats enable
stats uri /    stats realm Strictly\ Private    stats auth admin:secret*
Currently haproxy is listening on the first port specified* - 3306 - *but
not listening on port 80.

Observe:

[root@ha1:/etc/haproxy] #lsof -i :3306
COMMAND   PID    USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
*haproxy 11653 haproxy    4u  IPv4 7145270      0t0  TCP *:mysql (LISTEN)*

[root@ha1:/etc/haproxy] #lsof -i :80
[root@ha1:/etc/haproxy] #

[root@ha1:/etc/haproxy] #telnet localhost 80
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

Port 80 simply isn't listening.

And this time, I can't blame it on SELinux being on:

[root@ha1:/etc/haproxy] #getenforce
Permissive

I've grepped thru /var/log/messages but not turned up any clues to this one.

And I really would like to get the stats interface up and running.

Any thoughts here? I'm wondering what I can do to get stats working.

Thanks,
Tim



On Fri, Jul 24, 2015 at 10:52 PM, Gmail <longwuy...@gmail.com> wrote:

> Nice.
> Do you use selinux in prod.
> regards,
> ; Yuan
>
> On 07/25/2015 09:17 AM, Tim Dunphy wrote:
>
>> Bingo!!!
>>
>> The problem was with SELinux. Not sure what took me so long to think of
>> it...!!!
>>
>> So set the mysql listener back to port 3306. Turned off SELinux with
>> setenforce 0. Then it started right up!!! And port 3306 was listening.
>>
>> Then I consulted with audit2why and saw the following:
>>
>> type=AVC msg=audit(1437786617.963:28856863): avc:  denied  { name_connect
>> }
>> for  pid=29175 comm="haproxy" dest=3306
>> scontext=system_u:system_r:haproxy_t:s0
>> tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket
>>
>>          Was caused by:
>>          The boolean haproxy_connect_any was set incorrectly.
>>          Description:
>>          Allow haproxy to connect any
>>
>>          Allow access by executing:
>>          # *setsebool -P haproxy_connect_any 1*
>>
>>
>> I just ran that command you see above in bold, and then all was right with
>> the world.
>>
>> [root@ha1:/etc/haproxy] #systemctl status haproxy
>> haproxy.service - HAProxy Load Balancer
>>     Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled)
>>     Active: active (running) since Sat 2015-07-25 01:14:53 UTC; 33s ago
>>   Main PID: 30618 (haproxy-systemd)
>>     CGroup: /system.slice/haproxy.service
>>             ├─30618 /usr/sbin/haproxy-systemd-wrapper -f
>> /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
>>             ├─30619 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
>> /run/haproxy.pid -Ds
>>             └─30620 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
>> /run/haproxy.pid -Ds
>>
>> Jul 25 01:14:53 ha1 systemd[1]: Starting HAProxy Load Balancer...
>> Jul 25 01:14:53 ha1 systemd[1]: Started HAProxy Load Balancer.
>> Jul 25 01:14:53 ha1 haproxy-systemd-wrapper[30618]:
>> haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f
>> /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
>>
>> [root@ha1:/etc/haproxy] #lsof -i :3306
>> COMMAND   PID    USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
>> haproxy 30620 haproxy    1u  IPv4 7075172      0t0  TCP
>> ha1.example.com:55499->ec2-52-2-0-xxx.compute-1.amazonaws.com:mysql
>> (SYN_SENT)
>> haproxy 30620 haproxy    4u  IPv4 7074731      0t0  TCP *:mysql (LISTEN)
>>
>>
>> Thanks for nudging me in the right direction. All I had to hear was the
>> word 'selinux' and from there it all fell into place!
>>
>> Thanks!!
>> Tim
>>
>> On Fri, Jul 24, 2015 at 8:20 PM, Gmail <longwuy...@gmail.com> wrote:
>>
>>  I could be completely wrong here and I am curious to know the answer
>>> myself. Please don't take this as a solution, just my thoughts.
>>>
>>> First, you can not use backend ip-address of 10.x.x.x subnet because each
>>> account's VPC is seggregated. If you do want to use 10.X.X.X ipadress you
>>> have to setup a inter VPC endpoint in AWS. I would just use EIP.
>>>
>>> For the port 3306, try to use nc to listen on that port or iperf. Do yo
>>> uhave iptables turned on.
>>>
>>> I would check "systemctl -l status haproxy.service"
>>>
>>> I would check lsof -i why can't bind to 3306 on loopback ipaddress.
>>>
>>> I would check iptables or selinux preventing the bind.
>>>
>>> It wil be interesting to know the source ipaddress of MySQL client ec2
>>> instance.
>>> Interesting if you can Copy/paste output of "telnet
>>> <haproxynode_ipaddress> 3306" from mysql client ec2 instance , here.
>>> Interesting if you can  Copy/paste output of "telnet 10.10.10.10 3306"
>>> from haproxy ec2 instances, here.
>>> Interesting if you can  Copy/paste output of "telnet 10.10.10.11 3306"
>>> from haproxy ec2 instances, here.
>>>
>>> I I was doing this, maybe I would consider testing something like ;
>>> ..
>>> frontend mysql_lb_fe 0.0.0.0:3306
>>> ....
>>> acl host_myql_lb hdr(host) -i mysql-lb
>>> ..
>>> ..
>>> use_backend mysql_lb_backend if host mysql_lb
>>> ..
>>> ..
>>> backend  mysql_lb_be
>>> ..
>>> ..
>>>
>>> option mysql-check user haproxy_check
>>>       balance roundrobin
>>>       server mysql-1 10.10.10.10:3306 check
>>>       server mysql-2 10.10.10.11:3306 check
>>>
>>> Thanks,
>>> ; Yuan
>>>
>>>
>>> On 07/25/2015 06:41 AM, Tim Dunphy wrote:
>>>
>>>  Hello Nenad,
>>>>
>>>>      Jul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: [ALERT]
>>>>
>>>>             204/034418 (25035) : *Starting proxy mysql-cluster: cannot
>>>>> bind
>>>>>            s...:3306]*
>>>>>
>>>>>  Nothing listening on the port I'm trying to bind to: 3306
>>>>
>>>> [root@ha1:~] #ss -lpt | fgrep 3306
>>>> [root@ha1:~] #lsof -i :3306
>>>> [root@ha1:~] #netstat -tulpn | grep -i listen | grep 3306
>>>> [root@ha1:~] #
>>>>
>>>> While we're on the subject of listening ports, here's a list of all
>>>> listening ports on the haproxy host:
>>>>
>>>> [root@ha1:~] #netstat -tulpn | grep -i listen
>>>> tcp        0      0 0.0.0.0:35145           0.0.0.0:*
>>>> LISTEN      -
>>>> tcp        0      0 0.0.0.0:56814           0.0.0.0:*
>>>> LISTEN      16346/rpc.statd
>>>> tcp        0      0 0.0.0.0:111             0.0.0.0:*
>>>> LISTEN      16455/rpcbind
>>>> tcp        0      0 0.0.0.0:22              0.0.0.0:*
>>>> LISTEN      16396/sshd
>>>> tcp6       0      0 :::49349                :::*
>>>> LISTEN      16346/rpc.statd
>>>> tcp6       0      0 :::111                  :::*
>>>> LISTEN      16455/rpcbind
>>>> tcp6       0      0 :::47314                :::*
>>>> LISTEN      -
>>>> tcp6       0      0 :::22                   :::*
>>>> LISTEN      16396/sshd
>>>>
>>>> I thought I was beginning to understand this problem. That haproxy was
>>>> trying to bind on port 3306 from the mysql host on another machine. But
>>>> come to think of it, that doesn't make a lot of sense.
>>>>
>>>> Because I already have haproxy setup for some web servers, and there it
>>>> creates port 80 on the haproxy node. It's not trying to connect to a
>>>> foreign source. Not sure where I got that idea!!
>>>>
>>>> I also tried binding the mysql section to another port that wasn't in
>>>> use.
>>>> I tried port 3307,3308. I even tried binding the mysql section of the
>>>> config to a weird port I just grabbed off of the top of my head. I tried
>>>> binding it to port 4444.
>>>>
>>>> And there I still got a bind error:
>>>>
>>>>    [ALERT] 204/223303 (13081) : Starting proxy mysql-cluster: cannot
>>>> bind
>>>> socket [0.0.0.0:4444]
>>>>
>>>>
>>>>    Now watch this!! If I bind the mysql section to port 80 instead of
>>>> any
>>>> other port.. haproxy starts up without complaint!
>>>>
>>>> listen mysql-cluster
>>>>       bind 0.0.0.0:80
>>>>       mode tcp
>>>>       option mysql-check user haproxy_check
>>>>       balance roundrobin
>>>>       server mysql-1 10.0.0.xxx :3306 check
>>>>       server mysql-2 10.0.0.xxx:3306 check
>>>>
>>>> [root@ha1:/etc/haproxy] #systemctl status haproxy
>>>> haproxy.service - HAProxy Load Balancer
>>>>      Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled)
>>>>      Active: active (running) since Fri 2015-07-24 22:35:03 UTC; 4s ago
>>>>    Main PID: 13213 (haproxy-systemd)
>>>>      CGroup: /system.slice/haproxy.service
>>>>              ├─13213 /usr/sbin/haproxy-systemd-wrapper -f
>>>> /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
>>>>              ├─13214 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
>>>> /run/haproxy.pid -Ds
>>>>              └─13215 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p
>>>> /run/haproxy.pid -Ds
>>>>
>>>> Jul 24 22:35:03 ha1 systemd[1]: Starting HAProxy Load Balancer...
>>>> *Jul 24 22:35:03 ha1 systemd[1]: Started HAProxy Load Balancer.*
>>>>
>>>> Jul 24 22:35:03 ha1 haproxy-systemd-wrapper[13213]:
>>>> haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f
>>>> /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
>>>>
>>>> Ok. What...the...heck!!
>>>>
>>>> So why do you think that haproxy is only happy starting up on port 80? I
>>>> would think that I should be able to specify any arbitrary port for it
>>>> to
>>>> listen on in a 'listen' sub-block.
>>>>
>>>> I guess I could have my app contact the database using port 80. But
>>>> that's
>>>> a little... weird. I installed haproxy using yum from the 'updates'
>>>> repository. Is there any reason anyone can think of as to why haproxy
>>>> refuses to start on any port other than port 80??
>>>>
>>>> Thanks,
>>>> Tim
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Jul 24, 2015 at 4:59 PM, Nenad Merdanovic <ni...@nimzo.info>
>>>> wrote:
>>>>
>>>>   Hello Tim,
>>>>
>>>>>        On Fri, Jul 24, 2015 at 1:46 PM, Tim Dunphy <
>>>>> bluethu...@gmail.com
>>>>>
>>>>>>       <mailto:bluethu...@gmail.com>> wrote:
>>>>>>           listen mysql-cluster
>>>>>>               bind 127.0.0.1:3306 <http://127.0.0.1:3306>
>>>>>>
>>>>>>               mode tcp
>>>>>>               option mysql-check user haproxy_check
>>>>>>               balance roundrobin
>>>>>>               server mysql-1 10.10.10.10:3306 <
>>>>>> http://10.10.10.10:3306>
>>>>>>
>>>>>>  check
>>>>>
>>>>>                server mysql-2 10.10.10.11:3306 <
>>>>>> http://10.10.10.11:3306>
>>>>>>
>>>>>>  check
>>>>>
>>>>>            Jul 24 03:44:18 ha1 haproxy-systemd-wrapper[25034]: [ALERT]
>>>>>>           204/034418 (25035) : *Starting proxy mysql-cluster: cannot
>>>>>> bind
>>>>>>           s...:3306]*
>>>>>>
>>>>>>  Can you check if something is listening on 127.0.0.1:3306 (netstat,
>>>>> ss,
>>>>> lsof)? For example:
>>>>> ss -lpt | fgrep 3306
>>>>>
>>>>> Regards,
>>>>> Nenad
>>>>>
>>>>>
>>>>>
>>>>
>>
>


-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B

Reply via email to