RE: haproxy mysql-check
Hi, Verify this for the user you specified in option mysql-check: select plugin from mysql.user where user='monitor' \G *** 1. row *** plugin: sha256_password 1 row in set (0.00 sec) If you see sha256_password, it won't work. Since password authentication is not used in health checks by HAproxy, you can just create a new user that is not using this kind of authentication to be used by the HAproxy: CREATE USER monitor@haproxy_ip IDENTIFIED WITH 'mysql_native_password'; Verify it is working by doing: select plugin from mysql.user where user='monitor' \G *** 1. row *** plugin: mysql_native_password 1 row in set (0.00 sec) I moved to the latets version of HAProxy and now it is working. Need to do some more testing. Thanks for all the input/directions. Regards, Jayadevan DISCLAIMER: The information in this e-mail and any attachment is intended only for the person to whom it is addressed and may contain confidential and/or privileged material. If you have received this e-mail in error, kindly contact the sender and destroy all copies of the original communication. IBS makes no warranty, express or implied, nor guarantees the accuracy, adequacy or completeness of the information contained in this email or any attachment and is not liable for any errors, defects, omissions, viruses or for resultant loss or damage, if any, direct or indirect.
Re: haproxy mysql-check
On 06/18/2013 09:29 AM, Jayadevan M wrote: [...] CREATE USER monitor@haproxy_ip IDENTIFIED WITH 'mysql_native_password'; Verify it is working by doing: select plugin from mysql.user where user='monitor' \G *** 1. row *** plugin: mysql_native_password 1 row in set (0.00 sec) I moved to the latets version of HAProxy and now it is working. Need to do some more testing. Thanks for all the input/directions. Regards, Jayadevan Did you change authentication plugin to make it work ? If this is due to auth plugin, we definitely need to update the documentation. Regards, Hervé. -- Hervé COMMOWICK Ingénieur systèmes et réseaux. http://www.rezulteo.com by Lizeo Online Media Group http://www.lizeo-online-media-group.com/ 42 quai Rambaud - 69002 Lyon (France) ⎮ ☎ +33 (0)4 26 99 03 77
RE: haproxy mysql-check
Did you change authentication plugin to make it work ? If this is due to auth plugin, we definitely need to update the documentation. I did. But that did not help. So I used latest version of HAProxy. That worked. This is the status now - mysql select distinct user,HOST,plugin from mysql.user ; +-+---++ | user| HOST | plugin | +-+---++ | root| localhost || | root| 127.0.0.1 || | root| ::1 || | | localhost || | root| % || | galaxy | % || | haproxy | % || +-+---++ Regards, Jayadevan DISCLAIMER: The information in this e-mail and any attachment is intended only for the person to whom it is addressed and may contain confidential and/or privileged material. If you have received this e-mail in error, kindly contact the sender and destroy all copies of the original communication. IBS makes no warranty, express or implied, nor guarantees the accuracy, adequacy or completeness of the information contained in this email or any attachment and is not liable for any errors, defects, omissions, viruses or for resultant loss or damage, if any, direct or indirect.
Re: haproxy mysql-check
Ok so it looks like changes i made a while ago to support mysql =5.5, must be ok since haproxy 1.4.16 : http://git.1wt.eu/web?p=haproxy.git;a=commitdiff;h=212f778d6 Regards, Hervé. On 06/18/2013 12:01 PM, Jayadevan M wrote: Did you change authentication plugin to make it work ? If this is due to auth plugin, we definitely need to update the documentation. I did. But that did not help. So I used latest version of HAProxy. That worked. This is the status now - mysql select distinct user,HOST,plugin from mysql.user ; +-+---++ | user| HOST | plugin | +-+---++ | root| localhost || | root| 127.0.0.1 || | root| ::1 || | | localhost || | root| % || | galaxy | % || | haproxy | % || +-+---++ Regards, Jayadevan DISCLAIMER: The information in this e-mail and any attachment is intended only for the person to whom it is addressed and may contain confidential and/or privileged material. If you have received this e-mail in error, kindly contact the sender and destroy all copies of the original communication. IBS makes no warranty, express or implied, nor guarantees the accuracy, adequacy or completeness of the information contained in this email or any attachment and is not liable for any errors, defects, omissions, viruses or for resultant loss or damage, if any, direct or indirect. -- Hervé COMMOWICK Ingénieur systèmes et réseaux. http://www.rezulteo.com by Lizeo Online Media Group http://www.lizeo-online-media-group.com/ 42 quai Rambaud - 69002 Lyon (France) ⎮ ☎ +33 (0)4 26 99 03 77
Re: haproxy mysql-check
Hello, It is due to the plugin and it should be document IMHO (or even fixed, although not much benefit from it): haproxy[8258]: Server testback_mysql/localmysql-1 is UP, reason: Layer7 check passed, code: 0, info: 5.6.11-rc60.3, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. After doing: mysql drop user monitor@localhost; create user monitor@localhost identified with 'sha256_password'; flush privileges; haproxy[8258]: Server testback_mysql/localmysql-1 is DOWN, reason: Layer7 wrong status, code: 0, info: Client does not support authentication protocol requested by server; consider upgrading MySQL client, check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue Running the latest build with default options: ~/temp/haproxy$ ./haproxy -vv HA-Proxy version 1.5-dev19 2013/06/17 Copyright 2000-2013 Willy Tarreau w...@1wt.eu Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing OPTIONS = MySQL version used: 5.6.11-rc60.3 Percona Server (GPL) RSA keys have been properly configured. Relevant part of the HAproxy configuration: backend testback_mysql balance roundrobin mode tcp option mysql-check user monitor server localmysql-1 127.0.0.1:3306 weight 100 maxconn 768 inter 5000 fall 3 rise 3 check On 06/18/2013 12:08 PM, Hervé COMMOWICK wrote: Ok so it looks like changes i made a while ago to support mysql =5.5, must be ok since haproxy 1.4.16 : http://git.1wt.eu/web?p=haproxy.git;a=commitdiff;h=212f778d6 Regards, Hervé. On 06/18/2013 12:01 PM, Jayadevan M wrote: Did you change authentication plugin to make it work ? If this is due to auth plugin, we definitely need to update the documentation. I did. But that did not help. So I used latest version of HAProxy. That worked. This is the status now - mysql select distinct user,HOST,plugin from mysql.user ; +-+---++ | user| HOST | plugin | +-+---++ | root| localhost || | root| 127.0.0.1 || | root| ::1 || | | localhost || | root| % || | galaxy | % || | haproxy | % || +-+---++ Regards, Jayadevan DISCLAIMER: The information in this e-mail and any attachment is intended only for the person to whom it is addressed and may contain confidential and/or privileged material. If you have received this e-mail in error, kindly contact the sender and destroy all copies of the original communication. IBS makes no warranty, express or implied, nor guarantees the accuracy, adequacy or completeness of the information contained in this email or any attachment and is not liable for any errors, defects, omissions, viruses or for resultant loss or damage, if any, direct or indirect. Regards, -- Nenad Merdanovic | PGP: 0x423edcb2 | Web: http://nimzo.info Linkedin: http://www.linkedin.com/in/nenadmerdanovic
RE: haproxy mysql-check
Hi! I moved to the latets version of HAProxy and now it is working. Jayadevan, can you tell us what version you are running (which works) and what release you where using before? Just double checking that commit 212f778d6 fixed that problem ... I did. But that did not help. So I used latest version of HAProxy. That worked. I guess you had to do both things to make it work; or did you revert the the plugin change? Can you post: select user,HOST,plugin from mysql.user; It is due to the plugin and it should be document IMHO (or even fixed, although not much benefit from it): Agreed, but it looks like Jayadevan had a different issue (or both). Do you know for certain that the exact release MySQL broke backwards compatibility was 5.6.6? Could you make a full tcpdump of the new 5.6.x client authentication packet and post upload the .cap somewhere? It shouldn't be that difficult to adjust the mysql-check. Lukas
Re: haproxy mysql-check
Hello Lukas,
Yes, the plugin was introduced in 5.6.6 as seen on [1]. I have uploaded
the .cap file (tcpdump -A -r capture.cap) to
http://nimzo[dot]info/files/capture{dot}cap
I don't think it is worth the effort as sha256 is used for password auth
and HAproxy doesn't do that. Further, using plugin auth will require a
lot of changes to the code and could break compatibility with old MySQL
auth services.
Here are a few links to read:
http://dev.mysql.com/doc/internals/en/connection-phase.html#packet-Protocol::AuthSwitchRequest
http://dev.mysql.com/doc/internals/en/connection-phase.html#flag-CLIENT_PLUGIN_AUTH
http://dev.mysql.com/doc/internals/en/connection-phase.html#packet-Protocol::HandshakeResponse41
It is doable and would likely work flawlessly if we were to wait for the
AuthSwitchRequest packet and process it correctly in order not to break
backwards compatibility, but it all sounds too much for me in order to
do this.
Another option is to add plugin option to the configuration, like
option mysql-check user monitor sha256
[1]
http://dev.mysql.com/doc/refman/5.6/en/sha256-authentication-plugin.html
IHMO, this is a plugin auth module and going down the route to support
it is just not worth it. Tomorrow another one might pop up expecting
something different. We should just draw the line saying we support
pre-4.1 and post-4.1 authentication, without plugins. Especially since
password hashing done by sha256_password module is useless to HAproxy.
On 06/18/2013 04:32 PM, Lukas Tribus wrote:
Hi!
I moved to the latets version of HAProxy and now it is working.
Jayadevan, can you tell us what version you are running (which works)
and what release you where using before? Just double checking that
commit 212f778d6 fixed that problem ...
I did. But that did not help. So I used latest version of HAProxy.
That worked.
I guess you had to do both things to make it work; or did you revert the
the plugin change?
Can you post:
select user,HOST,plugin from mysql.user;
It is due to the plugin and it should be document IMHO (or even fixed,
although not much benefit from it):
Agreed, but it looks like Jayadevan had a different issue (or both).
Do you know for certain that the exact release MySQL broke backwards
compatibility was 5.6.6? Could you make a full tcpdump of the new 5.6.x
client authentication packet and post upload the .cap somewhere?
It shouldn't be that difficult to adjust the mysql-check.
Lukas
Regards,
--
Nenad Merdanovic | PGP: 0x423edcb2 | Web: http://nimzo.info
Linkedin: http://www.linkedin.com/in/nenadmerdanovic
RE: haproxy mysql-check
Hi, Jayadevan, can you tell us what version you are running (which works) and what release you where using before? Just double checking that commit 212f778d6 fixed that problem ... Initial - HA-Proxy version 1.4.9 2010/10/28 Now - HA-Proxy version 1.4.24 2013/06/17 I guess you had to do both things to make it work; or did you revert the the plugin change? Can you post: select user,HOST,plugin from mysql.user; mysql select user,HOST,plugin from mysql.user ; +-+---++ | user| HOST | plugin | +-+---++ | root| localhost || | root| 127.0.0.1 || | root| ::1 || | | localhost || | root| % || | galaxy | % || | haproxy | % || +-+---++ Regards, Jayadevan DISCLAIMER: The information in this e-mail and any attachment is intended only for the person to whom it is addressed and may contain confidential and/or privileged material. If you have received this e-mail in error, kindly contact the sender and destroy all copies of the original communication. IBS makes no warranty, express or implied, nor guarantees the accuracy, adequacy or completeness of the information contained in this email or any attachment and is not liable for any errors, defects, omissions, viruses or for resultant loss or damage, if any, direct or indirect.
Re: haproxy mysql-check
On Mon, Jun 17, 2013 at 05:50:29AM +, Jayadevan M wrote: What version of haproxy is this ? It is haproxy-1.4.22 OK. That's really strange. I'll recheck here by copy-pasting your data to a fake responder, just in case we missed something. What does MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = -1 EAGAIN (Resource temporarily unavailable) 0.14 mean? It means that a recv() attempt was made on the socket but that no data were available yet and that haproxy must poll the fd first, which is why this is followed by a call to poll(), and only once poll() returns OK on this FD, recv() is attempted again. Best regards, Willy
RE: haproxy mysql-check
Hi, OK. That's really strange. I'll recheck here by copy-pasting your data to a fake responder, just in case we missed something. Does the following parameter have anything to do with this? I tried changing a few parameters and now it seems to be working. Will test some more. In the real scenario, both MySQL servers will be on Windows. timeout connect 5 Regards, Jayadevan DISCLAIMER: The information in this e-mail and any attachment is intended only for the person to whom it is addressed and may contain confidential and/or privileged material. If you have received this e-mail in error, kindly contact the sender and destroy all copies of the original communication. IBS makes no warranty, express or implied, nor guarantees the accuracy, adequacy or completeness of the information contained in this email or any attachment and is not liable for any errors, defects, omissions, viruses or for resultant loss or damage, if any, direct or indirect.
Re: haproxy mysql-check
On Mon, Jun 17, 2013 at 06:29:55AM +, Jayadevan M wrote: Hi, OK. That's really strange. I'll recheck here by copy-pasting your data to a fake responder, just in case we missed something. Does the following parameter have anything to do with this? I tried changing a few parameters and now it seems to be working. Will test some more. In the real scenario, both MySQL servers will be on Windows. timeout connect 5 It should not because as you can see in the trace, the whole check happens in only 2 milliseconds, which is quite fast. What are the other parameters you changed ? Could you also please share all your timeouts ? Maybe some of them are wrong ? Regards, Willy
RE: haproxy mysql-check
HI, It should not because as you can see in the trace, the whole check happens in only 2 milliseconds, which is quite fast. What are the other parameters you changed ? Could you also please share all your timeouts ? Maybe some of them are wrong ? I am getting a layer 4 timeout in the environment where I should get this running. Here is my file - global pidfile /var/run/haproxy1.pid stats socket /tmp/haproxy defaults log global timeout connect 5 # default 10 second time out if a backend is not found timeout client 3 timeout server 3 listen stats :1936 mode http stats enable stats realm Haproxy\ Statistics stats uri / frontend mysql_proxy *:3309 mode tcp default_backend request_mysql backend request_mysql mode tcp option mysql-check user haproxy server svr2 192.168.8.37:3406 weight 1 check port 3406 inter 5000 rise 3 fall 3 server svr1 192.168.2.27:3306 weight 1 check port 3306 inter 5000 rise 3 fall 3 Please note that the IPs have changed and now they are both windows machines. Do we need MySQL client to be installed on the m/c for this to work? Regards, Jayadevan DISCLAIMER: The information in this e-mail and any attachment is intended only for the person to whom it is addressed and may contain confidential and/or privileged material. If you have received this e-mail in error, kindly contact the sender and destroy all copies of the original communication. IBS makes no warranty, express or implied, nor guarantees the accuracy, adequacy or completeness of the information contained in this email or any attachment and is not liable for any errors, defects, omissions, viruses or for resultant loss or damage, if any, direct or indirect.
Re: haproxy mysql-check
On Mon, Jun 17, 2013 at 06:57:15AM +, Jayadevan M wrote: HI, It should not because as you can see in the trace, the whole check happens in only 2 milliseconds, which is quite fast. What are the other parameters you changed ? Could you also please share all your timeouts ? Maybe some of them are wrong ? I am getting a layer 4 timeout in the environment where I should get this running. Here is my file - global pidfile /var/run/haproxy1.pid stats socket /tmp/haproxy defaults log global timeout connect 5 # default 10 second time out if a backend is not found timeout client 3 timeout server 3 What timeout did you have before timeout connect 5 ? Do you think it's possible that you've had timeout connect 10 ? Then it would be 10 ms and might explain what it failed (and running via strace can add enough latency to make it work more often or fail more often depending how things go). listen stats :1936 mode http stats enable stats realm Haproxy\ Statistics stats uri / frontend mysql_proxy *:3309 mode tcp default_backend request_mysql backend request_mysql mode tcp option mysql-check user haproxy server svr2 192.168.8.37:3406 weight 1 check port 3406 inter 5000 rise 3 fall 3 server svr1 192.168.2.27:3306 weight 1 check port 3306 inter 5000 rise 3 fall 3 Please note that the IPs have changed and now they are both windows machines. OK. Anyway your config is OK in my opinion. Do we need MySQL client to be installed on the m/c for this to work? no, not at all. Best regards, Willy
RE: haproxy mysql-check
Hi, listen stats :1936 mode http stats enable stats realm Haproxy\ Statistics stats uri / frontend mysql_proxy *:3309 mode tcp default_backend request_mysql One error to another - How can I make HAProxy use a new MySQL client/libraries? I am getting the error - [WARNING] 167/224803 (20008) : Server request_mysql/svr1 is DOWN, reason: Layer7 wrong status, code: 0, info: Client does not support authentication protocol requested by server; consider upgrading MySQL client, check duration: 2ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. I have MySQL 5.6.10 Regards, Jayadevan DISCLAIMER: The information in this e-mail and any attachment is intended only for the person to whom it is addressed and may contain confidential and/or privileged material. If you have received this e-mail in error, kindly contact the sender and destroy all copies of the original communication. IBS makes no warranty, express or implied, nor guarantees the accuracy, adequacy or completeness of the information contained in this email or any attachment and is not liable for any errors, defects, omissions, viruses or for resultant loss or damage, if any, direct or indirect.
Re: haproxy mysql-check
Hi Jayadevan, On Mon, Jun 17, 2013 at 11:50:10AM +, Jayadevan M wrote: One error to another - How can I make HAProxy use a new MySQL client/libraries? Haproxy doesn't use a mysql library, it only does health checks and verifies the handshake. I am getting the error - [WARNING] 167/224803 (20008) : Server request_mysql/svr1 is DOWN, reason: Layer7 wrong status, code: 0, info: Client does not support authentication protocol requested by server; consider upgrading MySQL client, check duration: 2ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. You should specify a username with the check I believe, please look at option mysql-check in the doc, it includes an example of how to set it up. Hoping this helps, Willy
Re: haproxy mysql-check
Hello, I have tracked this down to the new authentication method used in MySQL 5.6 since version 5.6.6. Verify this for the user you specified in option mysql-check: select plugin from mysql.user where user='monitor' \G *** 1. row *** plugin: sha256_password 1 row in set (0.00 sec) If you see sha256_password, it won't work. Since password authentication is not used in health checks by HAproxy, you can just create a new user that is not using this kind of authentication to be used by the HAproxy: CREATE USER monitor@haproxy_ip IDENTIFIED WITH 'mysql_native_password'; Verify it is working by doing: select plugin from mysql.user where user='monitor' \G *** 1. row *** plugin: mysql_native_password 1 row in set (0.00 sec) Let me know if this fixes the issue. Regards, Nenad On 06/17/2013 01:50 PM, Jayadevan M wrote: Hi, listen stats :1936 mode http stats enable stats realm Haproxy\ Statistics stats uri / frontend mysql_proxy *:3309 mode tcp default_backend request_mysql One error to another - How can I make HAProxy use a new MySQL client/libraries? I am getting the error - [WARNING] 167/224803 (20008) : Server request_mysql/svr1 is DOWN, reason: Layer7 wrong status, code: 0, info: Client does not support authentication protocol requested by server; consider upgrading MySQL client, check duration: 2ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. I have MySQL 5.6.10 Regards, Jayadevan DISCLAIMER: The information in this e-mail and any attachment is intended only for the person to whom it is addressed and may contain confidential and/or privileged material. If you have received this e-mail in error, kindly contact the sender and destroy all copies of the original communication. IBS makes no warranty, express or implied, nor guarantees the accuracy, adequacy or completeness of the information contained in this email or any attachment and is not liable for any errors, defects, omissions, viruses or for resultant loss or damage, if any, direct or indirect. -- Nenad Merdanovic | PGP: 0x423edcb2 | Web: http://nimzo.info Linkedin: http://www.linkedin.com/in/nenadmerdanovic
RE: haproxy mysql-check
Hi,
-Original Message-
From: Willy Tarreau [mailto:w...@1wt.eu]
Sent: Friday, June 14, 2013 5:53 PM
To: Jayadevan M
Cc: Jayadevan M; haproxy@formilux.org
Subject: Re: haproxy mysql-check
On Fri, Jun 14, 2013 at 10:30:14AM +, Jayadevan M wrote:
Telnet works -
-bash-3.00# telnet 192.168.8.37 3406
Trying 192.168.8.37...
Connected to 192.168.8.37.
Escape character is '^]'.
B
5.1.34-community-logÂBjon[]Ax`OI_;Qg3^CConnection to 192.168.8.37
closed by foreign host.
OK. Could you show us the complete strace output from haproxy, from the
connect() call to the close() call ? Ideally if you could take it with strace
-ttT, it
would be nice as we'd also have the timings.
It is possible that we're simply facing a bug in the timeout handling for
example.
Here is the output. 192.168.2.27 (MySQL on linux) works. 192.168.8.37 (MySQL on
Windows) does not.
10:25:57.474629 connect(5, {sa_family=AF_INET, sin_port=htons(3306),
sin_addr=inet_addr(192.168.2.27)}, 16) = -1 EINPROGRESS (Operation now in
progress) 0.000202
10:25:57.475000 sendto(5, \16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1, 23,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 23 0.99
10:25:57.475259 epoll_wait(3, {}, 6, 0) = 0 0.84
10:25:57.475488 recvfrom(5,
4\0\0\0\n5.1.52\0\253\27\0\0lUC0d,Jn\0\377\367!\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0mWKTA,VKVNe3\0\3\0\0\2\0\0\0,
16384, 0, NULL, NULL) = 63 0.71
10:25:57.475691 recvfrom(5, , 16321, 0, NULL, NULL) = 0 0.50
10:25:57.475824 shutdown(5, 2 /* send and receive */) = 0 0.63
10:25:57.475943 epoll_wait(3, {}, 6, 0) = 0 0.27
10:25:57.476021 close(5)= 0 0.19
10:25:57.476075 epoll_wait(3, {}, 5, 1000) = 0 0.44
10:25:58.476230 epoll_wait(3, {}, 5, 1000) = 0 0.999819
10:25:59.476230 epoll_wait(3, {}, 5, 507) = 0 0.506766
10:25:59.983158 epoll_wait(3, {}, 5, 2) = 0 0.001827
10:25:59.985133 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 5 0.000101
10:25:59.985375 fcntl(5, F_SETFL, O_RDONLY|O_NONBLOCK) = 0 0.83
10:25:59.985609 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0 0.83
10:25:59.985838 connect(5, {sa_family=AF_INET, sin_port=htons(3406),
sin_addr=inet_addr(192.168.8.37)}, 16) = -1 EINPROGRESS (Operation now in
progress) 0.000102
10:25:59.986031 sendto(5, \16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1, 23,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = -1 EAGAIN (Resource temporarily
unavailable) 0.14
10:25:59.986110 epoll_ctl(3, EPOLL_CTL_ADD, 5, {EPOLLOUT, {u32=5, u64=5}}) = 0
0.63
10:25:59.986250 epoll_wait(3, {{EPOLLOUT, {u32=5, u64=5}}}, 6, 1000) = 1
0.000173
10:25:59.986487 sendto(5, \16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1, 23,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 23 0.38
10:25:59.986609 recvfrom(5, 0x1e92770, 16384, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable) 0.66
10:25:59.986778 epoll_ctl(3, EPOLL_CTL_MOD, 5, {EPOLLIN, {u32=5, u64=5}}) = 0
0.64
10:25:59.986929 epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 6, 1000) = 1
0.000397
10:25:59.987373 recvfrom(5,
B\0\0\0\n5.1.34-community-log\0\20\0\0\0?)Lt9]I\0\377\367\10\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0007uj4an%h`*M%\0\3\0\0\2\0\0\0,
16384, 0, NULL, NULL) = 77 0.15
10:25:59.987440 recvfrom(5, , 16307, 0, NULL, NULL) = 0 0.29
10:25:59.987508 shutdown(5, 2 /* send and receive */) = 0 0.16
10:25:59.987560 close(5)= 0 0.30
10:25:59.987625 epoll_wait(3, {}, 5, 1000) = 0 0.999548
10:26:00.987263 epoll_wait(3, {}, 5, 1000) = 0 0.999853
10:26:01.987288 epoll_wait(3, {}, 5, 488) = 0 0.488027
10:26:02.476277 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 5 0.96
10:26:02.476452 fcntl(5, F_SETFL, O_RDONLY|O_NONBLOCK) = 0 0.11
10:26:02.476511 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0 0.12
10:26:02.476563 connect(5, {sa_family=AF_INET, sin_port=htons(3306),
sin_addr=inet_addr(192.168.2.27)}, 16) = -1 EINPROGRESS (Operation now in
progress) 0.000166
Connect from command line -
[root@n3170 ~]# mysql -h 192.168.8.37 --port=3406 -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 74
Server version: 5.1.34-community-log MySQL Community Server (GPL)
Regards,
Jayadevan
DISCLAIMER: The information in this e-mail and any attachment is intended only
for the person to whom it is addressed and may contain confidential and/or
privileged material. If you have received this e-mail in error, kindly contact
the sender and destroy all copies of the original communication. IBS makes no
warranty, express or implied, nor guarantees the accuracy, adequacy or
completeness of the information contained in this email or any attachment and
is not liable for any errors, defects, omissions, viruses or for resultant loss
or damage, if any, direct or indirect.
Re: haproxy mysql-check
Hi Jayadevan,
On Mon, Jun 17, 2013 at 04:55:42AM +, Jayadevan M wrote:
Here is the output. 192.168.2.27 (MySQL on linux) works. 192.168.8.37 (MySQL
on Windows) does not.
OK so linux part here :
10:25:57.474629 connect(5, {sa_family=AF_INET, sin_port=htons(3306),
sin_addr=inet_addr(192.168.2.27)}, 16) = -1 EINPROGRESS (Operation now in
progress) 0.000202
10:25:57.475000 sendto(5, \16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1, 23,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 23 0.99
10:25:57.475259 epoll_wait(3, {}, 6, 0) = 0 0.84
10:25:57.475488 recvfrom(5,
4\0\0\0\n5.1.52\0\253\27\0\0lUC0d,Jn\0\377\367!\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0mWKTA,VKVNe3\0\3\0\0\2\0\0\0,
16384, 0, NULL, NULL) = 63 0.71
10:25:57.475691 recvfrom(5, , 16321, 0, NULL, NULL) = 0 0.50
10:25:57.475824 shutdown(5, 2 /* send and receive */) = 0 0.63
10:25:57.475943 epoll_wait(3, {}, 6, 0) = 0 0.27
10:25:57.476021 close(5)= 0 0.19
And windows part here :
10:25:59.985133 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 5 0.000101
10:25:59.985375 fcntl(5, F_SETFL, O_RDONLY|O_NONBLOCK) = 0 0.83
10:25:59.985609 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0 0.83
10:25:59.985838 connect(5, {sa_family=AF_INET, sin_port=htons(3406),
sin_addr=inet_addr(192.168.8.37)}, 16) = -1 EINPROGRESS (Operation now in
progress) 0.000102
10:25:59.986031 sendto(5, \16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1, 23,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = -1 EAGAIN (Resource temporarily
unavailable) 0.14
10:25:59.986110 epoll_ctl(3, EPOLL_CTL_ADD, 5, {EPOLLOUT, {u32=5, u64=5}}) =
0 0.63
10:25:59.986250 epoll_wait(3, {{EPOLLOUT, {u32=5, u64=5}}}, 6, 1000) = 1
0.000173
10:25:59.986487 sendto(5, \16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1, 23,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 23 0.38
10:25:59.986609 recvfrom(5, 0x1e92770, 16384, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable) 0.66
10:25:59.986778 epoll_ctl(3, EPOLL_CTL_MOD, 5, {EPOLLIN, {u32=5, u64=5}}) = 0
0.64
10:25:59.986929 epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 6, 1000) = 1
0.000397
10:25:59.987373 recvfrom(5,
B\0\0\0\n5.1.34-community-log\0\20\0\0\0?)Lt9]I\0\377\367\10\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0007uj4an%h`*M%\0\3\0\0\2\0\0\0,
16384, 0, NULL, NULL) = 77 0.15
10:25:59.987440 recvfrom(5, , 16307, 0, NULL, NULL) = 0 0.29
10:25:59.987508 shutdown(5, 2 /* send and receive */) = 0 0.16
10:25:59.987560 close(5)= 0 0.30
So the response was received, including the shutdown. It seems like it's
the response contents which are not correctly parsed then.
I have compared the received data to what the check does and both checks
seem to parse correctly (both contain two response packets) and both should
return the major version. And I've double checked, both haproxy 1.4 and 1.5
use the same parser.
What version of haproxy is this ?
Regards,
Willy
RE: haproxy mysql-check
HI,
OK so linux part here :
10:25:57.474629 connect(5, {sa_family=AF_INET, sin_port=htons(3306),
sin_addr=inet_addr(192.168.2.27)}, 16) = -1 EINPROGRESS (Operation
now in progress) 0.000202
10:25:57.475000 sendto(5,
\16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1, 23,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 23 0.99
10:25:57.475259 epoll_wait(3, {}, 6, 0) = 0 0.84
10:25:57.475488 recvfrom(5,
4\0\0\0\n5.1.52\0\253\27\0\0lUC0d,Jn\0\377\367!\2\0\0\0\0\0\0\0\0\0\0
\0\0\0\0mWKTA,VKVNe3\0\3\0\0\2\0\0\0, 16384, 0, NULL, NULL) = 63
0.71
10:25:57.475691 recvfrom(5, , 16321, 0, NULL, NULL) = 0 0.50
10:25:57.475824 shutdown(5, 2 /* send and receive */) = 0 0.63
10:25:57.475943 epoll_wait(3, {}, 6, 0) = 0 0.27
10:25:57.476021 close(5)= 0 0.19
And windows part here :
10:25:59.985133 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 5
0.000101
10:25:59.985375 fcntl(5, F_SETFL, O_RDONLY|O_NONBLOCK) = 0 0.83
10:25:59.985609 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
0.83
10:25:59.985838 connect(5, {sa_family=AF_INET, sin_port=htons(3406),
sin_addr=inet_addr(192.168.8.37)}, 16) = -1 EINPROGRESS (Operation
now in progress) 0.000102
10:25:59.986031 sendto(5,
\16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1, 23,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = -1 EAGAIN (Resource
temporarily
unavailable) 0.14
10:25:59.986110 epoll_ctl(3, EPOLL_CTL_ADD, 5, {EPOLLOUT, {u32=5,
u64=5}}) = 0 0.63
10:25:59.986250 epoll_wait(3, {{EPOLLOUT, {u32=5, u64=5}}}, 6, 1000) =
1 0.000173
10:25:59.986487 sendto(5,
\16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1, 23,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 23 0.38
10:25:59.986609 recvfrom(5, 0x1e92770, 16384, 0, 0, 0) = -1 EAGAIN
(Resource temporarily unavailable) 0.66
10:25:59.986778 epoll_ctl(3, EPOLL_CTL_MOD, 5, {EPOLLIN, {u32=5,
u64=5}}) = 0 0.64
10:25:59.986929 epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 6, 1000) =
1 0.000397
10:25:59.987373 recvfrom(5,
B\0\0\0\n5.1.34-community-log\0\20\0\0\0?)Lt9]I\0\377\367\10\2\0\0\0
\0\0\0\0\0\0\0\0\0\0\0007uj4an%h`*M%\0\3\0\0\2\0\0\0, 16384, 0, NULL,
NULL) = 77 0.15
10:25:59.987440 recvfrom(5, , 16307, 0, NULL, NULL) = 0 0.29
10:25:59.987508 shutdown(5, 2 /* send and receive */) = 0 0.16
10:25:59.987560 close(5)= 0 0.30
So the response was received, including the shutdown. It seems like it's the
response contents which are not correctly parsed then.
I have compared the received data to what the check does and both checks
seem to parse correctly (both contain two response packets) and both should
return the major version. And I've double checked, both haproxy 1.4 and 1.5
use the same parser.
What version of haproxy is this ?
It is haproxy-1.4.22
What does MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = -1 EAGAIN (Resource
temporarily unavailable) 0.14 mean?
Regards,
Jayadevan
DISCLAIMER: The information in this e-mail and any attachment is intended only
for the person to whom it is addressed and may contain confidential and/or
privileged material. If you have received this e-mail in error, kindly contact
the sender and destroy all copies of the original communication. IBS makes no
warranty, express or implied, nor guarantees the accuracy, adequacy or
completeness of the information contained in this email or any attachment and
is not liable for any errors, defects, omissions, viruses or for resultant loss
or damage, if any, direct or indirect.
haproxy mysql-check
Hi,
I was trying to make HAProxy's mysql-check option work with the MySQL
Servers on Windows. HAProxy is on linux, MySQL on windows.There are 2 MySQL
servers - Server1 and Server2. It was not working (HAProxy was always
marking the servers down). I tried changing one of the MySQL server entries
point to MySQL on a linux box and it was working. Won't HAProxy's
mysql-check work if the target MySQL is on windows? Output from a strace of
HAProxy -
connect(5, {sa_family=AF_INET, sin_port=htons(3406),
sin_addr=inet_addr(192.168.8.37)}, 16) = -1 EINPROGRESS (Operation now in
progress)
sendto(5, \16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1, 23,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = -1 EAGAIN (Resource temporarily
unavailable)
connect(5, {sa_family=AF_INET, sin_port=htons(3306),
sin_addr=inet_addr(192.168.2.27)}, 16) = -1 EINPROGRESS (Operation now in
progress)
sendto(5, \16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1, 23,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 23
192.168.2.27 is linux and 192.168.8.37 windows. I am able to connect to the
MySQL server on windows, from mysql command line client on 192.168.2.27. So
it is not a port blocked issue.
Regards,
Jayadevan
Re: haproxy mysql-check
Hi,
On Fri, Jun 14, 2013 at 03:38:25PM +0530, Jayadevan M wrote:
Hi,
I was trying to make HAProxy's mysql-check option work with the MySQL
Servers on Windows. HAProxy is on linux, MySQL on windows.There are 2 MySQL
servers - Server1 and Server2. It was not working (HAProxy was always
marking the servers down). I tried changing one of the MySQL server entries
point to MySQL on a linux box and it was working. Won't HAProxy's
mysql-check work if the target MySQL is on windows? Output from a strace of
HAProxy -
connect(5, {sa_family=AF_INET, sin_port=htons(3406),
sin_addr=inet_addr(192.168.8.37)}, 16) = -1 EINPROGRESS (Operation now in
progress)
sendto(5, \16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1, 23,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = -1 EAGAIN (Resource temporarily
unavailable)
connect(5, {sa_family=AF_INET, sin_port=htons(3306),
sin_addr=inet_addr(192.168.2.27)}, 16) = -1 EINPROGRESS (Operation now in
progress)
sendto(5, \16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1, 23,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 23
192.168.2.27 is linux and 192.168.8.37 windows. I am able to connect to the
MySQL server on windows, from mysql command line client on 192.168.2.27. So
it is not a port blocked issue.
But from what I understand, it means that the windows server is not responding
to the connection request. I don't know if this is intentional or a mistake,
but the port is different on your windows config (3406). Could you check
with telnet from the haproxy host that the connection establishes on this
port :
# telnet 192.168.8.37 3406
That way we'll be sure that the same parameters are used. Also, it would be nice
to take a tcpdump capture from the haproxy machine, maybe it will reveal why
this does not work.
Regards,
Willy
Re: haproxy mysql-check
Hi,
The port is correct. I am running MySQL on 3406. Will try to take tcpdum.
Regards,
Jayadevan
On Fri, Jun 14, 2013 at 3:43 PM, Willy Tarreau w...@1wt.eu wrote:
Hi,
On Fri, Jun 14, 2013 at 03:38:25PM +0530, Jayadevan M wrote:
Hi,
I was trying to make HAProxy's mysql-check option work with the MySQL
Servers on Windows. HAProxy is on linux, MySQL on windows.There are 2
MySQL
servers - Server1 and Server2. It was not working (HAProxy was always
marking the servers down). I tried changing one of the MySQL server
entries
point to MySQL on a linux box and it was working. Won't HAProxy's
mysql-check work if the target MySQL is on windows? Output from a strace
of
HAProxy -
connect(5, {sa_family=AF_INET, sin_port=htons(3406),
sin_addr=inet_addr(192.168.8.37)}, 16) = -1 EINPROGRESS (Operation now
in
progress)
sendto(5, \16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1, 23,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = -1 EAGAIN (Resource temporarily
unavailable)
connect(5, {sa_family=AF_INET, sin_port=htons(3306),
sin_addr=inet_addr(192.168.2.27)}, 16) = -1 EINPROGRESS (Operation now
in
progress)
sendto(5, \16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1, 23,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 23
192.168.2.27 is linux and 192.168.8.37 windows. I am able to connect to
the
MySQL server on windows, from mysql command line client on 192.168.2.27.
So
it is not a port blocked issue.
But from what I understand, it means that the windows server is not
responding
to the connection request. I don't know if this is intentional or a
mistake,
but the port is different on your windows config (3406). Could you check
with telnet from the haproxy host that the connection establishes on this
port :
# telnet 192.168.8.37 3406
That way we'll be sure that the same parameters are used. Also, it would
be nice
to take a tcpdump capture from the haproxy machine, maybe it will reveal
why
this does not work.
Regards,
Willy
RE: haproxy mysql-check
Hi,
-Original Message-
From: Willy Tarreau [mailto:w...@1wt.eu]
Sent: Friday, June 14, 2013 3:44 PM
To: Jayadevan M
Cc: haproxy@formilux.org; Jayadevan M
Subject: Re: haproxy mysql-check
Hi,
On Fri, Jun 14, 2013 at 03:38:25PM +0530, Jayadevan M wrote:
Hi,
I was trying to make HAProxy's mysql-check option work with the MySQL
Servers on Windows. HAProxy is on linux, MySQL on windows.There are 2
MySQL servers - Server1 and Server2. It was not working (HAProxy was
always marking the servers down). I tried changing one of the MySQL
server entries point to MySQL on a linux box and it was working. Won't
HAProxy's mysql-check work if the target MySQL is on windows? Output
from a strace of HAProxy -
connect(5, {sa_family=AF_INET, sin_port=htons(3406),
sin_addr=inet_addr(192.168.8.37)}, 16) = -1 EINPROGRESS (Operation
now in
progress)
sendto(5, \16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1, 23,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = -1 EAGAIN (Resource
temporarily
unavailable)
connect(5, {sa_family=AF_INET, sin_port=htons(3306),
sin_addr=inet_addr(192.168.2.27)}, 16) = -1 EINPROGRESS (Operation
now in
progress)
sendto(5, \16\0\0\1\0\200\0\0\1haproxy\0\0\1\0\0\0\1, 23,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 23
192.168.2.27 is linux and 192.168.8.37 windows. I am able to connect
to the MySQL server on windows, from mysql command line client on
192.168.2.27. So it is not a port blocked issue.
But from what I understand, it means that the windows server is not
responding to the connection request. I don't know if this is intentional or a
mistake, but the port is different on your windows config (3406). Could you
check with telnet from the haproxy host that the connection establishes on
this port :
# telnet 192.168.8.37 3406
Telnet works -
-bash-3.00# telnet 192.168.8.37 3406
Trying 192.168.8.37...
Connected to 192.168.8.37.
Escape character is '^]'.
B
5.1.34-community-logÂBjon[]Ax`OI_;Qg3^CConnection to 192.168.8.37 closed by
foreign host.
Regards,
Jayadevan
DISCLAIMER: The information in this e-mail and any attachment is intended only
for the person to whom it is addressed and may contain confidential and/or
privileged material. If you have received this e-mail in error, kindly contact
the sender and destroy all copies of the original communication. IBS makes no
warranty, express or implied, nor guarantees the accuracy, adequacy or
completeness of the information contained in this email or any attachment and
is not liable for any errors, defects, omissions, viruses or for resultant loss
or damage, if any, direct or indirect.
Re: haproxy mysql-check
On Fri, Jun 14, 2013 at 10:30:14AM +, Jayadevan M wrote: Telnet works - -bash-3.00# telnet 192.168.8.37 3406 Trying 192.168.8.37... Connected to 192.168.8.37. Escape character is '^]'. B 5.1.34-community-logÂBjon[]Ax`OI_;Qg3^CConnection to 192.168.8.37 closed by foreign host. OK. Could you show us the complete strace output from haproxy, from the connect() call to the close() call ? Ideally if you could take it with strace -ttT, it would be nice as we'd also have the timings. It is possible that we're simply facing a bug in the timeout handling for example. Willy
