Re: [Openstack] Incredibly odd mysql permission error
Does anyone think this could be an openstack bug? I just want to check before submitting a bug report. Sam On Fri, Mar 8, 2013 at 4:02 PM, Jay Pipes jaypi...@gmail.com wrote: Sorry, I really can't think of anything :( On 03/08/2013 03:52 PM, Samuel Winchenbach wrote: I dropped those users and no change. I also set up general logging in mysql but it really doesn't provide any additional information. Any idea for a next step I could take? I am almost at the point of taking a tcpdump and trying to recreate the salted password. :/ Thanks for the help Sam On Fri, Mar 8, 2013 at 3:38 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: I'm stumped :( Looks like everything is set up correctly to me. What is interested is that your nova user access works from test2, but there is no nova@test2 user in the mysql.user table. What about doing a DROP USER nova@test1; FLUSH PRIVILEGES; and then see if that fixes things... since the nova@10.21.0.0/255.255.0.0 http://nova@10.21.0.0/255.255.0.0 user is clearly working for the access from test2. Also, I'd recommend highly removing the nova@% user. Best, -jay On 03/08/2013 03:09 PM, Samuel Winchenbach wrote: http://paste2.org/p/3085807 On Fri, Mar 8, 2013 at 2:46 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: Please paste the results of SELECT User, Host, Password FROM mysql.user when running as root... Thanks! -jay On 03/08/2013 02:25 PM, Samuel Winchenbach wrote: Here are my grants. I don't know if this helps, but I did verify that the password was identical for each grant: http://paste2.org/p/3085361 On Fri, Mar 8, 2013 at 2:17 PM, Samuel Winchenbach swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com wrote: root@test1:/var/log# mysql -hmysql-ha -unova -p -eSELECT User, Host, Password FROM mysql.user; ERROR 1142 (42000) at line 1: SELECT command denied to user 'nova'@'test1' for table 'user' On Fri, Mar 8, 2013 at 2:06 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: What does this show? mysql -hmysql-ha -unova -pPASS -eSELECT User, Host, Password FROM mysql.user -jay On 03/08/2013 01:46 PM, Samuel Winchenbach wrote: Sorry, that must have been a copy and paste error. Here is what I actually ran: http://paste2.org/p/3084996 On Fri, Mar 8, 2013 at 12:40 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: On 03/08/2013 12:19 PM, Samuel Winchenbach wrote: Hi All, I have two nodes (test1 and test2) that I am trying to set up in a highly available configuration. During the setup process I tried running nova-manage service list on both nodes. It worked fine on test2, but fails on test1 even though I can connect to the database with the mysql client from test1. Here is a screen capture that shows the setup on the two nodes are basically
Re: [Openstack] Incredibly odd mysql permission error
So as to reproduce the nova-manage SQL command, I would recommand to tcpdump -A port 3306 on the host and get the SQL trace on what's failing. Could you please explain further what is your HA config ? Are you using pacemaker/heartbeat or any VIP ? -Sylvain Le 11/03/2013 14:23, Samuel Winchenbach a écrit : Does anyone think this could be an openstack bug? I just want to check before submitting a bug report. Sam On Fri, Mar 8, 2013 at 4:02 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: Sorry, I really can't think of anything :( On 03/08/2013 03:52 PM, Samuel Winchenbach wrote: I dropped those users and no change. I also set up general logging in mysql but it really doesn't provide any additional information. Any idea for a next step I could take? I am almost at the point of taking a tcpdump and trying to recreate the salted password. :/ Thanks for the help Sam On Fri, Mar 8, 2013 at 3:38 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: I'm stumped :( Looks like everything is set up correctly to me. What is interested is that your nova user access works from test2, but there is no nova@test2 user in the mysql.user table. What about doing a DROP USER nova@test1; FLUSH PRIVILEGES; and then see if that fixes things... since the nova@10.21.0.0/255.255.0.0 http://nova@10.21.0.0/255.255.0.0 http://nova@10.21.0.0/255.255.0.0 user is clearly working for the access from test2. Also, I'd recommend highly removing the nova@% user. Best, -jay On 03/08/2013 03:09 PM, Samuel Winchenbach wrote: http://paste2.org/p/3085807 On Fri, Mar 8, 2013 at 2:46 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: Please paste the results of SELECT User, Host, Password FROM mysql.user when running as root... Thanks! -jay On 03/08/2013 02:25 PM, Samuel Winchenbach wrote: Here are my grants. I don't know if this helps, but I did verify that the password was identical for each grant: http://paste2.org/p/3085361 On Fri, Mar 8, 2013 at 2:17 PM, Samuel Winchenbach swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com wrote: root@test1:/var/log# mysql -hmysql-ha -unova -p -eSELECT User, Host, Password FROM mysql.user; ERROR 1142 (42000) at line 1: SELECT command denied to user 'nova'@'test1' for table 'user' On Fri, Mar 8, 2013 at 2:06 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: What does this show? mysql -hmysql-ha -unova -pPASS -eSELECT User, Host, Password FROM mysql.user -jay On 03/08/2013 01:46 PM, Samuel Winchenbach wrote: Sorry, that must have been a copy and paste error. Here is what I actually ran: http://paste2.org/p/3084996 On Fri, Mar 8, 2013 at 12:40 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com
Re: [Openstack] Incredibly odd mysql permission error
I enabled general_log in /etc/mysql/my.cnf Here are the results of connecting from test1, test2 and using the client: http://paste2.org/p/3115525 I purposefully used the real password in case there is a problem with it. I changed before submitting post. here is a raw packet TCP dump (tcpdump -w rawdump port 3306) of an attempted nova-manage service list from test1: https://www.dropbox.com/s/u4cjzxv6w6bwwe6/rawdump I looked at it with wireshark and couldn't see anything that jumped out at me as incorrect. I have not yet tried to recreate the salted password. Here is my pacemaker configuration for mysql. I stripped out openstack services, rabbitmq and others for clarity. All resources are currently disabled (other than MySQL): http://paste2.org/p/3115685 Please don't yell at me for having STONITH disabled :P This is a testing cluster and I am working on getting routed to the IPMI interface. /etc/hosts: http://paste2.org/p/3115713 /etc/nova/nova.conf: http://paste2.org/p/3115739 If there is anything else I can provide you, please let me know! I have pulled out most of my hair at this point! Sam On Mon, Mar 11, 2013 at 10:11 AM, Sylvain Bauza sylvain.ba...@digimind.comwrote: So as to reproduce the nova-manage SQL command, I would recommand to tcpdump -A port 3306 on the host and get the SQL trace on what's failing. Could you please explain further what is your HA config ? Are you using pacemaker/heartbeat or any VIP ? -Sylvain Le 11/03/2013 14:23, Samuel Winchenbach a écrit : Does anyone think this could be an openstack bug? I just want to check before submitting a bug report. Sam On Fri, Mar 8, 2013 at 4:02 PM, Jay Pipes jaypi...@gmail.com wrote: Sorry, I really can't think of anything :( On 03/08/2013 03:52 PM, Samuel Winchenbach wrote: I dropped those users and no change. I also set up general logging in mysql but it really doesn't provide any additional information. Any idea for a next step I could take? I am almost at the point of taking a tcpdump and trying to recreate the salted password. :/ Thanks for the help Sam On Fri, Mar 8, 2013 at 3:38 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: I'm stumped :( Looks like everything is set up correctly to me. What is interested is that your nova user access works from test2, but there is no nova@test2 user in the mysql.user table. What about doing a DROP USER nova@test1; FLUSH PRIVILEGES; and then see if that fixes things... since the nova@10.21.0.0/255.255.0.0 http://nova@10.21.0.0/255.255.0.0 user is clearly working for the access from test2. Also, I'd recommend highly removing the nova@% user. Best, -jay On 03/08/2013 03:09 PM, Samuel Winchenbach wrote: http://paste2.org/p/3085807 On Fri, Mar 8, 2013 at 2:46 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: Please paste the results of SELECT User, Host, Password FROM mysql.user when running as root... Thanks! -jay On 03/08/2013 02:25 PM, Samuel Winchenbach wrote: Here are my grants. I don't know if this helps, but I did verify that the password was identical for each grant: http://paste2.org/p/3085361 On Fri, Mar 8, 2013 at 2:17 PM, Samuel Winchenbach swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com wrote: root@test1:/var/log# mysql -hmysql-ha -unova -p -eSELECT User, Host, Password FROM mysql.user; ERROR 1142 (42000) at line 1: SELECT command denied to user 'nova'@'test1' for table 'user' On Fri, Mar 8, 2013 at 2:06 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: What does this show? mysql -hmysql-ha -unova -pPASS -eSELECT User, Host, Password FROM mysql.user -jay On 03/08/2013 01:46 PM, Samuel Winchenbach wrote: Sorry, that must have been a copy and paste error. Here is what I actually ran: http://paste2.org/p/3084996
Re: [Openstack] Incredibly odd mysql permission error
#1 - No change #2 - All of grants are in the ip/mask form such as: 'nova'@' 10.21.0.0/255.255.0.0' I have also tried adding 'nova'@'test1' and 'nova'@'10.21.0.1'. No change. #3 - I changed the SQL connection string over to IP instead of hostname. No change. I didn't restart nova-api because it isn't running. If I understand correctly nova-manage communicated directly with the db, bypassing nova-api. This would appear true seeing nova-manage service list works correctly on test2. :( Thanks for the help! Sam On Mon, Mar 11, 2013 at 12:24 PM, Sylvain Bauza sylvain.ba...@digimind.comwrote: When looking at MySQL 5.1 refman ( http://dev.mysql.com/doc/refman/5.1/en/access-denied.html ), I would suggest to follow the procedure : 1. 'mysqladmin flush-hosts' 2. replace DNS entries in mysql.user table by IP addresses instead 3. modify /etc/nova/nova.conf with IP address of HA Mysql instead (and restart nova-api !) I wouldn't bet on it, but I would say this is due to some name resolution which is incorrect. -Sylvain Le 11/03/2013 17:00, Sylvain Bauza a écrit : Ok, lemme try to summarize. You do have a DRBD setup for MySQL bound to a VIP 10.21.1.1 thanks to Pacemaker. This setup is relying on two hosts, test1 (10.21.0.1) and test2 (10.21.0.2). Your nova.conf is pointing to mysql://10.21.1.1 which is the VIP. Are you sure your my.cnf is actually the same in between both DRBD nodes ? (I would recommend to symlink it to a physical file hosted on the DRBD device). One thing is hurting me : you told me that nova is also pacemake'd. If so, why can I still see my_ip=10.21.0.2 (test2) ? It should be pointing to nova-ha (assuming 10.21.2.4 as per /etc/hosts). Also, as per my understanding of Pacemaker, DRBD partition is setup by default on test2, correct ? Sorry, as per my first reading, I can't see anything obvious. That said, I'm not sure this is a Nova bug, as the tcpdump trace is seeing a correct MySQL connection attempt. But maybe I'm wrong ? Anyway, are you sure you only have *one* MySQL engine running (either on test1 or test2) and nova-manage trying to access this right one ? Perms look good to me. As it a test setup, you could try to unleash the grants by deleting them and allowing nova@'%' to see if it's a basic dns mapping issue. -Sylvain Le 11/03/2013 16:09, Samuel Winchenbach a écrit : I enabled general_log in /etc/mysql/my.cnf Here are the results of connecting from test1, test2 and using the client: http://paste2.org/p/3115525 I purposefully used the real password in case there is a problem with it. I changed before submitting post. here is a raw packet TCP dump (tcpdump -w rawdump port 3306) of an attempted nova-manage service list from test1: https://www.dropbox.com/s/u4cjzxv6w6bwwe6/rawdump I looked at it with wireshark and couldn't see anything that jumped out at me as incorrect. I have not yet tried to recreate the salted password. Here is my pacemaker configuration for mysql. I stripped out openstack services, rabbitmq and others for clarity. All resources are currently disabled (other than MySQL): http://paste2.org/p/3115685 Please don't yell at me for having STONITH disabled :P This is a testing cluster and I am working on getting routed to the IPMI interface. /etc/hosts: http://paste2.org/p/3115713 /etc/nova/nova.conf: http://paste2.org/p/3115739 If there is anything else I can provide you, please let me know! I have pulled out most of my hair at this point! Sam On Mon, Mar 11, 2013 at 10:11 AM, Sylvain Bauza sylvain.ba...@digimind.com wrote: So as to reproduce the nova-manage SQL command, I would recommand to tcpdump -A port 3306 on the host and get the SQL trace on what's failing. Could you please explain further what is your HA config ? Are you using pacemaker/heartbeat or any VIP ? -Sylvain Le 11/03/2013 14:23, Samuel Winchenbach a écrit : Does anyone think this could be an openstack bug? I just want to check before submitting a bug report. Sam On Fri, Mar 8, 2013 at 4:02 PM, Jay Pipes jaypi...@gmail.com wrote: Sorry, I really can't think of anything :( On 03/08/2013 03:52 PM, Samuel Winchenbach wrote: I dropped those users and no change. I also set up general logging in mysql but it really doesn't provide any additional information. Any idea for a next step I could take? I am almost at the point of taking a tcpdump and trying to recreate the salted password. :/ Thanks for the help Sam On Fri, Mar 8, 2013 at 3:38 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: I'm stumped :( Looks like everything is set up correctly to me. What is interested is that your nova user access works from test2, but there is no nova@test2 user in the mysql.user table. What about doing a DROP USER nova@test1; FLUSH PRIVILEGES; and then see if that fixes things... since the
Re: [Openstack] Incredibly odd mysql permission error
For completeness here the routing table, and ip listing for both test1 and test2. Doubt this will help much: http://paste2.org/p/3117125 On Mon, Mar 11, 2013 at 1:52 PM, Samuel Winchenbach swinc...@gmail.comwrote: #1 - No change #2 - All of grants are in the ip/mask form such as: 'nova'@' 10.21.0.0/255.255.0.0' I have also tried adding 'nova'@'test1' and 'nova'@'10.21.0.1'. No change. #3 - I changed the SQL connection string over to IP instead of hostname. No change. I didn't restart nova-api because it isn't running. If I understand correctly nova-manage communicated directly with the db, bypassing nova-api. This would appear true seeing nova-manage service list works correctly on test2. :( Thanks for the help! Sam On Mon, Mar 11, 2013 at 12:24 PM, Sylvain Bauza sylvain.ba...@digimind.com wrote: When looking at MySQL 5.1 refman ( http://dev.mysql.com/doc/refman/5.1/en/access-denied.html ), I would suggest to follow the procedure : 1. 'mysqladmin flush-hosts' 2. replace DNS entries in mysql.user table by IP addresses instead 3. modify /etc/nova/nova.conf with IP address of HA Mysql instead (and restart nova-api !) I wouldn't bet on it, but I would say this is due to some name resolution which is incorrect. -Sylvain Le 11/03/2013 17:00, Sylvain Bauza a écrit : Ok, lemme try to summarize. You do have a DRBD setup for MySQL bound to a VIP 10.21.1.1 thanks to Pacemaker. This setup is relying on two hosts, test1 (10.21.0.1) and test2 (10.21.0.2). Your nova.conf is pointing to mysql://10.21.1.1 which is the VIP. Are you sure your my.cnf is actually the same in between both DRBD nodes ? (I would recommend to symlink it to a physical file hosted on the DRBD device). One thing is hurting me : you told me that nova is also pacemake'd. If so, why can I still see my_ip=10.21.0.2 (test2) ? It should be pointing to nova-ha (assuming 10.21.2.4 as per /etc/hosts). Also, as per my understanding of Pacemaker, DRBD partition is setup by default on test2, correct ? Sorry, as per my first reading, I can't see anything obvious. That said, I'm not sure this is a Nova bug, as the tcpdump trace is seeing a correct MySQL connection attempt. But maybe I'm wrong ? Anyway, are you sure you only have *one* MySQL engine running (either on test1 or test2) and nova-manage trying to access this right one ? Perms look good to me. As it a test setup, you could try to unleash the grants by deleting them and allowing nova@'%' to see if it's a basic dns mapping issue. -Sylvain Le 11/03/2013 16:09, Samuel Winchenbach a écrit : I enabled general_log in /etc/mysql/my.cnf Here are the results of connecting from test1, test2 and using the client: http://paste2.org/p/3115525 I purposefully used the real password in case there is a problem with it. I changed before submitting post. here is a raw packet TCP dump (tcpdump -w rawdump port 3306) of an attempted nova-manage service list from test1: https://www.dropbox.com/s/u4cjzxv6w6bwwe6/rawdump I looked at it with wireshark and couldn't see anything that jumped out at me as incorrect. I have not yet tried to recreate the salted password. Here is my pacemaker configuration for mysql. I stripped out openstack services, rabbitmq and others for clarity. All resources are currently disabled (other than MySQL): http://paste2.org/p/3115685 Please don't yell at me for having STONITH disabled :P This is a testing cluster and I am working on getting routed to the IPMI interface. /etc/hosts: http://paste2.org/p/3115713 /etc/nova/nova.conf: http://paste2.org/p/3115739 If there is anything else I can provide you, please let me know! I have pulled out most of my hair at this point! Sam On Mon, Mar 11, 2013 at 10:11 AM, Sylvain Bauza sylvain.ba...@digimind.com wrote: So as to reproduce the nova-manage SQL command, I would recommand to tcpdump -A port 3306 on the host and get the SQL trace on what's failing. Could you please explain further what is your HA config ? Are you using pacemaker/heartbeat or any VIP ? -Sylvain Le 11/03/2013 14:23, Samuel Winchenbach a écrit : Does anyone think this could be an openstack bug? I just want to check before submitting a bug report. Sam On Fri, Mar 8, 2013 at 4:02 PM, Jay Pipes jaypi...@gmail.com wrote: Sorry, I really can't think of anything :( On 03/08/2013 03:52 PM, Samuel Winchenbach wrote: I dropped those users and no change. I also set up general logging in mysql but it really doesn't provide any additional information. Any idea for a next step I could take? I am almost at the point of taking a tcpdump and trying to recreate the salted password. :/ Thanks for the help Sam On Fri, Mar 8, 2013 at 3:38 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: I'm stumped :( Looks like everything is set up correctly to me. What is
Re: [Openstack] Incredibly odd mysql permission error
OK Someone on the IRC channel got me closer, but we have no idea why this would happen: this works: root@test1:~# nova-manage --config-file=/etc/nova/nova.conf service list Why would I have to specify the config file though? It is in the standard place. Thanks, Sam On Mon, Mar 11, 2013 at 2:01 PM, Samuel Winchenbach swinc...@gmail.comwrote: For completeness here the routing table, and ip listing for both test1 and test2. Doubt this will help much: http://paste2.org/p/3117125 On Mon, Mar 11, 2013 at 1:52 PM, Samuel Winchenbach swinc...@gmail.comwrote: #1 - No change #2 - All of grants are in the ip/mask form such as: 'nova'@' 10.21.0.0/255.255.0.0' I have also tried adding 'nova'@'test1' and 'nova'@'10.21.0.1'. No change. #3 - I changed the SQL connection string over to IP instead of hostname. No change. I didn't restart nova-api because it isn't running. If I understand correctly nova-manage communicated directly with the db, bypassing nova-api. This would appear true seeing nova-manage service list works correctly on test2. :( Thanks for the help! Sam On Mon, Mar 11, 2013 at 12:24 PM, Sylvain Bauza sylvain.ba...@digimind.com wrote: When looking at MySQL 5.1 refman ( http://dev.mysql.com/doc/refman/5.1/en/access-denied.html ), I would suggest to follow the procedure : 1. 'mysqladmin flush-hosts' 2. replace DNS entries in mysql.user table by IP addresses instead 3. modify /etc/nova/nova.conf with IP address of HA Mysql instead (and restart nova-api !) I wouldn't bet on it, but I would say this is due to some name resolution which is incorrect. -Sylvain Le 11/03/2013 17:00, Sylvain Bauza a écrit : Ok, lemme try to summarize. You do have a DRBD setup for MySQL bound to a VIP 10.21.1.1 thanks to Pacemaker. This setup is relying on two hosts, test1 (10.21.0.1) and test2 (10.21.0.2). Your nova.conf is pointing to mysql://10.21.1.1 which is the VIP. Are you sure your my.cnf is actually the same in between both DRBD nodes ? (I would recommend to symlink it to a physical file hosted on the DRBD device). One thing is hurting me : you told me that nova is also pacemake'd. If so, why can I still see my_ip=10.21.0.2 (test2) ? It should be pointing to nova-ha (assuming 10.21.2.4 as per /etc/hosts). Also, as per my understanding of Pacemaker, DRBD partition is setup by default on test2, correct ? Sorry, as per my first reading, I can't see anything obvious. That said, I'm not sure this is a Nova bug, as the tcpdump trace is seeing a correct MySQL connection attempt. But maybe I'm wrong ? Anyway, are you sure you only have *one* MySQL engine running (either on test1 or test2) and nova-manage trying to access this right one ? Perms look good to me. As it a test setup, you could try to unleash the grants by deleting them and allowing nova@'%' to see if it's a basic dns mapping issue. -Sylvain Le 11/03/2013 16:09, Samuel Winchenbach a écrit : I enabled general_log in /etc/mysql/my.cnf Here are the results of connecting from test1, test2 and using the client: http://paste2.org/p/3115525 I purposefully used the real password in case there is a problem with it. I changed before submitting post. here is a raw packet TCP dump (tcpdump -w rawdump port 3306) of an attempted nova-manage service list from test1: https://www.dropbox.com/s/u4cjzxv6w6bwwe6/rawdump I looked at it with wireshark and couldn't see anything that jumped out at me as incorrect. I have not yet tried to recreate the salted password. Here is my pacemaker configuration for mysql. I stripped out openstack services, rabbitmq and others for clarity. All resources are currently disabled (other than MySQL): http://paste2.org/p/3115685 Please don't yell at me for having STONITH disabled :P This is a testing cluster and I am working on getting routed to the IPMI interface. /etc/hosts: http://paste2.org/p/3115713 /etc/nova/nova.conf: http://paste2.org/p/3115739 If there is anything else I can provide you, please let me know! I have pulled out most of my hair at this point! Sam On Mon, Mar 11, 2013 at 10:11 AM, Sylvain Bauza sylvain.ba...@digimind.com wrote: So as to reproduce the nova-manage SQL command, I would recommand to tcpdump -A port 3306 on the host and get the SQL trace on what's failing. Could you please explain further what is your HA config ? Are you using pacemaker/heartbeat or any VIP ? -Sylvain Le 11/03/2013 14:23, Samuel Winchenbach a écrit : Does anyone think this could be an openstack bug? I just want to check before submitting a bug report. Sam On Fri, Mar 8, 2013 at 4:02 PM, Jay Pipes jaypi...@gmail.com wrote: Sorry, I really can't think of anything :( On 03/08/2013 03:52 PM, Samuel Winchenbach wrote: I dropped those users and no change. I also set up general logging in mysql but it really doesn't provide any additional information. Any idea
Re: [Openstack] Incredibly odd mysql permission error
I ran into a similar problem with the Grizzly-3 Keystone release. I had to specify keystone.conf as the config-file with keystone-manage/ db_sync command otherwise it would not use the mysql statement in the keystone.conf file. Mark From: openstack-bounces+mark.m.miller=hp@lists.launchpad.net [mailto:openstack-bounces+mark.m.miller=hp@lists.launchpad.net] On Behalf Of Samuel Winchenbach Sent: Monday, March 11, 2013 11:18 AM To: Sylvain Bauza Cc: openstack@lists.launchpad.net Subject: Re: [Openstack] Incredibly odd mysql permission error OK Someone on the IRC channel got me closer, but we have no idea why this would happen: this works: root@test1:~# nova-manage --config-file=/etc/nova/nova.conf service list Why would I have to specify the config file though? It is in the standard place. Thanks, Sam On Mon, Mar 11, 2013 at 2:01 PM, Samuel Winchenbach swinc...@gmail.commailto:swinc...@gmail.com wrote: For completeness here the routing table, and ip listing for both test1 and test2. Doubt this will help much: http://paste2.org/p/3117125 On Mon, Mar 11, 2013 at 1:52 PM, Samuel Winchenbach swinc...@gmail.commailto:swinc...@gmail.com wrote: #1 - No change #2 - All of grants are in the ip/mask form such as: 'nova'@'10.21.0.0/255.255.0.0http://10.21.0.0/255.255.0.0' I have also tried adding 'nova'@'test1' and 'nova'@'10.21.0.1'. No change. #3 - I changed the SQL connection string over to IP instead of hostname. No change. I didn't restart nova-api because it isn't running. If I understand correctly nova-manage communicated directly with the db, bypassing nova-api. This would appear true seeing nova-manage service list works correctly on test2. :( Thanks for the help! Sam On Mon, Mar 11, 2013 at 12:24 PM, Sylvain Bauza sylvain.ba...@digimind.commailto:sylvain.ba...@digimind.com wrote: When looking at MySQL 5.1 refman (http://dev.mysql.com/doc/refman/5.1/en/access-denied.html ), I would suggest to follow the procedure : 1. 'mysqladmin flush-hosts' 2. replace DNS entries in mysql.user table by IP addresses instead 3. modify /etc/nova/nova.conf with IP address of HA Mysql instead (and restart nova-api !) I wouldn't bet on it, but I would say this is due to some name resolution which is incorrect. -Sylvain Le 11/03/2013 17:00, Sylvain Bauza a écrit : Ok, lemme try to summarize. You do have a DRBD setup for MySQL bound to a VIP 10.21.1.1 thanks to Pacemaker. This setup is relying on two hosts, test1 (10.21.0.1) and test2 (10.21.0.2). Your nova.conf is pointing to mysql://10.21.1.1http://10.21.1.1 which is the VIP. Are you sure your my.cnf is actually the same in between both DRBD nodes ? (I would recommend to symlink it to a physical file hosted on the DRBD device). One thing is hurting me : you told me that nova is also pacemake'd. If so, why can I still see my_ip=10.21.0.2 (test2) ? It should be pointing to nova-ha (assuming 10.21.2.4 as per /etc/hosts). Also, as per my understanding of Pacemaker, DRBD partition is setup by default on test2, correct ? Sorry, as per my first reading, I can't see anything obvious. That said, I'm not sure this is a Nova bug, as the tcpdump trace is seeing a correct MySQL connection attempt. But maybe I'm wrong ? Anyway, are you sure you only have *one* MySQL engine running (either on test1 or test2) and nova-manage trying to access this right one ? Perms look good to me. As it a test setup, you could try to unleash the grants by deleting them and allowing nova@'%' to see if it's a basic dns mapping issue. -Sylvain Le 11/03/2013 16:09, Samuel Winchenbach a écrit : I enabled general_log in /etc/mysql/my.cnf Here are the results of connecting from test1, test2 and using the client: http://paste2.org/p/3115525 I purposefully used the real password in case there is a problem with it. I changed before submitting post. here is a raw packet TCP dump (tcpdump -w rawdump port 3306) of an attempted nova-manage service list from test1: https://www.dropbox.com/s/u4cjzxv6w6bwwe6/rawdump I looked at it with wireshark and couldn't see anything that jumped out at me as incorrect. I have not yet tried to recreate the salted password. Here is my pacemaker configuration for mysql. I stripped out openstack services, rabbitmq and others for clarity. All resources are currently disabled (other than MySQL): http://paste2.org/p/3115685 Please don't yell at me for having STONITH disabled :P This is a testing cluster and I am working on getting routed to the IPMI interface. /etc/hosts: http://paste2.org/p/3115713 /etc/nova/nova.conf: http://paste2.org/p/3115739 If there is anything else I can provide you, please let me know! I have pulled out most of my hair at this point! Sam On Mon, Mar 11, 2013 at 10:11 AM, Sylvain Bauza sylvain.ba...@digimind.commailto:sylvain.ba...@digimind.com wrote: So as to reproduce the nova-manage SQL command, I would recommand to tcpdump -A port 3306 on the host and get
Re: [Openstack] Incredibly odd mysql permission error
ugh... I had an example file called nova.conf in /root Apparently nova-manage looks for that file first. Case closed. I wish I could get the last week back. Sam On Mon, Mar 11, 2013 at 2:37 PM, Miller, Mark M (EB SW Cloud - RD - Corvallis) mark.m.mil...@hp.com wrote: I ran into a similar problem with the Grizzly-3 Keystone release. I had to specify keystone.conf as the config-file with keystone-manage/ db_sync command otherwise it would not use the mysql statement in the keystone.conf file. ** ** Mark ** ** *From:* openstack-bounces+mark.m.miller=hp@lists.launchpad.net[mailto: openstack-bounces+mark.m.miller=hp@lists.launchpad.net] *On Behalf Of *Samuel Winchenbach *Sent:* Monday, March 11, 2013 11:18 AM *To:* Sylvain Bauza *Cc:* openstack@lists.launchpad.net *Subject:* Re: [Openstack] Incredibly odd mysql permission error ** ** OK Someone on the IRC channel got me closer, but we have no idea why this would happen: ** ** this works: root@test1:~# nova-manage --config-file=/etc/nova/nova.conf service list ** ** Why would I have to specify the config file though? It is in the standard place. ** ** Thanks, Sam ** ** ** ** ** ** On Mon, Mar 11, 2013 at 2:01 PM, Samuel Winchenbach swinc...@gmail.com wrote: For completeness here the routing table, and ip listing for both test1 and test2. Doubt this will help much: http://paste2.org/p/3117125 ** ** On Mon, Mar 11, 2013 at 1:52 PM, Samuel Winchenbach swinc...@gmail.com wrote: #1 - No change #2 - All of grants are in the ip/mask form such as: 'nova'@' 10.21.0.0/255.255.0.0' I have also tried adding 'nova'@'test1' and 'nova'@'10.21.0.1'. No change. #3 - I changed the SQL connection string over to IP instead of hostname. No change. I didn't restart nova-api because it isn't running. If I understand correctly nova-manage communicated directly with the db, bypassing nova-api. This would appear true seeing nova-manage service list works correctly on test2. ** ** ** ** :( ** ** Thanks for the help! Sam ** ** On Mon, Mar 11, 2013 at 12:24 PM, Sylvain Bauza sylvain.ba...@digimind.com wrote: When looking at MySQL 5.1 refman ( http://dev.mysql.com/doc/refman/5.1/en/access-denied.html ), I would suggest to follow the procedure : 1. 'mysqladmin flush-hosts' 2. replace DNS entries in mysql.user table by IP addresses instead 3. modify /etc/nova/nova.conf with IP address of HA Mysql instead (and restart nova-api !) I wouldn't bet on it, but I would say this is due to some name resolution which is incorrect. -Sylvain Le 11/03/2013 17:00, Sylvain Bauza a écrit : Ok, lemme try to summarize. You do have a DRBD setup for MySQL bound to a VIP 10.21.1.1 thanks to Pacemaker. This setup is relying on two hosts, test1 (10.21.0.1) and test2 (10.21.0.2). Your nova.conf is pointing to mysql://10.21.1.1 which is the VIP. Are you sure your my.cnf is actually the same in between both DRBD nodes ? (I would recommend to symlink it to a physical file hosted on the DRBD device). One thing is hurting me : you told me that nova is also pacemake'd. If so, why can I still see my_ip=10.21.0.2 (test2) ? It should be pointing to nova-ha (assuming 10.21.2.4 as per /etc/hosts). Also, as per my understanding of Pacemaker, DRBD partition is setup by default on test2, correct ? Sorry, as per my first reading, I can't see anything obvious. That said, I'm not sure this is a Nova bug, as the tcpdump trace is seeing a correct MySQL connection attempt. But maybe I'm wrong ? Anyway, are you sure you only have *one* MySQL engine running (either on test1 or test2) and nova-manage trying to access this right one ? Perms look good to me. As it a test setup, you could try to unleash the grants by deleting them and allowing nova@'%' to see if it's a basic dns mapping issue. -Sylvain Le 11/03/2013 16:09, Samuel Winchenbach a écrit : I enabled general_log in /etc/mysql/my.cnf Here are the results of connecting from test1, test2 and using the client: http://paste2.org/p/3115525 I purposefully used the real password in case there is a problem with it. I changed before submitting post. ** ** here is a raw packet TCP dump (tcpdump -w rawdump port 3306) of an attempted nova-manage service list from test1: https://www.dropbox.com/s/u4cjzxv6w6bwwe6/rawdump I looked at it with wireshark and couldn't see anything that jumped out at me as incorrect. I have not yet tried to recreate the salted password.*** * ** ** ** ** Here is my pacemaker configuration for mysql. I stripped out openstack services, rabbitmq and others for clarity. All resources are currently disabled (other than MySQL): http://paste2.org/p/3115685 ** ** Please don't yell at me for having STONITH disabled :P
[Openstack] Incredibly odd mysql permission error
Hi All, I have two nodes (test1 and test2) that I am trying to set up in a highly available configuration. During the setup process I tried running nova-manage service list on both nodes. It worked fine on test2, but fails on test1 even though I can connect to the database with the mysql client from test1. Here is a screen capture that shows the setup on the two nodes are basically identical: http://paste2.org/p/3084223 Here is one that shows that the command is working on test2: http://paste2.org/p/3084234 In the first paste I logged into the mysql server by copying and pasting the user name, host name and password from the nova.conf. user nova has the following grants in mysql: GRANT ALL PRIVILEGES ON nova.* to 'nova'@'test1' IDENTIFIED BY ' '; GRANT ALL PRIVILEGES ON nova.* to 'nova'@'%' IDENTIFIED BY ' '; GRANT ALL PRIVILEGES ON nova.* to 'nova'@'10.21.0.0/255.255.0.0' IDENTIFIED BY ''; GRANT ALL PRIVILEGES ON nova.* to 'nova'@'localhost' IDENTIFIED BY ' '; I have mysql controlled by pacemaker so I have tried running it on both test1, and test2 with the same exact results. If anyone can help me with this it would greatly appreciated. I am at wits end. Sam ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Incredibly odd mysql permission error
On 03/08/2013 12:19 PM, Samuel Winchenbach wrote: Hi All, I have two nodes (test1 and test2) that I am trying to set up in a highly available configuration. During the setup process I tried running nova-manage service list on both nodes. It worked fine on test2, but fails on test1 even though I can connect to the database with the mysql client from test1. Here is a screen capture that shows the setup on the two nodes are basically identical: http://paste2.org/p/3084223 In the above paste you are doing: mysql -unova -hmysql-ha -u root nova -p Note you are supplying 2 -u arguments, and mysql will take the second (root). -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Incredibly odd mysql permission error
Sorry, that must have been a copy and paste error. Here is what I actually ran: http://paste2.org/p/3084996 On Fri, Mar 8, 2013 at 12:40 PM, Jay Pipes jaypi...@gmail.com wrote: On 03/08/2013 12:19 PM, Samuel Winchenbach wrote: Hi All, I have two nodes (test1 and test2) that I am trying to set up in a highly available configuration. During the setup process I tried running nova-manage service list on both nodes. It worked fine on test2, but fails on test1 even though I can connect to the database with the mysql client from test1. Here is a screen capture that shows the setup on the two nodes are basically identical: http://paste2.org/p/3084223 In the above paste you are doing: mysql -unova - hmysql-ha -u root nova -p Note you are supplying 2 -u arguments, and mysql will take the second (root). -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Incredibly odd mysql permission error
oops. guess I need to change the root password now :P On Fri, Mar 8, 2013 at 3:09 PM, Samuel Winchenbach swinc...@gmail.comwrote: http://paste2.org/p/3085807 On Fri, Mar 8, 2013 at 2:46 PM, Jay Pipes jaypi...@gmail.com wrote: Please paste the results of SELECT User, Host, Password FROM mysql.user when running as root... Thanks! -jay On 03/08/2013 02:25 PM, Samuel Winchenbach wrote: Here are my grants. I don't know if this helps, but I did verify that the password was identical for each grant: http://paste2.org/p/3085361 On Fri, Mar 8, 2013 at 2:17 PM, Samuel Winchenbach swinc...@gmail.com mailto:swinc...@gmail.com wrote: root@test1:/var/log# mysql -hmysql-ha -unova -p -eSELECT User, Host, Password FROM mysql.user; ERROR 1142 (42000) at line 1: SELECT command denied to user 'nova'@'test1' for table 'user' On Fri, Mar 8, 2013 at 2:06 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: What does this show? mysql -hmysql-ha -unova -pPASS -eSELECT User, Host, Password FROM mysql.user -jay On 03/08/2013 01:46 PM, Samuel Winchenbach wrote: Sorry, that must have been a copy and paste error. Here is what I actually ran: http://paste2.org/p/3084996 On Fri, Mar 8, 2013 at 12:40 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: On 03/08/2013 12:19 PM, Samuel Winchenbach wrote: Hi All, I have two nodes (test1 and test2) that I am trying to set up in a highly available configuration. During the setup process I tried running nova-manage service list on both nodes. It worked fine on test2, but fails on test1 even though I can connect to the database with the mysql client from test1. Here is a screen capture that shows the setup on the two nodes are basically identical: http://paste2.org/p/3084223 In the above paste you are doing: mysql -unova - hmysql-ha -u root nova -p Note you are supplying 2 -u arguments, and mysql will take the second (root). -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Incredibly odd mysql permission error
http://paste2.org/p/3085807 On Fri, Mar 8, 2013 at 2:46 PM, Jay Pipes jaypi...@gmail.com wrote: Please paste the results of SELECT User, Host, Password FROM mysql.user when running as root... Thanks! -jay On 03/08/2013 02:25 PM, Samuel Winchenbach wrote: Here are my grants. I don't know if this helps, but I did verify that the password was identical for each grant: http://paste2.org/p/3085361 On Fri, Mar 8, 2013 at 2:17 PM, Samuel Winchenbach swinc...@gmail.com mailto:swinc...@gmail.com wrote: root@test1:/var/log# mysql -hmysql-ha -unova -p -eSELECT User, Host, Password FROM mysql.user; ERROR 1142 (42000) at line 1: SELECT command denied to user 'nova'@'test1' for table 'user' On Fri, Mar 8, 2013 at 2:06 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: What does this show? mysql -hmysql-ha -unova -pPASS -eSELECT User, Host, Password FROM mysql.user -jay On 03/08/2013 01:46 PM, Samuel Winchenbach wrote: Sorry, that must have been a copy and paste error. Here is what I actually ran: http://paste2.org/p/3084996 On Fri, Mar 8, 2013 at 12:40 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: On 03/08/2013 12:19 PM, Samuel Winchenbach wrote: Hi All, I have two nodes (test1 and test2) that I am trying to set up in a highly available configuration. During the setup process I tried running nova-manage service list on both nodes. It worked fine on test2, but fails on test1 even though I can connect to the database with the mysql client from test1. Here is a screen capture that shows the setup on the two nodes are basically identical: http://paste2.org/p/3084223 In the above paste you are doing: mysql -unova - hmysql-ha -u root nova -p Note you are supplying 2 -u arguments, and mysql will take the second (root). -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Incredibly odd mysql permission error
I'm stumped :( Looks like everything is set up correctly to me. What is interested is that your nova user access works from test2, but there is no nova@test2 user in the mysql.user table. What about doing a DROP USER nova@test1; FLUSH PRIVILEGES; and then see if that fixes things... since the nova@10.21.0.0/255.255.0.0 user is clearly working for the access from test2. Also, I'd recommend highly removing the nova@% user. Best, -jay On 03/08/2013 03:09 PM, Samuel Winchenbach wrote: http://paste2.org/p/3085807 On Fri, Mar 8, 2013 at 2:46 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: Please paste the results of SELECT User, Host, Password FROM mysql.user when running as root... Thanks! -jay On 03/08/2013 02:25 PM, Samuel Winchenbach wrote: Here are my grants. I don't know if this helps, but I did verify that the password was identical for each grant: http://paste2.org/p/3085361 On Fri, Mar 8, 2013 at 2:17 PM, Samuel Winchenbach swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com wrote: root@test1:/var/log# mysql -hmysql-ha -unova -p -eSELECT User, Host, Password FROM mysql.user; ERROR 1142 (42000) at line 1: SELECT command denied to user 'nova'@'test1' for table 'user' On Fri, Mar 8, 2013 at 2:06 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: What does this show? mysql -hmysql-ha -unova -pPASS -eSELECT User, Host, Password FROM mysql.user -jay On 03/08/2013 01:46 PM, Samuel Winchenbach wrote: Sorry, that must have been a copy and paste error. Here is what I actually ran: http://paste2.org/p/3084996 On Fri, Mar 8, 2013 at 12:40 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: On 03/08/2013 12:19 PM, Samuel Winchenbach wrote: Hi All, I have two nodes (test1 and test2) that I am trying to set up in a highly available configuration. During the setup process I tried running nova-manage service list on both nodes. It worked fine on test2, but fails on test1 even though I can connect to the database with the mysql client from test1. Here is a screen capture that shows the setup on the two nodes are basically identical: http://paste2.org/p/3084223 In the above paste you are doing: mysql -unova - hmysql-ha -u root nova -p Note you are supplying 2 -u arguments, and mysql will take the second (root). -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Incredibly odd mysql permission error
I dropped those users and no change. I also set up general logging in mysql but it really doesn't provide any additional information. Any idea for a next step I could take? I am almost at the point of taking a tcpdump and trying to recreate the salted password. :/ Thanks for the help Sam On Fri, Mar 8, 2013 at 3:38 PM, Jay Pipes jaypi...@gmail.com wrote: I'm stumped :( Looks like everything is set up correctly to me. What is interested is that your nova user access works from test2, but there is no nova@test2 user in the mysql.user table. What about doing a DROP USER nova@test1; FLUSH PRIVILEGES; and then see if that fixes things... since the nova@10.21.0.0/255.255.0.0 user is clearly working for the access from test2. Also, I'd recommend highly removing the nova@% user. Best, -jay On 03/08/2013 03:09 PM, Samuel Winchenbach wrote: http://paste2.org/p/3085807 On Fri, Mar 8, 2013 at 2:46 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: Please paste the results of SELECT User, Host, Password FROM mysql.user when running as root... Thanks! -jay On 03/08/2013 02:25 PM, Samuel Winchenbach wrote: Here are my grants. I don't know if this helps, but I did verify that the password was identical for each grant: http://paste2.org/p/3085361 On Fri, Mar 8, 2013 at 2:17 PM, Samuel Winchenbach swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com wrote: root@test1:/var/log# mysql -hmysql-ha -unova -p -eSELECT User, Host, Password FROM mysql.user; ERROR 1142 (42000) at line 1: SELECT command denied to user 'nova'@'test1' for table 'user' On Fri, Mar 8, 2013 at 2:06 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: What does this show? mysql -hmysql-ha -unova -pPASS -eSELECT User, Host, Password FROM mysql.user -jay On 03/08/2013 01:46 PM, Samuel Winchenbach wrote: Sorry, that must have been a copy and paste error. Here is what I actually ran: http://paste2.org/p/3084996 On Fri, Mar 8, 2013 at 12:40 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: On 03/08/2013 12:19 PM, Samuel Winchenbach wrote: Hi All, I have two nodes (test1 and test2) that I am trying to set up in a highly available configuration. During the setup process I tried running nova-manage service list on both nodes. It worked fine on test2, but fails on test1 even though I can connect to the database with the mysql client from test1. Here is a screen capture that shows the setup on the two nodes are basically identical: http://paste2.org/p/3084223 In the above paste you are doing: mysql -unova - hmysql-ha -u root nova -p Note you are supplying 2 -u arguments, and mysql will take the second (root). -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net mailto:openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help
Re: [Openstack] Incredibly odd mysql permission error
Sorry, I really can't think of anything :( On 03/08/2013 03:52 PM, Samuel Winchenbach wrote: I dropped those users and no change. I also set up general logging in mysql but it really doesn't provide any additional information. Any idea for a next step I could take? I am almost at the point of taking a tcpdump and trying to recreate the salted password. :/ Thanks for the help Sam On Fri, Mar 8, 2013 at 3:38 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: I'm stumped :( Looks like everything is set up correctly to me. What is interested is that your nova user access works from test2, but there is no nova@test2 user in the mysql.user table. What about doing a DROP USER nova@test1; FLUSH PRIVILEGES; and then see if that fixes things... since the nova@10.21.0.0/255.255.0.0 http://nova@10.21.0.0/255.255.0.0 user is clearly working for the access from test2. Also, I'd recommend highly removing the nova@% user. Best, -jay On 03/08/2013 03:09 PM, Samuel Winchenbach wrote: http://paste2.org/p/3085807 On Fri, Mar 8, 2013 at 2:46 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: Please paste the results of SELECT User, Host, Password FROM mysql.user when running as root... Thanks! -jay On 03/08/2013 02:25 PM, Samuel Winchenbach wrote: Here are my grants. I don't know if this helps, but I did verify that the password was identical for each grant: http://paste2.org/p/3085361 On Fri, Mar 8, 2013 at 2:17 PM, Samuel Winchenbach swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com mailto:swinc...@gmail.com wrote: root@test1:/var/log# mysql -hmysql-ha -unova -p -eSELECT User, Host, Password FROM mysql.user; ERROR 1142 (42000) at line 1: SELECT command denied to user 'nova'@'test1' for table 'user' On Fri, Mar 8, 2013 at 2:06 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: What does this show? mysql -hmysql-ha -unova -pPASS -eSELECT User, Host, Password FROM mysql.user -jay On 03/08/2013 01:46 PM, Samuel Winchenbach wrote: Sorry, that must have been a copy and paste error. Here is what I actually ran: http://paste2.org/p/3084996 On Fri, Mar 8, 2013 at 12:40 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: On 03/08/2013 12:19 PM, Samuel Winchenbach wrote: Hi All, I have two nodes (test1 and test2) that I am trying to set up in a highly available configuration. During the setup process I tried running nova-manage service list on both nodes. It worked fine on test2, but fails on test1 even though I can connect to the database with the mysql client from test1. Here is a screen capture that shows the setup on the two nodes are basically identical: http://paste2.org/p/3084223 In the above paste you are doing: mysql -unova - hmysql-ha -u root nova -p
