Re: [Openstack] nova-network and corosync
Dear Steven, it seems that nova-network breaks the network also on the interface it doesn't use, I've just done another fresh install using flat_interface eth1, and i get the issue of packets being sent with the public address. I think the effects of nova-network should be restricted to some interfaces, so it doesn't breaks other services. Best Alessandro Il giorno 19/lug/2012, alle ore 11:53, Alessandro Tagliapietra tagliapietra.alessan...@gmail.com ha scritto: Il giorno 19/lug/2012, alle ore 09:55, Alessandro Tagliapietra ha scritto: Il giorno 19/lug/2012, alle ore 09:40, Alessandro Tagliapietra ha scritto: Il giorno 19/lug/2012, alle ore 00:52, Steven Dake ha scritto: On 07/18/2012 06:51 AM, Alessandro Tagliapietra wrote: Hi Steve, the problem is not that it's not listening on the correct interface, as lsof shows corosync 1485 root9u IPv4 14890 0t0 UDP 226.94.1.1:5405 corosync 1485 root 10u IPv4 14891 0t0 UDP server1:5404 corosync 1485 root 11u IPv4 14892 0t0 UDP server1:5405 where server1 is 10.8.0.1, which is correct because it's the eth1 address. The problem is that for some reason, the packets it sends to eth1 has as source ip the ip of eth0, which is the public internet connected interface, so like: 15:44:34.135411 IP 5.9.x.x.5404 226.94.1.1.5405: UDP, length 82 15:44:34.238762 IP 5.9.x.x.5404 226.94.1.1.5405: UDP, length 82 which is wrong. my ip r is this: default via 5.9.x.x dev eth0 metric 100 5.9.x.x/27 via 5.9.x.x dev eth0 5.9.x.x/27 dev eth0 proto kernel scope link src 5.9.x.x 10.0.0.0/16 dev eth2 proto kernel scope link src 10.0.0.1 10.8.0.0/16 dev eth1 proto kernel scope link src 10.8.0.1 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 As you can see packets to eth1 should have 10.8.0.1 as source, not eth0 ip. Odd - Are you using udpu mode? Which version? Can you subscribe to the corosync list and we can follow-up there? When starting this discussion i was using ubuntu 12.04 repositories so 1.4.2, today i've installed 2.0.1, i've added corosync mailing list in cc. I have to specify that this problem happens only when i've installed nova network on both nodes (i'm using multi-host). I've tried with udpu mode specifying the nodelist and it works fine. I'm going to switch back to 1.4.2 to use regular ubuntu packages and updates and see if that works too. An update, i've removed eth2 (which was just a virtual interface) and set back flat_interface to eth1, now after a reboot i've got the same issue as before, packets are sent using public ip, but if a do a corosync restart after loggin in it works normally. Maybe it needs some delay to work? Regards Regards Let me paste some configs: nova.conf: http://pastie.org/private/c5tcutro6tp0s1te5yq (i've tried with flat_interface eth1 and eth2) ifconfig: http://pastie.org/private/7li8gwodr0ulgbafsi4edw corosync.conf: http://pastie.org/private/cjrtqx9bspgmff7rlye6ew (tried with 10.8.0.0 also as bindnetaddr) ip r: http://pastie.org/private/ckqhy0vqoiwzewuj17v7g iptables-save: http://pastie.org/private/yvypgi5ovs2rtcingrq5iw (all generated by nova-network) If you need any other things just ask. Btw, i'm going to try with udpu now. Regards http://lists.corosync.org/mailman/listinfo/discuss thanks -steve Regards Il giorno 18/lug/2012, alle ore 15:18, Steven Dake ha scritto: On 07/18/2012 03:50 AM, Alessandro Tagliapietra wrote: Hello, i've 2 machines, running ubuntu 12.04, i've installed corosync + pacemaker and it was working fine. Corosync is using eth1 with 10.8.0.1 and 10.8.0.2 as ip of the hosts, i've got keystone, glance, nova api-cert-scheduler, mysql, rabbitmq working in HA with pacemaker. The problem comes after installing nova-network and nova-compute, i've used this nova.conf: http://pastie.org/private/ddwva8kvaypqrxk7rifvba and after nova-compute started and hosts rebooted i can't get to work corosync, the problem seems that when hosts send packets in eth1 to multicast address, the source ip is the public one, not the 10.8.0.x one. After disabling nova-network on boot everything works. I've also tried to create a virtual eth2 device and set flat_interface to eth2, but it seems that still nova-network break the configuration as corosync still uses public ip for private lan. Any idea? Corosync goes to great pains to route packets across the interface identified in the corosync.conf file. If you are using a subnet definition ie: bindnetaddr: 10.8.0.0, it may be that the interface's netmask is causing a rebind to the new interface when nova network starts. One way to force binding to a specific interface when your network is not configured in a typical fashion is to identify the bindnetaddr exactly: ie: bindnetaddr:
Re: [Openstack] nova-network and corosync
Il giorno 19/lug/2012, alle ore 00:52, Steven Dake ha scritto: On 07/18/2012 06:51 AM, Alessandro Tagliapietra wrote: Hi Steve, the problem is not that it's not listening on the correct interface, as lsof shows corosync 1485 root9u IPv4 14890 0t0 UDP 226.94.1.1:5405 corosync 1485 root 10u IPv4 14891 0t0 UDP server1:5404 corosync 1485 root 11u IPv4 14892 0t0 UDP server1:5405 where server1 is 10.8.0.1, which is correct because it's the eth1 address. The problem is that for some reason, the packets it sends to eth1 has as source ip the ip of eth0, which is the public internet connected interface, so like: 15:44:34.135411 IP 5.9.x.x.5404 226.94.1.1.5405: UDP, length 82 15:44:34.238762 IP 5.9.x.x.5404 226.94.1.1.5405: UDP, length 82 which is wrong. my ip r is this: default via 5.9.x.x dev eth0 metric 100 5.9.x.x/27 via 5.9.x.x dev eth0 5.9.x.x/27 dev eth0 proto kernel scope link src 5.9.x.x 10.0.0.0/16 dev eth2 proto kernel scope link src 10.0.0.1 10.8.0.0/16 dev eth1 proto kernel scope link src 10.8.0.1 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 As you can see packets to eth1 should have 10.8.0.1 as source, not eth0 ip. Odd - Are you using udpu mode? Which version? Can you subscribe to the corosync list and we can follow-up there? When starting this discussion i was using ubuntu 12.04 repositories so 1.4.2, today i've installed 2.0.1, i've added corosync mailing list in cc. I have to specify that this problem happens only when i've installed nova network on both nodes (i'm using multi-host). Let me paste some configs: nova.conf: http://pastie.org/private/c5tcutro6tp0s1te5yq (i've tried with flat_interface eth1 and eth2) ifconfig: http://pastie.org/private/7li8gwodr0ulgbafsi4edw corosync.conf: http://pastie.org/private/cjrtqx9bspgmff7rlye6ew (tried with 10.8.0.0 also as bindnetaddr) ip r: http://pastie.org/private/ckqhy0vqoiwzewuj17v7g iptables-save: http://pastie.org/private/yvypgi5ovs2rtcingrq5iw (all generated by nova-network) If you need any other things just ask. Btw, i'm going to try with udpu now. Regards http://lists.corosync.org/mailman/listinfo/discuss thanks -steve Regards Il giorno 18/lug/2012, alle ore 15:18, Steven Dake ha scritto: On 07/18/2012 03:50 AM, Alessandro Tagliapietra wrote: Hello, i've 2 machines, running ubuntu 12.04, i've installed corosync + pacemaker and it was working fine. Corosync is using eth1 with 10.8.0.1 and 10.8.0.2 as ip of the hosts, i've got keystone, glance, nova api-cert-scheduler, mysql, rabbitmq working in HA with pacemaker. The problem comes after installing nova-network and nova-compute, i've used this nova.conf: http://pastie.org/private/ddwva8kvaypqrxk7rifvba and after nova-compute started and hosts rebooted i can't get to work corosync, the problem seems that when hosts send packets in eth1 to multicast address, the source ip is the public one, not the 10.8.0.x one. After disabling nova-network on boot everything works. I've also tried to create a virtual eth2 device and set flat_interface to eth2, but it seems that still nova-network break the configuration as corosync still uses public ip for private lan. Any idea? Corosync goes to great pains to route packets across the interface identified in the corosync.conf file. If you are using a subnet definition ie: bindnetaddr: 10.8.0.0, it may be that the interface's netmask is causing a rebind to the new interface when nova network starts. One way to force binding to a specific interface when your network is not configured in a typical fashion is to identify the bindnetaddr exactly: ie: bindnetaddr: 10.8.0.1 Regards -steve Best Regards -- Alessandro Tagliapietra | VISup srl piazza 4 novembre 7 20124 Milano http://www.visup.it ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Alessandro Tagliapietra | VISup srl piazza 4 novembre 7 20124 Milano http://www.visup.it___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] nova-network and corosync
Il giorno 19/lug/2012, alle ore 09:55, Alessandro Tagliapietra ha scritto: Il giorno 19/lug/2012, alle ore 09:40, Alessandro Tagliapietra ha scritto: Il giorno 19/lug/2012, alle ore 00:52, Steven Dake ha scritto: On 07/18/2012 06:51 AM, Alessandro Tagliapietra wrote: Hi Steve, the problem is not that it's not listening on the correct interface, as lsof shows corosync 1485 root9u IPv4 14890 0t0 UDP 226.94.1.1:5405 corosync 1485 root 10u IPv4 14891 0t0 UDP server1:5404 corosync 1485 root 11u IPv4 14892 0t0 UDP server1:5405 where server1 is 10.8.0.1, which is correct because it's the eth1 address. The problem is that for some reason, the packets it sends to eth1 has as source ip the ip of eth0, which is the public internet connected interface, so like: 15:44:34.135411 IP 5.9.x.x.5404 226.94.1.1.5405: UDP, length 82 15:44:34.238762 IP 5.9.x.x.5404 226.94.1.1.5405: UDP, length 82 which is wrong. my ip r is this: default via 5.9.x.x dev eth0 metric 100 5.9.x.x/27 via 5.9.x.x dev eth0 5.9.x.x/27 dev eth0 proto kernel scope link src 5.9.x.x 10.0.0.0/16 dev eth2 proto kernel scope link src 10.0.0.1 10.8.0.0/16 dev eth1 proto kernel scope link src 10.8.0.1 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 As you can see packets to eth1 should have 10.8.0.1 as source, not eth0 ip. Odd - Are you using udpu mode? Which version? Can you subscribe to the corosync list and we can follow-up there? When starting this discussion i was using ubuntu 12.04 repositories so 1.4.2, today i've installed 2.0.1, i've added corosync mailing list in cc. I have to specify that this problem happens only when i've installed nova network on both nodes (i'm using multi-host). I've tried with udpu mode specifying the nodelist and it works fine. I'm going to switch back to 1.4.2 to use regular ubuntu packages and updates and see if that works too. An update, i've removed eth2 (which was just a virtual interface) and set back flat_interface to eth1, now after a reboot i've got the same issue as before, packets are sent using public ip, but if a do a corosync restart after loggin in it works normally. Maybe it needs some delay to work? Regards Regards Let me paste some configs: nova.conf: http://pastie.org/private/c5tcutro6tp0s1te5yq (i've tried with flat_interface eth1 and eth2) ifconfig: http://pastie.org/private/7li8gwodr0ulgbafsi4edw corosync.conf: http://pastie.org/private/cjrtqx9bspgmff7rlye6ew (tried with 10.8.0.0 also as bindnetaddr) ip r: http://pastie.org/private/ckqhy0vqoiwzewuj17v7g iptables-save: http://pastie.org/private/yvypgi5ovs2rtcingrq5iw (all generated by nova-network) If you need any other things just ask. Btw, i'm going to try with udpu now. Regards http://lists.corosync.org/mailman/listinfo/discuss thanks -steve Regards Il giorno 18/lug/2012, alle ore 15:18, Steven Dake ha scritto: On 07/18/2012 03:50 AM, Alessandro Tagliapietra wrote: Hello, i've 2 machines, running ubuntu 12.04, i've installed corosync + pacemaker and it was working fine. Corosync is using eth1 with 10.8.0.1 and 10.8.0.2 as ip of the hosts, i've got keystone, glance, nova api-cert-scheduler, mysql, rabbitmq working in HA with pacemaker. The problem comes after installing nova-network and nova-compute, i've used this nova.conf: http://pastie.org/private/ddwva8kvaypqrxk7rifvba and after nova-compute started and hosts rebooted i can't get to work corosync, the problem seems that when hosts send packets in eth1 to multicast address, the source ip is the public one, not the 10.8.0.x one. After disabling nova-network on boot everything works. I've also tried to create a virtual eth2 device and set flat_interface to eth2, but it seems that still nova-network break the configuration as corosync still uses public ip for private lan. Any idea? Corosync goes to great pains to route packets across the interface identified in the corosync.conf file. If you are using a subnet definition ie: bindnetaddr: 10.8.0.0, it may be that the interface's netmask is causing a rebind to the new interface when nova network starts. One way to force binding to a specific interface when your network is not configured in a typical fashion is to identify the bindnetaddr exactly: ie: bindnetaddr: 10.8.0.1 Regards -steve Best Regards -- Alessandro Tagliapietra | VISup srl piazza 4 novembre 7 20124 Milano http://www.visup.it ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Alessandro Tagliapietra | VISup srl piazza 4 novembre 7 20124 Milano
[Openstack] nova-network and corosync
Hello, i've 2 machines, running ubuntu 12.04, i've installed corosync + pacemaker and it was working fine. Corosync is using eth1 with 10.8.0.1 and 10.8.0.2 as ip of the hosts, i've got keystone, glance, nova api-cert-scheduler, mysql, rabbitmq working in HA with pacemaker. The problem comes after installing nova-network and nova-compute, i've used this nova.conf: http://pastie.org/private/ddwva8kvaypqrxk7rifvba and after nova-compute started and hosts rebooted i can't get to work corosync, the problem seems that when hosts send packets in eth1 to multicast address, the source ip is the public one, not the 10.8.0.x one. After disabling nova-network on boot everything works. I've also tried to create a virtual eth2 device and set flat_interface to eth2, but it seems that still nova-network break the configuration as corosync still uses public ip for private lan. Any idea? Best Regards -- Alessandro Tagliapietra | VISup srl piazza 4 novembre 7 20124 Milano http://www.visup.it___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] nova-network and corosync
On 07/18/2012 03:50 AM, Alessandro Tagliapietra wrote: Hello, i've 2 machines, running ubuntu 12.04, i've installed corosync + pacemaker and it was working fine. Corosync is using eth1 with 10.8.0.1 and 10.8.0.2 as ip of the hosts, i've got keystone, glance, nova api-cert-scheduler, mysql, rabbitmq working in HA with pacemaker. The problem comes after installing nova-network and nova-compute, i've used this nova.conf: http://pastie.org/private/ddwva8kvaypqrxk7rifvba and after nova-compute started and hosts rebooted i can't get to work corosync, the problem seems that when hosts send packets in eth1 to multicast address, the source ip is the public one, not the 10.8.0.x one. After disabling nova-network on boot everything works. I've also tried to create a virtual eth2 device and set flat_interface to eth2, but it seems that still nova-network break the configuration as corosync still uses public ip for private lan. Any idea? Corosync goes to great pains to route packets across the interface identified in the corosync.conf file. If you are using a subnet definition ie: bindnetaddr: 10.8.0.0, it may be that the interface's netmask is causing a rebind to the new interface when nova network starts. One way to force binding to a specific interface when your network is not configured in a typical fashion is to identify the bindnetaddr exactly: ie: bindnetaddr: 10.8.0.1 Regards -steve Best Regards -- Alessandro Tagliapietra | VISup srl piazza 4 novembre 7 20124 Milano http://www.visup.it ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] nova-network and corosync
Hi Steve, the problem is not that it's not listening on the correct interface, as lsof shows corosync 1485 root9u IPv4 14890 0t0 UDP 226.94.1.1:5405 corosync 1485 root 10u IPv4 14891 0t0 UDP server1:5404 corosync 1485 root 11u IPv4 14892 0t0 UDP server1:5405 where server1 is 10.8.0.1, which is correct because it's the eth1 address. The problem is that for some reason, the packets it sends to eth1 has as source ip the ip of eth0, which is the public internet connected interface, so like: 15:44:34.135411 IP 5.9.x.x.5404 226.94.1.1.5405: UDP, length 82 15:44:34.238762 IP 5.9.x.x.5404 226.94.1.1.5405: UDP, length 82 which is wrong. my ip r is this: default via 5.9.x.x dev eth0 metric 100 5.9.x.x/27 via 5.9.x.x dev eth0 5.9.x.x/27 dev eth0 proto kernel scope link src 5.9.x.x 10.0.0.0/16 dev eth2 proto kernel scope link src 10.0.0.1 10.8.0.0/16 dev eth1 proto kernel scope link src 10.8.0.1 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 As you can see packets to eth1 should have 10.8.0.1 as source, not eth0 ip. Regards Il giorno 18/lug/2012, alle ore 15:18, Steven Dake ha scritto: On 07/18/2012 03:50 AM, Alessandro Tagliapietra wrote: Hello, i've 2 machines, running ubuntu 12.04, i've installed corosync + pacemaker and it was working fine. Corosync is using eth1 with 10.8.0.1 and 10.8.0.2 as ip of the hosts, i've got keystone, glance, nova api-cert-scheduler, mysql, rabbitmq working in HA with pacemaker. The problem comes after installing nova-network and nova-compute, i've used this nova.conf: http://pastie.org/private/ddwva8kvaypqrxk7rifvba and after nova-compute started and hosts rebooted i can't get to work corosync, the problem seems that when hosts send packets in eth1 to multicast address, the source ip is the public one, not the 10.8.0.x one. After disabling nova-network on boot everything works. I've also tried to create a virtual eth2 device and set flat_interface to eth2, but it seems that still nova-network break the configuration as corosync still uses public ip for private lan. Any idea? Corosync goes to great pains to route packets across the interface identified in the corosync.conf file. If you are using a subnet definition ie: bindnetaddr: 10.8.0.0, it may be that the interface's netmask is causing a rebind to the new interface when nova network starts. One way to force binding to a specific interface when your network is not configured in a typical fashion is to identify the bindnetaddr exactly: ie: bindnetaddr: 10.8.0.1 Regards -steve Best Regards -- Alessandro Tagliapietra | VISup srl piazza 4 novembre 7 20124 Milano http://www.visup.it ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
