Re: [openstack-dev] [Openstack][qa][Tempest][Network] Test for external connectivity
On 11/18/2013 08:19 PM, Salvatore Orlando wrote: Basically I think we can assume test machines have access to the Internet IMO, we should assume test machines doesn't. It's a too dangerous assumption to make. I assume you want to check servers such as www.google.com http://www.google.com are reachable I don't know why we would like to check for www.google.com by default, especially one which is geo-distributed, and for which the IPs are changing without any warning. Why not using a domain which we control, like www.openstack.org? but the routing from the external_gateway_ip to the final destination is beyond's Neutron control. DNS resolution might be interesting, but however I think resolution of external names is too beyond Neutron's control. Agreed. Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack][qa][Tempest][Network] Test for external connectivity
Thanks for the input. I apologize for the delay. 1. A working patch is here - https://review.openstack.org/#/c/55146. Reviews will be much appreciated. 2. Default setting has external_connectivity=False so tempest gate doesn't check this. I wonder if we could somehow set the neutron gate to configure external access and enable this feature for all tests? 3. Tomoe How can we test the internal network connectivity? -- I'm pinging router and dhcp ports from the VM via ssh and floating IP, since l2 and l3 agents might reside on different hosts. 4. Jeremy Stanley - test check for no fewer than three addresses -- Why? - Original Message - From: Jeremy Stanley fu...@yuggoth.org To: openstack-dev@lists.openstack.org Sent: Thursday, November 21, 2013 12:17:52 AM Subject: Re: [openstack-dev] [Openstack][qa][Tempest][Network] Test for external connectivity On 2013-11-20 14:07:49 -0800 (-0800), Sean Dague wrote: On 11/18/2013 02:41 AM, Yair Fried wrote: [...] 2. add fields in tempest.conf for * external connectivity = False/True * external ip to test against (ie 8.8.8.8) +1 for #2. In the gate we'll need to think about what that address can / should be. It may be different between different AZs. At this point I'd leave the rest of the options off the table until #2 is working reliably. [...] Having gone down this path in the past, I suggest the test check for no fewer than three addresses, sending several probes to each, and be considered successful if at least one gets a response. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack][qa][Tempest][Network] Test for external connectivity
On 2013-11-28 06:46:27 -0500 (-0500), Yair Fried wrote: [...] 4. Jeremy Stanley - test check for no fewer than three addresses -- Why? If your tests try to communicate with addresses which are truly outside your own network, and thus outside your sphere of control, you don't want them failing because of maintenance on the system being pinged (don't just trust in a third party's HA to be working around the clock--I've been burned there before as well). Also, the Internet is not generally trustworthy, least of all for a low-priority protocol like ICMP type echo/echo-reply. Send several each to several remote addresses and as long as at least one reply comes back, there is *probably* working egress. Also, even at that level of belt-and-braces paranoia, I'm pretty sure we won't want this test enabled in upstream CI unless we can fake an external address on a loopback within the DevStack server itself (or perhaps if the addresses belong to remote things on which the test already depends, such as our PyPI mirror and Git farm, but that still adds a requirement for layer 4 NAT on the DevStack VM). -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack][qa][Tempest][Network] Test for external connectivity
On 2013-11-21 13:59:16 +0100 (+0100), Salvatore Orlando wrote:
[...]
In its default configuration the traffic from the OS instance is
SNATed and the SRC IP will be rewritten to an address in the
neutron's public network range (172.24.4.224/28 by default). If
the OS instance is trying to reach a public server like
www.google.com, then, assuming ip_forward is enabled on the
devstack-gate VM, the traffic should be forwarded via the
default route with a src IP of 172.24.4.224/28.
If the above is correct, will it be possible for the IP traffic
to be correctly routed back to the Openstack instance?
We would probably need similar L4 NAT configuration on the
devstack-gate node to re-rewrite that outbound source address to the
global address of the interface (and then it will hit yet another
NAT egressing some providers, for example HPCloud).
--
{ PGP( 48F9961143495829 ); FINGER( fu...@cthulhu.yuggoth.org );
WWW( http://fungi.yuggoth.org/ ); IRC( fu...@irc.yuggoth.org#ccl );
WHOIS( STANL3-ARIN ); MUD( kin...@katarsis.mudpy.org:6669 ); }
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack][qa][Tempest][Network] Test for external connectivity
Forgive my ignorance, but I would like to make sure that packets generated from Openstack instances on neutron private networks will actually be able to reach public addresses. In its default configuration the traffic from the OS instance is SNATed and the SRC IP will be rewritten to an address in the neutron's public network range (172.24.4.224/28 by default). If the OS instance is trying to reach a public server like www.google.com, then, assuming ip_forward is enabled on the devstack-gate VM, the traffic should be forwarded via the default route with a src IP of 172.24.4.224/28. If the above is correct, will it be possible for the IP traffic to be correctly routed back to the Openstack instance? Regards, Salvatore On 20 November 2013 23:17, Jeremy Stanley fu...@yuggoth.org wrote: On 2013-11-20 14:07:49 -0800 (-0800), Sean Dague wrote: On 11/18/2013 02:41 AM, Yair Fried wrote: [...] 2. add fields in tempest.conf for * external connectivity = False/True * external ip to test against (ie 8.8.8.8) +1 for #2. In the gate we'll need to think about what that address can / should be. It may be different between different AZs. At this point I'd leave the rest of the options off the table until #2 is working reliably. [...] Having gone down this path in the past, I suggest the test check for no fewer than three addresses, sending several probes to each, and be considered successful if at least one gets a response. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack][qa][Tempest][Network] Test for external connectivity
For External connectivity beyond the network gateway, rather than pinging google.com, configuring the vm for an external DNS server and pinging it by IPaddress would be a good initial test of external connectivity. --Rocky From: Tomoe Sugihara [mailto:to...@midokura.com] Sent: Tuesday, November 19, 2013 7:31 PM To: OpenStack Development Mailing List (not for usage questions) Cc: Rami Vaknin Subject: Re: [openstack-dev] [Openstack][qa][Tempest][Network] Test for external connectivity Hi Salvatore, et al, On Mon, Nov 18, 2013 at 9:19 PM, Salvatore Orlando sorla...@nicira.commailto:sorla...@nicira.com wrote: Hi Yair, I had in mind of doing something similar. I also registered a tempest blueprint for it. Basically I think we can assume test machines have access to the Internet, but the devstack deployment might not be able to route packets from VMs to the Internet. Being able to ping the external network gateway, which by default is 172.24.4.225 is a valuable connectivity test IMHO (and that's your #1 item) For items #2 and #3 I'm not sure of your intentions; precisely so far I'm not sure if we're adding any coverage to Neutron. I assume you want to check servers such as www.google.comhttp://www.google.com are reachable, but the routing from the external_gateway_ip to the final destination is beyond's Neutron control. DNS resolution might be interesting, but however I think resolution of external names is too beyond Neutron's control. Two more things to consider on external network connectivity tests: 1) SNAT can be enabled or not. In this case we need a test that can tell us the SRC IP of the host connecting to the public external gateway, because I think that if SNAT kicks in, it should be an IP on the ext network, otherwise it should be an IP on the internal network. In this case we can use netcat to this aim, emulating a web server and use verbose output to print the source IP 2) When the connection happens from a port associated with a floating IP it is important that the SNAT happens with the floating IP address, and not with the default SNAT address. This is actually a test which would have avoided us a regression in the havana release cycle. As far as I know from the code (I'm new to tempest and might be missing something), test_network_basic_ops launches a single VM with a floating IP associated and test is performed by accessing from the tempest host to the guest VM using floating ip. So, I have some questions: - How can we test the internal network connectivity (when the tenant networks are not accessible from the host, which I believe is the case for most of the plugins)? - For external connectivity, how can we test connectivity without floating ip? - should we have another vm and control that from the access VM e.g. by ssh remote command? or - spawn specific VMs which sends traffic upon boot (e.g. metadata server + userdata with cloud init installed VM, etc) to public and assert traffics on the tempest host side? Thanks, Tomoe Regards, Salvatore On 18 November 2013 13:13, Giulio Fidente gfide...@redhat.commailto:gfide...@redhat.com wrote: On 11/18/2013 11:41 AM, Yair Fried wrote: I'm editing tempest/scenario/test_network_basic_ops.py for external connectivity as the TODO listed in its docstring. the test cases are for pinging against external ip and url to test connectivity and dns respectivly. since default deployement (devstack gate) doesn't have external connectivity I was thinking on one or all of the following I think it's a nice thing to have! 2. add fields in tempest.conf for * external connectivity = False/True * external ip to test against (ie 8.8.8.8) I like this option. One can easily disable it entirely OR pick a more relevant ip address if needed. Seems to me it would give the greatest flexibility. -- Giulio Fidente GPG KEY: 08D733BA | IRC: giulivo ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack][qa][Tempest][Network] Test for external connectivity
On 11/18/2013 02:41 AM, Yair Fried wrote: I'm editing tempest/scenario/test_network_basic_ops.py for external connectivity as the TODO listed in its docstring. the test cases are for pinging against external ip and url to test connectivity and dns respectivly. since default deployement (devstack gate) doesn't have external connectivity I was thinking on one or all of the following 1. test against public network gateway 2. add fields in tempest.conf for * external connectivity = False/True * external ip to test against (ie 8.8.8.8) +1 for #2. In the gate we'll need to think about what that address can / should be. It may be different between different AZs. At this point I'd leave the rest of the options off the table until #2 is working reliably. 3. Regarding DNS: 1. assume that external connectivity means dns is also configured 2. activly configure dns on subnet creation (based on tempest.conf Comments/suggestions will be much appriciated Regards Yair Fried ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack][qa][Tempest][Network] Test for external connectivity
On 2013-11-20 14:07:49 -0800 (-0800), Sean Dague wrote: On 11/18/2013 02:41 AM, Yair Fried wrote: [...] 2. add fields in tempest.conf for * external connectivity = False/True * external ip to test against (ie 8.8.8.8) +1 for #2. In the gate we'll need to think about what that address can / should be. It may be different between different AZs. At this point I'd leave the rest of the options off the table until #2 is working reliably. [...] Having gone down this path in the past, I suggest the test check for no fewer than three addresses, sending several probes to each, and be considered successful if at least one gets a response. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack][qa][Tempest][Network] Test for external connectivity
Hi Salvatore, et al, On Mon, Nov 18, 2013 at 9:19 PM, Salvatore Orlando sorla...@nicira.comwrote: Hi Yair, I had in mind of doing something similar. I also registered a tempest blueprint for it. Basically I think we can assume test machines have access to the Internet, but the devstack deployment might not be able to route packets from VMs to the Internet. Being able to ping the external network gateway, which by default is 172.24.4.225 is a valuable connectivity test IMHO (and that's your #1 item) For items #2 and #3 I'm not sure of your intentions; precisely so far I'm not sure if we're adding any coverage to Neutron. I assume you want to check servers such as www.google.com are reachable, but the routing from the external_gateway_ip to the final destination is beyond's Neutron control. DNS resolution might be interesting, but however I think resolution of external names is too beyond Neutron's control. Two more things to consider on external network connectivity tests: 1) SNAT can be enabled or not. In this case we need a test that can tell us the SRC IP of the host connecting to the public external gateway, because I think that if SNAT kicks in, it should be an IP on the ext network, otherwise it should be an IP on the internal network. In this case we can use netcat to this aim, emulating a web server and use verbose output to print the source IP 2) When the connection happens from a port associated with a floating IP it is important that the SNAT happens with the floating IP address, and not with the default SNAT address. This is actually a test which would have avoided us a regression in the havana release cycle. As far as I know from the code (I'm new to tempest and might be missing something), test_network_basic_ops launches a single VM with a floating IP associated and test is performed by accessing from the tempest host to the guest VM using floating ip. So, I have some questions: - How can we test the internal network connectivity (when the tenant networks are not accessible from the host, which I believe is the case for most of the plugins)? - For external connectivity, how can we test connectivity without floating ip? - should we have another vm and control that from the access VM e.g. by ssh remote command? or - spawn specific VMs which sends traffic upon boot (e.g. metadata server + userdata with cloud init installed VM, etc) to public and assert traffics on the tempest host side? Thanks, Tomoe Regards, Salvatore On 18 November 2013 13:13, Giulio Fidente gfide...@redhat.com wrote: On 11/18/2013 11:41 AM, Yair Fried wrote: I'm editing tempest/scenario/test_network_basic_ops.py for external connectivity as the TODO listed in its docstring. the test cases are for pinging against external ip and url to test connectivity and dns respectivly. since default deployement (devstack gate) doesn't have external connectivity I was thinking on one or all of the following I think it's a nice thing to have! 2. add fields in tempest.conf for * external connectivity = False/True * external ip to test against (ie 8.8.8.8) I like this option. One can easily disable it entirely OR pick a more relevant ip address if needed. Seems to me it would give the greatest flexibility. -- Giulio Fidente GPG KEY: 08D733BA | IRC: giulivo ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack][qa][Tempest][Network] Test for external connectivity
On 11/18/2013 11:41 AM, Yair Fried wrote: I'm editing tempest/scenario/test_network_basic_ops.py for external connectivity as the TODO listed in its docstring. the test cases are for pinging against external ip and url to test connectivity and dns respectivly. since default deployement (devstack gate) doesn't have external connectivity I was thinking on one or all of the following I think it's a nice thing to have! 2. add fields in tempest.conf for * external connectivity = False/True * external ip to test against (ie 8.8.8.8) I like this option. One can easily disable it entirely OR pick a more relevant ip address if needed. Seems to me it would give the greatest flexibility. -- Giulio Fidente GPG KEY: 08D733BA | IRC: giulivo ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack][qa][Tempest][Network] Test for external connectivity
Hi Yair, I had in mind of doing something similar. I also registered a tempest blueprint for it. Basically I think we can assume test machines have access to the Internet, but the devstack deployment might not be able to route packets from VMs to the Internet. Being able to ping the external network gateway, which by default is 172.24.4.225 is a valuable connectivity test IMHO (and that's your #1 item) For items #2 and #3 I'm not sure of your intentions; precisely so far I'm not sure if we're adding any coverage to Neutron. I assume you want to check servers such as www.google.com are reachable, but the routing from the external_gateway_ip to the final destination is beyond's Neutron control. DNS resolution might be interesting, but however I think resolution of external names is too beyond Neutron's control. Two more things to consider on external network connectivity tests: 1) SNAT can be enabled or not. In this case we need a test that can tell us the SRC IP of the host connecting to the public external gateway, because I think that if SNAT kicks in, it should be an IP on the ext network, otherwise it should be an IP on the internal network. In this case we can use netcat to this aim, emulating a web server and use verbose output to print the source IP 2) When the connection happens from a port associated with a floating IP it is important that the SNAT happens with the floating IP address, and not with the default SNAT address. This is actually a test which would have avoided us a regression in the havana release cycle. Regards, Salvatore On 18 November 2013 13:13, Giulio Fidente gfide...@redhat.com wrote: On 11/18/2013 11:41 AM, Yair Fried wrote: I'm editing tempest/scenario/test_network_basic_ops.py for external connectivity as the TODO listed in its docstring. the test cases are for pinging against external ip and url to test connectivity and dns respectivly. since default deployement (devstack gate) doesn't have external connectivity I was thinking on one or all of the following I think it's a nice thing to have! 2. add fields in tempest.conf for * external connectivity = False/True * external ip to test against (ie 8.8.8.8) I like this option. One can easily disable it entirely OR pick a more relevant ip address if needed. Seems to me it would give the greatest flexibility. -- Giulio Fidente GPG KEY: 08D733BA | IRC: giulivo ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
