Re: [Openstack] Weird nova-network bridging problem with precise/essex
Narayan, If you do net.bridge.bridge-nf-call-iptables = 0 on the network controller, does floating IP still work? For each tenant/network, a subnet is created, and the nova-network has a .1 gateway configured on the bridge with the vlan interface plugged in. The packets from VMs are actually sent to the bridge for NATting. But if you doesn't allow the bridges to call iptables, it might break public access all together. Don't know, maybe I'm not understanding the sysctl flag correctly... Maybe it only applies to the packet transiting the bridge, not impacting the ones destined to the nova-network? -Simon On Fri, Jul 20, 2012 at 9:57 PM, Narayan Desai narayan.de...@gmail.comwrote: Just for the record, we found the issue. There was some filtering being applied in the bridge code which randomly (?) dropped some DNS requests. Setting: net.bridge.bridge-nf-call-arptables = 0 net.bridge.bridge-nf-call-iptables = 0 net.bridge.bridge-nf-call-ip6tables = 0 completely resolved the problem. I've written up full details here: http://buriedlede.blogspot.com/2012/07/debugging-networking-problems-with.html -nld ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Weird nova-network bridging problem with precise/essex
On Sat, Jul 21, 2012 at 6:47 AM, Xu (Simon) Chen xche...@gmail.com wrote: Narayan, If you do net.bridge.bridge-nf-call-iptables = 0 on the network controller, does floating IP still work? For each tenant/network, a subnet is created, and the nova-network has a .1 gateway configured on the bridge with the vlan interface plugged in. The packets from VMs are actually sent to the bridge for NATting. But if you doesn't allow the bridges to call iptables, it might break public access all together. Don't know, maybe I'm not understanding the sysctl flag correctly... Maybe it only applies to the packet transiting the bridge, not impacting the ones destined to the nova-network? Do you mean floating (private) or fixed (public) IPs? I suspect that you mean fixed. Fixed IPs worked regardless of this setting. The crux of the issue was that packets transiting the bridge (ie being moved from vlan200 to the virtual br200) were hitting filtering rules. It looks to me like the sysctls only apply to traffic moving across the bridge (ie exactly between vlan200 and br200), but don't bypass iptables entirely. I don't think that should effect NAT/SNAT in any case. -nld ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Weird nova-network bridging problem with precise/essex
OK, that sounds good... I was talking about fixed IP to floating IP SNAT, which happens on the bridge interfaces. But if the sysctl flag only affects transiting packets, we should be good... -Simon On Sat, Jul 21, 2012 at 8:15 AM, Narayan Desai narayan.de...@gmail.comwrote: On Sat, Jul 21, 2012 at 6:47 AM, Xu (Simon) Chen xche...@gmail.com wrote: Narayan, If you do net.bridge.bridge-nf-call-iptables = 0 on the network controller, does floating IP still work? For each tenant/network, a subnet is created, and the nova-network has a .1 gateway configured on the bridge with the vlan interface plugged in. The packets from VMs are actually sent to the bridge for NATting. But if you doesn't allow the bridges to call iptables, it might break public access all together. Don't know, maybe I'm not understanding the sysctl flag correctly... Maybe it only applies to the packet transiting the bridge, not impacting the ones destined to the nova-network? Do you mean floating (private) or fixed (public) IPs? I suspect that you mean fixed. Fixed IPs worked regardless of this setting. The crux of the issue was that packets transiting the bridge (ie being moved from vlan200 to the virtual br200) were hitting filtering rules. It looks to me like the sysctls only apply to traffic moving across the bridge (ie exactly between vlan200 and br200), but don't bypass iptables entirely. I don't think that should effect NAT/SNAT in any case. -nld ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Networking issue with VlanManager and Floating IPs
Here is what happened on a different thread: http://buriedlede.blogspot.com/2012/07/debugging-networking-problems-with.html I feel that using this might solve your issue too without changing iptables drivers... On Fri, Jul 20, 2012 at 12:58 PM, Wael Ghandour (wghandou) wghan...@cisco.com wrote: Yup, that has definitely helped, thanks a bunch Xu. Regards, Wael On Jul 20, 2012, at 8:09 AM, Xu (Simon) Chen wrote: Yes, one solution is to modify the iptables driver, so that you don't SNAT for internal subnets... So, at the beginning of the nova-network-floating-snat rules, you add something like this: -A nova-network-floating-snat -s 10.0.0.0/24 -d 10.0.0.0/24 -j ACCEPT ... -A nova-network-floating-snat -s 10.0.88.16/32 -j SNAT --to-source pub1 -A nova-network-floating-snat -s 10.0.16.7/32 -j SNAT --to-source pub2 -A nova-network-floating-snat -s 10.0.4.11/32 -j SNAT --to-source pub3 Then it should solve the unnecessary NATting issue... On Fri, Jul 20, 2012 at 10:13 AM, Wael Ghandour (wghandou) wghan...@cisco.com wrote: I can confirm that the VM traffic is undergoing NAT with using its floating IP on the *private* interface of the nova-compute node when it tries to reach the private address of the VMs belonging to the same tenant and on other compute nodes. That obviously is breaking internal connectivity Regards, Wael On Jul 20, 2012, at 5:42 AM, Xu (Simon) Chen wrote: There was an issue that we saw in an earlier nova-network... Due to multi_host configuration, the nova-network runs on every nova-compute node. Therefore the floating IP assignment happens on the compute nodes directly. So between two VMs within the same tenant on different hosts, private-public SNAT happens unnecessarily. Not sure if this is fixed in Essex... On Fri, Jul 20, 2012 at 3:49 AM, Edgar Magana (eperdomo) eperd...@cisco.com wrote: Folks, ** ** We are using Essex for our multi-host OpenStack deployment with Vlan Manager. All the private IPs are working as expected in a multi-tenant scenario but the problem that we are seen is with Floating IPs. ** ** We have three tenants, all of them are able to use Floating IPs and then VMs are reachable from the public network but the inter VMs connectivity by private IPs is totally lost. Once we dissociate the Floating IPs to the corresponding VMs, the connectivity is back. The odd part is that we are seeing this behavior in just two of the three tenants that we have tested so far. ** ** Is anyone aware of any bug or misconfiguration in Nova-network that could explain this behavior? We will be running more tests and we can provide detailed information of our environment if needed. ** ** Thanks for your help, ** ** Edgar ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Improving logging
On 21/07/12 00:08, Jay Pipes wrote: Not that I've seen, but I think it would be good to standardize on one. How about just ops? Works for me. Mikal ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] [nova] a proposal to change metadata API data
Preamble: Until now, all data that is made available by the metadata server has been data that cannot be found anywhere else at the time it may be needed. In short, an instance can't be passed it's instance id before it's instance id has been allocated so a user cannot pass it to an instance that is being started up. So whether a user wants to jump through a few hoops or not to pass their instance the instance id of itself... they simply cannot without metadata api being there to provide it at creation time. This means that the metadata server holds an uneasy place as a necessary clearing house ( evil? ) of data that just doesn't have another place to be. It's not secure, it's not authenticated, and it's a little scary that it exists at all. I wish to add some data to the metadata server that can be found somewhere else. That a user could jump through a hoop or two to add to their instances. Esteemed personages are concerned that I would be crossing the rubicon in terms of opening up the metadata api for wanton abuse. They are not without a right or reason to be concerned. And that is why I am going to attempt to explicitly classify a new category of data that we might wish to allow into the metadata server. If we can be clear about what we are allowing we can avoid abuse. I want to provide a uniform ( standardized? ) way for instances in the openstack cloud to communicate back to the OpenStack APIs without having to be provided data by the users of the cloud services. Today the mechanism by which this is done is catastrophically difficult for a new user. This uniform way for instances to interact with the openstack API that I want already sort of exists in the keystone catalog service. The problem is that you need to know where the keystone server is in the world to access it. That of course changes from deployment to deployment. Especially with the way SSL endpoints are being handled. But the metadata API server is generally known as it uses a default ip address value that can be found on any amazon compatible deployment. In fact to my knowledge it is the only known way to query openstack for data relevant to interacting with it without user interaction. And that's the key to this whole thing. I want to direct users or automation baked into instances to the keystone api and catalog service. And the only way I know how to do that is the metadata service. This api data can be classified as being first and foremost OpenStack infrastructure related. Additionally it is not available without a user providing it anywhere else. And finally it is a catalog service. I'd love some more input on whether this makes sense, or can be improved upon as an idea and formalized as a rule for using the metadata api without abusing it. Cheers, Matt Joyce PS: My current work effort in regards to this is related to passing keystone credentials to instances via pam authentication. So I can do a number of API related queries into openstack because I have credentials available to the OS that are dynamically allocated. But to make my image portable I need to not be baking in the keystone API URI. If that gives any insight on why this is important to me. Or possibly you. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] CloudFoundry @ OpenStack Design Summit
hi all anyone have the slide of cloudfoundry or any slides @ OpenSTack Design Summir last april http://blog.cloudfoundry.org/2012/07/19/oscon2012/ wanna to know the OpenPAAS movement. thx F ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] [nova] Specifying Schedule Filters Per Request?
Hi All, http://docs.openstack.org/essex/openstack-compute/admin/content/compute-options-reference.html states that the scheduler_default_filters option in nova.conf controls which filter class names to use for filtering hosts when not specified in the request. How can I specify what filters to use in the request (or can I)? I've been experimenting with DifferentHostFilter and SameHostFilter both of which seem very expensive if left in the default list (even if not passing hints to engage either). It would be nice if these we only enabled for the few cases that want them but I don't see any options to the python-novaclient to pass in a filter list. Thanks, -Jon ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
