Re: [Openstack] Strange network behavior
I should add, that it looks like none of the iptables rules are being setup for the floating IP. It is bound to the right interface in ip addr, but my iptables look as follows: (You'll note that 10.0.40.129 is conspicuous by its absence) Ideas? Chain INPUT (policy ACCEPT 47238 packets, 20M bytes) pkts bytes target prot opt in out source destination 47393 20M nova-network-INPUT all -- * * 0.0.0.0/0 0.0.0.0/0 47238 20M nova-compute-INPUT all -- * * 0.0.0.0/0 0.0.0.0/0 47238 20M nova-api-INPUT all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT udp -- virbr0 * 0.0.0.0/0 0.0.0.0/0udp dpt:53 0 0 ACCEPT tcp -- virbr0 * 0.0.0.0/0 0.0.0.0/0tcp dpt:53 0 0 ACCEPT udp -- virbr0 * 0.0.0.0/0 0.0.0.0/0udp dpt:67 0 0 ACCEPT tcp -- virbr0 * 0.0.0.0/0 0.0.0.0/0tcp dpt:67 Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 225 18900 nova-filter-top all -- * * 0.0.0.0/0 0.0.0.0/0 208 17472 nova-network-FORWARD all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 nova-compute-FORWARD all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 nova-api-FORWARD all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- * virbr0 0.0.0.0/0 192.168.122.0/24 state RELATED,ESTABLISHED 0 0 ACCEPT all -- virbr0 * 192.168.122.0/24 0.0.0.0/0 0 0 ACCEPT all -- virbr0 virbr0 0.0.0.0/0 0.0.0.0/0 0 0 REJECT all -- * virbr0 0.0.0.0/0 0.0.0.0/0reject-with icmp-port-unreachable 0 0 REJECT all -- virbr0 * 0.0.0.0/0 0.0.0.0/0reject-with icmp-port-unreachable Chain OUTPUT (policy ACCEPT 38296 packets, 23M bytes) pkts bytes target prot opt in out source destination 38667 23M nova-filter-top all -- * * 0.0.0.0/0 0.0.0.0/0 38296 23M nova-network-OUTPUT all -- * * 0.0.0.0/0 0.0.0.0/0 38296 23M nova-compute-OUTPUT all -- * * 0.0.0.0/0 0.0.0.0/0 38296 23M nova-api-OUTPUT all -- * * 0.0.0.0/0 0.0.0.0/0 Chain nova-api-FORWARD (1 references) pkts bytes target prot opt in out source destination Chain nova-api-INPUT (1 references) pkts bytes target prot opt in out source destination 0 0 ACCEPT tcp -- * * 0.0.0.0/0 10.0.0.250 tcp dpt:8775 Chain nova-api-OUTPUT (1 references) pkts bytes target prot opt in out source destination Chain nova-api-local (1 references) pkts bytes target prot opt in out source destination Chain nova-compute-FORWARD (1 references) pkts bytes target prot opt in out source destination 0 0 ACCEPT all -- br41 * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- * br410.0.0.0/0 0.0.0.0/0 Chain nova-compute-INPUT (1 references) pkts bytes target prot opt in out source destination Chain nova-compute-OUTPUT (1 references) pkts bytes target prot opt in out source destination Chain nova-compute-inst-2 (1 references) pkts bytes target prot opt in out source destination 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0state INVALID 388 37807 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0state RELATED,ESTABLISHED 0 0 nova-compute-provider all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT udp -- * * 10.0.41.1 0.0.0.0/0udp spt:67 dpt:68 0 0 ACCEPT all -- * * 10.0.41.0/24 0.0.0.0/0 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0tcp dpt:22 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0tcp dpt:80 0 0 nova-compute-sg-fallback all -- * * 0.0.0.0/0 0.0.0.0/0 Chain nova-compute-local (1 references) pkts bytes target prot opt in out source destination 388 37807 nova-compute-inst-2 all -- * * 0.0.0.0/0 10.0.41.4 Chain nova-compute-provider (1 references) pkts bytes target prot opt in out source destination Chain nova-compute-sg-fallback (1 references) pkts bytes target prot opt in out source destination 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 Chain nova-filter-top (2 references) pkts bytes target prot opt in out source destination 38892 23M nova-network-local all -- * * 0.0.0.0/0 0.0.0.0/0 38892 23M nova-compute-local all -- * * 0.0.0.0/0 0.0.0.0/0 38504 23M nova-api-local all -- * * 0.0.0.0/0 0.0.0.0/0 Chain nova-network-FORWARD (1 references) pkts bytes target
Re: [Openstack] Strange network behavior
Hi Vish et al. I still can't make head nor tail of it. ICMP works in both directions fine, but when I try to ssh out from the VM (even with the dmz_cidr flags) the SYN gets through un-snatted ok, then my desktop SYN-ACKs back, but the virt never gets to see it. Instead, the snat layer sends a RST. I don't want any NAT at all. I just want the virts bridged on to the VLAN. Is there a way to do that? Kind regards -- joe. On 9 November 2012 19:56, Vishvananda Ishaya vishvana...@gmail.com wrote: What is the ip address of your workstation? You may be running into something similar to this issue: http://lists.openstack.org/pipermail/openstack-dev/2012-September/001212.html I suspect either: a) Traffic not getting snatted when it should. This is usually due to overlapping ranges between your internal network and fixed_range this would be fixed by limiting fixed_range in your config file to just the instances range: (fixed_range=10.0.41.0/24 ?) or b) Traffic getting snatted when it shouldn't. This is usually because your workstation ip is on an ip that is internally routable but not routable from the external network of the compute host, so it can't get back to the snatted ip this is fixed by stopping snatting to the workstation by setting dmz_cidr to a value that includes your workstation network: (dmz_cidr=10.0.0.0/24?) Vish On Nov 9, 2012, at 9:14 AM, Joe Warren-Meeks joe.warren.me...@gmail.com wrote: Hi all, I've managed to get Openstack pretty much up and running as I wanted it. I do have, however, a rather strange networking issue. I created the network with nova-manage network create --fixed_range_v4=10.0.41.0/24 --num_networks=1 --bridge=br41 --bridge_interface=eth0 --label=development --gateway=10.0.41.1 --dns1=10.0.0.2 --vlan=41 --project_id=XXX And i can boot instances fine. I've configured the default security group to allow port 22, 80 and ICMP -1 in and I can ping from my work station to the virtual instance ok: joe@kaneda:~$ ping 10.0.41.3 PING 10.0.41.3 (10.0.41.3) 56(84) bytes of data. 64 bytes from 10.0.41.3: icmp_req=1 ttl=63 time=1.18 ms And i can ping from the virt back too: ubuntu@test:~$ ping 10.0.0.240 PING 10.0.0.240 (10.0.0.240) 56(84) bytes of data. 64 bytes from 10.0.0.240: icmp_req=1 ttl=64 time=0.713 ms I can SSH out from the virt to a host in the outside world fine: ubuntu@test:~$ ssh joe@X joe@XX password: -bash: fortune: command not found joe@dixon:~ $ BUT I can't ssh from the virt to my workstation, nor from my workstation to the Virt. Neither does http work. What I am seeing in Tcpdump is a lot of incorrect cksums. This happens with all Tcp connections. 17:12:38.539784 IP (tos 0x0, ttl 64, id 53611, offset 0, flags [DF], proto TCP (6), length 60) 10.0.0.240.56791 10.0.41.3.22: Flags [S], cksum 0x3e21 (incorrect - 0x6de2), seq 2650163743, win 14600, options [mss 1460,sackOK,TS val 28089204 ecr 0,nop,wscale 6], length 0 17:12:38.585279 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.0.41.3.22 10.0.0.240.56791: Flags [S.], cksum 0x3e21 (incorrect - 0xe5c5), seq 1530502549, ack 3098447117, win 14480, options [mss 1460,sackOK,TS val 340493 ecr 28089204,nop,wscale 3], length 0 Anyone come across this before? -- joe. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Strange network behavior
Hey guys, Ignore this q. I didn't really have my head around how Openstack works and I think I get it now. Thanks for all your help. -- joe. On 12 November 2012 10:12, Joe Warren-Meeks joe.warren.me...@gmail.comwrote: Hi Vish et al. I still can't make head nor tail of it. ICMP works in both directions fine, but when I try to ssh out from the VM (even with the dmz_cidr flags) the SYN gets through un-snatted ok, then my desktop SYN-ACKs back, but the virt never gets to see it. Instead, the snat layer sends a RST. I don't want any NAT at all. I just want the virts bridged on to the VLAN. Is there a way to do that? Kind regards -- joe. On 9 November 2012 19:56, Vishvananda Ishaya vishvana...@gmail.comwrote: What is the ip address of your workstation? You may be running into something similar to this issue: http://lists.openstack.org/pipermail/openstack-dev/2012-September/001212.html I suspect either: a) Traffic not getting snatted when it should. This is usually due to overlapping ranges between your internal network and fixed_range this would be fixed by limiting fixed_range in your config file to just the instances range: (fixed_range=10.0.41.0/24 ?) or b) Traffic getting snatted when it shouldn't. This is usually because your workstation ip is on an ip that is internally routable but not routable from the external network of the compute host, so it can't get back to the snatted ip this is fixed by stopping snatting to the workstation by setting dmz_cidr to a value that includes your workstation network: (dmz_cidr=10.0.0.0/24?) Vish On Nov 9, 2012, at 9:14 AM, Joe Warren-Meeks joe.warren.me...@gmail.com wrote: Hi all, I've managed to get Openstack pretty much up and running as I wanted it. I do have, however, a rather strange networking issue. I created the network with nova-manage network create --fixed_range_v4=10.0.41.0/24--num_networks=1 --bridge=br41 --bridge_interface=eth0 --label=development --gateway=10.0.41.1 --dns1=10.0.0.2 --vlan=41 --project_id=XXX And i can boot instances fine. I've configured the default security group to allow port 22, 80 and ICMP -1 in and I can ping from my work station to the virtual instance ok: joe@kaneda:~$ ping 10.0.41.3 PING 10.0.41.3 (10.0.41.3) 56(84) bytes of data. 64 bytes from 10.0.41.3: icmp_req=1 ttl=63 time=1.18 ms And i can ping from the virt back too: ubuntu@test:~$ ping 10.0.0.240 PING 10.0.0.240 (10.0.0.240) 56(84) bytes of data. 64 bytes from 10.0.0.240: icmp_req=1 ttl=64 time=0.713 ms I can SSH out from the virt to a host in the outside world fine: ubuntu@test:~$ ssh joe@X joe@XX password: -bash: fortune: command not found joe@dixon:~ $ BUT I can't ssh from the virt to my workstation, nor from my workstation to the Virt. Neither does http work. What I am seeing in Tcpdump is a lot of incorrect cksums. This happens with all Tcp connections. 17:12:38.539784 IP (tos 0x0, ttl 64, id 53611, offset 0, flags [DF], proto TCP (6), length 60) 10.0.0.240.56791 10.0.41.3.22: Flags [S], cksum 0x3e21 (incorrect - 0x6de2), seq 2650163743, win 14600, options [mss 1460,sackOK,TS val 28089204 ecr 0,nop,wscale 6], length 0 17:12:38.585279 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.0.41.3.22 10.0.0.240.56791: Flags [S.], cksum 0x3e21 (incorrect - 0xe5c5), seq 1530502549, ack 3098447117, win 14480, options [mss 1460,sackOK,TS val 340493 ecr 28089204,nop,wscale 3], length 0 Anyone come across this before? -- joe. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Strange network behavior
Hi all, I've managed to get Openstack pretty much up and running as I wanted it. I do have, however, a rather strange networking issue. I created the network with nova-manage network create --fixed_range_v4=10.0.41.0/24 --num_networks=1 --bridge=br41 --bridge_interface=eth0 --label=development --gateway=10.0.41.1 --dns1=10.0.0.2 --vlan=41 --project_id=XXX And i can boot instances fine. I've configured the default security group to allow port 22, 80 and ICMP -1 in and I can ping from my work station to the virtual instance ok: joe@kaneda:~$ ping 10.0.41.3 PING 10.0.41.3 (10.0.41.3) 56(84) bytes of data. 64 bytes from 10.0.41.3: icmp_req=1 ttl=63 time=1.18 ms And i can ping from the virt back too: ubuntu@test:~$ ping 10.0.0.240 PING 10.0.0.240 (10.0.0.240) 56(84) bytes of data. 64 bytes from 10.0.0.240: icmp_req=1 ttl=64 time=0.713 ms I can SSH out from the virt to a host in the outside world fine: ubuntu@test:~$ ssh joe@X joe@XX password: -bash: fortune: command not found joe@dixon:~ $ BUT I can't ssh from the virt to my workstation, nor from my workstation to the Virt. Neither does http work. What I am seeing in Tcpdump is a lot of incorrect cksums. This happens with all Tcp connections. 17:12:38.539784 IP (tos 0x0, ttl 64, id 53611, offset 0, flags [DF], proto TCP (6), length 60) 10.0.0.240.56791 10.0.41.3.22: Flags [S], cksum 0x3e21 (incorrect - 0x6de2), seq 2650163743, win 14600, options [mss 1460,sackOK,TS val 28089204 ecr 0,nop,wscale 6], length 0 17:12:38.585279 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.0.41.3.22 10.0.0.240.56791: Flags [S.], cksum 0x3e21 (incorrect - 0xe5c5), seq 1530502549, ack 3098447117, win 14480, options [mss 1460,sackOK,TS val 340493 ecr 28089204,nop,wscale 3], length 0 Anyone come across this before? -- joe. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Strange network behavior
On 11/09/2012 09:14 AM, Joe Warren-Meeks wrote:
What I am seeing in Tcpdump is a lot of incorrect cksums. This happens
with all Tcp connections.
17:12:38.539784 IP (tos 0x0, ttl 64, id 53611, offset 0, flags [DF],
proto TCP (6), length 60)
10.0.0.240.56791 10.0.41.3.22: Flags [S], cksum 0x3e21 (incorrect
- 0x6de2), seq 2650163743, win 14600, options [mss 1460,sackOK,TS val
28089204 ecr 0,nop,wscale 6], length 0
17:12:38.585279 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
TCP (6), length 60)
10.0.41.3.22 10.0.0.240.56791: Flags [S.], cksum 0x3e21
(incorrect - 0xe5c5), seq 1530502549, ack 3098447117, win 14480,
options [mss 1460,sackOK,TS val 340493 ecr 28089204,nop,wscale 3], length 0
Anyone come across this before?
When a Network Interface card (NIC) offers ChecKsum Offload (CKO) in the
outbound/transmit direction, the computation of the Layer 4 (eg TCP,
UDP) checksum is deferred to the NIC. You can see if a given
interface/NIC has checksum offload, or other offloads, enabled via
ethtool -k interface
When the packet passes the promiscuous tap on the way down the stack to
a NIC offering CKO, the packet will be in essence unchecksummed and so
tcpdump will report that as an incorrect checksum. It is therefore
possibly a false positive. I say possibly because I just did a quick
netperf test on my Ubuntu 11.04 workstation to see what the SYN's looked
like there, and I didn't see an incorrect checksum warning out of
tcpdump though I know the egress interface is offering outbound CKO,
making me think that TCP may not bother with CKO for small segments like
SYNchronize segments. One way to check if the incorrect checksum report
is valid would be to run tcpdump on 10.0.41.3 as well. And/or disable
CKO if you see it is enabled in ethtool.
I would not have expected to see invalid checksums reported by tcpdump
for an inbound packet though. Might be good to cross-check with the
netstat statistics.
There is what appears to be an inconsistency between those two TCP
segments. The sequence number of the SYNchronize (that 'S' in flags)
segment from 10.0.0.240.56791 to 10.0.41.3.22 is 2650163743. The SYN
from 10.0.41.3.22 to 10.0.0.240.56791 though has the ACK flag set ('.')
but the ACKnowledgement number is 3098447117 rather than what I would
have expected - 2650163744.
FWIW, that there was a SYN-ACK sent in response to the SYN in the first
place suggests that 10.0.41.3 received what it thought was a properly
checksummed SYN segment. All the more reason I suspect to take traces
at both ends and compare the packets byte by byte.
rick jones
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Strange network behavior
What is the ip address of your workstation? You may be running into something similar to this issue: http://lists.openstack.org/pipermail/openstack-dev/2012-September/001212.html I suspect either: a) Traffic not getting snatted when it should. This is usually due to overlapping ranges between your internal network and fixed_range this would be fixed by limiting fixed_range in your config file to just the instances range: (fixed_range=10.0.41.0/24 ?) or b) Traffic getting snatted when it shouldn't. This is usually because your workstation ip is on an ip that is internally routable but not routable from the external network of the compute host, so it can't get back to the snatted ip this is fixed by stopping snatting to the workstation by setting dmz_cidr to a value that includes your workstation network: (dmz_cidr=10.0.0.0/24 ?) Vish On Nov 9, 2012, at 9:14 AM, Joe Warren-Meeks joe.warren.me...@gmail.com wrote: Hi all, I've managed to get Openstack pretty much up and running as I wanted it. I do have, however, a rather strange networking issue. I created the network with nova-manage network create --fixed_range_v4=10.0.41.0/24 --num_networks=1 --bridge=br41 --bridge_interface=eth0 --label=development --gateway=10.0.41.1 --dns1=10.0.0.2 --vlan=41 --project_id=XXX And i can boot instances fine. I've configured the default security group to allow port 22, 80 and ICMP -1 in and I can ping from my work station to the virtual instance ok: joe@kaneda:~$ ping 10.0.41.3 PING 10.0.41.3 (10.0.41.3) 56(84) bytes of data. 64 bytes from 10.0.41.3: icmp_req=1 ttl=63 time=1.18 ms And i can ping from the virt back too: ubuntu@test:~$ ping 10.0.0.240 PING 10.0.0.240 (10.0.0.240) 56(84) bytes of data. 64 bytes from 10.0.0.240: icmp_req=1 ttl=64 time=0.713 ms I can SSH out from the virt to a host in the outside world fine: ubuntu@test:~$ ssh joe@X joe@XX password: -bash: fortune: command not found joe@dixon:~ $ BUT I can't ssh from the virt to my workstation, nor from my workstation to the Virt. Neither does http work. What I am seeing in Tcpdump is a lot of incorrect cksums. This happens with all Tcp connections. 17:12:38.539784 IP (tos 0x0, ttl 64, id 53611, offset 0, flags [DF], proto TCP (6), length 60) 10.0.0.240.56791 10.0.41.3.22: Flags [S], cksum 0x3e21 (incorrect - 0x6de2), seq 2650163743, win 14600, options [mss 1460,sackOK,TS val 28089204 ecr 0,nop,wscale 6], length 0 17:12:38.585279 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.0.41.3.22 10.0.0.240.56791: Flags [S.], cksum 0x3e21 (incorrect - 0xe5c5), seq 1530502549, ack 3098447117, win 14480, options [mss 1460,sackOK,TS val 340493 ecr 28089204,nop,wscale 3], length 0 Anyone come across this before? -- joe. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
