Thank you Kevin.Works like a charm now.
Farhad.
On Friday, April 22, 2016 4:43 AM, Kevin Benton <[email protected]> wrote:
The status being DOWN is an issue. That means either the port binding failed
because the host_id was not set on the port that matches the hostname where the
openvswitch agent is running, or the openvswitch agent failed to identify the
port from the container that was plugged in.So there are two things. First do a
port-show on that port with admin credentials so you can see if binding:host_id
is set to the hostname of the node the container is on. Second, I assume
veth_h0 is the connection to the container? If that's the case, you will need
to add the external ID to OVSDB that indicates the Neutron port it corresponds
to. Here is an example command of how to add a port in a way that the
openvswitch agent can correctly wire it up. Apologies for it being so hacky,
it's just a quick script I had to test Neutron agent wiring without using
VMs.port_body=$(neutron port-show $PORT_UUID)port_id=$(echo "$port_body" | grep
"| id" | awk '{ print $4 }')port_mac=$(echo "$port_body" | grep "| mac_address"
| awk '{ print $4 }')port_name=${port_id:0:11}port_name="tap${port_name}"sudo
ovs-vsctl --timeout=120 -- --if-exists del-port $port_name -- add-port br-int
$port_name -- set Interface $port_name external-ids:iface-id=$port_id
external-ids:iface-status=active external-ids:attached-mac=$port_mac
type=internal
The port_name may not need to be changed to match the partial port UUID format,
but there used to be old code that depended on that name so it might be safest
to first start with making your veth pair name match that format if you can.The
giveaway is that your veth_h0 port is missing a tag on br-int. If it doesn't
have a tag (or if it has a tag of 4095), that means the openvswitch agent
hasn't correctly wired it.
On Apr 21, 2016 20:57, "Farhad Sunavala" <[email protected]> wrote:
Thanks Kevin and Kaustubh. Yes, just 1 node (controller, network and compute).
SRC C DST| | | OpenvSwitch SRC VM = 192.168.0.3DST VM =
192.168.0.4Container C = 192.168.0.5
root@fs-10-145-105-64:~# neutron
port-list+--------------------------------------+------+-------------------+-------------------------------------------------------+|
id | name | mac_address | fixed_ips
|+--------------------------------------+------+-------------------+-------------------------------------------------------+|
371b8923-efb7-499f-b344-a9f38d538a73 | | fa:16:3e:1e:bd:2c |
{"subnet_id": "762f6059-a06f-4b3f-a91d-d807bcf9f349", ||
| | | "ip_address": "192.168.0.3"}
|| 86883e00-9d97-4bed-b05e-c942227fb268 | |
fa:16:3e:2b:36:22 | {"subnet_id": "762f6059-a06f-4b3f-a91d-d807bcf9f349", ||
| | | "ip_address":
"192.168.0.2"} || a1213dae-ab39-41c0-b2cd-99b20714aa8b
| | fa:16:3e:c7:23:be | {"subnet_id":
"762f6059-a06f-4b3f-a91d-d807bcf9f349", ||
| | | "ip_address": "192.168.0.5"}
|| b2339046-da71-43e9-a6f8-cbadf24be654 | | fa:16:3e:06:c5:23 |
{"subnet_id": "762f6059-a06f-4b3f-a91d-d807bcf9f349", ||
| | | "ip_address": "192.168.0.1"}
|| b2b2c53a-c92e-47b6-88bc-db3d3e135b79 | |
fa:16:3e:89:0d:fa | {"subnet_id": "762f6059-a06f-4b3f-a91d-d807bcf9f349", ||
| | | "ip_address":
"192.168.0.4"}
|+--------------------------------------+------+-------------------+-------------------------------------------------------+
Just limiting output to SRC (192.168.0.3) and Container C (192.168.0.5)
root@fs-10-145-105-64:~# neutron port-show
371b8923-efb7-499f-b344-a9f38d538a73+-----------------------+------------------------------------------------------------------------------------+|
Field | Value
|+-----------------------+------------------------------------------------------------------------------------+|
admin_state_up | True
|| allowed_address_pairs |
|| binding:vnic_type
| normal
|| created_at | 2016-04-21T20:39:37
|| description |
||
device_id | b7995a2e-f5c7-4398-a801-8d71c9c05a54
|| device_owner | compute:None
|| dns_name
|
|| extra_dhcp_opts |
|| fixed_ips | {"subnet_id":
"762f6059-a06f-4b3f-a91d-d807bcf9f349", "ip_address": "192.168.0.3"} || id
| 371b8923-efb7-499f-b344-a9f38d538a73
|| mac_address | fa:16:3e:1e:bd:2c
|| name |
|| network_id | b2be2b73-6e17-427e-9ec3-5e6d1a1ef98d
|| port_security_enabled | False
|| security_groups
|
|| status | ACTIVE
|| tenant_id |
ba13e4a3e7724db8889852b398988587
|| updated_at | 2016-04-22T03:38:48
|+-----------------------+------------------------------------------------------------------------------------+
root@fs-10-145-105-64:~# neutron port-show
a1213dae-ab39-41c0-b2cd-99b20714aa8b+-----------------------+------------------------------------------------------------------------------------+|
Field | Value
|+-----------------------+------------------------------------------------------------------------------------+|
admin_state_up | True
|| allowed_address_pairs |
|| binding:vnic_type
| normal
|| created_at | 2016-04-21T20:41:50
|| description |
||
device_id |
|| device_owner |
|| dns_name
|
|| extra_dhcp_opts |
|| fixed_ips | {"subnet_id":
"762f6059-a06f-4b3f-a91d-d807bcf9f349", "ip_address": "192.168.0.5"} || id
| a1213dae-ab39-41c0-b2cd-99b20714aa8b
|| mac_address | fa:16:3e:c7:23:be
|| name |
|| network_id | b2be2b73-6e17-427e-9ec3-5e6d1a1ef98d
|| port_security_enabled | False
|| security_groups
|
|| status | DOWN
|| tenant_id |
ba13e4a3e7724db8889852b398988587
|| updated_at | 2016-04-21T20:41:50
|+-----------------------+------------------------------------------------------------------------------------+
The status=DOWN makes sense since the container is not under nova-control.
root@fs-10-145-105-64:~# docker psCONTAINER ID IMAGE
COMMAND CREATED STATUS PORTS
NAMESc8d99952695d ubuntu "/bin/bash" 7 hours ago
Up 7 hours
backstabbing_spenceroot@fs-10-145-105-64:~# docker inspect -f '{{.State.Pid}}'
c8d99952695d105529
root@fs-10-145-105-64:~# ip
netns105529qrouter-872d95f1-5ef7-4811-8232-8ced2322863bqdhcp-b2be2b73-6e17-427e-9ec3-5e6d1a1ef98d83376root@fs-10-145-105-64:~#
ip netns exec 105529 ifconfigeth0 Link encap:Ethernet HWaddr
fa:16:3e:c7:23:be inet addr:192.168.0.5 Bcast:192.168.0.255
Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fec7:23be/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX
packets:23908 errors:0 dropped:0 overruns:0 frame:0 TX packets:24514
errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000
RX bytes:2303668 (2.3 MB) TX bytes:2329412 (2.3 MB)
As you can see the MAC address of container matches what neutron has given
along withthe IP address of 192.168.0.5
root@fs-10-145-105-64:~# brctl showbridge name bridge id STP
enabled [email protected]
root@fs-10-145-105-64:~# iptables -LChain INPUT (policy ACCEPT)target prot
opt source destinationnova-api-INPUT all -- anywhere
anywhereACCEPT udp -- anywhere anywhere udp
dpt:domainACCEPT tcp -- anywhere anywhere tcp
dpt:domainACCEPT udp -- anywhere anywhere udp
dpt:bootpsACCEPT tcp -- anywhere anywhere tcp
dpt:bootps
Chain FORWARD (policy ACCEPT)target prot opt source
destinationnova-filter-top all -- anywhere
anywherenova-api-FORWARD all -- anywhere anywhereACCEPT all
-- anywhere 192.168.122.0/24 ctstate RELATED,ESTABLISHEDACCEPT
all -- 192.168.122.0/24 anywhereACCEPT all -- anywhere
anywhereREJECT all -- anywhere anywhere
reject-with icmp-port-unreachableREJECT all -- anywhere
anywhere reject-with icmp-port-unreachable
Chain OUTPUT (policy ACCEPT)target prot opt source
destinationnova-filter-top all -- anywhere
anywherenova-api-OUTPUT all -- anywhere anywhereACCEPT udp
-- anywhere anywhere udp dpt:bootpc
Chain nova-api-FORWARD (1 references)target prot opt source
destination
Chain nova-api-INPUT (1 references)target prot opt source
destinationACCEPT tcp -- anywhere fs-10.145.105.64 tcp
dpt:8775
Chain nova-api-OUTPUT (1 references)target prot opt source
destination
Chain nova-api-local (1 references)target prot opt source
destination
Chain nova-filter-top (2 references)target prot opt source
destinationnova-api-local all -- anywhere anywhere
======================================================================
Before sending 3 pings:
root@c8d99952695d:/# ifconfig eth0eth0 Link encap:Ethernet HWaddr
fa:16:3e:c7:23:be inet addr:192.168.0.5 Bcast:192.168.0.255
Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fec7:23be/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX
packets:23908 errors:0 dropped:0 overruns:0 frame:0 TX packets:24514
errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000
RX bytes:2303668 (2.3 MB) TX bytes:2329412 (2.3 MB)
root@fs-10-145-105-64:~# ovs-ofctl dump-flows br-intNXST_FLOW reply (xid=0x4):
cookie=0xb67c96fbae779e37, duration=26464.927s, table=0, n_packets=0,
n_bytes=0, idle_age=26464, priority=20,mpls actions=resubmit(,10)
cookie=0xb67c96fbae779e37, duration=25823.697s, table=0, n_packets=0,
n_bytes=0, idle_age=25823, priority=10,icmp6,in_port=7,icmp_type=136
actions=resubmit(,24) cookie=0xb67c96fbae779e37, duration=25803.737s, table=0,
n_packets=0, n_bytes=0, idle_age=25803,
priority=10,icmp6,in_port=8,icmp_type=136 actions=resubmit(,24)
cookie=0xb67c96fbae779e37, duration=26465.387s, table=0, n_packets=734,
n_bytes=39540, idle_age=668, priority=0 actions=NORMAL
cookie=0xb67c96fbae779e37, duration=26464.921s, table=10, n_packets=0,
n_bytes=0, idle_age=26464, priority=0 actions=drop cookie=0xb67c96fbae779e37,
duration=26465.381s, table=23, n_packets=0, n_bytes=0, idle_age=26465,
priority=0 actions=drop cookie=0xb67c96fbae779e37, duration=26465.375s,
table=24, n_packets=0, n_bytes=0, idle_age=26465, priority=0 actions=drop
root@c8d99952695d:/# ping 192.168.0.3 -c3PING 192.168.0.3 (192.168.0.3) 56(84)
bytes of data.From 192.168.0.5 icmp_seq=1 Destination Host UnreachableFrom
192.168.0.5 icmp_seq=2 Destination Host UnreachableFrom 192.168.0.5 icmp_seq=3
Destination Host Unreachable
--- 192.168.0.3 ping statistics ---3 packets transmitted, 0 received, +3
errors, 100% packet loss, time 2014ms
(TCPDUn veth_h0 - container's tap interface on br-int - Note: SRC MAC address
matches one given by neutron)root@fs-10-145-105-64:~# tcpdump -i veth_h0
-nnXXetcpdump: WARNING: veth_h0: no IPv4 address assignedtcpdump: verbose
output suppressed, use -v or -vv for full protocol decodelistening on veth_h0,
link-type EN10MB (Ethernet), capture size 65535 bytes20:51:58.629048
fa:16:3e:c7:23:be > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42:
Request who-has 192.168.0.3 tell 192.168.0.5, length 28 0x0000: ffff
ffff ffff fa16 3ec7 23be 0806 0001 ........>.#..... 0x0010: 0800 0604
0001 fa16 3ec7 23be c0a8 0005 ........>.#..... 0x0020: 0000 0000 0000
c0a8 0003 ..........20:51:59.625739 fa:16:3e:c7:23:be >
ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has
192.168.0.3 tell 192.168.0.5, length 28 0x0000: ffff ffff ffff fa16
3ec7 23be 0806 0001 ........>.#..... 0x0010: 0800 0604 0001 fa16 3ec7
23be c0a8 0005 ........>.#..... 0x0020: 0000 0000 0000 c0a8 0003
..........20:52:00.625741 fa:16:3e:c7:23:be > ff:ff:ff:ff:ff:ff,
ethertype ARP (0x0806), length 42: Request who-has 192.168.0.3 tell
192.168.0.5, length 28 0x0000: ffff ffff ffff fa16 3ec7 23be 0806 0001
........>.#..... 0x0010: 0800 0604 0001 fa16 3ec7 23be c0a8 0005
........>.#..... 0x0020: 0000 0000 0000 c0a8 0003
..........
root@fs-10-145-105-64:~# ovs-ofctl dump-flows br-intNXST_FLOW reply (xid=0x4):
cookie=0xb67c96fbae779e37, duration=26703.277s, table=0, n_packets=0,
n_bytes=0, idle_age=26703, priority=20,mpls actions=resubmit(,10)
cookie=0xb67c96fbae779e37, duration=26062.047s, table=0, n_packets=0,
n_bytes=0, idle_age=26062, priority=10,icmp6,in_port=7,icmp_type=136
actions=resubmit(,24) cookie=0xb67c96fbae779e37, duration=26042.087s, table=0,
n_packets=0, n_bytes=0, idle_age=26042,
priority=10,icmp6,in_port=8,icmp_type=136 actions=resubmit(,24)
cookie=0xb67c96fbae779e37, duration=26703.737s, table=0, n_packets=737,
n_bytes=39666, idle_age=123, priority=0 actions=NORMAL
cookie=0xb67c96fbae779e37, duration=26703.271s, table=10, n_packets=0,
n_bytes=0, idle_age=26703, priority=0 actions=drop cookie=0xb67c96fbae779e37,
duration=26703.731s, table=23, n_packets=0, n_bytes=0, idle_age=26703,
priority=0 actions=drop
The packet counter has incremented by 3 (737-734) on table=0, actions:NORMAL
But, the packets never make it out of the tap interface to SRC (192.168.0.3).
Totally stumped. It is reproducible easily.Just get the latest devstack from
master (or mitaka stable) latest openvswitch (2.5.90) or OVS 2.4.1.
Thanks,Farhad.
On Thursday, April 21, 2016 3:52 PM, Kevin Benton <[email protected]> wrote:
I don't see port details for the container port (50686370). That would be the
one that needs port_security_enabled=False if your container is not using the
MAC address defined in the Neutron Port for that container.
On Thu, Apr 21, 2016 at 3:30 PM, Kaustubh Kelkar
<[email protected]> wrote:
Hi, Can you post the output of:iptables –Lbrctl showovs-ofctl show br-int I
presume all the instances are on the same physical node? -Kaustubh From:
Farhad Sunavala [mailto:[email protected]]
Sent: Thursday, April 21, 2016 3:15 PM
To: [email protected]
Subject: [Openstack] Openstack security - port-security, security-groups with
added container Hi folks, My setup is as follows. SRC C DST|
| | OpenvSwitch SRC VM = 192.168.0.3DST VM = 192.168.0.4 Container C =
192.168.0.5 Openstack version: latest from masterOpenvSwitch: 2.4.1
Openstack brings up SRC (192.168.0.3) and DST VM (192.168.0.5)I manually
bringup container C using docker and connect it to br-int.I allocate a neutron
port (192.168.0.5) and configure this IP address on the container C. SRC,DST
VMs work fine.Now, my problem is that the container C is not able to pass
traffic. Troubleshooting reveals it is sending packet to br-int, it hits the
NORMAL action flows on br-int but then br-int does not transmit them forward.I
have disabled security-groups and port-security on all ports. It is definitely
something to do with security (anti-spoofing) but I am not able to narrow it
down.Any pointers will be appreciated. Thanks,Farhad. Details of
troubleshooting: Setup: SRC C DST| | | OpenvSwitch SRC VM
= 192.168.0.3DST VM = 192.168.0.4 Container C = 192.168.0.5
root@fs-10-145-105-64:~# neutron
port-list+--------------------------------------+------+-------------------+-------------------------------------------------------+|
id | name | mac_address | fixed_ips
|+--------------------------------------+------+-------------------+-------------------------------------------------------+|
31d7fef4-2689-40d5-886c-b0419295eabb | | fa:16:3e:e7:f8:47 |
{"subnet_id": "19c6dc03-b255-45e9-9bd7-366631b745b6", ||
| | | "ip_address": "192.168.0.4"}
|| 3466abf1-aa9a-420a-bb9b-c4ad55561f19 | |
fa:16:3e:84:62:af | {"subnet_id": "19c6dc03-b255-45e9-9bd7-366631b745b6", ||
| | | "ip_address":
"192.168.0.2"} || 50686370-de40-4665-a5a1-39b5b795122b
| | fa:16:3e:0d:ae:bc | {"subnet_id":
"19c6dc03-b255-45e9-9bd7-366631b745b6", ||
| | | "ip_address": "192.168.0.5"}
|| c6043a1e-ac4d-4138-af14-b91ae8e4d57e | | fa:16:3e:36:95:77 |
{"subnet_id": "19c6dc03-b255-45e9-9bd7-366631b745b6", ||
| | | "ip_address": "192.168.0.1"}
|| e8a167b1-8457-4bd7-a18b-82712e0c46dd | |
fa:16:3e:85:92:ee | {"subnet_id": "19c6dc03-b255-45e9-9bd7-366631b745b6", ||
| | | "ip_address":
"192.168.0.3"}
|+--------------------------------------+------+-------------------+-------------------------------------------------------+root@fs-10-145-105-64:~#
neutron port-show
e8a167b1-8457-4bd7-a18b-82712e0c46dd+-----------------------+------------------------------------------------------------------------------------+|
Field | Value
|+-----------------------+------------------------------------------------------------------------------------+|
admin_state_up | True
|| allowed_address_pairs |
|| binding:host_id
| fs-10-145-105-64
|| binding:profile | {}
|| binding:vif_details |
{"port_filter": true, "ovs_hybrid_plug": false}
|| binding:vif_type | ovs
|| binding:vnic_type | normal
|| created_at
| 2016-04-21T15:27:51
|| description |
|| device_id |
1a65bcfb-4607-4529-a313-6927d86c9302
|| device_owner | compute:nova
|| dns_name |
||
extra_dhcp_opts |
|| fixed_ips | {"subnet_id":
"19c6dc03-b255-45e9-9bd7-366631b745b6", "ip_address": "192.168.0.3"} || id
| e8a167b1-8457-4bd7-a18b-82712e0c46dd
|| mac_address | fa:16:3e:85:92:ee
|| name |
|| network_id | e4c38c74-c9c2-47a7-802d-446488b82e5f
|| port_security_enabled | True
|| security_groups
| fcf62b2a-1dd4-432a-b47b-5d9b79813da7
|| status | ACTIVE
|| tenant_id |
62b9eff93cc74b65a2b52a0d41604e2c
|| updated_at | 2016-04-21T15:27:58
|+-----------------------+------------------------------------------------------------------------------------+
root@fs-10-145-105-64:~# neutron port-show
31d7fef4-2689-40d5-886c-b0419295eabb+-----------------------+------------------------------------------------------------------------------------+|
Field | Value
|+-----------------------+------------------------------------------------------------------------------------+|
admin_state_up | True
|| allowed_address_pairs |
|| binding:host_id
| fs-10-145-105-64
|| binding:profile | {}
|| binding:vif_details |
{"port_filter": true, "ovs_hybrid_plug": false}
|| binding:vif_type | ovs
|| binding:vnic_type | normal
|| created_at
| 2016-04-21T15:28:37
|| description |
|| device_id |
9c8a5e3c-0867-4938-b26e-4834a24d0c5d
|| device_owner | compute:nova
|| dns_name |
||
extra_dhcp_opts |
|| fixed_ips | {"subnet_id":
"19c6dc03-b255-45e9-9bd7-366631b745b6", "ip_address": "192.168.0.4"} || id
| 31d7fef4-2689-40d5-886c-b0419295eabb
|| mac_address | fa:16:3e:e7:f8:47
|| name |
|| network_id | e4c38c74-c9c2-47a7-802d-446488b82e5f
|| port_security_enabled | False
|| security_groups
|
|| status | ACTIVE
|| tenant_id |
62b9eff93cc74b65a2b52a0d41604e2c
|| updated_at | 2016-04-21T18:25:20
|+-----------------------+------------------------------------------------------------------------------------+
root@fs-10-145-105-64:~# neutron port-show
31d7fef4-2689-40d5-886c-b0419295eabb+-----------------------+------------------------------------------------------------------------------------+|
Field | Value
|+-----------------------+------------------------------------------------------------------------------------+|
admin_state_up | True
|| allowed_address_pairs |
|| binding:host_id
| fs-10-145-105-64
|| binding:profile | {}
|| binding:vif_details |
{"port_filter": true, "ovs_hybrid_plug": false}
|| binding:vif_type | ovs
|| binding:vnic_type | normal
|| created_at
| 2016-04-21T15:28:37
|| description |
|| device_id |
9c8a5e3c-0867-4938-b26e-4834a24d0c5d
|| device_owner | compute:nova
|| dns_name |
||
extra_dhcp_opts |
|| fixed_ips | {"subnet_id":
"19c6dc03-b255-45e9-9bd7-366631b745b6", "ip_address": "192.168.0.4"} || id
| 31d7fef4-2689-40d5-886c-b0419295eabb
|| mac_address | fa:16:3e:e7:f8:47
|| name |
|| network_id | e4c38c74-c9c2-47a7-802d-446488b82e5f
|| port_security_enabled | False
|| security_groups
|
|| status | ACTIVE
|| tenant_id |
62b9eff93cc74b65a2b52a0d41604e2c
|| updated_at | 2016-04-21T18:25:20
|+-----------------------+------------------------------------------------------------------------------------+
root@fs-10-145-105-64:~# ovs-vsctl showcd040c9d-427b-418b-943c-7ad5522e1d8f
Bridge br-int fail_mode: secure Port "tap3466abf1-aa"
tag: 2 Interface "tap3466abf1-aa" type: internal
Port "tape8a167b1-84" tag: 2 Interface
"tape8a167b1-84" Port br-int Interface br-int
type: internal Port patch-tun
...
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : [email protected]
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
