Public bug reported: OpenStack Release: Newton Operating System: Ubuntu 16.04 LTS 4.4.0-45-generic OpenStack Distro: OpenStack-Ansible 14.0.2
While working to test/implement macvtap functionality, I found it was not possible to boot an instance when using the macvtap mech driver and macvtap agent. My goal is/was to boot an instance with a 'normal' (non- PCI) port, and have a macvtap interface created on the compute node. What I found was that that booting the instance resulted in an ERROR state, with errors reported in neutron-server.log[1] and nova- compute.log[2]: [1] Neutron Server: http://paste.openstack.org/show/596128/ [2] Nova Compute: http://paste.openstack.org/show/596129/ The Nova boot syntax can be seen here: nova boot --image 'Cirros Test Image' --flavor 'm1.test' --nic port- id=5f08dcec-6689-4c1c-9f95-ffb69548c606 --availability-zone=ZONE-B TEST-1 An availability zone consisting of a single node was specified. The interesting thing was observed (multiple times) in neutron- server.log during the boot process: 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers [req-f531783a-8f29-423e-9764-026f4cd4693f 3e202337454b4813b0ce6ecff74d43d6 b009912cbdcd45a0b4ef34fb3d22e7e1 - - -] Mechanism driver macvtap failed in bind_port 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers Traceback (most recent call last): 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers File "/openstack/venvs/neutron-14.0.2/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 787, in _bind_port_level 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers driver.obj.bind_port(context) 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers File "/openstack/venvs/neutron-14.0.2/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/mech_agent.py", line 109, in bind_port 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers agent): 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers File "/openstack/venvs/neutron-14.0.2/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/macvtap/mech_driver/mech_macvtap.py", line 106, in try_to_bind_segment_for_agent 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers if self._is_live_migration(context): 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers File "/openstack/venvs/neutron-14.0.2/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/macvtap/mech_driver/mech_macvtap.py", line 68, in _is_live_migration 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers port_profile = context.original.get(portbindings.PROFILE) 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers AttributeError: 'NoneType' object has no attribute 'get' 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers The _is_live_migration function was implemented in [3]: [3] https://review.openstack.org/#/c/361301/5/neutron/plugins/ml2/drivers/macvtap/mech_driver/mech_macvtap.py context.original is set to None, which is resulting in the Traceback observed here. I have logged the object data at the time it hits the function in [4]: [4] Object: http://paste.openstack.org/show/596130/ To work around this issue for the sake of booting the instance, I set 'port_profile' to None in mech_macvtap.py and restarted Neutron server. I was then able to boot the instance, and the agent created the VLAN and macvtap interfaces accordingly: +--------------------------------------+------------------------------------+--------+------------+-------------+------------------------------------------------------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------------------------------------+--------+------------+-------------+------------------------------------------------------------------------------+ | 0022df55-6500-44c0-8ff5-4433c37157c1 | TEST-1 | ERROR | - | NOSTATE | | | 69eda7de-00ad-4927-955f-9e5e1d9d6909 | TEST-1 | ACTIVE | - | Running | TEST=192.168.7.8 | 113: em50.62@em50: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 5c:b9:01:88:fd:a5 brd ff:ff:ff:ff:ff:ff inet6 fe80::5eb9:1ff:fe88:fda5/64 scope link valid_lft forever preferred_lft forever 115: [email protected]: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500 link/ether fa:16:3e:67:da:b8 brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:fe67:dab8/64 scope link valid_lft forever preferred_lft forever root@817769-i812846:~# tcpdump -i macvtap0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on macvtap0, link-type EN10MB (Ethernet), capture size 262144 bytes 15:30:52.524485 IP6 :: > ff02::1:ff67:dab8: ICMP6, neighbor solicitation, who has fe80::f816:3eff:fe67:dab8, length 24 15:30:52.994199 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:67:da:b8 (oui Unknown), length 290 15:30:52.998185 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 15:30:53.390218 IP6 :: > ff02::1:ff67:dab8: ICMP6, neighbor solicitation, who has fe80::f816:3eff:fe67:dab8, length 24 15:30:53.524385 IP6 fe80::f816:3eff:fe67:dab8 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 15:30:53.524420 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16 15:30:54.008276 IP6 fe80::f816:3eff:fe67:dab8 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 15:30:54.390184 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16 15:30:57.528300 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16 15:30:58.394100 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16 15:31:01.536286 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16 15:31:02.042017 IP6 fe80::f816:3eff:fe67:dab8 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 15:31:02.402014 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16 Port details can be seen here: +-----------------------+-------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------------+-------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:host_id | 817769-i812846.NewtonTest.com | | binding:profile | {} | | binding:vif_details | {"port_filter": false, "macvtap_mode": "bridge", "vlan": 62, "physical_interface": "em50", "macvtap_source": "em50.62"} | | binding:vif_type | macvtap | | binding:vnic_type | normal | | created_at | 2017-01-23T20:18:35Z | | description | | | device_id | 69eda7de-00ad-4927-955f-9e5e1d9d6909 | | device_owner | compute:ZONE-B | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "898171fa-903e-4549-bb82-fbb5f16c9bb1", "ip_address": "192.168.7.8"} | | id | 5f08dcec-6689-4c1c-9f95-ffb69548c606 | | mac_address | fa:16:3e:67:da:b8 | | name | TEST | | network_id | 508e4e86-0137-4774-8df5-bd6919080bea | | port_security_enabled | True | | project_id | a2bc030466504dea81191913f1071a10 | | revision_number | 23 | | security_groups | | | status | ACTIVE | | tenant_id | a2bc030466504dea81191913f1071a10 | | updated_at | 2017-01-23T20:29:07Z | +-----------------------+-------------------------------------------------------------------------------------------------------------------------+ Please let me know what additional information you need, including further troubleshooting or debug output. James ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1658802 Title: Issue booting instance with normal port and macvtap agent Status in neutron: New Bug description: OpenStack Release: Newton Operating System: Ubuntu 16.04 LTS 4.4.0-45-generic OpenStack Distro: OpenStack-Ansible 14.0.2 While working to test/implement macvtap functionality, I found it was not possible to boot an instance when using the macvtap mech driver and macvtap agent. My goal is/was to boot an instance with a 'normal' (non-PCI) port, and have a macvtap interface created on the compute node. What I found was that that booting the instance resulted in an ERROR state, with errors reported in neutron-server.log[1] and nova- compute.log[2]: [1] Neutron Server: http://paste.openstack.org/show/596128/ [2] Nova Compute: http://paste.openstack.org/show/596129/ The Nova boot syntax can be seen here: nova boot --image 'Cirros Test Image' --flavor 'm1.test' --nic port- id=5f08dcec-6689-4c1c-9f95-ffb69548c606 --availability-zone=ZONE-B TEST-1 An availability zone consisting of a single node was specified. The interesting thing was observed (multiple times) in neutron- server.log during the boot process: 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers [req-f531783a-8f29-423e-9764-026f4cd4693f 3e202337454b4813b0ce6ecff74d43d6 b009912cbdcd45a0b4ef34fb3d22e7e1 - - -] Mechanism driver macvtap failed in bind_port 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers Traceback (most recent call last): 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers File "/openstack/venvs/neutron-14.0.2/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 787, in _bind_port_level 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers driver.obj.bind_port(context) 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers File "/openstack/venvs/neutron-14.0.2/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/mech_agent.py", line 109, in bind_port 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers agent): 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers File "/openstack/venvs/neutron-14.0.2/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/macvtap/mech_driver/mech_macvtap.py", line 106, in try_to_bind_segment_for_agent 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers if self._is_live_migration(context): 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers File "/openstack/venvs/neutron-14.0.2/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/macvtap/mech_driver/mech_macvtap.py", line 68, in _is_live_migration 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers port_profile = context.original.get(portbindings.PROFILE) 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers AttributeError: 'NoneType' object has no attribute 'get' 2017-01-23 15:25:57.464 17265 ERROR neutron.plugins.ml2.managers The _is_live_migration function was implemented in [3]: [3] https://review.openstack.org/#/c/361301/5/neutron/plugins/ml2/drivers/macvtap/mech_driver/mech_macvtap.py context.original is set to None, which is resulting in the Traceback observed here. I have logged the object data at the time it hits the function in [4]: [4] Object: http://paste.openstack.org/show/596130/ To work around this issue for the sake of booting the instance, I set 'port_profile' to None in mech_macvtap.py and restarted Neutron server. I was then able to boot the instance, and the agent created the VLAN and macvtap interfaces accordingly: +--------------------------------------+------------------------------------+--------+------------+-------------+------------------------------------------------------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------------------------------------+--------+------------+-------------+------------------------------------------------------------------------------+ | 0022df55-6500-44c0-8ff5-4433c37157c1 | TEST-1 | ERROR | - | NOSTATE | | | 69eda7de-00ad-4927-955f-9e5e1d9d6909 | TEST-1 | ACTIVE | - | Running | TEST=192.168.7.8 | 113: em50.62@em50: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 5c:b9:01:88:fd:a5 brd ff:ff:ff:ff:ff:ff inet6 fe80::5eb9:1ff:fe88:fda5/64 scope link valid_lft forever preferred_lft forever 115: [email protected]: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500 link/ether fa:16:3e:67:da:b8 brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:fe67:dab8/64 scope link valid_lft forever preferred_lft forever root@817769-i812846:~# tcpdump -i macvtap0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on macvtap0, link-type EN10MB (Ethernet), capture size 262144 bytes 15:30:52.524485 IP6 :: > ff02::1:ff67:dab8: ICMP6, neighbor solicitation, who has fe80::f816:3eff:fe67:dab8, length 24 15:30:52.994199 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:67:da:b8 (oui Unknown), length 290 15:30:52.998185 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 15:30:53.390218 IP6 :: > ff02::1:ff67:dab8: ICMP6, neighbor solicitation, who has fe80::f816:3eff:fe67:dab8, length 24 15:30:53.524385 IP6 fe80::f816:3eff:fe67:dab8 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 15:30:53.524420 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16 15:30:54.008276 IP6 fe80::f816:3eff:fe67:dab8 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 15:30:54.390184 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16 15:30:57.528300 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16 15:30:58.394100 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16 15:31:01.536286 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16 15:31:02.042017 IP6 fe80::f816:3eff:fe67:dab8 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 15:31:02.402014 IP6 fe80::f816:3eff:fe67:dab8 > ip6-allrouters: ICMP6, router solicitation, length 16 Port details can be seen here: +-----------------------+-------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-----------------------+-------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:host_id | 817769-i812846.NewtonTest.com | | binding:profile | {} | | binding:vif_details | {"port_filter": false, "macvtap_mode": "bridge", "vlan": 62, "physical_interface": "em50", "macvtap_source": "em50.62"} | | binding:vif_type | macvtap | | binding:vnic_type | normal | | created_at | 2017-01-23T20:18:35Z | | description | | | device_id | 69eda7de-00ad-4927-955f-9e5e1d9d6909 | | device_owner | compute:ZONE-B | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "898171fa-903e-4549-bb82-fbb5f16c9bb1", "ip_address": "192.168.7.8"} | | id | 5f08dcec-6689-4c1c-9f95-ffb69548c606 | | mac_address | fa:16:3e:67:da:b8 | | name | TEST | | network_id | 508e4e86-0137-4774-8df5-bd6919080bea | | port_security_enabled | True | | project_id | a2bc030466504dea81191913f1071a10 | | revision_number | 23 | | security_groups | | | status | ACTIVE | | tenant_id | a2bc030466504dea81191913f1071a10 | | updated_at | 2017-01-23T20:29:07Z | +-----------------------+-------------------------------------------------------------------------------------------------------------------------+ Please let me know what additional information you need, including further troubleshooting or debug output. James To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1658802/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
