​Hi all.
I'm trying to setup test lab on Openstack Mitaka RDO working simultaneously
with VLAN and VXLAN network types without SDN. I'm trying to setup ESXi as a
hypervisor (not only ESXi, but it doesn't matter), and I've got troubles.
My setup:
1 controller node (db, keystone, horizon, nova-{api,conductor,schuduler},
glance, memcached, rabbit)
1 network node (neutron-server) (hostname: bc01 in logs/examples)
1 vCenter
2 ESXi hosts
2 neutron-ovsvapp-agents (as described on reference scheme:
https://wiki.openstack.org/wiki/Neutron/Networking-vSphere - one VM per ESXi
host; hostnames ovsvapp1, ovsvapp2 in logs/examples)
1 nova-compute VM for managing vSphere cluster (hostname: ovsvapp-nova in
logs/examples).
When I try to spawn VM, I got error "No valid host was found. There are not
enough hosts available.".
If I look in nova-compute logs, I see traces:
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager
[req-1c94993d-69a0-48d7-ac5f-262f99961b16 18db55ec3559448ba25299cb6bd90504
bb18be7ce13c44a78419769b4feed0fc - - -] Instance failed network setup after 1
attempt(s)
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager Traceback (most recent
call last):
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1570, in
_allocate_network_async
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager
bind_host_id=bind_host_id)
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager File
"/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 729, in
allocate_for_instance
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager
self._delete_ports(neutron, instance, created_port_ids)
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager
self.force_reraise()
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager
six.reraise(self.type_, self.value, self.tb)
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager File
"/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 718, in
allocate_for_instance
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager
security_group_ids, available_macs, dhcp_opts)
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager File
"/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 305, in
_create_port
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager raise
exception.PortBindingFailed(port_id=port_id)
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager PortBindingFailed:
Binding failed for port ef43e4f0-4db7-405c-b4d5-c621aaa3884b, please check
neutron logs for more information.
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager
[req-1c94993d-69a0-48d7-ac5f-262f99961b16 18db55ec3559448ba25299cb6bd90504
bb18be7ce13c44a78419769b4feed0fc - - -] [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] Instance failed to spawn
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] Traceback (most recent call last):
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2218, in
_build_resources
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] yield resources
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2064, in
_build_and_run_instance
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] block_device_info=block_device_info)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File
"/usr/lib/python2.7/site-packages/nova/virt/vmwareapi/driver.py", line 381, in
spawn
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] admin_password, network_info,
block_device_info)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File
"/usr/lib/python2.7/site-packages/nova/virt/vmwareapi/vmops.py", line 724, in
spawn
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] metadata)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File
"/usr/lib/python2.7/site-packages/nova/virt/vmwareapi/vmops.py", line 304, in
build_virtual_machine
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] network_info)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File
"/usr/lib/python2.7/site-packages/nova/virt/vmwareapi/vif.py", line 179, in
get_vif_info
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] for vif in network_info:
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File
"/usr/lib/python2.7/site-packages/nova/network/model.py", line 513, in __iter__
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] return self._sync_wrapper(fn, *args,
**kwargs)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File
"/usr/lib/python2.7/site-packages/nova/network/model.py", line 504, in
_sync_wrapper
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] self.wait()
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File
"/usr/lib/python2.7/site-packages/nova/network/model.py", line 536, in wait
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] self[:] = self._gt.wait()
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File
"/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] return self._exit_event.wait()
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File
"/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] return hubs.get_hub().switch()
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File
"/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] return self.greenlet.switch()
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File
"/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] result = function(*args, **kwargs)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File
"/usr/lib/python2.7/site-packages/nova/utils.py", line 1145, in context_wrapper
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] return func(*args, **kwargs)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1587, in
_allocate_network_async
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] six.reraise(*exc_info)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1570, in
_allocate_network_async
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] bind_host_id=bind_host_id)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File
"/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 729, in
allocate_for_instance
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] self._delete_ports(neutron, instance,
created_port_ids)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] self.force_reraise()
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] six.reraise(self.type_, self.value,
self.tb)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File
"/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 718, in
allocate_for_instance
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] security_group_ids, available_macs,
dhcp_opts)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File
"/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 305, in
_create_port
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] raise
exception.PortBindingFailed(port_id=port_id)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] PortBindingFailed: Binding failed for
port ef43e4f0-4db7-405c-b4d5-c621aaa3884b, please check neutron logs for more
information.
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a]
2016-11-07 11:47:12.718 10441 INFO nova.compute.manager
[req-1c94993d-69a0-48d7-ac5f-262f99961b16 18db55ec3559448ba25299cb6bd90504
bb18be7ce13c44a78419769b4feed0fc - - -] [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] Terminating instance
2016-11-07 11:47:12.719 10441 DEBUG nova.compute.manager
[req-1c94993d-69a0-48d7-ac5f-262f99961b16 18db55ec3559448ba25299cb6bd90504
bb18be7ce13c44a78419769b4feed0fc - - -] [instance:
5ea7de1b-06a2-4ba1-8f54-d00203496f2a] Start destroying the instance on the
hypervisor. _shutdown_instance
/usr/lib/python2.7/site-packages/nova/compute/manager.py:2330
On neutron-server I've got errors:
2016-11-07 11:47:12.268 30899 WARNING neutron.plugins.ml2.drivers.mech_agent
[req-8f6be9c7-12a3-42fe-9976-57655889f90c 942080d9793b4fe1a73b039706ca9496
fe2c5af094bd49b495b86a9dc89b9903 - - -] Port ef43e4f
0-4db7-405c-b4d5-c621aaa3884b on network 3db44ff7-1949-4cc9-8e04-e9982120513f
not bound, no agent registered on host ovsvapp-nova
2016-11-07 11:47:12.269 30899 ERROR neutron.plugins.ml2.managers
[req-8f6be9c7-12a3-42fe-9976-57655889f90c 942080d9793b4fe1a73b039706ca9496
fe2c5af094bd49b495b86a9dc89b9903 - - -] Failed to bind port ef43
e4f0-4db7-405c-b4d5-c621aaa3884b on host ovsvapp-nova for vnic_type normal
using segments [{'segmentation_id': 10094, 'physical_network': None, 'id':
u'f95a1a2e-f641-4541-90ae-f747ec912ae0', 'network_type
': u'vxlan'}]
If we look at neutron agent list, we'll see, than we have no such agent.
Moreover: we have agents for each ESXi hypervisor (ovsvapp1, ovsvapp2).
[root@bc01 ~]# neutron agent-list -c agent_type -c host -c alive -c
admin_state_up -c binary
+--------------------+----------+-------+----------------+---------------------------+
| agent_type | host | alive | admin_state_up | binary
|
+--------------------+----------+-------+----------------+---------------------------+
| OVSvApp Agent | ovsvapp1 | :-) | True | ovsvapp-agent
|
| Open vSwitch agent | bc01 | :-) | True |
neutron-openvswitch-agent |
| L3 agent | bc01 | :-) | True | neutron-l3-agent
|
| DHCP agent | bc01 | :-) | True | neutron-dhcp-agent
|
| Metadata agent | bc01 | :-) | True |
neutron-metadata-agent |
| Open vSwitch agent | bc10 | :-) | True |
neutron-openvswitch-agent |
| Open vSwitch agent | bc09 | :-) | True |
neutron-openvswitch-agent |
| OVSvApp Agent | ovsvapp2 | :-) | True | ovsvapp-agent
|
| Open vSwitch agent | bc11 | :-) | True |
neutron-openvswitch-agent |
| Open vSwitch agent | bc12 | :-) | True |
neutron-openvswitch-agent |
+--------------------+----------+-------+----------------+---------------------------+
Don't look at bc09,bc11,bc12 - these are other hypervisors, non-vsphere.
So, main question: HOW should it work? Does nova-compute need any specific
setup? Should nova-compute send any hostname with ovsvapp-agent to neutron
server? If yes, so how?
From sequence diagram we see, that nova-compute asks neutron-server to create
ports at first (when nova doesn't know, where the VM will reside).
If I install nova-compute on one of ovsvapp-agent VMs, so spawn is successfull,
but such architecture is not fault-tolerant and doesn't follow specs
recommendations.
Please, give me an advice... Or, whom should I contact with such questions?
Thanks.
---
My configs:
Neutron-server:
[root@bc01 ~]# grep -v ^$ /etc/neutron/neutron.conf | grep -v ^#
[DEFAULT]
verbose = True
debug = True
core_plugin = ml2
auth_strategy = keystone
service_plugins = router
rpc_backend = rabbit
nova_url = http://kvm-controller:8774/v2
nova_admin_username = nova
nova_admin_tenant_name = compute
nova_admin_password = compaq
nova_admin_auth_url = http://keystone-server:5000
router_scheduler_driver =
neutron.scheduler.l3_agent_scheduler.LeastRoutersScheduler
notify_nova_on_port_status_changes = True
notify_nova_on_port_data_changes = True
notification_driver = messagingv2
state_path = /var/lib/neutron
[keystone_authtoken]
username = neutron
password = compaq
project_name = services
auth_uri = http://keystone-server:5000
auth_url = http://keystone-server:35357
auth_plugin = password
user_domain_name = Default
project_domain_name = Default
memcached_servers = db-server:11211
[database]
connection = mysql+pymysql://neutron:compaq@db-server/neutron
[nova]
region_name = RegionOne
username = nova
password = compaq
project_name = services
insecure = true
auth_uri = http://keystone-server:5000
auth_url = http://keystone-server:35357
auth_plugin = password
user_domain_name = Default
project_domain_name = Default
endpoint_type = internal
[oslo_concurrency]
lock_path = $state_path/lock
[oslo_messaging_rabbit]
rabbit_host = db-server
rabbit_port = 5672
rabbit_userid = neutron
rabbit_password = compaq
rabbit_virtual_host = /
rabbit_use_ssl = false
[SECURITYGROUP]
firewall_driver = openvswitch
OVSVAPP agent 1:
[root@ovsvapp1 ~]# grep -v ^$ /etc/neutron/neutron.conf | grep -v ^#
[DEFAULT]
[agent]
[cors]
[cors.subdomain]
[database]
[keystone_authtoken]
auth_uri = http://keystone-server:5000
auth_url = http://keystone-server:35357
project_name = services
username = neutron
user_domain_name = Default
project_domain_name = Default
password = compaq
auth_plugin = password
memcached_servers = db-server:11211
[matchmaker_redis]
[nova]
[oslo_concurrency]
[oslo_messaging_amqp]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
rabbit_host = db-server
rabbit_userid = neutron
rabbit_password = compaq
[oslo_policy]
[quotas]
[ssl]
[root@ovsvapp1 ~]# grep -v ^$ /etc/neutron/plugins/ml2/ovsvapp_agent.ini | grep
-v ^#
[vmware]
vcenter_ip = 172.17.0.117
vcenter_username = Administrator@vsphere.local
vcenter_password = P@ssw0rd
wsdl_location=https://172.17.0.117:443/sdk/vimService.wsdl
cluster_dvs_mapping = VOL615/host/VOL615_IBM:dvSwitchInt
esx_hostname = 172.17.0.113
[ovsvapp]
tenant_network_types = vlan,vxlan
local_ip = 172.17.4.11
bridge_mappings = physnet1:br-vlan
[securitygroup]
security_bridge_mapping = br-sec:vsphere
ovsvapp_firewall_driver =
networking_vsphere.drivers.ovs_firewall.OVSFirewallDriver​
[root@ovsvapp-nova ~]# grep -v ^$ /etc/nova/nova.conf | grep -v ^#
[DEFAULT]
my_ip = 172.17.0.10
auth_strategy=keystone
compute_driver=vmwareapi.VMwareVCDriver
firewall_driver=nova.virt.firewall.NoopFirewallDriver
use_neutron=true
debug=True
[api_database]
[barbican]
[cache]
[cells]
[cinder]
[conductor]
[cors]
[cors.subdomain]
[database]
[ephemeral_storage_encryption]
[glance]
api_servers=http://glance-server:9292
[guestfs]
[hyperv]
[image_file_url]
[ironic]
[keymgr]
[keystone_authtoken]
auth_uri = http://keystone-server:5000/v3
auth_url = http://keystone-server:35357/v3
memcached_servers = db-server:11211
auth_type = password
project_domain_name = Default
user_domain_name = Default
project_name = services
username = nova
password = compaq
[libvirt]
[matchmaker_redis]
[metrics]
[neutron]
url = http://neutron-controller:9696/
auth_type=password
insecure = False
username = neutron
password = compaq
service_metadata_proxy=True
metadata_proxy_shared_secret = supersecret
auth_uri = http://keystone-server:5000/v3
auth_version = v3
auth_url = http://keystone-server:35357/v3
project_name = services
user_domain_name = Default
project_domain_name = Default
[osapi_v21]
[oslo_concurrency]
lock_path=/var/lib/nova/tmp
[oslo_messaging_amqp]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
rabbit_host=db-server
rabbit_userid=nova
rabbit_password=compaq
[oslo_middleware]
[oslo_policy]
[rdp]
[serial_console]
[spice]
[ssl]
[trusted_computing]
[upgrade_levels]
[vmware]
host_ip=172.17.0.117
host_username=Administrator@vsphere.local
host_password=P@ssw0rd
insecure=true
cluster_name=VOL615_IBM
datastore_regex=storwize*
[vnc]
enabled=true
vncserver_listen=0.0.0.0
vncserver_proxyclient_address=$my_ip
novncproxy_base_url=http://172.17.0.107:6080/vnc_auto.html
[workarounds]
[xenserver]
second ovsvapp-agent has similar config, just with another IPs.
________________________________
Regards,
Vladislav Odintsov
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
