Sailaja Mada created CLOUDSTACK-4584:
----------------------------------------
Summary: [VMWARE]System VM's failed to start if nexus is enabled
at cloud level before upgrade but all the VMWARE clusters are with Standard
vSwitch
Key: CLOUDSTACK-4584
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4584
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Network Devices, VMware
Affects Versions: 4.2.1
Reporter: Sailaja Mada
Fix For: 4.2.1
Steps:
1. Configure two zones with VMWARE cluster in 307 Patch B
2. Create two physical networks (1 for mgmt. 2 for public/guest )
3. Deploy VM's on these two zones
4. Enable nexus flag With no clusters on nexus vSwitch
5. Upgrade to 4.2
Observation:
System VM's failed to start if nexus is enabled at cloud level before upgrade
but all the VMWARE clusters are with Standard vSwitch
Root Cause: We are assuming that at cloud level if nexus is enabled then all
clusters switch type should get modified to Nexus only.
With that cluster_details table is updating the Switch type for public/Guest
traffic all the VMWARE clusters in the cloud to nexus type.
mysql> select * from cluster_details;
+----+------------+-----------------------+------------------------------------------+
| id | cluster_id | name | value
|
+----+------------+-----------------------+------------------------------------------+
| 1 | 1 | username | administrator
|
| 2 | 1 | password | XZCFHHifMgSGvFarPBx16cwqI9Y9Oadr
|
| 3 | 1 | url | http://10.102.192.248/307dc1/307c1
|
| 5 | 3 | username | administrator
|
| 6 | 3 | password | 6RJmckhfL31Y9BYv6DuCLZ094BWzE6ed
|
| 7 | 3 | url |
http://10.102.192.248/legacydc1/legacyc1 |
| 9 | 1 | guestvswitchtype | nexusdvs
|
| 10 | 1 | guestvswitchname | vSwitch1
|
| 11 | 1 | publicvswitchtype | nexusdvs
|
| 12 | 1 | publicvswitchname | vSwitch1
|
| 13 | 1 | guestvswitchtype | nexusdvs
|
| 14 | 1 | guestvswitchname | vSwitch1
|
| 15 | 1 | publicvswitchtype | nexusdvs
|
| 16 | 1 | publicvswitchname | vSwitch1
|
| 17 | 3 | guestvswitchtype | nexusdvs
|
| 18 | 3 | guestvswitchname | vSwitch1
|
| 19 | 3 | publicvswitchtype | nexusdvs
|
| 20 | 3 | publicvswitchname | vSwitch1
|
| 21 | 1 | cpuOvercommitRatio | 1
|
| 22 | 1 | memoryOvercommitRatio | 1
|
| 23 | 2 | cpuOvercommitRatio | 1
|
| 24 | 2 | memoryOvercommitRatio | 1
|
| 25 | 3 | cpuOvercommitRatio | 1
|
| 26 | 3 | memoryOvercommitRatio | 1
|
| 33 | 1 | NativeHA | false
|
| 34 | 3 | NativeHA | false
|
+----+------------+-----------------------+------------------------------------------+
26 rows in set (0.00 sec)
2013-08-31 16:10:33,109 INFO [vmware.resource.VmwareResource]
(DirectAgent-42:10.102.192.20) Preparing NIC device on network
cloud.private.untagged.0.1-vSwitch0
2013-08-31 16:10:33,109 DEBUG [vmware.resource.VmwareResource]
(DirectAgent-42:10.102.192.20) Prepare NIC at new device
{"operation":"ADD","device":{"addressType":"Manual","macAddress":"02:00:07:e2:00:02","key":-3,"backing":{"network":{"value":"network-12406","type":"Network"},"deviceName":"cloud.private.untagged.0.1-vSwitch0"},"connectable":{"startConnected":true,"allowGuestControl":true,"connected":true},"unitNumber":0}}
2013-08-31 16:10:33,109 INFO [vmware.resource.VmwareResource]
(DirectAgent-42:10.102.192.20) Prepare NIC device based on NicTO:
{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"b5a549b4-caed-4e1c-a55d-d512fb073ace","ip":"10.102.195.114","netmask":"255.255.252.0","gateway":"10.102.192.1","mac":"06:ca:4e:00:00:05","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false,"name":"vSwitch0"}
2013-08-31 16:10:33,113 INFO [vmware.resource.VmwareResource]
(DirectAgent-42:10.102.192.20) Prepare network on vmwaresvs
P[vSwitch0:untagged] with name prefix: cloud.private
2013-08-31 16:10:33,177 INFO [vmware.mo.HypervisorHostHelper]
(DirectAgent-42:10.102.192.20) Network cloud.private.untagged.0.1-vSwitch0 is
ready on vSwitch vSwitch0
2013-08-31 16:10:33,177 INFO [vmware.resource.VmwareResource]
(DirectAgent-42:10.102.192.20) Preparing NIC device on network
cloud.private.untagged.0.1-vSwitch0
2013-08-31 16:10:33,177 DEBUG [vmware.resource.VmwareResource]
(DirectAgent-42:10.102.192.20) Prepare NIC at new device
{"operation":"ADD","device":{"addressType":"Manual","macAddress":"06:ca:4e:00:00:05","key":-4,"backing":{"network":{"value":"network-12406","type":"Network"},"deviceName":"cloud.private.untagged.0.1-vSwitch0"},"connectable":{"startConnected":true,"allowGuestControl":true,"connected":true},"unitNumber":1}}
2013-08-31 16:10:33,177 INFO [vmware.resource.VmwareResource]
(DirectAgent-42:10.102.192.20) Prepare NIC device based on NicTO:
{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"uuid":"66657016-b62e-4061-840b-3515fa33f583","ip":"10.102.197.230","netmask":"255.255.255.0","gateway":"10.102.197.1","mac":"06:04:06:00:00:07","dns1":"10.103.128.15","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://101","isolationUri":"vlan://101","isSecurityGroupEnabled":false,"name":"vSwitch1"}
2013-08-31 16:10:33,181 INFO [vmware.resource.VmwareResource]
(DirectAgent-42:10.102.192.20) Prepare network on nexusdvs P[vSwitch1:untagged]
with name prefix: cloud.public
2013-08-31 16:10:33,188 ERROR [vmware.mo.HypervisorHostHelper]
(DirectAgent-42:10.102.192.20) Unable to find Ethernet port profile vSwitch1
2013-08-31 16:10:33,188 WARN [vmware.resource.VmwareResource]
(DirectAgent-42:10.102.192.20) StartCommand failed due to Exception:
java.lang.Exception
Message: Unable to find Ethernet port profile vSwitch1
java.lang.Exception: Unable to find Ethernet port profile vSwitch1
at
com.cloud.hypervisor.vmware.mo.HypervisorHostHelper.prepareNetwork(HypervisorHostHelper.java:567)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.prepareNetworkFromNicInfo(VmwareResource.java:3530)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:2896)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:519)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-08-31 16:10:33,190 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-42:null) Seq 6-340918310: Cancelling because one of the answers is
false and it is stop on error.
2013-08-31 16:10:33,190 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-42:null) Seq 6-340918310: Response Received:
2013-08-31 16:10:33,191 DEBUG [agent.transport.Request] (DirectAgent-42:null)
Seq 6-340918310: Processing: { Ans: , MgmtId: 227594284004867, via: 6, Ver:
v1, Flags: 110,
[{"com.cloud.agent.api.StartAnswer":{"vm":{"id":7,"name":"v-7-VM","bootloader":"HVM","type":"ConsoleProxy","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":1073741824,"maxRam":1073741824,"hostName":"v-7-VM","arch":"x86_64","os":"Debian
GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP type=consoleproxy
host=10.102.192.221 port=8250 name=v-7-VM premium=true zone=2 pod=2
guid=Proxy.7 proxy_vm=7 disable_rp_filter=true eth2ip=10.102.197.230
eth2mask=255.255.255.0 gateway=10.102.197.1 eth0ip=0.0.0.0 eth0mask=0.0.0.0
eth1ip=10.102.195.114 eth1mask=255.255.252.0 mgmtcidr=10.102.192.0/22
localgw=10.102.192.1 internaldns1=10.103.128.15
dns1=10.103.128.15","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"2857ec9fcb4d655d","params":{"nicAdapter":"E1000","vmware.reserve.cpu":"false","vmware.reserve.mem":"false"},"uuid":"de2ecf56-bb07-451c-9123-4be19fe97778","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"30d602e6-8b76-43f8-a0e4-d41b9ba503fc","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"55e6d095-7e22-3f4a-b533-874a9fede8df","id":202,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/vmwareps1","port":2049}},"name":"ROOT-7","size":0,"path":"ROOT-7","volumeId":20,"vmName":"v-7-VM","accountId":1,"format":"OVA","id":20,"hypervisorType":"VMware"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"uuid":"66657016-b62e-4061-840b-3515fa33f583","ip":"10.102.197.230","netmask":"255.255.255.0","gateway":"10.102.197.1","mac":"06:04:06:00:00:07","dns1":"10.103.128.15","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://101","isolationUri":"vlan://101","isSecurityGroupEnabled":false,"name":"vSwitch1"},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"uuid":"6c7a9d3a-a50f-4fa4-954f-dc57a7da9243","mac":"02:00:07:e2:00:02","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false,"name":"vSwitch0"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"b5a549b4-caed-4e1c-a55d-d512fb073ace","ip":"10.102.195.114","netmask":"255.255.252.0","gateway":"10.102.192.1","mac":"06:ca:4e:00:00:05","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false,"name":"vSwitch0"}]},"result":false,"details":"StartCommand
failed due to Exception: java.lang.Exception\nMessage: Unable to find Ethernet
port profile vSwitch1\n","wait":0}}] }
WorkAround:
1. Disable Zone
2.Update the Switch type to vmwaresvs - Standard Vswitch)
3 Stop and Start Management server
4. Enable Zone
After this workaround system VM's got started with no issues
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
