Sailaja Mada created CLOUDSTACK-5028:
----------------------------------------
Summary: Instance is failing to start after releasing the primary
storage from maintenance mode
Key: CLOUDSTACK-5028
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5028
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Storage Controller, VMware
Affects Versions: 4.2.1
Reporter: Sailaja Mada
Priority: Critical
Steps:
1. Configure two Adv zones using VMWARE hypervisor
2. Configuration with Zone wide primary storages
3. Deploy VM using a user account
4. Move the Primary storage into maintenance
5. With this VM got into stopped state
6. Release the Storage into maintenance
7. Now start the VM.
Observation:
Instance is failing to start after releasing the primary storage from
maintenance mode
2013-11-03 19:14:30,652 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-330:null) Seq 9-894186938: Executing request
2013-11-03 19:14:30,665 INFO [vmware.resource.VmwareResource]
(DirectAgent-330:10.102.192.18) Executing resource StartCommand:
{"vm":{"id":24,"name":"i-5-24-VM","bootloader":"HVM","type":"User","cpus":1,"minSpeed":200,"maxSpeed":200,"minRam":536870912,"maxRam":536870912,"hostName":"sailajaVM1","arch":"x86_64","os":"CentOS
5.3
(64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"e5e3f536d1a3d5cc","params":{"nicAdapter":"E1000","vmware.reserve.cpu":"false","nestedVirtualizationFlag":"false","Message.ReservedCapacityFreed.Flag":"false","rootDiskController":"ide","vmware.reserve.mem":"false"},"uuid":"1ec83c40-e0dc-48ad-90d8-f4000c0dfe91","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"1ffa6e41-00cc-47d5-958c-f8305ac06af4","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"b33e996a-444e-3685-9070-0865067454c4","id":6,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/sailajaps2","port":2049}},"name":"DATA-24","size":5368709120,"path":"a933eb3fa28a473ab5e28b99f5f2607e","volumeId":37,"vmName":"i-5-24-VM","accountId":5,"chainInfo":"{\"diskDeviceBusName\":\"scsi0:0\",\"diskChain\":[\"[7f18caf5397a340a934ed37c558aee2b]
i-5-24-VM/8d463fcfe15c43dea0cef4474864552e-000004.vmdk\",\"[7f18caf5397a340a934ed37c558aee2b]
i-5-24-VM/8d463fcfe15c43dea0cef4474864552e-000003.vmdk\",\"[7f18caf5397a340a934ed37c5","format":"OVA","id":37,"hypervisorType":"VMware"}},"diskSeq":1,"type":"DATADISK"},{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"91d3fff2-cc90-4122-b076-06588fa33401","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"7f18caf5-397a-340a-934e-d37c558aee2b","id":5,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/sailajaps1","port":2049}},"name":"ROOT-24","size":2147483648,"path":"ROOT-24","volumeId":38,"vmName":"i-5-24-VM","accountId":5,"format":"OVA","id":38,"hypervisorType":"VMware"}},"diskSeq":0,"type":"ROOT"},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"uuid":"257add99-2031-4ad8-b633-979f3812b1cd","id":200,"format":"ISO","accountId":1,"hvm":true,"displayText":"VMware
Tools Installer ISO","name":"vmware-tools.iso","guestOsType":"CentOS 4.5
(32-bit)","hypervisorType":"VMware"}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"f3987dbb-7e7e-4a50-8dc1-761b6bdd2f41","ip":"10.1.1.79","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:3a:5d:00:01","dns1":"10.103.128.15","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://777","isolationUri":"vlan://777","isSecurityGroupEnabled":false}]},"hostIp":"10.102.192.18","executeInSequence":true,"wait":0}
2013-11-03 19:14:30,747 DEBUG [vmware.mo.HostMO]
(DirectAgent-330:10.102.192.18) find VM i-5-24-VM on host
2013-11-03 19:14:30,747 INFO [vmware.mo.HostMO]
(DirectAgent-330:10.102.192.18) VM i-5-24-VM not found in host cache
2013-11-03 19:14:30,747 DEBUG [vmware.mo.HostMO]
(DirectAgent-330:10.102.192.18) load VM cache on host
2013-11-03 19:14:30,769 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null)
===START=== 10.104.255.45 -- GET
command=queryAsyncJobResult&jobId=fc31a268-ae48-49c7-8908-35c52ee1b922&response=json&sessionkey=ktQfh1aD0%2Ffcgyias7vCxMP6WIs%3D&_=1383486601934
2013-11-03 19:14:30,774 INFO [vmware.resource.VmwareResource]
(DirectAgent-330:10.102.192.18) VM i-5-24-VM already exists, tear down devices
for reconfiguration
2013-11-03 19:14:30,806 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null)
===END=== 10.104.255.45 -- GET
command=queryAsyncJobResult&jobId=fc31a268-ae48-49c7-8908-35c52ee1b922&response=json&sessionkey=ktQfh1aD0%2Ffcgyias7vCxMP6WIs%3D&_=1383486601934
2013-11-03 19:14:31,194 DEBUG [vmware.resource.VmwareResource]
(DirectAgent-330:10.102.192.18) Prepare ISO volume at existing device
{"key":3000,"deviceInfo":{"label":"CD/DVD drive 1","summary":"Remote device
/usr/lib/vmware/isoimages/linux.iso"},"backing":{"exclusive":false,"deviceName":""},"connectable":{"startConnected":true,"allowGuestControl":false,"connected":true},"controllerKey":200,"unitNumber":0}
2013-11-03 19:14:31,194 INFO [vmware.resource.VmwareResource]
(DirectAgent-330:10.102.192.18) Found existing disk info from volume path:
ROOT-24
2013-11-03 19:14:31,195 INFO [vmware.resource.VmwareResource]
(DirectAgent-330:10.102.192.18) Chose disk controller based on existing
information: ide0:1
2013-11-03 19:14:31,195 WARN [vmware.resource.VmwareResource]
(DirectAgent-330:10.102.192.18) StartCommand failed due to Exception:
com.google.gson.JsonSyntaxException
Message: com.google.gson.stream.MalformedJsonException: Unterminated string near
com.google.gson.JsonSyntaxException:
com.google.gson.stream.MalformedJsonException: Unterminated string near
at com.google.gson.Streams.parse(Streams.java:51)
at com.google.gson.Gson.fromJson(Gson.java:497)
at com.google.gson.Gson.fromJson(Gson.java:467)
at com.google.gson.Gson.fromJson(Gson.java:417)
at com.google.gson.Gson.fromJson(Gson.java:389)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.getMatchingExistingDisk(VmwareResource.java:3280)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:2855)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:513)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: com.google.gson.stream.MalformedJsonException: Unterminated string
near
at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1111)
at com.google.gson.stream.JsonReader.nextString(JsonReader.java:967)
at com.google.gson.stream.JsonReader.nextValue(JsonReader.java:802)
at com.google.gson.stream.JsonReader.nextInArray(JsonReader.java:706)
at com.google.gson.stream.JsonReader.quickPeek(JsonReader.java:373)
at com.google.gson.stream.JsonReader.hasNext(JsonReader.java:332)
at com.google.gson.Streams.parseRecursive(Streams.java:74)
at com.google.gson.Streams.parseRecursive(Streams.java:83)
at com.google.gson.Streams.parse(Streams.java:40)
... 16 more
2013-11-03 19:14:31,199 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-330:null) Seq 9-894186938: Response Received:
2013-11-03 19:14:31,201 DEBUG [agent.transport.Request] (DirectAgent-330:null)
Seq 9-894186938: Processing: { Ans: , MgmtId: 94838926819810, via: 9, Ver: v1,
Flags: 110,
[{"com.cloud.agent.api.StartAnswer":{"vm":{"id":24,"name":"i-5-24-VM","bootloader":"HVM","type":"User","cpus":1,"minSpeed":200,"maxSpeed":200,"minRam":536870912,"maxRam":536870912,"hostName":"sailajaVM1","arch":"x86_64","os":"CentOS
5.3
(64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"e5e3f536d1a3d5cc","params":{"nicAdapter":"E1000","vmware.reserve.cpu":"false","nestedVirtualizationFlag":"false","Message.ReservedCapacityFreed.Flag":"false","rootDiskController":"ide","vmware.reserve.mem":"false"},"uuid":"1ec83c40-e0dc-48ad-90d8-f4000c0dfe91","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"1ffa6e41-00cc-47d5-958c-f8305ac06af4","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"b33e996a-444e-3685-9070-0865067454c4","id":6,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/sailajaps2","port":2049}},"name":"DATA-24","size":5368709120,"path":"a933eb3fa28a473ab5e28b99f5f2607e","volumeId":37,"vmName":"i-5-24-VM","accountId":5,"chainInfo":"{\"diskDeviceBusName\":\"scsi0:0\",\"diskChain\":[\"[7f18caf5397a340a934ed37c558aee2b]
i-5-24-VM/8d463fcfe15c43dea0cef4474864552e-000004.vmdk\",\"[7f18caf5397a340a934ed37c558aee2b]
i-5-24-VM/8d463fcfe15c43dea0cef4474864552e-000003.vmdk\",\"[7f18caf5397a340a934ed37c5","format":"OVA","id":37,"hypervisorType":"VMware"}},"diskSeq":1,"type":"DATADISK"},{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"91d3fff2-cc90-4122-b076-06588fa33401","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"7f18caf5-397a-340a-934e-d37c558aee2b","id":5,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/sailajaps1","port":2049}},"name":"ROOT-24","size":2147483648,"path":"ROOT-24","volumeId":38,"vmName":"i-5-24-VM","accountId":5,"format":"OVA","id":38,"hypervisorType":"VMware"}},"diskSeq":0,"type":"ROOT"},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"uuid":"257add99-2031-4ad8-b633-979f3812b1cd","id":200,"format":"ISO","accountId":1,"hvm":true,"displayText":"VMware
Tools Installer ISO","name":"vmware-tools.iso","guestOsType":"CentOS 4.5
(32-bit)","hypervisorType":"VMware"}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"f3987dbb-7e7e-4a50-8dc1-761b6bdd2f41","ip":"10.1.1.79","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:3a:5d:00:01","dns1":"10.103.128.15","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://777","isolationUri":"vlan://777","isSecurityGroupEnabled":false}]},"result":false,"details":"StartCommand
failed due to Exception: com.google.gson.JsonSyntaxException\nMessage:
com.google.gson.stream.MalformedJsonException: Unterminated string near
\n","wait":0}}] }
2013-11-03 19:14:31,201 DEBUG [agent.transport.Request]
(Job-Executor-27:job-165 = [ fc31a268-ae48-49c7-8908-35c52ee1b922 ]) Seq
9-894186938: Received: { Ans: , MgmtId: 94838926819810, via: 9, Ver: v1,
Flags: 110, { StartAnswer } }
2013-11-03 19:14:31,204 DEBUG [agent.manager.AgentAttache]
(DirectAgent-330:null) Seq 9-894186938: No more commands found
2013-11-03 19:14:31,208 INFO [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-27:job-165 = [ fc31a268-ae48-49c7-8908-35c52ee1b922 ]) Unable to
start VM on Host[-9-Routing] due to StartCommand failed due to Exception:
com.google.gson.JsonSyntaxException
Message: com.google.gson.stream.MalformedJsonException: Unterminated string near
2013-11-03 19:14:31,219 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-27:job-165 = [ fc31a268-ae48-49c7-8908-35c52ee1b922 ]) Cleaning
up resources for the vm VM[User|sailajaVM1] in Starting state
2013-11-03 19:14:31,221 DEBUG [agent.transport.Request]
(Job-Executor-27:job-165 = [ fc31a268-ae48-49c7-8908-35c52ee1b922 ]) Seq
9-894186939: Sending { Cmd , MgmtId: 94838926819810, via: 9, Ver: v1, Flags:
100111,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":true,"vmName":"i-5-24-VM","wait":0}}]
}
2013-11-03 19:14:31,222 DEBUG [agent.transport.Request]
(Job-Executor-27:job-165 = [ fc31a268-ae48-49c7-8908-35c52ee1b922 ]) Seq
9-894186939: Executing: { Cmd , MgmtId: 94838926819810, via: 9, Ver: v1,
Flags: 100111,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":true,"vmName":"i-5-24-VM","wait":0}}]
}
2013-11-03 19:14:31,244 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-184:null) Seq 9-894186939: Executing request
2013-11-03 19:14:31,245 INFO [vmware.resource.VmwareResource]
(DirectAgent-184:10.102.192.18) Executing resource StopCommand:
{"isProxy":false,"executeInSequence":true,"vmName":"i-5-24-VM","wait":0}
2013-11-03 19:14:31,264 DEBUG [vmware.mo.HostMO]
(DirectAgent-184:10.102.192.18) find VM i-5-24-VM on host
2013-11-03 19:14:31,264 INFO [vmware.mo.HostMO]
(DirectAgent-184:10.102.192.18) VM i-5-24-VM not found in host cache
2013-11-03 19:14:31,264 DEBUG [vmware.mo.HostMO]
(DirectAgent-184:10.102.192.18) load VM cache on host
2013-11-03 19:14:31,322 INFO [vmware.resource.VmwareResource]
(DirectAgent-184:10.102.192.18) VM i-5-24-VM is already in stopped state
--
This message was sent by Atlassian JIRA
(v6.1#6144)
