[
https://issues.apache.org/jira/browse/CLOUDSTACK-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sailaja Mada updated CLOUDSTACK-5028:
-------------------------------------
Fix Version/s: 4.2.1
> Instance is failing to start after releasing the primary storage from
> maintenance mode
> ---------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-5028
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5028
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Storage Controller, VMware
> Affects Versions: 4.2.1
> Reporter: Sailaja Mada
> Priority: Critical
> Fix For: 4.2.1
>
>
> Steps:
> 1. Configure two Adv zones using VMWARE hypervisor
> 2. Configuration with Zone wide primary storages
> 3. Deploy VM using a user account
> 4. Move the Primary storage into maintenance
> 5. With this VM got into stopped state
> 6. Release the Storage into maintenance
> 7. Now start the VM.
> Observation:
> Instance is failing to start after releasing the primary storage from
> maintenance mode
> 2013-11-03 19:14:30,652 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-330:null) Seq 9-894186938: Executing request
> 2013-11-03 19:14:30,665 INFO [vmware.resource.VmwareResource]
> (DirectAgent-330:10.102.192.18) Executing resource StartCommand:
> {"vm":{"id":24,"name":"i-5-24-VM","bootloader":"HVM","type":"User","cpus":1,"minSpeed":200,"maxSpeed":200,"minRam":536870912,"maxRam":536870912,"hostName":"sailajaVM1","arch":"x86_64","os":"CentOS
> 5.3
> (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"e5e3f536d1a3d5cc","params":{"nicAdapter":"E1000","vmware.reserve.cpu":"false","nestedVirtualizationFlag":"false","Message.ReservedCapacityFreed.Flag":"false","rootDiskController":"ide","vmware.reserve.mem":"false"},"uuid":"1ec83c40-e0dc-48ad-90d8-f4000c0dfe91","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"1ffa6e41-00cc-47d5-958c-f8305ac06af4","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"b33e996a-444e-3685-9070-0865067454c4","id":6,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/sailajaps2","port":2049}},"name":"DATA-24","size":5368709120,"path":"a933eb3fa28a473ab5e28b99f5f2607e","volumeId":37,"vmName":"i-5-24-VM","accountId":5,"chainInfo":"{\"diskDeviceBusName\":\"scsi0:0\",\"diskChain\":[\"[7f18caf5397a340a934ed37c558aee2b]
>
> i-5-24-VM/8d463fcfe15c43dea0cef4474864552e-000004.vmdk\",\"[7f18caf5397a340a934ed37c558aee2b]
>
> i-5-24-VM/8d463fcfe15c43dea0cef4474864552e-000003.vmdk\",\"[7f18caf5397a340a934ed37c5","format":"OVA","id":37,"hypervisorType":"VMware"}},"diskSeq":1,"type":"DATADISK"},{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"91d3fff2-cc90-4122-b076-06588fa33401","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"7f18caf5-397a-340a-934e-d37c558aee2b","id":5,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/sailajaps1","port":2049}},"name":"ROOT-24","size":2147483648,"path":"ROOT-24","volumeId":38,"vmName":"i-5-24-VM","accountId":5,"format":"OVA","id":38,"hypervisorType":"VMware"}},"diskSeq":0,"type":"ROOT"},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"uuid":"257add99-2031-4ad8-b633-979f3812b1cd","id":200,"format":"ISO","accountId":1,"hvm":true,"displayText":"VMware
> Tools Installer ISO","name":"vmware-tools.iso","guestOsType":"CentOS 4.5
> (32-bit)","hypervisorType":"VMware"}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"f3987dbb-7e7e-4a50-8dc1-761b6bdd2f41","ip":"10.1.1.79","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:3a:5d:00:01","dns1":"10.103.128.15","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://777","isolationUri":"vlan://777","isSecurityGroupEnabled":false}]},"hostIp":"10.102.192.18","executeInSequence":true,"wait":0}
> 2013-11-03 19:14:30,747 DEBUG [vmware.mo.HostMO]
> (DirectAgent-330:10.102.192.18) find VM i-5-24-VM on host
> 2013-11-03 19:14:30,747 INFO [vmware.mo.HostMO]
> (DirectAgent-330:10.102.192.18) VM i-5-24-VM not found in host cache
> 2013-11-03 19:14:30,747 DEBUG [vmware.mo.HostMO]
> (DirectAgent-330:10.102.192.18) load VM cache on host
> 2013-11-03 19:14:30,769 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null)
> ===START=== 10.104.255.45 -- GET
> command=queryAsyncJobResult&jobId=fc31a268-ae48-49c7-8908-35c52ee1b922&response=json&sessionkey=ktQfh1aD0%2Ffcgyias7vCxMP6WIs%3D&_=1383486601934
> 2013-11-03 19:14:30,774 INFO [vmware.resource.VmwareResource]
> (DirectAgent-330:10.102.192.18) VM i-5-24-VM already exists, tear down
> devices for reconfiguration
> 2013-11-03 19:14:30,806 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null)
> ===END=== 10.104.255.45 -- GET
> command=queryAsyncJobResult&jobId=fc31a268-ae48-49c7-8908-35c52ee1b922&response=json&sessionkey=ktQfh1aD0%2Ffcgyias7vCxMP6WIs%3D&_=1383486601934
> 2013-11-03 19:14:31,194 DEBUG [vmware.resource.VmwareResource]
> (DirectAgent-330:10.102.192.18) Prepare ISO volume at existing device
> {"key":3000,"deviceInfo":{"label":"CD/DVD drive 1","summary":"Remote device
> /usr/lib/vmware/isoimages/linux.iso"},"backing":{"exclusive":false,"deviceName":""},"connectable":{"startConnected":true,"allowGuestControl":false,"connected":true},"controllerKey":200,"unitNumber":0}
> 2013-11-03 19:14:31,194 INFO [vmware.resource.VmwareResource]
> (DirectAgent-330:10.102.192.18) Found existing disk info from volume path:
> ROOT-24
> 2013-11-03 19:14:31,195 INFO [vmware.resource.VmwareResource]
> (DirectAgent-330:10.102.192.18) Chose disk controller based on existing
> information: ide0:1
> 2013-11-03 19:14:31,195 WARN [vmware.resource.VmwareResource]
> (DirectAgent-330:10.102.192.18) StartCommand failed due to Exception:
> com.google.gson.JsonSyntaxException
> Message: com.google.gson.stream.MalformedJsonException: Unterminated string
> near
> com.google.gson.JsonSyntaxException:
> com.google.gson.stream.MalformedJsonException: Unterminated string near
> at com.google.gson.Streams.parse(Streams.java:51)
> at com.google.gson.Gson.fromJson(Gson.java:497)
> at com.google.gson.Gson.fromJson(Gson.java:467)
> at com.google.gson.Gson.fromJson(Gson.java:417)
> at com.google.gson.Gson.fromJson(Gson.java:389)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.getMatchingExistingDisk(VmwareResource.java:3280)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:2855)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:513)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: com.google.gson.stream.MalformedJsonException: Unterminated string
> near
> at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1111)
> at com.google.gson.stream.JsonReader.nextString(JsonReader.java:967)
> at com.google.gson.stream.JsonReader.nextValue(JsonReader.java:802)
> at com.google.gson.stream.JsonReader.nextInArray(JsonReader.java:706)
> at com.google.gson.stream.JsonReader.quickPeek(JsonReader.java:373)
> at com.google.gson.stream.JsonReader.hasNext(JsonReader.java:332)
> at com.google.gson.Streams.parseRecursive(Streams.java:74)
> at com.google.gson.Streams.parseRecursive(Streams.java:83)
> at com.google.gson.Streams.parse(Streams.java:40)
> ... 16 more
> 2013-11-03 19:14:31,199 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-330:null) Seq 9-894186938: Response Received:
> 2013-11-03 19:14:31,201 DEBUG [agent.transport.Request]
> (DirectAgent-330:null) Seq 9-894186938: Processing: { Ans: , MgmtId:
> 94838926819810, via: 9, Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":24,"name":"i-5-24-VM","bootloader":"HVM","type":"User","cpus":1,"minSpeed":200,"maxSpeed":200,"minRam":536870912,"maxRam":536870912,"hostName":"sailajaVM1","arch":"x86_64","os":"CentOS
> 5.3
> (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"e5e3f536d1a3d5cc","params":{"nicAdapter":"E1000","vmware.reserve.cpu":"false","nestedVirtualizationFlag":"false","Message.ReservedCapacityFreed.Flag":"false","rootDiskController":"ide","vmware.reserve.mem":"false"},"uuid":"1ec83c40-e0dc-48ad-90d8-f4000c0dfe91","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"1ffa6e41-00cc-47d5-958c-f8305ac06af4","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"b33e996a-444e-3685-9070-0865067454c4","id":6,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/sailajaps2","port":2049}},"name":"DATA-24","size":5368709120,"path":"a933eb3fa28a473ab5e28b99f5f2607e","volumeId":37,"vmName":"i-5-24-VM","accountId":5,"chainInfo":"{\"diskDeviceBusName\":\"scsi0:0\",\"diskChain\":[\"[7f18caf5397a340a934ed37c558aee2b]
>
> i-5-24-VM/8d463fcfe15c43dea0cef4474864552e-000004.vmdk\",\"[7f18caf5397a340a934ed37c558aee2b]
>
> i-5-24-VM/8d463fcfe15c43dea0cef4474864552e-000003.vmdk\",\"[7f18caf5397a340a934ed37c5","format":"OVA","id":37,"hypervisorType":"VMware"}},"diskSeq":1,"type":"DATADISK"},{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"91d3fff2-cc90-4122-b076-06588fa33401","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"7f18caf5-397a-340a-934e-d37c558aee2b","id":5,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/sailajaps1","port":2049}},"name":"ROOT-24","size":2147483648,"path":"ROOT-24","volumeId":38,"vmName":"i-5-24-VM","accountId":5,"format":"OVA","id":38,"hypervisorType":"VMware"}},"diskSeq":0,"type":"ROOT"},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"uuid":"257add99-2031-4ad8-b633-979f3812b1cd","id":200,"format":"ISO","accountId":1,"hvm":true,"displayText":"VMware
> Tools Installer ISO","name":"vmware-tools.iso","guestOsType":"CentOS 4.5
> (32-bit)","hypervisorType":"VMware"}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"f3987dbb-7e7e-4a50-8dc1-761b6bdd2f41","ip":"10.1.1.79","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:3a:5d:00:01","dns1":"10.103.128.15","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://777","isolationUri":"vlan://777","isSecurityGroupEnabled":false}]},"result":false,"details":"StartCommand
> failed due to Exception: com.google.gson.JsonSyntaxException\nMessage:
> com.google.gson.stream.MalformedJsonException: Unterminated string near
> \n","wait":0}}] }
> 2013-11-03 19:14:31,201 DEBUG [agent.transport.Request]
> (Job-Executor-27:job-165 = [ fc31a268-ae48-49c7-8908-35c52ee1b922 ]) Seq
> 9-894186938: Received: { Ans: , MgmtId: 94838926819810, via: 9, Ver: v1,
> Flags: 110, { StartAnswer } }
> 2013-11-03 19:14:31,204 DEBUG [agent.manager.AgentAttache]
> (DirectAgent-330:null) Seq 9-894186938: No more commands found
> 2013-11-03 19:14:31,208 INFO [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-27:job-165 = [ fc31a268-ae48-49c7-8908-35c52ee1b922 ]) Unable
> to start VM on Host[-9-Routing] due to StartCommand failed due to Exception:
> com.google.gson.JsonSyntaxException
> Message: com.google.gson.stream.MalformedJsonException: Unterminated string
> near
> 2013-11-03 19:14:31,219 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-27:job-165 = [ fc31a268-ae48-49c7-8908-35c52ee1b922 ]) Cleaning
> up resources for the vm VM[User|sailajaVM1] in Starting state
> 2013-11-03 19:14:31,221 DEBUG [agent.transport.Request]
> (Job-Executor-27:job-165 = [ fc31a268-ae48-49c7-8908-35c52ee1b922 ]) Seq
> 9-894186939: Sending { Cmd , MgmtId: 94838926819810, via: 9, Ver: v1, Flags:
> 100111,
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":true,"vmName":"i-5-24-VM","wait":0}}]
> }
> 2013-11-03 19:14:31,222 DEBUG [agent.transport.Request]
> (Job-Executor-27:job-165 = [ fc31a268-ae48-49c7-8908-35c52ee1b922 ]) Seq
> 9-894186939: Executing: { Cmd , MgmtId: 94838926819810, via: 9, Ver: v1,
> Flags: 100111,
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":true,"vmName":"i-5-24-VM","wait":0}}]
> }
> 2013-11-03 19:14:31,244 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-184:null) Seq 9-894186939: Executing request
> 2013-11-03 19:14:31,245 INFO [vmware.resource.VmwareResource]
> (DirectAgent-184:10.102.192.18) Executing resource StopCommand:
> {"isProxy":false,"executeInSequence":true,"vmName":"i-5-24-VM","wait":0}
> 2013-11-03 19:14:31,264 DEBUG [vmware.mo.HostMO]
> (DirectAgent-184:10.102.192.18) find VM i-5-24-VM on host
> 2013-11-03 19:14:31,264 INFO [vmware.mo.HostMO]
> (DirectAgent-184:10.102.192.18) VM i-5-24-VM not found in host cache
> 2013-11-03 19:14:31,264 DEBUG [vmware.mo.HostMO]
> (DirectAgent-184:10.102.192.18) load VM cache on host
> 2013-11-03 19:14:31,322 INFO [vmware.resource.VmwareResource]
> (DirectAgent-184:10.102.192.18) VM i-5-24-VM is already in stopped state
--
This message was sent by Atlassian JIRA
(v6.1#6144)
