[ https://issues.apache.org/jira/browse/CLOUDSTACK-5573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sangeetha Hariharan closed CLOUDSTACK-5573. ------------------------------------------- Tested with latest build from 4.3 Set up: Advanced zone set up with 2 host (rhel 6.3 KVM hosts). Have 12 Vms deployed in both hosts. Tried network down and host power down uses multiple times. We are able to successfully HA SSVM and CPVM . Did not see them being stuck in "Starting" state. > KVM- SSVM/CPVM stuck in "Starting" state Caused by: > java.lang.NullPointerException. > ----------------------------------------------------------------------------------- > > Key: CLOUDSTACK-5573 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5573 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server > Affects Versions: 4.3.0 > Environment: Build from 4.3 > Reporter: Sangeetha Hariharan > Assignee: edison su > Priority: Critical > Fix For: 4.3.0 > > Attachments: kvm-ssvm.rar, kvmhost-down-up.rar > > > Set up: > Advanced zone set up with 2 KVM (rhel 6.3) hosts. > Few Vms running on both hosts. > I was testing use cases of bringing down hosts and brings them back up again . > During this testing , SSVM got stuck in "Starting" state for ever. > Following exception seen in agent log: > 2013-12-19 10:28:08,616 ERROR [agent.transport.Request] > (agentRequest-Handler-5:null) Caught problem with > [{"com.cloud.agent.api.StartCommand":{"vm":{"id":35,"name":"s-35-MyTestVM","type":"SecondaryStorageVm","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian > GNU/Linux 5.0 (32-bit)","bootArgs":" template\u003ddomP type\u003dsecstorage > host\u003d10.223.49.6 port\u003d8250 name\u003ds-35-MyTestVM zone\u003d1 > pod\u003d1 guid\u003ds-35-MyTestVM > resource\u003dcom.cloud.storage.resource.PremiumSecondaryStorageResource > instance\u003dSecStorage sslcopy\u003dtrue role\u003dtemplateProcessor > mtu\u003d1500 eth2ip\u003d10.223.138.133 eth2mask\u003d255.255.255.192 > gateway\u003d10.223.138.129 public.network.device\u003deth2 > eth0ip\u003d169.254.0.251 eth0mask\u003d255.255.0.0 eth1ip\u003d10.223.58.137 > eth1mask\u003d255.255.255.192 mgmtcidr\u003d10.223.49.0/26 > localgw\u003d10.223.58.129 private.network.device\u003deth1 > eth3ip\u003d10.223.58.147 eth3mask\u003d255.255.255.192 > storageip\u003d10.223.58.147 storagenetmask\u003d255.255.255.192 > storagegateway\u003d10.223.58.129 internaldns1\u003d10.223.240.234 > dns1\u003d10.223.240.232","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"6e5928251a8718f6","params":{},"uuid":"db3f9893-d98e-4fb4-a6a4-95f2c95ce407","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"488b20bf-e706-46b9-9039-4dd407aa23ba","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4aedbaff-1e54-37a2-a150-0b67dbe58ed5","id":3,"poolType":"NetworkFilesystem","host":"10.223.57.195","path":"/export/home/kvm/primary1","port":2049,"url":"NetworkFilesystem://10.223.57.195//export/home/kvm/primary1/?ROLE\u003dPrimary\u0026STOREUUID\u003d4aedbaff-1e54-37a2-a150-0b67dbe58ed5"}},"name":"ROOT-35","size":0,"path":"488b20bf-e706-46b9-9039-4dd407aa23ba","volumeId":35,"vmName":"s-35-MyTestVM","accountId":1,"format":"QCOW2","id":35,"deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"488b20bf-e706-46b9-9039-4dd407aa23ba","type":"ROOT","_details":{"managed":"false","storagePort":"2049","storageHost":"10.223.57.195","volumeSize":"0"}}],"nics":[{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"uuid":"35eb2804-514b-40d5-8e20-a2f423ca2625","ip":"10.223.138.133","netmask":"255.255.255.192","gateway":"10.223.138.129","mac":"06:66:22:00:00:15","dns1":"10.223.240.232","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://1382","isolationUri":"vlan://1382","isSecurityGroupEnabled":false},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"uuid":"ea38c96f-cc88-463b-812e-a4a69b59d7f7","ip":"169.254.0.251","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:fb","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"aa865f74-1add-476c-ad0f-86aed6b84155","ip":"10.223.58.137","netmask":"255.255.255.192","gateway":"10.223.58.129","mac":"06:10:f0:00:00:06","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false},{"deviceId":3,"networkRateMbps":-1,"defaultNic":false,"uuid":"032a8d30-fdf9-493d-ad2c-e3c2627e1869","ip":"10.223.58.147","netmask":"255.255.255.192","gateway":"10.223.58.129","mac":"06:40:ce:00:00:10","broadcastType":"Native","type":"Storage","isSecurityGroupEnabled":false}]},"hostIp":"10.223.58.131","executeInSequence":false,"contextMap":{},"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.251","port":3922,"interval":6,"retries":100,"name":"s-35-MyTestVM","contextMap":{},"wait":0}}] > com.google.gson.JsonParseException: The JsonDeserializer > com.cloud.agent.transport.InterfaceTypeAdaptor@6db22920 failed to deserialize > json object > {"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"488b20bf-e706-46b9-9039-4dd407aa23ba","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4aedbaff-1e54-37a2-a150-0b67dbe58ed5","id":3,"poolType":"NetworkFilesystem","host":"10.223.57.195","path":"/export/home/kvm/primary1","port":2049,"url":"NetworkFilesystem://10.223.57.195//export/home/kvm/primary1/?ROLE=Primary&STOREUUID=4aedbaff-1e54-37a2-a150-0b67dbe58ed5"}},"name":"ROOT-35","size":0,"path":"488b20bf-e706-46b9-9039-4dd407aa23ba","volumeId":35,"vmName":"s-35-MyTestVM","accountId":1,"format":"QCOW2","id":35,"deviceId":0,"hypervisorType":"KVM"}} > given the type interface com.cloud.agent.api.to.DataTO > at > com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:64) > at > com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92) > at > com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:117) > at > com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63) > at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120) > at > com.google.gson.JsonDeserializationVisitor.visitChild(JsonDeserializationVisitor.java:110) > at > com.google.gson.JsonDeserializationVisitor.visitChildAsObject(JsonDeserializationVisitor.java:99) > at > com.google.gson.JsonArrayDeserializationVisitor.visitArray(JsonArrayDeserializationVisitor.java:70) > at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:112) > at > com.google.gson.JsonDeserializationVisitor.visitChild(JsonDeserializationVisitor.java:110) > at > com.google.gson.JsonDeserializationVisitor.visitChildAsArray(JsonDeserializationVisitor.java:106) > at > com.google.gson.JsonObjectDeserializationVisitor.visitArrayField(JsonObjectDeserializationVisitor.java:82) > at > com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:67) > at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120) > at > com.google.gson.JsonDeserializationVisitor.visitChild(JsonDeserializationVisitor.java:110) > at > com.google.gson.JsonDeserializationVisitor.visitChildAsObject(JsonDeserializationVisitor.java:99) > at > com.google.gson.JsonObjectDeserializationVisitor.visitObjectField(JsonObjectDeserializationVisitor.java:63) > at > com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:69) > at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120) > at > com.google.gson.JsonDeserializationContextDefault.fromJsonObject(JsonDeserializationContextDefault.java:76) > at > com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:54) > at com.google.gson.Gson.fromJson(Gson.java:551) > at com.google.gson.Gson.fromJson(Gson.java:521) > at > com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:77) > at > com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:36) > at > com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51) > at > com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92) > at > com.google.gson.JsonDeserializationVisitor.visitUsingCustomHandler(JsonDeserializationVisitor.java:80) > at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:101) > at > com.google.gson.JsonDeserializationContextDefault.fromJsonArray(JsonDeserializationContextDefault.java:67) > at > com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:52) > at com.google.gson.Gson.fromJson(Gson.java:551) > at com.google.gson.Gson.fromJson(Gson.java:498) > at com.cloud.agent.transport.Request.getCommands(Request.java:246) > at com.cloud.agent.Agent.processRequest(Agent.java:438) > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:806) > at com.cloud.utils.nio.Task.run(Task.java:83) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > Caused by: java.lang.NullPointerException > at > com.google.gson.FieldAttributes.getAnnotationFromArray(FieldAttributes.java:231) > at > com.google.gson.FieldAttributes.getAnnotation(FieldAttributes.java:150) > at > com.google.gson.VersionExclusionStrategy.shouldSkipField(VersionExclusionStrategy.java:38) > at > com.google.gson.DisjunctionExclusionStrategy.shouldSkipField(DisjunctionExclusionStrategy.java:38) > at > com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:58) > at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120) > at > com.google.gson.JsonDeserializationContextDefault.fromJsonObject(JsonDeserializationContextDefault.java:76) > at > com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:54) > at com.google.gson.Gson.fromJson(Gson.java:551) > at com.google.gson.Gson.fromJson(Gson.java:521) > at > com.cloud.agent.transport.InterfaceTypeAdaptor.deserialize(InterfaceTypeAdaptor.java:64) > at > com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51) > ... 39 more > 2013-12-19 10:28:08,620 WARN [utils.nio.Task] (agentRequest-Handler-5:null) > Caught the following exception but pushing on > java.lang.NullPointerException > at com.cloud.agent.Agent.processRequest(Agent.java:525) > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:806) > at com.cloud.utils.nio.Task.run(Task.java:83) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > @ -- This message was sent by Atlassian JIRA (v6.1.5#6160)