New deployment of CloudStack 4.2.1 that was build for nonoss VMware. The
environment is currently only running KVM, but will have VMware in the future.
That being said, we deployed the packages with nonoss at build time.
The CPVM starts up fine and works as expected. The SSVM starts up, the GUI
recognizes that it is up, but the agent never starts on the SSVM and the GUI
shows that as well. It looks like CloudStack is deciding that the VM should
start up using the SS class for VMware and that is ultimately causing it to
fail.
Here is a snippet from the cloud.log on the SSVM it self:
2014-02-27 13:32:35,075 INFO [cloud.agent.AgentShell] (main:null) Agent started
2014-02-27 13:32:35,078 INFO [cloud.agent.AgentShell] (main:null)
Implementation Version is 4.2.1
2014-02-27 13:32:35,078 INFO [cloud.agent.AgentShell] (main:null)
agent.properties found at /usr/local/cloud/systemvm/conf/agent.properties
2014-02-27 13:32:35,080 DEBUG [cloud.agent.AgentShell] (main:null) Found
property: instance
2014-02-27 13:32:35,081 DEBUG [cloud.agent.AgentShell] (main:null) Found
property: resource
2014-02-27 13:32:35,081 INFO [cloud.agent.AgentShell] (main:null) Defaulting
to using properties file for storage
2014-02-27 13:32:35,084 INFO [cloud.agent.AgentShell] (main:null) Defaulting
to the constant time backoff algorithm
2014-02-27 13:32:35,088 INFO [cloud.utils.LogUtils] (main:null) log4j
configuration found at /usr/local/cloud/systemvm/conf/log4j-cloud.xml
2014-02-27 13:32:35,110 DEBUG [cloud.agent.AgentShell] (main:null) Checking to
see if agent.SecStorage.pid exists.
2014-02-27 13:32:35,112 DEBUG [cloud.utils.ProcessUtil] (main:null)
environment.properties could not be opened
2014-02-27 13:32:35,119 DEBUG [cloud.utils.ProcessUtil] (main:null) Executing:
bash -c echo $PPID
2014-02-27 13:32:35,123 DEBUG [cloud.utils.ProcessUtil] (main:null) Execution
is successful.
2014-02-27 13:32:35,124 ERROR [cloud.agent.AgentShell] (main:null) Unable to
start agent: Resource class not found:
com.cloud.storage.resource.PremiumSecondaryStorageResource due to:
java.lang.ClassNotFoundException:
com.cloud.storage.resource.PremiumSecondaryStorageResource
And here is what the log looks like from the management server, it looks as if
we are passing the wrong argument to the SSVM on boot for some reason:
2014-02-27 07:45:35,615 DEBUG [agent.transport.Request] (secstorage-1:null) Seq
4-444203058: Sending { Cmd , MgmtId: 161343762130, via: 4, Ver: v1, Flags:
100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"2f28ae31-db7c-45db-a605-fb9a36cb9fed","origUrl":"http://download.cloud.com/templates/4.2/systemvmtemplate-2013-06-12-master-kvm.qcow2.bz2","uuid":"48f12a3b-9e43-11e3-ada2-002590d66ed2","id":3,"format":"QCOW2","accountId":1,"checksum":"6cea42b2633841648040becb588bd8f0","hvm":false,"displayText":"SystemVM
Template
(KVM)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"e2808596-44b3-4568-81b5-0720cb817b31","id":1,"poolType":"SharedMountPoint","host":"localhost","path":"/mnt/primary/zone1-primary1","port":0}},"name":"routing-3","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"9e182aa3-61ab-4152-b28c-c848b1adbee2","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"e2808596-44b3-4568-81b5-0720cb817b31","id":1,"poolType":"SharedMountPoint","host":"localhost","path":"/mnt/primary/zone1-primary1","port":0}},"name":"ROOT-45","size":0,"volumeId":45,"vmName":"s-45-VM","accountId":1,"format":"QCOW2","id":45,"hypervisorType":"KVM"}},"executeInSequence":true,"wait":0}}]
}
2014-02-27 07:45:35,867 DEBUG [agent.transport.Request]
(AgentManager-Handler-1:null) Seq 4-444203058: Processing: { Ans: , MgmtId:
161343762130, via: 4, Ver: v1, Flags: 110,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":0,"path":"ac2b28be-a84a-4b0d-acd8-603f8995db1a","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
}
2014-02-27 07:45:35,867 DEBUG [agent.transport.Request] (secstorage-1:null) Seq
4-444203058: Received: { Ans: , MgmtId: 161343762130, via: 4, Ver: v1, Flags:
110, { CopyCmdAnswer } }
2014-02-27 07:45:35,867 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-1:null) Seq 4-444203058: No more commands found
2014-02-27 07:45:36,069 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
(secstorage-1:null) VmWare hypervisor configured, telling the ssvm to load the
PremiumSecondaryStorageResource
2014-02-27 07:45:36,071 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
(secstorage-1:null) Boot Args for VM[SecondaryStorageVm|s-45-VM]:
template=domP type=secstorage host=172.16.215.6 port=8250 name=s-45-VM zone=1
pod=1 guid=s-45-VM
resource=com.cloud.storage.resource.PremiumSecondaryStorageResource
instance=SecStorage sslcopy=true role=templateProcessor mtu=1500 eth2ip=removed
eth2mask=255.255.255.0 gateway=removed public.network.device=eth2
eth0ip=169.254.0.216 eth0mask=255.255.0.0 eth1ip=172.16.215.248
eth1mask=255.255.255.0 mgmtcidr=172.16.215.0/24 localgw=172.16.215.1
private.network.device=eth1 eth3ip=172.16.215.247 eth3mask=255.255.255.0
storageip=172.16.215.247 storagenetmask=255.255.255.0
storagegateway=172.16.215.1 internaldns1=removed internaldns2=removed
dns1=removed dns2=removed
2014-02-27 07:45:36,248 DEBUG [agent.transport.Request] (secstorage-1:null) Seq
4-444203059: Sending { Cmd , MgmtId: 161343762130, via: 4, Ver: v1, Flags:
100111,
[{"com.cloud.agent.api.StartCommand":{"vm":{"id":45,"name":"s-45-VM","type":"SecondaryStorageVm","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian
GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP type=secstorage
host=172.16.215.6 port=8250 name=s-45-VM zone=1 pod=1 guid=s-45-VM
resource=com.cloud.storage.resource.PremiumSecondaryStorageResource
instance=SecStorage sslcopy=true role=templateProcessor mtu=1500 eth2ip=removed
eth2mask=255.255.255.0 gateway=removed public.network.device=eth2
eth0ip=169.254.0.216 eth0mask=255.255.0.0 eth1ip=172.16.215.248
eth1mask=255.255.255.0 mgmtcidr=172.16.215.0/24 localgw=172.16.215.1
private.network.device=eth1 eth3ip=172.16.215.247 eth3mask=255.255.255.0
storageip=172.16.215.247 storagenetmask=255.255.255.0
storagegateway=172.16.215.1 internaldns1=removed internaldns2=removed
dns1=removed
dns2=removed","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"d0eba270fc5ae388","params":{},"uuid":"75e4863f-af35-4273-9743-4103de16bc84","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"9e182aa3-61ab-4152-b28c-c848b1adbee2","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"e2808596-44b3-4568-81b5-0720cb817b31","id":1,"poolType":"SharedMountPoint","host":"localhost","path":"/mnt/primary/zone1-primary1","port":0}},"name":"ROOT-45","size":0,"path":"ac2b28be-a84a-4b0d-acd8-603f8995db1a","volumeId":45,"vmName":"s-45-VM","accountId":1,"format":"QCOW2","id":45,"hypervisorType":"KVM"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"uuid":"d25285b0-3b15-4774-8060-e70d908d97ad","ip”:"removed","netmask":"255.255.255.0","gateway”:"removed","mac":"06:e0:66:00:00:0b","dns1”:"removed","dns2”:"removed","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://3500","isolationUri":"vlan://3500","isSecurityGroupEnabled":false,"name":"cloudbr0"},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"uuid":"401fafb0-28ab-48a7-99d3-7ed6d08d99f1","ip":"169.254.0.216","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:d8","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"efe02715-5f2b-4b59-8923-9515c34d9a1c","ip":"172.16.215.248","netmask":"255.255.255.0","gateway":"172.16.215.1","mac":"06:84:88:00:00:09","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false,"name":"br0"},{"deviceId":3,"networkRateMbps":-1,"defaultNic":false,"uuid":"7db64f75-962a-4d4a-9e6a-b878b18d1634","ip":"172.16.215.247","netmask":"255.255.255.0","gateway":"172.16.215.1","mac":"06:99:ae:00:00:08","broadcastType":"Native","type":"Storage","isSecurityGroupEnabled":false,"name":"br0"}]},"hostIp":"172.16.215.7","executeInSequence":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.216","port":3922,"interval":6,"retries":100,"name":"s-45-VM","wait":0}}]
}
I went through the issue list on Jira to see if anyone has seen something
similar, and I did not find anything. Has anyone else experienced something
similar?
Thanks for reading!
