Figured this out. The management server is running CentOS, and our KVM
hypervisor is running Ubuntu… the Ubuntu 4.2.1 packages were erroneously build
without the proper jars for VMware, where as the CentOS packages were correct.
However, we pull the systemvm.iso from the KVM node, and not the management
server when the SSVM starts which explains why they were missing. All better
now!
On Feb 27, 2014, at 8:51 AM, John Skinner <[email protected]> wrote:
> New deployment of CloudStack 4.2.1 that was build for nonoss VMware. The
> environment is currently only running KVM, but will have VMware in the
> future. That being said, we deployed the packages with nonoss at build time.
>
> The CPVM starts up fine and works as expected. The SSVM starts up, the GUI
> recognizes that it is up, but the agent never starts on the SSVM and the GUI
> shows that as well. It looks like CloudStack is deciding that the VM should
> start up using the SS class for VMware and that is ultimately causing it to
> fail.
>
> Here is a snippet from the cloud.log on the SSVM it self:
>
> 2014-02-27 13:32:35,075 INFO [cloud.agent.AgentShell] (main:null) Agent
> started
> 2014-02-27 13:32:35,078 INFO [cloud.agent.AgentShell] (main:null)
> Implementation Version is 4.2.1
> 2014-02-27 13:32:35,078 INFO [cloud.agent.AgentShell] (main:null)
> agent.properties found at /usr/local/cloud/systemvm/conf/agent.properties
> 2014-02-27 13:32:35,080 DEBUG [cloud.agent.AgentShell] (main:null) Found
> property: instance
> 2014-02-27 13:32:35,081 DEBUG [cloud.agent.AgentShell] (main:null) Found
> property: resource
> 2014-02-27 13:32:35,081 INFO [cloud.agent.AgentShell] (main:null) Defaulting
> to using properties file for storage
> 2014-02-27 13:32:35,084 INFO [cloud.agent.AgentShell] (main:null) Defaulting
> to the constant time backoff algorithm
> 2014-02-27 13:32:35,088 INFO [cloud.utils.LogUtils] (main:null) log4j
> configuration found at /usr/local/cloud/systemvm/conf/log4j-cloud.xml
> 2014-02-27 13:32:35,110 DEBUG [cloud.agent.AgentShell] (main:null) Checking
> to see if agent.SecStorage.pid exists.
> 2014-02-27 13:32:35,112 DEBUG [cloud.utils.ProcessUtil] (main:null)
> environment.properties could not be opened
> 2014-02-27 13:32:35,119 DEBUG [cloud.utils.ProcessUtil] (main:null)
> Executing: bash -c echo $PPID
> 2014-02-27 13:32:35,123 DEBUG [cloud.utils.ProcessUtil] (main:null) Execution
> is successful.
> 2014-02-27 13:32:35,124 ERROR [cloud.agent.AgentShell] (main:null) Unable to
> start agent: Resource class not found:
> com.cloud.storage.resource.PremiumSecondaryStorageResource due to:
> java.lang.ClassNotFoundException:
> com.cloud.storage.resource.PremiumSecondaryStorageResource
>
> And here is what the log looks like from the management server, it looks as
> if we are passing the wrong argument to the SSVM on boot for some reason:
>
> 2014-02-27 07:45:35,615 DEBUG [agent.transport.Request] (secstorage-1:null)
> Seq 4-444203058: Sending { Cmd , MgmtId: 161343762130, via: 4, Ver: v1,
> Flags: 100111,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"2f28ae31-db7c-45db-a605-fb9a36cb9fed","origUrl":"http://download.cloud.com/templates/4.2/systemvmtemplate-2013-06-12-master-kvm.qcow2.bz2","uuid":"48f12a3b-9e43-11e3-ada2-002590d66ed2","id":3,"format":"QCOW2","accountId":1,"checksum":"6cea42b2633841648040becb588bd8f0","hvm":false,"displayText":"SystemVM
> Template
> (KVM)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"e2808596-44b3-4568-81b5-0720cb817b31","id":1,"poolType":"SharedMountPoint","host":"localhost","path":"/mnt/primary/zone1-primary1","port":0}},"name":"routing-3","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"9e182aa3-61ab-4152-b28c-c848b1adbee2","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"e2808596-44b3-4568-81b5-0720cb817b31","id":1,"poolType":"SharedMountPoint","host":"localhost","path":"/mnt/primary/zone1-primary1","port":0}},"name":"ROOT-45","size":0,"volumeId":45,"vmName":"s-45-VM","accountId":1,"format":"QCOW2","id":45,"hypervisorType":"KVM"}},"executeInSequence":true,"wait":0}}]
> }
> 2014-02-27 07:45:35,867 DEBUG [agent.transport.Request]
> (AgentManager-Handler-1:null) Seq 4-444203058: Processing: { Ans: , MgmtId:
> 161343762130, via: 4, Ver: v1, Flags: 110,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":0,"path":"ac2b28be-a84a-4b0d-acd8-603f8995db1a","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
> }
> 2014-02-27 07:45:35,867 DEBUG [agent.transport.Request] (secstorage-1:null)
> Seq 4-444203058: Received: { Ans: , MgmtId: 161343762130, via: 4, Ver: v1,
> Flags: 110, { CopyCmdAnswer } }
> 2014-02-27 07:45:35,867 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-1:null) Seq 4-444203058: No more commands found
> 2014-02-27 07:45:36,069 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
> (secstorage-1:null) VmWare hypervisor configured, telling the ssvm to load
> the PremiumSecondaryStorageResource
> 2014-02-27 07:45:36,071 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
> (secstorage-1:null) Boot Args for VM[SecondaryStorageVm|s-45-VM]:
> template=domP type=secstorage host=172.16.215.6 port=8250 name=s-45-VM zone=1
> pod=1 guid=s-45-VM
> resource=com.cloud.storage.resource.PremiumSecondaryStorageResource
> instance=SecStorage sslcopy=true role=templateProcessor mtu=1500
> eth2ip=removed eth2mask=255.255.255.0 gateway=removed
> public.network.device=eth2 eth0ip=169.254.0.216 eth0mask=255.255.0.0
> eth1ip=172.16.215.248 eth1mask=255.255.255.0 mgmtcidr=172.16.215.0/24
> localgw=172.16.215.1 private.network.device=eth1 eth3ip=172.16.215.247
> eth3mask=255.255.255.0 storageip=172.16.215.247 storagenetmask=255.255.255.0
> storagegateway=172.16.215.1 internaldns1=removed internaldns2=removed
> dns1=removed dns2=removed
> 2014-02-27 07:45:36,248 DEBUG [agent.transport.Request] (secstorage-1:null)
> Seq 4-444203059: Sending { Cmd , MgmtId: 161343762130, via: 4, Ver: v1,
> Flags: 100111,
> [{"com.cloud.agent.api.StartCommand":{"vm":{"id":45,"name":"s-45-VM","type":"SecondaryStorageVm","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian
> GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP type=secstorage
> host=172.16.215.6 port=8250 name=s-45-VM zone=1 pod=1 guid=s-45-VM
> resource=com.cloud.storage.resource.PremiumSecondaryStorageResource
> instance=SecStorage sslcopy=true role=templateProcessor mtu=1500
> eth2ip=removed eth2mask=255.255.255.0 gateway=removed
> public.network.device=eth2 eth0ip=169.254.0.216 eth0mask=255.255.0.0
> eth1ip=172.16.215.248 eth1mask=255.255.255.0 mgmtcidr=172.16.215.0/24
> localgw=172.16.215.1 private.network.device=eth1 eth3ip=172.16.215.247
> eth3mask=255.255.255.0 storageip=172.16.215.247 storagenetmask=255.255.255.0
> storagegateway=172.16.215.1 internaldns1=removed internaldns2=removed
> dns1=removed
> dns2=removed","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"d0eba270fc5ae388","params":{},"uuid":"75e4863f-af35-4273-9743-4103de16bc84","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"9e182aa3-61ab-4152-b28c-c848b1adbee2","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"e2808596-44b3-4568-81b5-0720cb817b31","id":1,"poolType":"SharedMountPoint","host":"localhost","path":"/mnt/primary/zone1-primary1","port":0}},"name":"ROOT-45","size":0,"path":"ac2b28be-a84a-4b0d-acd8-603f8995db1a","volumeId":45,"vmName":"s-45-VM","accountId":1,"format":"QCOW2","id":45,"hypervisorType":"KVM"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"uuid":"d25285b0-3b15-4774-8060-e70d908d97ad","ip”:"removed","netmask":"255.255.255.0","gateway”:"removed","mac":"06:e0:66:00:00:0b","dns1”:"removed","dns2”:"removed","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://3500","isolationUri":"vlan://3500","isSecurityGroupEnabled":false,"name":"cloudbr0"},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"uuid":"401fafb0-28ab-48a7-99d3-7ed6d08d99f1","ip":"169.254.0.216","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:d8","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"efe02715-5f2b-4b59-8923-9515c34d9a1c","ip":"172.16.215.248","netmask":"255.255.255.0","gateway":"172.16.215.1","mac":"06:84:88:00:00:09","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false,"name":"br0"},{"deviceId":3,"networkRateMbps":-1,"defaultNic":false,"uuid":"7db64f75-962a-4d4a-9e6a-b878b18d1634","ip":"172.16.215.247","netmask":"255.255.255.0","gateway":"172.16.215.1","mac":"06:99:ae:00:00:08","broadcastType":"Native","type":"Storage","isSecurityGroupEnabled":false,"name":"br0"}]},"hostIp":"172.16.215.7","executeInSequence":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.0.216","port":3922,"interval":6,"retries":100,"name":"s-45-VM","wait":0}}]
> }
>
> I went through the issue list on Jira to see if anyone has seen something
> similar, and I did not find anything. Has anyone else experienced something
> similar?
>
> Thanks for reading!
