Chandan Purushothama created CLOUDSTACK-4485:
------------------------------------------------
Summary: 3.0.6 to ASF 4.2 Upgrade [VMWare]: System VMs Fail to
create in the Cluster due to invalid path information of the template on the
Primary Storage in the cluster
Key: CLOUDSTACK-4485
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4485
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server, VMware
Affects Versions: 4.2.0
Environment: 3.0.6-1 to 4.2 Upgraded Environment, VMWare ESXi 5.0
Hosts in two different clusters
Reporter: Chandan Purushothama
Priority: Critical
Fix For: 4.2.1
Upgraded Setup Configuration:
3.0.6-1 Management Server;
Two Clusters on the Setup: DC1-Cluster1; DC2-Cluster2;
Each Clusters have one Shared Primary storage
Each Cluster has one ESXi 5.0 Host.
Upgraded it to 4.2 Build
======================================
Observed the Bug when I did the following Steps:
======================================
1. Create a new account
2. Login as the new User
3. Create a VPC
============
Observations:
============
2013-08-23 12:19:24,280 DEBUG [agent.transport.Request]
(Job-Executor-23:job-124 = [ 5e3d7605-2f54-4a08-985d-4da655dd82ad ]) Seq
1-449844129: Sending { Cmd , MgmtId: 7471666038533, via: 1, Ver: v1, Flags:
100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"a873d86aa6cb3aec9d3510350a0717f1","origUrl":"http://download.cloud.com/templates/4.2/systemvmtemplate-4.2-vh7.ova","uuid":"5d970767-5a9d-47fe-8a48-7c5f3847ddc8","id":202,"format":"OVA","accountId":2,"checksum":"8fde62b1089e5844a9cd3b9b953f9596","hvm":true,"displayText":"systemvm-vmware-4.2","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"d89e306d-88e8-3997-ade6-fa97fed423bf","id":200,"poolType":"NetworkFilesystem","host":"10.223.110.232","path":"/export/home/chandan/307PB-195-103/primary1","port":2049}},"name":"202-2-b045d1fb-76d9-38e8-b569-3f7690565fb7","hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"2984549b-ab6f-41b5-9448-2c51bbb36290","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"d89e306d-88e8-3997-ade6-fa97fed423bf","id":200,"poolType":"NetworkFilesystem","host":"10.223.110.232","path":"/export/home/chandan/307PB-195-103/primary1","port":2049}},"name":"ROOT-20","size":2097152000,"volumeId":30,"vmName":"r-20-VMWARERETEST","accountId":5,"format":"OVA","id":30,"hypervisorType":"None"}},"executeInSequence":false,"wait":0}}]
}
2013-08-23 12:19:24,281 DEBUG [agent.transport.Request]
(Job-Executor-23:job-124 = [ 5e3d7605-2f54-4a08-985d-4da655dd82ad ]) Seq
1-449844129: Executing: { Cmd , MgmtId: 7471666038533, via: 1, Ver: v1, Flags:
100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"a873d86aa6cb3aec9d3510350a0717f1","origUrl":"http://download.cloud.com/templates/4.2/systemvmtemplate-4.2-vh7.ova","uuid":"5d970767-5a9d-47fe-8a48-7c5f3847ddc8","id":202,"format":"OVA","accountId":2,"checksum":"8fde62b1089e5844a9cd3b9b953f9596","hvm":true,"displayText":"systemvm-vmware-4.2","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"d89e306d-88e8-3997-ade6-fa97fed423bf","id":200,"poolType":"NetworkFilesystem","host":"10.223.110.232","path":"/export/home/chandan/307PB-195-103/primary1","port":2049}},"name":"202-2-b045d1fb-76d9-38e8-b569-3f7690565fb7","hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"2984549b-ab6f-41b5-9448-2c51bbb36290","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"d89e306d-88e8-3997-ade6-fa97fed423bf","id":200,"poolType":"NetworkFilesystem","host":"10.223.110.232","path":"/export/home/chandan/307PB-195-103/primary1","port":2049}},"name":"ROOT-20","size":2097152000,"volumeId":30,"vmName":"r-20-VMWARERETEST","accountId":5,"format":"OVA","id":30,"hypervisorType":"None"}},"executeInSequence":false,"wait":0}}]
}
2013-08-23 12:19:24,281 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-470:null) Seq 1-449844129: Executing request
2013-08-23 12:19:24,329 WARN [storage.resource.VmwareStorageProcessor]
(DirectAgent-470:10.223.59.66) Template host in vSphere is not in connected
state, request template reload
2013-08-23 12:19:24,330 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-470:null) Seq 1-449844129: Response Received:
2013-08-23 12:19:24,330 DEBUG [agent.transport.Request] (DirectAgent-470:null)
Seq 1-449844129: Processing: { Ans: , MgmtId: 7471666038533, via: 1, Ver: v1,
Flags: 10,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Template
host in vSphere is not in connected state, request template
reload","wait":0}}] }
2013-08-23 12:19:24,330 DEBUG [agent.transport.Request]
(Job-Executor-23:job-124 = [ 5e3d7605-2f54-4a08-985d-4da655dd82ad ]) Seq
1-449844129: Received: { Ans: , MgmtId: 7471666038533, via: 1, Ver: v1, Flags:
10, { CopyCmdAnswer } }
2013-08-23 12:19:24,338 WARN [storage.datastore.ObjectInDataStoreManagerImpl]
(Job-Executor-23:job-124 = [ 5e3d7605-2f54-4a08-985d-4da655dd82ad ])
Unsupported data object (VOLUME,
org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@10c976a0), no need
to delete from object in store ref table
2013-08-23 12:19:24,340 DEBUG [cloud.storage.VolumeManagerImpl]
(Job-Executor-23:job-124 = [ 5e3d7605-2f54-4a08-985d-4da655dd82ad ]) Unable to
create Vol[30|vm=20|ROOT]:Template host in vSphere is not in connected state,
request template reload
2013-08-23 12:19:24,340 INFO [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-23:job-124 = [ 5e3d7605-2f54-4a08-985d-4da655dd82ad ]) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:200] is
unreachable: Unable to create Vol[30|vm=20|ROOT]:Template host in vSphere is
not in connected state, request template reload
at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2534)
at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2582)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:885)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:575)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2740)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1872)
at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployRouter(VirtualNetworkApplianceManagerImpl.java:1664)
at
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.deployVpcRouter(VpcVirtualNetworkApplianceManagerImpl.java:331)
at
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.findOrDeployVirtualRouterInVpc(VpcVirtualNetworkApplianceManagerImpl.java:227)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.deployVirtualRouterInVpc(VpcVirtualNetworkApplianceManagerImpl.java:176)
at
com.cloud.network.element.VpcVirtualRouterElement.implementVpc(VpcVirtualRouterElement.java:126)
at
com.cloud.network.vpc.VpcManagerImpl.startVpc(VpcManagerImpl.java:994)
CopyCommand is searching for "path":"a873d86aa6cb3aec9d3510350a0717f1" of the
System VM Template to create the VPC Router. That is the name of the template
that is supposed to be present on the primary storage. But it is not present on
the primary storage but on the database it is mentioned as Downloaded.
mysql> select * from template_spool_ref where template_id=202;
+----+---------+-------------+---------------------+--------------+--------+--------------+----------------+-----------+----------------------------------+----------------------------------+---------------+---------------+-------+--------------+---------------------+
| id | pool_id | template_id | created | last_updated | job_id |
download_pct | download_state | error_str | local_path |
install_path | template_size | marked_for_gc | state |
update_count | updated |
+----+---------+-------------+---------------------+--------------+--------+--------------+----------------+-----------+----------------------------------+----------------------------------+---------------+---------------+-------+--------------+---------------------+
| 8 | 200 | 202 | 2013-08-20 22:41:18 | NULL | NULL |
100 | DOWNLOADED | NULL | a873d86aa6cb3aec9d3510350a0717f1 |
a873d86aa6cb3aec9d3510350a0717f1 | 0 | 0 | Ready |
2 | 2013-08-20 22:41:18 |
| 6 | 201 | 202 | 2013-08-20 22:07:23 | NULL | NULL |
100 | DOWNLOADED | NULL | d42e159e3a7b31b0ba7b5f978a992060 |
d42e159e3a7b31b0ba7b5f978a992060 | 0 | 0 | Ready |
2 | 2013-08-20 22:07:58 |
+----+---------+-------------+---------------------+--------------+--------+--------------+----------------+-----------+----------------------------------+----------------------------------+---------------+---------------+-------+--------------+---------------------+
2 rows in set (0.00 sec)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
