[
https://issues.apache.org/jira/browse/CLOUDSTACK-3498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Min Chen reassigned CLOUDSTACK-3498:
------------------------------------
Assignee: Min Chen
> [Object_Store_Refactor] Deleting and adding back the zone does not create NFS
> Cache store and Failure in Sytem VMs Start Up
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-3498
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3498
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.2.0
> Environment: Latest build from ACS 4.2 branch.
> Reporter: Sanjeev N
> Assignee: Min Chen
> Priority: Critical
> Fix For: 4.2.0
>
> Attachments: management-server.rar
>
>
> Deleting a zone does not delete the NFS Cache store in that zone and adding
> another zone with same NFS cache store details those were given in the
> deleted zone does not create another NFS Cache store in the new zone. Hence
> the Failure in System vms bring up.
> Steps to Reproduce:
> ================
> 1.Bring up CS with advanced zone say zone1.
> 2.Add secondary storage with S3 provider and with NFS Cache Store
> 3.Delete secondary storage
> 4.Delete zone1
> 5.Add another zone say zone2
> 6.Add secondary storage with s3 provider and NFS Cache storage in Zone2 with
> NFS Server and Path same as zone1
> Observations:
> ===========
> 1.Deleting zone did not delete NFS Cache store from image_store table.
> 2.Adding NFS Cache store with same details in new zone did not create another
> entry in image_store table for the new zone.
> Impact:
> ======
> CS downloads Routing template from S3 to NFS Cache store to bring up the
> System VMs. Since CS did not find NFS Cache store in the new zone to copy the
> routing template spinning system vms failed with following exception:
> 2013-07-12 06:28:14,731 DEBUG [storage.motion.AncientDataMotionStrategy]
> (consoleproxy-1:null) needCacheStorage true, dest at null dest role
> Primarytemplate/tmpl/1/3/routing-3/systemvmtemplate-2013-06-25-master-kvm.qcow2.bz2
> src role Image
> 2013-07-12 06:28:14,737 DEBUG [network.guru.PublicNetworkGuru]
> (secstorage-1:null) public network deallocate network: networkId: 210, ip:
> 10.147.48.3
> 2013-07-12 06:28:14,739 DEBUG [cache.allocator.StorageCacheRandomAllocator]
> (consoleproxy-1:null) Can't find cache storage in zone: 3
> 2013-07-12 06:28:14,739 DEBUG [storage.motion.AncientDataMotionStrategy]
> (consoleproxy-1:null) copy object failed:
> com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope
> id 3 type ZONE
> 2013-07-12 06:28:14,739 DEBUG [storage.motion.AncientDataMotionStrategy]
> (consoleproxy-1:null) copy failed
> com.cloud.utils.exception.CloudRuntimeException:
> com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope
> id 3 type ZONE
> at
> org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyObject(AncientDataMotionStrategy.java:204)
> at
> org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:351)
> at
> org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55)
> at
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:390)
> at
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:490)
> at
> com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2485)
> at
> com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2542)
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:851)
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:557)
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:550)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:927)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1664)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
> at
> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
> at
> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> at
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
> at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-07-12 06:28:14,752 DEBUG [cloud.storage.VolumeManagerImpl]
> (consoleproxy-1:null) Unable to create
> Vol[91|vm=79|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
> com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope
> id 3 type ZONE
> 2013-07-12 06:28:14,752 INFO [cloud.vm.VirtualMachineManagerImpl]
> (consoleproxy-1:null) Unable to contact resource.
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3] is
> unreachable: Unable to create
> Vol[91|vm=79|ROOT]:com.cloud.utils.exception.CloudRuntimeException:
> com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope
> id 3 type ZONE
> at
> com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2493)
> at
> com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2542)
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:851)
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:557)
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:550)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:927)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1664)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
> at
> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
> at
> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> at
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
> at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-07-12 06:28:14,758 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (consoleproxy-1:null) Cleaning up resources for the vm
> VM[ConsoleProxy|v-79-VM] in Starting state
> DB entries from Cloud table:
> ======================
> mysql> select * from image_store where role='Image' and removed is null\G;
> *************************** 1. row ***************************
> id: 3
> name: objstore
> image_provider_name: S3
> protocol: http
> url: NULL
> data_center_id: NULL
> scope: REGION
> role: Image
> uuid: 0c444304-7e5f-4137-8da1-a812c3b81cf7
> parent: NULL
> created: 2013-07-11 15:07:56
> removed: NULL
> total_size: NULL
> used_bytes: NULL
> *************************** 2. row ***************************
> id: 5
> name: obj_store
> image_provider_name: S3
> protocol: http
> url: NULL
> data_center_id: NULL
> scope: REGION
> role: Image
> uuid: 789b6388-ea95-41e6-98e6-e2ded85f119f
> parent: NULL
> created: 2013-07-12 10:23:09
> removed: NULL
> total_size: NULL
> used_bytes: NULL
> 2 rows in set (0.00 sec)
> mysql> select * from image_store where role='ImageCache' and removed is
> null\G;
> *************************** 1. row ***************************
> id: 1
> name: nfs://10.147.28.7/export/home/sanjeev/sec_xen_os
> image_provider_name: NFS
> protocol: nfs
> url: nfs://10.147.28.7/export/home/sanjeev/sec_xen_os
> data_center_id: 1
> scope: ZONE
> role: ImageCache
> uuid: 8e43fa4b-cf0c-46c2-9e68-ead6bd481742
> parent: NULL
> created: 2013-07-11 14:51:18
> removed: NULL
> total_size: NULL
> used_bytes: NULL
> *************************** 2. row ***************************
> id: 4
> name: nfs://10.147.28.7/export/home/sanjeev/sec_kvm_os
> image_provider_name: NFS
> protocol: nfs
> url: nfs://10.147.28.7/export/home/sanjeev/sec_kvm_os
> data_center_id: 2
> scope: ZONE
> role: ImageCache
> uuid: b42417d4-3e97-4f75-9439-25f97badcaa1
> parent: NULL
> created: 2013-07-11 16:41:51
> removed: NULL
> total_size: NULL
> used_bytes: NULL
> 2 rows in set (0.00 sec)
> mysql> select * from data_center;
> +----+---------+--------------------------------------+-------------+---------------+------+---------------+---------------+---------+---------+--------------------+-------------+--------------------+--------+-----------+-------------+---------------+------------------+-------------------+---------------+-----------------------+---------------+-------------------+------------------+--------------------------------------+---------------------------+--------------------------+---------------------+-------+---------+-------------+--------------+----------+----------+
> | id | name | uuid | description | dns1
> | dns2 | internal_dns1 | internal_dns2 | gateway | netmask |
> router_mac_address | mac_address | guest_network_cidr | domain | domain_id |
> networktype | dns_provider | gateway_provider | firewall_provider |
> dhcp_provider | lb_provider | vpn_provider | userdata_provider |
> allocation_state | zone_token |
> is_security_group_enabled | is_local_storage_enabled | removed |
> owner | created | lastUpdated | engine_state | ip6_dns1 | ip6_dns2 |
> +----+---------+--------------------------------------+-------------+---------------+------+---------------+---------------+---------+---------+--------------------+-------------+--------------------+--------+-----------+-------------+---------------+------------------+-------------------+---------------+-----------------------+---------------+-------------------+------------------+--------------------------------------+---------------------------+--------------------------+---------------------+-------+---------+-------------+--------------+----------+----------+
> | 1 | Basic | d234e7a5-5cd0-4029-9b80-07a47f470e79 | NULL |
> 10.103.128.16 | NULL | 10.103.128.16 | NULL | NULL | NULL |
> 02:00:00:00:00:01 | 21 | NULL | NULL | NULL |
> Basic | VirtualRouter | NULL | NULL |
> VirtualRouter | ElasticLoadBalancerVm | NULL | VirtualRouter |
> Enabled | 64c45795-5345-36e5-b52d-5530cd602a55 |
> 1 | 0 | NULL | NULL | NULL |
> NULL | Disabled | NULL | NULL |
> | 2 | NULL | 45072eac-10d8-499d-9325-d4ee79e45026 | NULL |
> 10.103.128.16 | NULL | 10.103.128.16 | NULL | NULL | NULL |
> 02:00:00:00:00:01 | 19 | 10.1.1.0/24 | NULL | NULL |
> Advanced | VirtualRouter | VirtualRouter | VirtualRouter |
> VirtualRouter | VirtualRouter | VirtualRouter | VirtualRouter |
> Disabled | b2359ba6-39c7-3dc6-a2ca-ac9091a6a06b |
> 0 | 1 | 2013-07-12 10:09:14 | NULL | NULL |
> NULL | Disabled | NULL | NULL |
> | 3 | Adv-kvm | 37b35302-af3c-43b9-bc6c-acb17642b0d3 | NULL |
> 10.103.128.16 | NULL | 10.103.128.16 | NULL | NULL | NULL |
> 02:00:00:00:00:01 | 21 | 10.1.1.0/24 | NULL | NULL |
> Advanced | VirtualRouter | VirtualRouter | VirtualRouter |
> VirtualRouter | VirtualRouter | VirtualRouter | VirtualRouter |
> Disabled | 8f2a7a84-dbfc-35e0-9201-f3066c65b414 |
> 0 | 1 | NULL | NULL | NULL |
> NULL | Disabled | NULL | NULL |
> +----+---------+--------------------------------------+-------------+---------------+------+---------------+---------------+---------+---------+--------------------+-------------+--------------------+--------+-----------+-------------+---------------+------------------+-------------------+---------------+-----------------------+---------------+-------------------+------------------+--------------------------------------+---------------------------+--------------------------+---------------------+-------+---------+-------------+--------------+----------+----------+
> 3 rows in set (0.00 sec)
> 1.Data_Center 2 was removed from the setup, however we can see the image
> store with role image_cache for data_center 2 in the image_store table.
> 2. Data_Center 3 was created after deleting DC 2 , and provided the same
> image_store details as DC2, hence there is no image store with role
> image_cache for DC3.
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
