Hi Prasanna, Good day to you, and thank you for your e-mail.
See my reply inline below. On Mon, Jul 15, 2013 at 12:38 PM, Prasanna Santhanam <[email protected]> wrote: > On Mon, Jul 15, 2013 at 11:58:47AM +0800, Indra Pramana wrote: > > Dear all, > > > > In addition to my previous e-mail, I just realised that the wrong > capacity > > usage information is only applicable to the Ceph RBD primary storage. I > did > > a check manually on the "storage_pool" table on the "cloud" MySQL > database: > > > > > +-----+-----------------+--------------------------------------+-------------------+------+----------------+--------+------------+-------------------+----------------+---------------------------+------------------------------------------------+------------------------+---------------------+---------------------+-------------+-------------+---------------------+-------+ > > | id | name | uuid | > > pool_type | port | data_center_id | pod_id | cluster_id | > > available_bytes | capacity_bytes | host_address | > > user_info | path | > > created | removed | update_time | status | > > storage_provider_id | scope | > > > +-----+-----------------+--------------------------------------+-------------------+------+----------------+--------+------------+-------------------+----------------+---------------------------+------------------------------------------------+------------------------+---------------------+---------------------+-------------+-------------+---------------------+-------+ > > | 209 | sc-image | bab81ce8-d53f-3a7d-b8f6-841702f65c89 | > > RBD | 6789 | 6 | 6 | 6 | > > 38283921137336466 | 6013522722816 | ceph-mon.xxx.com | admin:xxx | > > sc1 | 2013-07-13 08:58:27 | NULL | > > NULL | Up | NULL | NULL | > > > > The "available_bytes" column is wrong. > > > > My issue is similar to the one reported by Guangjian Liu here, but it > seems > > that so far there's no solutions available? > > > > > http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201304.mbox/%3ccakryd0b_qtjhtsfjk8mceqzdh6froxe6ekjmfrcmdn54e5z...@mail.gmail.com%3E > > It might be a bug on ceph but the thread you referenced was resolved. > See here: http://markmail.org/message/mkm2fqyawmwpufsc > Thanks for the email thread. However, the conversation doesn't mention on how was the problem resolved. I tried to execute the same command which have suggested by Wido on the email thread: === root@hv-kvm-02:~# virsh pool-list Name State Autostart ----------------------------------------- 87ba6ca3-1b46-3f36-b138-2fd3bffb3d71 active no bab81ce8-d53f-3a7d-b8f6-841702f65c89 active no ff06ae2a-ff27-4ff6-87b3-c2f942cf76d6 active no root@hv-kvm-02:~# virsh pool-info bab81ce8-d53f-3a7d-b8f6-841702f65c89 Name: bab81ce8-d53f-3a7d-b8f6-841702f65c89 UUID: bab81ce8-d53f-3a7d-b8f6-841702f65c89 State: running Persistent: no Autostart: no Capacity: 5.47 TiB Allocation: *34819.02 TiB* <-- wrong information Available: 5.47 TiB === If it still persists, can you please file a bug on JIRA? > Any specific instructions on how to file the bug? > > Is it safe for me to update the database record manually (using the > UPDATE > > MySQL command) to reflect the actual usage of the disk? > > Even if you do whatever is reporting the stats to CS about Ceph's > storage usage, will overwrite those values. > Noted, thanks. So it's an issue on the Ceph RBD side rather than on CloudStack side? Updating the Cloudstack database record manually will not help? In any case, I tried to update the "available_bytes" and "capacity_bytes" record on the "storage_pool" table of "cloud" database manually to reflect the actual size of the RBD image, which is 3 TB: === indra@cs-mgmt-01:~$ rbd --image sc-image -p sc1 info rbd image 'sc-image': size 3072 GB in 786432 objects order 22 (4096 KB objects) block_name_prefix: rb.0.1825.238e1f29 format: 1 mysql> UPDATE storage_pool SET available_bytes=3072000000, capacity_bytes=3072000000 WHERE id=209; Query OK, 1 row affected (0.03 sec) Rows matched: 1 Changed: 1 Warnings: 0 === I tried to re-run the creation of the VM instance again, and it's still failed even though the error message is a bit different. Not too sure where the "usedBytes" value is coming from? usedBytes: 38283921137336466 === 2013-07-15 12:10:46,232 DEBUG [cloud.storage.StorageManagerImpl] (Job-Executor-5:job-170) Checking pool 209 for storage, totalSize: 3072000000, usedBytes: 38283921137336466, usedPct: 1.246221391189338E7, disable threshold: 0.85 2013-07-15 12:10:46,232 DEBUG [cloud.storage.StorageManagerImpl] (Job-Executor-5:job-170) Insufficient space on pool: 209 since its usage percentage: 1.246221391189338E7 has crossed the pool.storage.capacity.disablethreshold: 0.85 2013-07-15 12:10:46,232 DEBUG [storage.allocator.FirstFitStoragePoolAllocator] (Job-Executor-5:job-170) FirstFitStoragePoolAllocator returning 0 suitable storage pools 2013-07-15 12:10:46,232 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-5:job-170) No suitable pools found for volume: Vol[231|vm=227|DATADISK] under cluster: 6 2013-07-15 12:10:46,233 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-5:job-170) No suitable pools found 2013-07-15 12:10:46,233 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-5:job-170) No suitable storagePools found under this Cluster: 6 2013-07-15 12:10:46,233 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-5:job-170) Could not find suitable Deployment Destination for this VM under any clusters, returning. 2013-07-15 12:10:46,235 DEBUG [cloud.vm.UserVmManagerImpl] (Job-Executor-5:job-170) Destroying vm VM[User|Indra-Test-6] as it failed to create on Host with Id:null 2013-07-15 12:10:46,358 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-5:job-170) VM state transitted from :Stopped to Error with event: OperationFailedToErrorvm's original host id: null new host id: null host id before state transition: null 2013-07-15 12:10:47,052 INFO [user.vm.DeployVMCmd] (Job-Executor-5:job-170) com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|Indra-Test-6]Scope=interface com.cloud.dc.DataCenter; id=6 2013-07-15 12:10:47,052 INFO [user.vm.DeployVMCmd] (Job-Executor-5:job-170) Unable to create a deployment for VM[User|Indra-Test-6] === > > > > > Looking forward to your reply, thank you. > > > > Cheers. > > > > > > > > > > On Mon, Jul 15, 2013 at 11:41 AM, Indra Pramana <[email protected]> wrote: > > > > > Dear all, > > > > > > I am using CloudStack 4.1.0. Just managed to get it setup over the > > > weekend. System VMs have been created and both (SSVM and CPVM) are > running > > > fine. The default CentOS template has also been downloaded and ready > to use. > > > > > > However, I am not able to launch my first VM instance because the > storage > > > capacity checker is reporting wrong usage information. I have a total > of 13 > > > TB of primary storage and the capacity checker is reporting my usage is > > > 36,510 TB (274879.22%), which is not supposed to be the case. > > > > > > === > > > 2013-07-15 11:27:31,632 DEBUG [cloud.alert.AlertManagerImpl] > > > (CapacityChecker:null) System Alert: Low Available Storage in cluster > > > Cluster-01 pod Pod-01 of availability zone 01 > > > 2013-07-15 11:27:31,632 DEBUG [cloud.alert.AlertManagerImpl] > > > (CapacityChecker:null) Available storage space is low, total: 13282342 > MB, > > > used: 36510398411 MB (274879.22%) > > > === > > > > > > As a result, VM instance creation fails since it's not able to find > > > available storage pool. > > > > > > === > > > 2013-07-15 11:15:28,313 DEBUG [cloud.storage.StorageManagerImpl] > > > (Job-Executor-3:job-168) Checking pool: 208 for volume allocation > > > [Vol[227|vm=225|ROOT]], maxSize : 15828044742656, totalAllocatedSize : > > > 1769538048, askingSize : 8589934592, allocated disable threshold: 0.85 > > > 2013-07-15 11:15:28,313 DEBUG > > > [storage.allocator.AbstractStoragePoolAllocator] > (Job-Executor-3:job-168) > > > Checking if storage pool is suitable, name: sc-image ,poolId: 209 > > > 2013-07-15 11:15:28,313 DEBUG > > > [storage.allocator.AbstractStoragePoolAllocator] > (Job-Executor-3:job-168) > > > Is localStorageAllocationNeeded? false > > > 2013-07-15 11:15:28,313 DEBUG > > > [storage.allocator.AbstractStoragePoolAllocator] > (Job-Executor-3:job-168) > > > Is storage pool shared? true > > > 2013-07-15 11:15:28,317 DEBUG [cloud.storage.StorageManagerImpl] > > > (Job-Executor-3:job-168) Checking pool 209 for storage, totalSize: > > > 6013522722816, usedBytes: 38283921137336466, usedPct: > 6366.305226067051, > > > disable threshold: 0.85 > > > 2013-07-15 11:15:28,317 DEBUG [cloud.storage.StorageManagerImpl] > > > (Job-Executor-3:job-168) Insufficient space on pool: 209 since its > usage > > > percentage: 6366.305226067051 has crossed the > > > pool.storage.capacity.disablethreshold: 0.85 > > > 2013-07-15 11:15:28,317 DEBUG > > > [storage.allocator.FirstFitStoragePoolAllocator] > (Job-Executor-3:job-168) > > > FirstFitStoragePoolAllocator returning 1 suitable storage pools > > > 2013-07-15 11:15:28,317 DEBUG [cloud.deploy.FirstFitPlanner] > > > (Job-Executor-3:job-168) Checking suitable pools for volume (Id, Type): > > > (228,DATADISK) > > > 2013-07-15 11:15:28,317 DEBUG [cloud.deploy.FirstFitPlanner] > > > (Job-Executor-3:job-168) We need to allocate new storagepool for this > volume > > > 2013-07-15 11:15:28,319 DEBUG [cloud.deploy.FirstFitPlanner] > > > (Job-Executor-3:job-168) Calling StoragePoolAllocators to find suitable > > > pools > > > 2013-07-15 11:15:28,319 DEBUG > > > [storage.allocator.FirstFitStoragePoolAllocator] > (Job-Executor-3:job-168) > > > Looking for pools in dc: 6 pod:6 cluster:6 having tags:[rbd] > > > 2013-07-15 11:15:28,322 DEBUG > > > [storage.allocator.FirstFitStoragePoolAllocator] > (Job-Executor-3:job-168) > > > FirstFitStoragePoolAllocator has 1 pools to check for allocation > > > 2013-07-15 11:15:28,322 DEBUG > > > [storage.allocator.AbstractStoragePoolAllocator] > (Job-Executor-3:job-168) > > > Checking if storage pool is suitable, name: sc-image ,poolId: 209 > > > 2013-07-15 11:15:28,322 DEBUG > > > [storage.allocator.AbstractStoragePoolAllocator] > (Job-Executor-3:job-168) > > > Is localStorageAllocationNeeded? false > > > 2013-07-15 11:15:28,322 DEBUG > > > [storage.allocator.AbstractStoragePoolAllocator] > (Job-Executor-3:job-168) > > > Is storage pool shared? true > > > 2013-07-15 11:15:28,326 DEBUG [cloud.storage.StorageManagerImpl] > > > (Job-Executor-3:job-168) Checking pool 209 for storage, totalSize: > > > 6013522722816, usedBytes: 38283921137336466, usedPct: > 6366.305226067051, > > > disable threshold: 0.85 > > > 2013-07-15 11:15:28,326 DEBUG [cloud.storage.StorageManagerImpl] > > > (Job-Executor-3:job-168) Insufficient space on pool: 209 since its > usage > > > percentage: 6366.305226067051 has crossed the > > > pool.storage.capacity.disablethreshold: 0.85 > > > 2013-07-15 11:15:28,326 DEBUG > > > [storage.allocator.FirstFitStoragePoolAllocator] > (Job-Executor-3:job-168) > > > FirstFitStoragePoolAllocator returning 0 suitable storage pools > > > 2013-07-15 11:15:28,326 DEBUG [cloud.deploy.FirstFitPlanner] > > > (Job-Executor-3:job-168) No suitable pools found for volume: > > > Vol[228|vm=225|DATADISK] under cluster: 6 > > > 2013-07-15 11:15:28,326 DEBUG [cloud.deploy.FirstFitPlanner] > > > (Job-Executor-3:job-168) No suitable pools found > > > 2013-07-15 11:15:28,326 DEBUG [cloud.deploy.FirstFitPlanner] > > > (Job-Executor-3:job-168) No suitable storagePools found under this > Cluster: > > > 6 > > > 2013-07-15 11:15:28,326 DEBUG [cloud.deploy.FirstFitPlanner] > > > (Job-Executor-3:job-168) Could not find suitable Deployment > Destination for > > > this VM under any clusters, returning. > > > 2013-07-15 11:15:28,332 DEBUG [cloud.vm.UserVmManagerImpl] > > > (Job-Executor-3:job-168) Destroying vm VM[User|Indra-Test-3] as it > failed > > > to create on Host with Id:null > > > 2013-07-15 11:15:28,498 DEBUG [cloud.capacity.CapacityManagerImpl] > > > (Job-Executor-3:job-168) VM state transitted from :Stopped to Error > with > > > event: OperationFailedToErrorvm's original host id: null new host id: > null > > > host id before state transition: null > > > 2013-07-15 11:15:29,125 INFO [user.vm.DeployVMCmd] > > > (Job-Executor-3:job-168) > > > com.cloud.exception.InsufficientServerCapacityException: Unable to > create a > > > deployment for VM[User|Indra-Test-3]Scope=interface > > > com.cloud.dc.DataCenter; id=6 > > > === > > > > > > Anyone can advise on how to resolve this problem? > > > > > > Looking forward to your reply, thank you. > > > > > > Cheers. > > > > > -- > Prasanna., > > ------------------------ > Powered by BigRock.com > >
