Hello,
We do have a strange problem with our Primary Storage Allocated. It is now at
189%! (Storage is at 30%). So we cannot deploy anymore, because of the
pool.storage.allocated.capacity.disabledthreshold=0.85
CloudStack version is 4.2.1
Hypervisor is VMware ESXi 5.1 with 2 x 2 TB VMFS volume in cluster
storage.overprovisioning.factor=2
If we do a list capacity, the capacitytotal for Primary Storage Allocated
(type=3) looks fine at 8 TB (8795019280384) but the capacityused value totally
doesn’t make sense.
We don’t know how this happened or how to fix it. In order to prove my point I
did the sum of all volumes ready/destroy (we do not have snapshots) and the
total is a little more than 4 TB (see SQL query below).
However, storage pools are reported with wrong disk used value on
storage_pool_view. Content of resource_count table makes more sense.
It looks like our 2 primary storage pools disk_used_capacity are not reporting
the right value. VMware reports then 30% full (same as CloudStack storage
value).
Do you have any idea why my 2 storage pool are not reporting allocated disk
properly?
I tried restarting CloudStack management server with no success.
> list capacity
count = 9
capacity:
+----------------+---------------+-------------+--------------------------------------+------+--------------+
| capacityused | capacitytotal | percentused | zoneid
| type | zonename |
+----------------+---------------+-------------+--------------------------------------+------+--------------+
| 287158829056 | 927422582784 | 30.96 |
83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 0 | HypertecProd |
| 106500 | 839904 | 12.68 |
83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 1 | HypertecProd |
| 16702508511567 | 8795019280384 | 189.91 |
83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 3 | HypertecProd |
| 2 | 231 | 0.87 |
83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 4 | HypertecProd |
| 6 | 201 | 2.99 |
83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 5 | HypertecProd |
| 0 | 100 | 0 |
83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 7 | HypertecProd |
| 58 | 883 | 6.57 |
83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 8 | HypertecProd |
| 61694476288 | 3248203235328 | 1.9 |
83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 6 | HypertecProd |
| 1338303840256 | 4397509640192 | 30.43 |
83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 2 | HypertecProd |
+----------------+---------------+-------------+--------------------------------------+------+--------------+
>
mysql> select state, sum(size) from volumes group by state;
+----------+----------------+
| state | sum(size) |
+----------+----------------+
| Destroy | 1030792151040 |
| Expunged | 22460481142784 |
| Ready | 4797277143040 |
+----------+----------------+
3 rows in set (0.00 sec)
mysql>
mysql> select * from storage_pool_view;
+----+--------------------------------------+--------------+--------+---------------------+-----------+-------------------------------------+---------------------+---------+----------------+---------------+-------+------------+------------+--------------+--------------+--------------+----------------+--------------------------------------+------------------+------------------+--------+----------+----------+--------------+--------------------+------------------------+--------+----------+------------+----------------+
| id | uuid | name | status | path
| pool_type | host_address | created
| removed | capacity_bytes | capacity_iops | scope | hypervisor | cluster_id
| cluster_uuid | cluster_name | cluster_type | data_center_id |
data_center_uuid | data_center_name | data_center_type |
pod_id | pod_uuid | pod_name | tag | disk_used_capacity |
disk_reserved_capacity | job_id | job_uuid | job_status | job_account_id |
+----+--------------------------------------+--------------+--------+---------------------+-----------+-------------------------------------+---------------------+---------+----------------+---------------+-------+------------+------------+--------------+--------------+--------------+----------------+--------------------------------------+------------------+------------------+--------+----------+----------+--------------+--------------------+------------------------+--------+----------+------------+----------------+
| 1 | e1efbd9d-123b-350f-a99b-21fb4b6b38f1 | CLOUD-PROD01 | Up |
/CLOUD/CLOUD-PROD01 | VMFS | VMFS datastore: /CLOUD/CLOUD-PROD01 |
2014-06-04 19:27:21 | NULL | 2198754820096 | NULL | ZONE | VMware
| NULL | NULL | NULL | NULL | 1
| 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | HypertecProd | Advanced |
NULL | NULL | NULL | CLOUD-PROD01 | 9036192478890 |
0 | NULL | NULL | NULL | NULL |
| 4 | 89506344-2c37-3eb0-8360-449a923a12c8 | CLOUD-PROD02 | Up |
/CLOUD/CLOUD-PROD02 | VMFS | VMFS datastore: /CLOUD/CLOUD-PROD02 |
2014-06-10 01:51:01 | NULL | 2198754820096 | NULL | ZONE | VMware
| NULL | NULL | NULL | NULL | 1
| 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | HypertecProd | Advanced |
NULL | NULL | NULL | CLOUD-PROD02 | 7666316032677 |
0 | NULL | NULL | NULL | NULL |
+----+--------------------------------------+--------------+--------+---------------------+-----------+-------------------------------------+---------------------+---------+----------------+---------------+-------+------------+------------+--------------+--------------+--------------+----------------+--------------------------------------+------------------+------------------+--------+----------+----------+--------------+--------------------+------------------------+--------+----------+------------+----------------+
2 rows in set (0.00 sec)
mysql> select * from resource_count where type='primary_storage';
+-----+------------+-----------+-----------------+---------------+
| id | account_id | domain_id | type | count |
+-----+------------+-----------+-----------------+---------------+
| 7 | NULL | 1 | primary_storage | 5817583534080 |
| 67 | 1 | NULL | primary_storage | 0 |
| 103 | 2 | NULL | primary_storage | -2097152000 |
| 139 | 4 | NULL | primary_storage | 214748364800 |
| 175 | 7 | NULL | primary_storage | 2491081031680 |
| 211 | 10 | NULL | primary_storage | 2093796556800 |
| 247 | 13 | NULL | primary_storage | 1020054732800 |
+-----+------------+-----------+-----------------+---------------+
7 rows in set (0.00 sec)
Thanks
François Bousquet
NOC Team Lead
Office: 514-286-2636 #2337
Mobile: 514-654-8920
[email protected]
*********************************************************************** This
e-mail and attachments are confidential, legally privileged, may be subject to
copyright and sent solely for the attention of the addressee(s). Any
unauthorized use or disclosure is prohibited. Statements and opinions expressed
in this e-mail may not represent those of Radialpoint.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Le contenu du présent
courriel est confidentiel, privilégié et peut être soumis à des droits
d'auteur. Il est envoyé à l'intention exclusive de son ou de ses destinataires.
Il est interdit de l'utiliser ou de le divulguer sans autorisation. Les
opinions exprimées dans le présent courriel peuvent diverger de celles de
Radialpoint.
