Hi Francois, Could it be possible after a certain period metric inside cloudstack get updated to match available resources? I'm wondering if it could be because of global settings like this one:
storage.stats.interval "The interval (in milliseconds) when storage stats (per host) are retrieved from agents." default=60000 Pierre-Luc On Fri, Aug 22, 2014 at 6:53 AM, François Bousquet < [email protected]> wrote: > That may explain how I get there, but do you have a suggestion how to fix > this issue if I can't find any volume big enough to justify all that > allocated space? > > The sum of all the volumes is much lower than my 189% (15TB/8TB) primary > allocated storage. > > I'm kind if stuck right now, not being able to fix the situation. > > > On Aug 22, 2014, at 0:33, "Prashant Kumar Mishra" < > [email protected]> wrote: > > > > Similar issue seen because of disk resize > https://issues.apache.org/jira/browse/CLOUDSTACK-6797 . > > > > Thanks > > prashant > > > > -----Original Message----- > > From: François Bousquet [mailto:[email protected]] > > Sent: Friday, August 22, 2014 7:27 AM > > To: [email protected] > > Subject: Primary Storage Allocated over 100% ?! > > > > Hello, > > > > We do have a strange problem with our Primary Storage Allocated. It is > now at 189%! (Storage is at 30%). So we cannot deploy anymore, because of > the pool.storage.allocated.capacity.disabledthreshold=0.85 > > > > CloudStack version is 4.2.1 > > Hypervisor is VMware ESXi 5.1 with 2 x 2 TB VMFS volume in cluster > > storage.overprovisioning.factor=2 > > > > If we do a list capacity, the capacitytotal for Primary Storage > Allocated (type=3) looks fine at 8 TB (8795019280384) but the capacityused > value totally doesn’t make sense. > > > > We don’t know how this happened or how to fix it. In order to prove my > point I did the sum of all volumes ready/destroy (we do not have snapshots) > and the total is a little more than 4 TB (see SQL query below). > > > > However, storage pools are reported with wrong disk used value on > storage_pool_view. Content of resource_count table makes more sense. > > > > It looks like our 2 primary storage pools disk_used_capacity are not > reporting the right value. VMware reports then 30% full (same as CloudStack > storage value). > > > > Do you have any idea why my 2 storage pool are not reporting allocated > disk properly? > > > > I tried restarting CloudStack management server with no success. > > > > > >> list capacity > > count = 9 > > capacity: > > > +----------------+---------------+-------------+--------------------------------------+------+--------------+ > > | capacityused | capacitytotal | percentused | zoneid > | type | zonename | > > > +----------------+---------------+-------------+--------------------------------------+------+--------------+ > > | 287158829056 | 927422582784 | 30.96 | > 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 0 | HypertecProd | > > | 106500 | 839904 | 12.68 | > 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 1 | HypertecProd | > > | 16702508511567 | 8795019280384 | 189.91 | > 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 3 | HypertecProd | > > | 2 | 231 | 0.87 | > 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 4 | HypertecProd | > > | 6 | 201 | 2.99 | > 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 5 | HypertecProd | > > | 0 | 100 | 0 | > 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 7 | HypertecProd | > > | 58 | 883 | 6.57 | > 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 8 | HypertecProd | > > | 61694476288 | 3248203235328 | 1.9 | > 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 6 | HypertecProd | > > | 1338303840256 | 4397509640192 | 30.43 | > 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 2 | HypertecProd | > > > +----------------+---------------+-------------+--------------------------------------+------+--------------+ > > > > > > mysql> select state, sum(size) from volumes group by state; > > +----------+----------------+ > > | state | sum(size) | > > +----------+----------------+ > > | Destroy | 1030792151040 | > > | Expunged | 22460481142784 | > > | Ready | 4797277143040 | > > +----------+----------------+ > > 3 rows in set (0.00 sec) > > > > mysql> > > mysql> select * from storage_pool_view; > > > +----+--------------------------------------+--------------+--------+---------------------+-----------+-------------------------------------+---------------------+---------+----------------+---------------+-------+------------+------------+--------------+--------------+--------------+----------------+--------------------------------------+------------------+------------------+--------+----------+----------+--------------+--------------------+------------------------+--------+----------+------------+----------------+ > > | id | uuid | name | status | > path | pool_type | host_address | > created | removed | capacity_bytes | capacity_iops | scope | > hypervisor | cluster_id | cluster_uuid | cluster_name | cluster_type | > data_center_id | data_center_uuid | data_center_name | > data_center_type | pod_id | pod_uuid | pod_name | tag | > disk_used_capacity | disk_reserved_capacity | job_id | job_uuid | > job_status | job_account_id | > > > +----+--------------------------------------+--------------+--------+---------------------+-----------+-------------------------------------+---------------------+---------+----------------+---------------+-------+------------+------------+--------------+--------------+--------------+----------------+--------------------------------------+------------------+------------------+--------+----------+----------+--------------+--------------------+------------------------+--------+----------+------------+----------------+ > > | 1 | e1efbd9d-123b-350f-a99b-21fb4b6b38f1 | CLOUD-PROD01 | Up | > /CLOUD/CLOUD-PROD01 | VMFS | VMFS datastore: /CLOUD/CLOUD-PROD01 | > 2014-06-04 19:27:21 | NULL | 2198754820096 | NULL | ZONE | > VMware | NULL | NULL | NULL | NULL | > 1 | 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | HypertecProd | > Advanced | NULL | NULL | NULL | CLOUD-PROD01 | > 9036192478890 | 0 | NULL | NULL | NULL | > NULL | > > | 4 | 89506344-2c37-3eb0-8360-449a923a12c8 | CLOUD-PROD02 | Up | > /CLOUD/CLOUD-PROD02 | VMFS | VMFS datastore: /CLOUD/CLOUD-PROD02 | > 2014-06-10 01:51:01 | NULL | 2198754820096 | NULL | ZONE | > VMware | NULL | NULL | NULL | NULL | > 1 | 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | HypertecProd | > Advanced | NULL | NULL | NULL | CLOUD-PROD02 | > 7666316032677 | 0 | NULL | NULL | NULL | > NULL | > > > +----+--------------------------------------+--------------+--------+---------------------+-----------+-------------------------------------+---------------------+---------+----------------+---------------+-------+------------+------------+--------------+--------------+--------------+----------------+--------------------------------------+------------------+------------------+--------+----------+----------+--------------+--------------------+------------------------+--------+----------+------------+----------------+ > > 2 rows in set (0.00 sec) > > > > mysql> select * from resource_count where type='primary_storage'; > > +-----+------------+-----------+-----------------+---------------+ > > | id | account_id | domain_id | type | count | > > +-----+------------+-----------+-----------------+---------------+ > > | 7 | NULL | 1 | primary_storage | 5817583534080 | > > | 67 | 1 | NULL | primary_storage | 0 | > > | 103 | 2 | NULL | primary_storage | -2097152000 | > > | 139 | 4 | NULL | primary_storage | 214748364800 | > > | 175 | 7 | NULL | primary_storage | 2491081031680 | > > | 211 | 10 | NULL | primary_storage | 2093796556800 | > > | 247 | 13 | NULL | primary_storage | 1020054732800 | > > +-----+------------+-----------+-----------------+---------------+ > > 7 rows in set (0.00 sec) > > > > Thanks > > > > > > François Bousquet > > NOC Team Lead > > Office: 514-286-2636 #2337 > > Mobile: 514-654-8920 > > [email protected] > > > > > > > > > > *********************************************************************** > This e-mail and attachments are confidential, legally privileged, may be > subject to copyright and sent solely for the attention of the addressee(s). > Any unauthorized use or disclosure is prohibited. Statements and opinions > expressed in this e-mail may not represent those of Radialpoint. > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Le contenu du présent > courriel est confidentiel, privilégié et peut être soumis à des droits > d'auteur. Il est envoyé à l'intention exclusive de son ou de ses > destinataires. Il est interdit de l'utiliser ou de le divulguer sans > autorisation. Les opinions exprimées dans le présent courriel peuvent > diverger de celles de Radialpoint. > *********************************************************************** > This e-mail and attachments are confidential, legally privileged, may be > subject to copyright and sent solely for the attention of the addressee(s). > Any unauthorized use or disclosure is prohibited. Statements and opinions > expressed in this e-mail may not represent those of Radialpoint. > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Le contenu du présent > courriel est confidentiel, privilégié et peut être soumis à des droits > d'auteur. Il est envoyé à l'intention exclusive de son ou de ses > destinataires. Il est interdit de l'utiliser ou de le divulguer sans > autorisation. Les opinions exprimées dans le présent courriel peuvent > diverger de celles de Radialpoint. >
