Salut Pierre-Luc, It's been more than 24h and it is still at 189% primary storage allocated.
I can see them running in the log, but they don’t seem to recalculate the allocated space. storage.stats.interval=60000 (it is the default value) capacity.check.period=300000 (it is the default value) -----Original Message----- From: Pierre-Luc Dion [mailto:[email protected]] Sent: Friday, August 22, 2014 07:57 To: [email protected] Subject: Re: Primary Storage Allocated over 100% ?! Hi Francois, Could it be possible after a certain period metric inside cloudstack get updated to match available resources? I'm wondering if it could be because of global settings like this one: storage.stats.interval "The interval (in milliseconds) when storage stats (per host) are retrieved from agents." default=60000 Pierre-Luc On Fri, Aug 22, 2014 at 6:53 AM, François Bousquet < [email protected]> wrote: > That may explain how I get there, but do you have a suggestion how to > fix this issue if I can't find any volume big enough to justify all > that allocated space? > > The sum of all the volumes is much lower than my 189% (15TB/8TB) > primary allocated storage. > > I'm kind if stuck right now, not being able to fix the situation. > > > On Aug 22, 2014, at 0:33, "Prashant Kumar Mishra" < > [email protected]> wrote: > > > > Similar issue seen because of disk resize > https://issues.apache.org/jira/browse/CLOUDSTACK-6797 . > > > > Thanks > > prashant > > > > -----Original Message----- > > From: François Bousquet [mailto:[email protected]] > > Sent: Friday, August 22, 2014 7:27 AM > > To: [email protected] > > Subject: Primary Storage Allocated over 100% ?! > > > > Hello, > > > > We do have a strange problem with our Primary Storage Allocated. It > > is > now at 189%! (Storage is at 30%). So we cannot deploy anymore, because > of the pool.storage.allocated.capacity.disabledthreshold=0.85 > > > > CloudStack version is 4.2.1 > > Hypervisor is VMware ESXi 5.1 with 2 x 2 TB VMFS volume in cluster > > storage.overprovisioning.factor=2 > > > > If we do a list capacity, the capacitytotal for Primary Storage > Allocated (type=3) looks fine at 8 TB (8795019280384) but the > capacityused value totally doesn’t make sense. > > > > We don’t know how this happened or how to fix it. In order to prove > > my > point I did the sum of all volumes ready/destroy (we do not have > snapshots) and the total is a little more than 4 TB (see SQL query below). > > > > However, storage pools are reported with wrong disk used value on > storage_pool_view. Content of resource_count table makes more sense. > > > > It looks like our 2 primary storage pools disk_used_capacity are not > reporting the right value. VMware reports then 30% full (same as > CloudStack storage value). > > > > Do you have any idea why my 2 storage pool are not reporting > > allocated > disk properly? > > > > I tried restarting CloudStack management server with no success. > > > > > >> list capacity > > count = 9 > > capacity: > > > +----------------+---------------+-------------+--------------------------------------+------+--------------+ > > | capacityused | capacitytotal | percentused | zoneid > | type | zonename | > > > +----------------+---------------+-------------+--------------------------------------+------+--------------+ > > | 287158829056 | 927422582784 | 30.96 | > 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 0 | HypertecProd | > > | 106500 | 839904 | 12.68 | > 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 1 | HypertecProd | > > | 16702508511567 | 8795019280384 | 189.91 | > 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 3 | HypertecProd | > > | 2 | 231 | 0.87 | > 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 4 | HypertecProd | > > | 6 | 201 | 2.99 | > 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 5 | HypertecProd | > > | 0 | 100 | 0 | > 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 7 | HypertecProd | > > | 58 | 883 | 6.57 | > 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 8 | HypertecProd | > > | 61694476288 | 3248203235328 | 1.9 | > 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 6 | HypertecProd | > > | 1338303840256 | 4397509640192 | 30.43 | > 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | 2 | HypertecProd | > > > +----------------+---------------+-------------+--------------------------------------+------+--------------+ > > > > > > mysql> select state, sum(size) from volumes group by state; > > +----------+----------------+ > > | state | sum(size) | > > +----------+----------------+ > > | Destroy | 1030792151040 | > > | Expunged | 22460481142784 | > > | Ready | 4797277143040 | > > +----------+----------------+ > > 3 rows in set (0.00 sec) > > > > mysql> > > mysql> select * from storage_pool_view; > > > +----+--------------------------------------+--------------+--------+---------------------+-----------+-------------------------------------+---------------------+---------+----------------+---------------+-------+------------+------------+--------------+--------------+--------------+----------------+--------------------------------------+------------------+------------------+--------+----------+----------+--------------+--------------------+------------------------+--------+----------+------------+----------------+ > > | id | uuid | name | status | > path | pool_type | host_address | > created | removed | capacity_bytes | capacity_iops | scope | > hypervisor | cluster_id | cluster_uuid | cluster_name | cluster_type | > data_center_id | data_center_uuid | data_center_name | > data_center_type | pod_id | pod_uuid | pod_name | tag | > disk_used_capacity | disk_reserved_capacity | job_id | job_uuid | > job_status | job_account_id | > > > +----+--------------------------------------+--------------+--------+---------------------+-----------+-------------------------------------+---------------------+---------+----------------+---------------+-------+------------+------------+--------------+--------------+--------------+----------------+--------------------------------------+------------------+------------------+--------+----------+----------+--------------+--------------------+------------------------+--------+----------+------------+----------------+ > > | 1 | e1efbd9d-123b-350f-a99b-21fb4b6b38f1 | CLOUD-PROD01 | Up | > /CLOUD/CLOUD-PROD01 | VMFS | VMFS datastore: /CLOUD/CLOUD-PROD01 | > 2014-06-04 19:27:21 | NULL | 2198754820096 | NULL | ZONE | > VMware | NULL | NULL | NULL | NULL | > 1 | 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | HypertecProd | > Advanced | NULL | NULL | NULL | CLOUD-PROD01 | > 9036192478890 | 0 | NULL | NULL | NULL | > NULL | > > | 4 | 89506344-2c37-3eb0-8360-449a923a12c8 | CLOUD-PROD02 | Up | > /CLOUD/CLOUD-PROD02 | VMFS | VMFS datastore: /CLOUD/CLOUD-PROD02 | > 2014-06-10 01:51:01 | NULL | 2198754820096 | NULL | ZONE | > VMware | NULL | NULL | NULL | NULL | > 1 | 83b05505-08ca-4fb9-be3c-9be2cb4a8388 | HypertecProd | > Advanced | NULL | NULL | NULL | CLOUD-PROD02 | > 7666316032677 | 0 | NULL | NULL | NULL | > NULL | > > > +----+--------------------------------------+--------------+--------+---------------------+-----------+-------------------------------------+---------------------+---------+----------------+---------------+-------+------------+------------+--------------+--------------+--------------+----------------+--------------------------------------+------------------+------------------+--------+----------+----------+--------------+--------------------+------------------------+--------+----------+------------+----------------+ > > 2 rows in set (0.00 sec) > > > > mysql> select * from resource_count where type='primary_storage'; > > +-----+------------+-----------+-----------------+---------------+ > > | id | account_id | domain_id | type | count | > > +-----+------------+-----------+-----------------+---------------+ > > | 7 | NULL | 1 | primary_storage | 5817583534080 | > > | 67 | 1 | NULL | primary_storage | 0 | > > | 103 | 2 | NULL | primary_storage | -2097152000 | > > | 139 | 4 | NULL | primary_storage | 214748364800 | > > | 175 | 7 | NULL | primary_storage | 2491081031680 | > > | 211 | 10 | NULL | primary_storage | 2093796556800 | > > | 247 | 13 | NULL | primary_storage | 1020054732800 | > > +-----+------------+-----------+-----------------+---------------+ > > 7 rows in set (0.00 sec) > > > > Thanks > > > > > > François Bousquet > > NOC Team Lead > > Office: 514-286-2636 #2337 > > Mobile: 514-654-8920 > > [email protected] > > > > > > > > > > ******************************************************************** > > *** > This e-mail and attachments are confidential, legally privileged, may > be subject to copyright and sent solely for the attention of the addressee(s). > Any unauthorized use or disclosure is prohibited. Statements and > opinions expressed in this e-mail may not represent those of Radialpoint. > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Le contenu du > présent courriel est confidentiel, privilégié et peut être soumis à > des droits d'auteur. Il est envoyé à l'intention exclusive de son ou > de ses destinataires. Il est interdit de l'utiliser ou de le divulguer > sans autorisation. Les opinions exprimées dans le présent courriel > peuvent diverger de celles de Radialpoint. > ********************************************************************** > * This e-mail and attachments are confidential, legally privileged, > may be subject to copyright and sent solely for the attention of the > addressee(s). > Any unauthorized use or disclosure is prohibited. Statements and > opinions expressed in this e-mail may not represent those of Radialpoint. > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Le contenu du > présent courriel est confidentiel, privilégié et peut être soumis à > des droits d'auteur. Il est envoyé à l'intention exclusive de son ou > de ses destinataires. Il est interdit de l'utiliser ou de le divulguer > sans autorisation. Les opinions exprimées dans le présent courriel > peuvent diverger de celles de Radialpoint. > *********************************************************************** This e-mail and attachments are confidential, legally privileged, may be subject to copyright and sent solely for the attention of the addressee(s). Any unauthorized use or disclosure is prohibited. Statements and opinions expressed in this e-mail may not represent those of Radialpoint. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Le contenu du présent courriel est confidentiel, privilégié et peut être soumis à des droits d'auteur. Il est envoyé à l'intention exclusive de son ou de ses destinataires. Il est interdit de l'utiliser ou de le divulguer sans autorisation. Les opinions exprimées dans le présent courriel peuvent diverger de celles de Radialpoint.
