Hi all,
in the meantime we have found out that obviously not only the usage
records are missing for migrated volumes but
even the event records. It looks as if a volume is migrated, for the new
volume record/id no events are written to
the database.
Example, a new volume record after migration:
mysql> select * from cloud.volumes where id=1213\G
*************************** 1. row ***************************
* id: 1213*
account_id: 78
domain_id: 1
pool_id: 8
last_pool_id: 5
instance_id: 581
device_id: 1
name: xxxxx
* uuid: a1a3668b-14cf-490e-bcbe-ef0d4f225574*
size: 214748364800
folder: cloudstack01
path: e886bea4-4c94-43a3-9b17-e712c130f1a7
pod_id: 2
data_center_id: 2
iscsi_name: NULL
host_ip: NULL
volume_type: DATADISK
pool_type: NULL
disk_offering_id: 6
template_id: NULL
first_snapshot_backup_uuid: NULL
recreatable: 0
created: 2015-06-10 06:45:53
attached: 2014-09-24 07:29:45
updated: 2015-06-10 08:53:12
removed: NULL
* state: Ready*
chain_info: NULL
update_count: 2
disk_type: NULL
vm_snapshot_chain_size: NULL
iso_id: NULL
display_volume: 1
format: RAW
min_iops: NULL
max_iops: NULL
hv_ss_reserve: NULL
1 row in set (0.00 sec)
==> No events in the database:
mysql> select * from cloud.event where description like '% *1213* %'\G
Empty set (0.00 sec)
Thus no usage records:
mysql> select * from cloud_usage.usage_event where resource_id=*1213*\G
Empty set (0.00 sec)
mysql> select * from cloud.usage_event where resource_id=*1213*\G
Empty set (0.01 sec)
----------------------------------------------
Example, volume record before migration (has now NULL for uuid):
mysql> select * from cloud.volumes where id=747\G
*************************** 1. row ***************************
* id: 747*
account_id: 78
domain_id: 1
pool_id: 5
last_pool_id: NULL
instance_id: 581
device_id: 1
name: xxxxxx
* uuid: NULL*
size: 214748364800
folder: NULL
path: f303f233-29ac-4b3f-9524-16c7e901dbdc
pod_id: NULL
data_center_id: 2
iscsi_name: NULL
host_ip: NULL
volume_type: DATADISK
pool_type: NULL
disk_offering_id: 6
template_id: NULL
first_snapshot_backup_uuid: NULL
recreatable: 0
created: 2014-08-27 06:49:25
attached: 2014-09-24 07:29:45
updated: 2015-06-10 08:53:15
removed: 2015-06-10 08:53:15
* state: Expunged*
chain_info: NULL
update_count: 8
disk_type: NULL
vm_snapshot_chain_size: NULL
iso_id: NULL
display_volume: 1
format: QCOW2
min_iops: NULL
max_iops: NULL
hv_ss_reserve: NULL
1 row in set (0.00 sec)
Events exist:
mysql> select * from cloud.event where description like '% *747* %'\G
*************************** 1. row ***************************
id: 24165
uuid: 6669a7b2-ab27-4fdb-b21c-5e6e0c2df003
type: VOLUME.MIGRATE
state: Scheduled
description: Attempting to migrate volume Id: *747* to storage pool Id: 8
user_id: 60
account_id: 78
domain_id: 9
created: 2015-06-10 06:45:51
level: INFO
start_id: 0
parameters: NULL
archived: 0
display: 1
1 row in set (0.00 sec)
And thus usage records exist:
mysql> select * from cloud_usage.usage_event where resource_id=*747*\G
*************************** 1. row ***************************
id: 3449
type: VOLUME.CREATE
account_id: 78
created: 2014-08-27 06:49:25
zone_id: 2
resource_id: *747*
resource_name: bts014-data01
offering_id: 6
template_id: NULL
size: 214748364800
resource_type: NULL
processed: 1
virtual_size: NULL
1 row in set (0.01 sec)
If some knows where and how to debug this, any ideas are very welcome.
thx
Norbert
Am 24.06.2015 um 17:16 schrieb Norbert Klein:
> Hi Praveen,
>
> we have come a little bit further with our research.
> To answer your questions, I have investigated this case for data
> volumes so far.
>
> The usage records which don't have the usageid point to (old) volume
> table records of migrated volumes (via the volume id in the
> description field).
> These records have NULL in their uuid and the state is Expunged. It
> looks like CS creates a new record for a mirgrated volume and
> keeps the old one but sets uuid to NULL. Unfortunately there are no
> new usage records for these
> new volume records. This leads to the situation that volumes cannot be
> charged any more after migration.
>
> Here is an example:
>
> I copied from the CS web page a virtualmachine name, a data volume
> name and the uuid of the volume.
>
> In the database I find two volumes with this name, one has NULL for
> uuid, the other one has a valid uuid:
>
> mysql> select * from cloud.volumes where name like 'xxxxx-data01'\G
> *************************** 1. row ***************************
> *id: 747*
> account_id: 78
> domain_id: 1
> pool_id: 5
> last_pool_id: NULL
> instance_id: 581
> device_id: 1
> name: xxxxx-data01
> * uuid: NULL*
> size: 214748364800
> folder: NULL
> path: f303f233-29ac-4b3f-9524-16c7e901dbdc
> pod_id: NULL
> data_center_id: 2
> iscsi_name: NULL
> host_ip: NULL
> volume_type: DATADISK
> pool_type: NULL
> disk_offering_id: 6
> template_id: NULL
> first_snapshot_backup_uuid: NULL
> recreatable: 0
> created: 2014-08-27 06:49:25
> attached: 2014-09-24 07:29:45
> updated: 2015-06-10 08:53:15
> removed: 2015-06-10 08:53:15
> state: Expunged
> chain_info: NULL
> update_count: 8
> disk_type: NULL
> vm_snapshot_chain_size: NULL
> iso_id: NULL
> display_volume: 1
> format: QCOW2
> min_iops: NULL
> max_iops: NULL
> hv_ss_reserve: NULL
> *************************** 2. row ***************************
> *id: 1213*
> account_id: 78
> domain_id: 1
> pool_id: 8
> last_pool_id: 5
> instance_id: 581
> device_id: 1
> name: xxxxx-data01
> * uuid: a1a3668b-14cf-490e-bcbe-ef0d4f225574*
> size: 214748364800
> folder: cloudstack01
> path: e886bea4-4c94-43a3-9b17-e712c130f1a7
> pod_id: 2
> data_center_id: 2
> iscsi_name: NULL
> host_ip: NULL
> volume_type: DATADISK
> pool_type: NULL
> disk_offering_id: 6
> template_id: NULL
> first_snapshot_backup_uuid: NULL
> recreatable: 0
> created: 2015-06-10 06:45:53
> attached: 2014-09-24 07:29:45
> updated: 2015-06-10 08:53:12
> removed: NULL
> state: Ready
> chain_info: NULL
> update_count: 2
> disk_type: NULL
> vm_snapshot_chain_size: NULL
> iso_id: NULL
> display_volume: 1
> format: RAW
> min_iops: NULL
> max_iops: NULL
> hv_ss_reserve: NULL
> 2 rows in set (0.00 sec)
>
>
> If I use the table ID of the first record (747) and search for usage
> records I find all old records and all miss the usageid:
>
> 🐵 > list usagerecords type=6
> projectid=4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e startdate=2015-06-01
> enddate=2015-06-24 | grep -A 10 'Volume Id: *747*'
> description = Volume Id: 747 usage time (DiskOffering: 6)
> domainid = 42541338-8704-4a6d-8b45-7f7eb2996e06
> enddate = 2015-06-23'T'23:59:59+00:00
> offeringid = 1b618634-73cf-4a95-99ba-1f6e536a6051
> project = xxxxx
> projectid = 4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e
> rawusage = 24
> size = 214748364800
> startdate = 2015-06-23'T'00:00:00+00:00
> usage = 24 Hrs
> usagetype = 6
> --
> description = Volume Id: 747 usage time (DiskOffering: 6)
> domainid = 42541338-8704-4a6d-8b45-7f7eb2996e06
> enddate = 2015-06-22'T'23:59:59+00:00
> offeringid = 1b618634-73cf-4a95-99ba-1f6e536a6051
> project = xxxxx
> projectid = 4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e
> rawusage = 24
> size = 214748364800
> startdate = 2015-06-22'T'00:00:00+00:00
> usage = 24 Hrs
> usagetype = 6
> --
> description = Volume Id: 747 usage time (DiskOffering: 6)
> domainid = 42541338-8704-4a6d-8b45-7f7eb2996e06
> enddate = 2015-06-21'T'23:59:59+00:00
> offeringid = 1b618634-73cf-4a95-99ba-1f6e536a6051
> project = xxxxx
> projectid = 4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e
> rawusage = 24
> size = 214748364800
> startdate = 2015-06-21'T'00:00:00+00:00
> usage = 24 Hrs
> usagetype = 6
> --
> description = Volume Id: 747 usage time (DiskOffering: 6)
> domainid = 42541338-8704-4a6d-8b45-7f7eb2996e06
> enddate = 2015-06-20'T'23:59:59+00:00
> offeringid = 1b618634-73cf-4a95-99ba-1f6e536a6051
> project = xxxxx
> projectid = 4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e
> rawusage = 24
> size = 214748364800
> startdate = 2015-06-20'T'00:00:00+00:00
> usage = 24 Hrs
> usagetype = 6
> --
> description = Volume Id: 747 usage time (DiskOffering: 6)
> domainid = 42541338-8704-4a6d-8b45-7f7eb2996e06
> enddate = 2015-06-19'T'23:59:59+00:00
> offeringid = 1b618634-73cf-4a95-99ba-1f6e536a6051
> project = xxxxx
> projectid = 4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e
> rawusage = 24
> size = 214748364800
> startdate = 2015-06-19'T'00:00:00+00:00
> usage = 24 Hrs
> usagetype = 6
> --
> description = Volume Id: 747 usage time (DiskOffering: 6)
> domainid = 42541338-8704-4a6d-8b45-7f7eb2996e06
> enddate = 2015-06-18'T'23:59:59+00:00
> offeringid = 1b618634-73cf-4a95-99ba-1f6e536a6051
> project = xxxxx
> projectid = 4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e
> rawusage = 24
> size = 214748364800
> startdate = 2015-06-18'T'00:00:00+00:00
> usage = 24 Hrs
> usagetype = 6
> --
> description = Volume Id: 747 usage time (DiskOffering: 6)
> domainid = 42541338-8704-4a6d-8b45-7f7eb2996e06
> enddate = 2015-06-17'T'23:59:59+00:00
> offeringid = 1b618634-73cf-4a95-99ba-1f6e536a6051
> project = xxxxx
> projectid = 4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e
> rawusage = 24
> size = 214748364800
> startdate = 2015-06-17'T'00:00:00+00:00
> usage = 24 Hrs
> usagetype = 6
> --
> description = Volume Id: 747 usage time (DiskOffering: 6)
> domainid = 42541338-8704-4a6d-8b45-7f7eb2996e06
> enddate = 2015-06-16'T'23:59:59+00:00
> offeringid = 1b618634-73cf-4a95-99ba-1f6e536a6051
> project = xxxxx
> projectid = 4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e
> rawusage = 24
> size = 214748364800
> startdate = 2015-06-16'T'00:00:00+00:00
> usage = 24 Hrs
> usagetype = 6
> --
> description = Volume Id: 747 usage time (DiskOffering: 6)
> domainid = 42541338-8704-4a6d-8b45-7f7eb2996e06
> enddate = 2015-06-15'T'23:59:59+00:00
> offeringid = 1b618634-73cf-4a95-99ba-1f6e536a6051
> project = xxxxx
> projectid = 4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e
> rawusage = 24
> size = 214748364800
> startdate = 2015-06-15'T'00:00:00+00:00
> usage = 24 Hrs
> usagetype = 6
>
>
> Now the real problem comes:
>
> We have to charge the migrated volumes of course and therefor I
> searched for the corresponding usage records. First
> I tried to find the id of the table record (*1213*) and then I tried
> to find the uuid. Both have no result, so billing is not possible:
> 🐵 > list usagerecords type=6
> projectid=4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e startdate=2015-06-01
> enddate=2015-06-24 | grep -A 10 'Volume Id: *1213*'
> 🐵 > list usagerecords type=6
> projectid=4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e startdate=2015-06-01
> enddate=2015-06-24 | grep 'a1a3668b-14cf-490e-bcbe-ef0d4f225574'
> 🐵 >
>
> But this volume is available through the API nevertheless as long as
> you use the volume name:
> 🐵 > list volumes listall=true
> projectid=4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e name=xxxxx-data01
> count = 1
> volume:
> id = a1a3668b-14cf-490e-bcbe-ef0d4f225574
> name = xxxxx-data01
> attached = 2014-09-24T09:29:45+0200
> created = 2015-06-10T08:45:53+0200
> destroyed = False
> deviceid = 1
> diskofferingdisplaytext = Custom Disk
> diskofferingid = 1b618634-73cf-4a95-99ba-1f6e536a6051
> diskofferingname = Custom
> displayvolume = True
> domain = ROOT
> domainid = a7b69499-a912-11e3-a7c1-5254005b13af
> hypervisor = KVM
> isextractable = True
> path = e886bea4-4c94-43a3-9b17-e712c130f1a7
> project = xxxxx
> projectid = 4865e6c6-b1ff-48ef-b908-1f5fde7d7b6e
> quiescevm = False
> size = 214748364800
> state = Ready
> storage = ceph-cloudstack01
> storageid = ccfacebe-19c3-38f4-a161-2bd9e4ac70a9
> storagetype = shared
> tags:
> type = DATADISK
> virtualmachineid = 2360e425-1171-4ad7-a6b3-eec403486b19
> vmdisplayname = xxxxx
> vmname = xxxxx
> vmstate = Running
> zoneid = 10992277-e637-4795-ab02-be57cca0c66a
> zonename = EU-GER-01
>
>
> How can we get usage records of migrated volumes?
> If this is a database inconsistancy, or even a bug, is there a way to
> repair it?
>
> thx
> Norbert
>
>
> -----Ursprüngliche Nachricht-----
> Von: Praveen B [mailto:[email protected]]
> Gesendet: Dienstag, 23. Juni 2015 08:53
> An: [email protected]
> Betreff: Re: usageid missing
>
> Hi Ingo,
>
> We haven't seen this type of issue anytime before. Need to analyze
> more on the issue.
>
> Can you upload output of usage records somewhere and provide me the
> URL link to access it?
>
> There could be multiple volumes in CloudStack
> 1) Root volumes 2) Data Volumes 3) Volumes created from Snapshots
> 4) Uploaded volumes...etc.
>
> You need to find out if there is any specific pattern for the missing
> usage id volumes. That will help us to understand the issue behaviour.
>
> Thaks,
> Praveen Kumar,
> Founder,PSI Cloud Solutions,
> psiclouds.com
>
> On Tue, Jun 23, 2015 at 2:51 AM, Jochim, Ingo <[email protected]>
> wrote:
>
> > I'm using the usagerecords to get usage of volumes. Unfortunately for
> > most of my usagerecords of volumes the usageid is missing.
> > For those I have a usageid I can use this to the the volume (it's the
> > same as the volume id)
> >
> > Why are there so many usageid's missing?
> >
> > We're running 4.4.
> >
> > Thanks and regards,
> > Ingo
> >
> > --
> > This email was Virus checked by UTM 9. http://www.sophos.com
> >
>
> --
> This email was Virus checked by UTM 9. http://www.sophos.com
>
--
www.infosecprojects.net <http://www.infosecprojects.net/>
