Hean, If your external devices support retrieving statistics data you could either do it directly of write a cloudstack extension for it. If not than you would have to set a router in between that does support it. regards,
On Wed, Jun 30, 2021 at 7:00 AM Hean Seng <heans...@gmail.com> wrote: > Hi Andrija, > > I enabled the value , vm.network.stats.interval to > 0 where it will > calculate the traffic from the network port assigned ( which i > suppose include local and external) , I am fine for this ( network usage > = lan + wan traffic), which similar to previous VPS data usage calculation > before migrated to Cloudstack . > > However , it seems not all calculated accurately , this especially > after the issue happened. > > For external devices, you mention, can you recommend software that may be > put into consideration as an alternative . > > > > > > > On Wed, Jun 30, 2021 at 4:06 AM Andrija Panic <andrija.pa...@gmail.com> > wrote: > > > stats are pulled from VR in case traffic passes THORUGH those VRs. > > In shared networks, traffic does NOT passes THROUGH the VR, but through > > some external device - thus no statistics is expected to appear - all > good, > > no problem. > > > > On Fri, 25 Jun 2021 at 17:13, Hean Seng <heans...@gmail.com> wrote: > > > > > Hi > > > > > > yes, there are traffic running at the interface . I not sure the logic > > > behind on calculating the data, if not would be easy to troubleshoot. > > > > > > > > > > > > > > > On Fri, Jun 25, 2021 at 10:54 PM Daan Hoogland < > daan.hoogl...@gmail.com> > > > wrote: > > > > > > > if you look at the hypervisors, do you see traffic data for the nics > > > there? > > > > > > > > On Fri, Jun 25, 2021 at 4:32 PM Hean Seng <heans...@gmail.com> > wrote: > > > > > > > > > Yes, thats right, I am Advance zone with Security Group, and > > Shared > > > > > Network with Public IP range, However, I had change value of > > > > > vm.network.stats.interval. > > > > > so that calculate the Lan traffic of the Network Interface . So it > > > should > > > > > work for calculating all traffic incur in the network interface . > > But > > > > > seems not, about 50% of it not calculating. > > > > > > > > > > > > > > > > > > > > On Fri, Jun 25, 2021 at 9:51 PM Daan Hoogland < > > daan.hoogl...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > The only thing I can think of is that the VM is not on a network > > that > > > > > > provides routing. for instance in a basic zone or on a shared > > network > > > > > with > > > > > > a public ip range. > > > > > > > > > > > > On Fri, Jun 25, 2021 at 12:23 PM Hean Seng <heans...@gmail.com> > > > wrote: > > > > > > > > > > > > > HI Daan > > > > > > > > > > > > > > > > > > > > > My usage record have this , this is one of the example that do > > not > > > > > have. > > > > > > > record of type 4 and 5, but have type 1, > > > > > > > > > > > > > > select * from usage_event where account_id=3391; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------+-------------------------+------------+---------------------+---------+-------------+--------------------+-------------+-------------+-------------+----------------+-----------+--------------+ > > > > > > > > > > > > > > | id | type | account_id | created > > > > | > > > > > > > zone_id | resource_id | resource_name | offering_id | > > > > template_id > > > > > | > > > > > > > size | resource_type | processed | virtual_size | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +-------+-------------------------+------------+---------------------+---------+-------------+--------------------+-------------+-------------+-------------+----------------+-----------+--------------+ > > > > > > > > > > > > > > | 75910 | NET.IPASSIGN | 3391 | 2021-05-06 > > > 10:16:55 > > > > | > > > > > > > 1 | 2396 | 11.11.1.11 | NULL | > 0 | > > > > > > > 0 > > > > > > > | DirectAttached | 1 | NULL | > > > > > > > > > > > > > > | 75911 | VOLUME.CREATE | 3391 | 2021-05-06 > > > 10:16:55 > > > > | > > > > > > > 1 | 6472 | ROOT-3649 | NULL | > > 210 > > > | > > > > > > > 21474836480 | NULL | 1 | NULL | > > > > > > > > > > > > > > | 75912 | VM.CREATE | 3391 | 2021-05-06 > > > 10:16:55 > > > > | > > > > > > > 1 | 3649 | VM-38629-vasuflor | 51 | > > 210 | > > > > > > > NULL > > > > > > > | KVM | 1 | NULL | > > > > > > > > > > > > > > | 75913 | NETWORK.OFFERING.ASSIGN | 3391 | 2021-05-06 > > > 10:17:06 > > > > | > > > > > > > 1 | 3649 | 3753 | 6 | > > NULL > > > | > > > > > > > 1 | NULL | 1 | NULL | > > > > > > > > > > > > > > | 75914 | SG.ASSIGN | 3391 | 2021-05-06 > > > 10:17:06 > > > > | > > > > > > > 1 | 3649 | NULL | 3664 | > > NULL > > > | > > > > > > > NULL > > > > > > > | NULL | 1 | NULL | > > > > > > > > > > > > > > | 75915 | VM.START | 3391 | 2021-05-06 > > > 10:17:06 > > > > | > > > > > > > 1 | 3649 | VM-38629-vasuflor | 51 | > > 210 | > > > > > > > NULL > > > > > > > | KVM | 1 | NULL | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +-------+-------------------------+------------+---------------------+---------+-------------+--------------------+-------------+-------------+-------------+----------------+-----------+--------------+ > > > > > > > > > > > > > > 6 row > > > > > > > > > > > > > > mysql> select * from usage_network where account_id=3391; > > > > > > > > > > > > > > Empty set (0.00 sec) > > > > > > > > > > > > > > > > > > > > > > > > > > > > mysql> select * from usage_network_offering where > > account_id=3391 > > > > > > > > > > > > > > -> ; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +---------+------------+-----------+----------------+---------------------+------------+---------------------+---------+--------+ > > > > > > > > > > > > > > | zone_id | account_id | domain_id | vm_instance_id | > > > > > > network_offering_id | > > > > > > > is_default | created | deleted | nic_id | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +---------+------------+-----------+----------------+---------------------+------------+---------------------+---------+--------+ > > > > > > > > > > > > > > | 1 | 3391 | 3383 | 3649 | > > > > > > 6 | > > > > > > > 1 | 2021-05-06 10:17:06 | NULL | 3753 | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > +---------+------------+-----------+----------------+---------------------+------------+---------------------+---------+--------+ > > > > > > > > > > > > > > 1 row in set (0.01 sec) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jun 25, 2021 at 3:40 PM Daan Hoogland < > > > > daan.hoogl...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hean, > > > > > > > > To be honest, no i don't. You can start looking at any log > > > records > > > > > > > related > > > > > > > > to the VMs and than track to their nics. Also look at the > > > > > > > > usage_event-table, to see if any records are there. First in > > the > > > > > cloud > > > > > > > DB, > > > > > > > > and if these are there in the cloud_usage DB. > > > > > > > > Sorry to not have an obviously useful clue for you. > > > > > > > > > > > > > > > > On Thu, Jun 24, 2021 at 7:49 PM Hean Seng < > heans...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Daan, > > > > > > > > > > > > > > > > > > Thanks for reply. I deleted the null record, and manage to > > > make > > > > it > > > > > > > work > > > > > > > > > again. The it will auto generate back the record with > > correct > > > > ID . > > > > > > > > > > > > > > > > > > > > > > > > > > > However, I am facing the issue is the Usage Type 4, and 5 > > > > > (Network > > > > > > > sent > > > > > > > > > and received) are not populated . Other Type like Type1 is > > > > always > > > > > > > > working. > > > > > > > > > > > > > > > > > > Fyi, I am using share network, and set the parameter > > > > > > > > > vm.network.stats.interval=300 > > > > > > > > > so that the VM statistic is updated . > > > > > > > > > > > > > > > > > > This issue not happen on all VM, but part of it. ( about > > 50%) . > > > > I > > > > > > > > > tried look at the DB, and seem cannot find the clue what > > > actually > > > > > > diff > > > > > > > , > > > > > > > > in > > > > > > > > > term of those updated and those is not. > > > > > > > > > > > > > > > > > > The record is appearing in usage_network_offering and seems > > > good. > > > > > Do > > > > > > > you > > > > > > > > > have any idea of this ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jun 21, 2021 at 3:19 PM Daan Hoogland < > > > > > > daan.hoogl...@gmail.com > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Sorry Hean, wasn't on line most of the weekend, Is it > > running > > > > > now? > > > > > > > > > > as you wrote this two days ago, it should already have > run > > > > twice > > > > > > > > > (assuming > > > > > > > > > > you didn't change a lot) > > > > > > > > > > > > > > > > > > > > On Sat, Jun 19, 2021 at 8:41 PM Hean Seng < > > > heans...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > And it will insert another success is NULL to usage_job > > > > table , > > > > > > > > right ? > > > > > > > > > > > > > > > > > > > > > > i deleted the null record and restart usage servoce > and > > > see > > > > > > > > following > > > > > > > > > > in > > > > > > > > > > > usage log : > > > > > > > > > > > > > > > > > > > > > > 2021-06-19 18:37:52,912 INFO > > > [cloud.usage.UsageManagerImpl] > > > > > > > > > (main:null) > > > > > > > > > > > (logid:) Implementation Version is 4.15.0.0 > > > > > > > > > > > > > > > > > > > > > > 2021-06-19 18:37:55,942 DEBUG > > > [cloud.usage.UsageManagerImpl] > > > > > > > > > (main:null) > > > > > > > > > > > (logid:) Usage stats aggregation time zone: GMT > > > > > > > > > > > > > > > > > > > > > > 2021-06-19 18:37:55,952 DEBUG > > > [cloud.usage.UsageManagerImpl] > > > > > > > > > (main:null) > > > > > > > > > > > (logid:) Execution Time: Sat Jun 19 00:15:00 UTC 2021 > > > > > > > > > > > > > > > > > > > > > > 2021-06-19 18:37:55,953 DEBUG > > > [cloud.usage.UsageManagerImpl] > > > > > > > > > (main:null) > > > > > > > > > > > (logid:) Current Time: Sat Jun 19 18:37:55 UTC 2021 > > > > > > > > > > > > > > > > > > > > > > It seem execution time before current time ? Is that > > > normal > > > > ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Jun 20, 2021 at 2:39 AM Hean Seng < > > > > heans...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > So, I shall stop the Cloudstack-Usage services and > > delete > > > > all > > > > > > > > record > > > > > > > > > > > where > > > > > > > > > > > > success is NULL , and start back the usage service ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Jun 19, 2021 at 11:00 PM Daan Hoogland < > > > > > > > > > > daan.hoogl...@gmail.com> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > >> Heanm, > > > > > > > > > > > >> > > > > > > > > > > > >> On Sat, Jun 19, 2021 at 5:27 AM Hean Seng < > > > > > heans...@gmail.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > >> > > > > > > > > > > > >> > i have no unprocess job , but still have this > issue: > > > > > > > > > > > >> > > > > > > > > > > > > >> > 2021-06-19 10:09:57,657 DEBUG > > > > > [cloud.usage.UsageManagerImpl] > > > > > > > > > > > >> > (Usage-HB-1:null) (logid:) Scheduling Usage job... > > > > > > > > > > > >> > > > > > > > > > > > > >> > 2021-06-19 10:09:57,657 INFO > > > > > [cloud.usage.UsageManagerImpl] > > > > > > > > > > > >> > (Usage-Job-1:null) (logid:) starting usage job... > > > > > > > > > > > >> > > > > > > > > > > > > >> > 2021-06-19 10:09:57,661 DEBUG > > > > > [cloud.usage.UsageManagerImpl] > > > > > > > > > > > >> > (Usage-Job-1:null) (logid:) Not owner of usage > job, > > > > > > > skipping... > > > > > > > > > > > >> > > > > > > > > > > > > >> this line ^^ means there is a unprocessed job owned > by > > > > > another > > > > > > > > > > proccess > > > > > > > > > > > >> ... > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > 2021-06-19 10:09:57,661 INFO > > > > > [cloud.usage.UsageManagerImpl] > > > > > > > > > > > >> > (Usage-Job-1:null) (logid:) usage job complete > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > mysql> select id, pid, start_date ,end_date , > > > success, > > > > > > > > heartbeat > > > > > > > > > > from > > > > > > > > > > > >> > usage_job where success=0 order by start_date desc > > > limit > > > > > > 100; > > > > > > > > > > > >> > > > > > > > > > > > > >> you should search for `where success is null` > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > Empty set (0.00 sec) > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > On Fri, Jun 18, 2021 at 11:01 PM Hean Seng < > > > > > > > heans...@gmail.com> > > > > > > > > > > > wrote: > > > > > > > > > > > >> > > > > > > > > > > > > >> > > yes, I restarted the mgmt server, wait till > can > > > > login > > > > > > web > > > > > > > > > > > interface > > > > > > > > > > > >> . > > > > > > > > > > > >> > > then only restart usage . > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > I did follwoig also : > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > * stop usage server > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > * remove the record from usage_job for the > > > unprocessed > > > > > job > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > * start usage server again > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > Still same thing. Not sure where the Cloudstack > > > keep > > > > > the > > > > > > > > Usage > > > > > > > > > > > server > > > > > > > > > > > >> > PID > > > > > > > > > > > >> > > id , and keep getting wrong id. > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > On Fri, Jun 18, 2021 at 4:31 PM Boris Stoyanov < > > > > > > > > > > > >> > > boris.stoya...@shapeblue.com> wrote: > > > > > > > > > > > >> > > > > > > > > > > > > > >> > >> Hi Hean Seng, > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> Have you waited the management service to come > up > > > and > > > > > > then > > > > > > > > > > > restarted > > > > > > > > > > > >> > >> usage service? It might be some race condition? > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> Bobby. > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> From: Hean Seng <heans...@gmail.com> > > > > > > > > > > > >> > >> Date: Friday, 18 June 2021, 7:31 > > > > > > > > > > > >> > >> To: users@cloudstack.apache.org < > > > > > > > users@cloudstack.apache.org > > > > > > > > > > > > > > > > > > > > >> > >> Subject: Cloudstack Usage --- not owner > > > > > > > > > > > >> > >> Hi > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> I have mysql hang, and restarted mysql, > restarted > > > > > > > cloudstack > > > > > > > > > > mgmt, > > > > > > > > > > > >> and > > > > > > > > > > > >> > >> usage. > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> After that facing issue of following: > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> duration is 120 minutes) > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> 2021-06-18 04:25:49,656 DEBUG > > > > > > > [cloud.usage.UsageManagerImpl] > > > > > > > > > > > >> > >> (Usage-HB-1:null) (logid:) Scheduling Usage > > job... > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> 2021-06-18 04:25:49,657 INFO > > > > > > > [cloud.usage.UsageManagerImpl] > > > > > > > > > > > >> > >> (Usage-Job-1:null) (logid:) starting usage > job... > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> 2021-06-18 04:25:49,669 DEBUG > > > > > > > [cloud.usage.UsageManagerImpl] > > > > > > > > > > > >> > >> (Usage-Job-1:null) (logid:) Not owner of usage > > job, > > > > > > > > skipping... > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> 2021-06-18 04:25:49,669 INFO > > > > > > > [cloud.usage.UsageManagerImpl] > > > > > > > > > > > >> > >> (Usage-Job-1:null) (logid:) usage job complete > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> Seems usage pid is not updated to db . > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> Andybody know how to fix this ? > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> -- > > > > > > > > > > > >> > >> Regards, > > > > > > > > > > > >> > >> Hean Seng > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > -- > > > > > > > > > > > >> > > Regards, > > > > > > > > > > > >> > > Hean Seng > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > -- > > > > > > > > > > > >> > Regards, > > > > > > > > > > > >> > Hean Seng > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> -- > > > > > > > > > > > >> Daan > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Regards, > > > > > > > > > > > > Hean Seng > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Regards, > > > > > > > > > > > Hean Seng > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Daan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Regards, > > > > > > > > > Hean Seng > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Daan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Regards, > > > > > > > Hean Seng > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Daan > > > > > > > > > > > > > > > > > > > > > -- > > > > > Regards, > > > > > Hean Seng > > > > > > > > > > > > > > > > > -- > > > > Daan > > > > > > > > > > > > > -- > > > Regards, > > > Hean Seng > > > > > > > > > -- > > > > Andrija Panić > > > > > -- > Regards, > Hean Seng > -- Daan
