[
https://issues.apache.org/jira/browse/CLOUDSTACK-962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13552554#comment-13552554
]
Wei Zhou edited comment on CLOUDSTACK-962 at 1/14/13 10:21 AM:
---------------------------------------------------------------
As Ilia described, we noticed that the job running at 0:15(default) today
collected the data traffic at the day before yesterday, not yesterday.
I had another test. I found some problems
(1) After I set usage.stats.job.aggregation.range=15, and restart
cloud-management and cloud-usage services. The interval of usage job is 16
minutes (not 15 minutes). After 0:15( 23:45 GMT), it became right.
mysql> select id, start_date, end_date,success,heartbeat from
cloud_usage.usage_job;
| 448 | 2013-01-10 22:11:10 | 2013-01-10 22:27:10 | 1 | 2013-01-10
22:27:10 |
| 449 | 2013-01-10 22:27:10 | 2013-01-10 22:43:10 | 1 | 2013-01-10
22:43:10 |
| 450 | 2013-01-10 22:43:10 | 2013-01-10 22:59:10 | 1 | 2013-01-10
22:59:10 |
| 451 | 2013-01-10 22:59:10 | 2013-01-10 23:15:00 | 1 | 2013-01-10
23:14:10 |
| 452 | 2013-01-10 23:15:00 | 2013-01-10 23:30:00 | 1 | 2013-01-10
23:29:10 |
| 453 | 2013-01-10 23:30:00 | 2013-01-10 23:45:00 | 1 | 2013-01-10
23:44:10 |
| 454 | 2013-01-10 23:45:00 | 2013-01-11 00:00:00 | 1 | 2013-01-10
23:59:10 |
(2) Suppose the usage job executed at 9:15/30/45, I generated some data traffic
at 9:25, the AgentManager on Host ( which run every 5 minutes) collect the data
traffic and save to database(cloud.user_statistics) at 9:28.
However, The job running at 9:30 did not calculate it, and the job at 9:45
caculated it.
The jobs regarded it as a data traffic between 9:30 and 9:45. this is wrong.
I found some source codes which caculated data traffic only deal with the data
traffic of aggregate data, not the current data(new data).
In com.cloud.usage.UsageManagerImpl, Line 548-549,
hostAggregatedStat.setAggBytesSent(hostAggregatedStat.getAggBytesSent() +
userStat.getAggBytesSent());
hostAggregatedStat.setAggBytesReceived(hostAggregatedStat.getAggBytesReceived()
+ userStat.getAggBytesReceived());
In com.cloud.user.dao.UserStatisticsDaoImpl, Line 40
private static final String ACTIVE_AND_RECENTLY_DELETED_SEARCH = "SELECT
us.id, us.data_center_id, us.account_id, us.public_ip_address, us.device_id,
us.device_type, us.network_id, us.agg_bytes_received, us.agg_bytes_sent "
+"FROM user_statistics us, account a " + "WHERE us.account_id = a.id AND
(a.removed IS NULL OR a.removed >= ?) " + "ORDER BY us.id";
(3) After a network is deleted, the current value in cloud.user_statistics will
be set to 0. The data traffic after the last usage job execution time will not
be caculated.
was (Author: weizhou):
As Ilia described, we noticed that the job running at 0:15(default) today
collected the data traffic at the day before yesterday, not yesterday.
I had another test. I found some problems
(1) After I set usage.stats.job.aggregation.range=15, and restart
cloud-management and cloud-usage services. The interval of usage job is 16
minutes (not 15 minutes). After 0:15( 23:45 GMT), it became right.
mysql> select id, start_date, end_date,success,heartbeat from
cloud_usage.usage_job;
| 448 | 2013-01-10 22:11:10 | 2013-01-10 22:27:10 | 1 | 2013-01-10
22:27:10 |
| 449 | 2013-01-10 22:27:10 | 2013-01-10 22:43:10 | 1 | 2013-01-10
22:43:10 |
| 450 | 2013-01-10 22:43:10 | 2013-01-10 22:59:10 | 1 | 2013-01-10
22:59:10 |
| 451 | 2013-01-10 22:59:10 | 2013-01-10 23:15:00 | 1 | 2013-01-10
23:14:10 |
| 452 | 2013-01-10 23:15:00 | 2013-01-10 23:30:00 | 1 | 2013-01-10
23:29:10 |
| 453 | 2013-01-10 23:30:00 | 2013-01-10 23:45:00 | 1 | 2013-01-10
23:44:10 |
| 454 | 2013-01-10 23:45:00 | 2013-01-11 00:00:00 | 1 | 2013-01-10
23:59:10 |
(2) Suppose the usage job executed at 9:15/30/45, I generated some data traffic
at 9:25, the AgentManager on Host ( which run every 5 minutes) collect the data
traffic and save to database(cloud.user_statistics) at 9:28.
However, The job running at 9:30 did not calculate it, and the job at 9:45
caculated it.
The jobs regarded it as a data traffic between 9:30 and 9:45. this is wrong.
I found some source codes which caculated data traffic only deal with the data
traffic of aggregate data, not the current data(new data).
In com.cloud.usage.UsageManagerImpl, Line 548-549,
hostAggregatedStat.setAggBytesSent(hostAggregatedStat.getAggBytesSent() +
userStat.getCurrentBytesSent());
hostAggregatedStat.setAggBytesReceived(hostAggregatedStat.getAggBytesReceived()
+ userStat.getCurrentBytesReceived());
In com.cloud.user.dao.UserStatisticsDaoImpl, Line 40
private static final String ACTIVE_AND_RECENTLY_DELETED_SEARCH = "SELECT
us.id, us.data_center_id, us.account_id, us.public_ip_address, us.device_id,
us.device_type, us.network_id, us.agg_bytes_received, us.agg_bytes_sent "
+"FROM user_statistics us, account a " + "WHERE us.account_id = a.id AND
(a.removed IS NULL OR a.removed >= ?) " + "ORDER BY us.id";
(3) After a network is deleted, the current value in cloud.user_statistics will
be set to 0. The data traffic after the last usage job execution time will not
be caculated.
> CloudStack usage server didn't store data to cloud_usage in correct time
> ------------------------------------------------------------------------
>
> Key: CLOUDSTACK-962
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-962
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Usage
> Affects Versions: 4.0.0
> Reporter: Ilia Shakitko
> Labels: bytesReceived, bytesSended, cloud_usage, usage,
> user_statistics
>
> We have found strange behavior of CloudStack usage server.
> There are two global settings values:
> usage.stats.job.aggregation.range 1440
> usage.stats.job.exec.time 00:15
> Here I will describe how we had determined this problem.
> 9th of January we‘ve generated some traffic, like 10Gb. But 10th of January
> we didn’t saw records with amount of traffic in cloud_usage.cloud_usage. And
> we got this values in cloud_usage db only at 11th.
> Looks like it works like as written below:
> ---------January---------------------------------9th----------------10th--------------11th----------12th-----------
> (traffic generetad ) 10Gb 2,5Gb
> 5Gb …
> (traffic stored in usage_db) 0 0
> 10Gb 12,5Gb …
> We changed the value 'usage.stats.job.aggregation.range' to 10 minutes to
> have more tests. Here's what we got:
> ---------Timeline--------------------------------10:00--------------10:10------------10:20--------10:30--------10:40
> (traffic generetad ) 5Gb 0
> 2,5Gb 0 0
> (traffic stored in usage_db) 0 0
> 5Gb 5Gb 7,5Gb
> My colleague Wei Zhou will adds more comments with what we found in
> CloudStack source and solution for it.
> Will keep updated information about tests and additional info in this thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
