Re: [vdsm] Proposal VDSM = Engine Data Statistics Retrieval Optimization
On 03/07/2013 01:25 PM, Vinzenz Feenstra wrote:
Often Changed but unused
This data does not seem to be used in the engine at all. It is *not*
even used in the data warehouse.
*memoryStats* = {'swap_out': '0', 'majflt': '0', 'mem_free': '1466884',
'swap_in': '0', 'pageflt': '0', 'mem_total': '2096736', 'mem_unused': '1466884'}
*balloonInfo* = {'balloon_max': 2097152, 'balloon_cur': 2097152}
these were added for better balloon control iirc. so api needs them for
mom probbaly. not sure if engine needs them or not.
*disks* = {'vda': {'readLatency': '0', 'apparentsize': '64424509440',
'writeLatency': '1754496', 'imageID': '28abb923-7b89-4638-84f8-1700f0b76482',
'flushLatency': '156549', 'readRate': '0.00', 'truesize': '18855059456',
'writeRate': '952.05'}, 'hdc': {'readLatency': '0', 'apparentsize': '0',
'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0',
'writeRate': '0.00'}}
these i remember we added per request to the system - are you sure they
are not in the dwh?
___
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
[vdsm] Proposal VDSM = Engine Data Statistics Retrieval Optimization
Please find the prettier version on the wiki:
http://www.ovirt.org/Proposal_VDSM_-_Engine_Data_Statistics_Retrieval
Proposal VDSM - Engine Data Statistics Retrieval
VDSM = Engine data retrieval optimization
Motivation:
Currently the RHEVM engine is polling the a lot of data from VDSM every
15 seconds. This should be optimized and the amount of data requested
should be more specific.
For each VM the data currently contains much more information than
actually needed which blows up the size of the XML content quite big. We
could optimize this by splitting the reply on the getVmStats based on
the request of the engine into sections. For this reason Omer Frenkel
and me have split up the data into parts based on their usage.
This data can and usually does change during the lifetime of the VM.
Rarely Changed:
This data is change not very frequent and it should be enough to update
this only once in a while. Most commonly this data changes after changes
made in the UI or after a migration of the VM to another Host.
*Status* = Running
*acpiEnable* = true
*vmType* = kvm
*guestName* = W864GUESTAGENTT
*displayType* = qxl
*guestOs* = Win 8
*kvmEnable* = true #/*this should be constant and never changed*/
*pauseCode* = NOERR
*monitorResponse* = 0
*session* = Locked # unused
*netIfaces* = [{'name': 'Realtek RTL8139C+ Fast Ethernet NIC', 'inet6':
['fe80::490c:92bb:bbcc:9f87'], 'inet': ['10.34.60.148'], 'hw':
'00:1a:4a:22:3c:db'}]
*appsList* = ['RHEV-Tools 3.2.4', 'RHEV-Agent64 3.2.3', 'RHEV-Serial64
3.2.3', 'RHEV-Network64 3.2.2', 'RHEV-Network64 3.2.3', 'RHEV-Block64 3.2.3',
'RHEV-Balloon64 3.2.3', 'RHEV-Balloon64 3.2.2', 'RHEV-Agent64 3.2.2', 'RHEV-USB
3.2.3', 'RHEV-Block64 3.2.2', 'RHEV-Serial64 3.2.2']
*pid* = 11314
*guestIPs* = 10.34.60.148 # duplicated info
*displayIp* = 0
*displayPort* = 5902
*displaySecurePort* = 5903
*username* = user@W864GUESTAGENTT
*clientIp* =
*lastLogin* = 1361976900.67
Often Changed:
This data is changed quite often however it is not necessary to update
this data every 15 seconds. As this is cumulative data and reflects the
current status, and it does not need to be snapshotted every 15 seconds
to retrieve statistics. The data can be retrieved in much more generous
time slices. (e.g. Every 5 minutes)
*network* = {'vnet1': {'macAddr': '00:1a:4a:22:3c:db', 'rxDropped': '0',
'txDropped': '0', 'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0',
'txErrors': '0', 'state': 'unknown', 'speed': '100', 'name': 'vnet1'}}
*disksUsage* = [{'path': 'c:\\', 'total': '64055406592', 'fs': 'NTFS',
'used': '19223846912'}, {'path': 'd:\\', 'total': '3490912256', 'fs': 'UDF',
'used': '3490912256'}]
*timeOffset* = 14422
*elapsedTime* = 68591
*hash* = 2335461227228498964
*statsAge* = 0.09 # unused
Often Changed but unused
This data does not seem to be used in the engine at all. It is *not*
even used in the data warehouse.
*memoryStats* = {'swap_out': '0', 'majflt': '0', 'mem_free': '1466884',
'swap_in': '0', 'pageflt': '0', 'mem_total': '2096736', 'mem_unused': '1466884'}
*balloonInfo* = {'balloon_max': 2097152, 'balloon_cur': 2097152}
*disks* = {'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '1754496', 'imageID': '28abb923-7b89-4638-84f8-1700f0b76482', 'flushLatency': '156549', 'readRate': '0.00', 'truesize': '18855059456', 'writeRate': '952.05'}, 'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}}
Very frequent uppdates needed by webadmin portal:
This data is mostly needed for the webadmin portal and might be required
to be updated quite often. An exception here is the statsAge field,
which seems to be unused by the Engine. This data could be requested
every 15 seconds to keep things as they are now.
*cpuSys* = 2.32
*cpuUser* = 1.34
*memUsage* = 30
Proposed Solution for VDSM Engine:
We will introduce new optional parameters to getVmStats, getAllVmStats
and list to allow a finer grained specification of data which should be
included.
*Parameter:* *statsType*=/*string*/ (getVmStats, getAllVmStats only)
*Allowed values:*
* full (default to keep backwards compatibility)
* app-list (Just send the application list)
* rare (include everything from rarely changed to very frequent)
* often (include everything from often changed to very frequent)
* frequent (only send the very frequently changed items)
*Parameter:* *clientId*=*string* The client id is specified by the
client and should be unique however constantly used.
*Parameter:* *diff*=*boolean* In combination with the clientId VDSM
will send only differences to the previous request from the named
clientId. (if diff=true)
Additional Change:
Besides the introduction of
Re: [vdsm] Proposal VDSM = Engine Data Statistics Retrieval Optimization
On 03/07/2013 07:25 PM, Vinzenz Feenstra wrote:
Please find the prettier version on the wiki:
http://www.ovirt.org/Proposal_VDSM_-_Engine_Data_Statistics_Retrieval
Proposal VDSM - Engine Data Statistics Retrieval
VDSM = Engine data retrieval optimization
Motivation:
Currently the RHEVM engine is polling the a lot of data from VDSM
every 15 seconds. This should be optimized and the amount of data
requested should be more specific.
If the data size really matters, we could also consider to pack the
information into binary. I am not sure if it's suitable in the
transmission of XMLRPC.
For each VM the data currently contains much more information than
actually needed which blows up the size of the XML content quite big.
We could optimize this by splitting the reply on the getVmStats based
on the request of the engine into sections. For this reason Omer
Frenkel and me have split up the data into parts based on their usage.
This data can and usually does change during the lifetime of the VM.
Rarely Changed:
This data is change not very frequent and it should be enough to
update this only once in a while. Most commonly this data changes
after changes made in the UI or after a migration of the VM to another
Host.
*Status* = Running
*acpiEnable* = true
*vmType* = kvm
*guestName* = W864GUESTAGENTT
*displayType* = qxl
*guestOs* = Win 8
*kvmEnable* = true #/*this should be constant and never changed*/
Then it should be removed from vm stats. In my opinion, any information
belongs to vm's static configuration, it shouldn't be included in vm
stats. For the fields above, except 'Status', engine can get the
information without querying the vdsm host. It could not be changed by
vdsm itself, right?
*pauseCode* = NOERR
*monitorResponse* = 0
*session* = Locked # unused
*netIfaces* = [{'name': 'Realtek RTL8139C+ Fast Ethernet NIC', 'inet6':
['fe80::490c:92bb:bbcc:9f87'], 'inet': ['10.34.60.148'], 'hw':
'00:1a:4a:22:3c:db'}]
*appsList* = ['RHEV-Tools 3.2.4', 'RHEV-Agent64 3.2.3', 'RHEV-Serial64
3.2.3', 'RHEV-Network64 3.2.2', 'RHEV-Network64 3.2.3', 'RHEV-Block64 3.2.3',
'RHEV-Balloon64 3.2.3', 'RHEV-Balloon64 3.2.2', 'RHEV-Agent64 3.2.2', 'RHEV-USB
3.2.3', 'RHEV-Block64 3.2.2', 'RHEV-Serial64 3.2.2']
*pid* = 11314
*guestIPs* = 10.34.60.148 # duplicated info
*displayIp* = 0
*displayPort* = 5902
*displaySecurePort* = 5903
*username* = user@W864GUESTAGENTT
*clientIp* =
*lastLogin* = 1361976900.67
Often Changed:
This data is changed quite often however it is not necessary to update
this data every 15 seconds. As this is cumulative data and reflects
the current status, and it does not need to be snapshotted every 15
seconds to retrieve statistics. The data can be retrieved in much more
generous time slices. (e.g. Every 5 minutes)
*network* = {'vnet1': {'macAddr': '00:1a:4a:22:3c:db', 'rxDropped': '0',
'txDropped': '0', 'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0',
'txErrors': '0', 'state': 'unknown', 'speed': '100', 'name': 'vnet1'}}
macAddr and name don't change either.
*disksUsage* = [{'path': 'c:\\', 'total': '64055406592', 'fs': 'NTFS',
'used': '19223846912'}, {'path': 'd:\\', 'total': '3490912256', 'fs': 'UDF',
'used': '3490912256'}]
*timeOffset* = 14422
*elapsedTime* = 68591
*hash* = 2335461227228498964
*statsAge* = 0.09 # unused
Often Changed but unused
This data does not seem to be used in the engine at all. It is *not*
even used in the data warehouse.
*memoryStats* = {'swap_out': '0', 'majflt': '0', 'mem_free': '1466884',
'swap_in': '0', 'pageflt': '0', 'mem_total': '2096736', 'mem_unused': '1466884'}
*balloonInfo* = {'balloon_max': 2097152, 'balloon_cur': 2097152}
It's used by mom to adjust memory overcommitment dynamically.
*disks* = {'vda': {'readLatency': '0', 'apparentsize': '64424509440',
'writeLatency': '1754496', 'imageID': '28abb923-7b89-4638-84f8-1700f0b76482',
'flushLatency': '156549', 'readRate': '0.00', 'truesize': '18855059456',
'writeRate': '952.05'}, 'hdc': {'readLatency': '0', 'apparentsize': '0',
'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0',
'writeRate': '0.00'}}
Very frequent uppdates needed by webadmin portal:
This data is mostly needed for the webadmin portal and might be
required to be updated quite often. An exception here is the statsAge
field, which seems to be unused by the Engine. This data could be
requested every 15 seconds to keep things as they are now.
*cpuSys* = 2.32
*cpuUser* = 1.34
*memUsage* = 30
Proposed Solution for VDSM Engine:
We will introduce new optional parameters to getVmStats, getAllVmStats
and list to allow a finer grained specification of data which should
be included.
*Parameter:* *statsType*=/*string*/ (getVmStats,
