Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo
- Original Message - From: Itamar Heim ih...@redhat.com To: Liran Zelkha lzel...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Tuesday, June 11, 2013 4:00:21 PM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo On 06/11/2013 03:26 PM, Liran Zelkha wrote: Hi all, I'm checking performance for VdsUpdateRunTimeInfo. Naturally, much of the performance surrounds database activity (getVmsRunningOnVds queries, updateDeviceRuntimeInfo, updateVmDynamic) Few questions: 1. I have implemented batch updates for procedure UpdateVmDeviceRuntimeInfo for improved performance. 2. Seems like the only parameters UpdateVmDeviceRuntimeInfo is getting are vm_id,vm_device_id,address and alias. Are those rapidly changing, or will it be beneficial to implement caching on those updates (to ensure not-required updates do not travel to the database). slowly changing, but how will you cover all flows changing these devices to invalidate the cache (iiuc, this table is modified by engine when adding devices to a VM as well?) I don't think that in the device run time info we need to invalidate once we add a device. This is a specific case where we actually get the information from the VDSM (addresses are received from libvirt) The commands IIRC are first send to VDSM and than update the runtime info only on changed info (we can also hash it), It may put the placeholder in the DB first but it still relies on the data received from VDSM. 3. Any additional known performance problems you know of in regards to this class? Thanks... ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo
On 06/12/2013 09:41 AM, Barak Azulay wrote: - Original Message - From: Itamar Heim ih...@redhat.com To: Liran Zelkha lzel...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Tuesday, June 11, 2013 4:00:21 PM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo On 06/11/2013 03:26 PM, Liran Zelkha wrote: Hi all, I'm checking performance for VdsUpdateRunTimeInfo. Naturally, much of the performance surrounds database activity (getVmsRunningOnVds queries, updateDeviceRuntimeInfo, updateVmDynamic) Few questions: 1. I have implemented batch updates for procedure UpdateVmDeviceRuntimeInfo for improved performance. 2. Seems like the only parameters UpdateVmDeviceRuntimeInfo is getting are vm_id,vm_device_id,address and alias. Are those rapidly changing, or will it be beneficial to implement caching on those updates (to ensure not-required updates do not travel to the database). slowly changing, but how will you cover all flows changing these devices to invalidate the cache (iiuc, this table is modified by engine when adding devices to a VM as well?) I don't think that in the device run time info we need to invalidate once we add a device. This is a specific case where we actually get the information from the VDSM (addresses are received from libvirt) The commands IIRC are first send to VDSM and than update the runtime info only on changed info (we can also hash it), It may put the placeholder in the DB first but it still relies on the data received from VDSM. if this table is only updated from vdsm to save it, i agree. but isn't the engine also manipulating it? wasn't there a request to be able to maybe edit the addresses some day? ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo
- Original Message - From: Itamar Heim ih...@redhat.com To: Barak Azulay bazu...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Wednesday, June 12, 2013 10:09:09 AM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo On 06/12/2013 09:41 AM, Barak Azulay wrote: - Original Message - From: Itamar Heim ih...@redhat.com To: Liran Zelkha lzel...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Tuesday, June 11, 2013 4:00:21 PM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo On 06/11/2013 03:26 PM, Liran Zelkha wrote: Hi all, I'm checking performance for VdsUpdateRunTimeInfo. Naturally, much of the performance surrounds database activity (getVmsRunningOnVds queries, updateDeviceRuntimeInfo, updateVmDynamic) Few questions: 1. I have implemented batch updates for procedure UpdateVmDeviceRuntimeInfo for improved performance. 2. Seems like the only parameters UpdateVmDeviceRuntimeInfo is getting are vm_id,vm_device_id,address and alias. Are those rapidly changing, or will it be beneficial to implement caching on those updates (to ensure not-required updates do not travel to the database). slowly changing, but how will you cover all flows changing these devices to invalidate the cache (iiuc, this table is modified by engine when adding devices to a VM as well?) I don't think that in the device run time info we need to invalidate once we add a device. This is a specific case where we actually get the information from the VDSM (addresses are received from libvirt) The commands IIRC are first send to VDSM and than update the runtime info only on changed info (we can also hash it), It may put the placeholder in the DB first but it still relies on the data received from VDSM. if this table is only updated from vdsm to save it, i agree. but isn't the engine also manipulating it? wasn't there a request to be able to maybe edit the addresses some day? Even if there was such a request, we still update when we receive the info from libvirt. Anyway there are obviously a few improvements to be done before caching. - update only when info received from VDSM change (need to verify it is the practice here) - do batch update for the new updated data received. - do the hashing (of vm device runtime info) on VDSM for the data , and update only when hash changes - caching ... Barak ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo
Hi all, Current status of VdsUpdateRuntimeInfo (please note - patch is not yet committed). Initial UpdateRuntimeInfo performance wasted roughly 57% of CPU time on DB activity. Of that, about 25% of CPU time wasted on update behavior. After modification (moving updates to batch update) UpdateRuntimeInfo performance wasted roughly 40% of CPU time on DB activity. And only 12% of that was wasted on update behavior. My next task is to try and optimize the rest of overall database performance. - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Barak Azulay bazu...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Wednesday, June 12, 2013 12:35:03 PM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo Hi guys, I'm working hard on this. I'll send summary mail and findings later today. On Jun 12, 2013, at 10:45 AM, Barak Azulay wrote: - Original Message - From: Itamar Heim ih...@redhat.com To: Barak Azulay bazu...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Wednesday, June 12, 2013 10:09:09 AM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo On 06/12/2013 09:41 AM, Barak Azulay wrote: - Original Message - From: Itamar Heim ih...@redhat.com To: Liran Zelkha lzel...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Tuesday, June 11, 2013 4:00:21 PM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo On 06/11/2013 03:26 PM, Liran Zelkha wrote: Hi all, I'm checking performance for VdsUpdateRunTimeInfo. Naturally, much of the performance surrounds database activity (getVmsRunningOnVds queries, updateDeviceRuntimeInfo, updateVmDynamic) Few questions: 1. I have implemented batch updates for procedure UpdateVmDeviceRuntimeInfo for improved performance. 2. Seems like the only parameters UpdateVmDeviceRuntimeInfo is getting are vm_id,vm_device_id,address and alias. Are those rapidly changing, or will it be beneficial to implement caching on those updates (to ensure not-required updates do not travel to the database). slowly changing, but how will you cover all flows changing these devices to invalidate the cache (iiuc, this table is modified by engine when adding devices to a VM as well?) I don't think that in the device run time info we need to invalidate once we add a device. This is a specific case where we actually get the information from the VDSM (addresses are received from libvirt) The commands IIRC are first send to VDSM and than update the runtime info only on changed info (we can also hash it), It may put the placeholder in the DB first but it still relies on the data received from VDSM. if this table is only updated from vdsm to save it, i agree. but isn't the engine also manipulating it? wasn't there a request to be able to maybe edit the addresses some day? Even if there was such a request, we still update when we receive the info from libvirt. Anyway there are obviously a few improvements to be done before caching. - update only when info received from VDSM change (need to verify it is the practice here) - do batch update for the new updated data received. - do the hashing (of vm device runtime info) on VDSM for the data , and update only when hash changes - caching ... Barak ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo
On 06/12/2013 04:03 PM, Liran Zelkha wrote: Hi all, Current status of VdsUpdateRuntimeInfo (please note - patch is not yet committed). Initial UpdateRuntimeInfo performance wasted roughly 57% of CPU time on DB activity. Of that, about 25% of CPU time wasted on update behavior. After modification (moving updates to batch update) UpdateRuntimeInfo performance wasted roughly 40% of CPU time on DB activity. And only 12% of that was wasted on update behavior. which changes are in place for above? My next task is to try and optimize the rest of overall database performance. - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Barak Azulay bazu...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Wednesday, June 12, 2013 12:35:03 PM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo Hi guys, I'm working hard on this. I'll send summary mail and findings later today. On Jun 12, 2013, at 10:45 AM, Barak Azulay wrote: - Original Message - From: Itamar Heim ih...@redhat.com To: Barak Azulay bazu...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Wednesday, June 12, 2013 10:09:09 AM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo On 06/12/2013 09:41 AM, Barak Azulay wrote: - Original Message - From: Itamar Heim ih...@redhat.com To: Liran Zelkha lzel...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Tuesday, June 11, 2013 4:00:21 PM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo On 06/11/2013 03:26 PM, Liran Zelkha wrote: Hi all, I'm checking performance for VdsUpdateRunTimeInfo. Naturally, much of the performance surrounds database activity (getVmsRunningOnVds queries, updateDeviceRuntimeInfo, updateVmDynamic) Few questions: 1. I have implemented batch updates for procedure UpdateVmDeviceRuntimeInfo for improved performance. 2. Seems like the only parameters UpdateVmDeviceRuntimeInfo is getting are vm_id,vm_device_id,address and alias. Are those rapidly changing, or will it be beneficial to implement caching on those updates (to ensure not-required updates do not travel to the database). slowly changing, but how will you cover all flows changing these devices to invalidate the cache (iiuc, this table is modified by engine when adding devices to a VM as well?) I don't think that in the device run time info we need to invalidate once we add a device. This is a specific case where we actually get the information from the VDSM (addresses are received from libvirt) The commands IIRC are first send to VDSM and than update the runtime info only on changed info (we can also hash it), It may put the placeholder in the DB first but it still relies on the data received from VDSM. if this table is only updated from vdsm to save it, i agree. but isn't the engine also manipulating it? wasn't there a request to be able to maybe edit the addresses some day? Even if there was such a request, we still update when we receive the info from libvirt. Anyway there are obviously a few improvements to be done before caching. - update only when info received from VDSM change (need to verify it is the practice here) - do batch update for the new updated data received. - do the hashing (of vm device runtime info) on VDSM for the data , and update only when hash changes - caching ... Barak ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo
- Original Message - From: Barak Azulay bazu...@redhat.com To: Itamar Heim ih...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Wednesday, June 12, 2013 10:45:25 AM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo - Original Message - From: Itamar Heim ih...@redhat.com To: Barak Azulay bazu...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Wednesday, June 12, 2013 10:09:09 AM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo On 06/12/2013 09:41 AM, Barak Azulay wrote: - Original Message - From: Itamar Heim ih...@redhat.com To: Liran Zelkha lzel...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Tuesday, June 11, 2013 4:00:21 PM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo On 06/11/2013 03:26 PM, Liran Zelkha wrote: Hi all, I'm checking performance for VdsUpdateRunTimeInfo. Naturally, much of the performance surrounds database activity (getVmsRunningOnVds queries, updateDeviceRuntimeInfo, updateVmDynamic) Few questions: 1. I have implemented batch updates for procedure UpdateVmDeviceRuntimeInfo for improved performance. 2. Seems like the only parameters UpdateVmDeviceRuntimeInfo is getting are vm_id,vm_device_id,address and alias. Are those rapidly changing, or will it be beneficial to implement caching on those updates (to ensure not-required updates do not travel to the database). slowly changing, but how will you cover all flows changing these devices to invalidate the cache (iiuc, this table is modified by engine when adding devices to a VM as well?) I don't think that in the device run time info we need to invalidate once we add a device. This is a specific case where we actually get the information from the VDSM (addresses are received from libvirt) The commands IIRC are first send to VDSM and than update the runtime info only on changed info (we can also hash it), It may put the placeholder in the DB first but it still relies on the data received from VDSM. if this table is only updated from vdsm to save it, i agree. but isn't the engine also manipulating it? wasn't there a request to be able to maybe edit the addresses some day? Even if there was such a request, we still update when we receive the info from libvirt. Anyway there are obviously a few improvements to be done before caching. - update only when info received from VDSM change (need to verify it is the practice here) This should be the behavior right now There is a hash column in vm_dynamic , this hash is passed by VDSM We are updating only VMs that change their hash value - do batch update for the new updated data received. - do the hashing (of vm device runtime info) on VDSM for the data , and update only when hash changes - caching ... Barak ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo
Batch updates for UpdateVmDeviceRuntimeInfo, UpdateVdsInterfaceStatistics and UpdateVdsDynamic. All improved execution performance by about 60-80%. On Jun 12, 2013, at 4:05 PM, Itamar Heim wrote: On 06/12/2013 04:03 PM, Liran Zelkha wrote: Hi all, Current status of VdsUpdateRuntimeInfo (please note - patch is not yet committed). Initial UpdateRuntimeInfo performance wasted roughly 57% of CPU time on DB activity. Of that, about 25% of CPU time wasted on update behavior. After modification (moving updates to batch update) UpdateRuntimeInfo performance wasted roughly 40% of CPU time on DB activity. And only 12% of that was wasted on update behavior. which changes are in place for above? My next task is to try and optimize the rest of overall database performance. - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Barak Azulay bazu...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Wednesday, June 12, 2013 12:35:03 PM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo Hi guys, I'm working hard on this. I'll send summary mail and findings later today. On Jun 12, 2013, at 10:45 AM, Barak Azulay wrote: - Original Message - From: Itamar Heim ih...@redhat.com To: Barak Azulay bazu...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Wednesday, June 12, 2013 10:09:09 AM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo On 06/12/2013 09:41 AM, Barak Azulay wrote: - Original Message - From: Itamar Heim ih...@redhat.com To: Liran Zelkha lzel...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Tuesday, June 11, 2013 4:00:21 PM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo On 06/11/2013 03:26 PM, Liran Zelkha wrote: Hi all, I'm checking performance for VdsUpdateRunTimeInfo. Naturally, much of the performance surrounds database activity (getVmsRunningOnVds queries, updateDeviceRuntimeInfo, updateVmDynamic) Few questions: 1. I have implemented batch updates for procedure UpdateVmDeviceRuntimeInfo for improved performance. 2. Seems like the only parameters UpdateVmDeviceRuntimeInfo is getting are vm_id,vm_device_id,address and alias. Are those rapidly changing, or will it be beneficial to implement caching on those updates (to ensure not-required updates do not travel to the database). slowly changing, but how will you cover all flows changing these devices to invalidate the cache (iiuc, this table is modified by engine when adding devices to a VM as well?) I don't think that in the device run time info we need to invalidate once we add a device. This is a specific case where we actually get the information from the VDSM (addresses are received from libvirt) The commands IIRC are first send to VDSM and than update the runtime info only on changed info (we can also hash it), It may put the placeholder in the DB first but it still relies on the data received from VDSM. if this table is only updated from vdsm to save it, i agree. but isn't the engine also manipulating it? wasn't there a request to be able to maybe edit the addresses some day? Even if there was such a request, we still update when we receive the info from libvirt. Anyway there are obviously a few improvements to be done before caching. - update only when info received from VDSM change (need to verify it is the practice here) - do batch update for the new updated data received. - do the hashing (of vm device runtime info) on VDSM for the data , and update only when hash changes - caching ... Barak ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
[Engine-devel] Updates in VdsUpdateRuntimeInfo
Hi all, I'm checking performance for VdsUpdateRunTimeInfo. Naturally, much of the performance surrounds database activity (getVmsRunningOnVds queries, updateDeviceRuntimeInfo, updateVmDynamic) Few questions: 1. I have implemented batch updates for procedure UpdateVmDeviceRuntimeInfo for improved performance. 2. Seems like the only parameters UpdateVmDeviceRuntimeInfo is getting are vm_id,vm_device_id,address and alias. Are those rapidly changing, or will it be beneficial to implement caching on those updates (to ensure not-required updates do not travel to the database). 3. Any additional known performance problems you know of in regards to this class? Thanks... ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo
On 06/11/2013 03:26 PM, Liran Zelkha wrote: Hi all, I'm checking performance for VdsUpdateRunTimeInfo. Naturally, much of the performance surrounds database activity (getVmsRunningOnVds queries, updateDeviceRuntimeInfo, updateVmDynamic) Few questions: 1. I have implemented batch updates for procedure UpdateVmDeviceRuntimeInfo for improved performance. 2. Seems like the only parameters UpdateVmDeviceRuntimeInfo is getting are vm_id,vm_device_id,address and alias. Are those rapidly changing, or will it be beneficial to implement caching on those updates (to ensure not-required updates do not travel to the database). slowly changing, but how will you cover all flows changing these devices to invalidate the cache (iiuc, this table is modified by engine when adding devices to a VM as well?) 3. Any additional known performance problems you know of in regards to this class? Thanks... ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel