Re: [Engine-devel] NUMA support action items
On 4/3/2014 7:11 AM, Gilad Chaplik wrote: - Original Message - From: Chegu Vinod chegu_vi...@hp.com To: Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com Cc: Einav Cohen eco...@redhat.com, Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com, Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com, msi...@redhat.com, Da-huai Tang (Gary, MCXS-CQ) da-huai.t...@hp.com, Malini Rao m...@redhat.com, Eldan Hildesheim ehild...@redhat.com, Doron Fediuck dfedi...@redhat.com, sher...@redhat.com, Alexander Wels aw...@redhat.com, Gilad Chaplik gchap...@redhat.com Sent: Thursday, April 3, 2014 3:28:03 PM Subject: RE: NUMA support action items Hi Bruce, The virtual NUMA layout in the guest is a very simple one (not multi-level etc). It is generated by qemu+seabios... and there is no relationship with the host NUMA node distances etc. Let us not worry about gathering Virtual NUMA node distances for now. Vinod CC'ing devel list as well. Having said that, I don't see a reason why not to prepare an infrastructure for that (if it's free) for future versions (guest agent will collect vNuma data in some point in time). If you think having this Virtual NUMA topology (along with the virtual numa node *distance* info.) really helps some future use cases then pl. go ahead... Vinod Thanks, Gilad. -Original Message- From: Shi, Xiao-Lei (Bruce, HP Servers-PSC-CQ) Sent: Thursday, April 03, 2014 12:41 AM To: Vinod, Chegu Cc: Einav Cohen; Liang, Shang-Chun (David Liang, HPservers-Core-OE-PSC); Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC); msi...@redhat.com; Tang, Da-huai (Gary, MCXS-CQ); Malini Rao; Eldan Hildesheim; Doron Fediuck; sher...@redhat.com; Alexander Wels; Gilad Chaplik Subject: RE: NUMA support action items Hi Vinod, Is it meaningful for us to collect the distance information of vm numa node (maybe in future, not now)? In my understanding, vm numa topology is a simulation of numa topology, since the vcpus are just threads, I don't know how the vm numa node distances are calculated in vm. Is there any relationship between the vNode distances and host node distances? Thanks Best Regards Shi, Xiao-Lei (Bruce) Hewlett-Packard Co., Ltd. HP Servers Core Platform Software China Telephone +86 23 65683093 Mobile +86 18696583447 Email xiao-lei@hp.com -Original Message- From: Vinod, Chegu Sent: Thursday, April 03, 2014 7:18 AM To: Gilad Chaplik Cc: Einav Cohen; Liang, Shang-Chun (David Liang, HPservers-Core-OE-PSC); Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC); msi...@redhat.com; Shi, Xiao-Lei (Bruce, HP Servers-PSC-CQ); Tang, Da-huai (Gary, MCXS-CQ); Malini Rao; Eldan Hildesheim; Doron Fediuck; sher...@redhat.com; Alexander Wels Subject: RE: NUMA support action items Not sure what the correct way to do this isbut here is a suggestion. Let a given host server diagram shown be very generic...i.e. show the N sockets/nodes numbered from 0 thru N-1. Show the amount of memory and the list of CPUs in each of those sockets/nodes. Draw a generic Interconnect fabric [box] in between which all the sockets connect to Ideally ... Under that host diagram we could show the NUMA node distances in text format (as you know this is derived from the numactl -H and then conveyed from VDSM- oVIrt engine etc). That distance info. will tell the user what the distance between a pair of sockets/nodes are (and they can then do what they wish after that :)). Vinod -Original Message- From: Gilad Chaplik [mailto:gchap...@redhat.com] Sent: Wednesday, April 02, 2014 4:09 PM To: Vinod, Chegu Cc: Einav Cohen; Liang, Shang-Chun (David Liang, HPservers-Core-OE-PSC); Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC); msi...@redhat.com; Shi, Xiao-Lei (Bruce, HP Servers-PSC-CQ); Tang, Da-huai (Gary, MCXS-CQ); Malini Rao; Eldan Hildesheim; Doron Fediuck; sher...@redhat.com; Alexander Wels Subject: Re: NUMA support action items Thank you Vinod for the much elaborate explanation. GUI-wise, do you want to show those numbers? maybe for first phase, enough to show them via API? A thought, According to your example there could be up to 2 distances, so maybe the 'closer' nodes can be on the same column or sth; I mean to try an illustrate it graphically rather than with numbers (we have enough of those :)). Thanks, Gilad. - Original Message - From: Chegu Vinod chegu_vi...@hp.com To: Einav Cohen eco...@redhat.com Cc: Gilad Chaplik gchap...@redhat.com, Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com, Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com, msi...@redhat.com, Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com, Da-huai Tang (Gary, MCXS-CQ) da-huai.t...@hp.com, Malini Rao m...@redhat.com, Eldan Hildesheim ehild...@redhat.com, Doron Fediuck dfedi...@redhat.com, sher...@redhat.com, Alexander Wels aw...@redhat.com Sent: Saturday, March 29, 2014 8:15:56 AM Subject: Re: NUMA support
Re: [Engine-devel] NUMA support action items
- Original Message - From: Chegu Vinod chegu_vi...@hp.com To: Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com Cc: Einav Cohen eco...@redhat.com, Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com, Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com, msi...@redhat.com, Da-huai Tang (Gary, MCXS-CQ) da-huai.t...@hp.com, Malini Rao m...@redhat.com, Eldan Hildesheim ehild...@redhat.com, Doron Fediuck dfedi...@redhat.com, sher...@redhat.com, Alexander Wels aw...@redhat.com, Gilad Chaplik gchap...@redhat.com Sent: Thursday, April 3, 2014 3:28:03 PM Subject: RE: NUMA support action items Hi Bruce, The virtual NUMA layout in the guest is a very simple one (not multi-level etc). It is generated by qemu+seabios... and there is no relationship with the host NUMA node distances etc. Let us not worry about gathering Virtual NUMA node distances for now. Vinod CC'ing devel list as well. Having said that, I don't see a reason why not to prepare an infrastructure for that (if it's free) for future versions (guest agent will collect vNuma data in some point in time). Thanks, Gilad. -Original Message- From: Shi, Xiao-Lei (Bruce, HP Servers-PSC-CQ) Sent: Thursday, April 03, 2014 12:41 AM To: Vinod, Chegu Cc: Einav Cohen; Liang, Shang-Chun (David Liang, HPservers-Core-OE-PSC); Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC); msi...@redhat.com; Tang, Da-huai (Gary, MCXS-CQ); Malini Rao; Eldan Hildesheim; Doron Fediuck; sher...@redhat.com; Alexander Wels; Gilad Chaplik Subject: RE: NUMA support action items Hi Vinod, Is it meaningful for us to collect the distance information of vm numa node (maybe in future, not now)? In my understanding, vm numa topology is a simulation of numa topology, since the vcpus are just threads, I don't know how the vm numa node distances are calculated in vm. Is there any relationship between the vNode distances and host node distances? Thanks Best Regards Shi, Xiao-Lei (Bruce) Hewlett-Packard Co., Ltd. HP Servers Core Platform Software China Telephone +86 23 65683093 Mobile +86 18696583447 Email xiao-lei@hp.com -Original Message- From: Vinod, Chegu Sent: Thursday, April 03, 2014 7:18 AM To: Gilad Chaplik Cc: Einav Cohen; Liang, Shang-Chun (David Liang, HPservers-Core-OE-PSC); Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC); msi...@redhat.com; Shi, Xiao-Lei (Bruce, HP Servers-PSC-CQ); Tang, Da-huai (Gary, MCXS-CQ); Malini Rao; Eldan Hildesheim; Doron Fediuck; sher...@redhat.com; Alexander Wels Subject: RE: NUMA support action items Not sure what the correct way to do this isbut here is a suggestion. Let a given host server diagram shown be very generic...i.e. show the N sockets/nodes numbered from 0 thru N-1. Show the amount of memory and the list of CPUs in each of those sockets/nodes. Draw a generic Interconnect fabric [box] in between which all the sockets connect to Ideally ... Under that host diagram we could show the NUMA node distances in text format (as you know this is derived from the numactl -H and then conveyed from VDSM- oVIrt engine etc). That distance info. will tell the user what the distance between a pair of sockets/nodes are (and they can then do what they wish after that :)). Vinod -Original Message- From: Gilad Chaplik [mailto:gchap...@redhat.com] Sent: Wednesday, April 02, 2014 4:09 PM To: Vinod, Chegu Cc: Einav Cohen; Liang, Shang-Chun (David Liang, HPservers-Core-OE-PSC); Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC); msi...@redhat.com; Shi, Xiao-Lei (Bruce, HP Servers-PSC-CQ); Tang, Da-huai (Gary, MCXS-CQ); Malini Rao; Eldan Hildesheim; Doron Fediuck; sher...@redhat.com; Alexander Wels Subject: Re: NUMA support action items Thank you Vinod for the much elaborate explanation. GUI-wise, do you want to show those numbers? maybe for first phase, enough to show them via API? A thought, According to your example there could be up to 2 distances, so maybe the 'closer' nodes can be on the same column or sth; I mean to try an illustrate it graphically rather than with numbers (we have enough of those :)). Thanks, Gilad. - Original Message - From: Chegu Vinod chegu_vi...@hp.com To: Einav Cohen eco...@redhat.com Cc: Gilad Chaplik gchap...@redhat.com, Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com, Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com, msi...@redhat.com, Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com, Da-huai Tang (Gary, MCXS-CQ) da-huai.t...@hp.com, Malini Rao m...@redhat.com, Eldan Hildesheim ehild...@redhat.com, Doron Fediuck dfedi...@redhat.com, sher...@redhat.com, Alexander Wels aw...@redhat.com Sent: Saturday, March 29, 2014 8:15:56 AM Subject: Re: NUMA support action items On 3/27/2014 10:42 AM, Einav Cohen wrote: Hi Vinod, thank you very
Re: [Engine-devel] Please help us to review our database schema design with NUMA feature on ovirt
- Original Message - From: Eli Mesika emes...@redhat.com To: Omer Frenkel ofren...@redhat.com Cc: Gilad Chaplik gchap...@redhat.com, Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com, Roy Golan rgo...@redhat.com, Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com, Doron Fediuck dfedi...@redhat.com, Chegu Vinod chegu_vi...@hp.com, Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com, Yaniv Dary yd...@redhat.com, engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:51:38 PM Subject: Re: Please help us to review our database schema design with NUMA feature on ovirt - Original Message - From: Omer Frenkel ofren...@redhat.com To: Gilad Chaplik gchap...@redhat.com Cc: Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com, Roy Golan rgo...@redhat.com, Eli Mesika emes...@redhat.com, Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com, Doron Fediuck dfedi...@redhat.com, Chegu Vinod chegu_vi...@hp.com, Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com, Yaniv Dary yd...@redhat.com, engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:45:49 PM Subject: Re: Please help us to review our database schema design with NUMA feature on ovirt - Original Message - From: Gilad Chaplik gchap...@redhat.com To: Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com, Roy Golan rgo...@redhat.com, Omer Frenkel ofren...@redhat.com Cc: Eli Mesika emes...@redhat.com, Roy Golan rgo...@redhat.com, Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com, Doron Fediuck dfedi...@redhat.com, Chegu Vinod chegu_vi...@hp.com, Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com, Yaniv Dary yd...@redhat.com, engine-devel@ovirt.org Sent: Monday, March 31, 2014 4:31:17 PM Subject: Re: Please help us to review our database schema design with NUMA feature on ovirt adding Roy Omer. why CPU topology is in dynamic? the guideline we (try to) follow today is (i know you can find exceptions to this, but this is how it suppose to be): vm/vds static - user configurable information (whatever you have in add/edit) vm dynamic - information related to a running instance, the is changed in a somewhat small frequency vds dynamic - host capabilities, information from the host that doesn't change while the host is up. statistics - all the rest, mostly statistics, and other fast changing information. IMO it is a little more complicated and the terms static/dynamic does not reflect today the meaning of those terms rather, if engine knows the values by itself , this is considered static and if it has to ask the VDSM for the value then this is considered dynamic. For example cpu_cores is a static information but it is stored in vds_dynamic since we get the value from VDSM. I found those terms very confusing and a good candidate for a serious refactoring in future +1, I've opened another thread about it in devel list, and cc'ed Omer. Thanks, Gilad. - Original Message - From: Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com To: Eli Mesika emes...@redhat.com Cc: Gilad Chaplik gchap...@redhat.com, Roy Golan rgo...@redhat.com, Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com, Doron Fediuck dfedi...@redhat.com, Chegu Vinod chegu_vi...@hp.com, Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com, Yaniv Dary yd...@redhat.com, engine-devel@ovirt.org Sent: Monday, March 31, 2014 3:20:33 PM Subject: RE: Please help us to review our database schema design with NUMA feature on ovirt Thanks Eli. I will move the vm level NUMA fields to vm_dynamic, and the related database schema will be updated accordingly. Thanks Best Regards Shi, Xiao-Lei (Bruce) Hewlett-Packard Co., Ltd. HP Servers Core Platform Software China Telephone +86 23 65683093 Mobile +86 18696583447 Email xiao-lei@hp.com -Original Message- From: Eli Mesika [mailto:emes...@redhat.com] Sent: Monday, March 31, 2014 5:49 PM To: Shi, Xiao-Lei (Bruce, HP Servers-PSC-CQ) Cc: Gilad Chaplik; Roy Golan; Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC); Doron Fediuck; Vinod, Chegu; Liang, Shang-Chun (David Liang, HPservers-Core-OE-PSC); Yaniv Dary; engine-devel@ovirt.org Subject: Re: Please help us to review our database schema design with NUMA feature on ovirt - Original Message - From: Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com To: Gilad Chaplik gchap...@redhat.com, Eli Mesika emes...@redhat.com, Roy Golan rgo...@redhat.com Cc: Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com,
Re: [Engine-devel] Please help us to review our database schema design with NUMA feature on ovirt
- Original Message - From: Omer Frenkel ofren...@redhat.com To: Gilad Chaplik gchap...@redhat.com Cc: Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com, Roy Golan rgo...@redhat.com, Eli Mesika emes...@redhat.com, Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com, Doron Fediuck dfedi...@redhat.com, Chegu Vinod chegu_vi...@hp.com, Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com, Yaniv Dary yd...@redhat.com, engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:45:49 PM Subject: Re: Please help us to review our database schema design with NUMA feature on ovirt - Original Message - From: Gilad Chaplik gchap...@redhat.com To: Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com, Roy Golan rgo...@redhat.com, Omer Frenkel ofren...@redhat.com Cc: Eli Mesika emes...@redhat.com, Roy Golan rgo...@redhat.com, Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com, Doron Fediuck dfedi...@redhat.com, Chegu Vinod chegu_vi...@hp.com, Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com, Yaniv Dary yd...@redhat.com, engine-devel@ovirt.org Sent: Monday, March 31, 2014 4:31:17 PM Subject: Re: Please help us to review our database schema design with NUMA feature on ovirt adding Roy Omer. why CPU topology is in dynamic? the guideline we (try to) follow today is (i know you can find exceptions to this, but this is how it suppose to be): vm/vds static - user configurable information (whatever you have in add/edit) vm dynamic - information related to a running instance, the is changed in a somewhat small frequency vds dynamic - host capabilities, information from the host that doesn't change while the host is up. statistics - all the rest, mostly statistics, and other fast changing information. IMO it is a little more complicated and the terms static/dynamic does not reflect today the meaning of those terms rather, if engine knows the values by itself , this is considered static and if it has to ask the VDSM for the value then this is considered dynamic. For example cpu_cores is a static information but it is stored in vds_dynamic since we get the value from VDSM. I found those terms very confusing and a good candidate for a serious refactoring in future Thanks, Gilad. - Original Message - From: Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com To: Eli Mesika emes...@redhat.com Cc: Gilad Chaplik gchap...@redhat.com, Roy Golan rgo...@redhat.com, Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com, Doron Fediuck dfedi...@redhat.com, Chegu Vinod chegu_vi...@hp.com, Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com, Yaniv Dary yd...@redhat.com, engine-devel@ovirt.org Sent: Monday, March 31, 2014 3:20:33 PM Subject: RE: Please help us to review our database schema design with NUMA feature on ovirt Thanks Eli. I will move the vm level NUMA fields to vm_dynamic, and the related database schema will be updated accordingly. Thanks Best Regards Shi, Xiao-Lei (Bruce) Hewlett-Packard Co., Ltd. HP Servers Core Platform Software China Telephone +86 23 65683093 Mobile +86 18696583447 Email xiao-lei@hp.com -Original Message- From: Eli Mesika [mailto:emes...@redhat.com] Sent: Monday, March 31, 2014 5:49 PM To: Shi, Xiao-Lei (Bruce, HP Servers-PSC-CQ) Cc: Gilad Chaplik; Roy Golan; Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC); Doron Fediuck; Vinod, Chegu; Liang, Shang-Chun (David Liang, HPservers-Core-OE-PSC); Yaniv Dary; engine-devel@ovirt.org Subject: Re: Please help us to review our database schema design with NUMA feature on ovirt - Original Message - From: Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com To: Gilad Chaplik gchap...@redhat.com, Eli Mesika emes...@redhat.com, Roy Golan rgo...@redhat.com Cc: Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com, Doron Fediuck dfedi...@redhat.com, Chegu Vinod chegu_vi...@hp.com, Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com, Yaniv Dary yd...@redhat.com, engine-devel@ovirt.org Sent: Monday, March 31, 2014 12:38:04 PM Subject: RE: Please help us to review our database schema design with NUMA feature on ovirt We put host level NUMA fields in vds_dynamic because these information are from host itself, and NUMA topology may be changed if the host's hardware make a change. NUMA information are similar to the host's cpu topology information like cpu_cores and cpu_sockets which are in vds_dynamic, we refer to this. VM level NUMA fields are configured by user, and actually we originally think they should be in vm_dynamic. But we found that the
Re: [Engine-devel] vds_dynamic refactor
Don't go down that road. Status shouldn't be saved in the db. But anyway statistics is rapidly changing. So it fits... On Apr 3, 2014 5:14 PM, Gilad Chaplik gchap...@redhat.com wrote: - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Eli Mesika emes...@redhat.com Cc: Gilad Chaplik gchap...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:36:07 PM Subject: Re: [Engine-devel] vds_dynamic refactor I would be happy if we can lose vds_dynamic all together, and just keep vds_static and vds_statistics. Our performance will be happy too ;-) @Liran, status and pending fields are very fragile ones, IMO need separate table. @Eli, iirc, you don't have a problem with adding more tables :-) On Thu, Apr 3, 2014 at 4:33 PM, Eli Mesika emes...@redhat.com wrote: - Original Message - From: Gilad Chaplik gchap...@redhat.com To: Yair Zaslavsky yzasl...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:00:25 PM Subject: Re: [Engine-devel] vds_dynamic refactor - Original Message - From: Yair Zaslavsky yzasl...@redhat.com To: Liran Zelkha liran.zel...@gmail.com Cc: Gilad Chaplik gchap...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 2:12:59 PM Subject: Re: [Engine-devel] vds_dynamic refactor - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 2:04:29 PM Subject: Re: [Engine-devel] vds_dynamic refactor Why not move it to vds_static? +1 on Liran's comment. +1 as well , vds_static is the place for that I would prefer not to add more complexity to the vds tables family. Such complexity may effect performs of queries/views. If you wish, you can create a view on top of vds_static named vds_on_boot for querying of vds on boot info. Yair That means moving almost all of vds_dynamic into vds_static except of memory, pending resources and status (maybe more but not much); and there will not be any db separation between user input and on_boot data. Why we should have such separation ? Thanks, Gilad. On Thu, Apr 3, 2014 at 2:00 PM, Gilad Chaplik gchap...@redhat.com wrote: Hi list, I propose to split vds_dynamic table into 2 tables: - vds_dynamic - vds_on_boot We need a place to put all non-dynamic data that gets updated once the host is booted, and I think dynamic isn't the place for it. In first phase we will put there NUMA host topoplogy, but later on migrate all non-dynamic host data (cpu, os, etc.). thoughts? Thanks, Gilad. ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] vds_dynamic refactor
- Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Eli Mesika emes...@redhat.com Cc: Gilad Chaplik gchap...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:36:07 PM Subject: Re: [Engine-devel] vds_dynamic refactor I would be happy if we can lose vds_dynamic all together, and just keep vds_static and vds_statistics. Our performance will be happy too ;-) @Liran, status and pending fields are very fragile ones, IMO need separate table. @Eli, iirc, you don't have a problem with adding more tables :-) On Thu, Apr 3, 2014 at 4:33 PM, Eli Mesika emes...@redhat.com wrote: - Original Message - From: Gilad Chaplik gchap...@redhat.com To: Yair Zaslavsky yzasl...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:00:25 PM Subject: Re: [Engine-devel] vds_dynamic refactor - Original Message - From: Yair Zaslavsky yzasl...@redhat.com To: Liran Zelkha liran.zel...@gmail.com Cc: Gilad Chaplik gchap...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 2:12:59 PM Subject: Re: [Engine-devel] vds_dynamic refactor - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 2:04:29 PM Subject: Re: [Engine-devel] vds_dynamic refactor Why not move it to vds_static? +1 on Liran's comment. +1 as well , vds_static is the place for that I would prefer not to add more complexity to the vds tables family. Such complexity may effect performs of queries/views. If you wish, you can create a view on top of vds_static named vds_on_boot for querying of vds on boot info. Yair That means moving almost all of vds_dynamic into vds_static except of memory, pending resources and status (maybe more but not much); and there will not be any db separation between user input and on_boot data. Why we should have such separation ? Thanks, Gilad. On Thu, Apr 3, 2014 at 2:00 PM, Gilad Chaplik gchap...@redhat.com wrote: Hi list, I propose to split vds_dynamic table into 2 tables: - vds_dynamic - vds_on_boot We need a place to put all non-dynamic data that gets updated once the host is booted, and I think dynamic isn't the place for it. In first phase we will put there NUMA host topoplogy, but later on migrate all non-dynamic host data (cpu, os, etc.). thoughts? Thanks, Gilad. ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Please help us to review our database schema design with NUMA feature on ovirt
- Original Message - From: Gilad Chaplik gchap...@redhat.com To: Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com, Roy Golan rgo...@redhat.com, Omer Frenkel ofren...@redhat.com Cc: Eli Mesika emes...@redhat.com, Roy Golan rgo...@redhat.com, Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com, Doron Fediuck dfedi...@redhat.com, Chegu Vinod chegu_vi...@hp.com, Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com, Yaniv Dary yd...@redhat.com, engine-devel@ovirt.org Sent: Monday, March 31, 2014 4:31:17 PM Subject: Re: Please help us to review our database schema design with NUMA feature on ovirt adding Roy Omer. why CPU topology is in dynamic? the guideline we (try to) follow today is (i know you can find exceptions to this, but this is how it suppose to be): vm/vds static - user configurable information (whatever you have in add/edit) vm dynamic - information related to a running instance, the is changed in a somewhat small frequency vds dynamic - host capabilities, information from the host that doesn't change while the host is up. statistics - all the rest, mostly statistics, and other fast changing information. Thanks, Gilad. - Original Message - From: Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com To: Eli Mesika emes...@redhat.com Cc: Gilad Chaplik gchap...@redhat.com, Roy Golan rgo...@redhat.com, Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com, Doron Fediuck dfedi...@redhat.com, Chegu Vinod chegu_vi...@hp.com, Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com, Yaniv Dary yd...@redhat.com, engine-devel@ovirt.org Sent: Monday, March 31, 2014 3:20:33 PM Subject: RE: Please help us to review our database schema design with NUMA feature on ovirt Thanks Eli. I will move the vm level NUMA fields to vm_dynamic, and the related database schema will be updated accordingly. Thanks Best Regards Shi, Xiao-Lei (Bruce) Hewlett-Packard Co., Ltd. HP Servers Core Platform Software China Telephone +86 23 65683093 Mobile +86 18696583447 Email xiao-lei@hp.com -Original Message- From: Eli Mesika [mailto:emes...@redhat.com] Sent: Monday, March 31, 2014 5:49 PM To: Shi, Xiao-Lei (Bruce, HP Servers-PSC-CQ) Cc: Gilad Chaplik; Roy Golan; Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC); Doron Fediuck; Vinod, Chegu; Liang, Shang-Chun (David Liang, HPservers-Core-OE-PSC); Yaniv Dary; engine-devel@ovirt.org Subject: Re: Please help us to review our database schema design with NUMA feature on ovirt - Original Message - From: Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com To: Gilad Chaplik gchap...@redhat.com, Eli Mesika emes...@redhat.com, Roy Golan rgo...@redhat.com Cc: Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com, Doron Fediuck dfedi...@redhat.com, Chegu Vinod chegu_vi...@hp.com, Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com, Yaniv Dary yd...@redhat.com, engine-devel@ovirt.org Sent: Monday, March 31, 2014 12:38:04 PM Subject: RE: Please help us to review our database schema design with NUMA feature on ovirt We put host level NUMA fields in vds_dynamic because these information are from host itself, and NUMA topology may be changed if the host's hardware make a change. NUMA information are similar to the host's cpu topology information like cpu_cores and cpu_sockets which are in vds_dynamic, we refer to this. VM level NUMA fields are configured by user, and actually we originally think they should be in vm_dynamic. But we found that the field of another feature cpuPin which is similar as NUMA feature is in vm_static, so we put vm NUMA fields in vm_static. Do you think we need to put VM level NUMA fields in vm_dynamic? I think that in this case we should fix cpuPin to be in vm_dynamic and put after that the other NUMA fields in vm_dynamic as well Thanks Best Regards Shi, Xiao-Lei (Bruce) Hewlett-Packard Co., Ltd. HP Servers Core Platform Software China Telephone +86 23 65683093 Mobile +86 18696583447 Email xiao-lei@hp.com -Original Message- From: Gilad Chaplik [mailto:gchap...@redhat.com] Sent: Monday, March 31, 2014 5:22 PM To: Eli Mesika; Roy Golan Cc: Shi, Xiao-Lei (Bruce, HP Servers-PSC-CQ); Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC); Doron Fediuck; Vinod, Chegu; Liang, Shang-Chun (David Liang, HPservers-Core-OE-PSC); Yaniv Dary; engine-devel@ovirt.org Subject: Re: Please help us to review our database schema design with NUMA feature on ovirt +1 IMO: vds data should reside in static VM need to think about it. Roy? Thanks, Gilad. - Original Message - From: Eli Mesika emes...@redhat.com
Re: [Engine-devel] vds_dynamic refactor
- Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com Cc: Eli Mesika emes...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 5:16:51 PM Subject: Re: [Engine-devel] vds_dynamic refactor Don't go down that road. Status shouldn't be saved in the db. But anyway statistics is rapidly changing. So it fits... First let's have a notion of caching, then notion of shared caching, then I can start thinking of not going down that road... On Apr 3, 2014 5:14 PM, Gilad Chaplik gchap...@redhat.com wrote: - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Eli Mesika emes...@redhat.com Cc: Gilad Chaplik gchap...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:36:07 PM Subject: Re: [Engine-devel] vds_dynamic refactor I would be happy if we can lose vds_dynamic all together, and just keep vds_static and vds_statistics. Our performance will be happy too ;-) @Liran, status and pending fields are very fragile ones, IMO need separate table. @Eli, iirc, you don't have a problem with adding more tables :-) On Thu, Apr 3, 2014 at 4:33 PM, Eli Mesika emes...@redhat.com wrote: - Original Message - From: Gilad Chaplik gchap...@redhat.com To: Yair Zaslavsky yzasl...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:00:25 PM Subject: Re: [Engine-devel] vds_dynamic refactor - Original Message - From: Yair Zaslavsky yzasl...@redhat.com To: Liran Zelkha liran.zel...@gmail.com Cc: Gilad Chaplik gchap...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 2:12:59 PM Subject: Re: [Engine-devel] vds_dynamic refactor - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 2:04:29 PM Subject: Re: [Engine-devel] vds_dynamic refactor Why not move it to vds_static? +1 on Liran's comment. +1 as well , vds_static is the place for that I would prefer not to add more complexity to the vds tables family. Such complexity may effect performs of queries/views. If you wish, you can create a view on top of vds_static named vds_on_boot for querying of vds on boot info. Yair That means moving almost all of vds_dynamic into vds_static except of memory, pending resources and status (maybe more but not much); and there will not be any db separation between user input and on_boot data. Why we should have such separation ? Thanks, Gilad. On Thu, Apr 3, 2014 at 2:00 PM, Gilad Chaplik gchap...@redhat.com wrote: Hi list, I propose to split vds_dynamic table into 2 tables: - vds_dynamic - vds_on_boot We need a place to put all non-dynamic data that gets updated once the host is booted, and I think dynamic isn't the place for it. In first phase we will put there NUMA host topoplogy, but later on migrate all non-dynamic host data (cpu, os, etc.). thoughts? Thanks, Gilad. ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] ovirtengine connect node
On 03/28/2014 09:36 AM, marco lin wrote: st node1 is compatible with versions (3.0,3.1,3.2,3.3) and cannot join Cluster Default which is set to version 3.4. How to solve? was this resolveD? which OS are you using for node1? ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Please help us to review our database schema design with NUMA feature on ovirt
On 4/3/2014 3:46 AM, Gilad Chaplik wrote: - Original Message - From: Eli Mesika emes...@redhat.com To: Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com Cc: Gilad Chaplik gchap...@redhat.com, Roy Golan rgo...@redhat.com, Omer Frenkel ofren...@redhat.com, Chegu Vinod chegu_vi...@hp.com, Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com, Doron Fediuck dfedi...@redhat.com, Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com, Yaniv Dary yd...@redhat.com, engine-devel@ovirt.org Sent: Thursday, April 3, 2014 10:54:54 AM Subject: Re: Please help us to review our database schema design with NUMA feature on ovirt - Original Message - From: Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com To: Gilad Chaplik gchap...@redhat.com, Eli Mesika emes...@redhat.com Cc: Roy Golan rgo...@redhat.com, Omer Frenkel ofren...@redhat.com, Chegu Vinod chegu_vi...@hp.com, Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com, Doron Fediuck dfedi...@redhat.com, Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com, Yaniv Dary yd...@redhat.com, engine-devel@ovirt.org Sent: Thursday, April 3, 2014 7:25:11 AM Subject: RE: Please help us to review our database schema design with NUMA feature on ovirt Hi all, Please see my comments in line. Thanks Best Regards Shi, Xiao-Lei (Bruce) Hewlett-Packard Co., Ltd. HP Servers Core Platform Software China Telephone +86 23 65683093 Mobile +86 18696583447 Email xiao-lei@hp.com -Original Message- From: Gilad Chaplik [mailto:gchap...@redhat.com] Sent: Thursday, April 03, 2014 9:05 AM To: Eli Mesika Cc: Shi, Xiao-Lei (Bruce, HP Servers-PSC-CQ); Roy Golan; Omer Frenkel; Vinod, Chegu; Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC); Doron Fediuck; Liang, Shang-Chun (David Liang, HPservers-Core-OE-PSC); Yaniv Dary; engine-devel@ovirt.org Subject: Re: Please help us to review our database schema design with NUMA feature on ovirt Hi all, Sorry for joining-in late. My comments (according to the db diagram section in https://docs.google.com/document/d/1-wdDkm6EDbwyoCIRPPcmbGWAcyQo_ISTY8ykDr0I6VY): 1) Join vm_numa_node and vds_numa_node to a single table (almost identical), one of the FKs can be null. [Bruce] I prefer two tables. Actually host level NUMA node and vm level NUMA node are different objects. In my understanding, vm level NUMA node is just a simulation of host level NUMA node, and host level NUMA node has more features that not in vm NUMA (like several levels of host NUMA topology mentioned by Vinod). We need to consider the extensions of host NUMA in the future. What future extension are you referring to ? Not sure how relevant this is to the discussion but a little bit of background info. here : A VM's Virtual NUMA node topology is generated by qemu+seabios and is based on options specified at the qemu command line (libvirt translates the information in the VM's xml file and invokes the qemu command line with the correct options).. Today there is no support in qemu+seabios for generating multiple levels of Virtual NUMA. A vast majority of the hosts out there (i.e. 2 socket and 4 socket hosts) have only single level of NUMA topology...so this is fine for now. (Multi-level NUMA support in the qemu+seabios is a slightly different topic...and may (or may not) be pursued as a potential future enhancement for qemu so for now let us not worry about such things over-engineer in oVirt infrastructure etc. for multi-level virtual NUMA nodes etc.) The values for the node distances in the virtual NUMA topology are auto-generated defaults (by qemu+seabios) and has no relation with the node distances in the host NUMA topology (which is extracted from the ACPI SLIT tables and are supposed to be representative of the underlying system fabric's inter node latency capabilities etc). All the guest OS needs to know is that there are multiple [virtual] NUMA nodes and these virtual nodes are a single hop away This helps the guest to do the right thing with per node data structure allocations/locking etc and helps it scale/perform better. As I mentioned in another email thread : If it makes sense for some [current/future] use cases to store this virtual NUMA topology info. (along with the node distances) somewhere in the oVirt infrastructure...then please feel free to do so. Let's open the discussion and consider them right now. vNode and Node are the same. Not really sure what I can say here... A VM's virtual NUMA node should be sized (i.e. cpu count in the node) no larger than the host NUMA node. (Ideally they should be of the same size). Vinod Vinod? I agree with Bruce, we have no problem with more tables and constrains should work as expected and remove entries when a Host or VM is removed. I do not like tables that have 2 UUIDs when one of them is null , this is against simple DB normalization We are going
Re: [Engine-devel] NUMA Support - GUI Technical Session
Hi Gilad, to my understanding, we already use _HTML5_ Drag'n'Drop support (exposed by GWT API since 2.4) in Setup Host Networks dialog. To utilize HTML5 Drag'n'Drop support in GWT widget, just extend FocusPanel and mark your widget's DOM element as draggable. For example, in UnassignedNetworksPanel constructor: getElement().setDraggable( ... ); addBitlessDomHandler(new DragEnterHandler() { ... }); addBitlessDomHandler(new DragOverHandler() { ... }); addBitlessDomHandler(new DragLeaveHandler() { ... }); addBitlessDomHandler(new DropHandler() { ... }); In other words, GWT already exposes API for working with HTML5 Drag'n'Drop spec, so you don't need any 3rd party libraries. The downside is that HTML5 Drag'n'Drop spec is supported only in recent browsers (but this isn't an issue for us, AFAIK): http://caniuse.com/#feat=dragndrop So in general we have two alternatives: 1, use standard HTML5 Drag'n'Drop spec pros: + no need for 3rd party library + compliant with existing code, i.e. Setup Host Networks cons (not too relevant IMHO): - requires browser support of HTML5 Drag'n'Drop spec - HTML5 Drag'n'Drop spec deals with dragging data, not widgets themselves (no HTML DOM re-parenting after drag finish) 2, use 3rd party gwt-dnd library pros (which I don't think we really need): + emulate Drag'n'Drop support in older browsers + more advanced functionality, i.e. allows dragging widgets so that HTML DOM is dynamically updated cons: - dependency on 3rd party library - this would mean we need to revisit existing code (we should do Drag'n'Drop in one consistent way) It's possible that I might be missing something, but I'd suggest to try using HTML5 Drag'n'Drop via GWT API as the first approach. If we find out that HTML5 Drag'n'Drop doesn't work for us in given browser(s) or if we need extra functionality, we can always add gwt-dnd dependency. Few more comments inline, let me know what you think. Regards, Vojtech - Original Message - From: Alexander Wels aw...@redhat.com To: Gilad Chaplik gchap...@redhat.com Cc: eco...@redhat.com, Vojtech Szocs vsz...@redhat.com, dfedi...@redhat.com, engine-devel@ovirt.org, chegu vinod chegu_vi...@hp.com, lver...@redhat.com Sent: Tuesday, April 1, 2014 7:24:12 PM Subject: Re: NUMA Support - GUI Technical Session On Tuesday, April 01, 2014 11:34:43 AM Gilad Chaplik wrote: Hi all, Here are the resolutions from the meeting: * option 1 1) Use gwt-dnd lib for oVirt's dnd (drag and drop) infrastructure. 2) Come up with a very simple POC that covers all of NUMA-support UX requirements. 3) Either do a POC, or get UX maintainers approval, that moving already existing dnd features to new infrastructure (setup-networks and scheduling policy dialogs) is feasible and possible. +1 for option 1. None of the drag and drop in the application now looks terribly hard. The gwt-dnd library simply makes things easier to control and maintain. Yeah, but the downside is adding 3rd party dependency which predates GWT's support for (API exposure of) HTML5 Drag'n'Drop spec. * option 2 Extract setup network dnd capabilities to a common general infrastructure and use that as an infrastructure. NOTE that setup-networks will not use it in oVirt-3.5. GWT's HTML5 Drag'n'Drop API works directly on DOM element level, it's just a thin API overlay on top of HTML5Drag'n'Drop spec. Do we really need custom DnD infra on top of that? Can't we just use GWT APIs like Element.setDraggable, Drag*Handler, Drop*Handler? I will start with option 1, just need UX team approval that [1] will be added to ovirt-3.5, and be used for dnd for now on. In case I fail to deliver option 1 (with the help and guidance of the UX team) in a quick cycle (a week or so), I will peruse option 2, which is trivial. Moving forward, all new dnd enabled features will use the new infrastructure, and the motivation is to migrate existing ones as well. I agree, there should be one consistent way to do DnD in oVirt UI. I'm just saying we should consider existing GWT HTML5 Drag'n'Drop API before jumping into gwt-dnd 3rd party dependency. If it turns out that GWT DnD API is too basic (lacks functionality) or has issues with some browser(s) - we can add gwt-dnd dependency anytime. Thanks, Gilad. [1] http://code.google.com/p/gwt-dnd/ - Original Message - From: Gilad Chaplik gchap...@redhat.com To: Einav Cohen eco...@redhat.com, Alexander Wels aw...@redhat.com, Eyal Edri ee...@redhat.com, Steve Gordon sgor...@redhat.com, Eli Mesika emes...@redhat.com, Otavio Luiz Ferranti otavio.ferra...@eldorado.org.br, Sandro Bonazzola sbona...@redhat.com, Greg Sheremeta gsher...@redhat.com, Doron Fediuck dfedi...@redhat.com, Lior Vernia lver...@redhat.com, engine-devel engine-devel@ovirt.org, Martin Sivak msi...@redhat.com, chuan
Re: [Engine-devel] NUMA support action items
- Original Message - From: Chegu Vinod chegu_vi...@hp.com To: Gilad Chaplik gchap...@redhat.com Cc: Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com, Einav Cohen eco...@redhat.com, Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com, Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com, msi...@redhat.com, Da-huai Tang (Gary, MCXS-CQ) da-huai.t...@hp.com, Malini Rao m...@redhat.com, Eldan Hildesheim ehild...@redhat.com, Doron Fediuck dfedi...@redhat.com, sher...@redhat.com, Alexander Wels aw...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 5:21:49 PM Subject: Re: NUMA support action items On 4/3/2014 7:11 AM, Gilad Chaplik wrote: - Original Message - From: Chegu Vinod chegu_vi...@hp.com To: Xiao-Lei Shi (Bruce, HP Servers-PSC-CQ) xiao-lei@hp.com Cc: Einav Cohen eco...@redhat.com, Shang-Chun Liang (David Liang, HPservers-Core-OE-PSC) shangchun.li...@hp.com, Chuan Liao (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com, msi...@redhat.com, Da-huai Tang (Gary, MCXS-CQ) da-huai.t...@hp.com, Malini Rao m...@redhat.com, Eldan Hildesheim ehild...@redhat.com, Doron Fediuck dfedi...@redhat.com, sher...@redhat.com, Alexander Wels aw...@redhat.com, Gilad Chaplik gchap...@redhat.com Sent: Thursday, April 3, 2014 3:28:03 PM Subject: RE: NUMA support action items Hi Bruce, The virtual NUMA layout in the guest is a very simple one (not multi-level etc). It is generated by qemu+seabios... and there is no relationship with the host NUMA node distances etc. Let us not worry about gathering Virtual NUMA node distances for now. Vinod CC'ing devel list as well. Having said that, I don't see a reason why not to prepare an infrastructure for that (if it's free) for future versions (guest agent will collect vNuma data in some point in time). If you think having this Virtual NUMA topology (along with the virtual numa node *distance* info.) really helps some future use cases then pl. go ahead... Vinod I really don't know. but IMO, me as a user that gets some machine (transparent to the fact it's VM or bare metal), it would be very nice to see the NUMA stats outside of the machine. Thanks, Gilad. Thanks, Gilad. -Original Message- From: Shi, Xiao-Lei (Bruce, HP Servers-PSC-CQ) Sent: Thursday, April 03, 2014 12:41 AM To: Vinod, Chegu Cc: Einav Cohen; Liang, Shang-Chun (David Liang, HPservers-Core-OE-PSC); Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC); msi...@redhat.com; Tang, Da-huai (Gary, MCXS-CQ); Malini Rao; Eldan Hildesheim; Doron Fediuck; sher...@redhat.com; Alexander Wels; Gilad Chaplik Subject: RE: NUMA support action items Hi Vinod, Is it meaningful for us to collect the distance information of vm numa node (maybe in future, not now)? In my understanding, vm numa topology is a simulation of numa topology, since the vcpus are just threads, I don't know how the vm numa node distances are calculated in vm. Is there any relationship between the vNode distances and host node distances? Thanks Best Regards Shi, Xiao-Lei (Bruce) Hewlett-Packard Co., Ltd. HP Servers Core Platform Software China Telephone +86 23 65683093 Mobile +86 18696583447 Email xiao-lei@hp.com -Original Message- From: Vinod, Chegu Sent: Thursday, April 03, 2014 7:18 AM To: Gilad Chaplik Cc: Einav Cohen; Liang, Shang-Chun (David Liang, HPservers-Core-OE-PSC); Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC); msi...@redhat.com; Shi, Xiao-Lei (Bruce, HP Servers-PSC-CQ); Tang, Da-huai (Gary, MCXS-CQ); Malini Rao; Eldan Hildesheim; Doron Fediuck; sher...@redhat.com; Alexander Wels Subject: RE: NUMA support action items Not sure what the correct way to do this isbut here is a suggestion. Let a given host server diagram shown be very generic...i.e. show the N sockets/nodes numbered from 0 thru N-1. Show the amount of memory and the list of CPUs in each of those sockets/nodes. Draw a generic Interconnect fabric [box] in between which all the sockets connect to Ideally ... Under that host diagram we could show the NUMA node distances in text format (as you know this is derived from the numactl -H and then conveyed from VDSM- oVIrt engine etc). That distance info. will tell the user what the distance between a pair of sockets/nodes are (and they can then do what they wish after that :)). Vinod -Original Message- From: Gilad Chaplik [mailto:gchap...@redhat.com] Sent: Wednesday, April 02, 2014 4:09 PM To: Vinod, Chegu Cc: Einav Cohen; Liang, Shang-Chun (David Liang, HPservers-Core-OE-PSC); Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC); msi...@redhat.com; Shi, Xiao-Lei (Bruce, HP Servers-PSC-CQ); Tang, Da-huai (Gary, MCXS-CQ); Malini Rao; Eldan Hildesheim; Doron Fediuck; sher...@redhat.com;
Re: [Engine-devel] vds_dynamic refactor
True but that's no reason to have a bad schema On Apr 3, 2014 5:18 PM, Gilad Chaplik gchap...@redhat.com wrote: - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com Cc: Eli Mesika emes...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 5:16:51 PM Subject: Re: [Engine-devel] vds_dynamic refactor Don't go down that road. Status shouldn't be saved in the db. But anyway statistics is rapidly changing. So it fits... First let's have a notion of caching, then notion of shared caching, then I can start thinking of not going down that road... On Apr 3, 2014 5:14 PM, Gilad Chaplik gchap...@redhat.com wrote: - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Eli Mesika emes...@redhat.com Cc: Gilad Chaplik gchap...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:36:07 PM Subject: Re: [Engine-devel] vds_dynamic refactor I would be happy if we can lose vds_dynamic all together, and just keep vds_static and vds_statistics. Our performance will be happy too ;-) @Liran, status and pending fields are very fragile ones, IMO need separate table. @Eli, iirc, you don't have a problem with adding more tables :-) On Thu, Apr 3, 2014 at 4:33 PM, Eli Mesika emes...@redhat.com wrote: - Original Message - From: Gilad Chaplik gchap...@redhat.com To: Yair Zaslavsky yzasl...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:00:25 PM Subject: Re: [Engine-devel] vds_dynamic refactor - Original Message - From: Yair Zaslavsky yzasl...@redhat.com To: Liran Zelkha liran.zel...@gmail.com Cc: Gilad Chaplik gchap...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 2:12:59 PM Subject: Re: [Engine-devel] vds_dynamic refactor - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 2:04:29 PM Subject: Re: [Engine-devel] vds_dynamic refactor Why not move it to vds_static? +1 on Liran's comment. +1 as well , vds_static is the place for that I would prefer not to add more complexity to the vds tables family. Such complexity may effect performs of queries/views. If you wish, you can create a view on top of vds_static named vds_on_boot for querying of vds on boot info. Yair That means moving almost all of vds_dynamic into vds_static except of memory, pending resources and status (maybe more but not much); and there will not be any db separation between user input and on_boot data. Why we should have such separation ? Thanks, Gilad. On Thu, Apr 3, 2014 at 2:00 PM, Gilad Chaplik gchap...@redhat.com wrote: Hi list, I propose to split vds_dynamic table into 2 tables: - vds_dynamic - vds_on_boot We need a place to put all non-dynamic data that gets updated once the host is booted, and I think dynamic isn't the place for it. In first phase we will put there NUMA host topoplogy, but later on migrate all non-dynamic host data (cpu, os, etc.). thoughts? Thanks, Gilad. ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] vds_dynamic refactor
On Thu, Apr 3, 2014 at 5:33 PM, Gilad Chaplik gchap...@redhat.com wrote: *From: *Liran Zelkha liran.zel...@gmail.com *To: *Gilad Chaplik gchap...@redhat.com *Cc: *Omer Frenkel ofren...@redhat.com, Eli Mesika emes...@redhat.com, engine-devel engine-devel@ovirt.org *Sent: *Thursday, April 3, 2014 5:27:56 PM *Subject: *Re: [Engine-devel] vds_dynamic refactor True but that's no reason to have a bad schema I'm open to new ideas and truly want to understand what is bad? The problem is with both updates and selects. For selects - to get all the information for the VDS we have multiple joins. Adding another one will hurt performance even more. For updates - we have vds_static thats hardly changed. vds_statistics that changes all the time. vds_dynamic is not changed allot - but is updated all the time because of the status. I think it's best to split it to the two existing tables (BTW - relevant for VM as well) On Apr 3, 2014 5:18 PM, Gilad Chaplik gchap...@redhat.com wrote: - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com Cc: Eli Mesika emes...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 5:16:51 PM Subject: Re: [Engine-devel] vds_dynamic refactor Don't go down that road. Status shouldn't be saved in the db. But anyway statistics is rapidly changing. So it fits... First let's have a notion of caching, then notion of shared caching, then I can start thinking of not going down that road... On Apr 3, 2014 5:14 PM, Gilad Chaplik gchap...@redhat.com wrote: - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Eli Mesika emes...@redhat.com Cc: Gilad Chaplik gchap...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:36:07 PM Subject: Re: [Engine-devel] vds_dynamic refactor I would be happy if we can lose vds_dynamic all together, and just keep vds_static and vds_statistics. Our performance will be happy too ;-) @Liran, status and pending fields are very fragile ones, IMO need separate table. @Eli, iirc, you don't have a problem with adding more tables :-) On Thu, Apr 3, 2014 at 4:33 PM, Eli Mesika emes...@redhat.com wrote: - Original Message - From: Gilad Chaplik gchap...@redhat.com To: Yair Zaslavsky yzasl...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:00:25 PM Subject: Re: [Engine-devel] vds_dynamic refactor - Original Message - From: Yair Zaslavsky yzasl...@redhat.com To: Liran Zelkha liran.zel...@gmail.com Cc: Gilad Chaplik gchap...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 2:12:59 PM Subject: Re: [Engine-devel] vds_dynamic refactor - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 2:04:29 PM Subject: Re: [Engine-devel] vds_dynamic refactor Why not move it to vds_static? +1 on Liran's comment. +1 as well , vds_static is the place for that I would prefer not to add more complexity to the vds tables family. Such complexity may effect performs of queries/views. If you wish, you can create a view on top of vds_static named vds_on_boot for querying of vds on boot info. Yair That means moving almost all of vds_dynamic into vds_static except of memory, pending resources and status (maybe more but not much); and there will not be any db separation between user input and on_boot data. Why we should have such separation ? Thanks, Gilad. On Thu, Apr 3, 2014 at 2:00 PM, Gilad Chaplik gchap...@redhat.com wrote: Hi list, I propose to split vds_dynamic table into 2 tables: - vds_dynamic - vds_on_boot We need a place to put all non-dynamic data that gets updated once the host is booted, and I think dynamic isn't the place for it. In first phase we will put there NUMA host topoplogy, but later on migrate all non-dynamic host data (cpu, os, etc.). thoughts? Thanks, Gilad. ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org
Re: [Engine-devel] vds_dynamic refactor
From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com Cc: Omer Frenkel ofren...@redhat.com, Eli Mesika emes...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 5:27:56 PM Subject: Re: [Engine-devel] vds_dynamic refactor True but that's no reason to have a bad schema I'm open to new ideas and truly want to understand what is bad? On Apr 3, 2014 5:18 PM, Gilad Chaplik gchap...@redhat.com wrote: - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com Cc: Eli Mesika emes...@redhat.com , engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 5:16:51 PM Subject: Re: [Engine-devel] vds_dynamic refactor Don't go down that road. Status shouldn't be saved in the db. But anyway statistics is rapidly changing. So it fits... First let's have a notion of caching, then notion of shared caching, then I can start thinking of not going down that road... On Apr 3, 2014 5:14 PM, Gilad Chaplik gchap...@redhat.com wrote: - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Eli Mesika emes...@redhat.com Cc: Gilad Chaplik gchap...@redhat.com , engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:36:07 PM Subject: Re: [Engine-devel] vds_dynamic refactor I would be happy if we can lose vds_dynamic all together, and just keep vds_static and vds_statistics. Our performance will be happy too ;-) @Liran, status and pending fields are very fragile ones, IMO need separate table. @Eli, iirc, you don't have a problem with adding more tables :-) On Thu, Apr 3, 2014 at 4:33 PM, Eli Mesika emes...@redhat.com wrote: - Original Message - From: Gilad Chaplik gchap...@redhat.com To: Yair Zaslavsky yzasl...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:00:25 PM Subject: Re: [Engine-devel] vds_dynamic refactor - Original Message - From: Yair Zaslavsky yzasl...@redhat.com To: Liran Zelkha liran.zel...@gmail.com Cc: Gilad Chaplik gchap...@redhat.com , engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 2:12:59 PM Subject: Re: [Engine-devel] vds_dynamic refactor - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 2:04:29 PM Subject: Re: [Engine-devel] vds_dynamic refactor Why not move it to vds_static? +1 on Liran's comment. +1 as well , vds_static is the place for that I would prefer not to add more complexity to the vds tables family. Such complexity may effect performs of queries/views. If you wish, you can create a view on top of vds_static named vds_on_boot for querying of vds on boot info. Yair That means moving almost all of vds_dynamic into vds_static except of memory, pending resources and status (maybe more but not much); and there will not be any db separation between user input and on_boot data. Why we should have such separation ? Thanks, Gilad. On Thu, Apr 3, 2014 at 2:00 PM, Gilad Chaplik gchap...@redhat.com wrote: Hi list, I propose to split vds_dynamic table into 2 tables: - vds_dynamic - vds_on_boot We need a place to put all non-dynamic data that gets updated once the host is booted, and I think dynamic isn't the place for it. In first phase we will put there NUMA host topoplogy, but later on migrate all non-dynamic host data (cpu, os, etc.). thoughts? Thanks, Gilad. ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] [Devel] vds_dynamic refactor
On Fri, Apr 4, 2014 at 3:15 PM, Itamar Heim ih...@redhat.com wrote: On 04/04/2014 04:30 AM, Liran Zelkha wrote: On Thu, Apr 3, 2014 at 8:40 PM, Gilad Chaplik gchap...@redhat.com mailto:gchap...@redhat.com wrote: - Original Message - From: Liran Zelkha liran.zel...@gmail.com mailto:liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com mailto:gchap...@redhat.com Cc: Omer Frenkel ofren...@redhat.com mailto:ofren...@redhat.com, Eli Mesika emes...@redhat.com mailto:emes...@redhat.com, engine-devel engine-devel@ovirt.org mailto:engine-devel@ovirt.org, de...@linode01.ovirt.org mailto:de...@linode01.ovirt.org Sent: Thursday, April 3, 2014 7:51:39 PM Subject: Re: [Engine-devel] vds_dynamic refactor On Thu, Apr 3, 2014 at 5:33 PM, Gilad Chaplik gchap...@redhat.com mailto:gchap...@redhat.com wrote: *From: *Liran Zelkha liran.zel...@gmail.com mailto:liran.zel...@gmail.com *To: *Gilad Chaplik gchap...@redhat.com mailto:gchap...@redhat.com *Cc: *Omer Frenkel ofren...@redhat.com mailto:ofren...@redhat.com, Eli Mesika emes...@redhat.com mailto:emes...@redhat.com, engine-devel engine-devel@ovirt.org mailto:engine-devel@ovirt.org *Sent: *Thursday, April 3, 2014 5:27:56 PM *Subject: *Re: [Engine-devel] vds_dynamic refactor True but that's no reason to have a bad schema I'm open to new ideas and truly want to understand what is bad? The problem is with both updates and selects. For selects - to get all the information for the VDS we have multiple joins. Adding another one will hurt performance even more. What about creating the VDS list in the db. i.e. your patch [1] about constructing VDS objects in the engine, should occur in the db within a SP, and should be transparent to the server. that will solve the multiple table join. The joins are happening on the server. We have a VDS view that brings in information from many tables and we retrieve rows from it all the time. Just run an explain on a select * from vds and see for yourself. thought the distinction was vds_static is updated by user (configuration), where as vds_dyanmic/statistics is reported from vdsm (slow/fast updates). True - vds_static hardly change. But vds_dynamic is split-brain - it has practically static data, but also has the status - which changes rapidly. And since we have 3 tables, we need to do 3 joins to get VDS (actually we do much more). please remember joining them to one table, also means DWH will ETA all of data each time. today it will only copy statistics if dynamic did not change. I'm not saying joining all 3 tables to 1 table, just make them 2 tables. And since the data is hardly changing - same logic of DWH will stay. ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Devel] [vdsm] vdsm sync meeting minutes April 1st, 2014
Hey all, back from France On 04/02/2014 11:29 AM, dan...@redhat.com wrote: Vdsm sync call April 1st 2014 = - cpopen: - Toni: there's a versioning mismatch: the version in pypi is higher than the one on https://github.com/ficoos/cpopen - Saggi: there shouldn't be any code difference - Yaniv should sync the two. https://github.com/ficoos/cpopen/pull/23 - committed 1.3 tag https://pypi.python.org/pypi?:action=displayname=cpopenversion=1.3 - uploaded latest code - We use two options that are missing from Python3's Popen: umask and deathSignal. - Toni to re-send his Python3 port for cpopen, with tests running on Python3, too. - Saggi to accept it. - Infra team to suggest missing features to Python. - fromani described his attempts of consolidating the two migration-monitoring threads into one. The motivation is a finer way of contolling the migration down time based on progress. Reducing thread numbers per se is not the main motivation. - pep8 has changed. Again. Version 1.5.1 has even more draconic requirements. We can comply with them, or, include our version of pep8/pyflakes/pylint in our git repo. danken shudders at the thought of copying the code, but having a git sub-module is a reasonable compromise. - Infra team to take this task (unless someone else is faster) - Until that happens, one need to use pep8-1.4.6 or --ignore offending errors. - We've been asked to kill the separate vdsm-devel mailing list, and join it into devel@ovirt.org. There's some resistence in the audience, so we'd do it softly. Next week, I'll have vdsm-devel members mass-subscribed to devel@ovirt. If you do NOT want to be subscribed, please send me a private request. Then, for several months, we'd see how it goes, and whether we drown in unrelated Engine chatter. - We had a very (too) heated debate about ignoring failures of setDomainRegularRole() in http://gerrit.ovirt.org/24495/ and http://gerrit.ovirt.org/25424. The pain point is that relying on domain role (master/regular) is faulty by design. We cannot avoid the cases where a pool has more than one domain with a master role written in its metadata. One side argued that oVirt should be fixed to handle this unescapable truth, or at least enumerate the places where Vdsm and Engine, both current and old, depend on master role uniqueness. The other side argued that this is not a priority task, and that we should try to garbage-collect known-bad master roles as a courtesy to people digging into domain metadata, and as a means to lessen the problem until we kill the pool concept in the upcoming version. I hope that I present the debate fairly enough. Dan. -- Yaniv Bronhaim. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [Devel] [Engine-devel] vds_dynamic refactor
On 04/03/2014 07:51 PM, Liran Zelkha wrote: The problem is with both updates and selects. For selects - to get all the information for the VDS we have multiple joins. Adding another one will hurt performance even more. For updates - we have vds_static thats hardly changed. vds_statistics that changes all the time. vds_dynamic is not changed allot - but is updated all the time because of the status. I think it's best to split it to the two existing tables (BTW - relevant for VM as well) but we don't update it unless the status has changed, which is a rare occurance? ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [Devel] NUMA Support - GUI Technical Session
[top posting] Thanks you Vojtech for the elaborate expatiation, I will follow on that and keep you all posted. Thanks, Gilad. - Original Message - From: Vojtech Szocs vsz...@redhat.com To: Gilad Chaplik gchap...@redhat.com Cc: devel@ovirt.org, chegu vinod chegu_vi...@hp.com, aw...@redhat.com Sent: Saturday, April 5, 2014 12:08:09 AM Subject: Re: [Devel] NUMA Support - GUI Technical Session Forgot to send some references on the matter: https://groups.google.com/forum/#!topic/google-web-toolkit/q0JUtbiUR8g http://stackoverflow.com/questions/13681541/best-way-to-implement-a-drag-and-drop-list-in-gwt Regards, Vojtech - Original Message - From: Vojtech Szocs vsz...@redhat.com To: aw...@redhat.com Cc: devel@ovirt.org, chegu vinod chegu_vi...@hp.com Sent: Friday, April 4, 2014 11:06:16 PM Subject: Re: [Devel] NUMA Support - GUI Technical Session Hi Gilad, to my understanding, we already use _HTML5_ Drag'n'Drop support (exposed by GWT API since 2.4) in Setup Host Networks dialog. To utilize HTML5 Drag'n'Drop support in GWT widget, just extend FocusPanel and mark your widget's DOM element as draggable. For example, in UnassignedNetworksPanel constructor: getElement().setDraggable( ... ); addBitlessDomHandler(new DragEnterHandler() { ... }); addBitlessDomHandler(new DragOverHandler() { ... }); addBitlessDomHandler(new DragLeaveHandler() { ... }); addBitlessDomHandler(new DropHandler() { ... }); In other words, GWT already exposes API for working with HTML5 Drag'n'Drop spec, so you don't need any 3rd party libraries. The downside is that HTML5 Drag'n'Drop spec is supported only in recent browsers (but this isn't an issue for us, AFAIK): http://caniuse.com/#feat=dragndrop So in general we have two alternatives: 1, use standard HTML5 Drag'n'Drop spec pros: + no need for 3rd party library + compliant with existing code, i.e. Setup Host Networks cons (not too relevant IMHO): - requires browser support of HTML5 Drag'n'Drop spec - HTML5 Drag'n'Drop spec deals with dragging data, not widgets themselves (no HTML DOM re-parenting after drag finish) 2, use 3rd party gwt-dnd library pros (which I don't think we really need): + emulate Drag'n'Drop support in older browsers + more advanced functionality, i.e. allows dragging widgets so that HTML DOM is dynamically updated cons: - dependency on 3rd party library - this would mean we need to revisit existing code (we should do Drag'n'Drop in one consistent way) It's possible that I might be missing something, but I'd suggest to try using HTML5 Drag'n'Drop via GWT API as the first approach. If we find out that HTML5 Drag'n'Drop doesn't work for us in given browser(s) or if we need extra functionality, we can always add gwt-dnd dependency. Few more comments inline, let me know what you think. Regards, Vojtech - Original Message - From: Alexander Wels aw...@redhat.com To: Gilad Chaplik gchap...@redhat.com Cc: eco...@redhat.com, Vojtech Szocs vsz...@redhat.com, dfedi...@redhat.com, engine-de...@ovirt.org, chegu vinod chegu_vi...@hp.com, lver...@redhat.com Sent: Tuesday, April 1, 2014 7:24:12 PM Subject: Re: NUMA Support - GUI Technical Session On Tuesday, April 01, 2014 11:34:43 AM Gilad Chaplik wrote: Hi all, Here are the resolutions from the meeting: * option 1 1) Use gwt-dnd lib for oVirt's dnd (drag and drop) infrastructure. 2) Come up with a very simple POC that covers all of NUMA-support UX requirements. 3) Either do a POC, or get UX maintainers approval, that moving already existing dnd features to new infrastructure (setup-networks and scheduling policy dialogs) is feasible and possible. +1 for option 1. None of the drag and drop in the application now looks terribly hard. The gwt-dnd library simply makes things easier to control and maintain. Yeah, but the downside is adding 3rd party dependency which predates GWT's support for (API exposure of) HTML5 Drag'n'Drop spec. * option 2 Extract setup network dnd capabilities to a common general infrastructure and use that as an infrastructure. NOTE that setup-networks will not use it in oVirt-3.5. GWT's HTML5 Drag'n'Drop API works directly on DOM element level, it's just a thin API overlay on top of HTML5Drag'n'Drop spec. Do we really need custom DnD infra on top of that? Can't we just use GWT APIs like Element.setDraggable, Drag*Handler, Drop*Handler? I will start with option 1, just need UX team approval that [1] will be added to ovirt-3.5, and be used for dnd for now on. In case I fail to deliver option 1 (with the help and guidance of the UX team) in a
Re: [Devel] [Engine-devel] vds_dynamic refactor
- Original Message - From: Itamar Heim ih...@redhat.com To: Liran Zelkha liran.zel...@gmail.com Cc: Gilad Chaplik gchap...@redhat.com, de...@linode01.ovirt.org, engine-devel engine-de...@ovirt.org Sent: Sunday, April 6, 2014 11:33:12 AM Subject: Re: [Engine-devel] vds_dynamic refactor On 04/06/2014 11:32 AM, Liran Zelkha wrote: On Sun, Apr 6, 2014 at 11:22 AM, Itamar Heim ih...@redhat.com mailto:ih...@redhat.com wrote: On 04/03/2014 07:51 PM, Liran Zelkha wrote: The problem is with both updates and selects. For selects - to get all the information for the VDS we have multiple joins. Adding another one will hurt performance even more. For updates - we have vds_static thats hardly changed. vds_statistics that changes all the time. vds_dynamic is not changed allot - but is updated all the time because of the status. I think it's best to split it to the two existing tables (BTW - relevant for VM as well) but we don't update it unless the status has changed, which is a rare occurance? Actually - no. We can definitely see times we are updating vds_dynamic with no reason at all. I tried to create patches for that - but it happens from many different places in the code. what would be updated vds_dyanmic for status not originating in update run time info? We have separate DB flows for that (updateStatus and updatePartialVdsDynamicCalc and more in VdsDynamicDAODbFacadeImpl). A question: do you know if we update status in updateVdsDynamic? :-) not sure but I found a possible race for pending resources (cpu, mem), LOL :-) Still holds my original thought for having vds_on_boot. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [Devel] [Engine-devel] vds_dynamic refactor
- Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com Cc: Itamar Heim ih...@redhat.com, de...@linode01.ovirt.org, engine-devel engine-de...@ovirt.org Sent: Sunday, April 6, 2014 3:26:24 PM Subject: Re: [Engine-devel] vds_dynamic refactor On Sun, Apr 6, 2014 at 3:18 PM, Gilad Chaplik gchap...@redhat.com wrote: - Original Message - From: Itamar Heim ih...@redhat.com To: Liran Zelkha liran.zel...@gmail.com Cc: Gilad Chaplik gchap...@redhat.com, de...@linode01.ovirt.org, engine-devel engine-de...@ovirt.org Sent: Sunday, April 6, 2014 11:33:12 AM Subject: Re: [Engine-devel] vds_dynamic refactor On 04/06/2014 11:32 AM, Liran Zelkha wrote: On Sun, Apr 6, 2014 at 11:22 AM, Itamar Heim ih...@redhat.com mailto:ih...@redhat.com wrote: On 04/03/2014 07:51 PM, Liran Zelkha wrote: The problem is with both updates and selects. For selects - to get all the information for the VDS we have multiple joins. Adding another one will hurt performance even more. For updates - we have vds_static thats hardly changed. vds_statistics that changes all the time. vds_dynamic is not changed allot - but is updated all the time because of the status. I think it's best to split it to the two existing tables (BTW - relevant for VM as well) but we don't update it unless the status has changed, which is a rare occurance? Actually - no. We can definitely see times we are updating vds_dynamic with no reason at all. I tried to create patches for that - but it happens from many different places in the code. what would be updated vds_dyanmic for status not originating in update run time info? We have separate DB flows for that (updateStatus and updatePartialVdsDynamicCalc and more in VdsDynamicDAODbFacadeImpl). A question: do you know if we update status in updateVdsDynamic? :-) not sure but I found a possible race for pending resources (cpu, mem), LOL :-) I think we do but not sure. Will check. Of course it is, that was a rhetorical question :-) (a lot of emoticons and LOLs ;-)) Still holds my original thought for having vds_on_boot. Let's talk f2f on Tuesday? I'd prefer to reach conclusions here, I'd like everyone to be involved in a root issue like this one. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [Devel] [Engine-devel] vds_dynamic refactor
Joining in... From my point of view, in real life a user should have that many VDSs on one Engine (from a DB point of view). Modern DB system handles tables with millions of records and many relations, Do we really have a performance issue here? We could prefer a more easy to maintain implantation in this case over DB performance - Original Message - From: Gilad Chaplik gchap...@redhat.com To: Liran Zelkha liran.zel...@gmail.com Cc: de...@linode01.ovirt.org, engine-devel engine-de...@ovirt.org Sent: Sunday, April 6, 2014 3:32:26 PM Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com Cc: Itamar Heim ih...@redhat.com, de...@linode01.ovirt.org, engine-devel engine-de...@ovirt.org Sent: Sunday, April 6, 2014 3:26:24 PM Subject: Re: [Engine-devel] vds_dynamic refactor On Sun, Apr 6, 2014 at 3:18 PM, Gilad Chaplik gchap...@redhat.com wrote: - Original Message - From: Itamar Heim ih...@redhat.com To: Liran Zelkha liran.zel...@gmail.com Cc: Gilad Chaplik gchap...@redhat.com, de...@linode01.ovirt.org, engine-devel engine-de...@ovirt.org Sent: Sunday, April 6, 2014 11:33:12 AM Subject: Re: [Engine-devel] vds_dynamic refactor On 04/06/2014 11:32 AM, Liran Zelkha wrote: On Sun, Apr 6, 2014 at 11:22 AM, Itamar Heim ih...@redhat.com mailto:ih...@redhat.com wrote: On 04/03/2014 07:51 PM, Liran Zelkha wrote: The problem is with both updates and selects. For selects - to get all the information for the VDS we have multiple joins. Adding another one will hurt performance even more. For updates - we have vds_static thats hardly changed. vds_statistics that changes all the time. vds_dynamic is not changed allot - but is updated all the time because of the status. I think it's best to split it to the two existing tables (BTW - relevant for VM as well) but we don't update it unless the status has changed, which is a rare occurance? Actually - no. We can definitely see times we are updating vds_dynamic with no reason at all. I tried to create patches for that - but it happens from many different places in the code. what would be updated vds_dyanmic for status not originating in update run time info? We have separate DB flows for that (updateStatus and updatePartialVdsDynamicCalc and more in VdsDynamicDAODbFacadeImpl). A question: do you know if we update status in updateVdsDynamic? :-) not sure but I found a possible race for pending resources (cpu, mem), LOL :-) I think we do but not sure. Will check. Of course it is, that was a rhetorical question :-) (a lot of emoticons and LOLs ;-)) Still holds my original thought for having vds_on_boot. Let's talk f2f on Tuesday? I'd prefer to reach conclusions here, I'd like everyone to be involved in a root issue like this one. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [Devel] [Engine-devel] vds_dynamic refactor
- Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com Cc: Kobi Ianko k...@redhat.com, de...@linode01.ovirt.org, engine-devel engine-de...@ovirt.org Sent: Sunday, April 6, 2014 8:51:02 PM Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor On Sun, Apr 6, 2014 at 6:32 PM, Gilad Chaplik gchap...@redhat.com wrote: - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Kobi Ianko k...@redhat.com Cc: Gilad Chaplik gchap...@redhat.com, de...@linode01.ovirt.org, engine-devel engine-de...@ovirt.org Sent: Sunday, April 6, 2014 3:40:13 PM Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor On Sun, Apr 6, 2014 at 3:37 PM, Kobi Ianko k...@redhat.com wrote: Joining in... From my point of view, in real life a user should have that many VDSs on one Engine (from a DB point of view). Modern DB system handles tables with millions of records and many relations, Do we really have a performance issue here? We could prefer a more easy to maintain implantation in this case over DB performance Yes we do. We make many queries on the VDS view, which is a VERY complex view. Actually I quite agree with Kobi, what is the plan for VMs? why do we start with VDS... what is the biggest deploy do you know of? We start with VDS because in an idle system, with 200 hosts and several thousands VMs, this is what you get as the top queries against the database. Look at how many times getvds is called. [image: Inline image 1] BTW - the second query is an example of abusing the dynamic query mechanism. The 4th query (an update command) is a set of useless update_vds_dynamic commands. For reference, the explain plan of get VDS is something like this: QUERY PLAN --- Nested Loop (cost=9.30..46.75 rows=6 width=9060) (actual time=0.063..0.068 rows=1 loops=1) Join Filter: (vds_static.vds_id = vds_statistics.vds_id) - Seq Scan on vds_statistics (cost=0.00..1.01 rows=1 width=109) (actual time=0.008..0.008 rows=1 loops=1) - Nested Loop (cost=9.30..45.64 rows=6 width=8983) (actual time=0.048..0.052 rows=1 loops=1) Join Filter: (vds_groups.vds_group_id = vds_static.vds_group_id) - Nested Loop Left Join (cost=0.00..9.29 rows=1 width=1389) (actual time=0.013..0.013 rows=1 loops=1) - Seq Scan on vds_groups (cost=0.00..1.01 rows=1 width=1271) (actual time=0.003..0.003 rows=1 loops=1) - Index Scan using pk_storage_pool on storage_pool (cost=0.00..8.27 rows=1 width=134) (actual time=0.008..0.008 rows=1 loops=1) Index Cond: (vds_groups.storage_pool_id = id) - Hash Right Join (cost=9.30..36.28 rows=6 width=7610) (actual time=0.033..0.037 rows=1 loops=1) Hash Cond: (vds_spm_id_map.vds_id = vds_static.vds_id) - Seq Scan on vds_spm_id_map (cost=0.00..22.30 rows=1230 width=20) (actual time=0.003..0.003 rows=1 loops=1) - Hash (cost=9.29..9.29 rows=1 width=7606) (actual time=0.019..0.019 rows=1 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 2kB - Nested Loop (cost=0.00..9.29 rows=1 width=7606) (actual time=0.012..0.013 rows=1 loops=1) - Seq Scan on vds_dynamic (cost=0.00..1.01 rows=1 width=1895) (actual time=0.006..0.006 rows=1 loops=1) - Index Scan using pk_vds_static on vds_static (cost=0.00..8.27 rows=1 width=5711) (actual time=0.005..0.006 rows=1 loops=1) Index Cond: (vds_id = vds_dynamic.vds_id) Total runtime: 0.299 ms (19 rows) It's terrible. Adding any additional join will make this worse. Please don't add any more tables... Thank you for the detailed explanation, my comments: * a very long time isn't an argument for not adding another table (should be neglectable); currently we have an unrelated problem, we need to solve it. * We start with VDS because in an idle system, with 200 hosts and several thousands VMs, this is what you get as the top queries against the database. so, if fetching VMs takes 10 minutes? and its get called a single time? * you didn't reply on my of my suggestion of constructing the VDS records in the DB without using joins. - Original Message - From: Gilad Chaplik gchap...@redhat.com To: Liran Zelkha liran.zel...@gmail.com Cc: de...@linode01.ovirt.org, engine-devel engine-de...@ovirt.org Sent: Sunday, April 6, 2014 3:32:26 PM Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com
Re: [Devel] Vdsm functional tests
On Sun, Apr 06, 2014 at 11:18:19AM +0300, ybronhei wrote: On 04/03/2014 07:31 PM, Douglas Schilling Landgraf wrote: On 04/03/2014 11:08 AM, Dan Kenigsberg wrote: Functional tests are intended to verify that a running Vdsm instance does what it should, when treated as a black box, over its public API. They should be comprehensive and representative of a typical field usage of Vdsm. It is a sin to break such a test - but we must be able to know when such a sin is committed. We currently have the following functional tests modules: - sosPluginTests.py supervdsmFuncTests.py Sure, count with me. any news about it ? need my help around it? Douglas still owes me a time estimate on when this be done. supervdsmFuncTests.py doesn't really check much. we need to add much more logic there if we actually want to test the communication between vdsm and supervdsm (not sure if its really required.. its like checking calls to libvirt or sanlock or testing api calls) At the moment supervdsmFuncTests do test that supervdsm is reponsive and that supervdsm.getProxy() works. It's not like testing libvirt api calls since supervdsm is inside our tree. So it's like testing libvirt api calls - within the libvirt project. I would embrace more smarter logic into the test - but I'm not sure what you have in mind. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [Devel] Vdsm functional tests
On 04/03/2014 12:24 PM, Dan Kenigsberg wrote: On Thu, Apr 03, 2014 at 12:31:44PM -0400, Douglas Schilling Landgraf wrote: On 04/03/2014 11:08 AM, Dan Kenigsberg wrote: Functional tests are intended to verify that a running Vdsm instance does what it should, when treated as a black box, over its public API. They should be comprehensive and representative of a typical field usage of Vdsm. It is a sin to break such a test - but we must be able to know when such a sin is committed. We currently have the following functional tests modules: - sosPluginTests.py supervdsmFuncTests.py Sure, count with me. Thanks! When do you think you could write a job similar to http://jenkins.ovirt.org/view/By%20Project/view/vdsm/job/vdsm_network_functional_tests/configure running whenever there's a change in the modules relevant to sosPluginTests and supervdsmFuncTests? Hi, Instead of to split by domain like (network, infra, storage), why not have a single functional test job? If something fail, it should trigger the volunteers. For while, I started a creation of infra based on the above one. http://jenkins.ovirt.org/job/vdsm_infra_functional_tests -- Cheers Douglas ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [Devel] [Engine-devel] vds_dynamic refactor
On Sun, Apr 6, 2014 at 11:03 PM, Gilad Chaplik gchap...@redhat.com wrote: - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com Cc: Kobi Ianko k...@redhat.com, de...@linode01.ovirt.org, engine-devel engine-de...@ovirt.org Sent: Sunday, April 6, 2014 8:51:02 PM Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor On Sun, Apr 6, 2014 at 6:32 PM, Gilad Chaplik gchap...@redhat.com wrote: - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Kobi Ianko k...@redhat.com Cc: Gilad Chaplik gchap...@redhat.com, de...@linode01.ovirt.org, engine-devel engine-de...@ovirt.org Sent: Sunday, April 6, 2014 3:40:13 PM Subject: Re: [Devel] [Engine-devel] vds_dynamic refactor On Sun, Apr 6, 2014 at 3:37 PM, Kobi Ianko k...@redhat.com wrote: Joining in... From my point of view, in real life a user should have that many VDSs on one Engine (from a DB point of view). Modern DB system handles tables with millions of records and many relations, Do we really have a performance issue here? We could prefer a more easy to maintain implantation in this case over DB performance Yes we do. We make many queries on the VDS view, which is a VERY complex view. Actually I quite agree with Kobi, what is the plan for VMs? why do we start with VDS... what is the biggest deploy do you know of? We start with VDS because in an idle system, with 200 hosts and several thousands VMs, this is what you get as the top queries against the database. Look at how many times getvds is called. [image: Inline image 1] BTW - the second query is an example of abusing the dynamic query mechanism. The 4th query (an update command) is a set of useless update_vds_dynamic commands. For reference, the explain plan of get VDS is something like this: QUERY PLAN --- Nested Loop (cost=9.30..46.75 rows=6 width=9060) (actual time=0.063..0.068 rows=1 loops=1) Join Filter: (vds_static.vds_id = vds_statistics.vds_id) - Seq Scan on vds_statistics (cost=0.00..1.01 rows=1 width=109) (actual time=0.008..0.008 rows=1 loops=1) - Nested Loop (cost=9.30..45.64 rows=6 width=8983) (actual time=0.048..0.052 rows=1 loops=1) Join Filter: (vds_groups.vds_group_id = vds_static.vds_group_id) - Nested Loop Left Join (cost=0.00..9.29 rows=1 width=1389) (actual time=0.013..0.013 rows=1 loops=1) - Seq Scan on vds_groups (cost=0.00..1.01 rows=1 width=1271) (actual time=0.003..0.003 rows=1 loops=1) - Index Scan using pk_storage_pool on storage_pool (cost=0.00..8.27 rows=1 width=134) (actual time=0.008..0.008 rows=1 loops=1) Index Cond: (vds_groups.storage_pool_id = id) - Hash Right Join (cost=9.30..36.28 rows=6 width=7610) (actual time=0.033..0.037 rows=1 loops=1) Hash Cond: (vds_spm_id_map.vds_id = vds_static.vds_id) - Seq Scan on vds_spm_id_map (cost=0.00..22.30 rows=1230 width=20) (actual time=0.003..0.003 rows=1 loops=1) - Hash (cost=9.29..9.29 rows=1 width=7606) (actual time=0.019..0.019 rows=1 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 2kB - Nested Loop (cost=0.00..9.29 rows=1 width=7606) (actual time=0.012..0.013 rows=1 loops=1) - Seq Scan on vds_dynamic (cost=0.00..1.01 rows=1 width=1895) (actual time=0.006..0.006 rows=1 loops=1) - Index Scan using pk_vds_static on vds_static (cost=0.00..8.27 rows=1 width=5711) (actual time=0.005..0.006 rows=1 loops=1) Index Cond: (vds_id = vds_dynamic.vds_id) Total runtime: 0.299 ms (19 rows) It's terrible. Adding any additional join will make this worse. Please don't add any more tables... Thank you for the detailed explanation, my comments: * a very long time isn't an argument for not adding another table (should be neglectable); currently we have an unrelated problem, we need to solve it. Of course it is. A very long time for a query that you execute many times is THE factor. Who said the join has no performance effect? Have you tested it? Under load? Under many writes/updates? * We start with VDS because in an idle system, with 200 hosts and several thousands VMs, this is what you get as the top queries against the database. so, if fetching VMs takes 10 minutes? and its get called a single time? Where do you see 10 minutes? If you are looking at the red bar it's the inherent time - total query time * number of queries. * you didn't reply on my of my suggestion of constructing