Re: [Engine-devel] vds_dynamic refactor
Don't go down that road. Status shouldn't be saved in the db. But anyway statistics is rapidly changing. So it fits... On Apr 3, 2014 5:14 PM, Gilad Chaplik gchap...@redhat.com wrote: - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Eli Mesika emes...@redhat.com Cc: Gilad Chaplik gchap...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:36:07 PM Subject: Re: [Engine-devel] vds_dynamic refactor I would be happy if we can lose vds_dynamic all together, and just keep vds_static and vds_statistics. Our performance will be happy too ;-) @Liran, status and pending fields are very fragile ones, IMO need separate table. @Eli, iirc, you don't have a problem with adding more tables :-) On Thu, Apr 3, 2014 at 4:33 PM, Eli Mesika emes...@redhat.com wrote: - Original Message - From: Gilad Chaplik gchap...@redhat.com To: Yair Zaslavsky yzasl...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:00:25 PM Subject: Re: [Engine-devel] vds_dynamic refactor - Original Message - From: Yair Zaslavsky yzasl...@redhat.com To: Liran Zelkha liran.zel...@gmail.com Cc: Gilad Chaplik gchap...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 2:12:59 PM Subject: Re: [Engine-devel] vds_dynamic refactor - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 2:04:29 PM Subject: Re: [Engine-devel] vds_dynamic refactor Why not move it to vds_static? +1 on Liran's comment. +1 as well , vds_static is the place for that I would prefer not to add more complexity to the vds tables family. Such complexity may effect performs of queries/views. If you wish, you can create a view on top of vds_static named vds_on_boot for querying of vds on boot info. Yair That means moving almost all of vds_dynamic into vds_static except of memory, pending resources and status (maybe more but not much); and there will not be any db separation between user input and on_boot data. Why we should have such separation ? Thanks, Gilad. On Thu, Apr 3, 2014 at 2:00 PM, Gilad Chaplik gchap...@redhat.com wrote: Hi list, I propose to split vds_dynamic table into 2 tables: - vds_dynamic - vds_on_boot We need a place to put all non-dynamic data that gets updated once the host is booted, and I think dynamic isn't the place for it. In first phase we will put there NUMA host topoplogy, but later on migrate all non-dynamic host data (cpu, os, etc.). thoughts? Thanks, Gilad. ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] vds_dynamic refactor
True but that's no reason to have a bad schema On Apr 3, 2014 5:18 PM, Gilad Chaplik gchap...@redhat.com wrote: - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com Cc: Eli Mesika emes...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 5:16:51 PM Subject: Re: [Engine-devel] vds_dynamic refactor Don't go down that road. Status shouldn't be saved in the db. But anyway statistics is rapidly changing. So it fits... First let's have a notion of caching, then notion of shared caching, then I can start thinking of not going down that road... On Apr 3, 2014 5:14 PM, Gilad Chaplik gchap...@redhat.com wrote: - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Eli Mesika emes...@redhat.com Cc: Gilad Chaplik gchap...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:36:07 PM Subject: Re: [Engine-devel] vds_dynamic refactor I would be happy if we can lose vds_dynamic all together, and just keep vds_static and vds_statistics. Our performance will be happy too ;-) @Liran, status and pending fields are very fragile ones, IMO need separate table. @Eli, iirc, you don't have a problem with adding more tables :-) On Thu, Apr 3, 2014 at 4:33 PM, Eli Mesika emes...@redhat.com wrote: - Original Message - From: Gilad Chaplik gchap...@redhat.com To: Yair Zaslavsky yzasl...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:00:25 PM Subject: Re: [Engine-devel] vds_dynamic refactor - Original Message - From: Yair Zaslavsky yzasl...@redhat.com To: Liran Zelkha liran.zel...@gmail.com Cc: Gilad Chaplik gchap...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 2:12:59 PM Subject: Re: [Engine-devel] vds_dynamic refactor - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 2:04:29 PM Subject: Re: [Engine-devel] vds_dynamic refactor Why not move it to vds_static? +1 on Liran's comment. +1 as well , vds_static is the place for that I would prefer not to add more complexity to the vds tables family. Such complexity may effect performs of queries/views. If you wish, you can create a view on top of vds_static named vds_on_boot for querying of vds on boot info. Yair That means moving almost all of vds_dynamic into vds_static except of memory, pending resources and status (maybe more but not much); and there will not be any db separation between user input and on_boot data. Why we should have such separation ? Thanks, Gilad. On Thu, Apr 3, 2014 at 2:00 PM, Gilad Chaplik gchap...@redhat.com wrote: Hi list, I propose to split vds_dynamic table into 2 tables: - vds_dynamic - vds_on_boot We need a place to put all non-dynamic data that gets updated once the host is booted, and I think dynamic isn't the place for it. In first phase we will put there NUMA host topoplogy, but later on migrate all non-dynamic host data (cpu, os, etc.). thoughts? Thanks, Gilad. ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] vds_dynamic refactor
On Thu, Apr 3, 2014 at 5:33 PM, Gilad Chaplik gchap...@redhat.com wrote: *From: *Liran Zelkha liran.zel...@gmail.com *To: *Gilad Chaplik gchap...@redhat.com *Cc: *Omer Frenkel ofren...@redhat.com, Eli Mesika emes...@redhat.com, engine-devel engine-devel@ovirt.org *Sent: *Thursday, April 3, 2014 5:27:56 PM *Subject: *Re: [Engine-devel] vds_dynamic refactor True but that's no reason to have a bad schema I'm open to new ideas and truly want to understand what is bad? The problem is with both updates and selects. For selects - to get all the information for the VDS we have multiple joins. Adding another one will hurt performance even more. For updates - we have vds_static thats hardly changed. vds_statistics that changes all the time. vds_dynamic is not changed allot - but is updated all the time because of the status. I think it's best to split it to the two existing tables (BTW - relevant for VM as well) On Apr 3, 2014 5:18 PM, Gilad Chaplik gchap...@redhat.com wrote: - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com Cc: Eli Mesika emes...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 5:16:51 PM Subject: Re: [Engine-devel] vds_dynamic refactor Don't go down that road. Status shouldn't be saved in the db. But anyway statistics is rapidly changing. So it fits... First let's have a notion of caching, then notion of shared caching, then I can start thinking of not going down that road... On Apr 3, 2014 5:14 PM, Gilad Chaplik gchap...@redhat.com wrote: - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Eli Mesika emes...@redhat.com Cc: Gilad Chaplik gchap...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:36:07 PM Subject: Re: [Engine-devel] vds_dynamic refactor I would be happy if we can lose vds_dynamic all together, and just keep vds_static and vds_statistics. Our performance will be happy too ;-) @Liran, status and pending fields are very fragile ones, IMO need separate table. @Eli, iirc, you don't have a problem with adding more tables :-) On Thu, Apr 3, 2014 at 4:33 PM, Eli Mesika emes...@redhat.com wrote: - Original Message - From: Gilad Chaplik gchap...@redhat.com To: Yair Zaslavsky yzasl...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:00:25 PM Subject: Re: [Engine-devel] vds_dynamic refactor - Original Message - From: Yair Zaslavsky yzasl...@redhat.com To: Liran Zelkha liran.zel...@gmail.com Cc: Gilad Chaplik gchap...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 2:12:59 PM Subject: Re: [Engine-devel] vds_dynamic refactor - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 2:04:29 PM Subject: Re: [Engine-devel] vds_dynamic refactor Why not move it to vds_static? +1 on Liran's comment. +1 as well , vds_static is the place for that I would prefer not to add more complexity to the vds tables family. Such complexity may effect performs of queries/views. If you wish, you can create a view on top of vds_static named vds_on_boot for querying of vds on boot info. Yair That means moving almost all of vds_dynamic into vds_static except of memory, pending resources and status (maybe more but not much); and there will not be any db separation between user input and on_boot data. Why we should have such separation ? Thanks, Gilad. On Thu, Apr 3, 2014 at 2:00 PM, Gilad Chaplik gchap...@redhat.com wrote: Hi list, I propose to split vds_dynamic table into 2 tables: - vds_dynamic - vds_on_boot We need a place to put all non-dynamic data that gets updated once the host is booted, and I think dynamic isn't the place for it. In first phase we will put there NUMA host topoplogy, but later on migrate all non-dynamic host data (cpu, os, etc.). thoughts? Thanks, Gilad. ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org
Re: [Engine-devel] [Devel] vds_dynamic refactor
On Fri, Apr 4, 2014 at 3:15 PM, Itamar Heim ih...@redhat.com wrote: On 04/04/2014 04:30 AM, Liran Zelkha wrote: On Thu, Apr 3, 2014 at 8:40 PM, Gilad Chaplik gchap...@redhat.com mailto:gchap...@redhat.com wrote: - Original Message - From: Liran Zelkha liran.zel...@gmail.com mailto:liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com mailto:gchap...@redhat.com Cc: Omer Frenkel ofren...@redhat.com mailto:ofren...@redhat.com, Eli Mesika emes...@redhat.com mailto:emes...@redhat.com, engine-devel engine-devel@ovirt.org mailto:engine-devel@ovirt.org, de...@linode01.ovirt.org mailto:de...@linode01.ovirt.org Sent: Thursday, April 3, 2014 7:51:39 PM Subject: Re: [Engine-devel] vds_dynamic refactor On Thu, Apr 3, 2014 at 5:33 PM, Gilad Chaplik gchap...@redhat.com mailto:gchap...@redhat.com wrote: *From: *Liran Zelkha liran.zel...@gmail.com mailto:liran.zel...@gmail.com *To: *Gilad Chaplik gchap...@redhat.com mailto:gchap...@redhat.com *Cc: *Omer Frenkel ofren...@redhat.com mailto:ofren...@redhat.com, Eli Mesika emes...@redhat.com mailto:emes...@redhat.com, engine-devel engine-devel@ovirt.org mailto:engine-devel@ovirt.org *Sent: *Thursday, April 3, 2014 5:27:56 PM *Subject: *Re: [Engine-devel] vds_dynamic refactor True but that's no reason to have a bad schema I'm open to new ideas and truly want to understand what is bad? The problem is with both updates and selects. For selects - to get all the information for the VDS we have multiple joins. Adding another one will hurt performance even more. What about creating the VDS list in the db. i.e. your patch [1] about constructing VDS objects in the engine, should occur in the db within a SP, and should be transparent to the server. that will solve the multiple table join. The joins are happening on the server. We have a VDS view that brings in information from many tables and we retrieve rows from it all the time. Just run an explain on a select * from vds and see for yourself. thought the distinction was vds_static is updated by user (configuration), where as vds_dyanmic/statistics is reported from vdsm (slow/fast updates). True - vds_static hardly change. But vds_dynamic is split-brain - it has practically static data, but also has the status - which changes rapidly. And since we have 3 tables, we need to do 3 joins to get VDS (actually we do much more). please remember joining them to one table, also means DWH will ETA all of data each time. today it will only copy statistics if dynamic did not change. I'm not saying joining all 3 tables to 1 table, just make them 2 tables. And since the data is hardly changing - same logic of DWH will stay. ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] vds_dynamic refactor
Why not move it to vds_static? On Thu, Apr 3, 2014 at 2:00 PM, Gilad Chaplik gchap...@redhat.com wrote: Hi list, I propose to split vds_dynamic table into 2 tables: - vds_dynamic - vds_on_boot We need a place to put all non-dynamic data that gets updated once the host is booted, and I think dynamic isn't the place for it. In first phase we will put there NUMA host topoplogy, but later on migrate all non-dynamic host data (cpu, os, etc.). thoughts? Thanks, Gilad. ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] vds_dynamic refactor
I would be happy if we can lose vds_dynamic all together, and just keep vds_static and vds_statistics. Our performance will be happy too ;-) On Thu, Apr 3, 2014 at 4:33 PM, Eli Mesika emes...@redhat.com wrote: - Original Message - From: Gilad Chaplik gchap...@redhat.com To: Yair Zaslavsky yzasl...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 4:00:25 PM Subject: Re: [Engine-devel] vds_dynamic refactor - Original Message - From: Yair Zaslavsky yzasl...@redhat.com To: Liran Zelkha liran.zel...@gmail.com Cc: Gilad Chaplik gchap...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 2:12:59 PM Subject: Re: [Engine-devel] vds_dynamic refactor - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Gilad Chaplik gchap...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Thursday, April 3, 2014 2:04:29 PM Subject: Re: [Engine-devel] vds_dynamic refactor Why not move it to vds_static? +1 on Liran's comment. +1 as well , vds_static is the place for that I would prefer not to add more complexity to the vds tables family. Such complexity may effect performs of queries/views. If you wish, you can create a view on top of vds_static named vds_on_boot for querying of vds on boot info. Yair That means moving almost all of vds_dynamic into vds_static except of memory, pending resources and status (maybe more but not much); and there will not be any db separation between user input and on_boot data. Why we should have such separation ? Thanks, Gilad. On Thu, Apr 3, 2014 at 2:00 PM, Gilad Chaplik gchap...@redhat.com wrote: Hi list, I propose to split vds_dynamic table into 2 tables: - vds_dynamic - vds_on_boot We need a place to put all non-dynamic data that gets updated once the host is booted, and I think dynamic isn't the place for it. In first phase we will put there NUMA host topoplogy, but later on migrate all non-dynamic host data (cpu, os, etc.). thoughts? Thanks, Gilad. ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Schema upgrade failure on master
Happens to me too, on engine_dao_tests. DB is not empty, one line: engine_dao_tests= select * from event_subscriber; subscriber_id | event_up_name | method_id | method_address | tag_name | notification_method --+---+---++--+- 9bf7c640-b620-456f-a550-0348f366544a | TestRun | 1 | | testrun | (1 row) On Tue, Mar 4, 2014 at 7:10 AM, Martin Perina mper...@redhat.com wrote: Hi Adam, I didn't notice any problem with this script. Was the database you tried to upgrade empty? If not could you please send me contents of your event_subscriber table? Thanks Martin Perina - Original Message - From: Adam Litke ali...@redhat.com To: engine-devel@ovirt.org Sent: Monday, March 3, 2014 11:26:28 PM Subject: [Engine-devel] Schema upgrade failure on master Hi, I've recently rebased to master and it looks like the 03_05_0050_event_notification_methods.sql script is failing on schema upgrade. Is this a bug or am I doing something wrong? To upgrade I did the normal proceedure with my development installation: make install-dev ... ~/ovirt/bin/engine-setup Got this result in the log file: psql:/home/alitke/ovirt-**FILTERED**/share/ovirt-**FILTERED**/dbscripts/upgrade/03_05_0050_event_notification_methods.sql:10: ERROR: column notification_method contains null values FATAL: Cannot execute sql command: --file=/home/alitke/ovirt-**FILTERED**/share/ovirt-**FILTERED**/dbscripts/upgrade/03_05_0050_event_notification_methods.sql 2014-03-03 17:20:34 DEBUG otopi.context context._executeMethod:152 method exception Traceback (most recent call last): File /usr/lib/python2.7/site-packages/otopi/context.py, line 142, in _executeMethod method['method']() File /home/alitke/ovirt-**FILTERED**/share/ovirt-**FILTERED**/setup/bin/../plugins/ovirt-**FILTERED**-setup/ovirt-**FILTERED**/db/schema.py, line 280, in _misc osetupcons.DBEnv.PGPASS_FILE File /usr/lib/python2.7/site-packages/otopi/plugin.py, line 451, in execute command=args[0], RuntimeError: Command '/home/alitke/ovirt-**FILTERED**/share/ovirt-**FILTERED**/dbscripts/schema.sh' failed to execute -- Adam Litke ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Schema upgrade failure on master
had the same issue. Run from psql: truncate table event_subscriber; Then run the script. Note that this will delete all the rows from your table... On Tue, Mar 4, 2014 at 3:18 PM, Adam Litke ali...@redhat.com wrote: On 04/03/14 00:10 -0500, Martin Perina wrote: Hi Adam, I didn't notice any problem with this script. Was the database you tried to upgrade empty? If not could you please send me contents of your event_subscriber table? engine= select * from event_subscriber ; subscriber_id |event_up_name| method_id | method_address | tag_name --+- +--- +--+-- fdfc627c-d875-11e0-90f0-83df133b58cc | VM_SET_TICKET | 0 | ali...@brewer.alitke.net | fdfc627c-d875-11e0-90f0-83df133b58cc | VM_DOWN_ERROR | 0 | ali...@brewer.alitke.net | fdfc627c-d875-11e0-90f0-83df133b58cc | VDS_INITIATED_RUN_VM_FAILED | 0 | ali...@brewer.alitke.net | (3 rows) Thanks Martin Perina - Original Message - From: Adam Litke ali...@redhat.com To: engine-devel@ovirt.org Sent: Monday, March 3, 2014 11:26:28 PM Subject: [Engine-devel] Schema upgrade failure on master Hi, I've recently rebased to master and it looks like the 03_05_0050_event_notification_methods.sql script is failing on schema upgrade. Is this a bug or am I doing something wrong? To upgrade I did the normal proceedure with my development installation: make install-dev ... ~/ovirt/bin/engine-setup Got this result in the log file: psql:/home/alitke/ovirt-**FILTERED**/share/ovirt-** FILTERED**/dbscripts/upgrade/03_05_0050_event_notification_ methods.sql:10: ERROR: column notification_method contains null values FATAL: Cannot execute sql command: --file=/home/alitke/ovirt-**FILTERED**/share/ovirt-** FILTERED**/dbscripts/upgrade/03_05_0050_event_notification_methods.sql 2014-03-03 17:20:34 DEBUG otopi.context context._executeMethod:152 method exception Traceback (most recent call last): File /usr/lib/python2.7/site-packages/otopi/context.py, line 142, in _executeMethod method['method']() File /home/alitke/ovirt-**FILTERED**/share/ovirt-** FILTERED**/setup/bin/../plugins/ovirt-**FILTERED**- setup/ovirt-**FILTERED**/db/schema.py, line 280, in _misc osetupcons.DBEnv.PGPASS_FILE File /usr/lib/python2.7/site-packages/otopi/plugin.py, line 451, in execute command=args[0], RuntimeError: Command '/home/alitke/ovirt-**FILTERED**/share/ovirt-** FILTERED**/dbscripts/schema.sh' failed to execute -- Adam Litke ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel -- Adam Litke ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
[Engine-devel] Usage of Dynamic Queries
Hi All, As part of an ongoing effort to improve oVirt performance, we're trying to reduce the number of Dynamic Queries usage in the code, and replace it with Postgres stored procedures. You wouldn't believe the performance gain. Please consider before making use of the Dynamic Queries feature - it should ONLY be used for places where users can enter search queries - and not for any other usage. Thanks. Liran ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] which data structure is better
Hi Please don't add rapidly changing data to VDSDynamic - it has major performance implications. So, choose option B. Actually, try to expose relevant data in VDSDynamic and VDSStatistics, and VDS should call VDSDynamic and VDSStatistics and merge the data from both entities. On Thu, Feb 20, 2014 at 11:31 AM, Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC) chuan.l...@hp.com wrote: Hi All, I am Jason Liao from HP who are in charge of NUMA and Virtual NUMA feature. Now I have some concept about the host NUMA topology data structure on engine core We have VDS, VDSDynamic, VDSStatic, VdsStatistics object on engine core. And we have NUMA topology information: ListNumaNode numaNodeList NumaNode String ID # update from GetCapabilitiesVDSCommand ListString cpuList # update from GetCapabilitiesVDSCommand Int totalMem# update from GetCapabilitiesVDSCommand Int freeMem # update from GetStatsVDSCommand A. Add this data structure into VDSDynamic We should change the GetStatsVDSCommand update the VDSDynamic data. B. Add this data structure into VDS, and build the data structure from VDSDynamic, VdsStatistics I prefer B. does anybody have some comments? *Best Regards, *Jason Liao ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
[Engine-devel] Installing RHEVM DWH 3.2
Anybody knows how to install RHEVM DWH version 3.2 using the subscription-manager? ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Polling vs Pushing engine events
Note that you can usually get all the information you want using 1 API call, which should still scale. For instance, /ovirt-engine/api/vms will give you a list of all VMs and their statuses, so you can just run an XPath and get the status of all of them. On Tue, Dec 17, 2013 at 10:18 AM, Itamar Heim ih...@redhat.com wrote: On 12/17/2013 03:08 AM, Sven Kieske wrote: Hi, we got the following problem: we create / start / stop hole vms /data centers / storage etc (basically: everything ovirt can handle via REST-API) But if you want to know e.g. the status of a vm (or anything) you need to constantly poll the API. This is not what we desire to do, as it does not scale very well (e.g. polling 100 vms). well, you can search events since last event you searched for, only for the specific type of event you are interested in, then check which vm its for. Is there a standardized way of pushing information from the engine? well, the notification service which sends emails on these actually polls for them every minute in order to send the emails. we are discussing snmptraps here[1] one of the options this could be implemented with is via log4j getting all the audit log events, then you could use any log4j appender (db table, jms queue, etc.) [1] Bug 1032661 - Add SNMP trap as notification method to to ovirt-engine-notification ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Ideas about engine clustering?
I've added a wiki page on Engine HA. http://www.ovirt.org/Features/Engine_High_Availability Please add your comments, we'll try to make this an ongoing effort. On Sat, Aug 24, 2013 at 3:25 PM, Liran Zelkha liran.zel...@gmail.comwrote: IMHO EJB clustering is not the direction we should take, especially considering our IoC plans (see http://gerrit.ovirt.org/#/c/5575/). QuartzScheduler can be easily fixed by using a DB for Quartz configuration. We had such a discussion a few weeks ago. mod_jk is probably better for our HA. Postgres clustering should probably be a different task, as it's more infra and less code. Another thing to consider is locking, currently done using synchronized, and will need to be changed somehow (probably using Infinispan). On Aug 22, 2013, at 10:22 AM, Itamar Heim wrote: On 08/21/2013 10:27 PM, plysan wrote: 2013/8/22 Itamar Heim ih...@redhat.com mailto:ih...@redhat.comih...@redhat.com On 08/21/2013 11:08 AM, plysan wrote: 2013/8/21 Laszlo Hornyak lhorn...@redhat.com mailto:lhorn...@redhat.com lhorn...@redhat.com mailto:lhorn...@redhat.com lhorn...@redhat.com mailto:lhorn...@redhat.com lhorn...@redhat.com Hi, - Original Message - From: plysan ply...@gmail.com mailto:ply...@gmail.com ply...@gmail.com mailto:ply...@gmail.com ply...@gmail.com mailto:ply...@gmail.com ply...@gmail.com To: engine-devel@ovirt.org mailto:engine-devel@ovirt.org engine-devel@ovirt.org mailto:engine-devel@ovirt.org engine-devel@ovirt.org mailto:engine-devel@ovirt.org engine-devel@ovirt.org__ Sent: Wednesday, August 21, 2013 3:49:31 PM Subject: [Engine-devel] Ideas about engine clustering? Hello, guys: I'm trying to make a jboss clustering environment for engine. But felt the difficuty too. So I want to know if any one is working on this? I googled it and haven't found anything. I know there will be lots of things to do for engine migrating to clustering environment. I just want to know what problem need to be solved? I would like to list the things I think that is needed to do(or solve): 1. run jboss in standalone-ha mode or domain mode. 2. add @Clustered annotation to each EJB, I think this will solve the replication problem in cluster, probably lots of details in it. 3. QuartzScheduler problem, only one node running a scheduler is enough at a time instead of each node running one. 4. use mod_cluster to handle load balancing. 5. postgresql clustering. For failover it sounds like a plan, just wondering if this could solve load balancing problems as well. Ovirt is generating a big load on the DB and I would be really interested if SQL DB clustering could solve the issues. It would be great if it could, big part of the scalablity issue is the evil things we did against that poor database. 6. There should be more, but hope anyone can think about it :D There are tons of data structures that are used in oVirt that store state. You will have to hunt all of these down and replace with a data structure that is shared between the cluster memebers. This may be a big lot of work from many parts of the application. Indeed, and like Yair Zaslavsky said, we can put the data structure in infinispan cache, but clustered EJB can do the same thing for us, right? If all those state values are inside EJBs. (iirc)** Any ideas? Thanks! _ Engine-devel mailing list Engine-devel@ovirt.org mailto:Engine-devel@ovirt.orgEngine-devel@ovirt.org mailto:Engine-devel@ovirt.org Engine-devel@ovirt.org mailto:Engine-devel@ovirt.org Engine-devel@ovirt.org__ http://lists.ovirt.org/__mailman/listinfo/engine-devel http://lists.ovirt.org/mailman/listinfo/engine-devel _ Engine-devel mailing list Engine-devel@ovirt.org mailto:Engine-devel@ovirt.orgEngine-devel@ovirt.org http://lists.ovirt.org/__mailman/listinfo/engine-devel http
Re: [Engine-devel] Java Development Lifecycle
+1 - Original Message - From: Mooli Tayer mta...@redhat.com To: engine-devel engine-devel@ovirt.org Sent: Tuesday, August 27, 2013 2:29:01 PM Subject: [Engine-devel] Java Development Lifecycle Hello, I've been working on the engine for a few month now, And I feel I have not yet been able to find a productive and fast approach for Java development in different engine components. This revolves mainly around: Compiling and deploying artifacts: == If I want to check a small change in the engine, Be it in an artifact that is part of engine.ear or when I am working on one of the side tools that might run as a service (e.g ovirt-engine-notifier) or just a single jar (engine-manage-domains) It is essential to compile only parts of the project and not all of it. This can be achieved usually quite easily with mvn, however afterwards I would like to be able to also deploy and run it locally or even take all jars comprising a project and run them exploded inside my IDE (I use intellij idea) so I can enjoy live code editing and other benefits idea jboss gwt integration offers. I have been using remote debug (which is available by default in the engine and non existent in other jars so I have to tamper manually with a python service file to add debug flags - see next) But it's not as powerful as running inside an IDE and I feel my current development lifecycle is not as fast and productive as it could have been. (I do understand the engine is complex regarding configuration and deployment, so it's a challenge) Configuration: == This might be more of a todo item, I'm not sure.. After deployment of an artifact to run on a machine(again, engine, notifier or whatever) if I want to change it's configuration (configure it for remote debug as mentioned or change it's logging behavior and so on) It seems I always have to do it in a different place which I always spend hours finding. It could be great if for all artifacts configuration could be streamlined and monolithic across different components. These issues might seem obvious to some of you or unneeded to others, but I've decided to shout out in engine-devel because my usual way of approaching individuals has not got me very far, and also because I'm convinced some of you have found ways to be productive I am unaware of - Please share them! Others may have their own good ideas approaches(or their own needs). Thanks, Mooli. ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Ideas about engine clustering?
IMHO EJB clustering is not the direction we should take, especially considering our IoC plans (see http://gerrit.ovirt.org/#/c/5575/). QuartzScheduler can be easily fixed by using a DB for Quartz configuration. We had such a discussion a few weeks ago. mod_jk is probably better for our HA. Postgres clustering should probably be a different task, as it's more infra and less code. Another thing to consider is locking, currently done using synchronized, and will need to be changed somehow (probably using Infinispan). On Aug 22, 2013, at 10:22 AM, Itamar Heim wrote: On 08/21/2013 10:27 PM, plysan wrote: 2013/8/22 Itamar Heim ih...@redhat.com mailto:ih...@redhat.com On 08/21/2013 11:08 AM, plysan wrote: 2013/8/21 Laszlo Hornyak lhorn...@redhat.com mailto:lhorn...@redhat.com mailto:lhorn...@redhat.com mailto:lhorn...@redhat.com Hi, - Original Message - From: plysan ply...@gmail.com mailto:ply...@gmail.com mailto:ply...@gmail.com mailto:ply...@gmail.com To: engine-devel@ovirt.org mailto:engine-devel@ovirt.org mailto:engine-devel@ovirt.org mailto:engine-devel@ovirt.org__ Sent: Wednesday, August 21, 2013 3:49:31 PM Subject: [Engine-devel] Ideas about engine clustering? Hello, guys: I'm trying to make a jboss clustering environment for engine. But felt the difficuty too. So I want to know if any one is working on this? I googled it and haven't found anything. I know there will be lots of things to do for engine migrating to clustering environment. I just want to know what problem need to be solved? I would like to list the things I think that is needed to do(or solve): 1. run jboss in standalone-ha mode or domain mode. 2. add @Clustered annotation to each EJB, I think this will solve the replication problem in cluster, probably lots of details in it. 3. QuartzScheduler problem, only one node running a scheduler is enough at a time instead of each node running one. 4. use mod_cluster to handle load balancing. 5. postgresql clustering. For failover it sounds like a plan, just wondering if this could solve load balancing problems as well. Ovirt is generating a big load on the DB and I would be really interested if SQL DB clustering could solve the issues. It would be great if it could, big part of the scalablity issue is the evil things we did against that poor database. 6. There should be more, but hope anyone can think about it :D There are tons of data structures that are used in oVirt that store state. You will have to hunt all of these down and replace with a data structure that is shared between the cluster memebers. This may be a big lot of work from many parts of the application. Indeed, and like Yair Zaslavsky said, we can put the data structure in infinispan cache, but clustered EJB can do the same thing for us, right? If all those state values are inside EJBs. (iirc)** Any ideas? Thanks! _ Engine-devel mailing list Engine-devel@ovirt.org mailto:Engine-devel@ovirt.org mailto:Engine-devel@ovirt.org mailto:Engine-devel@ovirt.org__ http://lists.ovirt.org/__mailman/listinfo/engine-devel http://lists.ovirt.org/mailman/listinfo/engine-devel _ Engine-devel mailing list Engine-devel@ovirt.org mailto:Engine-devel@ovirt.org http://lists.ovirt.org/__mailman/listinfo/engine-devel http://lists.ovirt.org/mailman/listinfo/engine-devel unless you need the load balancing, just for HA, hosted-engine should cover your needs? I think failover and more scalability is what i am interested here. Just thinking about the possibilities :) Imagine engine would never get restarted from outside view. Cool isn't it? active/passive failover you should get from hosted-engine (yes, with some downtime) scalability and active-active would be great. iirc, juan looked at implications of doing this a while back ___ Engine-devel mailing list
[Engine-devel] ovirt.org and profiliing
Hi all As part of our effort to get profiling licenses to the ovirt contributors, we need to place an image of JProfiler (the chosen profiler) on the home page of ovirt.org. Anyone here knows who has the rights to do that? Thanks ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] [rhevm-staff] Fwd: BatchUpdates and VdsUpdateRuntimeInfo
Loaded average of a system that only runs update runtime info On Jul 3, 2013 2:45 PM, Michal Skrivanek mskri...@redhat.com wrote: - Forwarded Message - From: Liran Zelkha liran.zel...@gmail.com To: engine-devel engine-devel@ovirt.org Cc: Itamar Heim ih...@redhat.com, Barak Azulay bazu...@redhat.com, Allon Mureinik amure...@redhat.com Sent: Wednesday, July 3, 2013 1:08:08 PM Subject: BatchUpdates and VdsUpdateRuntimeInfo Hi all, Batch updates are now merged, so please start using it. As usage example, please check VdsUpdateRuntimeInfo code. Attached doc shows performance benefits from using batch over regular update. Batch Updates in VdsUpdateRuntimeInfo.pdf Is the percentage per each VM update? Or a loaded average system? ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Guid improvements
Awesome!!! On Mon, Jul 1, 2013 at 10:29 AM, Allon Mureinik amure...@redhat.com wrote: - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Yair Zaslavsky yzasl...@redhat.com Cc: Allon Mureinik amure...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Sunday, June 30, 2013 11:37:26 AM Subject: Re: [Engine-devel] Guid improvements Great news. Allon - please note that GUID is being recreated from String by both DB calls and by data received from VDSM. It is VERY useful to cache Guid String--Guid, and not just Empty GUID. I generally agree about the high cost of sting-uuid operations, but I'm not sure caching is the way to go, at least not everywhere. At least for the database, there is a much easier solution - stop using strings to represent uuids. Here's an example of how it can be done: http://gerrit.ovirt.org/#/c/16281/ However, as the Guid class runs in GWT as well, you can't use Infinispan and you're limited in the HashMap implementations you can use. Personally, I don't think it's a memory leak, as GUID number in the system are finite and not too large. What do you think? Want to add it to your patch? On Jun 30, 2013, at 11:13 AM, Yair Zaslavsky wrote: Well done, should have been done ages ago :) Now, for the painful rebase of async_task_mgr changes :) - Original Message - From: Allon Mureinik amure...@redhat.com To: engine-devel engine-devel@ovirt.org, Barak Azulay bazu...@redhat.com Cc: Yair Zaslavsky yzasl...@redhat.com, Michael Pasternak mpast...@redhat.com, Tal Nisan tni...@redhat.com, Ayal Baron aba...@redhat.com Sent: Sunday, June 30, 2013 11:11:30 AM Subject: Guid improvements Hi all, I just merged a couple of improvements to the [N]Guid class [1] to improve it's performance both CPU-wise and memory-wise, based on a set of benchmarks presented by Liran. What this patchset achieves: 1. Clean up the code, so it's easier to understand and use 2. Eliminate the inflation in the memory foot print caused by the getValue() method 3. Eliminate all the heavy calls to UUID.fromString when creating a new/empty Guid instance as a default value 4. Note that the cleanups proposed in (1) will have minor performance benefits (e.g., eliminating useless conditional statements), but I doubt this would be anything to write home about. From a developer's perspective, here's what changed: 1. No more NGuid, just Guid. Both static methods to create a Guid from String still exist, and are named createGuidFromString and createGuidFromStringDefaultEmpty. 2. [N]Guid.getValue() was removed, it's no longer needed after (1) was implemented 3. The Guid() constructor was made private, as it forced a redundant call to UUID.fromString(String). If you need an empty Guid instance, just use Guid.Empty 4. The Guid.EMPTY_GUID_VALUE string constant was removed, as it was used for redundant calls to UUID.fromString. If you really, REALLY, need it, just call Guid.Empty.getValue() for a UUID or Guid.Empty.toString() for a String. 5. All sorts of ways to transform Strings to Guids were removed. If you have a literal you trust, just use new Guid(String). If you suspect it may be null, use Guid.createGuidFromString[DefaultEmpty] 6. NewGuid is now called newGuid. We're in Java, not C# :-) Many thanks to everyone who reviewed this patchset. You guys rock! Regards, Allon [1] http://gerrit.ovirt.org/#/q/project:ovirt-engine+branch:master+topic:guid-cleanup,n,z ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Guid improvements
Great news. Allon - please note that GUID is being recreated from String by both DB calls and by data received from VDSM. It is VERY useful to cache Guid String--Guid, and not just Empty GUID. However, as the Guid class runs in GWT as well, you can't use Infinispan and you're limited in the HashMap implementations you can use. Personally, I don't think it's a memory leak, as GUID number in the system are finite and not too large. What do you think? Want to add it to your patch? On Jun 30, 2013, at 11:13 AM, Yair Zaslavsky wrote: Well done, should have been done ages ago :) Now, for the painful rebase of async_task_mgr changes :) - Original Message - From: Allon Mureinik amure...@redhat.com To: engine-devel engine-devel@ovirt.org, Barak Azulay bazu...@redhat.com Cc: Yair Zaslavsky yzasl...@redhat.com, Michael Pasternak mpast...@redhat.com, Tal Nisan tni...@redhat.com, Ayal Baron aba...@redhat.com Sent: Sunday, June 30, 2013 11:11:30 AM Subject: Guid improvements Hi all, I just merged a couple of improvements to the [N]Guid class [1] to improve it's performance both CPU-wise and memory-wise, based on a set of benchmarks presented by Liran. What this patchset achieves: 1. Clean up the code, so it's easier to understand and use 2. Eliminate the inflation in the memory foot print caused by the getValue() method 3. Eliminate all the heavy calls to UUID.fromString when creating a new/empty Guid instance as a default value 4. Note that the cleanups proposed in (1) will have minor performance benefits (e.g., eliminating useless conditional statements), but I doubt this would be anything to write home about. From a developer's perspective, here's what changed: 1. No more NGuid, just Guid. Both static methods to create a Guid from String still exist, and are named createGuidFromString and createGuidFromStringDefaultEmpty. 2. [N]Guid.getValue() was removed, it's no longer needed after (1) was implemented 3. The Guid() constructor was made private, as it forced a redundant call to UUID.fromString(String). If you need an empty Guid instance, just use Guid.Empty 4. The Guid.EMPTY_GUID_VALUE string constant was removed, as it was used for redundant calls to UUID.fromString. If you really, REALLY, need it, just call Guid.Empty.getValue() for a UUID or Guid.Empty.toString() for a String. 5. All sorts of ways to transform Strings to Guids were removed. If you have a literal you trust, just use new Guid(String). If you suspect it may be null, use Guid.createGuidFromString[DefaultEmpty] 6. NewGuid is now called newGuid. We're in Java, not C# :-) Many thanks to everyone who reviewed this patchset. You guys rock! Regards, Allon [1] http://gerrit.ovirt.org/#/q/project:ovirt-engine+branch:master+topic:guid-cleanup,n,z ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Guid improvements
I checked such a solution using JProfiler. Creating the GUID object takes much more CPU cycles that checking the string in the Map. On Jun 30, 2013, at 12:06 PM, Michael Pasternak wrote: On 06/30/2013 11:37 AM, Liran Zelkha wrote: Great news. Allon - please note that GUID is being recreated from String by both DB calls and by data received from VDSM. It is VERY useful to cache Guid String--Guid, and not just Empty GUID. However, as the Guid class runs in GWT as well, you can't use Infinispan and you're limited in the HashMap implementations you can use. Personally, I don't think it's a memory leak, as GUID number in the system are finite and not too large. it's large, it's 128-bit random number, it's not about memory leaking, but cpu cost, as you'll face a lot of rehash'ings in the map, i'm not even sure that using indexing in the map can help, worth checking. What do you think? Want to add it to your patch? On Jun 30, 2013, at 11:13 AM, Yair Zaslavsky wrote: Well done, should have been done ages ago :) Now, for the painful rebase of async_task_mgr changes :) - Original Message - From: Allon Mureinik amure...@redhat.com To: engine-devel engine-devel@ovirt.org, Barak Azulay bazu...@redhat.com Cc: Yair Zaslavsky yzasl...@redhat.com, Michael Pasternak mpast...@redhat.com, Tal Nisan tni...@redhat.com, Ayal Baron aba...@redhat.com Sent: Sunday, June 30, 2013 11:11:30 AM Subject: Guid improvements Hi all, I just merged a couple of improvements to the [N]Guid class [1] to improve it's performance both CPU-wise and memory-wise, based on a set of benchmarks presented by Liran. What this patchset achieves: 1. Clean up the code, so it's easier to understand and use 2. Eliminate the inflation in the memory foot print caused by the getValue() method 3. Eliminate all the heavy calls to UUID.fromString when creating a new/empty Guid instance as a default value 4. Note that the cleanups proposed in (1) will have minor performance benefits (e.g., eliminating useless conditional statements), but I doubt this would be anything to write home about. From a developer's perspective, here's what changed: 1. No more NGuid, just Guid. Both static methods to create a Guid from String still exist, and are named createGuidFromString and createGuidFromStringDefaultEmpty. 2. [N]Guid.getValue() was removed, it's no longer needed after (1) was implemented 3. The Guid() constructor was made private, as it forced a redundant call to UUID.fromString(String). If you need an empty Guid instance, just use Guid.Empty 4. The Guid.EMPTY_GUID_VALUE string constant was removed, as it was used for redundant calls to UUID.fromString. If you really, REALLY, need it, just call Guid.Empty.getValue() for a UUID or Guid.Empty.toString() for a String. 5. All sorts of ways to transform Strings to Guids were removed. If you have a literal you trust, just use new Guid(String). If you suspect it may be null, use Guid.createGuidFromString[DefaultEmpty] 6. NewGuid is now called newGuid. We're in Java, not C# :-) Many thanks to everyone who reviewed this patchset. You guys rock! Regards, Allon [1] http://gerrit.ovirt.org/#/q/project:ovirt-engine+branch:master+topic:guid-cleanup,n,z ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel -- Michael Pasternak RedHat, ENG-Virtualization RD ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Guid improvements
Why synchronization? No need for it. Worst case scenario a put (which should be much less common then get) will occur twice on the same key. On Jun 30, 2013, at 1:00 PM, Michael Pasternak wrote: On 06/30/2013 12:45 PM, Liran Zelkha wrote: All is true. But average UUID.fromString execution is 1675us, and HashMap.put is 78us - so the benefit is clear when we're talking on 100K executions for 10minutes... even with synchronization? what about ConcurrentHashMap? On Sun, Jun 30, 2013 at 12:44 PM, Michael Pasternak mpast...@redhat.com mailto:mpast...@redhat.com wrote: On 06/30/2013 12:20 PM, Liran Zelkha wrote: I checked such a solution using JProfiler. Creating the GUID object takes much more CPU cycles that checking the string in the Map. of course it is, but what is the size of map that you checked?, check on worst scenario, i.e map is full of all possible guids, also problem a bit different,java map has a load factor (which is usually 0.75), when ratio increases beyond the load factor, occurs proses called re-hash so that the hash table will double amount of buckets. what can produce a cpu spikes (though it should not happen too often), to avoid this the initial capacity should be greater than the maximum number of entries / the load factor, and this is a huge map so basically this is a tradeoff between time and space costs against the new guid generation. On Jun 30, 2013, at 12:06 PM, Michael Pasternak wrote: On 06/30/2013 11:37 AM, Liran Zelkha wrote: Great news. Allon - please note that GUID is being recreated from String by both DB calls and by data received from VDSM. It is VERY useful to cache Guid String--Guid, and not just Empty GUID. However, as the Guid class runs in GWT as well, you can't use Infinispan and you're limited in the HashMap implementations you can use. Personally, I don't think it's a memory leak, as GUID number in the system are finite and not too large. it's large, it's 128-bit random number, it's not about memory leaking, but cpu cost, as you'll face a lot of rehash'ings in the map, i'm not even sure that using indexing in the map can help, worth checking. What do you think? Want to add it to your patch? On Jun 30, 2013, at 11:13 AM, Yair Zaslavsky wrote: Well done, should have been done ages ago :) Now, for the painful rebase of async_task_mgr changes :) - Original Message - From: Allon Mureinik amure...@redhat.com mailto:amure...@redhat.com To: engine-devel engine-devel@ovirt.org mailto:engine-devel@ovirt.org, Barak Azulay bazu...@redhat.com mailto:bazu...@redhat.com Cc: Yair Zaslavsky yzasl...@redhat.com mailto:yzasl...@redhat.com, Michael Pasternak mpast...@redhat.com mailto:mpast...@redhat.com, Tal Nisan tni...@redhat.com mailto:tni...@redhat.com, Ayal Baron aba...@redhat.com mailto:aba...@redhat.com Sent: Sunday, June 30, 2013 11:11:30 AM Subject: Guid improvements Hi all, I just merged a couple of improvements to the [N]Guid class [1] to improve it's performance both CPU-wise and memory-wise, based on a set of benchmarks presented by Liran. What this patchset achieves: 1. Clean up the code, so it's easier to understand and use 2. Eliminate the inflation in the memory foot print caused by the getValue() method 3. Eliminate all the heavy calls to UUID.fromString when creating a new/empty Guid instance as a default value 4. Note that the cleanups proposed in (1) will have minor performance benefits (e.g., eliminating useless conditional statements), but I doubt this would be anything to write home about. From a developer's perspective, here's what changed: 1. No more NGuid, just Guid. Both static methods to create a Guid from String still exist, and are named createGuidFromString and createGuidFromStringDefaultEmpty. 2. [N]Guid.getValue() was removed, it's no longer needed after (1) was implemented 3. The Guid() constructor was made private, as it forced a redundant call to UUID.fromString(String). If you need an empty Guid instance, just use Guid.Empty 4. The Guid.EMPTY_GUID_VALUE string constant was removed, as it was used for redundant calls to UUID.fromString. If you really, REALLY, need it, just call Guid.Empty.getValue() for a UUID or Guid.Empty.toString() for a String. 5. All sorts of ways to transform Strings to Guids were removed. If you have a literal you trust, just use new Guid(String). If you suspect it may be null, use Guid.createGuidFromString[DefaultEmpty] 6. NewGuid is now called newGuid. We're in Java, not C# :-) Many thanks to everyone who reviewed this patchset. You guys rock! Regards, Allon [1] http://gerrit.ovirt.org/#/q/project:ovirt-engine+branch:master+topic:guid-cleanup,n,z ___ Engine-devel mailing list Engine-devel@ovirt.org mailto:Engine-devel@ovirt.org http
Re: [Engine-devel] Guid improvements
Sure. I agree. I'd be happy to show you the results in the profiler, so we can make a correct decision. On Jun 30, 2013, at 1:23 PM, Michael Pasternak wrote: On 06/30/2013 01:15 PM, Michael Pasternak wrote: On 06/30/2013 01:08 PM, Liran Zelkha wrote: Why synchronization? No need for it. Worst case scenario a put (which should be much less common then get) will occur twice on the same key. why assuming a best not worst scenario? don't forget that every new insertion requires collision resolution which is triggers .equals() on the GUID. Liran, don't get me wrong, i'm not against the caching in general, obviously reads writes so actually i'm all with you, just we're going to significantly enlarge a memory footprint so i just want to make sure we're on a right track for the worst scenario where engine runs for ages and hashmap reaches it's load factor. On Jun 30, 2013, at 1:00 PM, Michael Pasternak wrote: On 06/30/2013 12:45 PM, Liran Zelkha wrote: All is true. But average UUID.fromString execution is 1675us, and HashMap.put is 78us - so the benefit is clear when we're talking on 100K executions for 10minutes... even with synchronization? what about ConcurrentHashMap? On Sun, Jun 30, 2013 at 12:44 PM, Michael Pasternak mpast...@redhat.com mailto:mpast...@redhat.com wrote: On 06/30/2013 12:20 PM, Liran Zelkha wrote: I checked such a solution using JProfiler. Creating the GUID object takes much more CPU cycles that checking the string in the Map. of course it is, but what is the size of map that you checked?, check on worst scenario, i.e map is full of all possible guids, also problem a bit different,java map has a load factor (which is usually 0.75), when ratio increases beyond the load factor, occurs proses called re-hash so that the hash table will double amount of buckets. what can produce a cpu spikes (though it should not happen too often), to avoid this the initial capacity should be greater than the maximum number of entries / the load factor, and this is a huge map so basically this is a tradeoff between time and space costs against the new guid generation. On Jun 30, 2013, at 12:06 PM, Michael Pasternak wrote: On 06/30/2013 11:37 AM, Liran Zelkha wrote: Great news. Allon - please note that GUID is being recreated from String by both DB calls and by data received from VDSM. It is VERY useful to cache Guid String--Guid, and not just Empty GUID. However, as the Guid class runs in GWT as well, you can't use Infinispan and you're limited in the HashMap implementations you can use. Personally, I don't think it's a memory leak, as GUID number in the system are finite and not too large. it's large, it's 128-bit random number, it's not about memory leaking, but cpu cost, as you'll face a lot of rehash'ings in the map, i'm not even sure that using indexing in the map can help, worth checking. What do you think? Want to add it to your patch? On Jun 30, 2013, at 11:13 AM, Yair Zaslavsky wrote: Well done, should have been done ages ago :) Now, for the painful rebase of async_task_mgr changes :) - Original Message - From: Allon Mureinik amure...@redhat.com mailto:amure...@redhat.com To: engine-devel engine-devel@ovirt.org mailto:engine-devel@ovirt.org, Barak Azulay bazu...@redhat.com mailto:bazu...@redhat.com Cc: Yair Zaslavsky yzasl...@redhat.com mailto:yzasl...@redhat.com, Michael Pasternak mpast...@redhat.com mailto:mpast...@redhat.com, Tal Nisan tni...@redhat.com mailto:tni...@redhat.com, Ayal Baron aba...@redhat.com mailto:aba...@redhat.com Sent: Sunday, June 30, 2013 11:11:30 AM Subject: Guid improvements Hi all, I just merged a couple of improvements to the [N]Guid class [1] to improve it's performance both CPU-wise and memory-wise, based on a set of benchmarks presented by Liran. What this patchset achieves: 1. Clean up the code, so it's easier to understand and use 2. Eliminate the inflation in the memory foot print caused by the getValue() method 3. Eliminate all the heavy calls to UUID.fromString when creating a new/empty Guid instance as a default value 4. Note that the cleanups proposed in (1) will have minor performance benefits (e.g., eliminating useless conditional statements), but I doubt this would be anything to write home about. From a developer's perspective, here's what changed: 1. No more NGuid, just Guid. Both static methods to create a Guid from String still exist, and are named createGuidFromString and createGuidFromStringDefaultEmpty. 2. [N]Guid.getValue() was removed, it's no longer needed after (1) was implemented 3. The Guid() constructor was made private, as it forced a redundant call to UUID.fromString(String). If you need an empty Guid instance, just use Guid.Empty 4. The Guid.EMPTY_GUID_VALUE string constant was removed, as it was used
Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo
Hi all, Current status of VdsUpdateRuntimeInfo (please note - patch is not yet committed). Initial UpdateRuntimeInfo performance wasted roughly 57% of CPU time on DB activity. Of that, about 25% of CPU time wasted on update behavior. After modification (moving updates to batch update) UpdateRuntimeInfo performance wasted roughly 40% of CPU time on DB activity. And only 12% of that was wasted on update behavior. My next task is to try and optimize the rest of overall database performance. - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Barak Azulay bazu...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Wednesday, June 12, 2013 12:35:03 PM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo Hi guys, I'm working hard on this. I'll send summary mail and findings later today. On Jun 12, 2013, at 10:45 AM, Barak Azulay wrote: - Original Message - From: Itamar Heim ih...@redhat.com To: Barak Azulay bazu...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Wednesday, June 12, 2013 10:09:09 AM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo On 06/12/2013 09:41 AM, Barak Azulay wrote: - Original Message - From: Itamar Heim ih...@redhat.com To: Liran Zelkha lzel...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Tuesday, June 11, 2013 4:00:21 PM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo On 06/11/2013 03:26 PM, Liran Zelkha wrote: Hi all, I'm checking performance for VdsUpdateRunTimeInfo. Naturally, much of the performance surrounds database activity (getVmsRunningOnVds queries, updateDeviceRuntimeInfo, updateVmDynamic) Few questions: 1. I have implemented batch updates for procedure UpdateVmDeviceRuntimeInfo for improved performance. 2. Seems like the only parameters UpdateVmDeviceRuntimeInfo is getting are vm_id,vm_device_id,address and alias. Are those rapidly changing, or will it be beneficial to implement caching on those updates (to ensure not-required updates do not travel to the database). slowly changing, but how will you cover all flows changing these devices to invalidate the cache (iiuc, this table is modified by engine when adding devices to a VM as well?) I don't think that in the device run time info we need to invalidate once we add a device. This is a specific case where we actually get the information from the VDSM (addresses are received from libvirt) The commands IIRC are first send to VDSM and than update the runtime info only on changed info (we can also hash it), It may put the placeholder in the DB first but it still relies on the data received from VDSM. if this table is only updated from vdsm to save it, i agree. but isn't the engine also manipulating it? wasn't there a request to be able to maybe edit the addresses some day? Even if there was such a request, we still update when we receive the info from libvirt. Anyway there are obviously a few improvements to be done before caching. - update only when info received from VDSM change (need to verify it is the practice here) - do batch update for the new updated data received. - do the hashing (of vm device runtime info) on VDSM for the data , and update only when hash changes - caching ... Barak ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo
Batch updates for UpdateVmDeviceRuntimeInfo, UpdateVdsInterfaceStatistics and UpdateVdsDynamic. All improved execution performance by about 60-80%. On Jun 12, 2013, at 4:05 PM, Itamar Heim wrote: On 06/12/2013 04:03 PM, Liran Zelkha wrote: Hi all, Current status of VdsUpdateRuntimeInfo (please note - patch is not yet committed). Initial UpdateRuntimeInfo performance wasted roughly 57% of CPU time on DB activity. Of that, about 25% of CPU time wasted on update behavior. After modification (moving updates to batch update) UpdateRuntimeInfo performance wasted roughly 40% of CPU time on DB activity. And only 12% of that was wasted on update behavior. which changes are in place for above? My next task is to try and optimize the rest of overall database performance. - Original Message - From: Liran Zelkha liran.zel...@gmail.com To: Barak Azulay bazu...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Wednesday, June 12, 2013 12:35:03 PM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo Hi guys, I'm working hard on this. I'll send summary mail and findings later today. On Jun 12, 2013, at 10:45 AM, Barak Azulay wrote: - Original Message - From: Itamar Heim ih...@redhat.com To: Barak Azulay bazu...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Wednesday, June 12, 2013 10:09:09 AM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo On 06/12/2013 09:41 AM, Barak Azulay wrote: - Original Message - From: Itamar Heim ih...@redhat.com To: Liran Zelkha lzel...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Tuesday, June 11, 2013 4:00:21 PM Subject: Re: [Engine-devel] Updates in VdsUpdateRuntimeInfo On 06/11/2013 03:26 PM, Liran Zelkha wrote: Hi all, I'm checking performance for VdsUpdateRunTimeInfo. Naturally, much of the performance surrounds database activity (getVmsRunningOnVds queries, updateDeviceRuntimeInfo, updateVmDynamic) Few questions: 1. I have implemented batch updates for procedure UpdateVmDeviceRuntimeInfo for improved performance. 2. Seems like the only parameters UpdateVmDeviceRuntimeInfo is getting are vm_id,vm_device_id,address and alias. Are those rapidly changing, or will it be beneficial to implement caching on those updates (to ensure not-required updates do not travel to the database). slowly changing, but how will you cover all flows changing these devices to invalidate the cache (iiuc, this table is modified by engine when adding devices to a VM as well?) I don't think that in the device run time info we need to invalidate once we add a device. This is a specific case where we actually get the information from the VDSM (addresses are received from libvirt) The commands IIRC are first send to VDSM and than update the runtime info only on changed info (we can also hash it), It may put the placeholder in the DB first but it still relies on the data received from VDSM. if this table is only updated from vdsm to save it, i agree. but isn't the engine also manipulating it? wasn't there a request to be able to maybe edit the addresses some day? Even if there was such a request, we still update when we receive the info from libvirt. Anyway there are obviously a few improvements to be done before caching. - update only when info received from VDSM change (need to verify it is the practice here) - do batch update for the new updated data received. - do the hashing (of vm device runtime info) on VDSM for the data , and update only when hash changes - caching ... Barak ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
[Engine-devel] Updates in VdsUpdateRuntimeInfo
Hi all, I'm checking performance for VdsUpdateRunTimeInfo. Naturally, much of the performance surrounds database activity (getVmsRunningOnVds queries, updateDeviceRuntimeInfo, updateVmDynamic) Few questions: 1. I have implemented batch updates for procedure UpdateVmDeviceRuntimeInfo for improved performance. 2. Seems like the only parameters UpdateVmDeviceRuntimeInfo is getting are vm_id,vm_device_id,address and alias. Are those rapidly changing, or will it be beneficial to implement caching on those updates (to ensure not-required updates do not travel to the database). 3. Any additional known performance problems you know of in regards to this class? Thanks... ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Performance and scalability
Will do. Thanks. On Mon, Jun 10, 2013 at 3:44 PM, Livnat Peer lp...@redhat.com wrote: On 06/04/2013 02:43 PM, Liran Zelkha wrote: Hi all, I've added a new feature page for Performance and Scalability. Please review and add your ideas... http://www.ovirt.org/Features/Performance_And_Scalability ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel Hi Liran, Handling performance and scale in oVirt seems very interesting. I would love to get a better understanding of what the problem is before viewing the different solutions to solve it. Can you please describe which scenarios you think we should tune the performance for and which scale are we looking at? I think that adding the above to the wiki page would give us a better ground for discussing the issue. In addition I suggest polling the users list for issues around performance and/or scale. Maybe the input would help us focus on tuning the right flows going forward. Thanks, Livnat ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
[Engine-devel] Performance and scalability
Hi all, I've added a new feature page for Performance and Scalability. Please review and add your ideas... http://www.ovirt.org/Features/Performance_And_Scalability ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Performance and scalability
Hi Laszlo I 100% agree with both of your remarks. Caching is caching the entities, but also attempting to cache the list of entities. Batch updates is done as an infrastructure (change was reviewed and now in fixes) and after that - we can start using it everywhere. - Original Message - From: Laszlo Hornyak lhorn...@redhat.com To: Liran Zelkha lzel...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Tuesday, June 4, 2013 5:01:37 PM Subject: Re: [Engine-devel] Performance and scalability Hi Liran, Some comments to start the discussion: - Caching: exactly how is this resultset caching going to work? Wouldnt it be more simple to just cache the mapped entities? The mapping is not a big overhead, but caching the resultset seems to be difficult. - Batch updates: sounds cool, transactions are a major overhead on performance, so if you can make the update in a single tx and few interactions that will rock, but what if we keep the dynamic and statistic data in the memory at the first place and synchronize to DB on a background process. So that when you look for dynamic/statistics data, you do not have to hit the DB. Thx, Laszlo - Original Message - From: Liran Zelkha lzel...@redhat.com To: engine-devel engine-devel@ovirt.org Sent: Tuesday, June 4, 2013 1:43:55 PM Subject: [Engine-devel] Performance and scalability Hi all, I've added a new feature page for Performance and Scalability. Please review and add your ideas... http://www.ovirt.org/Features/Performance_And_Scalability ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
[Engine-devel] DB performance optimization
Hi all, There is a WIP for caching and DB performance optimization. Please look at - http://gerrit.ovirt.org/#/c/14188/4 To summarize, it includes automatic caching for some business entities (*_static for instance), break up of VDS to VDS_static, VDS_dynamic and VDS_statistics, and support for batch updates. Please share if/where you think batch updates can be used (for a usage scenario, please check InterfaceDaoDbFacadeImpl.massUpdateStatisticsForVds()). Future tasks will include update sensitivity, better Collection caching and trying to minimize getConnection() calls. Would love your comments and ideas. Thanks. ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] VDSM Fake
Awesome. I really need this for the stress tests I'm running. How many hosts were you able to simulate with one web application? - Original Message - From: Libor Spevak lspe...@redhat.com To: engine-devel@ovirt.org Sent: Tuesday, April 30, 2013 12:50:52 PM Subject: [Engine-devel] VDSM Fake Hi, let me introduce an experimental project called VDSM Fake. This is a small Java web application simulating selected VDSM features, but only from the Engine point of view. The aim was to get oVirt-Engine top performance characteristics in core features simulating hundreds of fake hosts and VMs, and in ecologic way. More info: http://www.ovirt.org/VDSM_Fake Sources: git clone git://github.com/lspevak/ovirt-vdsmfake.git git clone git://github.com/lspevak/ovirt-restapiconf.git Regards, Libor ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Dropping encryption of database password
Why not do use the same technology like JBoss DataSource password encryption? http://docs.jboss.org/jbosssecurity/docs/6.0/security_guide/html/Encrypting_Data_Source_Passwords.html On Wed, May 1, 2013 at 3:45 AM, Eli Mesika emes...@redhat.com wrote: - Original Message - From: Alon Bar-Lev alo...@redhat.com To: engine-devel engine-devel@ovirt.org Cc: Yair Zaslavsky yzasl...@redhat.com, Eli Mesika emes...@redhat.com, Juan Hernandez jhern...@redhat.com Sent: Tuesday, April 30, 2013 10:41:20 PM Subject: Dropping encryption of database password Hello, Currently we store database password encrypted using org.picketbox.datasource.security.SecureIdentityLoginModule. This is reverse encryption with common knowledge shared secret. Using encryption with common knowledge shared secret is close to void protection. So far we also stored the password as plain text at /etc/ovirt-engine/.pgpass, this is going to be removed as no component actually uses the .pgpass, however we do need to store non-java specific password in for utilities. In master (aiming to 3.3), we store the database connection details in own file /etc/ovirt-engine/engine.conf.d/50-setup-database.conf owned by ovirt user and not world readable. I would like to use the same 50-setup-database.conf to store plain text password and remove the java specific reversible encrypted password usage. Bottom line... 1. We drop the .pgpass file. 2. We store database connection information in /etc/ovirt-engine/engine.conf.d/file that is readable only by ovirt usage. 3. We drop the java specific reversible encryption in favor of plain text. Thoughts? I see no problem in the .pgpass , only root can access it (it has 0600 mode , if it doesn't it is ignored by PG) Apart from that , this is the standard way used by PG so why not using it , AFAIK this is considered safe secured Alon ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
[Engine-devel] Database Caching
Hi I've created Change I04d7edea (http://gerrit.ovirt.org/#/c/14188/) that adds automatic database caching capabilities to the engine. To cache a business entity, just add an annotation called TimeToLiveInCache that accepts as a parameter the number of seconds this object can live in the cache until it is refreshed from the database. The next stage is improving the cache update mechanism, using Infinispan, and the ability to capture more query types. Would love input and suggestions. Thanks, Liran ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Introducing Infinispan cache and some required actions after change
Great news. I totally agree that Infinispan is the right way to go. Michael - let's sync on the database caching integration with Infinispan. Would love to hear your thoughts. - Original Message - From: Yair Zaslavsky yzasl...@redhat.com To: Michael Kublin mkub...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Tuesday, April 23, 2013 10:45:47 AM Subject: Re: [Engine-devel] Introducing Infinispan cache and some requiredactions after change - Original Message - From: Michael Kublin mkub...@redhat.com To: engine-devel engine-devel@ovirt.org Sent: Tuesday, April 23, 2013 10:30:09 AM Subject: [Engine-devel] Introducing Infinispan cache and some required actions after change Hi All, The following commit I2fbebb10c08c87d2c7fd4d7443d3e3b374541ed7 will introduce a in memory cache inside ovirt-engine The cache which was chosen is infinispan (http://www.jboss.org/infinispan/) The reasons are following: cache has all required functionality, it is open source project and has built-in integration with JBOSS, all cache configuration can be accessed or edited via JBOSS admin console. By introducing this commit the first stage is finished: 1. Cache implementation is added to project 2. Show example how to add and use a cache in ovirt-engine, if it is required 3. Solved bug with possible memory leak in audit log event The next stages are: 1. Second stage - replace all custom made implementation by standard central cache implementation 2. Third stage - introduce second level cache for work with DB 3. Fourth stage (Future) - make cluster wise cache or make cache as service, if we will have more than one instance of JBOSS Now, after the applying the following commit I2fbebb10c08c87d2c7fd4d7443d3e3b374541ed7, everyone who works with ovirt-engine will be required to perform the following operation 1. Run mvn clean install -Pdep,setup in order to replace an old standalone.xml with new version of it Or 1. Use regular deploy: mvn clean install -Pdep 2. Access to $JBOSS_HOME/bin/standalone.xml and replace or add infinispan subsystem: subsystem xmlns=urn:jboss:domain:infinispan:1.1 default-cache-container=ovirt-engine cache-container name=ovirt-engine default-cache=timeout-base jndi-name=java:jboss/infinispan/ovirt-engine start=EAGER local-cache name=timeout-base transaction mode=NONE/ eviction max-entries=1/ expiration interval=6/ /local-cache /cache-container /subsystem 3. Add or check that exists extension module=org.jboss.as.clustering.infinispan/ under extensions element Great! Minor addition - For those of you who are familiar with the term region (exists at JBoss Cache, and IIRC ehcache) - look at the configuration Michael provided for timeout-base. Infinspan supports transactions (see the transaction-mode) and supports distribution (currently at local mode - see the local-cache definition). Yair Thanks and Regards Michael ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Move SQL out of stored procedures
Hi Laszlo, I'm currently in the process of adding a caching layer on top of JdbcTemplate, which would greatly reduce the number of database activities we have, so that would solve the last item you raised. I didn't mean the ORM performance is caused by the mapping. I think the problem lies in the fact that we will modify our code to have batch updates for most insert activities - a thing that is impossible in JPA/Hibernate. So, if we'll have some code in SQL and some in ORM - I prefer we stick all code to SQL… On Apr 2, 2013, at 9:34 AM, Laszlo Hornyak wrote: Hi Liran, I agree that ORM tools in general have to add some mapping overhead, but that overhead is very small compared to the time needed by the database interaction. ORM tools sometimes generate SQL statements that we could imagine being better, I do not think they are as hard for the DB as for example the ones generated by searchbackend. Also, we can do rdbms specific optimizations when needed. Plus we could finally have some caching in ovirt engine and the code would not have to read e.g. the DC record again and again. There are some more like that. Therefore having a JPA could improve the performance in engine. Laszlo - Original Message - From: Liran Zelkha lzel...@redhat.com To: Eli Mesika emes...@redhat.com Cc: engine-devel@ovirt.org Sent: Tuesday, April 2, 2013 7:24:08 AM Subject: Re: [Engine-devel] Move SQL out of stored procedures I also apologize for jumping in late... I think concerning SQL injection we'll be covered by using PreparedStatements. Since we're using SpringJDBC, most of our code uses PreparedStatements anyway. Concerning ORM - I feel it won't really be beneficial to us. I know of very few projects who can actually be cross-database, and just maintaining schema creation scripts for different databases can be too difficult to maintain. Also, from a performance perspective, ORM performs worse than regular SQL (or stored procedures), so it wouldn't be the direction I choose. I think we should keep using SpringJDBC with either SQL or stored procedures (doesn't really matter, whatever is easier to maintain and performs faster) and maybe add a better, more generic, RowMapper class. - Original Message - From: Eli Mesika emes...@redhat.com To: engine-devel@ovirt.org Sent: Tuesday, April 2, 2013 12:35:03 AM Subject: Re: [Engine-devel] Move SQL out of stored procedures - Original Message - From: Laszlo Hornyak lhorn...@redhat.com To: Libor Spevak lspe...@redhat.com Cc: Juan Hernandez jhern...@redhat.com, engine-devel@ovirt.org Sent: Thursday, March 28, 2013 5:31:34 PM Subject: Re: [Engine-devel] Move SQL out of stored procedures - Original Message - From: Libor Spevak lspe...@redhat.com To: Itamar Heim ih...@redhat.com Cc: Juan Hernandez jhern...@redhat.com, engine-devel@ovirt.org Sent: Thursday, March 28, 2013 4:04:20 PM Subject: Re: [Engine-devel] Move SQL out of stored procedures Hi, apart from SQL vs. stored procedures discussion, I am trying to understand what we can get if we support more databases... Sorry for joining this discussion so late (I was in a vacation) anyway two points missing from SQL VS. SP are 1) security - With plain SQL we will have to handle SQL Injection 2) It is more economic to pass a call to SP than the full SQL on the wire... Some points: 1. Is there a real need by end-users/customers to run it on e.g. Oracle only? (performance, stability, easier administration). Usually companies have one database and they are trying to stick to that one. Having two doubles the resource needs, you need one more DBA team, care for mirrors, backups. So it almost doubles the costs. Generally, I agree with Alon B L , if you have to support X DBs you are not doubling the effort by X Actually, we had already experience with that when we supported both MS SQL Postgres I believe that as we have some customers with large installations, performance counts and the best way (and sometimes teh only way) id the DB layer This is why I frequently hear people asking if we plan to support XyDB in the future. PostgreSQL is cool, but those who already use MySQL/MariaDB, they just do not want one more. What is the future of PostgreSQL? 2. Is it decided by architectural board, what kind of databases we would like to support? (cannot support any db) With a JPA we could support most mainstream relational databases, but in my opinion 99 percent of people run oracle, mysql/mariadb or postgresql. So maybe we do not have to think in big number of database engines. This is theoretical since JPA is still on wishlist :( 3. Are we talking about the Engine only, or there will be a need to rewrite ETL mappings and upgrade DWH database, or maybe modify JasperReports templates (simply, some DB types behave differently)? Maybe we can look at JasperSoft solution, they support
Re: [Engine-devel] Move SQL out of stored procedures
Hi I think externalizing SQL can lead to a VERY difficult maintenance. But, as long as we stick to SQL (or stored procedures, just not ORM), I don't mind… On Apr 2, 2013, at 10:19 AM, Yair Zaslavsky wrote: - Original Message - From: Libor Spevak lspe...@redhat.com To: Juan Hernandez jhern...@redhat.com Cc: engine-devel@ovirt.org Sent: Wednesday, March 27, 2013 10:09:22 AM Subject: Re: [Engine-devel] Move SQL out of stored procedures Hi, I would recommend always to avoid hard coding SQL into Java code. It is very hard to maintain and read. If there is something, which prevents using JPA/Hibernate, e.g. the database relational model doesn't reflect the object-oriented domain very well or we have to live with many stored procedures concurrently, I would choose a framework, which enables to externalize the SQL code (into XML). I worked on a larger project(s) with a lot of PL/SQL code, we moved to myBatis (previously iBatis) very soon for Java backend: https://code.google.com/p/mybatis/ Libor I used a similar approach at past project - not with iBatis though, but a in house implementation of such framework. I think this idea is worth considering. On 26.3.2013 18:34, Juan Hernandez wrote: Hello, I would like to start a discussion about the subject. I think this is something we need to do if one day we want to be able to use any database other than PostgreSQL. I did an small example of what it takes and how it looks like to have the SQL code into the DAOs: http://gerrit.ovirt.org/13347 It isn't rocket science, it isn't an exciting task, it isn't fun, but something I think we should eventually do. I appreciate any comment about how and when to do this, including those saying that instead of this primitive approach we should use this or that ORM framework. Regards, Juan Hernandez ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Move SQL out of stored procedures
I also apologize for jumping in late... I think concerning SQL injection we'll be covered by using PreparedStatements. Since we're using SpringJDBC, most of our code uses PreparedStatements anyway. Concerning ORM - I feel it won't really be beneficial to us. I know of very few projects who can actually be cross-database, and just maintaining schema creation scripts for different databases can be too difficult to maintain. Also, from a performance perspective, ORM performs worse than regular SQL (or stored procedures), so it wouldn't be the direction I choose. I think we should keep using SpringJDBC with either SQL or stored procedures (doesn't really matter, whatever is easier to maintain and performs faster) and maybe add a better, more generic, RowMapper class. - Original Message - From: Eli Mesika emes...@redhat.com To: engine-devel@ovirt.org Sent: Tuesday, April 2, 2013 12:35:03 AM Subject: Re: [Engine-devel] Move SQL out of stored procedures - Original Message - From: Laszlo Hornyak lhorn...@redhat.com To: Libor Spevak lspe...@redhat.com Cc: Juan Hernandez jhern...@redhat.com, engine-devel@ovirt.org Sent: Thursday, March 28, 2013 5:31:34 PM Subject: Re: [Engine-devel] Move SQL out of stored procedures - Original Message - From: Libor Spevak lspe...@redhat.com To: Itamar Heim ih...@redhat.com Cc: Juan Hernandez jhern...@redhat.com, engine-devel@ovirt.org Sent: Thursday, March 28, 2013 4:04:20 PM Subject: Re: [Engine-devel] Move SQL out of stored procedures Hi, apart from SQL vs. stored procedures discussion, I am trying to understand what we can get if we support more databases... Sorry for joining this discussion so late (I was in a vacation) anyway two points missing from SQL VS. SP are 1) security - With plain SQL we will have to handle SQL Injection 2) It is more economic to pass a call to SP than the full SQL on the wire... Some points: 1. Is there a real need by end-users/customers to run it on e.g. Oracle only? (performance, stability, easier administration). Usually companies have one database and they are trying to stick to that one. Having two doubles the resource needs, you need one more DBA team, care for mirrors, backups. So it almost doubles the costs. Generally, I agree with Alon B L , if you have to support X DBs you are not doubling the effort by X Actually, we had already experience with that when we supported both MS SQL Postgres I believe that as we have some customers with large installations, performance counts and the best way (and sometimes teh only way) id the DB layer This is why I frequently hear people asking if we plan to support XyDB in the future. PostgreSQL is cool, but those who already use MySQL/MariaDB, they just do not want one more. What is the future of PostgreSQL? 2. Is it decided by architectural board, what kind of databases we would like to support? (cannot support any db) With a JPA we could support most mainstream relational databases, but in my opinion 99 percent of people run oracle, mysql/mariadb or postgresql. So maybe we do not have to think in big number of database engines. This is theoretical since JPA is still on wishlist :( 3. Are we talking about the Engine only, or there will be a need to rewrite ETL mappings and upgrade DWH database, or maybe modify JasperReports templates (simply, some DB types behave differently)? Maybe we can look at JasperSoft solution, they support more databases. IMHO , ETL DWH are perfect candidates for NO SQL which is already supported by Jasper 4. Current full/incremental upgrade process of PostgreSQL is IMHO very good tuned (it is similar to dbmaintain.org tool - Java implementation - I used successfully on one project - after some changes of course). I do not believe we can use or easily develop general upgrade/migration tool, and XML based (I am sorry Alissa, not sure about Liquibase, I haven't studied it deeply, but there is a need to incrementally change db objects, but sometimes also to migrate data to new structures, the most flexible and quickest is to do it using native SQL, but yes, it depends on the project needs...). I had evaluated Liquibase and I think that managing your DB upgrades via XML is very unfriendly and very limited as you reach complex upgrades as we had in the past. Just think of the tables in which we change the key from long to UUID , there is no way to do that in such tools 5. As a developer, with every new column I need to write upgrade scripts, prepare test environments and test all scenarios several times on different databases, so time-consuming. Did it also , again , since our SQL is 90% simple , the effort of writing a SP for more than one DB is not so high (and you have free converters you can use for that) Finally, embedded SQL in the Java code is not a good idea, it will be hard to maintain it
