Re: [Engine-devel] some cleanup
- Original Message - From: Laszlo Hornyak lhorn...@redhat.com To: engine-devel engine-devel@ovirt.org Sent: Monday, April 22, 2013 8:27:00 PM Subject: [Engine-devel] some cleanup Hi, Some of my patches are waiting for a brave reviewer to +2 them :) - entity cleanups http://gerrit.ovirt.org/13531 http://gerrit.ovirt.org/13705 http://gerrit.ovirt.org/13940 http://gerrit.ovirt.org/13705 - command cleanups http://gerrit.ovirt.org/13945 http://gerrit.ovirt.org/13946 http://gerrit.ovirt.org/13947 Thx, Laszlo ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel Done ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] DB Performance Monitoring
- Original Message - From: Eli Mesika emes...@redhat.com To: Libor Spevak lspe...@redhat.com Cc: engine-devel@ovirt.org Sent: Friday, February 22, 2013 12:46:07 AM Subject: Re: [Engine-devel] DB Performance Monitoring - Original Message - From: Libor Spevak lspe...@redhat.com To: Eli Mesika emes...@redhat.com Cc: engine-devel@ovirt.org, Yair Zaslavsky yzasl...@redhat.com, Barak Azulay bazu...@redhat.com Sent: Thursday, February 21, 2013 6:50:58 PM Subject: Re: [Engine-devel] DB Performance Monitoring On 21.2.2013 17:11, Eli Mesika wrote: - Original Message - From: Libor Spevak lspe...@redhat.com To: engine-devel@ovirt.org Sent: Tuesday, February 19, 2013 5:44:10 PM Subject: [Engine-devel] DB Performance Monitoring Hi, I just wanted to share a partial result from testing of engine db load during operation (SQL queries frequencies). Maybe, we can decide later, if some SQL queries result CACHING can boost throughput for larger deployments. The caching technology can be from a simple HashMap lookup to deployments of something like http://ehcache.org/ (memory database). I prepared two simple scenarios: - oVirt engine, 2 hosts, 1 VM, running 15 minutes (Power on + Up state) - oVirt engine, 2 hosts, 10 VMs from one pool, running 15 minutes (Power on + Up state) Appending 2 spreadsheets with data about the most used SQL queries (generated by PostgreSQL standard pg_statements_stat module). e.g. - 2nd row shows number of granted connections from the db pool (e.g. can be used to set optimal connection pool size) (check of connection health: select 1) - most of the queries are wrapped by a PLSQL function so we see a wrapper: select * from getvdsgroupbyvdsgroupid($1, $2, $3) and near to it the 'real' query: SELECT vds_groups_view.* FROM vds_groups_view WHERE vds_group_id = v_vds_group_id AND (NOT v_is_filtered OR EXISTS (SELECT 1 FROM user_vds_groups_permissions_view WHERE user_id = v_user_id AND entity_id = v_vds_group_id)) Just as an example, I selected in yellow color some queries, which probably do not change often, but are very frequent. Thanks Libor for taking te time to do that. The problematic queries are those that involves all kinds of *permissions* checks with the complicated expensive permission handling views. I believe that using Snapshot Materialized Views as defined in http://www.ovirt.org/OVirt-DB-Issues/MaterializedViews will solve the major part of those problems. We are currently verifying this approach and my recommendation is to have all *permissions* views as Snapshot Materialized Views that are updated via a cron job. I think there could be performace improvement with M-views, of course. There is some overhead just with their maintainance, e.g. refreshing, updating existing dependent objects (table name renames, ...), Refresh of *permission* views was tested with 2 views 160,000 records each and was about 2 sec not sure about NATIVE implementation level status in PostgreSQL, according to e.g. Oracle db capabilities (MV-logs + M-views). http://wiki.postgresql.org/wiki/Materialized_Views AFAIK Postgres is not going to implement that in the near future Still, if the Engine gathers statistics nearly in real time, my question would be, if there is a need of round-trip to the database and back at all for frequent queries. Depends what is the cost of this round-trip , if it is very cheep as I showed when I tested M-Views I think that DB query is still a good and natural choice Other tools like 'pg_top' can provide runtime statistics of db processes (cpu, mem, locks, ... views). By enabling debug level logging of PostgreSQL we can check real values to the queries. Of course, it would be useful to run such tests with many hosts and VMs to predict scaling issues. More info about tools configuration: http://www.ovirt.org/Engine_database_performance_monitoring Regards, Libor Due the following discussion I checked uses of select * from getvdsgroupbyvdsgroupid($1, $2, $3) query and find out that we have three call for such query during monitoring and we need only one, so the following patch should reduce a number of such queries during monitoring process http://gerrit.ovirt.org/#/c/12433/ The next step is, by my opinion, is start to use cache for not often changed object, VdsGroup is classical example of such object. ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Global transaction in ovirt-engine
- Original Message - From: Michael Kublin mkub...@redhat.com To: engine-devel engine-devel@ovirt.org Sent: Tuesday, February 12, 2013 11:38:33 AM Subject: [Engine-devel] Global transaction in ovirt-engine The issue is a following (I think I had raised it once), by default when any command is invoked in ovirt-engine a transaction is opened. Now, lets take a look on complicated command which is performing a lot of XML-RPC/HTTP calls and just simple update , such command is a main cause for the following problems: 1. Long transactions, each call increase a time of transaction 2. When transaction is opened a connection which is associated with it also in used, so a number of available DB connections reduced from connection pool. 3. A errors : transaction was cancelled and operation is failed on engine side, but successes at host side. 4. Performance - try to keep transactions as short as possible. 5. Performance - to many unneeded transactions 6. Bug debugging - it is very difficult to understand why something was committed and something not why transaction was reverted or not, when we have a lot of nested transactions. Now, what should be done by my opinion: 1. Change default behaviour - by default not to open a transaction (Today, transaction is opened and if we don't want a command should be marked by NonTransactiveCommandAttribute) - it is simple 2. The change described at 1 will require to rewrite and redesign some flows - more difficult, but also can be done pretty fast. By the way this can be or even should be done at any case. 3. All new commands should not be based on global transaction or keep in mind that global transaction can be disabled Benefits, 1. Better performance 2. Cleaner code 3. Less bugs 4. Increased scale Regards Michael ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel need to remember that transaction also gave us 10-min limit for command execution, not saying its good or bad, but disabling the transactions will allow commands to run forever, this 10 min limit actually, saved us few times when commands got stuck in all kind of locks etc.. Not exactly, if transaction has expired the JBOSS transaction framework (Arjuna) will not stop a running thread attached to such thread. We can configure it in such way that a running thread will be cancelled, but for now it is not. So current behavior is somewhat unexpected - it all determines if the thread will try to use the transaction after it was reaped off from him. Right? If this is the case - I think this is bad. Do we have a real requirement for commands to end their execution phase at 10 minutes? Propose patch for removing global transaction at endAction of CommandBase (this is a first step) http://gerrit.ovirt.org/#/c/12010/ ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
[Engine-devel] Global transaction in ovirt-engine
The issue is a following (I think I had raised it once), by default when any command is invoked in ovirt-engine a transaction is opened. Now, lets take a look on complicated command which is performing a lot of XML-RPC/HTTP calls and just simple update , such command is a main cause for the following problems: 1. Long transactions, each call increase a time of transaction 2. When transaction is opened a connection which is associated with it also in used, so a number of available DB connections reduced from connection pool. 3. A errors : transaction was cancelled and operation is failed on engine side, but successes at host side. 4. Performance - try to keep transactions as short as possible. 5. Performance - to many unneeded transactions 6. Bug debugging - it is very difficult to understand why something was committed and something not why transaction was reverted or not, when we have a lot of nested transactions. Now, what should be done by my opinion: 1. Change default behaviour - by default not to open a transaction (Today, transaction is opened and if we don't want a command should be marked by NonTransactiveCommandAttribute) - it is simple 2. The change described at 1 will require to rewrite and redesign some flows - more difficult, but also can be done pretty fast. By the way this can be or even should be done at any case. 3. All new commands should not be based on global transaction or keep in mind that global transaction can be disabled Benefits, 1. Better performance 2. Cleaner code 3. Less bugs 4. Increased scale Regards Michael ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Global transaction in ovirt-engine
- Original Message - From: Omer Frenkel ofren...@redhat.com To: Michael Kublin mkub...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Tuesday, February 12, 2013 4:32:44 PM Subject: Re: [Engine-devel] Global transaction in ovirt-engine - Original Message - From: Michael Kublin mkub...@redhat.com To: engine-devel engine-devel@ovirt.org Sent: Tuesday, February 12, 2013 11:38:33 AM Subject: [Engine-devel] Global transaction in ovirt-engine The issue is a following (I think I had raised it once), by default when any command is invoked in ovirt-engine a transaction is opened. Now, lets take a look on complicated command which is performing a lot of XML-RPC/HTTP calls and just simple update , such command is a main cause for the following problems: 1. Long transactions, each call increase a time of transaction 2. When transaction is opened a connection which is associated with it also in used, so a number of available DB connections reduced from connection pool. 3. A errors : transaction was cancelled and operation is failed on engine side, but successes at host side. 4. Performance - try to keep transactions as short as possible. 5. Performance - to many unneeded transactions 6. Bug debugging - it is very difficult to understand why something was committed and something not why transaction was reverted or not, when we have a lot of nested transactions. Now, what should be done by my opinion: 1. Change default behaviour - by default not to open a transaction (Today, transaction is opened and if we don't want a command should be marked by NonTransactiveCommandAttribute) - it is simple 2. The change described at 1 will require to rewrite and redesign some flows - more difficult, but also can be done pretty fast. By the way this can be or even should be done at any case. 3. All new commands should not be based on global transaction or keep in mind that global transaction can be disabled Benefits, 1. Better performance 2. Cleaner code 3. Less bugs 4. Increased scale Regards Michael ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel need to remember that transaction also gave us 10-min limit for command execution, not saying its good or bad, but disabling the transactions will allow commands to run forever, this 10 min limit actually, saved us few times when commands got stuck in all kind of locks etc.. Not exactly, if transaction has expired the JBOSS transaction framework (Arjuna) will not stop a running thread attached to such thread. We can configure it in such way that a running thread will be cancelled, but for now it is not. ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Guid NGuid
- Original Message - From: Laszlo Hornyak lhorn...@redhat.com To: Michael Kublin mkub...@redhat.com Cc: engine-devel@ovirt.org Sent: Monday, February 4, 2013 9:56:28 AM Subject: Re: [Engine-devel] Guid NGuid - Original Message - From: Michael Kublin mkub...@redhat.com To: engine-devel@ovirt.org Sent: Monday, February 4, 2013 8:47:29 AM Subject: Re: [Engine-devel] Guid NGuid - Original Message - From: Roy Golan rgo...@redhat.com To: engine-devel@ovirt.org Sent: Monday, February 4, 2013 9:18:21 AM Subject: Re: [Engine-devel] Guid NGuid On 02/03/2013 03:19 PM, Alon Bar-Lev wrote: - Original Message - From: Omer Frenkel ofren...@redhat.com To: Michael Kublin mkub...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Sunday, February 3, 2013 3:12:19 PM Subject: Re: [Engine-devel] Guid NGuid - Original Message - From: Michael Kublin mkub...@redhat.com To: engine-devel engine-devel@ovirt.org Sent: Sunday, February 3, 2013 3:10:14 PM Subject: [Engine-devel] Guid NGuid Hi, In ovirt-engine code we have Guid and NGuid objects. Guid is extends NGuid and also NGuid class has method getValue() which should return Guid. As for me these two classes are look like the same and I don't see to much differences between them. My proposal is to remove NGuid and move it functionality to Guid (Because of Guid is much more common) i agree, but we need to take another step forward and allow Guid to be null (as it should) and not assume its EMPTY or have a value (i'm pretty sure we have this assumption in many places) Hi, And for the new people out here... why not kill both and use plain standard java UUID[1]? +1 for using java.util.UUID NGuid functionality that should be extracted during the refactor is - 1. refactor DB and Java so empty or null return values are null and not EMPTY_GUID 2. the special constructor of NGuid for UUID return by Microsoft AD should be extracted to a factory/utility I think we should kill compat, I don't see any value in fixing anything about it while leaving it intact. Regards, Alon Bar-Lev. [1] http://docs.oracle.com/javase/6/docs/api/java/util/UUID.html Actually there is exists one reason not to use directly UUID at server side. The main operations on Guid today it is to convert object to Guid or Guid to string. Guid it is immutable object, number of Guids is limited and almost never changed, These sound like classical case for object that can be cached. Benefit - reduced number of string manipulations, reduced number of created instances, less work for garbage collection UUID has the same properties, it is immutable. So you could do the same with UUID, but I am not sure you could effectively cache these objects and prove that you are saving either CPU, heap, or GC time. These is a very common pattern: 1) implementation of valueOf() for most classes like Integer, Double, etc... has some kind of cache. 2) JVM has cache for strings that can be used and that cache can be tweaked by some JVM opts. 3) Most of our operations is perform query on DB, send request to host or parse response from host and Guid is very common object that is immutable. I am agree that these will not solve all our performance problems but can provide some benefit, especially when it is very easy to implement. ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Guid NGuid
- Original Message - From: Mike Kolesnik mkole...@redhat.com To: Michael Kublin mkub...@redhat.com Cc: engine-devel@ovirt.org Sent: Monday, February 4, 2013 11:48:45 AM Subject: Re: [Engine-devel] Guid NGuid - Original Message - - Original Message - From: Laszlo Hornyak lhorn...@redhat.com To: Michael Kublin mkub...@redhat.com Cc: engine-devel@ovirt.org Sent: Monday, February 4, 2013 9:56:28 AM Subject: Re: [Engine-devel] Guid NGuid - Original Message - From: Michael Kublin mkub...@redhat.com To: engine-devel@ovirt.org Sent: Monday, February 4, 2013 8:47:29 AM Subject: Re: [Engine-devel] Guid NGuid - Original Message - From: Roy Golan rgo...@redhat.com To: engine-devel@ovirt.org Sent: Monday, February 4, 2013 9:18:21 AM Subject: Re: [Engine-devel] Guid NGuid On 02/03/2013 03:19 PM, Alon Bar-Lev wrote: - Original Message - From: Omer Frenkel ofren...@redhat.com To: Michael Kublin mkub...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Sunday, February 3, 2013 3:12:19 PM Subject: Re: [Engine-devel] Guid NGuid - Original Message - From: Michael Kublin mkub...@redhat.com To: engine-devel engine-devel@ovirt.org Sent: Sunday, February 3, 2013 3:10:14 PM Subject: [Engine-devel] Guid NGuid Hi, In ovirt-engine code we have Guid and NGuid objects. Guid is extends NGuid and also NGuid class has method getValue() which should return Guid. As for me these two classes are look like the same and I don't see to much differences between them. My proposal is to remove NGuid and move it functionality to Guid (Because of Guid is much more common) i agree, but we need to take another step forward and allow Guid to be null (as it should) and not assume its EMPTY or have a value (i'm pretty sure we have this assumption in many places) Hi, And for the new people out here... why not kill both and use plain standard java UUID[1]? +1 for using java.util.UUID NGuid functionality that should be extracted during the refactor is - 1. refactor DB and Java so empty or null return values are null and not EMPTY_GUID 2. the special constructor of NGuid for UUID return by Microsoft AD should be extracted to a factory/utility I think we should kill compat, I don't see any value in fixing anything about it while leaving it intact. Regards, Alon Bar-Lev. [1] http://docs.oracle.com/javase/6/docs/api/java/util/UUID.html Actually there is exists one reason not to use directly UUID at server side. The main operations on Guid today it is to convert object to Guid or Guid to string. Guid it is immutable object, number of Guids is limited and almost never changed, These sound like classical case for object that can be cached. Benefit - reduced number of string manipulations, reduced number of created instances, less work for garbage collection UUID has the same properties, it is immutable. So you could do the same with UUID, but I am not sure you could effectively cache these objects and prove that you are saving either CPU, heap, or GC time. These is a very common pattern: 1) implementation of valueOf() for most classes like Integer, Double, etc... has some kind of cache. 2) JVM has cache for strings that can be used and that cache can be tweaked by some JVM opts. 3) Most of our operations is perform query on DB, send request to host or parse response from host and Guid is very common object that is immutable. I am agree that these will not solve all our performance problems but can provide some benefit, especially when it is very easy to implement. We could still achieve that using a UUIDCreator class and call it instead of Guid.fromString().. Whether this is cached or not is another question which can be solved later and checked to see if the cache improves performance or not. These is already implementation, if it will be Guid.fromString() or UUIDCreator. The issue is if we want to use cache, it should be implemented together with deleting/replacing of Guid/NGuid , because of it much more easilly ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
[Engine-devel] Guid NGuid
Hi, In ovirt-engine code we have Guid and NGuid objects. Guid is extends NGuid and also NGuid class has method getValue() which should return Guid. As for me these two classes are look like the same and I don't see to much differences between them. My proposal is to remove NGuid and move it functionality to Guid (Because of Guid is much more common) Regards Michael ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Guid NGuid
- Original Message - From: Roy Golan rgo...@redhat.com To: engine-devel@ovirt.org Sent: Monday, February 4, 2013 9:18:21 AM Subject: Re: [Engine-devel] Guid NGuid On 02/03/2013 03:19 PM, Alon Bar-Lev wrote: - Original Message - From: Omer Frenkel ofren...@redhat.com To: Michael Kublin mkub...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Sunday, February 3, 2013 3:12:19 PM Subject: Re: [Engine-devel] Guid NGuid - Original Message - From: Michael Kublin mkub...@redhat.com To: engine-devel engine-devel@ovirt.org Sent: Sunday, February 3, 2013 3:10:14 PM Subject: [Engine-devel] Guid NGuid Hi, In ovirt-engine code we have Guid and NGuid objects. Guid is extends NGuid and also NGuid class has method getValue() which should return Guid. As for me these two classes are look like the same and I don't see to much differences between them. My proposal is to remove NGuid and move it functionality to Guid (Because of Guid is much more common) i agree, but we need to take another step forward and allow Guid to be null (as it should) and not assume its EMPTY or have a value (i'm pretty sure we have this assumption in many places) Hi, And for the new people out here... why not kill both and use plain standard java UUID[1]? +1 for using java.util.UUID NGuid functionality that should be extracted during the refactor is - 1. refactor DB and Java so empty or null return values are null and not EMPTY_GUID 2. the special constructor of NGuid for UUID return by Microsoft AD should be extracted to a factory/utility I think we should kill compat, I don't see any value in fixing anything about it while leaving it intact. Regards, Alon Bar-Lev. [1] http://docs.oracle.com/javase/6/docs/api/java/util/UUID.html Actually there is exists one reason not to use directly UUID at server side. The main operations on Guid today it is to convert object to Guid or Guid to string. Guid it is immutable object, number of Guids is limited and almost never changed, These sound like classical case for object that can be cached. Benefit - reduced number of string manipulations, reduced number of created instances, less work for garbage collection ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
[Engine-devel] Improving of exist memory lock infrastructure in ovirt-engine
The following link http://www.ovirt.org/Features/DetailedLockMechanism contains description and design of in memory lock infrastructure. The design is describing already existing infrastructure with additional changes that should be done in order to improve it. The idea is to allow using of in memory locks with commands that can create asynchronous tasks. Regards Michael ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] ThreadPoolUtil with 500 threads and no queue?
- Original Message - From: Juan Hernandez jhern...@redhat.com To: engine-devel@ovirt.org Sent: Wednesday, December 12, 2012 6:06:30 PM Subject: [Engine-devel] ThreadPoolUtil with 500 threads and no queue? Hello all, What is the reasoning behind the decision to have a pool with a maximum of 500 threads and no job queue (see ThreadPoolUtil.java)? Wouldn't it make more sense to have a much smaller thread pool and a potentially large queue of jobs? Regards, Juan Hernandez There are three general strategies for queuing: 1) Direct handoffs. A good default choice for a work queue is a SynchronousQueue that hands off tasks to threads without otherwise holding them. Here, an attempt to queue a task will fail if no threads are immediately available to run it, so a new thread will be constructed. This policy avoids lockups when handling sets of requests that might have internal dependencies. Direct handoffs generally require unbounded maximumPoolSizes to avoid rejection of new submitted tasks. This in turn admits the possibility of unbounded thread growth when commands continue to arrive on average faster than they can be processed. 2) Unbounded queues. Using an unbounded queue (for example a LinkedBlockingQueue without a predefined capacity) will cause new tasks to wait in the queue when all corePoolSize threads are busy. Thus, no more than corePoolSize threads will ever be created. (And the value of the maximumPoolSize therefore doesn't have any effect.) This may be appropriate when each task is completely independent of others, so tasks cannot affect each others execution; for example, in a web page server. While this style of queuing can be useful in smoothing out transient bursts of requests, it admits the possibility of unbounded work queue growth when commands continue to arrive on average faster than they can be processed. 3) Bounded queues. A bounded queue (for example, an ArrayBlockingQueue) helps prevent resource exhaustion when used with finite maximumPoolSizes, but can be more difficult to tune and control. Queue sizes and maximum pool sizes may be traded off for each other: Using large queues and small pools minimizes CPU usage, OS resources, and context-switching overhead, but can lead to artificially low throughput. If tasks frequently block (for example if they are I/O bound), a system may be able to schedule time for more threads than you otherwise allow. Use of small queues generally requires larger pool sizes, which keeps CPUs busier but may encounter unacceptable scheduling overhead, which also decreases throughput. Why not? we are using 1). Actually 500 threads should be enough for very big applications ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
[Engine-devel] Task cancelation feature
Hi, I created a wiki page with design of task cancellation feature. The url is : http://www.ovirt.org/Features/TaskManagerCancelTask I can not call these design, I have not any requirements , except a name of the feature, so my wiki doesn't contains anything except open questions. Also, I think that it is impossible to make a good feature based on very problematic infrastructure, I think before we should fix all our infrastructure problems, and after that to add any cancellation task feature will be a meter of couple hours of work Regards Michael ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] [Engine-Devel] OvfDataUpdater
- Original Message - From: Liron Aravot lara...@redhat.com To: engine-devel@ovirt.org Sent: Monday, November 19, 2012 2:03:57 PM Subject: [Engine-devel] [Engine-Devel] OvfDataUpdater starting a new discussion thread. - Original Message - From: Mike Kolesnik mkole...@redhat.com To: Liron Aravot lara...@redhat.com Sent: Monday, November 19, 2012 12:42:10 PM Subject: Re: [Engine-devel] OvfAutoUpdater I think 'version' is a more standard term for the column names. Also in: 4. If succesfull - for each vm update the ovf_generation to equal the db_generation. I think you mean that the update should be to the entity version you initially selected. 1. I think should be version = version +1; 2. Now the quartz is automatic process, updateVmInSpm it is spm operation -it means it can trigger a reconstruct and spm election. 3. How your solution is handling a case of hotPlug/hotUnplug/remove/add of vm disks. vm_static is not usually updated. 4. Reconstruct/Recovery - updateVmInSpm should be called on all vms, no matter if they were updated. Regards, Mike - Original Message - http://wiki.ovirt.org/wiki/Feature/OvfAutoUpdater Hi all, i'll be glad if you could review the wiki page of OvfAutoUpdater, if you have any suggestions or ideas please let me know. http://wiki.ovirt.org/wiki/Feature/OvfAutoUpdater short preview from the wiki: vm/template configurations (including disks info) are stored on the master storage domain for backup purposes, import/export and also to provide the abillity to run VMs without having a running engine/db. Currently ovf update is done synchronously when performing various operations on vms/templates - update, save, adding/removing a disk, etc. What's more, currently updating the ovf (updateVM vdsm call) is usually done within a transcation. The idea behined OvfAutoUpdater is to perform batch ovf update operations that aggregate all outstanding updates per data center. These updates will be done in specifed time intervals which will reduce XML-RPC calls and will enable the removal of this syncronous vdsm call from all over the code. Thanks, Liron Aravot. ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] [Engine-Devel] OvfDataUpdater
- Original Message - From: Liron Aravot lara...@redhat.com To: Mike Kolesnik mkole...@redhat.com Cc: engine-devel@ovirt.org, Michael Kublin mkub...@redhat.com Sent: Monday, November 19, 2012 2:52:56 PM Subject: Re: [Engine-devel] [Engine-Devel] OvfDataUpdater - Original Message - From: Mike Kolesnik mkole...@redhat.com To: Liron Aravot lara...@redhat.com Cc: engine-devel@ovirt.org, Michael Kublin mkub...@redhat.com Sent: Monday, November 19, 2012 2:44:56 PM Subject: Re: [Engine-devel] [Engine-Devel] OvfDataUpdater - Original Message - From: Liron Aravot lara...@redhat.com To: engine-devel@ovirt.org Sent: Monday, November 19, 2012 2:03:57 PM Subject: [Engine-devel] [Engine-Devel] OvfDataUpdater starting a new discussion thread. - Original Message - From: Mike Kolesnik mkole...@redhat.com To: Liron Aravot lara...@redhat.com Sent: Monday, November 19, 2012 12:42:10 PM Subject: Re: [Engine-devel] OvfAutoUpdater I think 'version' is a more standard term for the column names. Also in: 4. If succesfull - for each vm update the ovf_generation to equal the db_generation. I think you mean that the update should be to the entity version you initially selected. 1. I think should be version = version +1; the version will be the db_generation that was loaded when we loaded the vm/template. for example - if the vm/template was updated twice between OvfUpdater runs, we will need to increment twice - so incrementing by one will cause another unneeded ovf update for that vm. I am talking about ovfVersion or how you called it. 2. Now the quartz is automatic process, updateVmInSpm it is spm operation -it means it can trigger a reconstruct and spm election. 2. updateVmInSpm command would be changed so it won't trigger failover, will add that to the wiki. 3. How your solution is handling a case of hotPlug/hotUnplug/remove/add of vm disks. vm_static is not usually updated. Whenever will be a command that locks a vm/template, you will have an option during the unlock to specify if the version need to be incremented, you'll be able to increment it also yourself. The HotUnPlug command does lock the vm and during endSuccesfully attempt to update the vm in spm, so it will increment the db_version/generation instead. addDisk, removeDisk, hotPlug/unPlug not locking a vm. Also, as far as I know LockVm/UnLockVm it is operation on vm_dynamic and not vm_static. Also consider snapshots vNICs which affect the VM state. Whatever affect the vm state and triggers updateVmInSpm today will continue to, it can be added to flows which don't do that today. 4. Reconstruct/Recovery - updateVmInSpm should be called on all vms, no matter if they were updated. Ofcourse, it's being taken care of. How? Regards, Mike - Original Message - http://wiki.ovirt.org/wiki/Feature/OvfAutoUpdater Hi all, i'll be glad if you could review the wiki page of OvfAutoUpdater, if you have any suggestions or ideas please let me know. http://wiki.ovirt.org/wiki/Feature/OvfAutoUpdater short preview from the wiki: vm/template configurations (including disks info) are stored on the master storage domain for backup purposes, import/export and also to provide the abillity to run VMs without having a running engine/db. Currently ovf update is done synchronously when performing various operations on vms/templates - update, save, adding/removing a disk, etc. What's more, currently updating the ovf (updateVM vdsm call) is usually done within a transcation. The idea behined OvfAutoUpdater is to perform batch ovf update operations that aggregate all outstanding updates per data center. These updates will be done in specifed time intervals which will reduce XML-RPC calls and will enable the removal of this syncronous vdsm call from all over the code. Thanks, Liron Aravot. ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] [Engine-Devel] OvfDataUpdater
- Original Message - From: Liron Aravot lara...@redhat.com To: Michael Kublin mkub...@redhat.com Cc: engine-devel@ovirt.org Sent: Monday, November 19, 2012 3:17:04 PM Subject: Re: [Engine-devel] [Engine-Devel] OvfDataUpdater - Original Message - From: Michael Kublin mkub...@redhat.com To: Liron Aravot lara...@redhat.com Cc: engine-devel@ovirt.org Sent: Monday, November 19, 2012 3:03:23 PM Subject: Re: [Engine-devel] [Engine-Devel] OvfDataUpdater - Original Message - From: Liron Aravot lara...@redhat.com To: Mike Kolesnik mkole...@redhat.com Cc: engine-devel@ovirt.org, Michael Kublin mkub...@redhat.com Sent: Monday, November 19, 2012 2:52:56 PM Subject: Re: [Engine-devel] [Engine-Devel] OvfDataUpdater - Original Message - From: Mike Kolesnik mkole...@redhat.com To: Liron Aravot lara...@redhat.com Cc: engine-devel@ovirt.org, Michael Kublin mkub...@redhat.com Sent: Monday, November 19, 2012 2:44:56 PM Subject: Re: [Engine-devel] [Engine-Devel] OvfDataUpdater - Original Message - From: Liron Aravot lara...@redhat.com To: engine-devel@ovirt.org Sent: Monday, November 19, 2012 2:03:57 PM Subject: [Engine-devel] [Engine-Devel] OvfDataUpdater starting a new discussion thread. - Original Message - From: Mike Kolesnik mkole...@redhat.com To: Liron Aravot lara...@redhat.com Sent: Monday, November 19, 2012 12:42:10 PM Subject: Re: [Engine-devel] OvfAutoUpdater I think 'version' is a more standard term for the column names. Also in: 4. If succesfull - for each vm update the ovf_generation to equal the db_generation. I think you mean that the update should be to the entity version you initially selected. 1. I think should be version = version +1; the version will be the db_generation that was loaded when we loaded the vm/template. for example - if the vm/template was updated twice between OvfUpdater runs, we will need to increment twice - so incrementing by one will cause another unneeded ovf update for that vm. I am talking about ovfVersion or how you called it. me too, if we update the vm three times between OvfAutoUpdater runs , we will have the following values in the loaded vm for example: db_generation 8 ovf_generation 5 when we perform the ovf update , ovf_generation should be set to 8 and not to 6, as version '8' is the version that we wrote the ovf metadata of. These should be written in wiki, with remarks that values which were retrieved from DB together. 2. Now the quartz is automatic process, updateVmInSpm it is spm operation -it means it can trigger a reconstruct and spm election. 2. updateVmInSpm command would be changed so it won't trigger failover, will add that to the wiki. 3. How your solution is handling a case of hotPlug/hotUnplug/remove/add of vm disks. vm_static is not usually updated. Whenever will be a command that locks a vm/template, you will have an option during the unlock to specify if the version need to be incremented, you'll be able to increment it also yourself. The HotUnPlug command does lock the vm and during endSuccesfully attempt to update the vm in spm, so it will increment the db_version/generation instead. addDisk, removeDisk, hotPlug/unPlug not locking a vm. Also, as far as I know LockVm/UnLockVm it is operation on vm_dynamic and not vm_static. hotplug acquires memory lock on the vm and has no async tasks if I recall correctly. Yes, correct and what? These what I said, vm_static is not updated. adddisk acquires shared lock on the vm which is problematic with today flow as well - as you have a race during updateVmInSpm, it should be fixed regardless of the OvfAutoUpdater. It is not answer on my question. How ovf will be updated? How you understand that ovf of vm should be updated? Also consider snapshots vNICs which affect the VM state. Whatever affect the vm state and triggers updateVmInSpm today will continue to, it can be added to flows which don't do that today. 4. Reconstruct/Recovery - updateVmInSpm should be called on all vms, no matter if they were updated. Ofcourse, it's being taken care of. How? we don't have ovf's of the vms when we reconstruct as we don't have master domain - so we should set the ovf_generation of the vms to be 0 - which will cause ovfautoupdater to copy the vm metadata. I did not understand that statement and how it is related. a performance improvement may be introduced later on in case of wrong master version, but that's not related to the ovfautoupdater. I don't understand connection to performance. Now you are replacing a calls
Re: [Engine-devel] [Engine-Devel] OvfDataUpdater
- Original Message - From: Liron Aravot lara...@redhat.com To: Michael Kublin mkub...@redhat.com Cc: engine-devel@ovirt.org Sent: Monday, November 19, 2012 3:34:11 PM Subject: Re: [Engine-devel] [Engine-Devel] OvfDataUpdater - Original Message - From: Michael Kublin mkub...@redhat.com To: Liron Aravot lara...@redhat.com Cc: engine-devel@ovirt.org Sent: Monday, November 19, 2012 3:28:06 PM Subject: Re: [Engine-devel] [Engine-Devel] OvfDataUpdater - Original Message - From: Liron Aravot lara...@redhat.com To: Michael Kublin mkub...@redhat.com Cc: engine-devel@ovirt.org Sent: Monday, November 19, 2012 3:17:04 PM Subject: Re: [Engine-devel] [Engine-Devel] OvfDataUpdater - Original Message - From: Michael Kublin mkub...@redhat.com To: Liron Aravot lara...@redhat.com Cc: engine-devel@ovirt.org Sent: Monday, November 19, 2012 3:03:23 PM Subject: Re: [Engine-devel] [Engine-Devel] OvfDataUpdater - Original Message - From: Liron Aravot lara...@redhat.com To: Mike Kolesnik mkole...@redhat.com Cc: engine-devel@ovirt.org, Michael Kublin mkub...@redhat.com Sent: Monday, November 19, 2012 2:52:56 PM Subject: Re: [Engine-devel] [Engine-Devel] OvfDataUpdater - Original Message - From: Mike Kolesnik mkole...@redhat.com To: Liron Aravot lara...@redhat.com Cc: engine-devel@ovirt.org, Michael Kublin mkub...@redhat.com Sent: Monday, November 19, 2012 2:44:56 PM Subject: Re: [Engine-devel] [Engine-Devel] OvfDataUpdater - Original Message - From: Liron Aravot lara...@redhat.com To: engine-devel@ovirt.org Sent: Monday, November 19, 2012 2:03:57 PM Subject: [Engine-devel] [Engine-Devel] OvfDataUpdater starting a new discussion thread. - Original Message - From: Mike Kolesnik mkole...@redhat.com To: Liron Aravot lara...@redhat.com Sent: Monday, November 19, 2012 12:42:10 PM Subject: Re: [Engine-devel] OvfAutoUpdater I think 'version' is a more standard term for the column names. Also in: 4. If succesfull - for each vm update the ovf_generation to equal the db_generation. I think you mean that the update should be to the entity version you initially selected. 1. I think should be version = version +1; the version will be the db_generation that was loaded when we loaded the vm/template. for example - if the vm/template was updated twice between OvfUpdater runs, we will need to increment twice - so incrementing by one will cause another unneeded ovf update for that vm. I am talking about ovfVersion or how you called it. me too, if we update the vm three times between OvfAutoUpdater runs , we will have the following values in the loaded vm for example: db_generation 8 ovf_generation 5 when we perform the ovf update , ovf_generation should be set to 8 and not to 6, as version '8' is the version that we wrote the ovf metadata of. These should be written in wiki, with remarks that values which were retrieved from DB together. 2. Now the quartz is automatic process, updateVmInSpm it is spm operation -it means it can trigger a reconstruct and spm election. 2. updateVmInSpm command would be changed so it won't trigger failover, will add that to the wiki. 3. How your solution is handling a case of hotPlug/hotUnplug/remove/add of vm disks. vm_static is not usually updated. Whenever will be a command that locks a vm/template, you will have an option during the unlock to specify if the version need to be incremented, you'll be able to increment it also yourself. The HotUnPlug command does lock the vm and during endSuccesfully attempt to update the vm in spm, so it will increment the db_version/generation instead. addDisk, removeDisk, hotPlug/unPlug not locking a vm. Also, as far as I know LockVm/UnLockVm it is operation on vm_dynamic and not vm_static. hotplug acquires memory lock on the vm and has no async tasks if I recall correctly. Yes, correct and what? These what I said, vm_static is not updated. you will just call the dao method for incrementing the db_generation, you don't have to perform vm_static update. adddisk acquires shared lock on the vm which is problematic with today flow as well - as you have a race during updateVmInSpm, it should be fixed regardless of the OvfAutoUpdater. It is not answer on my question. How ovf will be updated? How
Re: [Engine-devel] [Engine-Devel] OvfDataUpdater
- Original Message - From: Liron Aravot lara...@redhat.com To: Michael Kublin mkub...@redhat.com Cc: engine-devel@ovirt.org Sent: Monday, November 19, 2012 3:58:55 PM Subject: Re: [Engine-devel] [Engine-Devel] OvfDataUpdater - Original Message - From: Michael Kublin mkub...@redhat.com To: Liron Aravot lara...@redhat.com Cc: engine-devel@ovirt.org Sent: Monday, November 19, 2012 3:44:41 PM Subject: Re: [Engine-devel] [Engine-Devel] OvfDataUpdater - Original Message - From: Liron Aravot lara...@redhat.com To: Michael Kublin mkub...@redhat.com Cc: engine-devel@ovirt.org Sent: Monday, November 19, 2012 3:34:11 PM Subject: Re: [Engine-devel] [Engine-Devel] OvfDataUpdater - Original Message - From: Michael Kublin mkub...@redhat.com To: Liron Aravot lara...@redhat.com Cc: engine-devel@ovirt.org Sent: Monday, November 19, 2012 3:28:06 PM Subject: Re: [Engine-devel] [Engine-Devel] OvfDataUpdater - Original Message - From: Liron Aravot lara...@redhat.com To: Michael Kublin mkub...@redhat.com Cc: engine-devel@ovirt.org Sent: Monday, November 19, 2012 3:17:04 PM Subject: Re: [Engine-devel] [Engine-Devel] OvfDataUpdater - Original Message - From: Michael Kublin mkub...@redhat.com To: Liron Aravot lara...@redhat.com Cc: engine-devel@ovirt.org Sent: Monday, November 19, 2012 3:03:23 PM Subject: Re: [Engine-devel] [Engine-Devel] OvfDataUpdater - Original Message - From: Liron Aravot lara...@redhat.com To: Mike Kolesnik mkole...@redhat.com Cc: engine-devel@ovirt.org, Michael Kublin mkub...@redhat.com Sent: Monday, November 19, 2012 2:52:56 PM Subject: Re: [Engine-devel] [Engine-Devel] OvfDataUpdater - Original Message - From: Mike Kolesnik mkole...@redhat.com To: Liron Aravot lara...@redhat.com Cc: engine-devel@ovirt.org, Michael Kublin mkub...@redhat.com Sent: Monday, November 19, 2012 2:44:56 PM Subject: Re: [Engine-devel] [Engine-Devel] OvfDataUpdater - Original Message - From: Liron Aravot lara...@redhat.com To: engine-devel@ovirt.org Sent: Monday, November 19, 2012 2:03:57 PM Subject: [Engine-devel] [Engine-Devel] OvfDataUpdater starting a new discussion thread. - Original Message - From: Mike Kolesnik mkole...@redhat.com To: Liron Aravot lara...@redhat.com Sent: Monday, November 19, 2012 12:42:10 PM Subject: Re: [Engine-devel] OvfAutoUpdater I think 'version' is a more standard term for the column names. Also in: 4. If succesfull - for each vm update the ovf_generation to equal the db_generation. I think you mean that the update should be to the entity version you initially selected. 1. I think should be version = version +1; the version will be the db_generation that was loaded when we loaded the vm/template. for example - if the vm/template was updated twice between OvfUpdater runs, we will need to increment twice - so incrementing by one will cause another unneeded ovf update for that vm. I am talking about ovfVersion or how you called it. me too, if we update the vm three times between OvfAutoUpdater runs , we will have the following values in the loaded vm for example: db_generation 8 ovf_generation 5 when we perform the ovf update , ovf_generation should be set to 8 and not to 6, as version '8' is the version that we wrote the ovf metadata of. These should be written in wiki, with remarks that values which were retrieved from DB together. 2. Now the quartz is automatic process, updateVmInSpm it is spm operation -it means it can trigger a reconstruct and spm election. 2. updateVmInSpm command would be changed so it won't trigger failover, will add that to the wiki. 3. How your solution is handling a case of hotPlug/hotUnplug/remove/add of vm disks. vm_static is not usually updated. Whenever will be a command that locks a vm/template, you will have an option during the unlock to specify if the version need to be incremented, you'll be able to increment it also yourself
Re: [Engine-devel] [Engine-Devel] OvfDataUpdater
starting a new discussion thread. - Original Message - From: Mike Kolesnik mkole...@redhat.com To: Liron Aravot lara...@redhat.com Sent: Monday, November 19, 2012 12:42:10 PM Subject: Re: [Engine-devel] OvfAutoUpdater I think 'version' is a more standard term for the column names. Also in: 4. If succesfull - for each vm update the ovf_generation to equal the db_generation. I think you mean that the update should be to the entity version you initially selected. 1. I think should be version = version +1; the version will be the db_generation that was loaded when we loaded the vm/template. for example - if the vm/template was updated twice between OvfUpdater runs, we will need to increment twice - so incrementing by one will cause another unneeded ovf update for that vm. I am talking about ovfVersion or how you called it. me too, if we update the vm three times between OvfAutoUpdater runs , we will have the following values in the loaded vm for example: db_generation 8 ovf_generation 5 when we perform the ovf update , ovf_generation should be set to 8 and not to 6, as version '8' is the version that we wrote the ovf metadata of. These should be written in wiki, with remarks that values which were retrieved from DB together. 2. Now the quartz is automatic process, updateVmInSpm it is spm operation -it means it can trigger a reconstruct and spm election. 2. updateVmInSpm command would be changed so it won't trigger failover, will add that to the wiki. 3. How your solution is handling a case of hotPlug/hotUnplug/remove/add of vm disks. vm_static is not usually updated. Whenever will be a command that locks a vm/template, you will have an option during the unlock to specify if the version need to be incremented, you'll be able to increment it also yourself. The HotUnPlug command does lock the vm and during endSuccesfully attempt to update the vm in spm, so it will increment the db_version/generation instead. addDisk, removeDisk, hotPlug/unPlug not locking a vm. Also, as far as I know LockVm/UnLockVm it is operation on vm_dynamic and not vm_static. hotplug acquires memory lock on the vm and has no async tasks if I recall correctly. Yes, correct and what? These what I said, vm_static is not updated. you will just call the dao method for incrementing the db_generation, you don't have to perform vm_static update. adddisk acquires shared lock on the vm which is problematic with today flow as well - as you have a race during updateVmInSpm, it should be fixed regardless of the OvfAutoUpdater. It is not answer on my question. How ovf will be updated? How you understand that ovf of vm should be updated? the ovf will be updated when you increment the db_generation value instead of calling updateVmInSpm(). you will be able to do that from the dao directly when you want, or when you unlock a vm/template. when you will update the db_generation, the vm/template would be picked up by OvfUpdater as it's db_generation is different than its ovf_generation. 1. Update of ovfs is not related to locks and it should not be, I don't understand why you link them. 2. Races - your automatic process and couple of add disk commands. Sorry, part of my response was cutted by mistake, adding it As of today - there is a race. After adding ovfautoupdater , you won't update the ovf for vm which is locked or has a locked disk (written in the wiki). Vm/disk can be moved to locked after your query, at
Re: [Engine-devel] Serial Execution of Async Tasks
- Original Message -
From: Ayal Baron aba...@redhat.com
To: Michael Kublin mkub...@redhat.com
Cc: Liron Aravot lara...@redhat.com, engine-devel
engine-devel@ovirt.org, Eduardo Warszawski
ewars...@redhat.com, Allon Mureinik amure...@redhat.com
Sent: Sunday, September 23, 2012 5:27:54 PM
Subject: Re: [Engine-devel] Serial Execution of Async Tasks
- Original Message -
- Original Message -
From: Ayal Baron aba...@redhat.com
To: Allon Mureinik amure...@redhat.com, Michael Kublin
mkub...@redhat.com
Cc: Liron Aravot lara...@redhat.com, engine-devel
engine-devel@ovirt.org, Eduardo Warszawski
ewars...@redhat.com
Sent: Sunday, September 23, 2012 1:10:07 PM
Subject: Re: [Engine-devel] Serial Execution of Async Tasks
- Original Message -
- Original Message -
From: Michael Kublin mkub...@redhat.com
To: Allon Mureinik amure...@redhat.com
Cc: Eduardo Warszawski ewars...@redhat.com, Liron
Aravot
lara...@redhat.com, Maor Lipchuk
mlipc...@redhat.com, engine-devel
engine-devel@ovirt.org
Sent: Sunday, September 23, 2012 12:41:05 PM
Subject: Re: [Engine-devel] Serial Execution of Async Tasks
Hi guys,
As you may know the engine currently has the
ability
to
fire
an
SPM
task, and be asynchronously be woken-up when it
ends.
This is great, but we found the for the Live
Storage
Migration
feature we need something a bit complex - the
ability
to
have a
series of async tasks in a single control flow.
Here's my initial design for this, your comments
and
criticism
would
be welcome:
http://wiki.ovirt.org/wiki/Features/Serial_Execution_of_Asynchronous_Tasks_Detailed_Design
-successful execution -
* CommandBase iterates over its
SPMAsyncTaskHandlers
-
when?
This is the new suggested format of executeCommand().
I'll
clarify
this too.
* If the second task is an HSM command (vs. SPM
command),
I
think you
should explain in the design how to handle such
flows
as
well.
HSM commands do not create AsyncTasks, as they do
today
-
I
will
clarify this.
* Why do we need before task? can you give a
concrete
example
of what
would you do in such a method.
Basically, /today/, command look like this:
executeCommand() {
doStuffInTheDB();
runVdsCommand(someCommand);
}
endSuccessfully() {
doMoreStuffInTheDB();
}
endWithFailure() {
doMoreStuffForFailureInTheDB();
}
In the new design, the entire doStuffInTheDB() should
be
moved
to a
breforeTask of the (only) SPMAsyncTaskHandler.
- I see you added SPMAsyncTaskHandler, any reason
not
to
use
SPMAsyncTasK to manage it own life-cycle?
Conserving today's design - The SPMAsyncTaskHandler
is
the
place to
add additional, non-SPM, logic around the SPM task
execution,
like
CommandBase allows today.
- In the life-cycle managed by the
SPMAsyncTaskHandler
there
is a
step
'createTask - how to create the async task' can you
please
elaborate
what are the options.
new [any type of async task]
(I cleaned thread a little.)
The following design and it is implementation
http://gerrit.ovirt.org/#/c/7956/
is bad.
I don't see any reason for creating a new SPMAsyncTaskHandler
and
especially in the
way as it's done in patch.
The reason are following:
1. Performance , increased memory footprint, created CYCLIC
REFERENCE.
2. Readability and robust of code: the code which is written
as
cyclic references is unreadable
and difficult for debug.
3. Why I need a generic implementation and changes all over
whole
project because of
series of async commands, for me it is a private case?
What is the private case here exactly?
Every task can have multiple jobs. We've identified several such
places (e.g. live storage migration, move disk, move vm) and I
have
no doubt more will popup.
As Allon notes below, task handling is done at CommandBase, if
you
think task management should be for storage only, you're welcome
to
push it down to StorageHandlingCommandBase (or get rid of
inheritance here altogether).
Interesting , regards cyclic reference no response, but who cares,
it is difficult to answer , that's why better not to response?
There is no problem with cyclic references in general, GCs know how
to deal with these just
Re: [Engine-devel] Serial Execution of Async Tasks
Hi guys,
As you may know the engine currently has the ability to fire
an
SPM
task, and be asynchronously be woken-up when it ends.
This is great, but we found the for the Live Storage
Migration
feature we need something a bit complex - the ability to
have a
series of async tasks in a single control flow.
Here's my initial design for this, your comments and
criticism
would
be welcome:
http://wiki.ovirt.org/wiki/Features/Serial_Execution_of_Asynchronous_Tasks_Detailed_Design
-successful execution -
* CommandBase iterates over its SPMAsyncTaskHandlers - when?
This is the new suggested format of executeCommand(). I'll
clarify
this too.
* If the second task is an HSM command (vs. SPM command), I
think you
should explain in the design how to handle such flows as well.
HSM commands do not create AsyncTasks, as they do today - I
will
clarify this.
* Why do we need before task? can you give a concrete example
of what
would you do in such a method.
Basically, /today/, command look like this:
executeCommand() {
doStuffInTheDB();
runVdsCommand(someCommand);
}
endSuccessfully() {
doMoreStuffInTheDB();
}
endWithFailure() {
doMoreStuffForFailureInTheDB();
}
In the new design, the entire doStuffInTheDB() should be moved
to a
breforeTask of the (only) SPMAsyncTaskHandler.
- I see you added SPMAsyncTaskHandler, any reason not to use
SPMAsyncTasK to manage it own life-cycle?
Conserving today's design - The SPMAsyncTaskHandler is the
place to
add additional, non-SPM, logic around the SPM task execution,
like
CommandBase allows today.
- In the life-cycle managed by the SPMAsyncTaskHandler there
is a
step
'createTask - how to create the async task' can you please
elaborate
what are the options.
new [any type of async task]
(I cleaned thread a little.)
The following design and it is implementation http://gerrit.ovirt.org/#/c/7956/
is bad.
I don't see any reason for creating a new SPMAsyncTaskHandler and especially in
the
way as it's done in patch.
The reason are following:
1. Performance , increased memory footprint, created CYCLIC REFERENCE.
2. Readability and robust of code: the code which is written as cyclic
references is unreadable
and difficult for debug.
3. Why I need a generic implementation and changes all over whole project
because of
series of async commands, for me it is a private case?
___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Serial Execution of Async Tasks
- Original Message -
From: Ayal Baron aba...@redhat.com
To: Allon Mureinik amure...@redhat.com, Michael Kublin
mkub...@redhat.com
Cc: Liron Aravot lara...@redhat.com, engine-devel
engine-devel@ovirt.org, Eduardo Warszawski
ewars...@redhat.com
Sent: Sunday, September 23, 2012 1:10:07 PM
Subject: Re: [Engine-devel] Serial Execution of Async Tasks
- Original Message -
- Original Message -
From: Michael Kublin mkub...@redhat.com
To: Allon Mureinik amure...@redhat.com
Cc: Eduardo Warszawski ewars...@redhat.com, Liron Aravot
lara...@redhat.com, Maor Lipchuk
mlipc...@redhat.com, engine-devel engine-devel@ovirt.org
Sent: Sunday, September 23, 2012 12:41:05 PM
Subject: Re: [Engine-devel] Serial Execution of Async Tasks
Hi guys,
As you may know the engine currently has the ability
to
fire
an
SPM
task, and be asynchronously be woken-up when it
ends.
This is great, but we found the for the Live Storage
Migration
feature we need something a bit complex - the ability
to
have a
series of async tasks in a single control flow.
Here's my initial design for this, your comments and
criticism
would
be welcome:
http://wiki.ovirt.org/wiki/Features/Serial_Execution_of_Asynchronous_Tasks_Detailed_Design
-successful execution -
* CommandBase iterates over its SPMAsyncTaskHandlers -
when?
This is the new suggested format of executeCommand().
I'll
clarify
this too.
* If the second task is an HSM command (vs. SPM
command),
I
think you
should explain in the design how to handle such flows as
well.
HSM commands do not create AsyncTasks, as they do today -
I
will
clarify this.
* Why do we need before task? can you give a concrete
example
of what
would you do in such a method.
Basically, /today/, command look like this:
executeCommand() {
doStuffInTheDB();
runVdsCommand(someCommand);
}
endSuccessfully() {
doMoreStuffInTheDB();
}
endWithFailure() {
doMoreStuffForFailureInTheDB();
}
In the new design, the entire doStuffInTheDB() should be
moved
to a
breforeTask of the (only) SPMAsyncTaskHandler.
- I see you added SPMAsyncTaskHandler, any reason not to
use
SPMAsyncTasK to manage it own life-cycle?
Conserving today's design - The SPMAsyncTaskHandler is
the
place to
add additional, non-SPM, logic around the SPM task
execution,
like
CommandBase allows today.
- In the life-cycle managed by the SPMAsyncTaskHandler
there
is a
step
'createTask - how to create the async task' can you
please
elaborate
what are the options.
new [any type of async task]
(I cleaned thread a little.)
The following design and it is implementation
http://gerrit.ovirt.org/#/c/7956/
is bad.
I don't see any reason for creating a new SPMAsyncTaskHandler and
especially in the
way as it's done in patch.
The reason are following:
1. Performance , increased memory footprint, created CYCLIC
REFERENCE.
2. Readability and robust of code: the code which is written as
cyclic references is unreadable
and difficult for debug.
3. Why I need a generic implementation and changes all over whole
project because of
series of async commands, for me it is a private case?
What is the private case here exactly?
Every task can have multiple jobs. We've identified several such
places (e.g. live storage migration, move disk, move vm) and I have
no doubt more will popup.
As Allon notes below, task handling is done at CommandBase, if you
think task management should be for storage only, you're welcome to
push it down to StorageHandlingCommandBase (or get rid of
inheritance here altogether).
Interesting , regards cyclic reference no response, but who cares,
it is difficult to answer , that's why better not to response?
Regards private case:
1. We have command that not creating any task
2. We have command that will create a one task.
3. And we have 3 commands meanwhile which will create more than one task.
I think that 3 is private case and not common? (In the future, I removed too
many
lines of code that were preparation for future that never come)
The handling done at CommandBase it means that it is influence all system.
Now regards architecture why I need some handler which will be inside a command
and will call for command methods? Please explain.
This will occur all over the storage commands (which are the only
usages of tasks nowadys).
Moreover, async task handling is done at the Commandbase level (see
the end* methods) - instead of hacking it in X different places
whenever we need it, I'd prefer doing it once, properly.
___
Engine
Re: [Engine-devel] updating disk /api/disks/xxx
No, it is not my feature
- Original Message -
From: Michael Pasternak mpast...@redhat.com
To: Ayal Baron aba...@redhat.com
Cc: engine-devel engine-devel@ovirt.org, Michael Kublin
mkub...@redhat.com
Sent: Thursday, September 6, 2012 12:57:43 PM
Subject: Re: updating disk /api/disks/xxx
On 09/06/2012 12:41 PM, Ayal Baron wrote:
- Original Message -
i see in BackendDiskResource.update(), what is the status if this
item?
Is there a bug tracking this?
afaik no
I don't recall anyone is working on this.
iirc this feature was done by mkublin
@Override
public Disk update(Disk disk) {
//TODO: implmenet when Backend is ready. Implementation
stub:
// return performUpdate(this, new
QueryIdResolver(VdcQueryType.GetDiskByDiskId,
GetDiskByDiskIdParameters.class), ?, ?);
throw new NotImplementedException(Pending Backend support);
}
--
Michael Pasternak
RedHat, ENG-Virtualization RD
--
Michael Pasternak
RedHat, ENG-Virtualization RD
___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] oVirt UI: Legacy UserPortal removed
AWESOME !!! +0, -76860 - Original Message - From: Allon Mureinik amure...@redhat.com To: Laszlo Hornyak lhorn...@redhat.com Cc: engine-devel@ovirt.org Sent: Thursday, June 14, 2012 4:28:05 PM Subject: Re: [Engine-devel] oVirt UI: Legacy UserPortal removed awesome! - Original Message - From: Laszlo Hornyak lhorn...@redhat.com To: Vojtech Szocs vsz...@redhat.com, engine-devel@ovirt.org Sent: Thursday, June 14, 2012 2:29:40 PM Subject: Re: [Engine-devel] oVirt UI: Legacy UserPortal removed Hi, A visualization of how that counts (from sonar) https://twitter.com/kozka/status/213229060103483392/photo/1 Laszlo - Original Message - From: Vojtech Szocs vsz...@redhat.com To: engine-devel@ovirt.org Sent: Thursday, June 14, 2012 12:55:18 PM Subject: [Engine-devel] oVirt UI: Legacy UserPortal removed Hi everyone, please note that the original (SmartGWT-based) UserPortal application has been removed from oVirt repository, as part of patch [http://gerrit.ovirt.org/5317]. This has no impact on the current UserPortal application, which is part of the standard oVirt Maven build. Please let me know if you have any questions. Thanks, Vojtech ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
[Engine-devel] Do we need genericApi project?
Hi, I found that we have a ovirt-engine/frontend/api/genericapi project, which is for my opinion is not at use any more. If there are any reason that we want to keep it , maybe someone using it and I missed these? Thanks Michael ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Do we need genericApi project?
At least meanwhile, can I disable loading of GenericApiService bean during start up of jboss? - Original Message - From: Itamar Heim ih...@redhat.com To: Daniel Erez de...@redhat.com Cc: engine-devel engine-devel@ovirt.org, Michael Kublin mkub...@redhat.com Sent: Sunday, June 10, 2012 12:44:08 PM Subject: Re: [Engine-devel] Do we need genericApi project? On 06/10/2012 12:33 PM, Daniel Erez wrote: We may want to wait for the removal of the deprecated uicommon project (which is in use by the old UserPortal). uicommonweb doesn't use UiQueries any more (it's not depended on the genericapi project) - so I think it can be removed. I'd wait to see it really isn't needed any more. it was added to cover a gap in which the UI is using logic which exists in methods in the business entities (IsLinux/IsWindows/Is64bit, etc.) which is something we wanted to decouple, and the UI queries expose these as queries for the UI rather than use the entities directly. so i'd say this still isn't fully utilized, rather than deprecated. - Original Message - From: Itamar Heimih...@redhat.com To: Michael Kublinmkub...@redhat.com Cc: engine-develengine-devel@ovirt.org Sent: Sunday, June 10, 2012 11:48:53 AM Subject: Re: [Engine-devel] Do we need genericApi project? On 06/10/2012 11:23 AM, Michael Kublin wrote: Hi, I found that we have a ovirt-engine/frontend/api/genericapi project, which is for my opinion is not at use any more. If there are any reason that we want to keep it , maybe someone using it and I missed these? Thanks Michael ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel iirc, it's intended to allow to add queries needed for the UI without adding them into the core. and looking at GetUserActionGroups, I see it is used for example by uicommon. (GWT still uses the generic api) ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] DiskImage public mapper
Yep, it is should be id, I aware about these and Ori already asked me to change it, I will change these very soon - Original Message - From: Michael Pasternak mpast...@redhat.com To: Yair Zaslavsky yzasl...@redhat.com Cc: Michael Kublin mkub...@redhat.com, engine-devel engine-devel@ovirt.org Sent: Sunday, May 20, 2012 5:08:22 PM Subject: Re: DiskImage public mapper On 05/20/2012 04:55 PM, Michael Kublin wrote: both, id - it is a common id for all images that a disk point of them, and ImageId it is a current image if so i have doubts about cloning vm from template impl, as parameters class in BackendVmsResource:182 that receives HashMapGuid, DiskImage as collection of disks to clone has current image id as key, Yair shouldn't it be /id/ rather than /ImageId/? - Original Message - From: Michael Pasternak mpast...@redhat.com To: Michael Kublin mkub...@redhat.com, Yair Zaslavsky yzasl...@redhat.com Cc: engine-devel engine-devel@ovirt.org Sent: Sunday, May 20, 2012 4:57:13 PM Subject: DiskImage public mupper what should be exposed to user as disk.id: DiskImage.id or DiskImage.ImageId ? -- Michael Pasternak RedHat, ENG-Virtualization RD ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] VM disks
- Original Message - From: Oved Ourfalli ov...@redhat.com To: Livnat Peer lp...@redhat.com Cc: engine-devel@ovirt.org Sent: Sunday, February 19, 2012 9:48:31 AM Subject: Re: [Engine-devel] VM disks - Original Message - From: Livnat Peer lp...@redhat.com To: engine-devel@ovirt.org Sent: Saturday, February 18, 2012 7:07:01 PM Subject: [Engine-devel] VM disks Hi, These days we are working on various features around VM disks, in the different threads it was decided that we'll have the ability to attach a disk to a VM but it will be added as inactive, then the user can activate it for it to be accessible from within the guest. Flow of adding a new disk would be: - creating the disk - attaching the disk to the VM - activating it These should be in a one step, otherwise the clients (rest and gui) will need to pool us for every disk Flow of adding a shared disk (or any other existing disk): - attach the disk - activate it These is just simple as a hot plug , should be and it is easy implement as one step It seems to me a lot like adding a storage domain and I remember a lot of rejections on the storage domain flow (mostly about it being too cumbersome). After discussing the issue with various people we could not find a good reason for having a VM disk in attached but inactive mode. Of course we can wrap the above steps in one step for specific flows Agreed, should be in one step (add+attach within a VM context for example) but can anyone think on a good reason to support attached but inactive disk? I don't see a reason also. I would suggest that when attaching a disk to a VM it becomes part of the VM (active) like in 'real' machines. +1 on that (regardless of whether the disk is shared or not). IMO - in the case of shared disk we should make it as clear as possible to the user/admin that the added disk is shared, but the flow should be exactly the same. Also agreed Thank you, Livnat ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
[Engine-devel] Default TransactionScope at engine
Hi All, Today most of the flows of runAction at engine look like at the following way: 1. Perform some selects 2. Perform call to host (xml rpc) or call to some number of other actions in synchronous way 3. If a call success than perform update to DB. It is means that we actually need open transaction only at the step 3, it is mean that by default we are keeping transaction much more longer than it should be, the call to host can take a while. So, the possible plan of action is 1. Change a default scope of transaction from TransactionScopeOption.Required to TransactionScopeOption.Suppress 2. Remove annotation NonTransactiveCommandAttribute 3. The actions which are need to run in transaction - appropriate transaction scope should be passed via parameters It is a lot of dirty and not to interesting work, but I think it will provide a great benefit: 1. Reduced a time of open transaction 2. Better db connection utilization 3. I think that number of open transaction will also reduce 4. As a result performance will improve By the way, many of new actions which are written today by default marked as NonTransactiveCommandAttribute, if we will continue in such way almost every class will be marked with NonTransactiveCommandAttribute Regards Michael ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
