Re: [Engine-devel] Ideas about engine clustering?
I've added a wiki page on Engine HA. http://www.ovirt.org/Features/Engine_High_Availability Please add your comments, we'll try to make this an ongoing effort. On Sat, Aug 24, 2013 at 3:25 PM, Liran Zelkha liran.zel...@gmail.comwrote: IMHO EJB clustering is not the direction we should take, especially considering our IoC plans (see http://gerrit.ovirt.org/#/c/5575/). QuartzScheduler can be easily fixed by using a DB for Quartz configuration. We had such a discussion a few weeks ago. mod_jk is probably better for our HA. Postgres clustering should probably be a different task, as it's more infra and less code. Another thing to consider is locking, currently done using synchronized, and will need to be changed somehow (probably using Infinispan). On Aug 22, 2013, at 10:22 AM, Itamar Heim wrote: On 08/21/2013 10:27 PM, plysan wrote: 2013/8/22 Itamar Heim ih...@redhat.com mailto:ih...@redhat.comih...@redhat.com On 08/21/2013 11:08 AM, plysan wrote: 2013/8/21 Laszlo Hornyak lhorn...@redhat.com mailto:lhorn...@redhat.com lhorn...@redhat.com mailto:lhorn...@redhat.com lhorn...@redhat.com mailto:lhorn...@redhat.com lhorn...@redhat.com Hi, - Original Message - From: plysan ply...@gmail.com mailto:ply...@gmail.com ply...@gmail.com mailto:ply...@gmail.com ply...@gmail.com mailto:ply...@gmail.com ply...@gmail.com To: engine-devel@ovirt.org mailto:engine-devel@ovirt.org engine-devel@ovirt.org mailto:engine-devel@ovirt.org engine-devel@ovirt.org mailto:engine-devel@ovirt.org engine-devel@ovirt.org__ Sent: Wednesday, August 21, 2013 3:49:31 PM Subject: [Engine-devel] Ideas about engine clustering? Hello, guys: I'm trying to make a jboss clustering environment for engine. But felt the difficuty too. So I want to know if any one is working on this? I googled it and haven't found anything. I know there will be lots of things to do for engine migrating to clustering environment. I just want to know what problem need to be solved? I would like to list the things I think that is needed to do(or solve): 1. run jboss in standalone-ha mode or domain mode. 2. add @Clustered annotation to each EJB, I think this will solve the replication problem in cluster, probably lots of details in it. 3. QuartzScheduler problem, only one node running a scheduler is enough at a time instead of each node running one. 4. use mod_cluster to handle load balancing. 5. postgresql clustering. For failover it sounds like a plan, just wondering if this could solve load balancing problems as well. Ovirt is generating a big load on the DB and I would be really interested if SQL DB clustering could solve the issues. It would be great if it could, big part of the scalablity issue is the evil things we did against that poor database. 6. There should be more, but hope anyone can think about it :D There are tons of data structures that are used in oVirt that store state. You will have to hunt all of these down and replace with a data structure that is shared between the cluster memebers. This may be a big lot of work from many parts of the application. Indeed, and like Yair Zaslavsky said, we can put the data structure in infinispan cache, but clustered EJB can do the same thing for us, right? If all those state values are inside EJBs. (iirc)** Any ideas? Thanks! _ Engine-devel mailing list Engine-devel@ovirt.org mailto:Engine-devel@ovirt.orgEngine-devel@ovirt.org mailto:Engine-devel@ovirt.org Engine-devel@ovirt.org mailto:Engine-devel@ovirt.org Engine-devel@ovirt.org__ http://lists.ovirt.org/__mailman/listinfo/engine-devel http://lists.ovirt.org/mailman/listinfo/engine-devel _ Engine-devel mailing list Engine-devel@ovirt.org mailto:Engine-devel@ovirt.orgEngine-devel@ovirt.org http://lists.ovirt.org/__mailman/listinfo/engine-devel http
Re: [Engine-devel] Ideas about engine clustering?
IMHO EJB clustering is not the direction we should take, especially considering our IoC plans (see http://gerrit.ovirt.org/#/c/5575/). QuartzScheduler can be easily fixed by using a DB for Quartz configuration. We had such a discussion a few weeks ago. mod_jk is probably better for our HA. Postgres clustering should probably be a different task, as it's more infra and less code. Another thing to consider is locking, currently done using synchronized, and will need to be changed somehow (probably using Infinispan). On Aug 22, 2013, at 10:22 AM, Itamar Heim wrote: On 08/21/2013 10:27 PM, plysan wrote: 2013/8/22 Itamar Heim ih...@redhat.com mailto:ih...@redhat.com On 08/21/2013 11:08 AM, plysan wrote: 2013/8/21 Laszlo Hornyak lhorn...@redhat.com mailto:lhorn...@redhat.com mailto:lhorn...@redhat.com mailto:lhorn...@redhat.com Hi, - Original Message - From: plysan ply...@gmail.com mailto:ply...@gmail.com mailto:ply...@gmail.com mailto:ply...@gmail.com To: engine-devel@ovirt.org mailto:engine-devel@ovirt.org mailto:engine-devel@ovirt.org mailto:engine-devel@ovirt.org__ Sent: Wednesday, August 21, 2013 3:49:31 PM Subject: [Engine-devel] Ideas about engine clustering? Hello, guys: I'm trying to make a jboss clustering environment for engine. But felt the difficuty too. So I want to know if any one is working on this? I googled it and haven't found anything. I know there will be lots of things to do for engine migrating to clustering environment. I just want to know what problem need to be solved? I would like to list the things I think that is needed to do(or solve): 1. run jboss in standalone-ha mode or domain mode. 2. add @Clustered annotation to each EJB, I think this will solve the replication problem in cluster, probably lots of details in it. 3. QuartzScheduler problem, only one node running a scheduler is enough at a time instead of each node running one. 4. use mod_cluster to handle load balancing. 5. postgresql clustering. For failover it sounds like a plan, just wondering if this could solve load balancing problems as well. Ovirt is generating a big load on the DB and I would be really interested if SQL DB clustering could solve the issues. It would be great if it could, big part of the scalablity issue is the evil things we did against that poor database. 6. There should be more, but hope anyone can think about it :D There are tons of data structures that are used in oVirt that store state. You will have to hunt all of these down and replace with a data structure that is shared between the cluster memebers. This may be a big lot of work from many parts of the application. Indeed, and like Yair Zaslavsky said, we can put the data structure in infinispan cache, but clustered EJB can do the same thing for us, right? If all those state values are inside EJBs. (iirc)** Any ideas? Thanks! _ Engine-devel mailing list Engine-devel@ovirt.org mailto:Engine-devel@ovirt.org mailto:Engine-devel@ovirt.org mailto:Engine-devel@ovirt.org__ http://lists.ovirt.org/__mailman/listinfo/engine-devel http://lists.ovirt.org/mailman/listinfo/engine-devel _ Engine-devel mailing list Engine-devel@ovirt.org mailto:Engine-devel@ovirt.org http://lists.ovirt.org/__mailman/listinfo/engine-devel http://lists.ovirt.org/mailman/listinfo/engine-devel unless you need the load balancing, just for HA, hosted-engine should cover your needs? I think failover and more scalability is what i am interested here. Just thinking about the possibilities :) Imagine engine would never get restarted from outside view. Cool isn't it? active/passive failover you should get from hosted-engine (yes, with some downtime) scalability and active-active would be great. iirc, juan looked at implications of doing this a while back ___ Engine-devel mailing list Engine
Re: [Engine-devel] Ideas about engine clustering?
On 08/21/2013 10:27 PM, plysan wrote: 2013/8/22 Itamar Heim ih...@redhat.com mailto:ih...@redhat.com On 08/21/2013 11:08 AM, plysan wrote: 2013/8/21 Laszlo Hornyak lhorn...@redhat.com mailto:lhorn...@redhat.com mailto:lhorn...@redhat.com mailto:lhorn...@redhat.com Hi, - Original Message - From: plysan ply...@gmail.com mailto:ply...@gmail.com mailto:ply...@gmail.com mailto:ply...@gmail.com To: engine-devel@ovirt.org mailto:engine-devel@ovirt.org mailto:engine-devel@ovirt.org mailto:engine-devel@ovirt.org__ Sent: Wednesday, August 21, 2013 3:49:31 PM Subject: [Engine-devel] Ideas about engine clustering? Hello, guys: I'm trying to make a jboss clustering environment for engine. But felt the difficuty too. So I want to know if any one is working on this? I googled it and haven't found anything. I know there will be lots of things to do for engine migrating to clustering environment. I just want to know what problem need to be solved? I would like to list the things I think that is needed to do(or solve): 1. run jboss in standalone-ha mode or domain mode. 2. add @Clustered annotation to each EJB, I think this will solve the replication problem in cluster, probably lots of details in it. 3. QuartzScheduler problem, only one node running a scheduler is enough at a time instead of each node running one. 4. use mod_cluster to handle load balancing. 5. postgresql clustering. For failover it sounds like a plan, just wondering if this could solve load balancing problems as well. Ovirt is generating a big load on the DB and I would be really interested if SQL DB clustering could solve the issues. It would be great if it could, big part of the scalablity issue is the evil things we did against that poor database. 6. There should be more, but hope anyone can think about it :D There are tons of data structures that are used in oVirt that store state. You will have to hunt all of these down and replace with a data structure that is shared between the cluster memebers. This may be a big lot of work from many parts of the application. Indeed, and like Yair Zaslavsky said, we can put the data structure in infinispan cache, but clustered EJB can do the same thing for us, right? If all those state values are inside EJBs. (iirc)** Any ideas? Thanks! _ Engine-devel mailing list Engine-devel@ovirt.org mailto:Engine-devel@ovirt.org mailto:Engine-devel@ovirt.org mailto:Engine-devel@ovirt.org__ http://lists.ovirt.org/__mailman/listinfo/engine-devel http://lists.ovirt.org/mailman/listinfo/engine-devel _ Engine-devel mailing list Engine-devel@ovirt.org mailto:Engine-devel@ovirt.org http://lists.ovirt.org/__mailman/listinfo/engine-devel http://lists.ovirt.org/mailman/listinfo/engine-devel unless you need the load balancing, just for HA, hosted-engine should cover your needs? I think failover and more scalability is what i am interested here. Just thinking about the possibilities :) Imagine engine would never get restarted from outside view. Cool isn't it? active/passive failover you should get from hosted-engine (yes, with some downtime) scalability and active-active would be great. iirc, juan looked at implications of doing this a while back ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
[Engine-devel] Ideas about engine clustering?
Hello, guys: I'm trying to make a jboss clustering environment for engine. But felt the difficuty too. So I want to know if any one is working on this? I googled it and haven't found anything. I know there will be lots of things to do for engine migrating to clustering environment. I just want to know what problem need to be solved? I would like to list the things I think that is needed to do(or solve): 1. run jboss in standalone-ha mode or domain mode. 2. add @Clustered annotation to each EJB, I think this will solve the replication problem in cluster, probably lots of details in it. 3. QuartzScheduler problem, only one node running a scheduler is enough at a time instead of each node running one. 4. use mod_cluster to handle load balancing. 5. postgresql clustering. 6. There should be more, but hope anyone can think about it :D Any ideas? Thanks! ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Ideas about engine clustering?
Hi, From: plysan ply...@gmail.com To: engine-devel@ovirt.org Sent: Wednesday, August 21, 2013 4:49:31 PM Subject: [Engine-devel] Ideas about engine clustering? Hello, guys: I'm trying to make a jboss clustering environment for engine. But felt the difficuty too. So I want to know if any one is working on this? I googled it and haven't found anything. I know there will be lots of things to do for engine migrating to clustering environment. I just want to know what problem need to be solved? I would like to list the things I think that is needed to do(or solve): 1. run jboss in standalone-ha mode or domain mode. 2. add @Clustered annotation to each EJB, I think this will solve the replication problem in cluster, probably lots of details in it. 3. QuartzScheduler problem, only one node running a scheduler is enough at a time instead of each node running one. 4. use mod_cluster to handle load balancing. 5. postgresql clustering. 6. There should be more, but hope anyone can think about it :D Any ideas? Depending on your needs/interests, you might want to have a look at ovirt-hosted-engine-setup - allows creating a VM on a host and install an engine on that VM which will also manage the host ovirt-hosted-engine-ha - allows configuring two such hosts for high availability of the engine VM (in the VM level, not application level which is what you discussed above). Both are work-in-progress and are partially usable. Not ready for production, though. Regards, -- Didi ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Ideas about engine clustering?
Hi, - Original Message - From: plysan ply...@gmail.com To: engine-devel@ovirt.org Sent: Wednesday, August 21, 2013 3:49:31 PM Subject: [Engine-devel] Ideas about engine clustering? Hello, guys: I'm trying to make a jboss clustering environment for engine. But felt the difficuty too. So I want to know if any one is working on this? I googled it and haven't found anything. I know there will be lots of things to do for engine migrating to clustering environment. I just want to know what problem need to be solved? I would like to list the things I think that is needed to do(or solve): 1. run jboss in standalone-ha mode or domain mode. 2. add @Clustered annotation to each EJB, I think this will solve the replication problem in cluster, probably lots of details in it. 3. QuartzScheduler problem, only one node running a scheduler is enough at a time instead of each node running one. 4. use mod_cluster to handle load balancing. 5. postgresql clustering. For failover it sounds like a plan, just wondering if this could solve load balancing problems as well. Ovirt is generating a big load on the DB and I would be really interested if SQL DB clustering could solve the issues. It would be great if it could, big part of the scalablity issue is the evil things we did against that poor database. 6. There should be more, but hope anyone can think about it :D There are tons of data structures that are used in oVirt that store state. You will have to hunt all of these down and replace with a data structure that is shared between the cluster memebers. This may be a big lot of work from many parts of the application. Any ideas? Thanks! ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Ideas about engine clustering?
2013/8/21 Laszlo Hornyak lhorn...@redhat.com Hi, - Original Message - From: plysan ply...@gmail.com To: engine-devel@ovirt.org Sent: Wednesday, August 21, 2013 3:49:31 PM Subject: [Engine-devel] Ideas about engine clustering? Hello, guys: I'm trying to make a jboss clustering environment for engine. But felt the difficuty too. So I want to know if any one is working on this? I googled it and haven't found anything. I know there will be lots of things to do for engine migrating to clustering environment. I just want to know what problem need to be solved? I would like to list the things I think that is needed to do(or solve): 1. run jboss in standalone-ha mode or domain mode. 2. add @Clustered annotation to each EJB, I think this will solve the replication problem in cluster, probably lots of details in it. 3. QuartzScheduler problem, only one node running a scheduler is enough at a time instead of each node running one. 4. use mod_cluster to handle load balancing. 5. postgresql clustering. For failover it sounds like a plan, just wondering if this could solve load balancing problems as well. Ovirt is generating a big load on the DB and I would be really interested if SQL DB clustering could solve the issues. It would be great if it could, big part of the scalablity issue is the evil things we did against that poor database. 6. There should be more, but hope anyone can think about it :D There are tons of data structures that are used in oVirt that store state. You will have to hunt all of these down and replace with a data structure that is shared between the cluster memebers. This may be a big lot of work from many parts of the application. Indeed, and like Yair Zaslavsky said, we can put the data structure in infinispan cache, but clustered EJB can do the same thing for us, right? If all those state values are inside EJBs. (iirc)* * Any ideas? Thanks! ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Ideas about engine clustering?
On 08/21/2013 11:08 AM, plysan wrote: 2013/8/21 Laszlo Hornyak lhorn...@redhat.com mailto:lhorn...@redhat.com Hi, - Original Message - From: plysan ply...@gmail.com mailto:ply...@gmail.com To: engine-devel@ovirt.org mailto:engine-devel@ovirt.org Sent: Wednesday, August 21, 2013 3:49:31 PM Subject: [Engine-devel] Ideas about engine clustering? Hello, guys: I'm trying to make a jboss clustering environment for engine. But felt the difficuty too. So I want to know if any one is working on this? I googled it and haven't found anything. I know there will be lots of things to do for engine migrating to clustering environment. I just want to know what problem need to be solved? I would like to list the things I think that is needed to do(or solve): 1. run jboss in standalone-ha mode or domain mode. 2. add @Clustered annotation to each EJB, I think this will solve the replication problem in cluster, probably lots of details in it. 3. QuartzScheduler problem, only one node running a scheduler is enough at a time instead of each node running one. 4. use mod_cluster to handle load balancing. 5. postgresql clustering. For failover it sounds like a plan, just wondering if this could solve load balancing problems as well. Ovirt is generating a big load on the DB and I would be really interested if SQL DB clustering could solve the issues. It would be great if it could, big part of the scalablity issue is the evil things we did against that poor database. 6. There should be more, but hope anyone can think about it :D There are tons of data structures that are used in oVirt that store state. You will have to hunt all of these down and replace with a data structure that is shared between the cluster memebers. This may be a big lot of work from many parts of the application. Indeed, and like Yair Zaslavsky said, we can put the data structure in infinispan cache, but clustered EJB can do the same thing for us, right? If all those state values are inside EJBs. (iirc)** Any ideas? Thanks! ___ Engine-devel mailing list Engine-devel@ovirt.org mailto:Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel unless you need the load balancing, just for HA, hosted-engine should cover your needs? ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Ideas about engine clustering?
2013/8/22 Itamar Heim ih...@redhat.com On 08/21/2013 11:08 AM, plysan wrote: 2013/8/21 Laszlo Hornyak lhorn...@redhat.com mailto:lhorn...@redhat.com Hi, - Original Message - From: plysan ply...@gmail.com mailto:ply...@gmail.com To: engine-devel@ovirt.org mailto:engine-devel@ovirt.org** Sent: Wednesday, August 21, 2013 3:49:31 PM Subject: [Engine-devel] Ideas about engine clustering? Hello, guys: I'm trying to make a jboss clustering environment for engine. But felt the difficuty too. So I want to know if any one is working on this? I googled it and haven't found anything. I know there will be lots of things to do for engine migrating to clustering environment. I just want to know what problem need to be solved? I would like to list the things I think that is needed to do(or solve): 1. run jboss in standalone-ha mode or domain mode. 2. add @Clustered annotation to each EJB, I think this will solve the replication problem in cluster, probably lots of details in it. 3. QuartzScheduler problem, only one node running a scheduler is enough at a time instead of each node running one. 4. use mod_cluster to handle load balancing. 5. postgresql clustering. For failover it sounds like a plan, just wondering if this could solve load balancing problems as well. Ovirt is generating a big load on the DB and I would be really interested if SQL DB clustering could solve the issues. It would be great if it could, big part of the scalablity issue is the evil things we did against that poor database. 6. There should be more, but hope anyone can think about it :D There are tons of data structures that are used in oVirt that store state. You will have to hunt all of these down and replace with a data structure that is shared between the cluster memebers. This may be a big lot of work from many parts of the application. Indeed, and like Yair Zaslavsky said, we can put the data structure in infinispan cache, but clustered EJB can do the same thing for us, right? If all those state values are inside EJBs. (iirc)** Any ideas? Thanks! __**_ Engine-devel mailing list Engine-devel@ovirt.org mailto:Engine-devel@ovirt.org** http://lists.ovirt.org/**mailman/listinfo/engine-develhttp://lists.ovirt.org/mailman/listinfo/engine-devel __**_ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/**mailman/listinfo/engine-develhttp://lists.ovirt.org/mailman/listinfo/engine-devel unless you need the load balancing, just for HA, hosted-engine should cover your needs? I think failover and more scalability is what i am interested here. Just thinking about the possibilities :) Imagine engine would never get restarted from outside view. Cool isn't it? ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
Re: [Engine-devel] Ideas about engine clustering?
- Original Message - From: Laszlo Hornyak lhorn...@redhat.com To: plysan ply...@gmail.com Cc: engine-devel@ovirt.org Sent: Wednesday, August 21, 2013 6:30:36 PM Subject: Re: [Engine-devel] Ideas about engine clustering? - Original Message - From: plysan ply...@gmail.com To: Laszlo Hornyak lhorn...@redhat.com Cc: engine-devel@ovirt.org Sent: Wednesday, August 21, 2013 5:08:57 PM Subject: Re: [Engine-devel] Ideas about engine clustering? 2013/8/21 Laszlo Hornyak lhorn...@redhat.com Hi, - Original Message - From: plysan ply...@gmail.com To: engine-devel@ovirt.org Sent: Wednesday, August 21, 2013 3:49:31 PM Subject: [Engine-devel] Ideas about engine clustering? Hello, guys: I'm trying to make a jboss clustering environment for engine. But felt the difficuty too. So I want to know if any one is working on this? I googled it and haven't found anything. I know there will be lots of things to do for engine migrating to clustering environment. I just want to know what problem need to be solved? I would like to list the things I think that is needed to do(or solve): 1. run jboss in standalone-ha mode or domain mode. 2. add @Clustered annotation to each EJB, I think this will solve the replication problem in cluster, probably lots of details in it. 3. QuartzScheduler problem, only one node running a scheduler is enough at a time instead of each node running one. 4. use mod_cluster to handle load balancing. 5. postgresql clustering. For failover it sounds like a plan, just wondering if this could solve load balancing problems as well. Ovirt is generating a big load on the DB and I would be really interested if SQL DB clustering could solve the issues. It would be great if it could, big part of the scalablity issue is the evil things we did against that poor database. 6. There should be more, but hope anyone can think about it :D There are tons of data structures that are used in oVirt that store state. You will have to hunt all of these down and replace with a data structure that is shared between the cluster memebers. This may be a big lot of work from many parts of the application. Indeed, and like Yair Zaslavsky said, we can put the data structure in infinispan cache, but clustered EJB can do the same thing for us, right? If all those state values are inside EJBs. (iirc)* * There are only a very small number of real EJB's in oVirt, almost all and everything is done by singletons created by EJBs. (check InitBackendServicesOnStartupBean, Backend they are the biggest factories of these singletons) If you made a deeper research in this recently then your information may be more up to date than mine, but while working in various parts of the app I have found tons of local state. That's right. Not to mention there are some thoughts (at least by some of us) whether we really need EJBs , but that's a totally different discussion ;) Any ideas? Thanks! ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel ___ Engine-devel mailing list Engine-devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel
