Re: [vdsm] [RFC] An alternative way to provide a supported interface -- libvdsm
I'll clarify. Using gobject introspection means that you bindings are non existent for dynamic languages and auto-generated (JGIR for java) for strongly typed languages. This means that you don't maintain the bindings yourself, you maintain a schema that generates gobject code and it, in turn, generates the binding. You get integration for events, properties, and objects for free. As for binary compatibility. The way to would work is that all objects internally will be similar to python. All local object will hav an internal map and the getters and setters will modify these in a type safe manner. This means that you can expend the API and still keep BC. As for REST and AMQP. REST is pure request response, has high latency, and is pure CRUD which is very limiting. AMQP (or any brokered messaging) needs a lot of setting up and makes no sense in a supported API because it implies VDSM a cognizant entity in the cluster which makes it less of an API slave and has more BL that might change and be engine specific. The reason we are choosing to take the libvdsm approach with bridges is because remote binding usually let themselves to some underlying design philosophy. This means ALL bindings suffer. I've uploaded a small POC[1] to show how the gobject binding might work. The way it works is that you always have access to the send_command verb and the BaseObject This means that if you want to use something that you don't currently support you can get\set\query the objects you received. But, most objects will have types and their own type safe accessors to make everything easier. You could see that the python bindings "just work" because we use GObject no additional code has to be written to load libvdsm. http://gerrit.ovirt.org/6659 - Original Message - > From: "Itamar Heim" > To: "Adam Litke" > Cc: "Saggi Mizrahi" , vdsm-devel@lists.fedorahosted.org > Sent: Thursday, July 26, 2012 9:39:16 AM > Subject: Re: [vdsm] [RFC] An alternative way to provide a supported interface > -- libvdsm > > On 07/26/2012 04:33 PM, Adam Litke wrote: > > On Thu, Jul 26, 2012 at 11:47:51AM +0300, Itamar Heim wrote: > >> On 07/17/2012 01:19 AM, Itamar Heim wrote: > >>> On 07/09/2012 09:52 PM, Saggi Mizrahi wrote: > > > - Original Message - > > From: "Itamar Heim" > > To: "Saggi Mizrahi" > > Cc: "Adam Litke" , > > vdsm-devel@lists.fedorahosted.org > > Sent: Monday, July 9, 2012 11:03:43 AM > > Subject: Re: [vdsm] [RFC] An alternative way to provide a > > supported > > interface -- libvdsm > > > > On 07/09/2012 05:56 PM, Saggi Mizrahi wrote: > >> I don't think AMQP is a good low level supported protocol as > >> it's a > >> very complex protocol to set up and support. > >> Also brokers are known to have their differences in standard > >> implementation which means supporting them all is a mess. > >> > >> It looks like the most accepted route is the libvirt route of > >> having a c library abstracting away client server > >> communication > >> and having more advanced consumers build protocol specific > >> bridges > >> that may have different support standards. > >> > >> On a more personal note, I think brokerless messaging is the > >> way to > >> go in ovirt because, unlike traditional clustering, worker > >> nodes > >> are not interchangeable so direct communication is the way to > >> go, > >> rendering brokers pretty much useless. > > > > but brokerless doesn't let multiple consumers which a bus > > provides? > All consumers can connect to the host and *some* events can be > broadcasted to all connected clients. > > The real question is weather you want to depend on AMQP's > routing \ > message storing > Also, if you find it preferable to have a centralized host > (single > point of failure) to get all events from all hosts for the price > of > some clients (I assume read only clients) not needing to know > the > locations of all worker nodes. > But IMHO we already have something like that, it's called the > ovirt-engine, and it could send aggregated events about the > cluster > (maybe with some extra enginy data). > > The question is what does mandating a broker gives us something > that > an "AMQP bridge" wouldn't. > The only thing I can think of is vdsm can assume unmoderated > vdsm to > vdsm communication bypassing the engine. > This means that VDSM can have some clustered behavior that > requires no > engine intervention. > Further more, the engine can send a request and let the nodes > decide > who is performing the operation among themselves. > > Essentially: > > [ engine ] [ engine ] > | | VS | > [vdsm][vdsm] [ broker
Re: [vdsm] [RFC] An alternative way to provide a supported interface -- libvdsm
On 07/26/2012 04:33 PM, Adam Litke wrote: On Thu, Jul 26, 2012 at 11:47:51AM +0300, Itamar Heim wrote: On 07/17/2012 01:19 AM, Itamar Heim wrote: On 07/09/2012 09:52 PM, Saggi Mizrahi wrote: - Original Message - From: "Itamar Heim" To: "Saggi Mizrahi" Cc: "Adam Litke" , vdsm-devel@lists.fedorahosted.org Sent: Monday, July 9, 2012 11:03:43 AM Subject: Re: [vdsm] [RFC] An alternative way to provide a supported interface -- libvdsm On 07/09/2012 05:56 PM, Saggi Mizrahi wrote: I don't think AMQP is a good low level supported protocol as it's a very complex protocol to set up and support. Also brokers are known to have their differences in standard implementation which means supporting them all is a mess. It looks like the most accepted route is the libvirt route of having a c library abstracting away client server communication and having more advanced consumers build protocol specific bridges that may have different support standards. On a more personal note, I think brokerless messaging is the way to go in ovirt because, unlike traditional clustering, worker nodes are not interchangeable so direct communication is the way to go, rendering brokers pretty much useless. but brokerless doesn't let multiple consumers which a bus provides? All consumers can connect to the host and *some* events can be broadcasted to all connected clients. The real question is weather you want to depend on AMQP's routing \ message storing Also, if you find it preferable to have a centralized host (single point of failure) to get all events from all hosts for the price of some clients (I assume read only clients) not needing to know the locations of all worker nodes. But IMHO we already have something like that, it's called the ovirt-engine, and it could send aggregated events about the cluster (maybe with some extra enginy data). The question is what does mandating a broker gives us something that an "AMQP bridge" wouldn't. The only thing I can think of is vdsm can assume unmoderated vdsm to vdsm communication bypassing the engine. This means that VDSM can have some clustered behavior that requires no engine intervention. Further more, the engine can send a request and let the nodes decide who is performing the operation among themselves. Essentially: [ engine ] [ engine ] | | VS | [vdsm][vdsm] [ broker ] | | [vdsm][vdsm] *All links are two way links This has dire consequences on API usability and supportability. So we need to converge on that. There needs to be a good reason why the aforementioned logic code can't sit on a another ovirt specific entity (lets call it ovirt-dynamo) that uses VDSM's supported API but it's own APIs (or more likely messaging algorithms) are unsupported. [engine ] ||| | [ broker ] | ||| | [vdsm]-[dynamo] : [dynamo]-[vdsm] Host A : Host B *All links are two way links 1. we have engine today 'in the path' to the history db. but it makes no sense for engine to be aware of each statistic we want to keep in the history db. same would be for an event/stats correlation service. they don't need to depend on each other for availability/redundancy. 2. we are already looking at quantum integration, which is doing engine to nodes communication via amqp. 3. with somewhat of a forward looking - moving some scheduling logic "down to vdsm" will probably mean we'll want one of the nodes to listen to statistics and state from the other nodes. to all of these, setting up a bus which allows multiple peer listeners seems more robust I'm still against developing a C level binding for amqp and rest support over a codebase which is in python. rest and amqp allow for both local and remote bindings in any language. C bindings should/could be a parallel implementation, but they seem like an unneeded overhead and complexity in the middle of the codebase. Sure, it's probably possible to bind a REST or AMQP API in other languages but I don't think there is an automatic way of doing it. That means having to keep up with maintenance of each and every binding every time the API changes. If we look at libvirt, they will say this is a large source of pain that they have recommended we avoid. we'd need this on top of the C api as well - but it would probably be simpler doing it over the python api, rather than the C one. For the C/gobject approach, we write a single API schema file. From that, we automatically generate the C API and bindings. Sure, the generation could be a bit complex but much of it will be someone else's codebase (and one that is used by lots of Gnome projects). that's a critical part of the question for the C api - what is needed for adding a verb / parameter. I'm not against having c bindings. what i don't understand is why to put them in the 'middle',
Re: [vdsm] [RFC] An alternative way to provide a supported interface -- libvdsm
On Thu, Jul 26, 2012 at 11:47:51AM +0300, Itamar Heim wrote: > On 07/17/2012 01:19 AM, Itamar Heim wrote: > >On 07/09/2012 09:52 PM, Saggi Mizrahi wrote: > >> > >> > >>- Original Message - > >>>From: "Itamar Heim" > >>>To: "Saggi Mizrahi" > >>>Cc: "Adam Litke" , vdsm-devel@lists.fedorahosted.org > >>>Sent: Monday, July 9, 2012 11:03:43 AM > >>>Subject: Re: [vdsm] [RFC] An alternative way to provide a supported > >>>interface -- libvdsm > >>> > >>>On 07/09/2012 05:56 PM, Saggi Mizrahi wrote: > I don't think AMQP is a good low level supported protocol as it's a > very complex protocol to set up and support. > Also brokers are known to have their differences in standard > implementation which means supporting them all is a mess. > > It looks like the most accepted route is the libvirt route of > having a c library abstracting away client server communication > and having more advanced consumers build protocol specific bridges > that may have different support standards. > > On a more personal note, I think brokerless messaging is the way to > go in ovirt because, unlike traditional clustering, worker nodes > are not interchangeable so direct communication is the way to go, > rendering brokers pretty much useless. > >>> > >>>but brokerless doesn't let multiple consumers which a bus provides? > >>All consumers can connect to the host and *some* events can be > >>broadcasted to all connected clients. > >> > >>The real question is weather you want to depend on AMQP's routing \ > >>message storing > >>Also, if you find it preferable to have a centralized host (single > >>point of failure) to get all events from all hosts for the price of > >>some clients (I assume read only clients) not needing to know the > >>locations of all worker nodes. > >>But IMHO we already have something like that, it's called the > >>ovirt-engine, and it could send aggregated events about the cluster > >>(maybe with some extra enginy data). > >> > >>The question is what does mandating a broker gives us something that > >>an "AMQP bridge" wouldn't. > >>The only thing I can think of is vdsm can assume unmoderated vdsm to > >>vdsm communication bypassing the engine. > >>This means that VDSM can have some clustered behavior that requires no > >>engine intervention. > >>Further more, the engine can send a request and let the nodes decide > >>who is performing the operation among themselves. > >> > >>Essentially: > >> > >>[ engine ] [ engine ] > >>| | VS | > >>[vdsm][vdsm] [ broker ] > >> | | > >> [vdsm][vdsm] > >> > >>*All links are two way links > >> > >>This has dire consequences on API usability and supportability. So we > >>need to converge on that. > >> > >>There needs to be a good reason why the aforementioned logic code > >>can't sit on a another ovirt specific entity (lets call it > >>ovirt-dynamo) that uses VDSM's supported API but it's own APIs (or > >>more likely messaging algorithms) are unsupported. > >> > >> [engine ] > >>||| > >>| [ broker ] | > >>||| | > >>[vdsm]-[dynamo] : [dynamo]-[vdsm] > >> Host A : Host B > >> > >>*All links are two way links > > > >1. we have engine today 'in the path' to the history db. but it makes no > >sense for engine to be aware of each statistic we want to keep in the > >history db. > >same would be for an event/stats correlation service. > >they don't need to depend on each other for availability/redundancy. > > > >2. we are already looking at quantum integration, which is doing engine > >to nodes communication via amqp. > > > >3. with somewhat of a forward looking - moving some scheduling logic > >"down to vdsm" will probably mean we'll want one of the nodes to listen > >to statistics and state from the other nodes. > > > >to all of these, setting up a bus which allows multiple peer listeners > >seems more robust > > > I'm still against developing a C level binding for amqp and rest > support over a codebase which is in python. > rest and amqp allow for both local and remote bindings in any language. > C bindings should/could be a parallel implementation, but they seem > like an unneeded overhead and complexity in the middle of the > codebase. Sure, it's probably possible to bind a REST or AMQP API in other languages but I don't think there is an automatic way of doing it. That means having to keep up with maintenance of each and every binding every time the API changes. If we look at libvirt, they will say this is a large source of pain that they have recommended we avoid. For the C/gobject approach, we write a single API schema file. From that, we automatically generate the C API and bindings. Sure, the generation could be a bit complex but much of it will be someone else's codebase (and one that is used by lots of Gno
Re: [vdsm] [RFC] An alternative way to provide a supported interface -- libvdsm
On 07/17/2012 01:19 AM, Itamar Heim wrote: On 07/09/2012 09:52 PM, Saggi Mizrahi wrote: - Original Message - From: "Itamar Heim" To: "Saggi Mizrahi" Cc: "Adam Litke" , vdsm-devel@lists.fedorahosted.org Sent: Monday, July 9, 2012 11:03:43 AM Subject: Re: [vdsm] [RFC] An alternative way to provide a supported interface -- libvdsm On 07/09/2012 05:56 PM, Saggi Mizrahi wrote: I don't think AMQP is a good low level supported protocol as it's a very complex protocol to set up and support. Also brokers are known to have their differences in standard implementation which means supporting them all is a mess. It looks like the most accepted route is the libvirt route of having a c library abstracting away client server communication and having more advanced consumers build protocol specific bridges that may have different support standards. On a more personal note, I think brokerless messaging is the way to go in ovirt because, unlike traditional clustering, worker nodes are not interchangeable so direct communication is the way to go, rendering brokers pretty much useless. but brokerless doesn't let multiple consumers which a bus provides? All consumers can connect to the host and *some* events can be broadcasted to all connected clients. The real question is weather you want to depend on AMQP's routing \ message storing Also, if you find it preferable to have a centralized host (single point of failure) to get all events from all hosts for the price of some clients (I assume read only clients) not needing to know the locations of all worker nodes. But IMHO we already have something like that, it's called the ovirt-engine, and it could send aggregated events about the cluster (maybe with some extra enginy data). The question is what does mandating a broker gives us something that an "AMQP bridge" wouldn't. The only thing I can think of is vdsm can assume unmoderated vdsm to vdsm communication bypassing the engine. This means that VDSM can have some clustered behavior that requires no engine intervention. Further more, the engine can send a request and let the nodes decide who is performing the operation among themselves. Essentially: [ engine ] [ engine ] | | VS | [vdsm][vdsm] [ broker ] | | [vdsm][vdsm] *All links are two way links This has dire consequences on API usability and supportability. So we need to converge on that. There needs to be a good reason why the aforementioned logic code can't sit on a another ovirt specific entity (lets call it ovirt-dynamo) that uses VDSM's supported API but it's own APIs (or more likely messaging algorithms) are unsupported. [engine ] ||| | [ broker ] | ||| | [vdsm]-[dynamo] : [dynamo]-[vdsm] Host A : Host B *All links are two way links 1. we have engine today 'in the path' to the history db. but it makes no sense for engine to be aware of each statistic we want to keep in the history db. same would be for an event/stats correlation service. they don't need to depend on each other for availability/redundancy. 2. we are already looking at quantum integration, which is doing engine to nodes communication via amqp. 3. with somewhat of a forward looking - moving some scheduling logic "down to vdsm" will probably mean we'll want one of the nodes to listen to statistics and state from the other nodes. to all of these, setting up a bus which allows multiple peer listeners seems more robust I'm still against developing a C level binding for amqp and rest support over a codebase which is in python. rest and amqp allow for both local and remote bindings in any language. C bindings should/could be a parallel implementation, but they seem like an unneeded overhead and complexity in the middle of the codebase. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
