Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
On Tue, Nov 27, 2012 at 03:24:58PM -0500, Alon Bar-Lev wrote: Management interface configuration is a separate issue. But it is an important issue that has to be discussed.. If we perform changes of this interface when host is in maintenance we reduce the complexity of the problem. For your specific issue, if there are two interfaces, one which is up during boot and one which is down during boot, there is no problem to bond them after boot without persisting configuration. how would you know which bond mode to use? which MTU? I don't understand the question. I think I do: Alon suggests that on boot, the management interface would not have bonding at all, and use a single nic. The switch would have to assume that other nics in the bond are dead, and will use the only one which is alive to transfer packets. There is no doubt that we have to persist the vlan tag of the management interface, and its MTU, in the extremely rare case where the network would not alow Linux's default of 1500. Dan. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
On 11/28/2012 03:53 AM, Dan Kenigsberg wrote: On Tue, Nov 27, 2012 at 03:24:58PM -0500, Alon Bar-Lev wrote: Management interface configuration is a separate issue. But it is an important issue that has to be discussed.. If we perform changes of this interface when host is in maintenance we reduce the complexity of the problem. For your specific issue, if there are two interfaces, one which is up during boot and one which is down during boot, there is no problem to bond them after boot without persisting configuration. how would you know which bond mode to use? which MTU? I don't understand the question. I think I do: Alon suggests that on boot, the management interface would not have bonding at all, and use a single nic. The switch would have to assume that other nics in the bond are dead, and will use the only one which is alive to transfer packets. There is no doubt that we have to persist the vlan tag of the management interface, and its MTU, in the extremely rare case where the network would not alow Linux's default of 1500. i was thinking manager may be using jumbo frames to talk to host, and host will have an issue with them since it is set to 1500 instead of 8k. jumbo frames isn't a rare case. as for bond, are you sure you can use a nic in a non bonded mode for all bond modes? next, what if we're using openvswitch, and you need some flow definitions for the management interface? ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
On 11/28/2012 05:34 AM, Roni Luxenberg wrote: - Original Message - From: Itamar Heim ih...@redhat.com To: Dan Kenigsberg dan...@redhat.com Cc: Alon Bar-Lev alo...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org, Yaniv Kaul yk...@redhat.com Sent: Wednesday, November 28, 2012 11:01:35 AM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary On 11/28/2012 03:53 AM, Dan Kenigsberg wrote: On Tue, Nov 27, 2012 at 03:24:58PM -0500, Alon Bar-Lev wrote: Management interface configuration is a separate issue. But it is an important issue that has to be discussed.. If we perform changes of this interface when host is in maintenance we reduce the complexity of the problem. For your specific issue, if there are two interfaces, one which is up during boot and one which is down during boot, there is no problem to bond them after boot without persisting configuration. how would you know which bond mode to use? which MTU? I don't understand the question. I think I do: Alon suggests that on boot, the management interface would not have bonding at all, and use a single nic. The switch would have to assume that other nics in the bond are dead, and will use the only one which is alive to transfer packets. There is no doubt that we have to persist the vlan tag of the management interface, and its MTU, in the extremely rare case where the network would not alow Linux's default of 1500. i was thinking manager may be using jumbo frames to talk to host, and host will have an issue with them since it is set to 1500 instead of 8k. jumbo frames isn't a rare case. as for bond, are you sure you can use a nic in a non bonded mode for all bond modes? all bond modes have to cope with a situation where only a single nic is active and the rest are down, so one can boot with a single active nic and only activate the rest and promote to the desired bond mode upon getting the full network configuration from the manager. of course they need to handle single active nic, but iirc, the host must be configured for a matching bond as the switch. i.e., you can't configure the switch to be in bond, then boot the host with a single active nic in a non bonded config Changing the master interface mtu for either vlan or bond is required for management interface and non management interface. So the logic would probably be set max(mtu(slaves)) regardless it is management interface or not. I discussed this with Livnat, if there are applications that access the master interface directly we may break them, as the destination may not support non standard mtu. This is true in current implementation and any future implementation. It is bad practice to use the master interface directly (mixed tagged/untagged), better to define in switch that untagged communication belongs to vlanX, then use this explicit vlanX at host. next, what if we're using openvswitch, and you need some flow definitions for the management interface? I cannot answer that as I don't know openvswitch very well and don't know what flow definitions are, however, I do guess that it has non persistent mode that can effect any interface on its control. If you like I can research this one. you mainly need OVS for provisioning VM networks so here too you can completely bypass OVS during boot and only configure it in a transactional manner upon getting the full network configuration from the manager. a general question, why would you need to configure VM networks on the host (assuming a persistent cached configuration) upon boot if it cannot talk to the manager? after-all, in this case no resources would be scheduled to run on this host until connection to the manager is restored and up-to-date network configuration is applied. thanks, Roni ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
- Original Message - From: Itamar Heim ih...@redhat.com To: Roni Luxenberg rluxe...@redhat.com Cc: Alon Bar-Lev alo...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Wednesday, November 28, 2012 2:01:45 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary On 11/28/2012 05:34 AM, Roni Luxenberg wrote: - Original Message - From: Itamar Heim ih...@redhat.com To: Dan Kenigsberg dan...@redhat.com Cc: Alon Bar-Lev alo...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org, Yaniv Kaul yk...@redhat.com Sent: Wednesday, November 28, 2012 11:01:35 AM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary On 11/28/2012 03:53 AM, Dan Kenigsberg wrote: On Tue, Nov 27, 2012 at 03:24:58PM -0500, Alon Bar-Lev wrote: Management interface configuration is a separate issue. But it is an important issue that has to be discussed.. If we perform changes of this interface when host is in maintenance we reduce the complexity of the problem. For your specific issue, if there are two interfaces, one which is up during boot and one which is down during boot, there is no problem to bond them after boot without persisting configuration. how would you know which bond mode to use? which MTU? I don't understand the question. I think I do: Alon suggests that on boot, the management interface would not have bonding at all, and use a single nic. The switch would have to assume that other nics in the bond are dead, and will use the only one which is alive to transfer packets. There is no doubt that we have to persist the vlan tag of the management interface, and its MTU, in the extremely rare case where the network would not alow Linux's default of 1500. i was thinking manager may be using jumbo frames to talk to host, and host will have an issue with them since it is set to 1500 instead of 8k. jumbo frames isn't a rare case. as for bond, are you sure you can use a nic in a non bonded mode for all bond modes? all bond modes have to cope with a situation where only a single nic is active and the rest are down, so one can boot with a single active nic and only activate the rest and promote to the desired bond mode upon getting the full network configuration from the manager. of course they need to handle single active nic, but iirc, the host must be configured for a matching bond as the switch. i.e., you can't configure the switch to be in bond, then boot the host with a single active nic in a non bonded config as far as I know as long as the 2nd nic is down there is no problem. it is as if the cord is out. Changing the master interface mtu for either vlan or bond is required for management interface and non management interface. So the logic would probably be set max(mtu(slaves)) regardless it is management interface or not. I discussed this with Livnat, if there are applications that access the master interface directly we may break them, as the destination may not support non standard mtu. This is true in current implementation and any future implementation. It is bad practice to use the master interface directly (mixed tagged/untagged), better to define in switch that untagged communication belongs to vlanX, then use this explicit vlanX at host. next, what if we're using openvswitch, and you need some flow definitions for the management interface? I cannot answer that as I don't know openvswitch very well and don't know what flow definitions are, however, I do guess that it has non persistent mode that can effect any interface on its control. If you like I can research this one. you mainly need OVS for provisioning VM networks so here too you can completely bypass OVS during boot and only configure it in a transactional manner upon getting the full network configuration from the manager. a general question, why would you need to configure VM networks on the host (assuming a persistent cached configuration) upon boot if it cannot talk to the manager? after-all, in this case no resources would be scheduled to run on this host until connection to the manager is restored and up-to-date network configuration is applied. thanks, Roni ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
- Original Message - From: Itamar Heim ih...@redhat.com To: Roni Luxenberg rluxe...@redhat.com Cc: Alon Bar-Lev alo...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Wednesday, November 28, 2012 2:01:45 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary On 11/28/2012 05:34 AM, Roni Luxenberg wrote: - Original Message - From: Itamar Heim ih...@redhat.com To: Dan Kenigsberg dan...@redhat.com Cc: Alon Bar-Lev alo...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org, Yaniv Kaul yk...@redhat.com Sent: Wednesday, November 28, 2012 11:01:35 AM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary On 11/28/2012 03:53 AM, Dan Kenigsberg wrote: On Tue, Nov 27, 2012 at 03:24:58PM -0500, Alon Bar-Lev wrote: Management interface configuration is a separate issue. But it is an important issue that has to be discussed.. If we perform changes of this interface when host is in maintenance we reduce the complexity of the problem. For your specific issue, if there are two interfaces, one which is up during boot and one which is down during boot, there is no problem to bond them after boot without persisting configuration. how would you know which bond mode to use? which MTU? I don't understand the question. I think I do: Alon suggests that on boot, the management interface would not have bonding at all, and use a single nic. The switch would have to assume that other nics in the bond are dead, and will use the only one which is alive to transfer packets. There is no doubt that we have to persist the vlan tag of the management interface, and its MTU, in the extremely rare case where the network would not alow Linux's default of 1500. i was thinking manager may be using jumbo frames to talk to host, and host will have an issue with them since it is set to 1500 instead of 8k. jumbo frames isn't a rare case. as for bond, are you sure you can use a nic in a non bonded mode for all bond modes? all bond modes have to cope with a situation where only a single nic is active and the rest are down, so one can boot with a single active nic and only activate the rest and promote to the desired bond mode upon getting the full network configuration from the manager. of course they need to handle single active nic, but iirc, the host must be configured for a matching bond as the switch. i.e., you can't configure the switch to be in bond, then boot the host with a single active nic in a non bonded config as far as I know as long as the 2nd nic is down there is no problem. it is as if the cord is out. regular port grouping or trunking on adjacent switch does not mandate bond configuration on the host as long as only a single nic is active. OTAH, 802.3ad port grouping might require host configuration as link aggregation control packets are exchanged on the wire. If this is the case, you'd need to persist your L2 bonding config. Changing the master interface mtu for either vlan or bond is required for management interface and non management interface. So the logic would probably be set max(mtu(slaves)) regardless it is management interface or not. I discussed this with Livnat, if there are applications that access the master interface directly we may break them, as the destination may not support non standard mtu. This is true in current implementation and any future implementation. It is bad practice to use the master interface directly (mixed tagged/untagged), better to define in switch that untagged communication belongs to vlanX, then use this explicit vlanX at host. next, what if we're using openvswitch, and you need some flow definitions for the management interface? I cannot answer that as I don't know openvswitch very well and don't know what flow definitions are, however, I do guess that it has non persistent mode that can effect any interface on its control. If you like I can research this one. you mainly need OVS for provisioning VM networks so here too you can completely bypass OVS during boot and only configure it in a transactional manner upon getting the full network configuration from the manager. a general question, why would you need to configure VM networks on the host (assuming a persistent cached configuration) upon boot if it cannot talk to the manager? after-all, in this case no resources would be scheduled to run on this host until connection to the manager is restored and up-to-date network configuration is applied. thanks, Roni
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
On 26/11/12 16:59, Adam Litke wrote: On Mon, Nov 26, 2012 at 02:57:19PM +0200, Livnat Peer wrote: On 26/11/12 03:15, Shu Ming wrote: Livnat, Thanks for your summary. I got comments below. 2012-11-25 18:53, Livnat Peer: Hi All, We have been discussing $subject for a while and I'd like to summarized what we agreed and disagreed on thus far. The way I see it there are two related discussions: 1. Getting VDSM networking stack to be distribution agnostic. - We are all in agreement that VDSM API should be generic enough to incorporate multiple implementation. (discussed on this thread: Alon's suggestion, Mark's patch for adding support for netcf etc.) - We would like to maintain at least one implementation as the working/up-to-date implementation for our users, this implementation should be distribution agnostic (as we all acknowledge this is an important goal for VDSM). I also think that with the agreement of this community we can choose to change our focus, from time to time, from one implementation to another as we see fit (today it can be OVS+netcf and in a few months we'll use the quantum based implementation if we agree it is better) 2. The second discussion is about persisting the network configuration on the host vs. dynamically retrieving it from a centralized location like the engine. Danken raised a concern that even if going with the dynamic approach the host should persist the management network configuration. About dynamical retrieving from a centralized location, when will the retrieving start? Just in the very early stage of host booting before network functions? Or after the host startup and in the normal running state of the host? Before retrieving the configuration, how does the host network connecting to the engine? I think we need a basic well known network between hosts and the engine first. Then after the retrieving, hosts should reconfigure the network for later management. However, the timing to retrieve and reconfigure are challenging. We did not discuss the dynamic approach in details on the list so far and I think this is a good opportunity to start this discussion... From what was discussed previously I can say that the need for a well known network was raised by danken, it was referred to as the management network, this network would be used for pulling the full host network configuration from the centralized location, at this point the engine. About the timing for retrieving the configuration, there are several approaches. One of them was described by Alon, and I think he'll join this discussion and maybe put it in his own words, but the idea was to 'keep' the network synchronized at all times. When the host have communication channel to the engine and the engine detects there is a mismatch in the host configuration, the engine initiates 'apply network configuration' action on the host. Using this approach we'll have a single path of code to maintain and that would reduce code complexity and bugs - That's quoting Alon Bar Lev (Alon I hope I did not twisted your words/idea). On the other hand the above approach makes local tweaks on the host (done manually by the administrator) much harder. I worry a lot about the above if we take the dynamic approach. It seems we'd need to introduce before/after 'apply network configuration' hooks where the admin could add custom config commands that aren't yet modeled by engine. yes, and I'm not sure the administrators would like the fact that we are 'forcing' them to write everything in a script and getting familiar with VDSM hooking mechanism (which in some cases require the use of custom properties on the engine level) instead of running a simple command line. Any other approaches ? Static configuration has the advantage of allowing a host to bring itself back online independent of the engine. This is also useful for anyone who may want to deploy a vdsm node in standalone mode. I think it would be possible to easily support a quasi-static configuration mode simply by extending the design of the dynamic approach slightly. In dynamic mode, the network configuration is passed down as a well-defined data structure. When a particular configuration has been committed, vdsm could write a copy of that configuration data structure to /var/run/vdsm/network-config.json. During a subsequent boot, if the engine cannot be contacted after activating the management network, the cached configuration can be applied using the same code as for dynamic mode. We'd have to flesh out the circumstances under which this would happen. I like this approach a lot but we need to consider that network configuration is an accumulated state, for example - 1. The engine sends a setup-network command with the full host network configuration 2. The user configures new network on the host, the engine sends a new setup-network request to VDSM which includes only the
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
- Original Message - From: Livnat Peer lp...@redhat.com To: Adam Litke a...@us.ibm.com Cc: Alon Bar-Lev abar...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Tuesday, November 27, 2012 10:42:00 AM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary On 26/11/12 16:59, Adam Litke wrote: On Mon, Nov 26, 2012 at 02:57:19PM +0200, Livnat Peer wrote: On 26/11/12 03:15, Shu Ming wrote: Livnat, Thanks for your summary. I got comments below. 2012-11-25 18:53, Livnat Peer: Hi All, We have been discussing $subject for a while and I'd like to summarized what we agreed and disagreed on thus far. The way I see it there are two related discussions: 1. Getting VDSM networking stack to be distribution agnostic. - We are all in agreement that VDSM API should be generic enough to incorporate multiple implementation. (discussed on this thread: Alon's suggestion, Mark's patch for adding support for netcf etc.) - We would like to maintain at least one implementation as the working/up-to-date implementation for our users, this implementation should be distribution agnostic (as we all acknowledge this is an important goal for VDSM). I also think that with the agreement of this community we can choose to change our focus, from time to time, from one implementation to another as we see fit (today it can be OVS+netcf and in a few months we'll use the quantum based implementation if we agree it is better) 2. The second discussion is about persisting the network configuration on the host vs. dynamically retrieving it from a centralized location like the engine. Danken raised a concern that even if going with the dynamic approach the host should persist the management network configuration. About dynamical retrieving from a centralized location, when will the retrieving start? Just in the very early stage of host booting before network functions? Or after the host startup and in the normal running state of the host? Before retrieving the configuration, how does the host network connecting to the engine? I think we need a basic well known network between hosts and the engine first. Then after the retrieving, hosts should reconfigure the network for later management. However, the timing to retrieve and reconfigure are challenging. We did not discuss the dynamic approach in details on the list so far and I think this is a good opportunity to start this discussion... From what was discussed previously I can say that the need for a well known network was raised by danken, it was referred to as the management network, this network would be used for pulling the full host network configuration from the centralized location, at this point the engine. About the timing for retrieving the configuration, there are several approaches. One of them was described by Alon, and I think he'll join this discussion and maybe put it in his own words, but the idea was to 'keep' the network synchronized at all times. When the host have communication channel to the engine and the engine detects there is a mismatch in the host configuration, the engine initiates 'apply network configuration' action on the host. Using this approach we'll have a single path of code to maintain and that would reduce code complexity and bugs - That's quoting Alon Bar Lev (Alon I hope I did not twisted your words/idea). On the other hand the above approach makes local tweaks on the host (done manually by the administrator) much harder. I worry a lot about the above if we take the dynamic approach. It seems we'd need to introduce before/after 'apply network configuration' hooks where the admin could add custom config commands that aren't yet modeled by engine. yes, and I'm not sure the administrators would like the fact that we are 'forcing' them to write everything in a script and getting familiar with VDSM hooking mechanism (which in some cases require the use of custom properties on the engine level) instead of running a simple command line. In which case will we force? Please be more specific. If we can pass most of the iproute2, brctl, bond parameters via key/value pairs via the API, what in your view that is common or even seldom should be used? This hook mechanism is only as fallback, provided to calm people down. Any other approaches ? Static configuration has the advantage of allowing a host to bring itself back online independent of the engine. This is also useful for anyone who may want to deploy a vdsm node in standalone mode. I think it would be possible to easily support a quasi-static configuration mode simply by extending the design of the dynamic approach slightly. In dynamic mode, the network configuration is passed down as a well
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
On 27/11/12 10:53, Alon Bar-Lev wrote: - Original Message - From: Livnat Peer lp...@redhat.com To: Adam Litke a...@us.ibm.com Cc: Alon Bar-Lev abar...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Tuesday, November 27, 2012 10:42:00 AM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary On 26/11/12 16:59, Adam Litke wrote: On Mon, Nov 26, 2012 at 02:57:19PM +0200, Livnat Peer wrote: On 26/11/12 03:15, Shu Ming wrote: Livnat, Thanks for your summary. I got comments below. 2012-11-25 18:53, Livnat Peer: Hi All, We have been discussing $subject for a while and I'd like to summarized what we agreed and disagreed on thus far. The way I see it there are two related discussions: 1. Getting VDSM networking stack to be distribution agnostic. - We are all in agreement that VDSM API should be generic enough to incorporate multiple implementation. (discussed on this thread: Alon's suggestion, Mark's patch for adding support for netcf etc.) - We would like to maintain at least one implementation as the working/up-to-date implementation for our users, this implementation should be distribution agnostic (as we all acknowledge this is an important goal for VDSM). I also think that with the agreement of this community we can choose to change our focus, from time to time, from one implementation to another as we see fit (today it can be OVS+netcf and in a few months we'll use the quantum based implementation if we agree it is better) 2. The second discussion is about persisting the network configuration on the host vs. dynamically retrieving it from a centralized location like the engine. Danken raised a concern that even if going with the dynamic approach the host should persist the management network configuration. About dynamical retrieving from a centralized location, when will the retrieving start? Just in the very early stage of host booting before network functions? Or after the host startup and in the normal running state of the host? Before retrieving the configuration, how does the host network connecting to the engine? I think we need a basic well known network between hosts and the engine first. Then after the retrieving, hosts should reconfigure the network for later management. However, the timing to retrieve and reconfigure are challenging. We did not discuss the dynamic approach in details on the list so far and I think this is a good opportunity to start this discussion... From what was discussed previously I can say that the need for a well known network was raised by danken, it was referred to as the management network, this network would be used for pulling the full host network configuration from the centralized location, at this point the engine. About the timing for retrieving the configuration, there are several approaches. One of them was described by Alon, and I think he'll join this discussion and maybe put it in his own words, but the idea was to 'keep' the network synchronized at all times. When the host have communication channel to the engine and the engine detects there is a mismatch in the host configuration, the engine initiates 'apply network configuration' action on the host. Using this approach we'll have a single path of code to maintain and that would reduce code complexity and bugs - That's quoting Alon Bar Lev (Alon I hope I did not twisted your words/idea). On the other hand the above approach makes local tweaks on the host (done manually by the administrator) much harder. I worry a lot about the above if we take the dynamic approach. It seems we'd need to introduce before/after 'apply network configuration' hooks where the admin could add custom config commands that aren't yet modeled by engine. yes, and I'm not sure the administrators would like the fact that we are 'forcing' them to write everything in a script and getting familiar with VDSM hooking mechanism (which in some cases require the use of custom properties on the engine level) instead of running a simple command line. In which case will we force? Please be more specific. If we can pass most of the iproute2, brctl, bond parameters via key/value pairs via the API, what in your view that is common or even seldom should be used? This hook mechanism is only as fallback, provided to calm people down. I understand, I'm saying it can irritate the administrators that needs to use it, it does not help that we are calmed down ;) Just to make it clear I'm not against the stateless approach, I'm trying to understand it better and make sure we are all aware of the drawbacks this approach has. Complicating local tweaks to the admin is one of them. I'll reply on your original mail with the questions I have on your proposal. Any other approaches ? Static configuration has the advantage of allowing a host to bring itself
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
- Original Message - From: Livnat Peer lp...@redhat.com To: Alon Bar-Lev alo...@redhat.com Cc: VDSM Project Development vdsm-devel@lists.fedorahosted.org, Shu Ming shum...@linux.vnet.ibm.com, Saggi Mizrahi smizr...@redhat.com, Dan Kenigsberg dan...@redhat.com Sent: Tuesday, November 27, 2012 12:18:31 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary On 26/11/12 22:18, Alon Bar-Lev wrote: - Original Message - From: Livnat Peer lp...@redhat.com To: Shu Ming shum...@linux.vnet.ibm.com Cc: Alon Bar-Lev abar...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Monday, November 26, 2012 2:57:19 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary On 26/11/12 03:15, Shu Ming wrote: Livnat, Thanks for your summary. I got comments below. 2012-11-25 18:53, Livnat Peer: Hi All, We have been discussing $subject for a while and I'd like to summarized what we agreed and disagreed on thus far. The way I see it there are two related discussions: 1. Getting VDSM networking stack to be distribution agnostic. - We are all in agreement that VDSM API should be generic enough to incorporate multiple implementation. (discussed on this thread: Alon's suggestion, Mark's patch for adding support for netcf etc.) - We would like to maintain at least one implementation as the working/up-to-date implementation for our users, this implementation should be distribution agnostic (as we all acknowledge this is an important goal for VDSM). I also think that with the agreement of this community we can choose to change our focus, from time to time, from one implementation to another as we see fit (today it can be OVS+netcf and in a few months we'll use the quantum based implementation if we agree it is better) 2. The second discussion is about persisting the network configuration on the host vs. dynamically retrieving it from a centralized location like the engine. Danken raised a concern that even if going with the dynamic approach the host should persist the management network configuration. About dynamical retrieving from a centralized location, when will the retrieving start? Just in the very early stage of host booting before network functions? Or after the host startup and in the normal running state of the host? Before retrieving the configuration, how does the host network connecting to the engine? I think we need a basic well known network between hosts and the engine first. Then after the retrieving, hosts should reconfigure the network for later management. However, the timing to retrieve and reconfigure are challenging. We did not discuss the dynamic approach in details on the list so far and I think this is a good opportunity to start this discussion... From what was discussed previously I can say that the need for a well known network was raised by danken, it was referred to as the management network, this network would be used for pulling the full host network configuration from the centralized location, at this point the engine. About the timing for retrieving the configuration, there are several approaches. One of them was described by Alon, and I think he'll join this discussion and maybe put it in his own words, but the idea was to 'keep' the network synchronized at all times. When the host have communication channel to the engine and the engine detects there is a mismatch in the host configuration, the engine initiates 'apply network configuration' action on the host. Using this approach we'll have a single path of code to maintain and that would reduce code complexity and bugs - That's quoting Alon Bar Lev (Alon I hope I did not twisted your words/idea). On the other hand the above approach makes local tweaks on the host (done manually by the administrator) much harder. Any other approaches ? I'd like to add a more general question to the discussion what are the advantages of taking the dynamic approach? So far I collected two reasons: -It is a 'cleaner' design, removes complexity on VDSM code, easier to maintain going forward, and less bug prone (I agree with that one, as long as we keep the retrieving configuration mechanism/algorithm simple). -It adheres to the idea of having a stateless hypervisor - some more input on this point would be appreciated Any other advantages? discussing the benefits of having the persisted Livnat Sorry for the delay. Some more expansion. ASSUMPTION After boot a host running vdsm is able to receive communication from engine. This means that host has legitimate layer 2 configuration and layer 3 configuration for the interface used to communicate to engine. MISSION Reduce
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
On Tue, Nov 27, 2012 at 05:38:25AM -0500, Alon Bar-Lev wrote: snip ASSUMPTION After boot a host running vdsm is able to receive communication from engine. This means that host has legitimate layer 2 configuration and layer 3 configuration for the interface used to communicate to engine. MISSION Reduce complexity of implementation, so that only one algorithm is used in order to reach to operative state as far as networking is concerned. (Storage is extremely similar I can s/network/storage/ and still be relevant). For reaching the mission above we can also use the approach suggested by Adam. start from a clean configuration and execute setup network to set the host networking configuration. In Adam's proposal VDSM itself is issuing the setupNetwork and in your approach the engine does. Right. we can do this 100+ ways, question is which implementation will be the simplest. My problem with Adam's idea is http://xkcd.com/927/ : it amounts to an (n+1)th way of persisting network configuration on disk. We may have to take that way, but as with VM definitions and storage connections, I would like to keep it in a smaller service on top of vdsm. DESIGN FOCAL POINT Host running vdsm is a complete slave of its master, will it be ovirt-engine or other engine. Having a complete slave ease implementation: 1. Master always apply the setting as-is. 2. No need to consider slave state. 3. No need to implement AI to reach from unknown state X to known state Y + delta. 4. After reboot (or fence) host is always in known state. ALGORITHM A. Given communication to vdsm, I think we should not brush this permise aside. Current Vdsm API lets Engine tweak the means of communication for next boot. We had customers that wanted to add a bond, or change the vlan, or fix the IP address of the management interface. They could have used Engine for this, and declare the new configuration as safe (setSafeNetConfig). In many cases, the latter step has to be done out of band. But there are cases where this can be done completely remotely. It seems that you suggest to take this crucial configuration completely out-of-band. construct required vlan, bonding, bridge setup on machine. B. Reboot/Fence - host is reset, apply A. C. Network configuration is changed at engine: (1) Drop all resources that are not used by active VMs. I'm not sure what you mean by the above, drop all resources *not* used by VMs? Let's say we have running VM using bridge bridge1. We cannot modify this bridge1 as long as VM is operative. So we drop all network configuration except of bridge1 to allow VM to survive the upgrade. I was tempted to write something else but I did not want to alarm people But... when network configuration is changed on a host with running VMs, first move the VMs to a different host, then recycle configuration (simplest: reboot). We've been doing that until v3.1... (2) Apply A. D. Host in maintenance - network configuration can be changed, will be applied when host go into active, apply C (no resources are used by VMs, all resources are dropped). E. Critical network is down (Host not operative) - network configuration is not changed. F. Host unreachable (None responsive) - network configuration cannot be changed. What happens if we have a host that is added to the engine (or used to be non-operational and now returns to up) and reports a network configuration different than what is configured in the engine? This is a sign of totally malicious node! A trigger to fencing, active rebooting. Can you please describe a valid sequence in which it can happen? I'm not sure this example flies all, but how about a sysadmin that wants to replace our bonding definition with teaming. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
On Tue, Nov 27, 2012 at 11:56:54AM +0200, Livnat Peer wrote: On 27/11/12 10:53, Alon Bar-Lev wrote: - Original Message - From: Livnat Peer lp...@redhat.com To: Adam Litke a...@us.ibm.com Cc: Alon Bar-Lev abar...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Tuesday, November 27, 2012 10:42:00 AM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary On 26/11/12 16:59, Adam Litke wrote: On Mon, Nov 26, 2012 at 02:57:19PM +0200, Livnat Peer wrote: On 26/11/12 03:15, Shu Ming wrote: Livnat, Thanks for your summary. I got comments below. 2012-11-25 18:53, Livnat Peer: Hi All, We have been discussing $subject for a while and I'd like to summarized what we agreed and disagreed on thus far. The way I see it there are two related discussions: 1. Getting VDSM networking stack to be distribution agnostic. - We are all in agreement that VDSM API should be generic enough to incorporate multiple implementation. (discussed on this thread: Alon's suggestion, Mark's patch for adding support for netcf etc.) - We would like to maintain at least one implementation as the working/up-to-date implementation for our users, this implementation should be distribution agnostic (as we all acknowledge this is an important goal for VDSM). I also think that with the agreement of this community we can choose to change our focus, from time to time, from one implementation to another as we see fit (today it can be OVS+netcf and in a few months we'll use the quantum based implementation if we agree it is better) 2. The second discussion is about persisting the network configuration on the host vs. dynamically retrieving it from a centralized location like the engine. Danken raised a concern that even if going with the dynamic approach the host should persist the management network configuration. About dynamical retrieving from a centralized location, when will the retrieving start? Just in the very early stage of host booting before network functions? Or after the host startup and in the normal running state of the host? Before retrieving the configuration, how does the host network connecting to the engine? I think we need a basic well known network between hosts and the engine first. Then after the retrieving, hosts should reconfigure the network for later management. However, the timing to retrieve and reconfigure are challenging. We did not discuss the dynamic approach in details on the list so far and I think this is a good opportunity to start this discussion... From what was discussed previously I can say that the need for a well known network was raised by danken, it was referred to as the management network, this network would be used for pulling the full host network configuration from the centralized location, at this point the engine. About the timing for retrieving the configuration, there are several approaches. One of them was described by Alon, and I think he'll join this discussion and maybe put it in his own words, but the idea was to 'keep' the network synchronized at all times. When the host have communication channel to the engine and the engine detects there is a mismatch in the host configuration, the engine initiates 'apply network configuration' action on the host. Using this approach we'll have a single path of code to maintain and that would reduce code complexity and bugs - That's quoting Alon Bar Lev (Alon I hope I did not twisted your words/idea). On the other hand the above approach makes local tweaks on the host (done manually by the administrator) much harder. I worry a lot about the above if we take the dynamic approach. It seems we'd need to introduce before/after 'apply network configuration' hooks where the admin could add custom config commands that aren't yet modeled by engine. yes, and I'm not sure the administrators would like the fact that we are 'forcing' them to write everything in a script and getting familiar with VDSM hooking mechanism (which in some cases require the use of custom properties on the engine level) instead of running a simple command line. In which case will we force? Please be more specific. If we can pass most of the iproute2, brctl, bond parameters via key/value pairs via the API, what in your view that is common or even seldom should be used? This hook mechanism is only as fallback, provided to calm people down. I understand, I'm saying it can irritate the administrators that needs to use it, it does not help that we are calmed down ;) Just to make it clear I'm not against the stateless approach, I'm trying to understand it better and make sure we are all aware of the drawbacks this approach has. Complicating local tweaks
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
- Original Message - From: Dan Kenigsberg dan...@redhat.com To: Livnat Peer lp...@redhat.com Cc: Alon Bar-Lev alo...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Tuesday, November 27, 2012 4:22:19 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary snip Livnat, I don't see any argument of persistence vs non persistence as the above is common to any approach taken. Only this manual configuration argument keeps poping, which as I wrote is irrelevant in large scale and we do want to go into large scale. Well, we call it manual configuration, but it applies just as well to puppet-based configuration. Dan. There can by only one (manager to each host). Alon. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
On Tue, Nov 27, 2012 at 10:42:00AM +0200, Livnat Peer wrote: On 26/11/12 16:59, Adam Litke wrote: On Mon, Nov 26, 2012 at 02:57:19PM +0200, Livnat Peer wrote: On 26/11/12 03:15, Shu Ming wrote: Livnat, Thanks for your summary. I got comments below. 2012-11-25 18:53, Livnat Peer: Hi All, We have been discussing $subject for a while and I'd like to summarized what we agreed and disagreed on thus far. The way I see it there are two related discussions: 1. Getting VDSM networking stack to be distribution agnostic. - We are all in agreement that VDSM API should be generic enough to incorporate multiple implementation. (discussed on this thread: Alon's suggestion, Mark's patch for adding support for netcf etc.) - We would like to maintain at least one implementation as the working/up-to-date implementation for our users, this implementation should be distribution agnostic (as we all acknowledge this is an important goal for VDSM). I also think that with the agreement of this community we can choose to change our focus, from time to time, from one implementation to another as we see fit (today it can be OVS+netcf and in a few months we'll use the quantum based implementation if we agree it is better) 2. The second discussion is about persisting the network configuration on the host vs. dynamically retrieving it from a centralized location like the engine. Danken raised a concern that even if going with the dynamic approach the host should persist the management network configuration. About dynamical retrieving from a centralized location, when will the retrieving start? Just in the very early stage of host booting before network functions? Or after the host startup and in the normal running state of the host? Before retrieving the configuration, how does the host network connecting to the engine? I think we need a basic well known network between hosts and the engine first. Then after the retrieving, hosts should reconfigure the network for later management. However, the timing to retrieve and reconfigure are challenging. We did not discuss the dynamic approach in details on the list so far and I think this is a good opportunity to start this discussion... From what was discussed previously I can say that the need for a well known network was raised by danken, it was referred to as the management network, this network would be used for pulling the full host network configuration from the centralized location, at this point the engine. About the timing for retrieving the configuration, there are several approaches. One of them was described by Alon, and I think he'll join this discussion and maybe put it in his own words, but the idea was to 'keep' the network synchronized at all times. When the host have communication channel to the engine and the engine detects there is a mismatch in the host configuration, the engine initiates 'apply network configuration' action on the host. Using this approach we'll have a single path of code to maintain and that would reduce code complexity and bugs - That's quoting Alon Bar Lev (Alon I hope I did not twisted your words/idea). On the other hand the above approach makes local tweaks on the host (done manually by the administrator) much harder. I worry a lot about the above if we take the dynamic approach. It seems we'd need to introduce before/after 'apply network configuration' hooks where the admin could add custom config commands that aren't yet modeled by engine. yes, and I'm not sure the administrators would like the fact that we are 'forcing' them to write everything in a script and getting familiar with VDSM hooking mechanism (which in some cases require the use of custom properties on the engine level) instead of running a simple command line. Any other approaches ? Static configuration has the advantage of allowing a host to bring itself back online independent of the engine. This is also useful for anyone who may want to deploy a vdsm node in standalone mode. I think it would be possible to easily support a quasi-static configuration mode simply by extending the design of the dynamic approach slightly. In dynamic mode, the network configuration is passed down as a well-defined data structure. When a particular configuration has been committed, vdsm could write a copy of that configuration data structure to /var/run/vdsm/network-config.json. During a subsequent boot, if the engine cannot be contacted after activating the management network, the cached configuration can be applied using the same code as for dynamic mode. We'd have to flesh out the circumstances under which this would happen. I like this approach a lot but we need to consider that network configuration is an accumulated state, for example - 1. The engine sends a
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
On Mon, Nov 26, 2012 at 06:13:01PM -0500, Alon Bar-Lev wrote: Hello, - Original Message - From: Adam Litke a...@us.ibm.com To: Alon Bar-Lev alo...@redhat.com Cc: Livnat Peer lp...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Tuesday, November 27, 2012 12:51:36 AM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary Nice writeup! I like where this is going but see my comments inline below. On Mon, Nov 26, 2012 at 03:18:22PM -0500, Alon Bar-Lev wrote: - Original Message - From: Livnat Peer lp...@redhat.com To: Shu Ming shum...@linux.vnet.ibm.com Cc: Alon Bar-Lev abar...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Monday, November 26, 2012 2:57:19 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary On 26/11/12 03:15, Shu Ming wrote: Livnat, Thanks for your summary. I got comments below. 2012-11-25 18:53, Livnat Peer: Hi All, We have been discussing $subject for a while and I'd like to summarized what we agreed and disagreed on thus far. The way I see it there are two related discussions: 1. Getting VDSM networking stack to be distribution agnostic. - We are all in agreement that VDSM API should be generic enough to incorporate multiple implementation. (discussed on this thread: Alon's suggestion, Mark's patch for adding support for netcf etc.) - We would like to maintain at least one implementation as the working/up-to-date implementation for our users, this implementation should be distribution agnostic (as we all acknowledge this is an important goal for VDSM). I also think that with the agreement of this community we can choose to change our focus, from time to time, from one implementation to another as we see fit (today it can be OVS+netcf and in a few months we'll use the quantum based implementation if we agree it is better) 2. The second discussion is about persisting the network configuration on the host vs. dynamically retrieving it from a centralized location like the engine. Danken raised a concern that even if going with the dynamic approach the host should persist the management network configuration. About dynamical retrieving from a centralized location, when will the retrieving start? Just in the very early stage of host booting before network functions? Or after the host startup and in the normal running state of the host? Before retrieving the configuration, how does the host network connecting to the engine? I think we need a basic well known network between hosts and the engine first. Then after the retrieving, hosts should reconfigure the network for later management. However, the timing to retrieve and reconfigure are challenging. We did not discuss the dynamic approach in details on the list so far and I think this is a good opportunity to start this discussion... From what was discussed previously I can say that the need for a well known network was raised by danken, it was referred to as the management network, this network would be used for pulling the full host network configuration from the centralized location, at this point the engine. About the timing for retrieving the configuration, there are several approaches. One of them was described by Alon, and I think he'll join this discussion and maybe put it in his own words, but the idea was to 'keep' the network synchronized at all times. When the host have communication channel to the engine and the engine detects there is a mismatch in the host configuration, the engine initiates 'apply network configuration' action on the host. Using this approach we'll have a single path of code to maintain and that would reduce code complexity and bugs - That's quoting Alon Bar Lev (Alon I hope I did not twisted your words/idea). On the other hand the above approach makes local tweaks on the host (done manually by the administrator) much harder. Any other approaches ? I'd like to add a more general question to the discussion what are the advantages of taking the dynamic approach? So far I collected two reasons: -It is a 'cleaner' design, removes complexity on VDSM code, easier to maintain going forward, and less bug prone (I agree with that one, as long as we keep the retrieving configuration mechanism/algorithm simple). -It adheres to the idea of having a stateless hypervisor - some more input on this point would be appreciated Any other advantages? discussing the benefits of having the persisted Livnat Sorry for the delay. Some more
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
On 11/26/2012 03:18 PM, Alon Bar-Lev wrote: - Original Message - From: Livnat Peer lp...@redhat.com To: Shu Ming shum...@linux.vnet.ibm.com Cc: Alon Bar-Lev abar...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Monday, November 26, 2012 2:57:19 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary On 26/11/12 03:15, Shu Ming wrote: Livnat, Thanks for your summary. I got comments below. 2012-11-25 18:53, Livnat Peer: Hi All, We have been discussing $subject for a while and I'd like to summarized what we agreed and disagreed on thus far. The way I see it there are two related discussions: 1. Getting VDSM networking stack to be distribution agnostic. - We are all in agreement that VDSM API should be generic enough to incorporate multiple implementation. (discussed on this thread: Alon's suggestion, Mark's patch for adding support for netcf etc.) - We would like to maintain at least one implementation as the working/up-to-date implementation for our users, this implementation should be distribution agnostic (as we all acknowledge this is an important goal for VDSM). I also think that with the agreement of this community we can choose to change our focus, from time to time, from one implementation to another as we see fit (today it can be OVS+netcf and in a few months we'll use the quantum based implementation if we agree it is better) 2. The second discussion is about persisting the network configuration on the host vs. dynamically retrieving it from a centralized location like the engine. Danken raised a concern that even if going with the dynamic approach the host should persist the management network configuration. About dynamical retrieving from a centralized location, when will the retrieving start? Just in the very early stage of host booting before network functions? Or after the host startup and in the normal running state of the host? Before retrieving the configuration, how does the host network connecting to the engine? I think we need a basic well known network between hosts and the engine first. Then after the retrieving, hosts should reconfigure the network for later management. However, the timing to retrieve and reconfigure are challenging. We did not discuss the dynamic approach in details on the list so far and I think this is a good opportunity to start this discussion... From what was discussed previously I can say that the need for a well known network was raised by danken, it was referred to as the management network, this network would be used for pulling the full host network configuration from the centralized location, at this point the engine. About the timing for retrieving the configuration, there are several approaches. One of them was described by Alon, and I think he'll join this discussion and maybe put it in his own words, but the idea was to 'keep' the network synchronized at all times. When the host have communication channel to the engine and the engine detects there is a mismatch in the host configuration, the engine initiates 'apply network configuration' action on the host. Using this approach we'll have a single path of code to maintain and that would reduce code complexity and bugs - That's quoting Alon Bar Lev (Alon I hope I did not twisted your words/idea). On the other hand the above approach makes local tweaks on the host (done manually by the administrator) much harder. Any other approaches ? I'd like to add a more general question to the discussion what are the advantages of taking the dynamic approach? So far I collected two reasons: -It is a 'cleaner' design, removes complexity on VDSM code, easier to maintain going forward, and less bug prone (I agree with that one, as long as we keep the retrieving configuration mechanism/algorithm simple). -It adheres to the idea of having a stateless hypervisor - some more input on this point would be appreciated Any other advantages? discussing the benefits of having the persisted Livnat Sorry for the delay. Some more expansion. ASSUMPTION After boot a host running vdsm is able to receive communication from engine. This means that host has legitimate layer 2 configuration and layer 3 configuration for the interface used to communicate to engine. MISSION Reduce complexity of implementation, so that only one algorithm is used in order to reach to operative state as far as networking is concerned. (Storage is extremely similar I can s/network/storage/ and still be relevant). DESIGN FOCAL POINT Host running vdsm is a complete slave of its master, will it be ovirt-engine or other engine. Having a complete slave ease implementation: 1. Master always apply the setting as-is. 2. No need to consider slave state. 3. No need to implement AI to reach from unknown state X to known state Y + delta. 4. After reboot (or fence) host is always in known state. ALGORITHM A. Given communication to vdsm, construct
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
- Original Message - From: Itamar Heim ih...@redhat.com To: Alon Bar-Lev alo...@redhat.com Cc: Livnat Peer lp...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Tuesday, November 27, 2012 10:08:34 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary On 11/26/2012 03:18 PM, Alon Bar-Lev wrote: - Original Message - From: Livnat Peer lp...@redhat.com To: Shu Ming shum...@linux.vnet.ibm.com Cc: Alon Bar-Lev abar...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Monday, November 26, 2012 2:57:19 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary On 26/11/12 03:15, Shu Ming wrote: Livnat, Thanks for your summary. I got comments below. 2012-11-25 18:53, Livnat Peer: Hi All, We have been discussing $subject for a while and I'd like to summarized what we agreed and disagreed on thus far. The way I see it there are two related discussions: 1. Getting VDSM networking stack to be distribution agnostic. - We are all in agreement that VDSM API should be generic enough to incorporate multiple implementation. (discussed on this thread: Alon's suggestion, Mark's patch for adding support for netcf etc.) - We would like to maintain at least one implementation as the working/up-to-date implementation for our users, this implementation should be distribution agnostic (as we all acknowledge this is an important goal for VDSM). I also think that with the agreement of this community we can choose to change our focus, from time to time, from one implementation to another as we see fit (today it can be OVS+netcf and in a few months we'll use the quantum based implementation if we agree it is better) 2. The second discussion is about persisting the network configuration on the host vs. dynamically retrieving it from a centralized location like the engine. Danken raised a concern that even if going with the dynamic approach the host should persist the management network configuration. About dynamical retrieving from a centralized location, when will the retrieving start? Just in the very early stage of host booting before network functions? Or after the host startup and in the normal running state of the host? Before retrieving the configuration, how does the host network connecting to the engine? I think we need a basic well known network between hosts and the engine first. Then after the retrieving, hosts should reconfigure the network for later management. However, the timing to retrieve and reconfigure are challenging. We did not discuss the dynamic approach in details on the list so far and I think this is a good opportunity to start this discussion... From what was discussed previously I can say that the need for a well known network was raised by danken, it was referred to as the management network, this network would be used for pulling the full host network configuration from the centralized location, at this point the engine. About the timing for retrieving the configuration, there are several approaches. One of them was described by Alon, and I think he'll join this discussion and maybe put it in his own words, but the idea was to 'keep' the network synchronized at all times. When the host have communication channel to the engine and the engine detects there is a mismatch in the host configuration, the engine initiates 'apply network configuration' action on the host. Using this approach we'll have a single path of code to maintain and that would reduce code complexity and bugs - That's quoting Alon Bar Lev (Alon I hope I did not twisted your words/idea). On the other hand the above approach makes local tweaks on the host (done manually by the administrator) much harder. Any other approaches ? I'd like to add a more general question to the discussion what are the advantages of taking the dynamic approach? So far I collected two reasons: -It is a 'cleaner' design, removes complexity on VDSM code, easier to maintain going forward, and less bug prone (I agree with that one, as long as we keep the retrieving configuration mechanism/algorithm simple). -It adheres to the idea of having a stateless hypervisor - some more input on this point would be appreciated Any other advantages? discussing the benefits of having the persisted Livnat Sorry for the delay. Some more expansion. ASSUMPTION After boot a host running vdsm is able to receive communication from engine. This means that host has legitimate layer 2 configuration and layer 3 configuration for the interface used to communicate to engine. MISSION Reduce complexity of implementation, so that only one algorithm is used in order
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
- Original Message - From: Itamar Heim ih...@redhat.com To: Alon Bar-Lev alo...@redhat.com Cc: Livnat Peer lp...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Tuesday, November 27, 2012 10:19:51 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary On 11/27/2012 03:17 PM, Alon Bar-Lev wrote: - Original Message - From: Itamar Heim ih...@redhat.com To: Alon Bar-Lev alo...@redhat.com Cc: Livnat Peer lp...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Tuesday, November 27, 2012 10:08:34 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary On 11/26/2012 03:18 PM, Alon Bar-Lev wrote: - Original Message - From: Livnat Peer lp...@redhat.com To: Shu Ming shum...@linux.vnet.ibm.com Cc: Alon Bar-Lev abar...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Monday, November 26, 2012 2:57:19 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary On 26/11/12 03:15, Shu Ming wrote: Livnat, Thanks for your summary. I got comments below. 2012-11-25 18:53, Livnat Peer: Hi All, We have been discussing $subject for a while and I'd like to summarized what we agreed and disagreed on thus far. The way I see it there are two related discussions: 1. Getting VDSM networking stack to be distribution agnostic. - We are all in agreement that VDSM API should be generic enough to incorporate multiple implementation. (discussed on this thread: Alon's suggestion, Mark's patch for adding support for netcf etc.) - We would like to maintain at least one implementation as the working/up-to-date implementation for our users, this implementation should be distribution agnostic (as we all acknowledge this is an important goal for VDSM). I also think that with the agreement of this community we can choose to change our focus, from time to time, from one implementation to another as we see fit (today it can be OVS+netcf and in a few months we'll use the quantum based implementation if we agree it is better) 2. The second discussion is about persisting the network configuration on the host vs. dynamically retrieving it from a centralized location like the engine. Danken raised a concern that even if going with the dynamic approach the host should persist the management network configuration. About dynamical retrieving from a centralized location, when will the retrieving start? Just in the very early stage of host booting before network functions? Or after the host startup and in the normal running state of the host? Before retrieving the configuration, how does the host network connecting to the engine? I think we need a basic well known network between hosts and the engine first. Then after the retrieving, hosts should reconfigure the network for later management. However, the timing to retrieve and reconfigure are challenging. We did not discuss the dynamic approach in details on the list so far and I think this is a good opportunity to start this discussion... From what was discussed previously I can say that the need for a well known network was raised by danken, it was referred to as the management network, this network would be used for pulling the full host network configuration from the centralized location, at this point the engine. About the timing for retrieving the configuration, there are several approaches. One of them was described by Alon, and I think he'll join this discussion and maybe put it in his own words, but the idea was to 'keep' the network synchronized at all times. When the host have communication channel to the engine and the engine detects there is a mismatch in the host configuration, the engine initiates 'apply network configuration' action on the host. Using this approach we'll have a single path of code to maintain and that would reduce code complexity and bugs - That's quoting Alon Bar Lev (Alon I hope I did not twisted your words/idea). On the other hand the above approach makes local tweaks on the host (done manually by the administrator) much harder. Any other approaches ? I'd like to add a more general question to the discussion what are the advantages of taking the dynamic approach? So far I collected two reasons: -It is a 'cleaner' design, removes complexity on VDSM code, easier to maintain going forward, and less bug prone (I agree with that one, as long as we keep the retrieving configuration mechanism/algorithm simple). -It adheres to the idea of having a stateless hypervisor - some more input on this point would be appreciated Any other advantages? discussing the benefits
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
On 26/11/12 03:15, Shu Ming wrote: Livnat, Thanks for your summary. I got comments below. 2012-11-25 18:53, Livnat Peer: Hi All, We have been discussing $subject for a while and I'd like to summarized what we agreed and disagreed on thus far. The way I see it there are two related discussions: 1. Getting VDSM networking stack to be distribution agnostic. - We are all in agreement that VDSM API should be generic enough to incorporate multiple implementation. (discussed on this thread: Alon's suggestion, Mark's patch for adding support for netcf etc.) - We would like to maintain at least one implementation as the working/up-to-date implementation for our users, this implementation should be distribution agnostic (as we all acknowledge this is an important goal for VDSM). I also think that with the agreement of this community we can choose to change our focus, from time to time, from one implementation to another as we see fit (today it can be OVS+netcf and in a few months we'll use the quantum based implementation if we agree it is better) 2. The second discussion is about persisting the network configuration on the host vs. dynamically retrieving it from a centralized location like the engine. Danken raised a concern that even if going with the dynamic approach the host should persist the management network configuration. About dynamical retrieving from a centralized location, when will the retrieving start? Just in the very early stage of host booting before network functions? Or after the host startup and in the normal running state of the host? Before retrieving the configuration, how does the host network connecting to the engine? I think we need a basic well known network between hosts and the engine first. Then after the retrieving, hosts should reconfigure the network for later management. However, the timing to retrieve and reconfigure are challenging. We did not discuss the dynamic approach in details on the list so far and I think this is a good opportunity to start this discussion... From what was discussed previously I can say that the need for a well known network was raised by danken, it was referred to as the management network, this network would be used for pulling the full host network configuration from the centralized location, at this point the engine. About the timing for retrieving the configuration, there are several approaches. One of them was described by Alon, and I think he'll join this discussion and maybe put it in his own words, but the idea was to 'keep' the network synchronized at all times. When the host have communication channel to the engine and the engine detects there is a mismatch in the host configuration, the engine initiates 'apply network configuration' action on the host. Using this approach we'll have a single path of code to maintain and that would reduce code complexity and bugs - That's quoting Alon Bar Lev (Alon I hope I did not twisted your words/idea). On the other hand the above approach makes local tweaks on the host (done manually by the administrator) much harder. Any other approaches ? I'd like to add a more general question to the discussion what are the advantages of taking the dynamic approach? So far I collected two reasons: -It is a 'cleaner' design, removes complexity on VDSM code, easier to maintain going forward, and less bug prone (I agree with that one, as long as we keep the retrieving configuration mechanism/algorithm simple). -It adheres to the idea of having a stateless hypervisor - some more input on this point would be appreciated Any other advantages? discussing the benefits of having the persisted Livnat ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
On Mon, Nov 26, 2012 at 02:57:19PM +0200, Livnat Peer wrote: On 26/11/12 03:15, Shu Ming wrote: Livnat, Thanks for your summary. I got comments below. 2012-11-25 18:53, Livnat Peer: Hi All, We have been discussing $subject for a while and I'd like to summarized what we agreed and disagreed on thus far. The way I see it there are two related discussions: 1. Getting VDSM networking stack to be distribution agnostic. - We are all in agreement that VDSM API should be generic enough to incorporate multiple implementation. (discussed on this thread: Alon's suggestion, Mark's patch for adding support for netcf etc.) - We would like to maintain at least one implementation as the working/up-to-date implementation for our users, this implementation should be distribution agnostic (as we all acknowledge this is an important goal for VDSM). I also think that with the agreement of this community we can choose to change our focus, from time to time, from one implementation to another as we see fit (today it can be OVS+netcf and in a few months we'll use the quantum based implementation if we agree it is better) 2. The second discussion is about persisting the network configuration on the host vs. dynamically retrieving it from a centralized location like the engine. Danken raised a concern that even if going with the dynamic approach the host should persist the management network configuration. About dynamical retrieving from a centralized location, when will the retrieving start? Just in the very early stage of host booting before network functions? Or after the host startup and in the normal running state of the host? Before retrieving the configuration, how does the host network connecting to the engine? I think we need a basic well known network between hosts and the engine first. Then after the retrieving, hosts should reconfigure the network for later management. However, the timing to retrieve and reconfigure are challenging. We did not discuss the dynamic approach in details on the list so far and I think this is a good opportunity to start this discussion... From what was discussed previously I can say that the need for a well known network was raised by danken, it was referred to as the management network, this network would be used for pulling the full host network configuration from the centralized location, at this point the engine. About the timing for retrieving the configuration, there are several approaches. One of them was described by Alon, and I think he'll join this discussion and maybe put it in his own words, but the idea was to 'keep' the network synchronized at all times. When the host have communication channel to the engine and the engine detects there is a mismatch in the host configuration, the engine initiates 'apply network configuration' action on the host. Using this approach we'll have a single path of code to maintain and that would reduce code complexity and bugs - That's quoting Alon Bar Lev (Alon I hope I did not twisted your words/idea). On the other hand the above approach makes local tweaks on the host (done manually by the administrator) much harder. I worry a lot about the above if we take the dynamic approach. It seems we'd need to introduce before/after 'apply network configuration' hooks where the admin could add custom config commands that aren't yet modeled by engine. Any other approaches ? Static configuration has the advantage of allowing a host to bring itself back online independent of the engine. This is also useful for anyone who may want to deploy a vdsm node in standalone mode. I think it would be possible to easily support a quasi-static configuration mode simply by extending the design of the dynamic approach slightly. In dynamic mode, the network configuration is passed down as a well-defined data structure. When a particular configuration has been committed, vdsm could write a copy of that configuration data structure to /var/run/vdsm/network-config.json. During a subsequent boot, if the engine cannot be contacted after activating the management network, the cached configuration can be applied using the same code as for dynamic mode. We'd have to flesh out the circumstances under which this would happen. I'd like to add a more general question to the discussion what are the advantages of taking the dynamic approach? So far I collected two reasons: -It is a 'cleaner' design, removes complexity on VDSM code, easier to maintain going forward, and less bug prone (I agree with that one, as long as we keep the retrieving configuration mechanism/algorithm simple). -It adheres to the idea of having a stateless hypervisor - some more input on this point would be appreciated Any other advantages? discussing the benefits of having the persisted As I mentioned above, the main benefit I see of having some sort of
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
- Original Message - From: Livnat Peer lp...@redhat.com To: Shu Ming shum...@linux.vnet.ibm.com Cc: Alon Bar-Lev abar...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Monday, November 26, 2012 2:57:19 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary On 26/11/12 03:15, Shu Ming wrote: Livnat, Thanks for your summary. I got comments below. 2012-11-25 18:53, Livnat Peer: Hi All, We have been discussing $subject for a while and I'd like to summarized what we agreed and disagreed on thus far. The way I see it there are two related discussions: 1. Getting VDSM networking stack to be distribution agnostic. - We are all in agreement that VDSM API should be generic enough to incorporate multiple implementation. (discussed on this thread: Alon's suggestion, Mark's patch for adding support for netcf etc.) - We would like to maintain at least one implementation as the working/up-to-date implementation for our users, this implementation should be distribution agnostic (as we all acknowledge this is an important goal for VDSM). I also think that with the agreement of this community we can choose to change our focus, from time to time, from one implementation to another as we see fit (today it can be OVS+netcf and in a few months we'll use the quantum based implementation if we agree it is better) 2. The second discussion is about persisting the network configuration on the host vs. dynamically retrieving it from a centralized location like the engine. Danken raised a concern that even if going with the dynamic approach the host should persist the management network configuration. About dynamical retrieving from a centralized location, when will the retrieving start? Just in the very early stage of host booting before network functions? Or after the host startup and in the normal running state of the host? Before retrieving the configuration, how does the host network connecting to the engine? I think we need a basic well known network between hosts and the engine first. Then after the retrieving, hosts should reconfigure the network for later management. However, the timing to retrieve and reconfigure are challenging. We did not discuss the dynamic approach in details on the list so far and I think this is a good opportunity to start this discussion... From what was discussed previously I can say that the need for a well known network was raised by danken, it was referred to as the management network, this network would be used for pulling the full host network configuration from the centralized location, at this point the engine. About the timing for retrieving the configuration, there are several approaches. One of them was described by Alon, and I think he'll join this discussion and maybe put it in his own words, but the idea was to 'keep' the network synchronized at all times. When the host have communication channel to the engine and the engine detects there is a mismatch in the host configuration, the engine initiates 'apply network configuration' action on the host. Using this approach we'll have a single path of code to maintain and that would reduce code complexity and bugs - That's quoting Alon Bar Lev (Alon I hope I did not twisted your words/idea). On the other hand the above approach makes local tweaks on the host (done manually by the administrator) much harder. Any other approaches ? I'd like to add a more general question to the discussion what are the advantages of taking the dynamic approach? So far I collected two reasons: -It is a 'cleaner' design, removes complexity on VDSM code, easier to maintain going forward, and less bug prone (I agree with that one, as long as we keep the retrieving configuration mechanism/algorithm simple). -It adheres to the idea of having a stateless hypervisor - some more input on this point would be appreciated Any other advantages? discussing the benefits of having the persisted Livnat Sorry for the delay. Some more expansion. ASSUMPTION After boot a host running vdsm is able to receive communication from engine. This means that host has legitimate layer 2 configuration and layer 3 configuration for the interface used to communicate to engine. MISSION Reduce complexity of implementation, so that only one algorithm is used in order to reach to operative state as far as networking is concerned. (Storage is extremely similar I can s/network/storage/ and still be relevant). DESIGN FOCAL POINT Host running vdsm is a complete slave of its master, will it be ovirt-engine or other engine. Having a complete slave ease implementation: 1. Master always apply the setting as-is. 2. No need to consider slave state. 3. No need to implement AI to reach from unknown state X to known state Y + delta. 4
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
Nice writeup! I like where this is going but see my comments inline below. On Mon, Nov 26, 2012 at 03:18:22PM -0500, Alon Bar-Lev wrote: - Original Message - From: Livnat Peer lp...@redhat.com To: Shu Ming shum...@linux.vnet.ibm.com Cc: Alon Bar-Lev abar...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Monday, November 26, 2012 2:57:19 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary On 26/11/12 03:15, Shu Ming wrote: Livnat, Thanks for your summary. I got comments below. 2012-11-25 18:53, Livnat Peer: Hi All, We have been discussing $subject for a while and I'd like to summarized what we agreed and disagreed on thus far. The way I see it there are two related discussions: 1. Getting VDSM networking stack to be distribution agnostic. - We are all in agreement that VDSM API should be generic enough to incorporate multiple implementation. (discussed on this thread: Alon's suggestion, Mark's patch for adding support for netcf etc.) - We would like to maintain at least one implementation as the working/up-to-date implementation for our users, this implementation should be distribution agnostic (as we all acknowledge this is an important goal for VDSM). I also think that with the agreement of this community we can choose to change our focus, from time to time, from one implementation to another as we see fit (today it can be OVS+netcf and in a few months we'll use the quantum based implementation if we agree it is better) 2. The second discussion is about persisting the network configuration on the host vs. dynamically retrieving it from a centralized location like the engine. Danken raised a concern that even if going with the dynamic approach the host should persist the management network configuration. About dynamical retrieving from a centralized location, when will the retrieving start? Just in the very early stage of host booting before network functions? Or after the host startup and in the normal running state of the host? Before retrieving the configuration, how does the host network connecting to the engine? I think we need a basic well known network between hosts and the engine first. Then after the retrieving, hosts should reconfigure the network for later management. However, the timing to retrieve and reconfigure are challenging. We did not discuss the dynamic approach in details on the list so far and I think this is a good opportunity to start this discussion... From what was discussed previously I can say that the need for a well known network was raised by danken, it was referred to as the management network, this network would be used for pulling the full host network configuration from the centralized location, at this point the engine. About the timing for retrieving the configuration, there are several approaches. One of them was described by Alon, and I think he'll join this discussion and maybe put it in his own words, but the idea was to 'keep' the network synchronized at all times. When the host have communication channel to the engine and the engine detects there is a mismatch in the host configuration, the engine initiates 'apply network configuration' action on the host. Using this approach we'll have a single path of code to maintain and that would reduce code complexity and bugs - That's quoting Alon Bar Lev (Alon I hope I did not twisted your words/idea). On the other hand the above approach makes local tweaks on the host (done manually by the administrator) much harder. Any other approaches ? I'd like to add a more general question to the discussion what are the advantages of taking the dynamic approach? So far I collected two reasons: -It is a 'cleaner' design, removes complexity on VDSM code, easier to maintain going forward, and less bug prone (I agree with that one, as long as we keep the retrieving configuration mechanism/algorithm simple). -It adheres to the idea of having a stateless hypervisor - some more input on this point would be appreciated Any other advantages? discussing the benefits of having the persisted Livnat Sorry for the delay. Some more expansion. ASSUMPTION After boot a host running vdsm is able to receive communication from engine. This means that host has legitimate layer 2 configuration and layer 3 configuration for the interface used to communicate to engine. MISSION Reduce complexity of implementation, so that only one algorithm is used in order to reach to operative state as far as networking is concerned. (Storage is extremely similar I can s/network/storage/ and still be relevant). DESIGN FOCAL POINT Host running vdsm is a complete slave
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
Hello, - Original Message - From: Adam Litke a...@us.ibm.com To: Alon Bar-Lev alo...@redhat.com Cc: Livnat Peer lp...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Tuesday, November 27, 2012 12:51:36 AM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary Nice writeup! I like where this is going but see my comments inline below. On Mon, Nov 26, 2012 at 03:18:22PM -0500, Alon Bar-Lev wrote: - Original Message - From: Livnat Peer lp...@redhat.com To: Shu Ming shum...@linux.vnet.ibm.com Cc: Alon Bar-Lev abar...@redhat.com, VDSM Project Development vdsm-devel@lists.fedorahosted.org Sent: Monday, November 26, 2012 2:57:19 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary On 26/11/12 03:15, Shu Ming wrote: Livnat, Thanks for your summary. I got comments below. 2012-11-25 18:53, Livnat Peer: Hi All, We have been discussing $subject for a while and I'd like to summarized what we agreed and disagreed on thus far. The way I see it there are two related discussions: 1. Getting VDSM networking stack to be distribution agnostic. - We are all in agreement that VDSM API should be generic enough to incorporate multiple implementation. (discussed on this thread: Alon's suggestion, Mark's patch for adding support for netcf etc.) - We would like to maintain at least one implementation as the working/up-to-date implementation for our users, this implementation should be distribution agnostic (as we all acknowledge this is an important goal for VDSM). I also think that with the agreement of this community we can choose to change our focus, from time to time, from one implementation to another as we see fit (today it can be OVS+netcf and in a few months we'll use the quantum based implementation if we agree it is better) 2. The second discussion is about persisting the network configuration on the host vs. dynamically retrieving it from a centralized location like the engine. Danken raised a concern that even if going with the dynamic approach the host should persist the management network configuration. About dynamical retrieving from a centralized location, when will the retrieving start? Just in the very early stage of host booting before network functions? Or after the host startup and in the normal running state of the host? Before retrieving the configuration, how does the host network connecting to the engine? I think we need a basic well known network between hosts and the engine first. Then after the retrieving, hosts should reconfigure the network for later management. However, the timing to retrieve and reconfigure are challenging. We did not discuss the dynamic approach in details on the list so far and I think this is a good opportunity to start this discussion... From what was discussed previously I can say that the need for a well known network was raised by danken, it was referred to as the management network, this network would be used for pulling the full host network configuration from the centralized location, at this point the engine. About the timing for retrieving the configuration, there are several approaches. One of them was described by Alon, and I think he'll join this discussion and maybe put it in his own words, but the idea was to 'keep' the network synchronized at all times. When the host have communication channel to the engine and the engine detects there is a mismatch in the host configuration, the engine initiates 'apply network configuration' action on the host. Using this approach we'll have a single path of code to maintain and that would reduce code complexity and bugs - That's quoting Alon Bar Lev (Alon I hope I did not twisted your words/idea). On the other hand the above approach makes local tweaks on the host (done manually by the administrator) much harder. Any other approaches ? I'd like to add a more general question to the discussion what are the advantages of taking the dynamic approach? So far I collected two reasons: -It is a 'cleaner' design, removes complexity on VDSM code, easier to maintain going forward, and less bug prone (I agree with that one, as long as we keep the retrieving configuration mechanism/algorithm simple). -It adheres to the idea of having a stateless hypervisor - some more input on this point would be appreciated Any other advantages? discussing the benefits of having the persisted Livnat Sorry for the delay. Some
Re: [vdsm] Future of Vdsm network configuration - Thread mid-summary
Livnat, Thanks for your summary. I got comments below. 2012-11-25 18:53, Livnat Peer: Hi All, We have been discussing $subject for a while and I'd like to summarized what we agreed and disagreed on thus far. The way I see it there are two related discussions: 1. Getting VDSM networking stack to be distribution agnostic. - We are all in agreement that VDSM API should be generic enough to incorporate multiple implementation. (discussed on this thread: Alon's suggestion, Mark's patch for adding support for netcf etc.) - We would like to maintain at least one implementation as the working/up-to-date implementation for our users, this implementation should be distribution agnostic (as we all acknowledge this is an important goal for VDSM). I also think that with the agreement of this community we can choose to change our focus, from time to time, from one implementation to another as we see fit (today it can be OVS+netcf and in a few months we'll use the quantum based implementation if we agree it is better) 2. The second discussion is about persisting the network configuration on the host vs. dynamically retrieving it from a centralized location like the engine. Danken raised a concern that even if going with the dynamic approach the host should persist the management network configuration. About dynamical retrieving from a centralized location, when will the retrieving start? Just in the very early stage of host booting before network functions? Or after the host startup and in the normal running state of the host? Before retrieving the configuration, how does the host network connecting to the engine? I think we need a basic well known network between hosts and the engine first. Then after the retrieving, hosts should reconfigure the network for later management. However, the timing to retrieve and reconfigure are challenging. Obviously the second discussion influences the API modeling. Since I think it would be challenging to add support for generic API and change the current implementation to match the dynamic configuration approach simultaneously I suggest we'll focus our efforts on one change at a time. I suggest to have a discussion on the pro's and con's of dynamic configuration and after we get to a consensus around that we can start modeling the generic API. thoughts? comments? Livnat ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel -- --- 舒明 Shu Ming Open Virtualization Engineerning; CSTL, IBM Corp. Tel: 86-10-82451626 Tieline: 9051626 E-mail: shum...@cn.ibm.com or shum...@linux.vnet.ibm.com Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193, PRC ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Future of Vdsm network configuration
On 11/18/2012 10:52 AM, Itamar Heim wrote: On 11/18/2012 07:55 AM, Gary Kotton wrote: On 11/17/2012 09:13 PM, Itamar Heim wrote: On 11/17/2012 11:56 AM, Gary Kotton wrote: On 11/17/2012 11:00 AM, Alon Bar-Lev wrote: Hello, After discussion calm down, I want to once again to ask a question. Why isn't this discussion focusing on the interface vdsm will use to access network provider? Why should vdsm core care which network technology it actually uses? Quantum? 1. that's still a specific implementation. I tend to disgree, Quantum is an interface enabling one to manage virtual networks. If I understand correctly this is similar to what Alon is suggesting. At the end of the day VDSM will need to interface with linuxbridge, openvswitch, nics that provide SRIOV etc. This may be done either by VDSM or Quantum agents (in some case there may be no Quantum agents - for example if a NVP controller is used). Quantum enables VDSM and oVirt to consume external technologies that are currently not supported today. For example, if one want to use openvswicth. There is a open source implementation of OVS that is managed by Quantum. That is, a Quantum agent builds and manages all flows. Do you want VDSM to do this? 2. last i checked, it is far from covering the API needed by vdsm for provisioning network configurations, rather than just consuming them? (i.e., i don't remember quantum ever intends to provide an api to bond physical interfaces, etc). Quantum agents may do this. Yes, it will entail some hooks in VDSM but it will provide a large majority of the work that you guys are talking about. The added bonus is that it works with a number of technologies that are not supported by VDSM. I have yet to understand why VDSM has to invent the wheel again. At the moment there is a lot of work being done in Quantum to expose additional services - for example LBaaS. It would be interesting to know if the current networking plans address this. This should be something on the radar and in my opinion is something essential to any networking infrastructure. i didn't see anything in quantum leading me to feel it plans to expose a stable api for configuring/provisioning itself? I do not understand your comment. Via Quantum provider networks Quantum enables one to connect a specific network interface to a virtual network. At the end of the day this connection is done by configuring the agent. If the community ever decides to adopt Quantum, which I would consider a healthy and forward moving decision, then this is something that would need to be managed by VDSM (my understanding is that the only free lunch is one at a youth hostel in the outback in Australia - one needs to by his/her drink). This is why I am in favor of what Dan and Mark have suggested regarding the OVS integration. At the end of the day someone needs to do the wiring. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Future of Vdsm network configuration
On Wed, Nov 14, 2012 at 10:54:34AM -0500, Saggi Mizrahi wrote: I'm late to the party as usual... I'm all for dynamic set up of hosts, I think it's the only way to go. I don't understand how it can work anyway else. I did not expect the thread to go this way, but I agree that network setup is an exception: for storage and virtual machines, we do not persist anything on the node. For networking we need to persist only the management connection. Everything else can be volatile, created by the client after the node boots. That being said, if everything is set up dynamically it doesn't matter what backend we use to set it up as long as we can query the state. We can even mix and match. Or am I missing something. Choosing a backend is important, as all implementations are. We have the setupNetwork API. We could change its semantics to mean do not persist. Now is the time to consider implementations, too. Dan. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Future of Vdsm network configuration
On 11/17/2012 11:56 AM, Gary Kotton wrote: On 11/17/2012 11:00 AM, Alon Bar-Lev wrote: Hello, After discussion calm down, I want to once again to ask a question. Why isn't this discussion focusing on the interface vdsm will use to access network provider? Why should vdsm core care which network technology it actually uses? Quantum? 1. that's still a specific implementation. 2. last i checked, it is far from covering the API needed by vdsm for provisioning network configurations, rather than just consuming them? (i.e., i don't remember quantum ever intends to provide an api to bond physical interfaces, etc). With proper design of such interface, and the ability to select interface implementation using configuration, vdsm will be able to work with various of technologies without a change. Technologies can be either network manager, ovs, libvirt or basic. What popular now can be unpopular in future, what is considered stable enough for now, may be not stable enough for future uses, what is maintained now may be unmaintained in future. Developing tightly coupled software is something I would avoid if not absolutely required. People may vote which interface they like to have now and we can implement one, while in time we may see other implementations as contributions. This will also allow us to move from one technology to another with decent effort/costs if required for any reason. Best Regards, Alon Bar-Lev. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Future of Vdsm network configuration
On 11/15/2012 08:56 PM, huntxu wrote: On Thu, 15 Nov 2012 17:54:42 +0800, Gary Kotton gkot...@redhat.com wrote: On 11/14/2012 05:42 PM, Mark Wu wrote: On 11/14/2012 07:53 PM, Gary Kotton wrote: On 11/14/2012 11:53 AM, Livnat Peer wrote: On 14/11/12 00:28, Adam Litke wrote: On Sun, Nov 11, 2012 at 09:46:43AM -0500, Alon Bar-Lev wrote: snip Can we just start with running ovs in standalone mode at first? Yes, most certainly. It could have the basic forward function based on MAC-learning and bond/vlans/tunnel function by specifying related options when adding a new port. We could connect each physical nic for vm network with an ovs bridge, and then the VM can get external network access. I agree with that without adding a controller, we can't get a unified control panel. But I think the standalone mode could fit current oVirt network model well. Gary, please correct me if I am wrong or any suggestions from you? You are correct. This is certainly one way of achieving a first step for integrating with the OVS. My concerns are as follows (maybe some of them do not exist :)): +1 for a standalone ovs as a first step. 1. Boot process with binding to physical NICS to the OVS Both ifup/down scripts shipped with upstream ovs and bridge compatible mode work well in my test. 2. The OVS maintains a database. This may need to be cleaned of tap devices when the appliance reboots - lets take an edge case into account - say the appliance has a number of VMs running - there will be tap devices for these VMs registered with the OVS. If there is a exception or power failure the appliance will reset. These devices will still be registered when the appliance reboots. Who cleans them and when? Yes, I also prefer the ovs database be clean every time it starts. It should know nothing about the configuration when starting, except for essential ones for the host to connect to the management end. In my mind, vdsm/ovs should configure the machine only if requested by the management end. The configuration, however, is centralized. I think it's better to continue this discussion after we get the first draft of ovs integration page done as Gary suggested. 3. What about the traditional bridged network - will these be migrated to OVS. I don't think we are going to drop the traditional bridged network support. Isn't providing another choice better than only one? Could we implement a generic layer providing consistent APIs for management, as well as calling different low-level libs/tools among environments which requirements varies from one to another? I have submit a patch for it: http://gerrit.ovirt.org/#/c/7915/ It would be appreciated if you could review it. Thanks Gary Thanks Mark. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Future of Vdsm network configuration
On 11/14/2012 05:42 PM, Mark Wu wrote: On 11/14/2012 07:53 PM, Gary Kotton wrote: On 11/14/2012 11:53 AM, Livnat Peer wrote: On 14/11/12 00:28, Adam Litke wrote: On Sun, Nov 11, 2012 at 09:46:43AM -0500, Alon Bar-Lev wrote: - Original Message - From: Dan Kenigsbergdan...@redhat.com To: vdsm-de...@fedorahosted.org Sent: Sunday, November 11, 2012 4:07:30 PM Subject: [vdsm] Future of Vdsm network configuration Hi, Nowadays, when vdsm receives the setupNetowrk verb, it mangles /etc/sysconfig/network-scripts/ifcfg-* files and restarts the network service, so they are read by the responsible SysV service. This is very much Fedora-oriented, and not up with the new themes in Linux network configuration. Since we want oVirt and Vdsm to be distribution agnostic, and support new features, we have to change. setupNetwork is responsible for two different things: (1) configure the host networking interfaces, and (2) create virtual networks for guests and connect the to the world over (1). Functionality (2) is provided by building Linux software bridges, and vlan devices. I'd like to explore moving it to Open vSwitch, which would enable a host of functionalities that we currently lack (e.g. tunneling). One thing that worries me is the need to reimplement our config snapshot/recovery on ovs's database. As far as I know, ovs is unable to maintain host level parameters of interfaces (e.g. eth0's IPv4 address), so we need another tool for functionality (1): either speak to NetworkManager directly, or to use NetCF, via its libvirt virInterface* wrapper. I have minor worries about NetCF's breadth of testing and usage; I know it is intended to be cross-platform, but unlike ovs, I am not aware of a wide Debian usage thereof. On the other hand, its API is ready for vdsm's usage for quite a while. NetworkManager has become ubiquitous, and we'd better integrate with it better than our current setting of NM_CONTROLLED=no. But as DPB tells us, https://lists.fedorahosted.org/pipermail/vdsm-devel/2012-November/001677.html we'd better offload integration with NM to libvirt. We would like to take Network configuration in VDSM to the next level and make it distribution agnostic in addition for setting the infrastructure for more advanced features to be used going forward. The path we think of taking is to integrate with OVS and for feature completeness use NetCF, via its libvirt virInterface* wrapper. Any comments or feedback on this proposal is welcomed. Thanks to the oVirt net team members who's input has helped writing this email. Hi, As far as I see this, network manager is a monster that is a huge dependency to have just to create bridges or configure network interfaces... It is true that on a host where network manager lives it would be not polite to define network resources not via its interface, however I don't like we force network manager. libvirt is long not used as virtualization library but system management agent, I am not sure this is the best system agent I would have chosen. I think that all the terms and building blocks got lost in time... and the result integration became more and more complex. Stabilizing such multi-layered component environment is much harder than monolithic environment. I would really want to see vdsm as monolithic component with full control over its resources, I believe this is the only way vdsm can be stable enough to be production grade. Hypervisor should be a total slave of manager (or cluster), so I have no problem in bypassing/disabling any distribution specific tool in favour of atoms (brctl, iproute), in non persistence mode. I know this derive some more work, but I don't think it is that complex to implement and maintain. Just my 2 cents... I couldn't disagree more. What you are suggesting requires that we reimplement every single networking feature in oVirt by ourselves. If we want to support the (absolutely critical) goal of being distro agnostic, then we need to implement the same functionality across multiple distros too. This is more work than we will ever be able to keep up with. If you think it's hard to stabilize the integration of an external networking library, imagine how hard it will be to stabilize our own rewritten and buggy version. This is not how open source is supposed to work. We should be assembling distinct, modular, pre-existing components together when they are available. If NetworkManager has integration problems, let's work upstream to fix them. If it's dependencies are too great, let's modularize it so we don't need to ship the parts that we don't need. I agree with Adam on this one, reimplementing the networking management layer by ourselves using only atoms seems like duplication of work that was already done and available for our use both by NM and by libvirt. Yes, it is not perfect (far from it actually) but I think we better focus our efforts around adding new
Re: [vdsm] Future of Vdsm network configuration
On Thu, 15 Nov 2012 17:54:42 +0800, Gary Kotton gkot...@redhat.com wrote: On 11/14/2012 05:42 PM, Mark Wu wrote: On 11/14/2012 07:53 PM, Gary Kotton wrote: On 11/14/2012 11:53 AM, Livnat Peer wrote: On 14/11/12 00:28, Adam Litke wrote: On Sun, Nov 11, 2012 at 09:46:43AM -0500, Alon Bar-Lev wrote: snip Can we just start with running ovs in standalone mode at first? Yes, most certainly. It could have the basic forward function based on MAC-learning and bond/vlans/tunnel function by specifying related options when adding a new port. We could connect each physical nic for vm network with an ovs bridge, and then the VM can get external network access. I agree with that without adding a controller, we can't get a unified control panel. But I think the standalone mode could fit current oVirt network model well. Gary, please correct me if I am wrong or any suggestions from you? You are correct. This is certainly one way of achieving a first step for integrating with the OVS. My concerns are as follows (maybe some of them do not exist :)): +1 for a standalone ovs as a first step. 1. Boot process with binding to physical NICS to the OVS Both ifup/down scripts shipped with upstream ovs and bridge compatible mode work well in my test. 2. The OVS maintains a database. This may need to be cleaned of tap devices when the appliance reboots - lets take an edge case into account - say the appliance has a number of VMs running - there will be tap devices for these VMs registered with the OVS. If there is a exception or power failure the appliance will reset. These devices will still be registered when the appliance reboots. Who cleans them and when? Yes, I also prefer the ovs database be clean every time it starts. It should know nothing about the configuration when starting, except for essential ones for the host to connect to the management end. In my mind, vdsm/ovs should configure the machine only if requested by the management end. The configuration, however, is centralized. 3. What about the traditional bridged network - will these be migrated to OVS. I don't think we are going to drop the traditional bridged network support. Isn't providing another choice better than only one? Could we implement a generic layer providing consistent APIs for management, as well as calling different low-level libs/tools among environments which requirements varies from one to another? Thanks Gary Thanks Mark. -- regards, hunt ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] Future of Vdsm network configuration
On 14/11/12 00:28, Adam Litke wrote: On Sun, Nov 11, 2012 at 09:46:43AM -0500, Alon Bar-Lev wrote: - Original Message - From: Dan Kenigsberg dan...@redhat.com To: vdsm-de...@fedorahosted.org Sent: Sunday, November 11, 2012 4:07:30 PM Subject: [vdsm] Future of Vdsm network configuration Hi, Nowadays, when vdsm receives the setupNetowrk verb, it mangles /etc/sysconfig/network-scripts/ifcfg-* files and restarts the network service, so they are read by the responsible SysV service. This is very much Fedora-oriented, and not up with the new themes in Linux network configuration. Since we want oVirt and Vdsm to be distribution agnostic, and support new features, we have to change. setupNetwork is responsible for two different things: (1) configure the host networking interfaces, and (2) create virtual networks for guests and connect the to the world over (1). Functionality (2) is provided by building Linux software bridges, and vlan devices. I'd like to explore moving it to Open vSwitch, which would enable a host of functionalities that we currently lack (e.g. tunneling). One thing that worries me is the need to reimplement our config snapshot/recovery on ovs's database. As far as I know, ovs is unable to maintain host level parameters of interfaces (e.g. eth0's IPv4 address), so we need another tool for functionality (1): either speak to NetworkManager directly, or to use NetCF, via its libvirt virInterface* wrapper. I have minor worries about NetCF's breadth of testing and usage; I know it is intended to be cross-platform, but unlike ovs, I am not aware of a wide Debian usage thereof. On the other hand, its API is ready for vdsm's usage for quite a while. NetworkManager has become ubiquitous, and we'd better integrate with it better than our current setting of NM_CONTROLLED=no. But as DPB tells us, https://lists.fedorahosted.org/pipermail/vdsm-devel/2012-November/001677.html we'd better offload integration with NM to libvirt. We would like to take Network configuration in VDSM to the next level and make it distribution agnostic in addition for setting the infrastructure for more advanced features to be used going forward. The path we think of taking is to integrate with OVS and for feature completeness use NetCF, via its libvirt virInterface* wrapper. Any comments or feedback on this proposal is welcomed. Thanks to the oVirt net team members who's input has helped writing this email. Hi, As far as I see this, network manager is a monster that is a huge dependency to have just to create bridges or configure network interfaces... It is true that on a host where network manager lives it would be not polite to define network resources not via its interface, however I don't like we force network manager. libvirt is long not used as virtualization library but system management agent, I am not sure this is the best system agent I would have chosen. I think that all the terms and building blocks got lost in time... and the result integration became more and more complex. Stabilizing such multi-layered component environment is much harder than monolithic environment. I would really want to see vdsm as monolithic component with full control over its resources, I believe this is the only way vdsm can be stable enough to be production grade. Hypervisor should be a total slave of manager (or cluster), so I have no problem in bypassing/disabling any distribution specific tool in favour of atoms (brctl, iproute), in non persistence mode. I know this derive some more work, but I don't think it is that complex to implement and maintain. Just my 2 cents... I couldn't disagree more. What you are suggesting requires that we reimplement every single networking feature in oVirt by ourselves. If we want to support the (absolutely critical) goal of being distro agnostic, then we need to implement the same functionality across multiple distros too. This is more work than we will ever be able to keep up with. If you think it's hard to stabilize the integration of an external networking library, imagine how hard it will be to stabilize our own rewritten and buggy version. This is not how open source is supposed to work. We should be assembling distinct, modular, pre-existing components together when they are available. If NetworkManager has integration problems, let's work upstream to fix them. If it's dependencies are too great, let's modularize it so we don't need to ship the parts that we don't need. I agree with Adam on this one, reimplementing the networking management layer by ourselves using only atoms seems like duplication of work that was already done and available for our use both by NM and by libvirt. Yes, it is not perfect (far from it actually) but I think we better focus our efforts around adding new functionalities to VDSM and improving the current robustness of the code (we have
Re: [vdsm] Future of Vdsm network configuration
On 11/14/2012 11:53 AM, Livnat Peer wrote: On 14/11/12 00:28, Adam Litke wrote: On Sun, Nov 11, 2012 at 09:46:43AM -0500, Alon Bar-Lev wrote: - Original Message - From: Dan Kenigsbergdan...@redhat.com To: vdsm-de...@fedorahosted.org Sent: Sunday, November 11, 2012 4:07:30 PM Subject: [vdsm] Future of Vdsm network configuration Hi, Nowadays, when vdsm receives the setupNetowrk verb, it mangles /etc/sysconfig/network-scripts/ifcfg-* files and restarts the network service, so they are read by the responsible SysV service. This is very much Fedora-oriented, and not up with the new themes in Linux network configuration. Since we want oVirt and Vdsm to be distribution agnostic, and support new features, we have to change. setupNetwork is responsible for two different things: (1) configure the host networking interfaces, and (2) create virtual networks for guests and connect the to the world over (1). Functionality (2) is provided by building Linux software bridges, and vlan devices. I'd like to explore moving it to Open vSwitch, which would enable a host of functionalities that we currently lack (e.g. tunneling). One thing that worries me is the need to reimplement our config snapshot/recovery on ovs's database. As far as I know, ovs is unable to maintain host level parameters of interfaces (e.g. eth0's IPv4 address), so we need another tool for functionality (1): either speak to NetworkManager directly, or to use NetCF, via its libvirt virInterface* wrapper. I have minor worries about NetCF's breadth of testing and usage; I know it is intended to be cross-platform, but unlike ovs, I am not aware of a wide Debian usage thereof. On the other hand, its API is ready for vdsm's usage for quite a while. NetworkManager has become ubiquitous, and we'd better integrate with it better than our current setting of NM_CONTROLLED=no. But as DPB tells us, https://lists.fedorahosted.org/pipermail/vdsm-devel/2012-November/001677.html we'd better offload integration with NM to libvirt. We would like to take Network configuration in VDSM to the next level and make it distribution agnostic in addition for setting the infrastructure for more advanced features to be used going forward. The path we think of taking is to integrate with OVS and for feature completeness use NetCF, via its libvirt virInterface* wrapper. Any comments or feedback on this proposal is welcomed. Thanks to the oVirt net team members who's input has helped writing this email. Hi, As far as I see this, network manager is a monster that is a huge dependency to have just to create bridges or configure network interfaces... It is true that on a host where network manager lives it would be not polite to define network resources not via its interface, however I don't like we force network manager. libvirt is long not used as virtualization library but system management agent, I am not sure this is the best system agent I would have chosen. I think that all the terms and building blocks got lost in time... and the result integration became more and more complex. Stabilizing such multi-layered component environment is much harder than monolithic environment. I would really want to see vdsm as monolithic component with full control over its resources, I believe this is the only way vdsm can be stable enough to be production grade. Hypervisor should be a total slave of manager (or cluster), so I have no problem in bypassing/disabling any distribution specific tool in favour of atoms (brctl, iproute), in non persistence mode. I know this derive some more work, but I don't think it is that complex to implement and maintain. Just my 2 cents... I couldn't disagree more. What you are suggesting requires that we reimplement every single networking feature in oVirt by ourselves. If we want to support the (absolutely critical) goal of being distro agnostic, then we need to implement the same functionality across multiple distros too. This is more work than we will ever be able to keep up with. If you think it's hard to stabilize the integration of an external networking library, imagine how hard it will be to stabilize our own rewritten and buggy version. This is not how open source is supposed to work. We should be assembling distinct, modular, pre-existing components together when they are available. If NetworkManager has integration problems, let's work upstream to fix them. If it's dependencies are too great, let's modularize it so we don't need to ship the parts that we don't need. I agree with Adam on this one, reimplementing the networking management layer by ourselves using only atoms seems like duplication of work that was already done and available for our use both by NM and by libvirt. Yes, it is not perfect (far from it actually) but I think we better focus our efforts around adding new functionalities to VDSM and improving the current robustness of the code (we have issues regardless of any external component we're using
Re: [vdsm] Future of Vdsm network configuration
On Wed, Nov 14, 2012 at 11:53:06AM +0200, Livnat Peer wrote: On 14/11/12 00:28, Adam Litke wrote: On Sun, Nov 11, 2012 at 09:46:43AM -0500, Alon Bar-Lev wrote: - Original Message - From: Dan Kenigsberg dan...@redhat.com To: vdsm-de...@fedorahosted.org Sent: Sunday, November 11, 2012 4:07:30 PM Subject: [vdsm] Future of Vdsm network configuration Hi, Nowadays, when vdsm receives the setupNetowrk verb, it mangles /etc/sysconfig/network-scripts/ifcfg-* files and restarts the network service, so they are read by the responsible SysV service. This is very much Fedora-oriented, and not up with the new themes in Linux network configuration. Since we want oVirt and Vdsm to be distribution agnostic, and support new features, we have to change. setupNetwork is responsible for two different things: (1) configure the host networking interfaces, and (2) create virtual networks for guests and connect the to the world over (1). Functionality (2) is provided by building Linux software bridges, and vlan devices. I'd like to explore moving it to Open vSwitch, which would enable a host of functionalities that we currently lack (e.g. tunneling). One thing that worries me is the need to reimplement our config snapshot/recovery on ovs's database. As far as I know, ovs is unable to maintain host level parameters of interfaces (e.g. eth0's IPv4 address), so we need another tool for functionality (1): either speak to NetworkManager directly, or to use NetCF, via its libvirt virInterface* wrapper. I have minor worries about NetCF's breadth of testing and usage; I know it is intended to be cross-platform, but unlike ovs, I am not aware of a wide Debian usage thereof. On the other hand, its API is ready for vdsm's usage for quite a while. NetworkManager has become ubiquitous, and we'd better integrate with it better than our current setting of NM_CONTROLLED=no. But as DPB tells us, https://lists.fedorahosted.org/pipermail/vdsm-devel/2012-November/001677.html we'd better offload integration with NM to libvirt. We would like to take Network configuration in VDSM to the next level and make it distribution agnostic in addition for setting the infrastructure for more advanced features to be used going forward. The path we think of taking is to integrate with OVS and for feature completeness use NetCF, via its libvirt virInterface* wrapper. Any comments or feedback on this proposal is welcomed. Thanks to the oVirt net team members who's input has helped writing this email. Hi, As far as I see this, network manager is a monster that is a huge dependency to have just to create bridges or configure network interfaces... It is true that on a host where network manager lives it would be not polite to define network resources not via its interface, however I don't like we force network manager. libvirt is long not used as virtualization library but system management agent, I am not sure this is the best system agent I would have chosen. I think that all the terms and building blocks got lost in time... and the result integration became more and more complex. Stabilizing such multi-layered component environment is much harder than monolithic environment. I would really want to see vdsm as monolithic component with full control over its resources, I believe this is the only way vdsm can be stable enough to be production grade. Hypervisor should be a total slave of manager (or cluster), so I have no problem in bypassing/disabling any distribution specific tool in favour of atoms (brctl, iproute), in non persistence mode. I know this derive some more work, but I don't think it is that complex to implement and maintain. Just my 2 cents... I couldn't disagree more. What you are suggesting requires that we reimplement every single networking feature in oVirt by ourselves. If we want to support the (absolutely critical) goal of being distro agnostic, then we need to implement the same functionality across multiple distros too. This is more work than we will ever be able to keep up with. If you think it's hard to stabilize the integration of an external networking library, imagine how hard it will be to stabilize our own rewritten and buggy version. This is not how open source is supposed to work. We should be assembling distinct, modular, pre-existing components together when they are available. If NetworkManager has integration problems, let's work upstream to fix them. If it's dependencies are too great, let's modularize it so we don't need to ship the parts that we don't need. I agree with Adam on this one, reimplementing the networking management layer by ourselves using only atoms seems like duplication of work that was already done and available for our use both by NM
Re: [vdsm] Future of Vdsm network configuration
Hi Alon, Alon Bar-Lev píše v Ne 11. 11. 2012 v 13:28 -0500: - Original Message - From: Antoni Segura Puimedon asegu...@redhat.com To: Alon Bar-Lev alo...@redhat.com Cc: vdsm-de...@fedorahosted.org, Dan Kenigsberg dan...@redhat.com Sent: Sunday, November 11, 2012 5:47:54 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Original Message - From: Alon Bar-Lev alo...@redhat.com To: Dan Kenigsberg dan...@redhat.com Cc: vdsm-de...@fedorahosted.org Sent: Sunday, November 11, 2012 3:46:43 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Original Message - From: Dan Kenigsberg dan...@redhat.com To: vdsm-de...@fedorahosted.org Sent: Sunday, November 11, 2012 4:07:30 PM Subject: [vdsm] Future of Vdsm network configuration Hi, Nowadays, when vdsm receives the setupNetowrk verb, it mangles /etc/sysconfig/network-scripts/ifcfg-* files and restarts the network service, so they are read by the responsible SysV service. This is very much Fedora-oriented, and not up with the new themes in Linux network configuration. Since we want oVirt and Vdsm to be distribution agnostic, and support new features, we have to change. setupNetwork is responsible for two different things: (1) configure the host networking interfaces, and (2) create virtual networks for guests and connect the to the world over (1). Functionality (2) is provided by building Linux software bridges, and vlan devices. I'd like to explore moving it to Open vSwitch, which would enable a host of functionalities that we currently lack (e.g. tunneling). One thing that worries me is the need to reimplement our config snapshot/recovery on ovs's database. As far as I know, ovs is unable to maintain host level parameters of interfaces (e.g. eth0's IPv4 address), so we need another tool for functionality (1): either speak to NetworkManager directly, or to use NetCF, via its libvirt virInterface* wrapper. I have minor worries about NetCF's breadth of testing and usage; I know it is intended to be cross-platform, but unlike ovs, I am not aware of a wide Debian usage thereof. On the other hand, its API is ready for vdsm's usage for quite a while. NetworkManager has become ubiquitous, and we'd better integrate with it better than our current setting of NM_CONTROLLED=no. But as DPB tells us, https://lists.fedorahosted.org/pipermail/vdsm-devel/2012-November/001677.html we'd better offload integration with NM to libvirt. We would like to take Network configuration in VDSM to the next level and make it distribution agnostic in addition for setting the infrastructure for more advanced features to be used going forward. The path we think of taking is to integrate with OVS and for feature completeness use NetCF, via its libvirt virInterface* wrapper. Any comments or feedback on this proposal is welcomed. Thanks to the oVirt net team members who's input has helped writing this email. Hi, As far as I see this, network manager is a monster that is a huge dependency to have just to create bridges or configure network interfaces... It is true that on a host where network manager lives it would be not polite to define network resources not via its interface, however I don't like we force network manager. NM is a default way of network configuration from F17 on and it's available on all platforms. It isn't exactly small but it wouldn't pull any dependency AFAICT because all its dependencies are on Fedora initramfs already... libvirt is long not used as virtualization library but system management agent, I am not sure this is the best system agent I would have chosen. I think that all the terms and building blocks got lost in time... and the result integration became more and more complex. Stabilizing such multi-layered component environment is much harder than monolithic environment. I would really want to see vdsm as monolithic component with full control over its resources, I believe this is the only way vdsm can be stable enough to be production grade. Hypervisor should be a total slave of manager (or cluster), so I have no problem in bypassing/disabling any distribution specific tool in favour of atoms (brctl, iproute), in non persistence mode. So you propose that we would keep the network configuration database ourselves (something like sqlite maybe), disable network.service and networkmanager.service and put up and down the interfaces we need via brctl/iproute, sysfs and other netlink talking interfaces right? I won't deny that for hypervisor nodes it sounds really
Re: [vdsm] Future of Vdsm network configuration
- Original Message - From: David Jaša dj...@redhat.com To: Alon Bar-Lev alo...@redhat.com Cc: vdsm-de...@fedorahosted.org Sent: Monday, November 12, 2012 7:13:19 PM Subject: Re: [vdsm] Future of Vdsm network configuration Hi Alon, Alon Bar-Lev píše v Ne 11. 11. 2012 v 13:28 -0500: - Original Message - From: Antoni Segura Puimedon asegu...@redhat.com To: Alon Bar-Lev alo...@redhat.com Cc: vdsm-de...@fedorahosted.org, Dan Kenigsberg dan...@redhat.com Sent: Sunday, November 11, 2012 5:47:54 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Original Message - From: Alon Bar-Lev alo...@redhat.com To: Dan Kenigsberg dan...@redhat.com Cc: vdsm-de...@fedorahosted.org Sent: Sunday, November 11, 2012 3:46:43 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Original Message - From: Dan Kenigsberg dan...@redhat.com To: vdsm-de...@fedorahosted.org Sent: Sunday, November 11, 2012 4:07:30 PM Subject: [vdsm] Future of Vdsm network configuration Hi, Nowadays, when vdsm receives the setupNetowrk verb, it mangles /etc/sysconfig/network-scripts/ifcfg-* files and restarts the network service, so they are read by the responsible SysV service. This is very much Fedora-oriented, and not up with the new themes in Linux network configuration. Since we want oVirt and Vdsm to be distribution agnostic, and support new features, we have to change. setupNetwork is responsible for two different things: (1) configure the host networking interfaces, and (2) create virtual networks for guests and connect the to the world over (1). Functionality (2) is provided by building Linux software bridges, and vlan devices. I'd like to explore moving it to Open vSwitch, which would enable a host of functionalities that we currently lack (e.g. tunneling). One thing that worries me is the need to reimplement our config snapshot/recovery on ovs's database. As far as I know, ovs is unable to maintain host level parameters of interfaces (e.g. eth0's IPv4 address), so we need another tool for functionality (1): either speak to NetworkManager directly, or to use NetCF, via its libvirt virInterface* wrapper. I have minor worries about NetCF's breadth of testing and usage; I know it is intended to be cross-platform, but unlike ovs, I am not aware of a wide Debian usage thereof. On the other hand, its API is ready for vdsm's usage for quite a while. NetworkManager has become ubiquitous, and we'd better integrate with it better than our current setting of NM_CONTROLLED=no. But as DPB tells us, https://lists.fedorahosted.org/pipermail/vdsm-devel/2012-November/001677.html we'd better offload integration with NM to libvirt. We would like to take Network configuration in VDSM to the next level and make it distribution agnostic in addition for setting the infrastructure for more advanced features to be used going forward. The path we think of taking is to integrate with OVS and for feature completeness use NetCF, via its libvirt virInterface* wrapper. Any comments or feedback on this proposal is welcomed. Thanks to the oVirt net team members who's input has helped writing this email. Hi, As far as I see this, network manager is a monster that is a huge dependency to have just to create bridges or configure network interfaces... It is true that on a host where network manager lives it would be not polite to define network resources not via its interface, however I don't like we force network manager. NM is a default way of network configuration from F17 on and it's available on all platforms. It isn't exactly small but it wouldn't pull any dependency AFAICT because all its dependencies are on Fedora initramfs already... libvirt is long not used as virtualization library but system management agent, I am not sure this is the best system agent I would have chosen. I think that all the terms and building blocks got lost in time... and the result integration became more and more complex. Stabilizing such multi-layered component environment is much harder than monolithic environment. I would really want to see vdsm as monolithic component with full control over its resources, I believe this is the only way vdsm can be stable enough to be production grade. Hypervisor should be a total slave of manager (or cluster), so I have no problem
Re: [vdsm] Future of Vdsm network configuration
Hi Alon, Alon Bar-Lev píše v Po 12. 11. 2012 v 12:22 -0500: - Original Message - From: David Jaša dj...@redhat.com To: Alon Bar-Lev alo...@redhat.com Cc: vdsm-de...@fedorahosted.org Sent: Monday, November 12, 2012 7:13:19 PM Subject: Re: [vdsm] Future of Vdsm network configuration Hi Alon, Alon Bar-Lev píše v Ne 11. 11. 2012 v 13:28 -0500: - Original Message - From: Antoni Segura Puimedon asegu...@redhat.com To: Alon Bar-Lev alo...@redhat.com Cc: vdsm-de...@fedorahosted.org, Dan Kenigsberg dan...@redhat.com Sent: Sunday, November 11, 2012 5:47:54 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Original Message - From: Alon Bar-Lev alo...@redhat.com To: Dan Kenigsberg dan...@redhat.com Cc: vdsm-de...@fedorahosted.org Sent: Sunday, November 11, 2012 3:46:43 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Original Message - From: Dan Kenigsberg dan...@redhat.com To: vdsm-de...@fedorahosted.org Sent: Sunday, November 11, 2012 4:07:30 PM Subject: [vdsm] Future of Vdsm network configuration Hi, Nowadays, when vdsm receives the setupNetowrk verb, it mangles /etc/sysconfig/network-scripts/ifcfg-* files and restarts the network service, so they are read by the responsible SysV service. This is very much Fedora-oriented, and not up with the new themes in Linux network configuration. Since we want oVirt and Vdsm to be distribution agnostic, and support new features, we have to change. setupNetwork is responsible for two different things: (1) configure the host networking interfaces, and (2) create virtual networks for guests and connect the to the world over (1). Functionality (2) is provided by building Linux software bridges, and vlan devices. I'd like to explore moving it to Open vSwitch, which would enable a host of functionalities that we currently lack (e.g. tunneling). One thing that worries me is the need to reimplement our config snapshot/recovery on ovs's database. As far as I know, ovs is unable to maintain host level parameters of interfaces (e.g. eth0's IPv4 address), so we need another tool for functionality (1): either speak to NetworkManager directly, or to use NetCF, via its libvirt virInterface* wrapper. I have minor worries about NetCF's breadth of testing and usage; I know it is intended to be cross-platform, but unlike ovs, I am not aware of a wide Debian usage thereof. On the other hand, its API is ready for vdsm's usage for quite a while. NetworkManager has become ubiquitous, and we'd better integrate with it better than our current setting of NM_CONTROLLED=no. But as DPB tells us, https://lists.fedorahosted.org/pipermail/vdsm-devel/2012-November/001677.html we'd better offload integration with NM to libvirt. We would like to take Network configuration in VDSM to the next level and make it distribution agnostic in addition for setting the infrastructure for more advanced features to be used going forward. The path we think of taking is to integrate with OVS and for feature completeness use NetCF, via its libvirt virInterface* wrapper. Any comments or feedback on this proposal is welcomed. Thanks to the oVirt net team members who's input has helped writing this email. Hi, As far as I see this, network manager is a monster that is a huge dependency to have just to create bridges or configure network interfaces... It is true that on a host where network manager lives it would be not polite to define network resources not via its interface, however I don't like we force network manager. NM is a default way of network configuration from F17 on and it's available on all platforms. It isn't exactly small but it wouldn't pull any dependency AFAICT because all its dependencies are on Fedora initramfs already... libvirt is long not used as virtualization library but system management agent, I am not sure this is the best system agent I would have chosen. I think that all the terms and building blocks got lost in time... and the result integration became more and more complex. Stabilizing such multi-layered component environment is much harder than monolithic environment. I would really want to see vdsm as monolithic component with full
Re: [vdsm] Future of Vdsm network configuration
- Original Message - From: Alon Bar-Lev alo...@redhat.com To: Dan Kenigsberg dan...@redhat.com Cc: vdsm-de...@fedorahosted.org Sent: Sunday, November 11, 2012 3:46:43 PM Subject: Re: [vdsm] Future of Vdsm network configuration - Original Message - From: Dan Kenigsberg dan...@redhat.com To: vdsm-de...@fedorahosted.org Sent: Sunday, November 11, 2012 4:07:30 PM Subject: [vdsm] Future of Vdsm network configuration Hi, Nowadays, when vdsm receives the setupNetowrk verb, it mangles /etc/sysconfig/network-scripts/ifcfg-* files and restarts the network service, so they are read by the responsible SysV service. This is very much Fedora-oriented, and not up with the new themes in Linux network configuration. Since we want oVirt and Vdsm to be distribution agnostic, and support new features, we have to change. setupNetwork is responsible for two different things: (1) configure the host networking interfaces, and (2) create virtual networks for guests and connect the to the world over (1). Functionality (2) is provided by building Linux software bridges, and vlan devices. I'd like to explore moving it to Open vSwitch, which would enable a host of functionalities that we currently lack (e.g. tunneling). One thing that worries me is the need to reimplement our config snapshot/recovery on ovs's database. As far as I know, ovs is unable to maintain host level parameters of interfaces (e.g. eth0's IPv4 address), so we need another tool for functionality (1): either speak to NetworkManager directly, or to use NetCF, via its libvirt virInterface* wrapper. I have minor worries about NetCF's breadth of testing and usage; I know it is intended to be cross-platform, but unlike ovs, I am not aware of a wide Debian usage thereof. On the other hand, its API is ready for vdsm's usage for quite a while. NetworkManager has become ubiquitous, and we'd better integrate with it better than our current setting of NM_CONTROLLED=no. But as DPB tells us, https://lists.fedorahosted.org/pipermail/vdsm-devel/2012-November/001677.html we'd better offload integration with NM to libvirt. We would like to take Network configuration in VDSM to the next level and make it distribution agnostic in addition for setting the infrastructure for more advanced features to be used going forward. The path we think of taking is to integrate with OVS and for feature completeness use NetCF, via its libvirt virInterface* wrapper. Any comments or feedback on this proposal is welcomed. Thanks to the oVirt net team members who's input has helped writing this email. Hi, As far as I see this, network manager is a monster that is a huge dependency to have just to create bridges or configure network interfaces... It is true that on a host where network manager lives it would be not polite to define network resources not via its interface, however I don't like we force network manager. libvirt is long not used as virtualization library but system management agent, I am not sure this is the best system agent I would have chosen. I think that all the terms and building blocks got lost in time... and the result integration became more and more complex. Stabilizing such multi-layered component environment is much harder than monolithic environment. I would really want to see vdsm as monolithic component with full control over its resources, I believe this is the only way vdsm can be stable enough to be production grade. Hypervisor should be a total slave of manager (or cluster), so I have no problem in bypassing/disabling any distribution specific tool in favour of atoms (brctl, iproute), in non persistence mode. So you propose that we would keep the network configuration database ourselves (something like sqlite maybe), disable network.service and networkmanager.service and put up and down the interfaces we need via brctl/iproute, sysfs and other netlink talking interfaces right? I won't deny that for hypervisor nodes it sounds really well. For installations on machines that maybe serve other purposes as well, it could be slightly problematic. Not the part of managing the network, but the part of disabling network manager and network.service. Since what you said was bypass NM and network.service, maybe it would be better instead to leave whichever is default enabled and let the user define which interfaces we should manage, and make those unavailable to NM and network.service. Thre are four cases here: NM enabled network.service disabled: Simply create ifcfg-* for the interfaces that we want to manage that include NM_CONTROLLED=no and the MAC address of the interface. NM disabled and network.service disabled: Just make sure that the interfaces we are to manage do not have a ifcfg-* file. NM disabled and network.service disabled: No special
