Good idea. Looks fine to me. --Justin
On May 4, 2011, at 2:04 PM, Ben Pfaff wrote: > It seems more likely that interested users and administrators will be able > to find it here. > --- > DESIGN | 163 +++++++++++++++++++++++++++++++++++++++++++++++++++++ > ofproto/in-band.c | 160 ---------------------------------------------------- > 2 files changed, 163 insertions(+), 160 deletions(-) > > diff --git a/DESIGN b/DESIGN > index 6e25f01..2e3fced 100644 > --- a/DESIGN > +++ b/DESIGN > @@ -71,6 +71,169 @@ nodes that do not connect to link with such large MTUs. > Currently, Open > vSwitch doesn't process jumbograms. > > > +In-Band Control > +=============== > + > +In-band control allows a single network to be used for OpenFlow traffic and > +other data traffic. See ovs-vswitchd.conf.db(5) for a description of > +configuring in-band control. > + > +This comment is an attempt to describe how in-band control works at a > +wire- and implementation-level. Correctly implementing in-band > +control has proven difficult due to its many subtleties, and has thus > +gone through many iterations. Please read through and understand the > +reasoning behind the chosen rules before making modifications. > + > +In Open vSwitch, in-band control is implemented as "hidden" flows (in that > +they are not visible through OpenFlow) and at a higher priority than > +wildcarded flows can be set up by through OpenFlow. This is done so that > +the OpenFlow controller cannot interfere with them and possibly break > +connectivity with its switches. It is possible to see all flows, including > +in-band ones, with the ovs-appctl "bridge/dump-flows" command. > + > +The Open vSwitch implementation of in-band control can hide traffic to > +arbitrary "remotes", where each remote is one TCP port on one IP address. > +Currently the remotes are automatically configured as the in-band OpenFlow > +controllers plus the OVSDB managers, if any. (The latter is a requirement > +because OVSDB managers are responsible for configuring OpenFlow controllers, > +so if the manager cannot be reached then OpenFlow cannot be reconfigured.) > + > +The following rules (with the OFPP_NORMAL action) are set up on any bridge > +that has any remotes: > + > + (a) DHCP requests sent from the local port. > + (b) ARP replies to the local port's MAC address. > + (c) ARP requests from the local port's MAC address. > + > +In-band also sets up the following rules for each unique next-hop MAC > +address for the remotes' IPs (the "next hop" is either the remote > +itself, if it is on a local subnet, or the gateway to reach the remote): > + > + (d) ARP replies to the next hop's MAC address. > + (e) ARP requests from the next hop's MAC address. > + > +In-band also sets up the following rules for each unique remote IP address: > + > + (f) ARP replies containing the remote's IP address as a target. > + (g) ARP requests containing the remote's IP address as a source. > + > +In-band also sets up the following rules for each unique remote (IP,port) > +pair: > + > + (h) TCP traffic to the remote's IP and port. > + (i) TCP traffic from the remote's IP and port. > + > +The goal of these rules is to be as narrow as possible to allow a > +switch to join a network and be able to communicate with the > +remotes. As mentioned earlier, these rules have higher priority > +than the controller's rules, so if they are too broad, they may > +prevent the controller from implementing its policy. As such, > +in-band actively monitors some aspects of flow and packet processing > +so that the rules can be made more precise. > + > +In-band control monitors attempts to add flows into the datapath that > +could interfere with its duties. The datapath only allows exact > +match entries, so in-band control is able to be very precise about > +the flows it prevents. Flows that miss in the datapath are sent to > +userspace to be processed, so preventing these flows from being > +cached in the "fast path" does not affect correctness. The only type > +of flow that is currently prevented is one that would prevent DHCP > +replies from being seen by the local port. For example, a rule that > +forwarded all DHCP traffic to the controller would not be allowed, > +but one that forwarded to all ports (including the local port) would. > + > +As mentioned earlier, packets that miss in the datapath are sent to > +the userspace for processing. The userspace has its own flow table, > +the "classifier", so in-band checks whether any special processing > +is needed before the classifier is consulted. If a packet is a DHCP > +response to a request from the local port, the packet is forwarded to > +the local port, regardless of the flow table. Note that this requires > +L7 processing of DHCP replies to determine whether the 'chaddr' field > +matches the MAC address of the local port. > + > +It is interesting to note that for an L3-based in-band control > +mechanism, the majority of rules are devoted to ARP traffic. At first > +glance, some of these rules appear redundant. However, each serves an > +important role. First, in order to determine the MAC address of the > +remote side (controller or gateway) for other ARP rules, we must allow > +ARP traffic for our local port with rules (b) and (c). If we are > +between a switch and its connection to the remote, we have to > +allow the other switch's ARP traffic to through. This is done with > +rules (d) and (e), since we do not know the addresses of the other > +switches a priori, but do know the remote's or gateway's. Finally, > +if the remote is running in a local guest VM that is not reached > +through the local port, the switch that is connected to the VM must > +allow ARP traffic based on the remote's IP address, since it will > +not know the MAC address of the local port that is sending the traffic > +or the MAC address of the remote in the guest VM. > + > +With a few notable exceptions below, in-band should work in most > +network setups. The following are considered "supported' in the > +current implementation: > + > + - Locally Connected. The switch and remote are on the same > + subnet. This uses rules (a), (b), (c), (h), and (i). > + > + - Reached through Gateway. The switch and remote are on > + different subnets and must go through a gateway. This uses > + rules (a), (b), (c), (h), and (i). > + > + - Between Switch and Remote. This switch is between another > + switch and the remote, and we want to allow the other > + switch's traffic through. This uses rules (d), (e), (h), and > + (i). It uses (b) and (c) indirectly in order to know the MAC > + address for rules (d) and (e). Note that DHCP for the other > + switch will not work unless an OpenFlow controller explicitly lets this > + switch pass the traffic. > + > + - Between Switch and Gateway. This switch is between another > + switch and the gateway, and we want to allow the other switch's > + traffic through. This uses the same rules and logic as the > + "Between Switch and Remote" configuration described earlier. > + > + - Remote on Local VM. The remote is a guest VM on the > + system running in-band control. This uses rules (a), (b), (c), > + (h), and (i). > + > + - Remote on Local VM with Different Networks. The remote > + is a guest VM on the system running in-band control, but the > + local port is not used to connect to the remote. For > + example, an IP address is configured on eth0 of the switch. The > + remote's VM is connected through eth1 of the switch, but an > + IP address has not been configured for that port on the switch. > + As such, the switch will use eth0 to connect to the remote, > + and eth1's rules about the local port will not work. In the > + example, the switch attached to eth0 would use rules (a), (b), > + (c), (h), and (i) on eth0. The switch attached to eth1 would use > + rules (f), (g), (h), and (i). > + > +The following are explicitly *not* supported by in-band control: > + > + - Specify Remote by Name. Currently, the remote must be > + identified by IP address. A naive approach would be to permit > + all DNS traffic. Unfortunately, this would prevent the > + controller from defining any policy over DNS. Since switches > + that are located behind us need to connect to the remote, > + in-band cannot simply add a rule that allows DNS traffic from > + the local port. The "correct" way to support this is to parse > + DNS requests to allow all traffic related to a request for the > + remote's name through. Due to the potential security > + problems and amount of processing, we decided to hold off for > + the time-being. > + > + - Differing Remotes for Switches. All switches must know > + the L3 addresses for all the remotes that other switches > + may use, since rules need to be set up to allow traffic related > + to those remotes through. See rules (f), (g), (h), and (i). > + > + - Differing Routes for Switches. In order for the switch to > + allow other switches to connect to a remote through a > + gateway, it allows the gateway's traffic through with rules (d) > + and (e). If the routes to the remote differ for the two > + switches, we will not know the MAC address of the alternate > + gateway. > + > + > Suggestions > =========== > > diff --git a/ofproto/in-band.c b/ofproto/in-band.c > index e75d19e..9aa03af 100644 > --- a/ofproto/in-band.c > +++ b/ofproto/in-band.c > @@ -40,166 +40,6 @@ > > VLOG_DEFINE_THIS_MODULE(in_band); > > -/* In-band control allows a single network to be used for OpenFlow traffic > and > - * other data traffic. See ovs-vswitchd.conf.db(5) for a description of > - * configuring in-band control. > - * > - * This comment is an attempt to describe how in-band control works at a > - * wire- and implementation-level. Correctly implementing in-band > - * control has proven difficult due to its many subtleties, and has thus > - * gone through many iterations. Please read through and understand the > - * reasoning behind the chosen rules before making modifications. > - * > - * In Open vSwitch, in-band control is implemented as "hidden" flows (in that > - * they are not visible through OpenFlow) and at a higher priority than > - * wildcarded flows can be set up by through OpenFlow. This is done so that > - * the OpenFlow controller cannot interfere with them and possibly break > - * connectivity with its switches. It is possible to see all flows, > including > - * in-band ones, with the ovs-appctl "bridge/dump-flows" command. > - * > - * The Open vSwitch implementation of in-band control can hide traffic to > - * arbitrary "remotes", where each remote is one TCP port on one IP address. > - * Currently the remotes are automatically configured as the in-band OpenFlow > - * controllers plus the OVSDB managers, if any. (The latter is a requirement > - * because OVSDB managers are responsible for configuring OpenFlow > controllers, > - * so if the manager cannot be reached then OpenFlow cannot be reconfigured.) > - * > - * The following rules (with the OFPP_NORMAL action) are set up on any bridge > - * that has any remotes: > - * > - * (a) DHCP requests sent from the local port. > - * (b) ARP replies to the local port's MAC address. > - * (c) ARP requests from the local port's MAC address. > - * > - * In-band also sets up the following rules for each unique next-hop MAC > - * address for the remotes' IPs (the "next hop" is either the remote > - * itself, if it is on a local subnet, or the gateway to reach the remote): > - * > - * (d) ARP replies to the next hop's MAC address. > - * (e) ARP requests from the next hop's MAC address. > - * > - * In-band also sets up the following rules for each unique remote IP > address: > - * > - * (f) ARP replies containing the remote's IP address as a target. > - * (g) ARP requests containing the remote's IP address as a source. > - * > - * In-band also sets up the following rules for each unique remote (IP,port) > - * pair: > - * > - * (h) TCP traffic to the remote's IP and port. > - * (i) TCP traffic from the remote's IP and port. > - * > - * The goal of these rules is to be as narrow as possible to allow a > - * switch to join a network and be able to communicate with the > - * remotes. As mentioned earlier, these rules have higher priority > - * than the controller's rules, so if they are too broad, they may > - * prevent the controller from implementing its policy. As such, > - * in-band actively monitors some aspects of flow and packet processing > - * so that the rules can be made more precise. > - * > - * In-band control monitors attempts to add flows into the datapath that > - * could interfere with its duties. The datapath only allows exact > - * match entries, so in-band control is able to be very precise about > - * the flows it prevents. Flows that miss in the datapath are sent to > - * userspace to be processed, so preventing these flows from being > - * cached in the "fast path" does not affect correctness. The only type > - * of flow that is currently prevented is one that would prevent DHCP > - * replies from being seen by the local port. For example, a rule that > - * forwarded all DHCP traffic to the controller would not be allowed, > - * but one that forwarded to all ports (including the local port) would. > - * > - * As mentioned earlier, packets that miss in the datapath are sent to > - * the userspace for processing. The userspace has its own flow table, > - * the "classifier", so in-band checks whether any special processing > - * is needed before the classifier is consulted. If a packet is a DHCP > - * response to a request from the local port, the packet is forwarded to > - * the local port, regardless of the flow table. Note that this requires > - * L7 processing of DHCP replies to determine whether the 'chaddr' field > - * matches the MAC address of the local port. > - * > - * It is interesting to note that for an L3-based in-band control > - * mechanism, the majority of rules are devoted to ARP traffic. At first > - * glance, some of these rules appear redundant. However, each serves an > - * important role. First, in order to determine the MAC address of the > - * remote side (controller or gateway) for other ARP rules, we must allow > - * ARP traffic for our local port with rules (b) and (c). If we are > - * between a switch and its connection to the remote, we have to > - * allow the other switch's ARP traffic to through. This is done with > - * rules (d) and (e), since we do not know the addresses of the other > - * switches a priori, but do know the remote's or gateway's. Finally, > - * if the remote is running in a local guest VM that is not reached > - * through the local port, the switch that is connected to the VM must > - * allow ARP traffic based on the remote's IP address, since it will > - * not know the MAC address of the local port that is sending the traffic > - * or the MAC address of the remote in the guest VM. > - * > - * With a few notable exceptions below, in-band should work in most > - * network setups. The following are considered "supported' in the > - * current implementation: > - * > - * - Locally Connected. The switch and remote are on the same > - * subnet. This uses rules (a), (b), (c), (h), and (i). > - * > - * - Reached through Gateway. The switch and remote are on > - * different subnets and must go through a gateway. This uses > - * rules (a), (b), (c), (h), and (i). > - * > - * - Between Switch and Remote. This switch is between another > - * switch and the remote, and we want to allow the other > - * switch's traffic through. This uses rules (d), (e), (h), and > - * (i). It uses (b) and (c) indirectly in order to know the MAC > - * address for rules (d) and (e). Note that DHCP for the other > - * switch will not work unless an OpenFlow controller explicitly lets > this > - * switch pass the traffic. > - * > - * - Between Switch and Gateway. This switch is between another > - * switch and the gateway, and we want to allow the other switch's > - * traffic through. This uses the same rules and logic as the > - * "Between Switch and Remote" configuration described earlier. > - * > - * - Remote on Local VM. The remote is a guest VM on the > - * system running in-band control. This uses rules (a), (b), (c), > - * (h), and (i). > - * > - * - Remote on Local VM with Different Networks. The remote > - * is a guest VM on the system running in-band control, but the > - * local port is not used to connect to the remote. For > - * example, an IP address is configured on eth0 of the switch. The > - * remote's VM is connected through eth1 of the switch, but an > - * IP address has not been configured for that port on the switch. > - * As such, the switch will use eth0 to connect to the remote, > - * and eth1's rules about the local port will not work. In the > - * example, the switch attached to eth0 would use rules (a), (b), > - * (c), (h), and (i) on eth0. The switch attached to eth1 would use > - * rules (f), (g), (h), and (i). > - * > - * The following are explicitly *not* supported by in-band control: > - * > - * - Specify Remote by Name. Currently, the remote must be > - * identified by IP address. A naive approach would be to permit > - * all DNS traffic. Unfortunately, this would prevent the > - * controller from defining any policy over DNS. Since switches > - * that are located behind us need to connect to the remote, > - * in-band cannot simply add a rule that allows DNS traffic from > - * the local port. The "correct" way to support this is to parse > - * DNS requests to allow all traffic related to a request for the > - * remote's name through. Due to the potential security > - * problems and amount of processing, we decided to hold off for > - * the time-being. > - * > - * - Differing Remotes for Switches. All switches must know > - * the L3 addresses for all the remotes that other switches > - * may use, since rules need to be set up to allow traffic related > - * to those remotes through. See rules (f), (g), (h), and (i). > - * > - * - Differing Routes for Switches. In order for the switch to > - * allow other switches to connect to a remote through a > - * gateway, it allows the gateway's traffic through with rules (d) > - * and (e). If the routes to the remote differ for the two > - * switches, we will not know the MAC address of the alternate > - * gateway. > - */ > - > /* Priorities used in classifier for in-band rules. These values are higher > * than any that may be set with OpenFlow, and "18" kind of looks like "IB". > * The ordering of priorities is not important because all of the rules set up > -- > 1.7.4.4 > > _______________________________________________ > dev mailing list > [email protected] > http://openvswitch.org/mailman/listinfo/dev _______________________________________________ dev mailing list [email protected] http://openvswitch.org/mailman/listinfo/dev
