Re: [openstack-dev] [neutron][taas] On TaaS not capturing some packets between two VMs on the same host
Hi Takashi, Yes, this update was for bug 1529595. Thanks, Anil -Original Message- From: Takashi Yamamoto [mailto:yamam...@midokura.com] Sent: Monday, September 05, 2016 11:29 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron][taas] On TaaS not capturing some packets between two VMs on the same host hi, On Sat, Sep 3, 2016 at 7:32 AM, Anil Rao wrote: > Hello, > > Here is an update on the issue of not being able to capture packets > flowing between two VMs on the same host. it it about bug 1529595 ? https://bugs.launchpad.net/tap-as-a-service/+bug/1529595 > > Isolating the problem: > > It has been confirmed that the TaaS OVS driver cannot capture packets > ingressing (flowing into) a VM’s vNIC under the following conditions: > > - The sender port and the receiver port (which is being monitored) > reside on the same host. > > - The sender port and the receiver port are attached to the same > (virtual) network. > > We do not have a problem in the above situation for traffic egressing > (flowing out) of the VM’s vNIC. We also do not have a problem if the > sender port and receiver port are on different hosts, or if the sender > and receiver ports belong to different (virtual) networks. > > Trafffic Capture Technique: > > The implementation uses the following methods to capture packets > flowing into and out of a VM’s vNIC. > > Egress Side (flowing out of the vNIC): > > The ‘in_port=’ check is used to identify packets coming > in from the port associated with the VM’s vNIC. This allows us to > capture all packets flowing out of the vNIC under question. No problems here. > > Ingress Side (flowing into the vNIC): > > The ‘dl_vlan=, dl_dst=’ check is used > to identify packets destined for the port associated with the VM’s vNIC. > > The VLAN tag check was put in place because Neutron port MAC addresses > are required to be unique only within a (virtual) network. Now each > instance port in br-int gets tagged with a VLAN id that is the same > for all ports on that host, which are associated with a particular > (virtual) network. Ports belonging to different (virtual) networks > therefore get tagged with different VLAN ids. > > The combination of “VLAN tag + destination MAC address” was felt to be > a sufficient check to ensure that we are truly dealing with traffic > heading out of the port being monitored. > > Root Cause of the problem: > > The br-int bridge operates using the legacy (or normal) mode of operation. > We have observed that under these circumstances, OVS does not add VLAN > tags to packets, as long as they are flowing within the bridge. > Packets escaping from br-int into br-tun (via the patch ports), > however, get tagged as expected when they arrive in br-tun. > > Due to this behavior, our ingress side flow (described above) fails to > capture packets flowing into a VM’s vNIC, when it is originating from > another port on the same host and on the same virtual network. > > Possible Solutions: i think the it can be fixed by having rules to "tag" packets explicitly. ie. match on inport and record the logical network in openflow metadata. (or ovs registers, which is basically the same thing) as it's planned to be done for ovs-agent extensions: https://review.openstack.org/#/c/320439/ > > OVS documentation (refer: Open vSwitch FAQ) seems to recommend not > using the normal operating mode, but handling VLANs explicitly, if > flows related to VLAN ids are going to be used. As mentioned above, > to ensure that we correctly handle the Neutron requirement that port > MAC addresses are unique only within a virtual network, we need to add > VLAN related checks to the TaaS flows. However, br-int today relies on > the normal operation mode for most of its work. > > This has left us now in a quandary situation. Here is a proposal to > move us > forward: > > 1. As a temporary fix, we can convert the TaaS ingress side flow in > br-int from > > dl_vlan=, dl_dst= > > to > > dl_dst= > > This will obviously mean that we no longer support the notion that > Neutron port MAC addresses are unique only within a virtual network. > However, given that the probability of two port MAC addresses on a > host being the same is low, this should suffice for the short term. i agree this is the easiest workaround. > > [We will also disable the flows used to capture broadcast/multicast > traffic flowing into a VM’s vNIC] > > 2. Implement VLAN handling explicitly in br-int and thereby move away > from the normal operating mode. This will be a core Neutron change and > not limited to just TaaS. However,
Re: [openstack-dev] [neutron][taas] On TaaS not capturing some packets between two VMs on the same host
hi, On Sat, Sep 3, 2016 at 7:32 AM, Anil Rao wrote: > Hello, > > Here is an update on the issue of not being able to capture packets flowing > between two VMs on the same host. it it about bug 1529595 ? https://bugs.launchpad.net/tap-as-a-service/+bug/1529595 > > Isolating the problem: > > It has been confirmed that the TaaS OVS driver cannot capture packets > ingressing (flowing into) a VM’s vNIC under the following conditions: > > - The sender port and the receiver port (which is being monitored) reside on > the same host. > > - The sender port and the receiver port are attached to the same (virtual) > network. > > We do not have a problem in the above situation for traffic egressing > (flowing out) of the VM’s vNIC. We also do not have a problem if the sender > port and receiver port are on different hosts, or if the sender and receiver > ports belong to different (virtual) networks. > > Trafffic Capture Technique: > > The implementation uses the following methods to capture packets flowing > into and out of a VM’s vNIC. > > Egress Side (flowing out of the vNIC): > > The ‘in_port=’ check is used to identify packets coming in from > the port associated with the VM’s vNIC. This allows us to capture all > packets flowing out of the vNIC under question. No problems here. > > Ingress Side (flowing into the vNIC): > > The ‘dl_vlan=, dl_dst=’ check is used to > identify packets destined for the port associated with the VM’s vNIC. > > The VLAN tag check was put in place because Neutron port MAC addresses are > required to be unique only within a (virtual) network. Now each instance > port in br-int gets tagged with a VLAN id that is the same for all ports on > that host, which are associated with a particular (virtual) network. Ports > belonging to different (virtual) networks therefore get tagged with > different VLAN ids. > > The combination of “VLAN tag + destination MAC address” was felt to be a > sufficient check to ensure that we are truly dealing with traffic heading > out of the port being monitored. > > Root Cause of the problem: > > The br-int bridge operates using the legacy (or normal) mode of operation. > We have observed that under these circumstances, OVS does not add VLAN tags > to packets, as long as they are flowing within the bridge. Packets escaping > from br-int into br-tun (via the patch ports), however, get tagged as > expected when they arrive in br-tun. > > Due to this behavior, our ingress side flow (described above) fails to > capture packets flowing into a VM’s vNIC, when it is originating from > another port on the same host and on the same virtual network. > > Possible Solutions: i think the it can be fixed by having rules to "tag" packets explicitly. ie. match on inport and record the logical network in openflow metadata. (or ovs registers, which is basically the same thing) as it's planned to be done for ovs-agent extensions: https://review.openstack.org/#/c/320439/ > > OVS documentation (refer: Open vSwitch FAQ) seems to recommend not using the > normal operating mode, but handling VLANs explicitly, if flows related to > VLAN ids are going to be used. As mentioned above, to ensure that we > correctly handle the Neutron requirement that port MAC addresses are unique > only within a virtual network, we need to add VLAN related checks to the > TaaS flows. However, br-int today relies on the normal operation mode for > most of its work. > > This has left us now in a quandary situation. Here is a proposal to move us > forward: > > 1. As a temporary fix, we can convert the TaaS ingress side flow in > br-int from > > dl_vlan=, dl_dst= > > to > > dl_dst= > > This will obviously mean that we no longer support the notion that Neutron > port MAC addresses are unique only within a virtual network. However, given > that the probability of two port MAC addresses on a host being the same is > low, this should suffice for the short term. i agree this is the easiest workaround. > > [We will also disable the flows used to capture broadcast/multicast traffic > flowing into a VM’s vNIC] > > 2. Implement VLAN handling explicitly in br-int and thereby move away > from the normal operating mode. This will be a core Neutron change and not > limited to just TaaS. However, I have the feeling that there will be other > projects (outside of TaaS) that will sooner or later have a need to detect > traffic flowing into a vNIC and they will also run into the same issue we > are presently facing. i'm all for moving away from port based "internal" vlans for longer term. > > > > 3. Go back to the (current) TaaS ingress side flow > (dl_vlan=, dl_dst=) when (2) is > implemented. We will then be able to once again support the Neutron > requirement that port MAC addresses need be unique only within a (virtual) > network. > > > > [We will re-enable the flows used to capture broadcast/multicast traffic > flowing into a VM’s vNIC] > > > > Thoughts /comments are welcome. > > T
[openstack-dev] [neutron][taas] On TaaS not capturing some packets between two VMs on the same host
Hello, Here is an update on the issue of not being able to capture packets flowing between two VMs on the same host. Isolating the problem: It has been confirmed that the TaaS OVS driver cannot capture packets ingressing (flowing into) a VM's vNIC under the following conditions: - The sender port and the receiver port (which is being monitored) reside on the same host. - The sender port and the receiver port are attached to the same (virtual) network. We do not have a problem in the above situation for traffic egressing (flowing out) of the VM's vNIC. We also do not have a problem if the sender port and receiver port are on different hosts, or if the sender and receiver ports belong to different (virtual) networks. Trafffic Capture Technique: The implementation uses the following methods to capture packets flowing into and out of a VM's vNIC. Egress Side (flowing out of the vNIC): The 'in_port=' check is used to identify packets coming in from the port associated with the VM's vNIC. This allows us to capture all packets flowing out of the vNIC under question. No problems here. Ingress Side (flowing into the vNIC): The 'dl_vlan=, dl_dst=' check is used to identify packets destined for the port associated with the VM's vNIC. The VLAN tag check was put in place because Neutron port MAC addresses are required to be unique only within a (virtual) network. Now each instance port in br-int gets tagged with a VLAN id that is the same for all ports on that host, which are associated with a particular (virtual) network. Ports belonging to different (virtual) networks therefore get tagged with different VLAN ids. The combination of "VLAN tag + destination MAC address" was felt to be a sufficient check to ensure that we are truly dealing with traffic heading out of the port being monitored. Root Cause of the problem: The br-int bridge operates using the legacy (or normal) mode of operation. We have observed that under these circumstances, OVS does not add VLAN tags to packets, as long as they are flowing within the bridge. Packets escaping from br-int into br-tun (via the patch ports), however, get tagged as expected when they arrive in br-tun. Due to this behavior, our ingress side flow (described above) fails to capture packets flowing into a VM's vNIC, when it is originating from another port on the same host and on the same virtual network. Possible Solutions: OVS documentation (refer: Open vSwitch FAQ) seems to recommend not using the normal operating mode, but handling VLANs explicitly, if flows related to VLAN ids are going to be used. As mentioned above, to ensure that we correctly handle the Neutron requirement that port MAC addresses are unique only within a virtual network, we need to add VLAN related checks to the TaaS flows. However, br-int today relies on the normal operation mode for most of its work. This has left us now in a quandary situation. Here is a proposal to move us forward: 1. As a temporary fix, we can convert the TaaS ingress side flow in br-int from dl_vlan=, dl_dst= to dl_dst= This will obviously mean that we no longer support the notion that Neutron port MAC addresses are unique only within a virtual network. However, given that the probability of two port MAC addresses on a host being the same is low, this should suffice for the short term. [We will also disable the flows used to capture broadcast/multicast traffic flowing into a VM's vNIC] 2. Implement VLAN handling explicitly in br-int and thereby move away from the normal operating mode. This will be a core Neutron change and not limited to just TaaS. However, I have the feeling that there will be other projects (outside of TaaS) that will sooner or later have a need to detect traffic flowing into a vNIC and they will also run into the same issue we are presently facing. 3. Go back to the (current) TaaS ingress side flow (dl_vlan=, dl_dst=) when (2) is implemented. We will then be able to once again support the Neutron requirement that port MAC addresses need be unique only within a (virtual) network. [We will re-enable the flows used to capture broadcast/multicast traffic flowing into a VM's vNIC] Thoughts /comments are welcome. Thanks and regards, Anil __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
