Haha, you're not wrong, definitely a bit of a unique scenario, but we have a
specific use-case we're exploring.
This one is back to running 2.5.4 as I did not see any change in symptoms up to
2.9.1. Also, because 2.5.4 is labeled as the current LTS release (although I
do see 2.5.5 was just made available this week).
I've added the requested outputs as well consolidated some others to hopefully
help paint the full picture in one grouping. Please let me know if there are
any other questions.
ovs-vsctl show;
38955fc8-079e-46bc-a27b-633c192f795b
Bridge "wan1"
Port "wan1"
Interface "wan1"
type: internal
Port "ens3"
Interface "ens3"
Bridge br-int
Controller "tcp:40.138.35.150:6653"
is_connected: true
Port "test-tun-2"
Interface "test-tun-2"
type: vxlan
options: {key="102", local_ip="198.15.1.20",
remote_ip="40.138.35.166"}
Port "test-tun-1"
Interface "test-tun-1"
type: vxlan
options: {key="101", local_ip="198.15.0.22",
remote_ip="40.138.35.166"}
Port br-int
Interface br-int
type: internal
Bridge "wan2"
Port "wan2"
Interface "wan2"
type: internal
Port "ens4"
Interface "ens4"
ovs_version: "2.5.4"
ip addr show;
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group
default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master
ovs-system state UP group default qlen 1000
link/ether fa:16:3e:68:f2:db brd ff:ff:ff:ff:ff:ff
inet6 fe80::f816:3eff:fe68:f2db/64 scope link
valid_lft forever preferred_lft forever
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master
ovs-system state UP group default qlen 1000
link/ether fa:16:3e:00:72:72 brd ff:ff:ff:ff:ff:ff
inet6 fe80::f816:3eff:fe00:7272/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state
DOWN group default
link/ether 02:42:af:bc:d4:8f brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 scope global docker0
valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1
link/ether 1e:ff:6b:19:59:c1 brd ff:ff:ff:ff:ff:ff
6: br-int: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state
UNKNOWN group default qlen 1
link/ether 7a:e4:bf:b8:75:40 brd ff:ff:ff:ff:ff:ff
inet6 fe80::78e4:bfff:feb8:7540/64 scope link
valid_lft forever preferred_lft forever
10: wan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state
UNKNOWN group default qlen 1
link/ether fa:16:3e:68:f2:db brd ff:ff:ff:ff:ff:ff
inet 198.15.0.22/24 brd 198.15.0.255 scope global wan1
valid_lft forever preferred_lft forever
inet6 fe80::8065:7aff:fe92:e543/64 scope link
valid_lft forever preferred_lft forever
11: wan2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state
UNKNOWN group default qlen 1
link/ether fa:16:3e:00:72:72 brd ff:ff:ff:ff:ff:ff
inet 198.15.1.20/24 brd 198.15.1.255 scope global wan2
valid_lft forever preferred_lft forever
inet6 fe80::3c2b:eff:fecb:1047/64 scope link
valid_lft forever preferred_lft forever
12: vxlan_sys_4789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65485 qdisc noqueue
master ovs-system state UNKNOWN group default qlen 1000
link/ether d2:59:ff:d9:7c:af brd ff:ff:ff:ff:ff:ff
inet6 fe80::d059:ffff:fed9:7caf/64 scope link
valid_lft forever preferred_lft forever
ip route show;
default via 198.15.0.1 dev wan1 metric 10
default via 198.15.1.1 dev wan2 metric 40
169.254.169.254 via 198.15.0.1 dev wan1 metric 10
169.254.169.254 via 198.15.1.1 dev wan2 metric 40
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
198.15.0.0/24 dev ens3 proto kernel scope link src 198.15.0.22
198.15.0.0/24 dev wan1 proto kernel scope link src 198.15.0.22
198.15.1.0/24 dev wan2 proto kernel scope link src 198.15.1.20
ip route show table wan1;
default via 198.15.0.1 dev wan1 src 198.15.0.22
ip route show table wan2;
default via 198.15.1.1 dev wan2 src 198.15.1.20
iptables -L -n -t mangle;
iptables -t mangle -A OUTPUT -p udp --dport 4789 -s 198.15.0.22 -j MARK
--set-mark 0xa1
iptables -t mangle -A OUTPUT -p udp --dport 4789 -s 198.15.1.20 -j MARK
--set-mark 0xa2
ip rule show;
0: from all fwmark 0xa1 lookup wan1
0: from all fwmark 0xa2 lookup wan2
10: from all lookup local
32766: from all lookup main
32767: from all lookup default
Here's a diagram if it helps as well;
[cid:[email protected]]
All interfaces are behind 1-to-1 NAT, and are translated as such;
WAN1 = 198.15.0.1 = 40.138.35.146
WAN2 = 198.15.1.1 = 40.138.35.155
Data Center side is the 40.138.35.166
Thank you,
Marcus Padgett
Sr Engineer - Service Architecture | Windstream
From: Gregory Rose [mailto:[email protected]]
Sent: Wednesday, May 23, 2018 7:06 PM
To: Padgett, Marcus <[email protected]>; [email protected]
Subject: Re: [ovs-discuss] OVS VXLAN local_ip does not seem to function - Bug
On 5/21/2018 4:42 AM, Padgett, Marcus wrote:
My responses are inline.
Thank you,
Marcus Padgett
Sr Engineer - Service Architecture | Windstream
From: Gregory Rose [mailto:[email protected]]
Sent: Friday, May 18, 2018 5:55 PM
To: Padgett, Marcus
<[email protected]><mailto:[email protected]>;
[email protected]<mailto:[email protected]>
Subject: Re: [ovs-discuss] OVS VXLAN local_ip does not seem to function - Bug
Hi Marcus,
I have a couple questions inline below...
On 4/26/2018 6:17 AM, Padgett, Marcus wrote:
I have an issue with OVS VXLAN and using the local_ip option for the interface.
Some of the IP addresses have been changed in the email since I don't know
exactly who sees this.
What you did that make the problem appear.
Using a server with two interfaces with two default routes, I want to build a
VXLAN tunnel over each link to a destination switch.
They both go the same switch? Is it a real switch or a SW switch like OVS?
[MP] - They do both go to the same switch (single interface). Could be either
hardware or SW switch long term, but right now I'm terminating them to another
OVS bridge.
I am utilizing iptables to mark and ip rules to re-direct the traffic for the
second tunnel out the correct interface. I have built the VXLAN tunnel in OVS
trying to use the "local_ip" option.
Please provide the iptables and ip rules you used to redirect the traffic. [MP]
- Iptables and ip rules below;
ip route add default via $WAN1_GW dev wan1 src $WAN1_IP table wan1
ip route add default via $WAN2_GW dev wan2 src $WAN2_IP table wan2
iptables -t mangle -A OUTPUT -p udp --dport 4789 -s $WAN1_IP -j MARK --set-mark
0xa1
iptables -t mangle -A OUTPUT -p udp --dport 4789 -s $WAN2_IP -j MARK --set-mark
0xa2
ip rule add table local priority 10
ip rule del table local priority 0
ip rule add fwmark 0xa1 table wan1 priority 0
ip rule add fwmark 0xa2 table wan2 priority 0
So, I'm trying to capture the packet as it's generated by the process and add a
MARK, prior to any routing decisions being made (OUTPUT chain in MANGLE table).
What I'm seeing (and my assumption) is that the VXLAN packet is generating its
source IP address based on the local or main routing table while still being
handled by some OVS process. I can use the same setup with GRE (change the
protocol type and remote destination port) and it works as expected.
Wow. OK, it's quite the setup and not something simple to configure and get
going in an hour or two. I'm working on it... but let me ask something else.
What is the output of 'ovs-vsctl show' and 'ip addr show' and 'ip route show'?
Please substitute the IP addresses shown with the $WAN1_IP and $WAN1_GW
strings so that I can associate interfaces to addresses without you having to
publish the actual IP address values.
Thanks,
- Greg
I set something up similar to this situation and didn't see a problem but I
wasn't using a real switch.
Also, did you ever have a chance to try out a new version of OVS? As Ben
mentioned 2.5.2 is pretty old. [MP] - Yes. I was able to try this in 2.5.2,
2.5.4, 2.6.1, and 2.9.1. All seemed to have the same issue.
Thanks,
- Greg
EXAMPLE:
sudo ovs-vsctl add-port ovs-br1 tun1 -- set Interface tun1 type=vxlan
options:remote_ip=172.17.253.1 options:key=testflow2
options:local_ip=172.16.253.16
What you expected to happen.
I expected the VXLAN tunnel to be sent using the specified source
IP address of 172.16.253.16 out the correct interface.
What actually happened.
The VXLAN tunnel exits the correct interface based on my routing
rules, however was formed utilizing the other interface's IP address
(172.16.252.108).
EXAMPLE:
08:24:42.779236 IP 172.16.252.108.57418 > 172.17.253.1.4789: VXLAN, flags [I]
(0x08), vni 0
LLDP, length 79
08:24:42.779437 IP 172.16.252.108.34566 > 172.17.253.1.4789: VXLAN, flags [I]
(0x08), vni 0
02:eb:86:0d:38:74 (oui Unknown) > Broadcast, ethertype Unknown (0x8942), length
93:
0x0000: 0207 0486 e5fb d3da 4204 0502 0000 000e ........B.......
0x0010: 0602 0078 fe12 a423 0501 4f4e 4f53 2044 ...x...#..ONOS.D
0x0020: 6973 636f 7665 7279 fe17 a423 0502 6f66 iscovery...#..of
0x0030: 3a30 3030 3038 3665 3566 6264 3364 6134 :000086e5fbd3da4
0x0040: 3208 0a75 6370 652d 7475 6e2d 3200 00 2..tun1..
The Open vSwitch version number (as output by ovs-vswitchd --version).
ovs-vswitchd (Open vSwitch) 2.5.2
Compiled Oct 17 2017 16:38:57
The kernel version on which Open vSwitch is running (from /proc/version) and
the distribution and version number of your OS (e.g. "Centos 5.0").
Linux version 4.4.0-87-generic (buildd@lcy01-31) (gcc version 5.4.0
20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) #110-Ubuntu SMP Tue Jul 18 12:55:35
UTC 2017
The output of ovs-dpctl show.
system@ovs-system:
lookups: hit:191881 missed:5814 lost:0
flows: 4
masks: hit:471168 total:4 hit/pkt:2.38
port 0: ovs-system (internal)
port 1: ovs-br1 (internal)
port 2: vxlan_sys_4789 (vxlan)
port 3: ovs-lan (internal)
port 4: k8s-br (internal)
port 5: mirror-br (internal)
port 6: ens6
port 7: wan2 (internal)
Any other information that you think might be relevant.
Everything works fine when building with GRE instead of VXLAN, all the same
routing and firewall rules. The rules are not matching any protocol specific
parameters, just matching on destination IP address to mark the traffic.
Thank you,
Marcus Padgett
Sr Engineer - Service Architecture | Windstream
[email protected]<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.windstreambusiness.com%2F&data=02%7C01%7CMarcus.Padgett%40windstream.com%7C32c54daf46044a07ab7008d5bd0a2310%7C2567b4c1b0ed40f5aee358d7c5f3e2b2%7C1%7C0%7C636622773597849997&sdata=%2BHevGor9WR8uzFbSuS%2FuALFmZM74B6nZ91u2Fl%2F%2Bd3k%3D&reserved=0>
This email message and any attachments are for the sole use of the intended
recipient(s). Any unauthorized review, use, disclosure or distribution is
prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message and any attachments.
_______________________________________________
discuss mailing list
[email protected]<mailto:[email protected]>
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.openvswitch.org%2Fmailman%2Flistinfo%2Fovs-discuss&data=02%7C01%7CMarcus.Padgett%40windstream.com%7C32c54daf46044a07ab7008d5bd0a2310%7C2567b4c1b0ed40f5aee358d7c5f3e2b2%7C1%7C0%7C636622773597860005&sdata=WsT%2BIoPQIOjKD%2FdjtKmyh3YKe7jqsuY6AqP4zl9J2G0%3D&reserved=0>
_______________________________________________
discuss mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
