[Bug 1967856] Re: Hairpin traffic does not work with centralized NAT gw
** Changed in: ovn (Ubuntu) Importance: High => Undecided -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1967856 Title: Hairpin traffic does not work with centralized NAT gw To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1967856/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1967856] Re: Hairpin traffic does not work with centralized NAT gw
** Changed in: ovn (Ubuntu) Status: Triaged => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1967856 Title: Hairpin traffic does not work with centralized NAT gw To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1967856/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1967856] Re: Hairpin traffic does not work with centralized NAT gw
** Also affects: openvswitch (Ubuntu) Importance: Undecided Status: New ** Changed in: openvswitch (Ubuntu) Status: New => Triaged ** Changed in: openvswitch (Ubuntu) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1967856 Title: Hairpin traffic does not work with centralized NAT gw To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1967856/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1967856] Re: Hairpin traffic does not work with centralized NAT gw
A possible fix is being discussed in [7]. 7: https://mail.openvswitch.org/pipermail/ovs-dev/2022-May/393981.html -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1967856 Title: Hairpin traffic does not work with centralized NAT gw To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1967856] Re: Hairpin traffic does not work with centralized NAT gw
A update on some findings. If we either revert OVS commit [4], OR change a open vswitch kernel data path function [5] to always return 'false' (credits to Numan), the problem goes away. This also appears to be a root of a different issue previously reported to the ovs-discuss list [6]. 4: https://github.com/openvswitch/ovs/commit/355fef6f2 5: https://elixir.bootlin.com/linux/latest/source/net/openvswitch/conntrack.c#L683 6: https://mail.openvswitch.org/pipermail/ovs-discuss/2022-March/051771.html -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1967856 Title: Hairpin traffic does not work with centralized NAT gw To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1967856] Re: Hairpin traffic does not work with centralized NAT gw
The current line of thought is that the change in OVN has uncovered a conntrack related bug in either OVS, the OVS kernel datapath or kernel CT in general ref [3]. 3: https://mail.openvswitch.org/pipermail/ovs-dev/2022-April/393426.html -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1967856 Title: Hairpin traffic does not work with centralized NAT gw To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1967856] Re: Hairpin traffic does not work with centralized NAT gw
** Patch added: "test-synthesis.patch" https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+attachment/5579267/+files/test-synthesis.patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1967856 Title: Hairpin traffic does not work with centralized NAT gw To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1967856] Re: Hairpin traffic does not work with centralized NAT gw
Updated OVN to main and it unfortunately made no difference. The combination of stateless on the NAT rule and the allow-related ACLs does indeed look strange, but this is how OpenStack sets it up. Have not looked into whether that makes sense or not yet. To ensure we're looking at the same thing I made this modification to the `DNAT LR hairpin IPv4` system test [2] And executed it like this: sudo make check-kernel TESTSUITEFLAGS="337" It fails consistently here. If I either revert [1] or remove the check for the second ping from the test it succeeds. 2: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+attachment/5579267/+files/test-synthesis.patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1967856 Title: Hairpin traffic does not work with centralized NAT gw To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1967856] Re: Hairpin traffic does not work with centralized NAT gw
It works fine for me - [root@ovn-chassis-1 data]# ip netns exec vm1 ping 10.78.95.196 PING 10.78.95.196 (10.78.95.196) 56(84) bytes of data. 64 bytes from 10.78.95.196: icmp_seq=1 ttl=62 time=1.18 ms 64 bytes from 10.78.95.196: icmp_seq=2 ttl=62 time=0.651 ms 64 bytes from 10.78.95.196: icmp_seq=3 ttl=62 time=0.102 ms 64 bytes from 10.78.95.196: icmp_seq=4 ttl=62 time=0.141 ms ^C --- 10.78.95.196 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3044ms rtt min/avg/max/mdev = 0.102/0.518/1.179/0.438 ms [root@ovn-chassis-1 data]# [root@ovn-chassis-1 data]# [root@ovn-chassis-1 data]# ip netns exec vm1 ping 10.78.95.196 PING 10.78.95.196 (10.78.95.196) 56(84) bytes of data. 64 bytes from 10.78.95.196: icmp_seq=1 ttl=62 time=0.113 ms 64 bytes from 10.78.95.196: icmp_seq=2 ttl=62 time=0.339 ms 64 bytes from 10.78.95.196: icmp_seq=3 ttl=62 time=0.242 ms 64 bytes from 10.78.95.196: icmp_seq=4 ttl=62 time=0.110 ms 64 bytes from 10.78.95.196: icmp_seq=5 ttl=62 time=0.251 ms 64 bytes from 10.78.95.196: icmp_seq=6 ttl=62 time=0.213 ms 64 bytes from 10.78.95.196: icmp_seq=7 ttl=62 time=0.260 ms 64 bytes from 10.78.95.196: icmp_seq=8 ttl=62 time=0.258 ms 64 bytes from 10.78.95.196: icmp_seq=9 ttl=62 time=0.259 ms 64 bytes from 10.78.95.196: icmp_seq=10 ttl=62 time=0.257 ms 64 bytes from 10.78.95.196: icmp_seq=11 ttl=62 time=0.264 ms 64 bytes from 10.78.95.196: icmp_seq=12 ttl=62 time=0.258 ms 64 bytes from 10.78.95.196: icmp_seq=13 ttl=62 time=0.311 ms 64 bytes from 10.78.95.196: icmp_seq=14 ttl=62 time=0.257 ms 64 bytes from 10.78.95.196: icmp_seq=15 ttl=62 time=0.264 ms 64 bytes from 10.78.95.196: icmp_seq=16 ttl=62 time=0.253 ms 64 bytes from 10.78.95.196: icmp_seq=17 ttl=62 time=0.249 ms 64 bytes from 10.78.95.196: icmp_seq=18 ttl=62 time=0.286 ms 64 bytes from 10.78.95.196: icmp_seq=19 ttl=62 time=0.264 ms 64 bytes from 10.78.95.196: icmp_seq=20 ttl=62 time=0.252 ms 64 bytes from 10.78.95.196: icmp_seq=21 ttl=62 time=0.239 ms ^C --- 10.78.95.196 ping statistics --- 21 packets transmitted, 21 received, 0% packet loss, time 20515ms rtt min/avg/max/mdev = 0.110/0.247/0.339/0.050 ms [root@ovn-chassis-1 data]# ip netns exec vm1 ping 10.78.95.196 PING 10.78.95.196 (10.78.95.196) 56(84) bytes of data. 64 bytes from 10.78.95.196: icmp_seq=1 ttl=62 time=0.816 ms 64 bytes from 10.78.95.196: icmp_seq=2 ttl=62 time=0.258 ms 64 bytes from 10.78.95.196: icmp_seq=3 ttl=62 time=0.265 ms 64 bytes from 10.78.95.196: icmp_seq=4 ttl=62 time=0.269 ms 64 bytes from 10.78.95.196: icmp_seq=5 ttl=62 time=0.256 ms 64 bytes from 10.78.95.196: icmp_seq=6 ttl=62 time=0.273 ms 64 bytes from 10.78.95.196: icmp_seq=7 ttl=62 time=0.260 ms 64 bytes from 10.78.95.196: icmp_seq=8 ttl=62 time=0.239 ms ^C --- 10.78.95.196 ping statistics --- 8 packets transmitted, 8 received, 0% packet loss, time 7165ms rtt min/avg/max/mdev = 0.239/0.329/0.816/0.184 ms [root@ovn-chassis-1 data]# ip netns exec vm1 ping 10.78.95.196 PING 10.78.95.196 (10.78.95.196) 56(84) bytes of data. 64 bytes from 10.78.95.196: icmp_seq=1 ttl=62 time=1.41 ms 64 bytes from 10.78.95.196: icmp_seq=2 ttl=62 time=2.10 ms 64 bytes from 10.78.95.196: icmp_seq=3 ttl=62 time=0.275 ms 64 bytes from 10.78.95.196: icmp_seq=4 ttl=62 time=0.262 ms ^C --- 10.78.95.196 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3023ms rtt min/avg/max/mdev = 0.262/1.012/2.102/0.783 ms conntrack v1.4.5 (conntrack-tools): 11 flow entries have been shown. icmp 1 23 src=10.78.95.196 dst=10.78.95.196 type=8 code=0 id=44853 src=192.168.0.52 dst=10.78.95.196 type=0 code=0 id=44853 mark=0 secctx=system_u:object_r:unlabeled_t:s0 zone=1 use=1 icmp 1 29 src=192.168.0.52 dst=10.78.95.196 type=8 code=0 id=41407 src=10.78.95.196 dst=10.78.95.196 type=0 code=0 id=41407 mark=0 secctx=system_u:object_r:unlabeled_t:s0 zone=4 use=1 icmp 1 2 src=192.168.0.52 dst=10.78.95.196 type=8 code=0 id=50072 src=10.78.95.196 dst=10.78.95.196 type=0 code=0 id=50072 mark=0 secctx=system_u:object_r:unlabeled_t:s0 zone=4 use=1 icmp 1 29 src=10.78.95.196 dst=10.78.95.196 type=8 code=0 id=41407 src=192.168.0.52 dst=10.78.95.196 type=0 code=0 id=41407 mark=0 secctx=system_u:object_r:unlabeled_t:s0 zone=1 use=1 icmp 1 23 src=192.168.0.52 dst=10.78.95.196 type=8 code=0 id=44853 src=10.78.95.196 dst=10.78.95.196 type=0 code=0 id=44853 mark=0 secctx=system_u:object_r:unlabeled_t:s0 zone=4 use=1 icmp 1 2 src=10.78.95.196 dst=10.78.95.196 type=8 code=0 id=50072 src=192.168.0.52 dst=10.78.95.196 type=0 code=0 id=50072 mark=0 secctx=system_u:object_r:unlabeled_t:s0 zone=1 use=1 --- I noticed that NAT's had the option stateless=true set. Is that intentional ? If so, the packet should not be sent to conntrack at all. For me it worked both for stateless=true and stateless=false. I tested with the latest main. Maybe you can test with the latest main ? Thanks -- You received this bug notification because you are a
[Bug 1967856] Re: Hairpin traffic does not work with centralized NAT gw
Sure thing! In this DB the active gateway chassis is `deep-ferret.maas` and the instance on `comic-perch.maas` is unable to have two ping sessions to itself using non-distributed FIP 10.78.95.196. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1967856 Title: Hairpin traffic does not work with centralized NAT gw To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1967856] Re: Hairpin traffic does not work with centralized NAT gw
** Attachment added: "ovnsb_db.db" https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+attachment/5577953/+files/ovnsb_db.db -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1967856 Title: Hairpin traffic does not work with centralized NAT gw To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1967856] Re: Hairpin traffic does not work with centralized NAT gw
** Attachment added: "ovnnb_db.db" https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+attachment/5577952/+files/ovnnb_db.db -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1967856 Title: Hairpin traffic does not work with centralized NAT gw To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1967856] Re: Hairpin traffic does not work with centralized NAT gw
Is it possible to attach the OVN dbs ? I'm not able to reproduce it locally. For me a different zone for snat is used on the gateway chassis for the hairpin traffic. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1967856 Title: Hairpin traffic does not work with centralized NAT gw To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1967856] Re: Hairpin traffic does not work with centralized NAT gw
** Changed in: ovn (Ubuntu) Status: New => Triaged ** Changed in: ovn (Ubuntu) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1967856 Title: Hairpin traffic does not work with centralized NAT gw To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/1967856/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
