On Tue, Jul 14, 2026 at 08:42:01PM +0300, Ido Schimmel wrote: > It's unclear to me why you need the UDP port based demux. On a single > node, do you have multiple pods that belong to the same tenant and > therefore share the same VNI? From the above, my understanding is that > the answer is "no".
Good question, and I think my earlier framing sent you down the wrong path, sorry about that. The honest answer is actually "yes". By design a node can hold several VTEPs on the same VNI, as long as they belong to different HA sets. Anti-affinity only keeps the two replicas within one HA set off the same node. Two pods from different HA sets on the same VNI are free to co-locate, and that co-location is the case that needs a per-pod port. > Assuming the answer is "no", instead of having multiple VXLAN sockets > in the host netns (each bound to a different port), why not have a > single VXLAN socket bound to the standard port (4789) and let the > VXLAN driver perform the demux based on the VNI? [...] creating > multiple VXLAN devices in the host netns that only differ in their > VNI and then move each to the relevant pod netns. For different-VNI pods that is a clean fit, and I want to try it out regardless. The outer packet still carries the VNI, so a shared socket can steer by VNI without decapsulating, and the host never touches the tenant frame. Where it stops working for me is two same-VNI pods on one node. They share the underlay IP, the port (4789), and the VNI, so the outer header is the same for both: outer: NodeIP:4789, VNI=X -> pod A (HA set 1) outer: NodeIP:4789, VNI=X -> pod B (HA set 2) There is nothing left in the outer header to steer on, so the host would have to look past it and terminate the tunnel itself, which is the VTEP role I would rather keep inside the pod. A distinct port per pod puts the distinguishing bit back in the outer header, so the host can stay a stateless outer-UDP forwarder. > Given the anti-affinity constraint, you will only need to load balance > between pods on different nodes that use the standard port and that is > already supported by the FDB nexthop groups. Agreed that the across-node load balancing already works. What the VNI-demux version costs is packing density. With one shared underlay IP per node, VNI demux allows at most one VTEP per (VNI, node), so N same-VNI HA sets of two replicas need 2N nodes. With a per-pod port the same sets fit on two nodes (replica 1 on node A, replica 2 on node B, each pod on its own port), and anti-affinity still holds. For 5 sets on one VNI that is 10 nodes versus 2. So the port is not strictly required, and your VNI-demux model is a real option I will prototype. The tradeoff I am weighing is density plus keeping the host out of the tenant datapath, against the uAPI cost, which is small: NHA_FDB_PORT is the nexthop analog of the per-entry NDA_PORT that already exists, it is control-plane only, and a leg with no port behaves exactly as today. Happy to keep digging if it is useful. Thanks again for thinking it through with me.
