Re: [ovs-discuss] Encapsulate VXLAN and then process other flows
On 1/26/24 09:37, Lim, Derrick wrote:
> Hi Ilya Maximets,
>
> Thank you for explanation. I've create the two bridges, but the packet
> seems to be dropped looking at the `ovs-appctl dpctl/dump-flows` output.
> I didn’t receive it on the remote host either.
>
> In my setup, the two physical hosts are separated by a L3 network
> (local=100.87.18.6/32, remote=100.87.18.3/32). The routes are learnt by
> a routing agent and exported to the kernel. The kernel has the correct
> routes, but this information does not seem to be synced to OVS.
>
> $ ip route get 100.87.18.3
> 100.87.18.3 via inet6 fe80::920a:84ff:fe9e:9570 dev br-phy src 100.87.18.6
> uid 0
> cache
>
> $ ovs-appctl ovs/route/lookup 100.87.18.3
> src 100.87.2.168
> gateway 100.87.2.129
> dev ens3f1v1
>
> Perhaps the problem is that because I'm using BGP unnumbered, so the IPv4
> destination has an IPv6 next-hop. I tried adding the route statically but
> it seems not to be accepted.
>
> $ ovs-appctl ovs/route/add 100.87.18.3/32 br-phy fe80::920a:84ff:fe9e:9570
> Invalid pkt_mark or gateway
> ovs-appctl: ovs-vswitchd: server returned an error
>
> I've included some additional outputs from my setup below if you find them
> helpful.
>
> Is the routing where I'm going wrong or do you have any other advice about my
> setup?
Hmm. Interesting. I looked through the code and I see that OVS
router module that is responsible for syncing routes from the
kernel to userspace doesn't expect routes with different families.
Such routes are ignored.
The manual ovs/route/add command also expects the next hop to be of
the same IP family, so it refuses to add an v4-via-v6 static route.
So, unfortunately, the setup would work fine with the kernel datapath,
since kernel does all the routing in that case, but it will not work
with userspace.
We need to add support for v4 via v6 routing to ovs-vswitchd in order
to make the tunnels work.
If you're interested in making the change, I could point you to
the right places in the code. :)
Otherwise, maybe someone else from the community will pick it up.
Best regards, Ilya Maximets.
>
> $ ovs-vsctl show
> [abbreviated]
> Bridge br-int
> datapath_type: netdev
> Port vxlan0
> Interface vxlan0
> type: vxlan
> options: {dst_port="4789", key="1", local_ip="100.87.18.6",
> remote_ip="100.87.18.3"}
> Port dpdk-vm101
> Interface dpdk-vm101
> type: dpdkvhostuserclient
> options:
> {vhost-server-path="/var/run/vhost-sockets/dpdk-vm101"}
>
> Bridge br-phy
> fail_mode: standalone
> datapath_type: netdev
> Port br-phy
> tag: 304
> Interface br-phy
> type: internal
> Port exit_p0
> Interface exit_p0
> type: dpdk
> options: {dpdk-devargs=":c4:01.0"}
> ovs_version: "3.1.1"
>
> $ ip addr show
> [abbreviated]
> 27: br-phy: mtu 1500 qdisc fq_codel
> state UP group default qlen 1000
> link/ether de:03:37:e2:1f:ef brd ff:ff:ff:ff:ff:ff
> inet 100.87.18.6/32 scope global br-phy
> valid_lft forever preferred_lft forever
> inet6 fe80::dc03:37ff:fee2:1fef/64 scope link
> valid_lft forever preferred_lft forever
>
> $ ovs-ofctl dump-flows br-int
> cookie=0x0, duration=221.063s, table=0, n_packets=24, n_bytes=1008,
> priority=50,in_port="dpdk-vm101" actions=output:vxlan0
>
> $ ovs-appctl dpctl/dump-flows -m netdev@ovs-netdev
> [abbreviated]
> ufid:7f6b377d-8ee1-4605-91a7-34b1076068f2,
> skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(dpdk-vm101),packet_type(ns=0,id=0),eth(src=52:54:00:3d:cd:0c/00:00:00:00:00:00,dst=ff:ff:ff:ff:ff:ff/00:00:00:00:00:00),eth_type(0x0806),arp(sip=192.168.1.34/0.0.0.0,tip=192.168.1.33/0.0.0.0,op=1/0,sha=52:54:00:3d:cd:0c/00:00:00:00:00:00,tha=00:00:00:00:00:00/00:00:00:00:00:00),
> packets:5, bytes:210, used:0.447s, dp:ovs, actions:drop,
> dp-extra-info:miniflow_bits(4,0)
>
> Thank you,
> Derrick
>
> *From: *Ilya Maximets
> *Date: *Thursday, January 25, 2024 at 20:16
> *To: *Lim, Derrick | Derrick | CMD ,
> ovs-discuss@openvswitch.org
> *Cc: *i.maxim...@ovn.org
> *Subject: *Re: [ovs-discuss] Encapsulate VXLAN and then process other flows
>
> [EXTERNAL] This message comes from an external organization.
>
> On 1/25/24 10:42, Lim, Derrick via discuss wrote:
>> Hey all,
>>
>> Is there a way I can encapsulate a packet with VXLAN, and then resubmit
>> it through OVS again to run other flow actions based on this encapsulated
>> packet?
>>
>> Currently, I have a OVS-DPDK setup where in the final step, before a packet
>> leaves the host, the group action is used to pick between multiple physical
>> ports, and then rewrite the mac address (mod_dl_dst) to that of the
>> destination's, as well as apply the appropriate vlan tag (mod_vlan_vid).
>>
>> I would like the
Re: [ovs-discuss] Encapsulate VXLAN and then process other flows
Hi Ilya Maximets,
Thank you for explanation. I've create the two bridges, but the packet seems to
be dropped looking at the `ovs-appctl dpctl/dump-flows` output. I didn’t
receive it on the remote host either.
In my setup, the two physical hosts are separated by a L3 network
(local=100.87.18.6/32, remote=100.87.18.3/32). The routes are learnt by a
routing agent and exported to the kernel. The kernel has the correct routes,
but this information does not seem to be synced to OVS.
$ ip route get 100.87.18.3
100.87.18.3 via inet6 fe80::920a:84ff:fe9e:9570 dev br-phy src 100.87.18.6 uid 0
cache
$ ovs-appctl ovs/route/lookup 100.87.18.3
src 100.87.2.168
gateway 100.87.2.129
dev ens3f1v1
Perhaps the problem is that because I'm using BGP unnumbered, so the IPv4
destination has an IPv6 next-hop. I tried adding the route statically but it
seems not to be accepted.
$ ovs-appctl ovs/route/add 100.87.18.3/32 br-phy fe80::920a:84ff:fe9e:9570
Invalid pkt_mark or gateway
ovs-appctl: ovs-vswitchd: server returned an error
I've included some additional outputs from my setup below if you find them
helpful.
Is the routing where I'm going wrong or do you have any other advice about my
setup?
$ ovs-vsctl show
[abbreviated]
Bridge br-int
datapath_type: netdev
Port vxlan0
Interface vxlan0
type: vxlan
options: {dst_port="4789", key="1", local_ip="100.87.18.6",
remote_ip="100.87.18.3"}
Port dpdk-vm101
Interface dpdk-vm101
type: dpdkvhostuserclient
options: {vhost-server-path="/var/run/vhost-sockets/dpdk-vm101"}
Bridge br-phy
fail_mode: standalone
datapath_type: netdev
Port br-phy
tag: 304
Interface br-phy
type: internal
Port exit_p0
Interface exit_p0
type: dpdk
options: {dpdk-devargs=":c4:01.0"}
ovs_version: "3.1.1"
$ ip addr show
[abbreviated]
27: br-phy: mtu 1500 qdisc fq_codel
state UP group default qlen 1000
link/ether de:03:37:e2:1f:ef brd ff:ff:ff:ff:ff:ff
inet 100.87.18.6/32 scope global br-phy
valid_lft forever preferred_lft forever
inet6 fe80::dc03:37ff:fee2:1fef/64 scope link
valid_lft forever preferred_lft forever
$ ovs-ofctl dump-flows br-int
cookie=0x0, duration=221.063s, table=0, n_packets=24, n_bytes=1008,
priority=50,in_port="dpdk-vm101" actions=output:vxlan0
$ ovs-appctl dpctl/dump-flows -m netdev@ovs-netdev
[abbreviated]
ufid:7f6b377d-8ee1-4605-91a7-34b1076068f2,
skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(dpdk-vm101),packet_type(ns=0,id=0),eth(src=52:54:00:3d:cd:0c/00:00:00:00:00:00,dst=ff:ff:ff:ff:ff:ff/00:00:00:00:00:00),eth_type(0x0806),arp(sip=192.168.1.34/0.0.0.0,tip=192.168.1.33/0.0.0.0,op=1/0,sha=52:54:00:3d:cd:0c/00:00:00:00:00:00,tha=00:00:00:00:00:00/00:00:00:00:00:00),
packets:5, bytes:210, used:0.447s, dp:ovs, actions:drop,
dp-extra-info:miniflow_bits(4,0)
Thank you,
Derrick
From: Ilya Maximets
Date: Thursday, January 25, 2024 at 20:16
To: Lim, Derrick | Derrick | CMD ,
ovs-discuss@openvswitch.org
Cc: i.maxim...@ovn.org
Subject: Re: [ovs-discuss] Encapsulate VXLAN and then process other flows
[EXTERNAL] This message comes from an external organization.
On 1/25/24 10:42, Lim, Derrick via discuss wrote:
> Hey all,
>
> Is there a way I can encapsulate a packet with VXLAN, and then resubmit
> it through OVS again to run other flow actions based on this encapsulated
> packet?
>
> Currently, I have a OVS-DPDK setup where in the final step, before a packet
> leaves the host, the group action is used to pick between multiple physical
> ports, and then rewrite the mac address (mod_dl_dst) to that of the
> destination's, as well as apply the appropriate vlan tag (mod_vlan_vid).
>
> I would like the encapsulate action to take place before the step mentioned
> above. I created a tunnel port (eg. vxlan0). But if I set the action to this
> port, the packet basically leaves OVS and I can't resubmit it.
>
> In the userspace tunneling example, two bridges are used so that information
> from the kernel can be used for routing and ARP resolution. Is there a way I
> can populate these fields through various flow actions if I already know what
> they should be without going through the kernel? Or is going through the
> kernel
> absolutely required to create the data structure for encapsulation?
The output to a tunnel is an 'output' action, i.e. the packet always leaves
the bridge. And so, it requires routing after encapsulation in order to
identify where it should go next. For routing we need IP addresses and
a routing table. This information is normally synced from the kernel.
You can add static routes via ovs-appctl ovs/route/add, but you still need
IP addresses configured on bridges.
Normally the problem of applying actions after encap
