Hi Toke, On Mon, Jun 21, 2021 at 04:27:08PM +0200, Toke Høiland-Jørgensen wrote: > Daniel Golle <[email protected]> writes: > > > On Fri, Jun 18, 2021 at 02:24:29PM +0200, Jason A. Donenfeld wrote: > >> Hey Toke, > >> > >> On Fri, Jun 18, 2021 at 1:05 AM Toke Høiland-Jørgensen <[email protected]> > >> wrote: > >> > > I think you can achieve something similar using BPF filters, by relying > >> > > on wireguard passing through the skb->hash value when encrypting. > >> > > > >> > > Simply attach a TC-BPF filter to the wireguard netdev, pull out the > >> > > DSCP > >> > > value and store it in a map keyed on skb->hash. Then, run a second BPF > >> > > filter on the physical interface that shares that same map, lookup the > >> > > DSCP value based on the skb->hash value, and rewrite the outer IP > >> > > header. > >> > > > >> > > The read-side filter will need to use bpf_get_hash_recalc() to make > >> > > sure > >> > > the hash is calculated before the packet gets handed to wireguard, and > >> > > it'll be subject to hash collisions, but I think it should generally > >> > > work fairly well (for anything that's flow-based of course). And it can > >> > > be done without patching wireguard itself :) > >> > > >> > Just for fun I implemented such a pair of eBPF filters, and tested that > >> > it does indeed work for preserving DSCP marks on a Wireguard tunnel. The > >> > PoC is here: > >> > > >> > https://github.com/xdp-project/bpf-examples/tree/master/preserve-dscp > >> > > >> > To try it out (you'll need a recent-ish kernel and clang version) run: > >> > > >> > git clone --recurse-submodules > >> > https://github.com/xdp-project/bpf-examples > >> > cd bpf-examples/preserve-dscp > >> > make > >> > ./preserve-dscp wg0 eth0 > >> > > >> > (assuming wg0 and eth0 are the wireguard and physical interfaces in > >> > question, respectively). > >> > > >> > To actually deploy this it would probably need a few tweaks; in > >> > particular the second filter that rewrites packets should probably check > >> > that the packets are actually part of the Wireguard tunnel in question > >> > (by parsing the UDP header and checking the source port) before writing > >> > anything to the packet. > >> > > >> > -Toke > >> > >> That is a super cool approach. Thanks for writing that! Sounds like a > >> good approach, and one pretty easy to deploy, without the need to > >> patch kernels and such. > >> > >> Also, nice usage of BPF_MAP_TYPE_LRU_HASH for this. > >> > >> Daniel -- can you let the list know if this works for your use case? > > > > Turns out not exactly easy to deploy (on OpenWrt), as it depends on an > > extremely recent environment. I will try pushing to that direction, but > > it doesn't look like it's going to be ready very soon. > > > > In terms of toolchain: LLVM/Clang is a very bulky beast, I gave up on > > that and started working on integrating GCC-10's BPF target in our build > > system... > > I saw that, but I have no idea if GCC's BPF target support will support > this. My tentative guess would be no, unfortunately :(
Probably you are right. When building the BPF object with GCC, the result is: root@OpenWrt:/usr/lib/bpf# preserve-dscp wg0 eth0 libbpf: elf: skipping unrecognized data section(4) .stab libbpf: elf: skipping relo section(5) .rel.stab for section(4) .stab libbpf: elf: skipping unrecognized data section(13) .comment libbpf: BTF is required, but is missing or corrupted. Couldn't open file: preserve_dscp_kern.o Using the LLVM/Clang compiled object also doesn't work: root@OpenWrt:/usr/lib/bpf# preserve-dscp wg0 eth0 libbpf: Error in bpf_create_map_xattr(flow_dscps):Operation not permitted(-1). Retrying without BTF. libbpf: map 'flow_dscps': failed to create: Operation not permitted(-1) libbpf: permission error while running as root; try raising 'ulimit -l'? current value: 512.0 KiB libbpf: failed to load object 'preserve_dscp_kern.o' Failed to load object Probably Kernel 5.4.124 is too old...? > > An alternative to getting LLVM built as part of the OpenWrt toolchain is > to just use the host clang to build the BPF binaries. It doesn't > actually need to be cross-compiled with a special compiler, the BPF byte > code format is the same on all architectures except for endianness, so > just passing that to the host clang should theoretically be enough... I believe that having a way to build BPF objects compatible with the target built-into our toolchain would be a huge step forward. And given that gcc already get's pretty far, I think it'd be worth fixing/patching what ever is missing (I haven't even tried GCC-11 yet) Find my staging tree including 'preserve-dscp' ready to play with: https://git.openwrt.org/?p=openwrt/staging/dangole.git;a=shortlog;h=refs/heads/gcc10-bpf Select 'Enable experimental features by default', but note that toolchain doesn't build when selecting Linux 5.10 for x86, so you need to un-select 'Use testing Kernel' if building for x86. And have a look at the patch for allow building bpf-examples BPF objects with GCC in package/network/utils/bpf-examples/patches > > > In terms of kernel support: recent kernels don't build yet because of > > gelf_getsymshndx, so we got to update libelf first for that. Recent > > libelf doesn't seem to be an option yet on many of the build hosts we > > currently support (Darwin and such). > > > > In terms of library support: our build of libbpf comes from Linux > > release tarballs. There isn't yet a release supporting bpf_tc_attach, > > the easiest would be to wait for Linux 5.13 to be released. > > I used the libbpf TC loading support for convenience, but it's possible > to load it using 'tc' as well without too much trouble (right now the > userspace component sets a config variable before loading the program, > but it can be restructured to not need that). > > Alternatively, the bpf-examples repository is setup with a libbpf > submodule that it can link statically against, so you could use that for > now? I've updated to 5.13 + patches on top, so now it builds :) Library-embedding is a no-go for OpenWrt. Having different ABI-versions of libraries installed simultanously works, so we can just ship with a more recent version of libbpf. > > > I (of course ;) also tried and spend almost a day looking for a > > quick-and-dirty path for temporary deployment, so I could at least give > > feedback -- bpf-examples also isn't exactly made to be cross-compiled > > manually, so I have failed with that as well so far. > > Heh, no, it isn't, really. Anything in particular you need to make this > easier? We already added some bits to xdp-tools for supporting > cross-compilation (and that shares some lineage with bpf-examples), so > porting those over should not be too difficult. I found my way around, see the packaging for bpf-examples in the tree (link above, at path stated above) > > See: https://github.com/xdp-project/xdp-tools/pull/78 and > https://github.com/xdp-project/xdp-tools/issues/74 > > Unfortunately I don't have a lot of time to poke more at this right now, > but feel free to open up an issue / pull request to the bpf-examples > repository with any changes you need :) I guess I'll just go ahead then and package xdp-tools :) Cheers Daniel > > -Toke
