On Tue, Aug 16, 2016 at 07:38:45PM -0700, Alexei Starovoitov wrote: > On Tue, Aug 16, 2016 at 5:09 PM, Sargun Dhillon <[email protected]> wrote: > > On Fri, Aug 12, 2016 at 6:16 PM, Alexei Starovoitov > > <[email protected]> wrote: > >> On Thu, Aug 11, 2016 at 12:06 PM, Sargun Dhillon via iovisor-dev > >> <[email protected]> wrote: > >>> Hello all, > >>> Sorry for my bumbling on the phone yesterday. I promised to share the > >>> use cases for the LSM. I'd love to get opinions. > >>> > >>> There is a large enterprise that runs between 8-24 containers on each > >>> server. These apps don't always play by the rules -- They don't always > >>> bind to the right ports, sometimes they make too many connections that > >>> exhaust ephemeral ports, they use up too much bandwidth, etc.. In > >>> addition, this organization has an infrastructure that's leveraging a > >>> legacy DC, and they're very performance sensitive. This legacy DC is > >>> IPv4 only. > >>> > >>> Right now, they employ a mechanism by which they run multiple network > >>> namespaces that use tc mirred action to tie together machines, you can > >>> find out more about how it works here: > >>> http://mesos.apache.org/documentation/latest/port-mapping-isolator/. > >>> This is suboptimal because it interferes with operation of things like > >>> ICMP (packets are fanned out to all network namespaces, resulting in > >>> dups), and it has a noticeable performance overhead. > >>> > >>> Specifically, with data-intensive use cases, they've mentioned > >>> overhead numbers of 20-30%. > >>> > >>> They have the following use cases: > >>> Preventing Incorrect Binds > >>> They prevent apps from interfering that bind to the incorrect port by > >>> not forwarding traffic to that app's given network namespace, but this > >>> has the downside of quiet failures. They would rather have a set of > >>> ports to bind to, and if the app bound out of those, to -EPERM on the > >>> bind() syscall. > >>> > >>> Preventing Resource Exhaustion > >>> They prevent apps from exhausting the ephemeral port range by carving > >>> up 1000 port ephemeral ranges per network namespace, and only > >>> mirroring these ports on ingress. This has interesting issues when > >>> this range gets dense, but it means that 32 containers becomes the cap > >>> on a given machine, because that exhausts 32k set of ports that are > >>> dedicated to ephemeral ports. If instead they just counted the number > >>> of unique connections to a given ip:port for a container, they could > >>> sensibly limit it to 1000, and return EAGAIN, or some such. > >>> > >>> Accounting > >>> They use their current network isolator to account for traffic send > >>> and received by a container. Right now this is done by monitoring the > >>> veth between the container netns, and the host netns. Unfortunately, > >>> this has overhead. On the other hand if we did this using XDP / tc + > >>> rcv_skb, we could save on a lot of overhead. > >>> > >>> Filtering > >>> They run a lot of non-production apps along-side production systems. > >>> They want to be able to limit the egress access of production apps to > >>> non-production apps. This could potentially be done with XDP, but > >>> having multiple XDP filters, or a single complex TC/XDP filter for the > >>> entire system could prove inflexible. Since these filters are > >>> constantly churning, doing filtering to ensure existing connections > >>> are not severed would require connection tracking, and that's yet more > >>> overhead and complexity. Doing this at the syscall level would push > >>> that complexity down. > >>> ----- > >>> > >>> There exists other organizations that want to use helpers with alter > >>> kernel memory. This is done for security, as well as capability. > >>> > >>> Security > >>> The organization uses DSCP markings, and packet marks for filtering on > >>> the system, and on the network. This would require a helper that's > >>> accessible to the LSM to change the mark of packets generated by the > >>> sk and therefore they'd need a helper specifically to access > >>> information beyond just the SK. This could also be done using the > >>> netlabels framework for their use case, but netlabels requires opening > >>> up quite a few more APIs. > >>> > >>> Capability: Load Balancing > >>> There are a ton of container load balancing solutions right now. > >>> Unfortunately, all of them have caveats - IPVS only works with NAT in > >>> the cloud, HAProxy has a ton of overhead kubeproxy + iptables is slow > >>> and requires NAT. In addition to this, all of these solutions lose > >>> fidelity and make the BSD socket API not-a-thing for introspection of > >>> peer addresses. This makes logs hard to use. > >>> > >>> The organizations want the helper to be able to write to the > >>> sockaddr_in while intercepting the connect syscall to redirect that > >>> connect elsewhere. This is (1) much lower overhead than doing this at > >>> the XDP / TC layer (2) allows for logging to keep working. > >>> > >>> Capability: Port Remapping (DNAT) > >>> A lot of folks run Docker in the bridge / Port binding mode > >>> (https://docs.docker.com/engine/userguide/networking/default_network/binding/). > >>> Unfortunately, this has a lot of downsides such as speed, and > >>> requiring a separate ns. The customer would like to rewrite the struct > >>> sockaddr during the bind syscall. They're happier doing this once the > >>> data is copied to a kernel address rather than doing it in a probe to > >>> prevent non-cooperating programs from binding to addresses that > >>> shouldn't be available to them. (plus the aforementioned reasons about > >>> Load Balancing) > >> > >> Thanks a lot for describing the use cases. > >> The problem statement is clear. > >> I suspect there cannot be a single solution to all of the above. > >> Sounds like all containers already use netns. > >> By itself netns has non-trivial overhead. Hannes is working > >> on something new that should solve veth/netns performance issues. > >> Netns makes above problem to be solvable at L2 level. > >> Sometimes it doesn't fit. The solution without netns are being > >> developed. > >> The upcoming cgroup+bpf will allow ingress/egress socket > >> filtering without netns. All processes of a container will be > >> under a cgroup and bpf program will enforce operation for all > >> sockets of all processes. > >> Sounds like what you're saying is that checmate lsm can > >> solve all of the above? I think theoretically it can, but bpf > >> programs will become very complex. > >> imo cgroup+bpf approach is easier to manage and operate. > >> cgroup+bpf won't work for load balancing and nat, but they're > >> solvable at tc+bpf layer like cilium does. > > > > I'm not familiar with Cilium? > > > > I'm implementing some of the programs that have the aforementioned > > behaviour, and some of them (limiting usage of ephemeral ports) gets > > complicated, but it's still in the realm of possibility. I plan to > > share: > > -Rx / Tx statistics per container > > -Limiting usage of ephemeral ports > > -Rewriting bind port > > -Limiting filesystem access > > > > Do you have any advice on the API? > > For networking bits Daniel Mack is working on cgroup+bpf patches > that will allow 1 and 2. > Rewriting bind port would need write access which is a possible > extension. > fs is tricky. Probably should be cgroup based as well, > but that will be different cgroup controller. The one Daniel does > is network only, since the hook point is similar to sk_filter(). > I read the thread on netdev by the fellow writing a cgroup isolator to do similar things to me. Given this, do you the approach of building cgroups controllers to attach hooks to BPF programs is better, and I should abandon the LSM for that instead -- or just sit tight for Daniel Mack's patches?
Perhaps it makes sense to have a network cgroup, where there is a per-cgroup hook to get into sk_filter, bind, listen, so that one can filter there instead of at the LSM level? Do you think adding such hooks would perhaps be a better approach, as opposed to doing a global LSM? > > Currently, I've created some new bits around prctl, but I feel like > > those extensions are a bit awkward, since prctl is supposed to work on > > a process-by-process basis. On the other hand, a VFS API seems complex > > -- do I make it similar to kprobes, where someone can echo an fd # > > into some file, and we add the probe? _______________________________________________ iovisor-dev mailing list [email protected] https://lists.iovisor.org/mailman/listinfo/iovisor-dev
