On Thu, Aug 11, 2016 at 12:06 PM, Sargun Dhillon via iovisor-dev <[email protected]> wrote: > Hello all, > Sorry for my bumbling on the phone yesterday. I promised to share the > use cases for the LSM. I'd love to get opinions. > > There is a large enterprise that runs between 8-24 containers on each > server. These apps don't always play by the rules -- They don't always > bind to the right ports, sometimes they make too many connections that > exhaust ephemeral ports, they use up too much bandwidth, etc.. In > addition, this organization has an infrastructure that's leveraging a > legacy DC, and they're very performance sensitive. This legacy DC is > IPv4 only. > > Right now, they employ a mechanism by which they run multiple network > namespaces that use tc mirred action to tie together machines, you can > find out more about how it works here: > http://mesos.apache.org/documentation/latest/port-mapping-isolator/. > This is suboptimal because it interferes with operation of things like > ICMP (packets are fanned out to all network namespaces, resulting in > dups), and it has a noticeable performance overhead. > > Specifically, with data-intensive use cases, they've mentioned > overhead numbers of 20-30%. > > They have the following use cases: > Preventing Incorrect Binds > They prevent apps from interfering that bind to the incorrect port by > not forwarding traffic to that app's given network namespace, but this > has the downside of quiet failures. They would rather have a set of > ports to bind to, and if the app bound out of those, to -EPERM on the > bind() syscall. > > Preventing Resource Exhaustion > They prevent apps from exhausting the ephemeral port range by carving > up 1000 port ephemeral ranges per network namespace, and only > mirroring these ports on ingress. This has interesting issues when > this range gets dense, but it means that 32 containers becomes the cap > on a given machine, because that exhausts 32k set of ports that are > dedicated to ephemeral ports. If instead they just counted the number > of unique connections to a given ip:port for a container, they could > sensibly limit it to 1000, and return EAGAIN, or some such. > > Accounting > They use their current network isolator to account for traffic send > and received by a container. Right now this is done by monitoring the > veth between the container netns, and the host netns. Unfortunately, > this has overhead. On the other hand if we did this using XDP / tc + > rcv_skb, we could save on a lot of overhead. > > Filtering > They run a lot of non-production apps along-side production systems. > They want to be able to limit the egress access of production apps to > non-production apps. This could potentially be done with XDP, but > having multiple XDP filters, or a single complex TC/XDP filter for the > entire system could prove inflexible. Since these filters are > constantly churning, doing filtering to ensure existing connections > are not severed would require connection tracking, and that's yet more > overhead and complexity. Doing this at the syscall level would push > that complexity down. > ----- > > There exists other organizations that want to use helpers with alter > kernel memory. This is done for security, as well as capability. > > Security > The organization uses DSCP markings, and packet marks for filtering on > the system, and on the network. This would require a helper that's > accessible to the LSM to change the mark of packets generated by the > sk and therefore they'd need a helper specifically to access > information beyond just the SK. This could also be done using the > netlabels framework for their use case, but netlabels requires opening > up quite a few more APIs. > > Capability: Load Balancing > There are a ton of container load balancing solutions right now. > Unfortunately, all of them have caveats - IPVS only works with NAT in > the cloud, HAProxy has a ton of overhead kubeproxy + iptables is slow > and requires NAT. In addition to this, all of these solutions lose > fidelity and make the BSD socket API not-a-thing for introspection of > peer addresses. This makes logs hard to use. > > The organizations want the helper to be able to write to the > sockaddr_in while intercepting the connect syscall to redirect that > connect elsewhere. This is (1) much lower overhead than doing this at > the XDP / TC layer (2) allows for logging to keep working. > > Capability: Port Remapping (DNAT) > A lot of folks run Docker in the bridge / Port binding mode > (https://docs.docker.com/engine/userguide/networking/default_network/binding/). > Unfortunately, this has a lot of downsides such as speed, and > requiring a separate ns. The customer would like to rewrite the struct > sockaddr during the bind syscall. They're happier doing this once the > data is copied to a kernel address rather than doing it in a probe to > prevent non-cooperating programs from binding to addresses that > shouldn't be available to them. (plus the aforementioned reasons about > Load Balancing)
Thanks a lot for describing the use cases. The problem statement is clear. I suspect there cannot be a single solution to all of the above. Sounds like all containers already use netns. By itself netns has non-trivial overhead. Hannes is working on something new that should solve veth/netns performance issues. Netns makes above problem to be solvable at L2 level. Sometimes it doesn't fit. The solution without netns are being developed. The upcoming cgroup+bpf will allow ingress/egress socket filtering without netns. All processes of a container will be under a cgroup and bpf program will enforce operation for all sockets of all processes. Sounds like what you're saying is that checmate lsm can solve all of the above? I think theoretically it can, but bpf programs will become very complex. imo cgroup+bpf approach is easier to manage and operate. cgroup+bpf won't work for load balancing and nat, but they're solvable at tc+bpf layer like cilium does. _______________________________________________ iovisor-dev mailing list [email protected] https://lists.iovisor.org/mailman/listinfo/iovisor-dev
