Hello all, Sorry for my bumbling on the phone yesterday. I promised to share the use cases for the LSM. I'd love to get opinions.
There is a large enterprise that runs between 8-24 containers on each server. These apps don't always play by the rules -- They don't always bind to the right ports, sometimes they make too many connections that exhaust ephemeral ports, they use up too much bandwidth, etc.. In addition, this organization has an infrastructure that's leveraging a legacy DC, and they're very performance sensitive. This legacy DC is IPv4 only. Right now, they employ a mechanism by which they run multiple network namespaces that use tc mirred action to tie together machines, you can find out more about how it works here: http://mesos.apache.org/documentation/latest/port-mapping-isolator/. This is suboptimal because it interferes with operation of things like ICMP (packets are fanned out to all network namespaces, resulting in dups), and it has a noticeable performance overhead. Specifically, with data-intensive use cases, they've mentioned overhead numbers of 20-30%. They have the following use cases: Preventing Incorrect Binds They prevent apps from interfering that bind to the incorrect port by not forwarding traffic to that app's given network namespace, but this has the downside of quiet failures. They would rather have a set of ports to bind to, and if the app bound out of those, to -EPERM on the bind() syscall. Preventing Resource Exhaustion They prevent apps from exhausting the ephemeral port range by carving up 1000 port ephemeral ranges per network namespace, and only mirroring these ports on ingress. This has interesting issues when this range gets dense, but it means that 32 containers becomes the cap on a given machine, because that exhausts 32k set of ports that are dedicated to ephemeral ports. If instead they just counted the number of unique connections to a given ip:port for a container, they could sensibly limit it to 1000, and return EAGAIN, or some such. Accounting They use their current network isolator to account for traffic send and received by a container. Right now this is done by monitoring the veth between the container netns, and the host netns. Unfortunately, this has overhead. On the other hand if we did this using XDP / tc + rcv_skb, we could save on a lot of overhead. Filtering They run a lot of non-production apps along-side production systems. They want to be able to limit the egress access of production apps to non-production apps. This could potentially be done with XDP, but having multiple XDP filters, or a single complex TC/XDP filter for the entire system could prove inflexible. Since these filters are constantly churning, doing filtering to ensure existing connections are not severed would require connection tracking, and that's yet more overhead and complexity. Doing this at the syscall level would push that complexity down. ----- There exists other organizations that want to use helpers with alter kernel memory. This is done for security, as well as capability. Security The organization uses DSCP markings, and packet marks for filtering on the system, and on the network. This would require a helper that's accessible to the LSM to change the mark of packets generated by the sk and therefore they'd need a helper specifically to access information beyond just the SK. This could also be done using the netlabels framework for their use case, but netlabels requires opening up quite a few more APIs. Capability: Load Balancing There are a ton of container load balancing solutions right now. Unfortunately, all of them have caveats - IPVS only works with NAT in the cloud, HAProxy has a ton of overhead kubeproxy + iptables is slow and requires NAT. In addition to this, all of these solutions lose fidelity and make the BSD socket API not-a-thing for introspection of peer addresses. This makes logs hard to use. The organizations want the helper to be able to write to the sockaddr_in while intercepting the connect syscall to redirect that connect elsewhere. This is (1) much lower overhead than doing this at the XDP / TC layer (2) allows for logging to keep working. Capability: Port Remapping (DNAT) A lot of folks run Docker in the bridge / Port binding mode (https://docs.docker.com/engine/userguide/networking/default_network/binding/). Unfortunately, this has a lot of downsides such as speed, and requiring a separate ns. The customer would like to rewrite the struct sockaddr during the bind syscall. They're happier doing this once the data is copied to a kernel address rather than doing it in a probe to prevent non-cooperating programs from binding to addresses that shouldn't be available to them. (plus the aforementioned reasons about Load Balancing) _______________________________________________ iovisor-dev mailing list [email protected] https://lists.iovisor.org/mailman/listinfo/iovisor-dev
