On Tue, Aug 16, 2016 at 5:09 PM, Sargun Dhillon <[email protected]> wrote: > On Fri, Aug 12, 2016 at 6:16 PM, Alexei Starovoitov > <[email protected]> wrote: >> On Thu, Aug 11, 2016 at 12:06 PM, Sargun Dhillon via iovisor-dev >> <[email protected]> wrote: >>> Hello all, >>> Sorry for my bumbling on the phone yesterday. I promised to share the >>> use cases for the LSM. I'd love to get opinions. >>> >>> There is a large enterprise that runs between 8-24 containers on each >>> server. These apps don't always play by the rules -- They don't always >>> bind to the right ports, sometimes they make too many connections that >>> exhaust ephemeral ports, they use up too much bandwidth, etc.. In >>> addition, this organization has an infrastructure that's leveraging a >>> legacy DC, and they're very performance sensitive. This legacy DC is >>> IPv4 only. >>> >>> Right now, they employ a mechanism by which they run multiple network >>> namespaces that use tc mirred action to tie together machines, you can >>> find out more about how it works here: >>> http://mesos.apache.org/documentation/latest/port-mapping-isolator/. >>> This is suboptimal because it interferes with operation of things like >>> ICMP (packets are fanned out to all network namespaces, resulting in >>> dups), and it has a noticeable performance overhead. >>> >>> Specifically, with data-intensive use cases, they've mentioned >>> overhead numbers of 20-30%. >>> >>> They have the following use cases: >>> Preventing Incorrect Binds >>> They prevent apps from interfering that bind to the incorrect port by >>> not forwarding traffic to that app's given network namespace, but this >>> has the downside of quiet failures. They would rather have a set of >>> ports to bind to, and if the app bound out of those, to -EPERM on the >>> bind() syscall. >>> >>> Preventing Resource Exhaustion >>> They prevent apps from exhausting the ephemeral port range by carving >>> up 1000 port ephemeral ranges per network namespace, and only >>> mirroring these ports on ingress. This has interesting issues when >>> this range gets dense, but it means that 32 containers becomes the cap >>> on a given machine, because that exhausts 32k set of ports that are >>> dedicated to ephemeral ports. If instead they just counted the number >>> of unique connections to a given ip:port for a container, they could >>> sensibly limit it to 1000, and return EAGAIN, or some such. >>> >>> Accounting >>> They use their current network isolator to account for traffic send >>> and received by a container. Right now this is done by monitoring the >>> veth between the container netns, and the host netns. Unfortunately, >>> this has overhead. On the other hand if we did this using XDP / tc + >>> rcv_skb, we could save on a lot of overhead. >>> >>> Filtering >>> They run a lot of non-production apps along-side production systems. >>> They want to be able to limit the egress access of production apps to >>> non-production apps. This could potentially be done with XDP, but >>> having multiple XDP filters, or a single complex TC/XDP filter for the >>> entire system could prove inflexible. Since these filters are >>> constantly churning, doing filtering to ensure existing connections >>> are not severed would require connection tracking, and that's yet more >>> overhead and complexity. Doing this at the syscall level would push >>> that complexity down. >>> ----- >>> >>> There exists other organizations that want to use helpers with alter >>> kernel memory. This is done for security, as well as capability. >>> >>> Security >>> The organization uses DSCP markings, and packet marks for filtering on >>> the system, and on the network. This would require a helper that's >>> accessible to the LSM to change the mark of packets generated by the >>> sk and therefore they'd need a helper specifically to access >>> information beyond just the SK. This could also be done using the >>> netlabels framework for their use case, but netlabels requires opening >>> up quite a few more APIs. >>> >>> Capability: Load Balancing >>> There are a ton of container load balancing solutions right now. >>> Unfortunately, all of them have caveats - IPVS only works with NAT in >>> the cloud, HAProxy has a ton of overhead kubeproxy + iptables is slow >>> and requires NAT. In addition to this, all of these solutions lose >>> fidelity and make the BSD socket API not-a-thing for introspection of >>> peer addresses. This makes logs hard to use. >>> >>> The organizations want the helper to be able to write to the >>> sockaddr_in while intercepting the connect syscall to redirect that >>> connect elsewhere. This is (1) much lower overhead than doing this at >>> the XDP / TC layer (2) allows for logging to keep working. >>> >>> Capability: Port Remapping (DNAT) >>> A lot of folks run Docker in the bridge / Port binding mode >>> (https://docs.docker.com/engine/userguide/networking/default_network/binding/). >>> Unfortunately, this has a lot of downsides such as speed, and >>> requiring a separate ns. The customer would like to rewrite the struct >>> sockaddr during the bind syscall. They're happier doing this once the >>> data is copied to a kernel address rather than doing it in a probe to >>> prevent non-cooperating programs from binding to addresses that >>> shouldn't be available to them. (plus the aforementioned reasons about >>> Load Balancing) >> >> Thanks a lot for describing the use cases. >> The problem statement is clear. >> I suspect there cannot be a single solution to all of the above. >> Sounds like all containers already use netns. >> By itself netns has non-trivial overhead. Hannes is working >> on something new that should solve veth/netns performance issues. >> Netns makes above problem to be solvable at L2 level. >> Sometimes it doesn't fit. The solution without netns are being >> developed. >> The upcoming cgroup+bpf will allow ingress/egress socket >> filtering without netns. All processes of a container will be >> under a cgroup and bpf program will enforce operation for all >> sockets of all processes. >> Sounds like what you're saying is that checmate lsm can >> solve all of the above? I think theoretically it can, but bpf >> programs will become very complex. >> imo cgroup+bpf approach is easier to manage and operate. >> cgroup+bpf won't work for load balancing and nat, but they're >> solvable at tc+bpf layer like cilium does. > > I'm not familiar with Cilium? > > I'm implementing some of the programs that have the aforementioned > behaviour, and some of them (limiting usage of ephemeral ports) gets > complicated, but it's still in the realm of possibility. I plan to > share: > -Rx / Tx statistics per container > -Limiting usage of ephemeral ports > -Rewriting bind port > -Limiting filesystem access > > Do you have any advice on the API?
For networking bits Daniel Mack is working on cgroup+bpf patches that will allow 1 and 2. Rewriting bind port would need write access which is a possible extension. fs is tricky. Probably should be cgroup based as well, but that will be different cgroup controller. The one Daniel does is network only, since the hook point is similar to sk_filter(). > Currently, I've created some new bits around prctl, but I feel like > those extensions are a bit awkward, since prctl is supposed to work on > a process-by-process basis. On the other hand, a VFS API seems complex > -- do I make it similar to kprobes, where someone can echo an fd # > into some file, and we add the probe? _______________________________________________ iovisor-dev mailing list [email protected] https://lists.iovisor.org/mailman/listinfo/iovisor-dev
