Re: [RFC 0/4] RFC: Add Checmate, BPF-driven minor LSM
On Mon, Aug 15, 2016 at 12:59:13PM +0200, Mickaël Salaün wrote: > > On 15/08/2016 05:09, Sargun Dhillon wrote: > > On Mon, Aug 15, 2016 at 12:57:44AM +0200, Mickaël Salaün wrote: > >> Our approaches have some common points (i.e. use eBPF in an LSM, stacked > >> filters like seccomp) but I'm focused on a kind of unprivileged LSM (i.e. > >> no > >> CAP_SYS_ADMIN), to make standalone sandboxes, which brings more > >> constraints > >> (e.g. no use of unsafe functions like bpf_probe_read(), take care of > >> privacy, > >> SUID exec, stable ABI…). However, I don't want to handle resource limits, > >> which should be the job of cgroups. > >> > > Kind of. Sometimes describing these resource limits is difficult. For > > example, I > > have a customer who is trying to restrict containers from burning up all > > the > > ephemeral ports on the machine. In this, they have an incredibly elaborate > > chain > > of wiring to prevent a given container from connecting to the same (proto, > > destip, destport) more than 1000 times. > > > > I'm unsure of how you'd model that in a cgroup. > > This looks like a Netfilter rule. Have you tried applying this limitation > with the connlimit module? > > I could do this by adding a new Netfilter match, but with the existing matches, the only ones that "select" by cgroup2 don't have the ability to connlimit by cgroup. Potentially, I could wire up something with the cgroup2 match, but this comes with a lot of overhead. If you know of a low-overhead way of doing this, I'd love to hear. Have you ever user Kubernetes? (http://kubernetes.io/docs/whatisk8s/)? You usually have a bunch of independent systems running together under what's called a "Pod". You can think of this as an old style "lxc" container, or a VM, and in each of these pods there is nesting where you want to not only limit the pod's resources, but you also want to limit the resources of each application. Doing this without some layer of programmability in resource management layer can be difficult. > > > >> For now, I'm focusing on file-system access control which is one of the > >> more > >> complex system to properly filter. I also plan to support basic network > >> access > >> control. > >> > >> What you are trying to accomplish seems more related to a Netfilter > >> extension > >> (something like ipset but with eBPF maybe?). > >> > > I don't only want to do network access control, I also want to write to the > > value once it's copied into kernel space. There are lot of benefits of > > doing > > this at the syscall level, but the two primary ones are performance, and > > capability. > > > > One of the biggest complaints with our current approach to filtering & load > > balancing (iptables) is that it hides information. When people connect > > through > > the load balancer, they want to find out who they connected to, and without > > some > > high application-level mechanism, this isn't possible. On the other hand, > > if we > > just rewrite the destination address in the connect hook, we can pretty > > easily > > allow them to do getpeername. > > What exactly is not doable with Netfilter (e.g. REDIRECT or TPROXY)? > > Is there a way to "load balance" or "proxy" a connection where getpeername() tells you the real IP of the node you're connected to? > > > > I'm curious about your filesystem access limiter. Do you have a way to make > > it so > > that a given container can only write, say, 100mb of data to disk? > > It's a filesystem access control. It doesn't deal with quota and is not > focused on container but process hierarchies (which is more generic). > > What is not doable with a quota mount option? It may be more appropriate to > enhance the VFS (or overlayfs) to apply this kind of limitation, if needed. > Your overlayfs suggesion is on point. Since a lot of my containers look similar to Kubernetes though, quota isn't very well aligned with them (within a Pod, there are usually a bunch of independent things that need their usage limited). I think quota / overlayfs with labeling that comes from an LSM, or some other smart classifier would be ideal.
Re: [RFC 0/4] RFC: Add Checmate, BPF-driven minor LSM
On 15/08/2016 05:09, Sargun Dhillon wrote: > On Mon, Aug 15, 2016 at 12:57:44AM +0200, Mickaël Salaün wrote: >> Our approaches have some common points (i.e. use eBPF in an LSM, stacked >> filters like seccomp) but I'm focused on a kind of unprivileged LSM (i.e. no >> CAP_SYS_ADMIN), to make standalone sandboxes, which brings more constraints >> (e.g. no use of unsafe functions like bpf_probe_read(), take care of >> privacy, >> SUID exec, stable ABI…). However, I don't want to handle resource limits, >> which should be the job of cgroups. >> > Kind of. Sometimes describing these resource limits is difficult. For > example, I > have a customer who is trying to restrict containers from burning up all the > ephemeral ports on the machine. In this, they have an incredibly elaborate > chain > of wiring to prevent a given container from connecting to the same (proto, > destip, destport) more than 1000 times. > > I'm unsure of how you'd model that in a cgroup. This looks like a Netfilter rule. Have you tried applying this limitation with the connlimit module? > >> For now, I'm focusing on file-system access control which is one of the more >> complex system to properly filter. I also plan to support basic network >> access >> control. >> >> What you are trying to accomplish seems more related to a Netfilter >> extension >> (something like ipset but with eBPF maybe?). >> > I don't only want to do network access control, I also want to write to the > value once it's copied into kernel space. There are lot of benefits of doing > this at the syscall level, but the two primary ones are performance, and > capability. > > One of the biggest complaints with our current approach to filtering & load > balancing (iptables) is that it hides information. When people connect > through > the load balancer, they want to find out who they connected to, and without > some > high application-level mechanism, this isn't possible. On the other hand, if > we > just rewrite the destination address in the connect hook, we can pretty easily > allow them to do getpeername. What exactly is not doable with Netfilter (e.g. REDIRECT or TPROXY)? > > I'm curious about your filesystem access limiter. Do you have a way to make > it so > that a given container can only write, say, 100mb of data to disk? It's a filesystem access control. It doesn't deal with quota and is not focused on container but process hierarchies (which is more generic). What is not doable with a quota mount option? It may be more appropriate to enhance the VFS (or overlayfs) to apply this kind of limitation, if needed. signature.asc Description: OpenPGP digital signature
Re: [RFC 0/4] RFC: Add Checmate, BPF-driven minor LSM
Hi, I've been working on an extension to seccomp-bpf since last year and published a first RFC about it [1]. I'm working on a second RFC/PoC which use eBPF instead of cBPF and is more close to a common LSM than the first RFC. I plan to publish this second RFC by the end of the month. Our approaches have some common points (i.e. use eBPF in an LSM, stacked filters like seccomp) but I'm focused on a kind of unprivileged LSM (i.e. no CAP_SYS_ADMIN), to make standalone sandboxes, which brings more constraints (e.g. no use of unsafe functions like bpf_probe_read(), take care of privacy, SUID exec, stable ABI…). However, I don't want to handle resource limits, which should be the job of cgroups. For now, I'm focusing on file-system access control which is one of the more complex system to properly filter. I also plan to support basic network access control. What you are trying to accomplish seems more related to a Netfilter extension (something like ipset but with eBPF maybe?). Mickaël [1] http://www.openwall.com/lists/kernel-hardening/2016/03/24/2 On 09/08/2016 02:22, Kees Cook wrote: > On Mon, Aug 8, 2016 at 5:00 PM, Sargun Dhillon wrote: >> On Mon, Aug 08, 2016 at 04:44:02PM -0700, Kees Cook wrote: >>> On Thu, Aug 4, 2016 at 12:11 AM, Sargun Dhillon wrote: I distributed this patchset to linux-security-mod...@vger.kernel.org earlier, but based on the fact that the archive is down, and this is a fairly broad-sweeping proposal, I figured I'd grow the audience a little bit. Sorry if you received this multiple times. I've begun building out the skeleton of a Linux Security Module, and I'd like to get feedback on it. It's a skeleton, and I've only populated a few hooks, so I'm mostly looking for input on the general proposal, interest, and design. It's a minor LSM. My particular use case is one in which containers are being dynamically deployed to machines by internal developers in a different group. The point of Checmate is to act as an extensible bed for _safe_, complex security policies. It's nice to enable dynamic security policies that can be defined in C, and change as neccessary, without ever having to patch, or rebuild the kernel. For many of these containers, the security policies can be fairly nuanced. One particular one to take into account is network security. Often times, administrators want to prevent ingress, and egress connectivity except from a few select IPs. Egress filtering can be managed using net_cls, but without modifying running software, it's non-trivial to attach a filter to all sockets being created within a container. The inet_conn_request, socket_recvmsg, socket_sock_rcv_skb hooks make this trivial to implement. Other times, containers need to be throttled in places where there's not really a good place to impose that policy for software which isn't built in-house. If one wants to limit file creations/sec, or reject I/O under certain characteristics, there's not a great place to do it now. This gives engineers a mechanism to write those policies. This same flexibility can be used to take existing programs and enable safe BPF helpers to modify memory to allow rules to pass. One example that I prototyped was Docker's port mapping, which has an overhead (DNAT), and there's some loss of fidelity in the BSD Socket API to identify what's going on. Instead, we can just rewrite the port in a bind, based upon some data in a BPF map, and a cgroup match. I can actually see other minor security modules being implemented in Checmate, for example, Yama, or the recently proposed Hardchroot could be reimplemented in BPF. Potentially, they could even be API compatible. Although, at first, much of this sounds like seccomp, it's quite different. For one, what we can do in the security hooks is more complex (access to kernel pointers). The other side of this is we can have effects on a system-wide, or cgroup level. This also circumvents the need for CRIU-friendly policies. Lastly, the flexibility of this mechanism allows for prevention of security vulnerabilities which are often complex in nature and require the interaction of multiple hooks (CVE-2014-9717 is a good example), and although ksplice, and livepatch exist, they're not always easy to use, as compared to loading a single bpf program across all kernels. The user-facing API is exposed via prctl as it's meant to be very simple (at least the kernel components). It only has three operations. For a given security hook, you can attach a BPF program to it, which will add it to the set of programs that are execute
Re: [RFC 0/4] RFC: Add Checmate, BPF-driven minor LSM
On Mon, Aug 15, 2016 at 12:57:44AM +0200, Mickaël Salaün wrote: > Hi, > > I've been working on an extension to seccomp-bpf since last year and > published > a first RFC about it [1]. I'm working on a second RFC/PoC which use eBPF > instead of cBPF and is more close to a common LSM than the first RFC. I plan > to publish this second RFC by the end of the month. > Interesting. I plan on dropping another RFC close to the end of the month as well. > Our approaches have some common points (i.e. use eBPF in an LSM, stacked > filters like seccomp) but I'm focused on a kind of unprivileged LSM (i.e. no > CAP_SYS_ADMIN), to make standalone sandboxes, which brings more constraints > (e.g. no use of unsafe functions like bpf_probe_read(), take care of privacy, > SUID exec, stable ABI…). However, I don't want to handle resource limits, > which should be the job of cgroups. > Kind of. Sometimes describing these resource limits is difficult. For example, I have a customer who is trying to restrict containers from burning up all the ephemeral ports on the machine. In this, they have an incredibly elaborate chain of wiring to prevent a given container from connecting to the same (proto, destip, destport) more than 1000 times. I'm unsure of how you'd model that in a cgroup. > For now, I'm focusing on file-system access control which is one of the more > complex system to properly filter. I also plan to support basic network > access > control. > > What you are trying to accomplish seems more related to a Netfilter extension > (something like ipset but with eBPF maybe?). > I don't only want to do network access control, I also want to write to the value once it's copied into kernel space. There are lot of benefits of doing this at the syscall level, but the two primary ones are performance, and capability. One of the biggest complaints with our current approach to filtering & load balancing (iptables) is that it hides information. When people connect through the load balancer, they want to find out who they connected to, and without some high application-level mechanism, this isn't possible. On the other hand, if we just rewrite the destination address in the connect hook, we can pretty easily allow them to do getpeername. I'm curious about your filesystem access limiter. Do you have a way to make it so that a given container can only write, say, 100mb of data to disk? > Mickaël > > > [1] http://www.openwall.com/lists/kernel-hardening/2016/03/24/2 > > > > On 09/08/2016 02:22, Kees Cook wrote: > > On Mon, Aug 8, 2016 at 5:00 PM, Sargun Dhillon wrote: > >> On Mon, Aug 08, 2016 at 04:44:02PM -0700, Kees Cook wrote: > >>> On Thu, Aug 4, 2016 at 12:11 AM, Sargun Dhillon wrote: > I distributed this patchset to linux-security-mod...@vger.kernel.org > earlier, > but based on the fact that the archive is down, and this is a fairly > broad-sweeping proposal, I figured I'd grow the audience a little bit. > Sorry > if you received this multiple times. > > I've begun building out the skeleton of a Linux Security Module, and I'd > like to > get feedback on it. It's a skeleton, and I've only populated a few > hooks, so I'm > mostly looking for input on the general proposal, interest, and design. > It's a > minor LSM. My particular use case is one in which containers are being > dynamically deployed to machines by internal developers in a different > group. > The point of Checmate is to act as an extensible bed for _safe_, complex > security policies. It's nice to enable dynamic security policies that > can be > defined in C, and change as neccessary, without ever having to patch, or > rebuild > the kernel. > > For many of these containers, the security policies can be fairly > nuanced. One > particular one to take into account is network security. Often times, > administrators want to prevent ingress, and egress connectivity except > from a > few select IPs. Egress filtering can be managed using net_cls, but > without > modifying running software, it's non-trivial to attach a filter to all > sockets > being created within a container. The inet_conn_request, socket_recvmsg, > socket_sock_rcv_skb hooks make this trivial to implement. > > Other times, containers need to be throttled in places where there's not > really > a good place to impose that policy for software which isn't built > in-house. If > one wants to limit file creations/sec, or reject I/O under certain > characteristics, there's not a great place to do it now. This gives > engineers a > mechanism to write those policies. > > This same flexibility can be used to take existing programs and enable > safe BPF > helpers to modify memory to allow rules to pass. One example that I >
Re: [RFC 0/4] RFC: Add Checmate, BPF-driven minor LSM
On Mon, Aug 8, 2016 at 5:00 PM, Sargun Dhillon wrote: > On Mon, Aug 08, 2016 at 04:44:02PM -0700, Kees Cook wrote: >> On Thu, Aug 4, 2016 at 12:11 AM, Sargun Dhillon wrote: >> > I distributed this patchset to linux-security-mod...@vger.kernel.org >> > earlier, >> > but based on the fact that the archive is down, and this is a fairly >> > broad-sweeping proposal, I figured I'd grow the audience a little bit. >> > Sorry >> > if you received this multiple times. >> > >> > I've begun building out the skeleton of a Linux Security Module, and I'd >> > like to >> > get feedback on it. It's a skeleton, and I've only populated a few hooks, >> > so I'm >> > mostly looking for input on the general proposal, interest, and design. >> > It's a >> > minor LSM. My particular use case is one in which containers are being >> > dynamically deployed to machines by internal developers in a different >> > group. >> > The point of Checmate is to act as an extensible bed for _safe_, complex >> > security policies. It's nice to enable dynamic security policies that can >> > be >> > defined in C, and change as neccessary, without ever having to patch, or >> > rebuild >> > the kernel. >> > >> > For many of these containers, the security policies can be fairly nuanced. >> > One >> > particular one to take into account is network security. Often times, >> > administrators want to prevent ingress, and egress connectivity except >> > from a >> > few select IPs. Egress filtering can be managed using net_cls, but without >> > modifying running software, it's non-trivial to attach a filter to all >> > sockets >> > being created within a container. The inet_conn_request, socket_recvmsg, >> > socket_sock_rcv_skb hooks make this trivial to implement. >> > >> > Other times, containers need to be throttled in places where there's not >> > really >> > a good place to impose that policy for software which isn't built >> > in-house. If >> > one wants to limit file creations/sec, or reject I/O under certain >> > characteristics, there's not a great place to do it now. This gives >> > engineers a >> > mechanism to write those policies. >> > >> > This same flexibility can be used to take existing programs and enable >> > safe BPF >> > helpers to modify memory to allow rules to pass. One example that I >> > prototyped >> > was Docker's port mapping, which has an overhead (DNAT), and there's some >> > loss >> > of fidelity in the BSD Socket API to identify what's going on. Instead, we >> > can >> > just rewrite the port in a bind, based upon some data in a BPF map, and a >> > cgroup >> > match. >> > >> > I can actually see other minor security modules being implemented in >> > Checmate, >> > for example, Yama, or the recently proposed Hardchroot could be >> > reimplemented in >> > BPF. Potentially, they could even be API compatible. >> > >> > Although, at first, much of this sounds like seccomp, it's quite >> > different. For >> > one, what we can do in the security hooks is more complex (access to kernel >> > pointers). The other side of this is we can have effects on a system-wide, >> > or cgroup level. This also circumvents the need for CRIU-friendly policies. >> > >> > Lastly, the flexibility of this mechanism allows for prevention of security >> > vulnerabilities which are often complex in nature and require the >> > interaction >> > of multiple hooks (CVE-2014-9717 is a good example), and although ksplice, >> > and livepatch exist, they're not always easy to use, as compared to loading >> > a single bpf program across all kernels. >> > >> > The user-facing API is exposed via prctl as it's meant to be very simple >> > (at >> > least the kernel components). It only has three operations. For a given >> > security >> > hook, you can attach a BPF program to it, which will add it to the set of >> > programs that are executed over when the hook is hit. You can reset a hook, >> > which removes all program associated with a given hook, and you can set a >> > deny_reset flag on a hook to prevent anyone from resetting it. It's likely >> > that >> > an individual would want to set this in any production use case. >> >> One fairly serious problem that seccomp had to overcome was dealing >> with exec+setuid in the face of an attacker. The main example is "what >> if we refuse to allow a program to drop privileges via a filter rule?" >> For seccomp, no-new-privs was introduced for non-root users of >> seccomp. Programmatic syscall (or LSM) filters need to deal with this, >> and it's a bit ungainly. :) >> > Couldn't someone do the same with SELinux, or Apparmor? The "big" LSMs aren't defined programmatically by non-root users, so there is no risk of elevating privileges (they are already root). >> Also, if you have a prctl API that already has 3 operations, you might >> want to use a new syscall anyway. :) >> > Looking at other LSMs, they appear to expose their API via a virtual > filesystem, > or prctl. I followed the model of YA
Re: [RFC 0/4] RFC: Add Checmate, BPF-driven minor LSM
On Mon, Aug 08, 2016 at 04:44:02PM -0700, Kees Cook wrote: > On Thu, Aug 4, 2016 at 12:11 AM, Sargun Dhillon wrote: > > I distributed this patchset to linux-security-mod...@vger.kernel.org > > earlier, > > but based on the fact that the archive is down, and this is a fairly > > broad-sweeping proposal, I figured I'd grow the audience a little bit. Sorry > > if you received this multiple times. > > > > I've begun building out the skeleton of a Linux Security Module, and I'd > > like to > > get feedback on it. It's a skeleton, and I've only populated a few hooks, > > so I'm > > mostly looking for input on the general proposal, interest, and design. > > It's a > > minor LSM. My particular use case is one in which containers are being > > dynamically deployed to machines by internal developers in a different > > group. > > The point of Checmate is to act as an extensible bed for _safe_, complex > > security policies. It's nice to enable dynamic security policies that can be > > defined in C, and change as neccessary, without ever having to patch, or > > rebuild > > the kernel. > > > > For many of these containers, the security policies can be fairly nuanced. > > One > > particular one to take into account is network security. Often times, > > administrators want to prevent ingress, and egress connectivity except from > > a > > few select IPs. Egress filtering can be managed using net_cls, but without > > modifying running software, it's non-trivial to attach a filter to all > > sockets > > being created within a container. The inet_conn_request, socket_recvmsg, > > socket_sock_rcv_skb hooks make this trivial to implement. > > > > Other times, containers need to be throttled in places where there's not > > really > > a good place to impose that policy for software which isn't built in-house. > > If > > one wants to limit file creations/sec, or reject I/O under certain > > characteristics, there's not a great place to do it now. This gives > > engineers a > > mechanism to write those policies. > > > > This same flexibility can be used to take existing programs and enable safe > > BPF > > helpers to modify memory to allow rules to pass. One example that I > > prototyped > > was Docker's port mapping, which has an overhead (DNAT), and there's some > > loss > > of fidelity in the BSD Socket API to identify what's going on. Instead, we > > can > > just rewrite the port in a bind, based upon some data in a BPF map, and a > > cgroup > > match. > > > > I can actually see other minor security modules being implemented in > > Checmate, > > for example, Yama, or the recently proposed Hardchroot could be > > reimplemented in > > BPF. Potentially, they could even be API compatible. > > > > Although, at first, much of this sounds like seccomp, it's quite different. > > For > > one, what we can do in the security hooks is more complex (access to kernel > > pointers). The other side of this is we can have effects on a system-wide, > > or cgroup level. This also circumvents the need for CRIU-friendly policies. > > > > Lastly, the flexibility of this mechanism allows for prevention of security > > vulnerabilities which are often complex in nature and require the > > interaction > > of multiple hooks (CVE-2014-9717 is a good example), and although ksplice, > > and livepatch exist, they're not always easy to use, as compared to loading > > a single bpf program across all kernels. > > > > The user-facing API is exposed via prctl as it's meant to be very simple (at > > least the kernel components). It only has three operations. For a given > > security > > hook, you can attach a BPF program to it, which will add it to the set of > > programs that are executed over when the hook is hit. You can reset a hook, > > which removes all program associated with a given hook, and you can set a > > deny_reset flag on a hook to prevent anyone from resetting it. It's likely > > that > > an individual would want to set this in any production use case. > > One fairly serious problem that seccomp had to overcome was dealing > with exec+setuid in the face of an attacker. The main example is "what > if we refuse to allow a program to drop privileges via a filter rule?" > For seccomp, no-new-privs was introduced for non-root users of > seccomp. Programmatic syscall (or LSM) filters need to deal with this, > and it's a bit ungainly. :) > Couldn't someone do the same with SELinux, or Apparmor? > Also, if you have a prctl API that already has 3 operations, you might > want to use a new syscall anyway. :) > Looking at other LSMs, they appear to expose their API via a virtual filesystem, or prctl. I followed the model of YAMA. I think there may be two more operations (detach program, and mark a hook as append-only / read-only / disabled). It seems like overkill to implement my own syscall. > > On the BPF side of it, all that's involved in the work in progress is to > > move some of the tracing helpers into the shared help
Re: [RFC 0/4] RFC: Add Checmate, BPF-driven minor LSM
On Thu, Aug 4, 2016 at 12:11 AM, Sargun Dhillon wrote: > I distributed this patchset to linux-security-mod...@vger.kernel.org earlier, > but based on the fact that the archive is down, and this is a fairly > broad-sweeping proposal, I figured I'd grow the audience a little bit. Sorry > if you received this multiple times. > > I've begun building out the skeleton of a Linux Security Module, and I'd like > to > get feedback on it. It's a skeleton, and I've only populated a few hooks, so > I'm > mostly looking for input on the general proposal, interest, and design. It's a > minor LSM. My particular use case is one in which containers are being > dynamically deployed to machines by internal developers in a different group. > The point of Checmate is to act as an extensible bed for _safe_, complex > security policies. It's nice to enable dynamic security policies that can be > defined in C, and change as neccessary, without ever having to patch, or > rebuild > the kernel. > > For many of these containers, the security policies can be fairly nuanced. One > particular one to take into account is network security. Often times, > administrators want to prevent ingress, and egress connectivity except from a > few select IPs. Egress filtering can be managed using net_cls, but without > modifying running software, it's non-trivial to attach a filter to all sockets > being created within a container. The inet_conn_request, socket_recvmsg, > socket_sock_rcv_skb hooks make this trivial to implement. > > Other times, containers need to be throttled in places where there's not > really > a good place to impose that policy for software which isn't built in-house. > If > one wants to limit file creations/sec, or reject I/O under certain > characteristics, there's not a great place to do it now. This gives engineers > a > mechanism to write those policies. > > This same flexibility can be used to take existing programs and enable safe > BPF > helpers to modify memory to allow rules to pass. One example that I prototyped > was Docker's port mapping, which has an overhead (DNAT), and there's some loss > of fidelity in the BSD Socket API to identify what's going on. Instead, we can > just rewrite the port in a bind, based upon some data in a BPF map, and a > cgroup > match. > > I can actually see other minor security modules being implemented in Checmate, > for example, Yama, or the recently proposed Hardchroot could be reimplemented > in > BPF. Potentially, they could even be API compatible. > > Although, at first, much of this sounds like seccomp, it's quite different. > For > one, what we can do in the security hooks is more complex (access to kernel > pointers). The other side of this is we can have effects on a system-wide, > or cgroup level. This also circumvents the need for CRIU-friendly policies. > > Lastly, the flexibility of this mechanism allows for prevention of security > vulnerabilities which are often complex in nature and require the interaction > of multiple hooks (CVE-2014-9717 is a good example), and although ksplice, > and livepatch exist, they're not always easy to use, as compared to loading > a single bpf program across all kernels. > > The user-facing API is exposed via prctl as it's meant to be very simple (at > least the kernel components). It only has three operations. For a given > security > hook, you can attach a BPF program to it, which will add it to the set of > programs that are executed over when the hook is hit. You can reset a hook, > which removes all program associated with a given hook, and you can set a > deny_reset flag on a hook to prevent anyone from resetting it. It's likely > that > an individual would want to set this in any production use case. One fairly serious problem that seccomp had to overcome was dealing with exec+setuid in the face of an attacker. The main example is "what if we refuse to allow a program to drop privileges via a filter rule?" For seccomp, no-new-privs was introduced for non-root users of seccomp. Programmatic syscall (or LSM) filters need to deal with this, and it's a bit ungainly. :) Also, if you have a prctl API that already has 3 operations, you might want to use a new syscall anyway. :) > On the BPF side of it, all that's involved in the work in progress is to > move some of the tracing helpers into the shared helpers. For example, > it's very valuable to have access to current when enforcing a hook. > BPF programs also have access to maps, which somewhat works around > the need for security blobs in some cases. Just from a compatibility perspective, doesn't this end up exposing kernel structures to userspace? What happens when the structures change? And from a security perspective, programmatic examination of kernel structures means you can trivially leak kernel memory locations and contents. Resisting these sorts of leaks needs to be addressed too. This looks like a subset of kprobes but available to non-root users, which looks r
Re: [RFC 0/4] RFC: Add Checmate, BPF-driven minor LSM
On Thu, Aug 04, 2016 at 11:45:08AM +0200, Daniel Borkmann wrote:
> Hi Sargun,
>
> On 08/04/2016 09:11 AM, Sargun Dhillon wrote:
> [...]
> >[It's a] minor LSM. My particular use case is one in which containers are
> >being
> >dynamically deployed to machines by internal developers in a different group.
> [...]
> >For many of these containers, the security policies can be fairly nuanced.
> >One
> >particular one to take into account is network security. Often times,
> >administrators want to prevent ingress, and egress connectivity except from a
> >few select IPs. Egress filtering can be managed using net_cls, but without
> >modifying running software, it's non-trivial to attach a filter to all
> >sockets
> >being created within a container. The inet_conn_request, socket_recvmsg,
> >socket_sock_rcv_skb hooks make this trivial to implement.
>
> I'm not too familiar with LSMs, but afaik, when you install such policies they
> are effectively global, right? How would you install/manage such policies per
> container?
>
> On a quick glance, this would then be the job of the BPF proglet from the
> global
> hook, no? If yes, then the BPF contexts the BPF prog works with seem rather
> quite
> limited ...
You're right. They are global hooks. If you'd want the policy to be specific to
a given cgroup, or namespace, you'd have to introduce a level of indirection
through a prog_array, or some such. There are still cases (the CVE, and Docker
bind) case where you want global isolation. The other big aspect is being able
to implement application-specific LSMs without requiring kmods. (A la
hardchroot).
>
> +struct checmate_file_open_ctx {
> + struct file *file;
> + const struct cred *cred;
> +};
> +
> +struct checmate_task_create_ctx {
> + unsigned long clone_flags;
> +};
> +
> +struct checmate_task_free_ctx {
> + struct task_struct *task;
> +};
> +
> +struct checmate_socket_connect_ctx {
> + struct socket *sock;
> + struct sockaddr *address;
> + int addrlen;
> +};
>
> ... or are you using bpf_probe_read() in some way to walk 'current' to
> retrieve
> a namespace from there somehow? Like via nsproxy? But how you make sense of
> this
> for defining a per container policy?
In my prototype code, I'm using uts namespace + hostname, and I'm extracting
that via the bpf_probe_read walk. You're right, that's less than awesome. In
the
longer-term, I'd hope we'd be able to add a helper like bpf_current_in_cgroup
(a
la bpf_skb_in_cgroup). The idea is that we'd add enough helpers to avoid this.
I
can submit some more example BPF programs if that'd help. Off the top of my
head:
* current_in_cgroup
* introduce struct pid map
* introduce helpers to inspect common datatypes passed to the helper -- if you
look at something like the the net hooks, there aren't actually that many
datatypes being passed around
* Introduce an example top-level cgroup that maps cgroup -> tail_call into
other programs
>
> Do you see a way where we don't need to define so many different ctx each
> time?
>
> My other concern from a security PoV is that when using things like
> bpf_probe_read()
> we're dependent on kernel structs and there's a risk that when people migrate
> such
> policies that expectations break due to underlying structs changed. I see
> you've
> addressed that in patch 4 to place a small stone in the way, yeah kinda
> works. It's
> mostly a reminder that this is not stable ABI.
>
> Thanks,
> Daniel
Re: [RFC 0/4] RFC: Add Checmate, BPF-driven minor LSM
Hi Sargun,
On 08/04/2016 09:11 AM, Sargun Dhillon wrote:
[...]
[It's a] minor LSM. My particular use case is one in which containers are being
dynamically deployed to machines by internal developers in a different group.
[...]
For many of these containers, the security policies can be fairly nuanced. One
particular one to take into account is network security. Often times,
administrators want to prevent ingress, and egress connectivity except from a
few select IPs. Egress filtering can be managed using net_cls, but without
modifying running software, it's non-trivial to attach a filter to all sockets
being created within a container. The inet_conn_request, socket_recvmsg,
socket_sock_rcv_skb hooks make this trivial to implement.
I'm not too familiar with LSMs, but afaik, when you install such policies they
are effectively global, right? How would you install/manage such policies per
container?
On a quick glance, this would then be the job of the BPF proglet from the global
hook, no? If yes, then the BPF contexts the BPF prog works with seem rather
quite
limited ...
+struct checmate_file_open_ctx {
+ struct file *file;
+ const struct cred *cred;
+};
+
+struct checmate_task_create_ctx {
+ unsigned long clone_flags;
+};
+
+struct checmate_task_free_ctx {
+ struct task_struct *task;
+};
+
+struct checmate_socket_connect_ctx {
+ struct socket *sock;
+ struct sockaddr *address;
+ int addrlen;
+};
... or are you using bpf_probe_read() in some way to walk 'current' to retrieve
a namespace from there somehow? Like via nsproxy? But how you make sense of this
for defining a per container policy?
Do you see a way where we don't need to define so many different ctx each time?
My other concern from a security PoV is that when using things like
bpf_probe_read()
we're dependent on kernel structs and there's a risk that when people migrate
such
policies that expectations break due to underlying structs changed. I see you've
addressed that in patch 4 to place a small stone in the way, yeah kinda works.
It's
mostly a reminder that this is not stable ABI.
Thanks,
Daniel
Re: [RFC 0/4] RFC: Add Checmate, BPF-driven minor LSM
On Thu, Aug 04, 2016 at 10:41:17AM +0200, Richard Weinberger wrote: > Sargun, > > On Thu, Aug 4, 2016 at 9:11 AM, Sargun Dhillon wrote: > > I distributed this patchset to linux-security-mod...@vger.kernel.org > > earlier, > > but based on the fact that the archive is down, and this is a fairly > > broad-sweeping proposal, I figured I'd grow the audience a little bit. Sorry > > if you received this multiple times. > > > > I've begun building out the skeleton of a Linux Security Module, and I'd > > like to > > get feedback on it. It's a skeleton, and I've only populated a few hooks, > > so I'm > > mostly looking for input on the general proposal, interest, and design. > > It's a > > minor LSM. My particular use case is one in which containers are being > > dynamically deployed to machines by internal developers in a different > > group. > > The point of Checmate is to act as an extensible bed for _safe_, complex > > security policies. It's nice to enable dynamic security policies that can be > > defined in C, and change as neccessary, without ever having to patch, or > > rebuild > > the kernel. > > > > For many of these containers, the security policies can be fairly nuanced. > > One > > particular one to take into account is network security. Often times, > > administrators want to prevent ingress, and egress connectivity except from > > a > > few select IPs. Egress filtering can be managed using net_cls, but without > > modifying running software, it's non-trivial to attach a filter to all > > sockets > > being created within a container. The inet_conn_request, socket_recvmsg, > > socket_sock_rcv_skb hooks make this trivial to implement. > > What is wrong with having firewall rules per container? > Either by matching the container IP or an interface... > This requires infrastructure that's not always available. For one, this approach typically requires a network namespace per container, and therefore a dedicated IP. It's pretty common [1][2] to not have an IP/container solution, nor a network namespace per container solution. The alternatives to have a network namespace without IP/container typically involve bifurcating traffic using TC mirred actions, and friends. This isn't really great for debuggability. Twitter does this with their Mesos network isolator [3]. Cgroups / net_cls is great for egress traffic, but not ingress. > > Other times, containers need to be throttled in places where there's not > > really > > a good place to impose that policy for software which isn't built in-house. > > If > > one wants to limit file creations/sec, or reject I/O under certain > > characteristics, there's not a great place to do it now. This gives > > engineers a > > mechanism to write those policies. > > Hmm, not sure if resource control is something we want to do with an LSM. > This is just an example I brought up. I know of a fairly large security vendor that has abuse "patterns", and locks software down if it looks "abusive". They do it for VMs, but it'd be nice to do similar for containers. > > This same flexibility can be used to take existing programs and enable safe > > BPF > > helpers to modify memory to allow rules to pass. One example that I > > prototyped > > was Docker's port mapping, which has an overhead (DNAT), and there's some > > loss > > of fidelity in the BSD Socket API to identify what's going on. Instead, we > > can > > just rewrite the port in a bind, based upon some data in a BPF map, and a > > cgroup > > match. > > > > I can actually see other minor security modules being implemented in > > Checmate, > > for example, Yama, or the recently proposed Hardchroot could be > > reimplemented in > > BPF. Potentially, they could even be API compatible. > > > > Although, at first, much of this sounds like seccomp, it's quite different. > > For > > one, what we can do in the security hooks is more complex (access to kernel > > pointers). The other side of this is we can have effects on a system-wide, > > or cgroup level. This also circumvents the need for CRIU-friendly policies. > > It is like seccomp except that you have a single rule set and target LSM hooks > instead of syscalls, right? You're right, it's very similar. I like to think of Checmate as nftables for syscalls. It turns out having this on LSM hooks is a very big difference. Since LSM hooks are executed after data is copied to the kernel, you can safely dereference pointers and inspect the user's intentions. In one of the attached patches, I block traffic to AF_INET, port 1 -- there's no way to do that with seccomp(-bpf) today. This could also be used for things like filesystem path based filtering. You also have full access to the gambit of eBPF, as opposed to seccomp's cBPF. This allows you do to a variety of things, like write your programs in C, and compile them down to BPF via LLVM. You also have access to maps to share information between programs, and tail calls to chain together po
Re: [RFC 0/4] RFC: Add Checmate, BPF-driven minor LSM
Sargun, On Thu, Aug 4, 2016 at 9:11 AM, Sargun Dhillon wrote: > I distributed this patchset to linux-security-mod...@vger.kernel.org earlier, > but based on the fact that the archive is down, and this is a fairly > broad-sweeping proposal, I figured I'd grow the audience a little bit. Sorry > if you received this multiple times. > > I've begun building out the skeleton of a Linux Security Module, and I'd like > to > get feedback on it. It's a skeleton, and I've only populated a few hooks, so > I'm > mostly looking for input on the general proposal, interest, and design. It's a > minor LSM. My particular use case is one in which containers are being > dynamically deployed to machines by internal developers in a different group. > The point of Checmate is to act as an extensible bed for _safe_, complex > security policies. It's nice to enable dynamic security policies that can be > defined in C, and change as neccessary, without ever having to patch, or > rebuild > the kernel. > > For many of these containers, the security policies can be fairly nuanced. One > particular one to take into account is network security. Often times, > administrators want to prevent ingress, and egress connectivity except from a > few select IPs. Egress filtering can be managed using net_cls, but without > modifying running software, it's non-trivial to attach a filter to all sockets > being created within a container. The inet_conn_request, socket_recvmsg, > socket_sock_rcv_skb hooks make this trivial to implement. What is wrong with having firewall rules per container? Either by matching the container IP or an interface... > Other times, containers need to be throttled in places where there's not > really > a good place to impose that policy for software which isn't built in-house. > If > one wants to limit file creations/sec, or reject I/O under certain > characteristics, there's not a great place to do it now. This gives engineers > a > mechanism to write those policies. Hmm, not sure if resource control is something we want to do with an LSM. > This same flexibility can be used to take existing programs and enable safe > BPF > helpers to modify memory to allow rules to pass. One example that I prototyped > was Docker's port mapping, which has an overhead (DNAT), and there's some loss > of fidelity in the BSD Socket API to identify what's going on. Instead, we can > just rewrite the port in a bind, based upon some data in a BPF map, and a > cgroup > match. > > I can actually see other minor security modules being implemented in Checmate, > for example, Yama, or the recently proposed Hardchroot could be reimplemented > in > BPF. Potentially, they could even be API compatible. > > Although, at first, much of this sounds like seccomp, it's quite different. > For > one, what we can do in the security hooks is more complex (access to kernel > pointers). The other side of this is we can have effects on a system-wide, > or cgroup level. This also circumvents the need for CRIU-friendly policies. It is like seccomp except that you have a single rule set and target LSM hooks instead of syscalls, right? -- Thanks, //richard
[RFC 0/4] RFC: Add Checmate, BPF-driven minor LSM
I distributed this patchset to linux-security-mod...@vger.kernel.org earlier, but based on the fact that the archive is down, and this is a fairly broad-sweeping proposal, I figured I'd grow the audience a little bit. Sorry if you received this multiple times. I've begun building out the skeleton of a Linux Security Module, and I'd like to get feedback on it. It's a skeleton, and I've only populated a few hooks, so I'm mostly looking for input on the general proposal, interest, and design. It's a minor LSM. My particular use case is one in which containers are being dynamically deployed to machines by internal developers in a different group. The point of Checmate is to act as an extensible bed for _safe_, complex security policies. It's nice to enable dynamic security policies that can be defined in C, and change as neccessary, without ever having to patch, or rebuild the kernel. For many of these containers, the security policies can be fairly nuanced. One particular one to take into account is network security. Often times, administrators want to prevent ingress, and egress connectivity except from a few select IPs. Egress filtering can be managed using net_cls, but without modifying running software, it's non-trivial to attach a filter to all sockets being created within a container. The inet_conn_request, socket_recvmsg, socket_sock_rcv_skb hooks make this trivial to implement. Other times, containers need to be throttled in places where there's not really a good place to impose that policy for software which isn't built in-house. If one wants to limit file creations/sec, or reject I/O under certain characteristics, there's not a great place to do it now. This gives engineers a mechanism to write those policies. This same flexibility can be used to take existing programs and enable safe BPF helpers to modify memory to allow rules to pass. One example that I prototyped was Docker's port mapping, which has an overhead (DNAT), and there's some loss of fidelity in the BSD Socket API to identify what's going on. Instead, we can just rewrite the port in a bind, based upon some data in a BPF map, and a cgroup match. I can actually see other minor security modules being implemented in Checmate, for example, Yama, or the recently proposed Hardchroot could be reimplemented in BPF. Potentially, they could even be API compatible. Although, at first, much of this sounds like seccomp, it's quite different. For one, what we can do in the security hooks is more complex (access to kernel pointers). The other side of this is we can have effects on a system-wide, or cgroup level. This also circumvents the need for CRIU-friendly policies. Lastly, the flexibility of this mechanism allows for prevention of security vulnerabilities which are often complex in nature and require the interaction of multiple hooks (CVE-2014-9717 is a good example), and although ksplice, and livepatch exist, they're not always easy to use, as compared to loading a single bpf program across all kernels. The user-facing API is exposed via prctl as it's meant to be very simple (at least the kernel components). It only has three operations. For a given security hook, you can attach a BPF program to it, which will add it to the set of programs that are executed over when the hook is hit. You can reset a hook, which removes all program associated with a given hook, and you can set a deny_reset flag on a hook to prevent anyone from resetting it. It's likely that an individual would want to set this in any production use case. On the BPF side of it, all that's involved in the work in progress is to move some of the tracing helpers into the shared helpers. For example, it's very valuable to have access to current when enforcing a hook. BPF programs also have access to maps, which somewhat works around the need for security blobs in some cases. I would love to know what y'all think. Sargun Dhillon (4): bpf: move tracing helpers to shared helpers bpf, security: Add Checmate security/checmate: Add Checmate sample bpf: Restrict Checmate bpf programs to current kernel ABI include/linux/bpf.h | 2 + include/linux/checmate.h | 38 + include/uapi/linux/Kbuild| 1 + include/uapi/linux/bpf.h | 1 + include/uapi/linux/checmate.h| 65 + include/uapi/linux/prctl.h | 3 + kernel/bpf/helpers.c | 34 + kernel/bpf/syscall.c | 2 +- kernel/trace/bpf_trace.c | 33 - samples/bpf/Makefile | 4 + samples/bpf/bpf_load.c | 11 +- samples/bpf/checmate1_kern.c | 28 samples/bpf/checmate1_user.c | 54 +++ security/Kconfig | 1 + security/Makefile| 2 + security/checmate/Kconfig| 6 + security/checmate/Makefile | 3 + security/checmate/checmate_bpf.c | 67 + security/checmate/checmate_lsm.c | 304 +++
