Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Christian Brauner]

On Thu, Jun 11, 2020 at 05:17:09PM -0700, Matt Helsley wrote:
> On Fri, May 22, 2020 at 07:53:50AM +0200, Adrian Reber wrote:
> > This enables CRIU to checkpoint and restore a process as non-root.
> > 
> > Over the last years CRIU upstream has been asked a couple of time if it
> > is possible to checkpoint and restore a process as non-root. The answer
> > usually was: 'almost'.
> > 
> > The main blocker to restore a process was that selecting the PID of the
> > restored process, which is necessary for CRIU, is guarded by CAP_SYS_ADMIN.
> > 
> > In the last two years the questions about checkpoint/restore as non-root
> > have increased and especially in the last few months we have seen
> > multiple people inventing workarounds.
> > 
> > The use-cases so far and their workarounds:
> > 
> >  * Checkpoint/Restore in an HPC environment in combination with
> >a resource manager distributing jobs. Users are always running
> >as non root, but there was the desire to provide a way to
> >checkpoint and restore long running jobs.
> >Workaround: setuid wrapper to start CRIU as root as non-root
> >
> > https://github.com/FredHutch/slurm-examples/blob/master/checkpointer/lib/checkpointer/checkpointer-suid.c
> >  * Another use case to checkpoint/restore processes as non-root
> >uses as workaround a non privileged process which cycles through
> >PIDs by calling fork() as fast as possible with a rate of
> >100,000 pids/s instead of writing to ns_last_pid
> >https://github.com/twosigma/set_ns_last_pid
> >  * Fast Java startup using checkpoint/restore.
> >We have been in contact with JVM developers who are integrating
> >CRIU into a JVM to decrease the startup time.
> >Workaround so far: patch out CAP_SYS_ADMIN checks in the kernel
> >  * Container migration as non root. There are people already
> >using CRIU to migrate containers as non-root. The solution
> >there is to run it in a user namespace. So if you are able
> >to carefully setup your environment with the namespaces
> >it is already possible to restore a container/process as non-root.
> >Unfortunately it is not always possible to setup an environment
> >in such a way and for easier access to non-root based container
> >migration this patch is also required.
> > 
> > There are probably a few more things guarded by CAP_SYS_ADMIN required
> > to run checkpoint/restore as non-root, but by applying this patch I can
> > already checkpoint and restore processes as non-root. As there are
> > already multiple workarounds I would prefer to do it correctly in the
> > kernel to avoid that CRIU users are starting to invent more workarounds.
> > 
> > I have used the following tests to verify that this change works as
> > expected by setting the new capability CAP_RESTORE on the two resulting
> > test binaries:
> > 
> > $ cat ns_last_pid.c
> >  // 
> > http://efiop-notes.blogspot.com/2014/06/how-to-set-pid-using-nslastpid.html
> >  #include 
> >  #include 
> >  #include 
> >  #include 
> >  #include 
> >  #include 
> > 
> > int main(int argc, char *argv[])
> > {
> > pid_t pid, new_pid;
> > char buf[32];
> > int fd;
> > 
> > if (argc != 2)
> > return 1;
> > 
> > printf("Opening ns_last_pid...\n");
> > fd = open("/proc/sys/kernel/ns_last_pid", O_RDWR | O_CREAT, 0644);
> > if (fd < 0) {
> > perror("Cannot open ns_last_pid");
> > return 1;
> > }
> > 
> > printf("Locking ns_last_pid...\n");
> > if (flock(fd, LOCK_EX)) {
> > close(fd);
> > printf("Cannot lock ns_last_pid\n");
> > return 1;
> > }
> > 
> > pid = atoi(argv[1]);
> > snprintf(buf, sizeof(buf), "%d", pid - 1);
> > printf("Writing pid-1 to ns_last_pid...\n");
> > if (write(fd, buf, strlen(buf)) != strlen(buf)) {
> > printf("Cannot write to buf\n");
> > return 1;
> > }
> > 
> > printf("Forking...\n");
> > new_pid = fork();
> > if (new_pid == 0) {
> > printf("I am the child!\n");
> > exit(0);
> > } else if (new_pid == pid)
> > printf("I am the parent. My child got the pid %d!\n", new_pid);
> > else
> > printf("pid (%d) does not match expected pid (%d)\n", new_pid, 
> > pid);
> > 
> > printf("Cleaning up...\n");
> > if (flock(fd, LOCK_UN))
> > printf("Cannot unlock\n");
> > close(fd);
> > return 0;
> > }
> > $ id -u; /home/libcap/ns_last_pid 30
> > 1001
> > Opening ns_last_pid...
> > Locking ns_last_pid...
> > Writing pid-1 to ns_last_pid...
> > Forking...
> > I am the parent. My child got the pid 30!
> > I am the child!
> > Cleaning up...
> > 
> > For the clone3() based approach:
> > $ cat clone3_set_tid.c
> >  #define _GNU_SOURCE
> >  #include 
> >  #include 
> >  #include 
> >  #include 
> >  #include 
> >  #include 
> >  #include 
> >  #include 
> >  #include 
> > 
> >  #define 

Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Matt Helsley]

On Fri, May 22, 2020 at 07:53:50AM +0200, Adrian Reber wrote:
> This enables CRIU to checkpoint and restore a process as non-root.
> 
> Over the last years CRIU upstream has been asked a couple of time if it
> is possible to checkpoint and restore a process as non-root. The answer
> usually was: 'almost'.
> 
> The main blocker to restore a process was that selecting the PID of the
> restored process, which is necessary for CRIU, is guarded by CAP_SYS_ADMIN.
> 
> In the last two years the questions about checkpoint/restore as non-root
> have increased and especially in the last few months we have seen
> multiple people inventing workarounds.
> 
> The use-cases so far and their workarounds:
> 
>  * Checkpoint/Restore in an HPC environment in combination with
>a resource manager distributing jobs. Users are always running
>as non root, but there was the desire to provide a way to
>checkpoint and restore long running jobs.
>Workaround: setuid wrapper to start CRIU as root as non-root
>
> https://github.com/FredHutch/slurm-examples/blob/master/checkpointer/lib/checkpointer/checkpointer-suid.c
>  * Another use case to checkpoint/restore processes as non-root
>uses as workaround a non privileged process which cycles through
>PIDs by calling fork() as fast as possible with a rate of
>100,000 pids/s instead of writing to ns_last_pid
>https://github.com/twosigma/set_ns_last_pid
>  * Fast Java startup using checkpoint/restore.
>We have been in contact with JVM developers who are integrating
>CRIU into a JVM to decrease the startup time.
>Workaround so far: patch out CAP_SYS_ADMIN checks in the kernel
>  * Container migration as non root. There are people already
>using CRIU to migrate containers as non-root. The solution
>there is to run it in a user namespace. So if you are able
>to carefully setup your environment with the namespaces
>it is already possible to restore a container/process as non-root.
>Unfortunately it is not always possible to setup an environment
>in such a way and for easier access to non-root based container
>migration this patch is also required.
> 
> There are probably a few more things guarded by CAP_SYS_ADMIN required
> to run checkpoint/restore as non-root, but by applying this patch I can
> already checkpoint and restore processes as non-root. As there are
> already multiple workarounds I would prefer to do it correctly in the
> kernel to avoid that CRIU users are starting to invent more workarounds.
> 
> I have used the following tests to verify that this change works as
> expected by setting the new capability CAP_RESTORE on the two resulting
> test binaries:
> 
> $ cat ns_last_pid.c
>  // 
> http://efiop-notes.blogspot.com/2014/06/how-to-set-pid-using-nslastpid.html
>  #include 
>  #include 
>  #include 
>  #include 
>  #include 
>  #include 
> 
> int main(int argc, char *argv[])
> {
>   pid_t pid, new_pid;
>   char buf[32];
>   int fd;
> 
>   if (argc != 2)
>   return 1;
> 
>   printf("Opening ns_last_pid...\n");
>   fd = open("/proc/sys/kernel/ns_last_pid", O_RDWR | O_CREAT, 0644);
>   if (fd < 0) {
>   perror("Cannot open ns_last_pid");
>   return 1;
>   }
> 
>   printf("Locking ns_last_pid...\n");
>   if (flock(fd, LOCK_EX)) {
>   close(fd);
>   printf("Cannot lock ns_last_pid\n");
>   return 1;
>   }
> 
>   pid = atoi(argv[1]);
>   snprintf(buf, sizeof(buf), "%d", pid - 1);
>   printf("Writing pid-1 to ns_last_pid...\n");
>   if (write(fd, buf, strlen(buf)) != strlen(buf)) {
>   printf("Cannot write to buf\n");
>   return 1;
>   }
> 
>   printf("Forking...\n");
>   new_pid = fork();
>   if (new_pid == 0) {
>   printf("I am the child!\n");
>   exit(0);
>   } else if (new_pid == pid)
>   printf("I am the parent. My child got the pid %d!\n", new_pid);
>   else
>   printf("pid (%d) does not match expected pid (%d)\n", new_pid, 
> pid);
> 
>   printf("Cleaning up...\n");
>   if (flock(fd, LOCK_UN))
>   printf("Cannot unlock\n");
>   close(fd);
>   return 0;
> }
> $ id -u; /home/libcap/ns_last_pid 30
> 1001
> Opening ns_last_pid...
> Locking ns_last_pid...
> Writing pid-1 to ns_last_pid...
> Forking...
> I am the parent. My child got the pid 30!
> I am the child!
> Cleaning up...
> 
> For the clone3() based approach:
> $ cat clone3_set_tid.c
>  #define _GNU_SOURCE
>  #include 
>  #include 
>  #include 
>  #include 
>  #include 
>  #include 
>  #include 
>  #include 
>  #include 
> 
>  #define ptr_to_u64(ptr) ((__u64)((uintptr_t)(ptr)))
> 
> int main(int argc, char *argv[])
> {
>   struct clone_args c_args = { };
>   pid_t pid, new_pid;
> 
>   if (argc != 2)
>   return 1;
> 
>   pid = atoi(argv[1]);
>   c_args.set_tid = 

Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Andrei Vagin]

> >
> > I would argue that setting the current process exe file check should just 
> > be reduced to a "can you ptrace a children" check.
> > Here's why: any process can masquerade into another executable with ptrace.
> > One can fork a child, ptrace it, have the child execve("target_exe"), then 
> > replace its memory content with an arbitrary program.
>
> Then it should probably be relaxed to CAP_SYS_PTRACE in the user
> namespace and not CAP_CHECKPOINT_RESTORE. (But apparently you also have
> a way of achieving what you want anyway. Imho, it's not necessarily
> wrong to require a bit more work when you want something like fully
> unprivileged c/r that's a rather special interest group.)
>
> > With CRIU's libcompel parasite mechanism (https://criu.org/Compel) this is 
> > fairly easy to implement.
> > In fact, we could modify CRIU to do just that (but with a fair amount of 
> > efforts due to the way CRIU is written),
> > and not rely on being able to SET_MM_EXE_FILE via prctl(). In turn, that 
> > would give an easy way to masquerade any process
> > into another one, provided that one can ptrace a child.
> >

I think you misunderstand this. In the case of malicious processes,
when only one or two processes must be hidden, they can use this trick
with execve+ptrace and this is relatively simple. But in the case of
CRIU, where we need to restore a process tree with cow memory
mappings, shared mappings, file descriptors and etc, this trick with
execve+ptrace doesn't work at all. We are in a weird situation when
malicious processes can do some operations, but useful tools like CRIU
needed to be running with extra capabilities that actually reduces the
security of the entire system.

Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Christian Brauner]

On Wed, May 27, 2020 at 06:05:55PM +, Nicolas Viennot wrote:
> > > Also in this thread Kamil mentioned that they also need calling prctl 
> > > with PR_SET_MM during restore in their production setup.
> >
> > We're using that as well but it really feels like this:
> >
> > prctl_map = (struct prctl_mm_map){
> > .start_code = start_code,
> > .end_code = end_code,
> > .start_stack = start_stack,
> > .start_data = start_data,
> > .end_data = end_data,
> > .start_brk = start_brk,
> > .brk = brk_val,
> > .arg_start = arg_start,
> > .arg_end = arg_end,
> > .env_start = env_start,
> > .env_end = env_end,
> > .auxv = NULL,
> > .auxv_size = 0,
> > .exe_fd = -1,
> > };
> >
> > should belong under ns_capable(CAP_SYS_ADMIN). Why is that necessary to 
> > relax?
> 
> When the prctl(PR_SET_MM_MAP...), the only privileged operation is to change 
> the symlink of /proc/self/exe via set_mm_exe_file().
> See 
> https://github.com/torvalds/linux/blob/444fc5cde64330661bf59944c43844e7d4c2ccd8/kernel/sys.c#L2001-L2004
> It needs CAP_SYS_ADMIN of the current namespace.

This already has been relaxed before (see commit below) and why I'm at
least symbolically pushing back is that I'm getting worried that we're
just removing restrictions left and right and making kernel interfaces
available to fully unprivileged user that very much seem to belong into
the realm of local cap_sys_admin.

prctl: Allow local CAP_SYS_ADMIN changing exe_file

During checkpointing and restore of userspace tasks
we bumped into the situation, that it's not possible
to restore the tasks, which user namespace does not
have uid 0 or gid 0 mapped.

People create user namespace mappings like they want,
and there is no a limitation on obligatory uid and gid
"must be mapped". So, if there is no uid 0 or gid 0
in the mapping, it's impossible to restore mm->exe_file
of the processes belonging to this user namespace.

Also, there is no a workaround. It's impossible
to create a temporary uid/gid mapping, because
only one write to /proc/[pid]/uid_map and gid_map
is allowed during a namespace lifetime.
If there is an entry, then no more mapings can't be
written. If there isn't an entry, we can't write
there too, otherwise user task won't be able
to do that in the future.

The patch changes the check, and looks for CAP_SYS_ADMIN
instead of zero uid and gid. This allows to restore
a task independently of its user namespace mappings.

> 
> I would argue that setting the current process exe file check should just be 
> reduced to a "can you ptrace a children" check.
> Here's why: any process can masquerade into another executable with ptrace.
> One can fork a child, ptrace it, have the child execve("target_exe"), then 
> replace its memory content with an arbitrary program.

Then it should probably be relaxed to CAP_SYS_PTRACE in the user
namespace and not CAP_CHECKPOINT_RESTORE. (But apparently you also have
a way of achieving what you want anyway. Imho, it's not necessarily
wrong to require a bit more work when you want something like fully
unprivileged c/r that's a rather special interest group.)

> With CRIU's libcompel parasite mechanism (https://criu.org/Compel) this is 
> fairly easy to implement.
> In fact, we could modify CRIU to do just that (but with a fair amount of 
> efforts due to the way CRIU is written),
> and not rely on being able to SET_MM_EXE_FILE via prctl(). In turn, that 
> would give an easy way to masquerade any process
> into another one, provided that one can ptrace a child.
> 
> When not using PR_SET_MM_MAP, but using SET_MM_EXE_FILE, the CAP_RESOURCES at 
> the root namespace level is required:
> https://github.com/torvalds/linux/blob/444fc5cde64330661bf59944c43844e7d4c2ccd8/kernel/sys.c#L2109
> This seems inconsistent. Also for some reason changing auxv is not privileged 
> if using prctl via the MM_MAP mechanism, but is privileged otherwise.

Fwiw, it always helps if people take the time to dig through the history
of specifc changes. That usually helps explain why things ended up as
confusing as they are now:

commit f606b77f1a9e362451aca8f81d8f36a3a112139e
Author: Cyrill Gorcunov 
Date:   Thu Oct 9 15:27:37 2014 -0700

prctl: PR_SET_MM -- introduce PR_SET_MM_MAP operation

During development of c/r we've noticed that in case if we need to 
support
user namespaces we face a problem with capabilities in 
prctl(PR_SET_MM,
...) call, in particular once new user namespace is created
capable(CAP_SYS_RESOURCE) no longer passes.

A approach is to eliminate CAP_SYS_RESOURCE check but pass all new 
values
in one bundle, which would allow the kernel to make more intensive 
test

[snip]

Still note that 

RE: [PATCH] capabilities: Introduce CAP_RESTORE

[Nicolas Viennot]

> > Also in this thread Kamil mentioned that they also need calling prctl 
> > with PR_SET_MM during restore in their production setup.
>
> We're using that as well but it really feels like this:
>
>   prctl_map = (struct prctl_mm_map){
>   .start_code = start_code,
>   .end_code = end_code,
>   .start_stack = start_stack,
>   .start_data = start_data,
>   .end_data = end_data,
>   .start_brk = start_brk,
>   .brk = brk_val,
>   .arg_start = arg_start,
>   .arg_end = arg_end,
>   .env_start = env_start,
>   .env_end = env_end,
>   .auxv = NULL,
>   .auxv_size = 0,
>   .exe_fd = -1,
>   };
>
> should belong under ns_capable(CAP_SYS_ADMIN). Why is that necessary to relax?

When the prctl(PR_SET_MM_MAP...), the only privileged operation is to change 
the symlink of /proc/self/exe via set_mm_exe_file().
See 
https://github.com/torvalds/linux/blob/444fc5cde64330661bf59944c43844e7d4c2ccd8/kernel/sys.c#L2001-L2004
It needs CAP_SYS_ADMIN of the current namespace.

I would argue that setting the current process exe file check should just be 
reduced to a "can you ptrace a children" check.
Here's why: any process can masquerade into another executable with ptrace.
One can fork a child, ptrace it, have the child execve("target_exe"), then 
replace its memory content with an arbitrary program.
With CRIU's libcompel parasite mechanism (https://criu.org/Compel) this is 
fairly easy to implement.
In fact, we could modify CRIU to do just that (but with a fair amount of 
efforts due to the way CRIU is written),
and not rely on being able to SET_MM_EXE_FILE via prctl(). In turn, that would 
give an easy way to masquerade any process
into another one, provided that one can ptrace a child.

When not using PR_SET_MM_MAP, but using SET_MM_EXE_FILE, the CAP_RESOURCES at 
the root namespace level is required:
https://github.com/torvalds/linux/blob/444fc5cde64330661bf59944c43844e7d4c2ccd8/kernel/sys.c#L2109
This seems inconsistent. Also for some reason changing auxv is not privileged 
if using prctl via the MM_MAP mechanism, but is privileged otherwise.

Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Casey Schaufler]

On 5/27/2020 9:37 AM, Nicolas Viennot wrote:
>>> If I understand part of CRIU correctly, then we only need read-access 
>>> for the current user. I am sure Andrei, Pavel or Cyrill will correct 
>>> me if I am wrong concerning map_files.
>> If I do "ls -l /proc/self/map_files" I get the link name and link content.
>> While I can't open /proc/self/map_files/7fbde0c3200-7fbde0c3300 I can read 
>> that it points to /usr/lib64/ld-2.30.so, which is something I can open and 
>> read. Sure, it's an extra step, but it's no big deal. It does raise the 
>> question of what value comes from disallowing open via the symlink.
> Reading the symlink doesn't work in two cases:
> 1) The file has been deleted

In which case you won't be able to read it directly from
the symlink, either.

> 2) The file is a memfd file

Ditto? Or is there some other problem?

RE: [PATCH] capabilities: Introduce CAP_RESTORE

[Nicolas Viennot]

> > If I understand part of CRIU correctly, then we only need read-access 
> > for the current user. I am sure Andrei, Pavel or Cyrill will correct 
> > me if I am wrong concerning map_files.
> If I do "ls -l /proc/self/map_files" I get the link name and link content.
> While I can't open /proc/self/map_files/7fbde0c3200-7fbde0c3300 I can read 
> that it points to /usr/lib64/ld-2.30.so, which is something I can open and 
> read. Sure, it's an extra step, but it's no big deal. It does raise the 
> question of what value comes from disallowing open via the symlink.

Reading the symlink doesn't work in two cases:
1) The file has been deleted
2) The file is a memfd file

Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Casey Schaufler]

On 5/27/2020 6:48 AM, Adrian Reber wrote:
> On Mon, May 25, 2020 at 11:55:20AM -0700, Casey Schaufler wrote:
>> On 5/25/2020 1:05 AM, Adrian Reber wrote:
>>> On Fri, May 22, 2020 at 09:40:37AM -0700, Casey Schaufler wrote:
 On 5/21/2020 10:53 PM, Adrian Reber wrote:
> This enables CRIU to checkpoint and restore a process as non-root.
 I know it sounds pedantic, but could you spell out CRIU once?
 While I know that everyone who cares either knows or can guess
 what you're talking about, it may be a mystery to some of the
 newer kernel developers.
>>> Sure. CRIU - Checkpoint/Restore In Userspace.
>> Thanks. I blew out my acronym processor in the 1990's while
>> working on trusted Unix system security evaluations.
>>
> Over the last years CRIU upstream has been asked a couple of time if it
> is possible to checkpoint and restore a process as non-root. The answer
> usually was: 'almost'.
>
> The main blocker to restore a process was that selecting the PID of the
> restored process, which is necessary for CRIU, is guarded by 
> CAP_SYS_ADMIN.
 What are the other blockers? Are you going to suggest additional new
 capabilities to clear them?
>>> As mentioned somewhere else access to /proc//map_files/ would be
>>> helpful. Right now I am testing with a JVM and it works without root
>>> just with the attached patch. Without access to /proc//map_files/
>>> not everything CRIU can do will actually work, but we are a lot closer
>>> to what our users have been asking for.
>> Are you talking about read access to map_files owned by other users
>> or write access to map_files for the current user?
> If I understand part of CRIU correctly, then we only need read-access
> for the current user. I am sure Andrei, Pavel or Cyrill will correct me
> if I am wrong concerning map_files.

If I do "ls -l /proc/self/map_files" I get the link name and link content.
While I can't open /proc/self/map_files/7fbde0c3200-7fbde0c3300 I can read
that it points to /usr/lib64/ld-2.30.so, which is something I can open 
and read. Sure, it's an extra step, but it's no big deal. It does raise the
question of what value comes from disallowing open via the symlink.

Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Christian Brauner]

On Wed, May 27, 2020 at 04:14:03PM +0200, Adrian Reber wrote:
> On Tue, May 26, 2020 at 08:59:29AM -0500, Eric W. Biederman wrote:
> > Adrian Reber  writes:
> > 
> > > On Fri, May 22, 2020 at 09:40:37AM -0700, Casey Schaufler wrote:
> > 
> > >> What are the other blockers? Are you going to suggest additional new
> > >> capabilities to clear them?
> > >
> > > As mentioned somewhere else access to /proc//map_files/ would be
> > > helpful. Right now I am testing with a JVM and it works without root
> > > just with the attached patch. Without access to /proc//map_files/
> > > not everything CRIU can do will actually work, but we are a lot closer
> > > to what our users have been asking for.
> > 
> > The current permission checks on /proc//map_files/ are simply
> > someone being over-cautious.
> > 
> > Someone needs to think through the threat landscape and figure out what
> > permission checks are actually needed.
> > 
> > Making the permission check ns_capable instead of capable is a
> > no-brainer.  Figuring out which user_ns to test against might be a
> > we bit harder.
> > 
> > We could probably even allow the owner of the process to open the files
> > but that requires someone doing the work of thinking through how
> > being able to opening files that you have mmaped might be a problem.
> 
> As mentioned in the other thread, CRIU can work with read access to
> map_files.
> 
> > >> > There are probably a few more things guarded by CAP_SYS_ADMIN required
> > >> > to run checkpoint/restore as non-root,
> > >> 
> > >> If you need CAP_SYS_ADMIN anyway you're not gaining anything by
> > >> separating out CAP_RESTORE.
> > >
> > > No, as described we can checkpoint and restore a JVM with this patch and
> > > it also solves the problem the set_ns_last_pid fork() loop daemon tries
> > > to solve. It is not enough to support the full functionality of CRIU as
> > > map_files is also important, but we do not need CAP_SYS_ADMIN and
> > > CAP_RESTORE. Only CAP_RESTORE would be necessary.
> > >
> > > With a new capability users can enable checkpoint/restore as non-root
> > > without giving CRIU access to any of the other possibilities offered by
> > > CAP_SYS_ADMIN. Setting a PID and map_files have been introduced for CRIU
> > > and used to live behind CONFIG_CHECKPOINT_RESTORE. Having a capability
> > > for checkpoint/restore would make it easier for CRIU users to run it as
> > > non-root and make it very clear what is possible when giving CRIU the
> > > new capability. No other things would be allowed than necessary for
> > > checkpoint/restore. Setting a PID is most important for the restore part
> > > and reading map_files would be helpful during checkpoint. So it actually
> > > should be called CAP_CHECKPOINT_RESTORE as Christian mentioned in
> > > another email.
> > 
> > Please if one is for checkpoint and one is for restore asking for a pair
> > of capabilities is probably more appropriate.
> 
> I will send out a v2 with a renamed capability soon and also include
> map_files to be readable with that capability.
> 
> > >> >  but by applying this patch I can
> > >> > already checkpoint and restore processes as non-root. As there are
> > >> > already multiple workarounds I would prefer to do it correctly in the
> > >> > kernel to avoid that CRIU users are starting to invent more 
> > >> > workarounds.
> > >> 
> > >> You've presented a couple of really inappropriate implementations
> > >> that would qualify as workarounds. But the other two are completely
> > >> appropriate within the system security policy. They don't "get around"
> > >> the problem, they use existing mechanisms as they are intended.
> > >
> > > I agree with the user namespace approach to be appropriate, but not the
> > > CAP_SYS_ADMIN approach as CRIU only needs a tiny subset (2 things) of
> > > what CAP_SYS_ADMIN allows.
> > 
> > 
> > If we are only talking 2 things can you please include in your patchset
> > a patch enabling those 2 things?
> 
> The two things are setting a PID via ns_last_pid/clone3() and reading
> map_files.
> 
> > But even more than this we need a request that asks not for the least
> > you can possibly ask for but asks for what you need to do a good job.
> 
> Also in this thread Kamil mentioned that they also need calling prctl
> with PR_SET_MM during restore in their production setup.

We're using that as well but it really feels like this:

prctl_map = (struct prctl_mm_map){
.start_code = start_code,
.end_code = end_code,
.start_stack = start_stack,
.start_data = start_data,
.end_data = end_data,
.start_brk = start_brk,
.brk = brk_val,
.arg_start = arg_start,
.arg_end = arg_end,
.env_start = env_start,
.env_end = env_end,
.auxv = NULL,
.auxv_size = 0,
.exe_fd = -1,
};

should belong under ns_capable(CAP_SYS_ADMIN). Why is that necessary to

Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Adrian Reber]

On Tue, May 26, 2020 at 08:59:29AM -0500, Eric W. Biederman wrote:
> Adrian Reber  writes:
> 
> > On Fri, May 22, 2020 at 09:40:37AM -0700, Casey Schaufler wrote:
> 
> >> What are the other blockers? Are you going to suggest additional new
> >> capabilities to clear them?
> >
> > As mentioned somewhere else access to /proc//map_files/ would be
> > helpful. Right now I am testing with a JVM and it works without root
> > just with the attached patch. Without access to /proc//map_files/
> > not everything CRIU can do will actually work, but we are a lot closer
> > to what our users have been asking for.
> 
> The current permission checks on /proc//map_files/ are simply
> someone being over-cautious.
> 
> Someone needs to think through the threat landscape and figure out what
> permission checks are actually needed.
> 
> Making the permission check ns_capable instead of capable is a
> no-brainer.  Figuring out which user_ns to test against might be a
> we bit harder.
> 
> We could probably even allow the owner of the process to open the files
> but that requires someone doing the work of thinking through how
> being able to opening files that you have mmaped might be a problem.

As mentioned in the other thread, CRIU can work with read access to
map_files.

> >> > There are probably a few more things guarded by CAP_SYS_ADMIN required
> >> > to run checkpoint/restore as non-root,
> >> 
> >> If you need CAP_SYS_ADMIN anyway you're not gaining anything by
> >> separating out CAP_RESTORE.
> >
> > No, as described we can checkpoint and restore a JVM with this patch and
> > it also solves the problem the set_ns_last_pid fork() loop daemon tries
> > to solve. It is not enough to support the full functionality of CRIU as
> > map_files is also important, but we do not need CAP_SYS_ADMIN and
> > CAP_RESTORE. Only CAP_RESTORE would be necessary.
> >
> > With a new capability users can enable checkpoint/restore as non-root
> > without giving CRIU access to any of the other possibilities offered by
> > CAP_SYS_ADMIN. Setting a PID and map_files have been introduced for CRIU
> > and used to live behind CONFIG_CHECKPOINT_RESTORE. Having a capability
> > for checkpoint/restore would make it easier for CRIU users to run it as
> > non-root and make it very clear what is possible when giving CRIU the
> > new capability. No other things would be allowed than necessary for
> > checkpoint/restore. Setting a PID is most important for the restore part
> > and reading map_files would be helpful during checkpoint. So it actually
> > should be called CAP_CHECKPOINT_RESTORE as Christian mentioned in
> > another email.
> 
> Please if one is for checkpoint and one is for restore asking for a pair
> of capabilities is probably more appropriate.

I will send out a v2 with a renamed capability soon and also include
map_files to be readable with that capability.

> >> >  but by applying this patch I can
> >> > already checkpoint and restore processes as non-root. As there are
> >> > already multiple workarounds I would prefer to do it correctly in the
> >> > kernel to avoid that CRIU users are starting to invent more workarounds.
> >> 
> >> You've presented a couple of really inappropriate implementations
> >> that would qualify as workarounds. But the other two are completely
> >> appropriate within the system security policy. They don't "get around"
> >> the problem, they use existing mechanisms as they are intended.
> >
> > I agree with the user namespace approach to be appropriate, but not the
> > CAP_SYS_ADMIN approach as CRIU only needs a tiny subset (2 things) of
> > what CAP_SYS_ADMIN allows.
> 
> 
> If we are only talking 2 things can you please include in your patchset
> a patch enabling those 2 things?

The two things are setting a PID via ns_last_pid/clone3() and reading
map_files.

> But even more than this we need a request that asks not for the least
> you can possibly ask for but asks for what you need to do a good job.

Also in this thread Kamil mentioned that they also need calling prctl
with PR_SET_MM during restore in their production setup.

> I am having visions of a recurring discussion that says can we add one
> more permission check to CAP_RESTORE or CAP_CHECKPOINT when they are
> things we could know today.

I will prepare a new version of this patch using CAP_CHECKPOINT_RESTORE
for ns_last_pid/clone3(), map_files, and prctl with PR_SET_MM.

Adrian

Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Adrian Reber]

On Mon, May 25, 2020 at 11:55:20AM -0700, Casey Schaufler wrote:
> On 5/25/2020 1:05 AM, Adrian Reber wrote:
> > On Fri, May 22, 2020 at 09:40:37AM -0700, Casey Schaufler wrote:
> >> On 5/21/2020 10:53 PM, Adrian Reber wrote:
> >>> This enables CRIU to checkpoint and restore a process as non-root.
> >> I know it sounds pedantic, but could you spell out CRIU once?
> >> While I know that everyone who cares either knows or can guess
> >> what you're talking about, it may be a mystery to some of the
> >> newer kernel developers.
> > Sure. CRIU - Checkpoint/Restore In Userspace.
> 
> Thanks. I blew out my acronym processor in the 1990's while
> working on trusted Unix system security evaluations.
> 
> >>> Over the last years CRIU upstream has been asked a couple of time if it
> >>> is possible to checkpoint and restore a process as non-root. The answer
> >>> usually was: 'almost'.
> >>>
> >>> The main blocker to restore a process was that selecting the PID of the
> >>> restored process, which is necessary for CRIU, is guarded by 
> >>> CAP_SYS_ADMIN.
> >> What are the other blockers? Are you going to suggest additional new
> >> capabilities to clear them?
> > As mentioned somewhere else access to /proc//map_files/ would be
> > helpful. Right now I am testing with a JVM and it works without root
> > just with the attached patch. Without access to /proc//map_files/
> > not everything CRIU can do will actually work, but we are a lot closer
> > to what our users have been asking for.
> 
> Are you talking about read access to map_files owned by other users
> or write access to map_files for the current user?

If I understand part of CRIU correctly, then we only need read-access
for the current user. I am sure Andrei, Pavel or Cyrill will correct me
if I am wrong concerning map_files.

> >>> In the last two years the questions about checkpoint/restore as non-root
> >>> have increased and especially in the last few months we have seen
> >>> multiple people inventing workarounds.
> >> Giving a process CAP_SYS_ADMIN is a non-root solution.
> > Yes, but like mentioned somewhere else not a solution that actually
> > works,
> 
> It's a solution that will execute and do what you're asking of it ...
> 
> >  because CAP_SYS_ADMIN allows too much.
> 
> ... but apparently not one that your users find satisfactory.
> 
> >  Especially for the
> > checkpoint/restore case, we really need one (setting the PID of a new
> > process) and to make it complete a second (reading map_files).
> >
> > Reading the comments in include/uapi/linux/capability.h concerning
> > CAP_SYS_ADMIN it allows the binary to do at least 35 things. The two
> > (three) I mentioned above (ns_last_pid (clone3) map_files) are not
> > mentioned in that list, so CAP_SYS_ADMIN allows probably much more.
> >
> > To allow checkpoint/restore as non-root nobody will give CRIU
> > CAP_SYS_ADMIN because it is too wide.
> 
> CAP_SYS_ADMIN exists for system behaviors that are not policy enforcement,
> but important to the system nonetheless. If you argue that checkpoint/restart
> is system policy enforcement rather then an administrative task it would
> be easier to sell.
> 
> Nobody likes CAP_SYS_ADMIN, but usually a process that does one of the
> things it covers will do more (sometimes many more) of the things it
> covers. The longstanding problem with breaking up CAP_SYS_ADMIN is that
> most breakouts result in programs that still need CAP_SYS_ADMIN anyway.
> 
> >>> The use-cases so far and their workarounds:
> >>>
> >>>  * Checkpoint/Restore in an HPC environment in combination with
> >>>a resource manager distributing jobs. Users are always running
> >>>as non root, but there was the desire to provide a way to
> >>>checkpoint and restore long running jobs.
> >>>Workaround: setuid wrapper to start CRIU as root as non-root
> >>>
> >>> https://github.com/FredHutch/slurm-examples/blob/master/checkpointer/lib/checkpointer/checkpointer-suid.c
> >> This is a classic and well understood mechanism for dealing with
> >> this kind of situation. You could have checkpointer-filecap-sys_admin.c
> >> instead, if you want to reduce use of the super-user.
> >>
> >>> * Another use case to checkpoint/restore processes as non-root
> >>>uses as workaround a non privileged process which cycles through
> >>>PIDs by calling fork() as fast as possible with a rate of
> >>>100,000 pids/s instead of writing to ns_last_pid
> >>>https://github.com/twosigma/set_ns_last_pid
> >> Oh dear.
> >>
> >>>  * Fast Java startup using checkpoint/restore.
> >>>We have been in contact with JVM developers who are integrating
> >>>CRIU into a JVM to decrease the startup time.
> >>>Workaround so far: patch out CAP_SYS_ADMIN checks in the kernel
> >> That's not a workaround, it's a policy violation.
> >> Bad JVM! No biscuit!
> > This was used as a proof of concept to see if we can checkpoint and
> > restore a JVM without root. Only the ns_last_pid check was removed to

Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Jann Horn]

On Tue, May 26, 2020 at 9:01 PM Christine Flood  wrote:
> Java applications suffer from slow startup times due to dynamic class loading 
> and warming up the Just In Time compilers.  Not all Java users have root 
> access on their machines.  Enabling CRIU in user mode solves this problem for 
> us.  We are about to release a user library that will allow check pointing 
> Java from within Java.  Having to run this as root would severely limit its 
> utility.

Have you looked into whether it would be practical to restore the
saved process state with different PIDs, and then fix up all places
that might have stored the old PIDs? As long as all threads are
managed by the JVM, that might be doable, right?

If you did that, you would also solve the problem of not being able to
start two copies of the same image (because their PIDs would collide)
or randomly not being able to start processes (because their PIDs
collide with other existing things).

Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Casey Schaufler]

On 5/26/2020 12:01 PM, Christine Flood wrote:

Please do not top-post on this list.

> Java applications suffer from slow startup times due to dynamic class loading 
> and warming up the Just In Time compilers.  Not all Java users have root 
> access on their machines.  Enabling CRIU in user mode solves this problem for 
> us.  We are about to release a user library that will allow check pointing 
> Java from within Java.  Having to run this as root would severely limit its 
> utility.

The performance of dynamic loading is a well understood issue.
Please don't conflate that with the security issues involved.
Security is *not* the basic problem. If you are having problems
with application start-up performance you really should be be
addressing that directly rather than implementing sophisticated
workarounds that require system security changes.

>
>
> Christine
>
> On Tue, May 26, 2020 at 10:05 AM Eric W. Biederman  > wrote:
>
> Adrian Reber mailto:are...@redhat.com>> writes:
>
> > On Fri, May 22, 2020 at 09:40:37AM -0700, Casey Schaufler wrote:
>
> >> What are the other blockers? Are you going to suggest additional new
> >> capabilities to clear them?
> >
> > As mentioned somewhere else access to /proc//map_files/ would be
> > helpful. Right now I am testing with a JVM and it works without root
> > just with the attached patch. Without access to /proc//map_files/
> > not everything CRIU can do will actually work, but we are a lot closer
> > to what our users have been asking for.
>
> The current permission checks on /proc//map_files/ are simply
> someone being over-cautious.
>
> Someone needs to think through the threat landscape and figure out what
> permission checks are actually needed.
>
> Making the permission check ns_capable instead of capable is a
> no-brainer.  Figuring out which user_ns to test against might be a
> we bit harder.
>
> We could probably even allow the owner of the process to open the files
> but that requires someone doing the work of thinking through how
> being able to opening files that you have mmaped might be a problem.
>
> >> > There are probably a few more things guarded by CAP_SYS_ADMIN 
> required
> >> > to run checkpoint/restore as non-root,
> >>
> >> If you need CAP_SYS_ADMIN anyway you're not gaining anything by
> >> separating out CAP_RESTORE.
> >
> > No, as described we can checkpoint and restore a JVM with this patch and
> > it also solves the problem the set_ns_last_pid fork() loop daemon tries
> > to solve. It is not enough to support the full functionality of CRIU as
> > map_files is also important, but we do not need CAP_SYS_ADMIN and
> > CAP_RESTORE. Only CAP_RESTORE would be necessary.
> >
> > With a new capability users can enable checkpoint/restore as non-root
> > without giving CRIU access to any of the other possibilities offered by
> > CAP_SYS_ADMIN. Setting a PID and map_files have been introduced for CRIU
> > and used to live behind CONFIG_CHECKPOINT_RESTORE. Having a capability
> > for checkpoint/restore would make it easier for CRIU users to run it as
> > non-root and make it very clear what is possible when giving CRIU the
> > new capability. No other things would be allowed than necessary for
> > checkpoint/restore. Setting a PID is most important for the restore part
> > and reading map_files would be helpful during checkpoint. So it actually
> > should be called CAP_CHECKPOINT_RESTORE as Christian mentioned in
> > another email.
>
> Please if one is for checkpoint and one is for restore asking for a pair
> of capabilities is probably more appropriate.
>
> >> >  but by applying this patch I can
> >> > already checkpoint and restore processes as non-root. As there are
> >> > already multiple workarounds I would prefer to do it correctly in the
> >> > kernel to avoid that CRIU users are starting to invent more 
> workarounds.
> >>
> >> You've presented a couple of really inappropriate implementations
> >> that would qualify as workarounds. But the other two are completely
> >> appropriate within the system security policy. They don't "get around"
> >> the problem, they use existing mechanisms as they are intended.
> >
> > I agree with the user namespace approach to be appropriate, but not the
> > CAP_SYS_ADMIN approach as CRIU only needs a tiny subset (2 things) of
> > what CAP_SYS_ADMIN allows.
>
>
> If we are only talking 2 things can you please include in your patchset
> a patch enabling those 2 things?
>
> But even more than this we need a request that asks not for the least
> you can possibly ask for but asks for what you need to do a good job.
>
> I am having visions of a recurring discussion that says can we add one
> more permission check to CAP_RESTORE or 

Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Eric W. Biederman]

Adrian Reber  writes:

> On Fri, May 22, 2020 at 09:40:37AM -0700, Casey Schaufler wrote:

>> What are the other blockers? Are you going to suggest additional new
>> capabilities to clear them?
>
> As mentioned somewhere else access to /proc//map_files/ would be
> helpful. Right now I am testing with a JVM and it works without root
> just with the attached patch. Without access to /proc//map_files/
> not everything CRIU can do will actually work, but we are a lot closer
> to what our users have been asking for.

The current permission checks on /proc//map_files/ are simply
someone being over-cautious.

Someone needs to think through the threat landscape and figure out what
permission checks are actually needed.

Making the permission check ns_capable instead of capable is a
no-brainer.  Figuring out which user_ns to test against might be a
we bit harder.

We could probably even allow the owner of the process to open the files
but that requires someone doing the work of thinking through how
being able to opening files that you have mmaped might be a problem.

>> > There are probably a few more things guarded by CAP_SYS_ADMIN required
>> > to run checkpoint/restore as non-root,
>> 
>> If you need CAP_SYS_ADMIN anyway you're not gaining anything by
>> separating out CAP_RESTORE.
>
> No, as described we can checkpoint and restore a JVM with this patch and
> it also solves the problem the set_ns_last_pid fork() loop daemon tries
> to solve. It is not enough to support the full functionality of CRIU as
> map_files is also important, but we do not need CAP_SYS_ADMIN and
> CAP_RESTORE. Only CAP_RESTORE would be necessary.
>
> With a new capability users can enable checkpoint/restore as non-root
> without giving CRIU access to any of the other possibilities offered by
> CAP_SYS_ADMIN. Setting a PID and map_files have been introduced for CRIU
> and used to live behind CONFIG_CHECKPOINT_RESTORE. Having a capability
> for checkpoint/restore would make it easier for CRIU users to run it as
> non-root and make it very clear what is possible when giving CRIU the
> new capability. No other things would be allowed than necessary for
> checkpoint/restore. Setting a PID is most important for the restore part
> and reading map_files would be helpful during checkpoint. So it actually
> should be called CAP_CHECKPOINT_RESTORE as Christian mentioned in
> another email.

Please if one is for checkpoint and one is for restore asking for a pair
of capabilities is probably more appropriate.

>> >  but by applying this patch I can
>> > already checkpoint and restore processes as non-root. As there are
>> > already multiple workarounds I would prefer to do it correctly in the
>> > kernel to avoid that CRIU users are starting to invent more workarounds.
>> 
>> You've presented a couple of really inappropriate implementations
>> that would qualify as workarounds. But the other two are completely
>> appropriate within the system security policy. They don't "get around"
>> the problem, they use existing mechanisms as they are intended.
>
> I agree with the user namespace approach to be appropriate, but not the
> CAP_SYS_ADMIN approach as CRIU only needs a tiny subset (2 things) of
> what CAP_SYS_ADMIN allows.


If we are only talking 2 things can you please include in your patchset
a patch enabling those 2 things?

But even more than this we need a request that asks not for the least
you can possibly ask for but asks for what you need to do a good job.

I am having visions of a recurring discussion that says can we add one
more permission check to CAP_RESTORE or CAP_CHECKPOINT when they are
things we could know today.

Eric

Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Radostin Stoyanov]

On 25/05/2020 22:53, Jann Horn wrote:

On Fri, May 22, 2020 at 7:55 AM Adrian Reber  wrote:

This enables CRIU to checkpoint and restore a process as non-root.

Over the last years CRIU upstream has been asked a couple of time if it
is possible to checkpoint and restore a process as non-root. The answer
usually was: 'almost'.

The main blocker to restore a process was that selecting the PID of the
restored process, which is necessary for CRIU, is guarded by CAP_SYS_ADMIN.

And if you were restoring the process into your own PID namespace, so
that you actually have a guarantee that this isn't going to blow up in
your face because one of your PIDs is allocated for a different
process, this part of the problem could be simplified.

I don't get why your users are fine with a "oh it kinda works 99% of
the time but sometimes it randomly doesn't and then you have to go
reboot or whatever" model.
Transparent checkpoint and restore of a process tree is not simple, 
especially when it is done entirely in user-space. To best of my 
knowledge, CRIU is the only tool out there that is able to achieve this, 
it is actively being tested and maintained, and it has been integrated 
into several container runtimes. Like any other software, CRIU has 
limitations but, as said in the README file, contributions are welcome.

Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Jann Horn]

On Fri, May 22, 2020 at 7:55 AM Adrian Reber  wrote:
> This enables CRIU to checkpoint and restore a process as non-root.
>
> Over the last years CRIU upstream has been asked a couple of time if it
> is possible to checkpoint and restore a process as non-root. The answer
> usually was: 'almost'.
>
> The main blocker to restore a process was that selecting the PID of the
> restored process, which is necessary for CRIU, is guarded by CAP_SYS_ADMIN.

And if you were restoring the process into your own PID namespace, so
that you actually have a guarantee that this isn't going to blow up in
your face because one of your PIDs is allocated for a different
process, this part of the problem could be simplified.

I don't get why your users are fine with a "oh it kinda works 99% of
the time but sometimes it randomly doesn't and then you have to go
reboot or whatever" model.

Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Casey Schaufler]

On 5/25/2020 1:05 AM, Adrian Reber wrote:
> On Fri, May 22, 2020 at 09:40:37AM -0700, Casey Schaufler wrote:
>> On 5/21/2020 10:53 PM, Adrian Reber wrote:
>>> This enables CRIU to checkpoint and restore a process as non-root.
>> I know it sounds pedantic, but could you spell out CRIU once?
>> While I know that everyone who cares either knows or can guess
>> what you're talking about, it may be a mystery to some of the
>> newer kernel developers.
> Sure. CRIU - Checkpoint/Restore In Userspace.

Thanks. I blew out my acronym processor in the 1990's while
working on trusted Unix system security evaluations.

>>> Over the last years CRIU upstream has been asked a couple of time if it
>>> is possible to checkpoint and restore a process as non-root. The answer
>>> usually was: 'almost'.
>>>
>>> The main blocker to restore a process was that selecting the PID of the
>>> restored process, which is necessary for CRIU, is guarded by CAP_SYS_ADMIN.
>> What are the other blockers? Are you going to suggest additional new
>> capabilities to clear them?
> As mentioned somewhere else access to /proc//map_files/ would be
> helpful. Right now I am testing with a JVM and it works without root
> just with the attached patch. Without access to /proc//map_files/
> not everything CRIU can do will actually work, but we are a lot closer
> to what our users have been asking for.

Are you talking about read access to map_files owned by other users
or write access to map_files for the current user?
 

>>> In the last two years the questions about checkpoint/restore as non-root
>>> have increased and especially in the last few months we have seen
>>> multiple people inventing workarounds.
>> Giving a process CAP_SYS_ADMIN is a non-root solution.
> Yes, but like mentioned somewhere else not a solution that actually
> works,

It's a solution that will execute and do what you're asking of it ...

>  because CAP_SYS_ADMIN allows too much.

... but apparently not one that your users find satisfactory.

>  Especially for the
> checkpoint/restore case, we really need one (setting the PID of a new
> process) and to make it complete a second (reading map_files).
>
> Reading the comments in include/uapi/linux/capability.h concerning
> CAP_SYS_ADMIN it allows the binary to do at least 35 things. The two
> (three) I mentioned above (ns_last_pid (clone3) map_files) are not
> mentioned in that list, so CAP_SYS_ADMIN allows probably much more.
>
> To allow checkpoint/restore as non-root nobody will give CRIU
> CAP_SYS_ADMIN because it is too wide.

CAP_SYS_ADMIN exists for system behaviors that are not policy enforcement,
but important to the system nonetheless. If you argue that checkpoint/restart
is system policy enforcement rather then an administrative task it would
be easier to sell.

Nobody likes CAP_SYS_ADMIN, but usually a process that does one of the
things it covers will do more (sometimes many more) of the things it
covers. The longstanding problem with breaking up CAP_SYS_ADMIN is that
most breakouts result in programs that still need CAP_SYS_ADMIN anyway.

>>> The use-cases so far and their workarounds:
>>>
>>>  * Checkpoint/Restore in an HPC environment in combination with
>>>a resource manager distributing jobs. Users are always running
>>>as non root, but there was the desire to provide a way to
>>>checkpoint and restore long running jobs.
>>>Workaround: setuid wrapper to start CRIU as root as non-root
>>>
>>> https://github.com/FredHutch/slurm-examples/blob/master/checkpointer/lib/checkpointer/checkpointer-suid.c
>> This is a classic and well understood mechanism for dealing with
>> this kind of situation. You could have checkpointer-filecap-sys_admin.c
>> instead, if you want to reduce use of the super-user.
>>
>>> * Another use case to checkpoint/restore processes as non-root
>>>uses as workaround a non privileged process which cycles through
>>>PIDs by calling fork() as fast as possible with a rate of
>>>100,000 pids/s instead of writing to ns_last_pid
>>>https://github.com/twosigma/set_ns_last_pid
>> Oh dear.
>>
>>>  * Fast Java startup using checkpoint/restore.
>>>We have been in contact with JVM developers who are integrating
>>>CRIU into a JVM to decrease the startup time.
>>>Workaround so far: patch out CAP_SYS_ADMIN checks in the kernel
>> That's not a workaround, it's a policy violation.
>> Bad JVM! No biscuit!
> This was used as a proof of concept to see if we can checkpoint and
> restore a JVM without root. Only the ns_last_pid check was removed to
> see if it works and it does.
>
>>>  * Container migration as non root. There are people already
>>>using CRIU to migrate containers as non-root. The solution
>>>there is to run it in a user namespace. So if you are able
>>>to carefully setup your environment with the namespaces
>>>it is already possible to restore a container/process as non-root.
>> This is exactly the kind of situation that user namespaces are

Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Adrian Reber]

On Fri, May 22, 2020 at 09:40:37AM -0700, Casey Schaufler wrote:
> On 5/21/2020 10:53 PM, Adrian Reber wrote:
> > This enables CRIU to checkpoint and restore a process as non-root.
> 
> I know it sounds pedantic, but could you spell out CRIU once?
> While I know that everyone who cares either knows or can guess
> what you're talking about, it may be a mystery to some of the
> newer kernel developers.

Sure. CRIU - Checkpoint/Restore In Userspace.

> > Over the last years CRIU upstream has been asked a couple of time if it
> > is possible to checkpoint and restore a process as non-root. The answer
> > usually was: 'almost'.
> >
> > The main blocker to restore a process was that selecting the PID of the
> > restored process, which is necessary for CRIU, is guarded by CAP_SYS_ADMIN.
> 
> What are the other blockers? Are you going to suggest additional new
> capabilities to clear them?

As mentioned somewhere else access to /proc//map_files/ would be
helpful. Right now I am testing with a JVM and it works without root
just with the attached patch. Without access to /proc//map_files/
not everything CRIU can do will actually work, but we are a lot closer
to what our users have been asking for.

> > In the last two years the questions about checkpoint/restore as non-root
> > have increased and especially in the last few months we have seen
> > multiple people inventing workarounds.
> 
> Giving a process CAP_SYS_ADMIN is a non-root solution.

Yes, but like mentioned somewhere else not a solution that actually
works, because CAP_SYS_ADMIN allows too much. Especially for the
checkpoint/restore case, we really need one (setting the PID of a new
process) and to make it complete a second (reading map_files).

Reading the comments in include/uapi/linux/capability.h concerning
CAP_SYS_ADMIN it allows the binary to do at least 35 things. The two
(three) I mentioned above (ns_last_pid (clone3) map_files) are not
mentioned in that list, so CAP_SYS_ADMIN allows probably much more.

To allow checkpoint/restore as non-root nobody will give CRIU
CAP_SYS_ADMIN because it is too wide.

> > The use-cases so far and their workarounds:
> >
> >  * Checkpoint/Restore in an HPC environment in combination with
> >a resource manager distributing jobs. Users are always running
> >as non root, but there was the desire to provide a way to
> >checkpoint and restore long running jobs.
> >Workaround: setuid wrapper to start CRIU as root as non-root
> >
> > https://github.com/FredHutch/slurm-examples/blob/master/checkpointer/lib/checkpointer/checkpointer-suid.c
> 
> This is a classic and well understood mechanism for dealing with
> this kind of situation. You could have checkpointer-filecap-sys_admin.c
> instead, if you want to reduce use of the super-user.
> 
> > * Another use case to checkpoint/restore processes as non-root
> >uses as workaround a non privileged process which cycles through
> >PIDs by calling fork() as fast as possible with a rate of
> >100,000 pids/s instead of writing to ns_last_pid
> >https://github.com/twosigma/set_ns_last_pid
> 
> Oh dear.
> 
> >  * Fast Java startup using checkpoint/restore.
> >We have been in contact with JVM developers who are integrating
> >CRIU into a JVM to decrease the startup time.
> >Workaround so far: patch out CAP_SYS_ADMIN checks in the kernel
> 
> That's not a workaround, it's a policy violation.
> Bad JVM! No biscuit!

This was used as a proof of concept to see if we can checkpoint and
restore a JVM without root. Only the ns_last_pid check was removed to
see if it works and it does.

> >  * Container migration as non root. There are people already
> >using CRIU to migrate containers as non-root. The solution
> >there is to run it in a user namespace. So if you are able
> >to carefully setup your environment with the namespaces
> >it is already possible to restore a container/process as non-root.
> 
> This is exactly the kind of situation that user namespaces are
> supposed to address.
> 
> >Unfortunately it is not always possible to setup an environment
> >in such a way and for easier access to non-root based container
> >migration this patch is also required.
> 
> If a user namespace solution is impossible or (more likely) too
> expensive, there's always the checkpointer-filecap-sys_admin option.

But then again we open up all of CAP_SYS_ADMIN, which is not necessary.

> > There are probably a few more things guarded by CAP_SYS_ADMIN required
> > to run checkpoint/restore as non-root,
> 
> If you need CAP_SYS_ADMIN anyway you're not gaining anything by
> separating out CAP_RESTORE.

No, as described we can checkpoint and restore a JVM with this patch and
it also solves the problem the set_ns_last_pid fork() loop daemon tries
to solve. It is not enough to support the full functionality of CRIU as
map_files is also important, but we do not need CAP_SYS_ADMIN and
CAP_RESTORE. Only CAP_RESTORE would be necessary.

Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Casey Schaufler]

On 5/22/2020 9:27 PM, Andrei Vagin wrote:
> On Fri, May 22, 2020 at 09:40:37AM -0700, Casey Schaufler wrote:
>> On 5/21/2020 10:53 PM, Adrian Reber wrote:
>>> There are probably a few more things guarded by CAP_SYS_ADMIN required
>>> to run checkpoint/restore as non-root,
>> If you need CAP_SYS_ADMIN anyway you're not gaining anything by
>> separating out CAP_RESTORE.
>>
>>>  but by applying this patch I can
>>> already checkpoint and restore processes as non-root. As there are
>>> already multiple workarounds I would prefer to do it correctly in the
>>> kernel to avoid that CRIU users are starting to invent more workarounds.
>> You've presented a couple of really inappropriate implementations
>> that would qualify as workarounds. But the other two are completely
>> appropriate within the system security policy. They don't "get around"
>> the problem, they use existing mechanisms as they are intended.
>>
> With CAP_CHECKPOINT_RESTORE, we will need to use the same mechanisms.

Then why call them out as objectionable "workarounds"?

> The problem is that CAP_SYS_ADMIN is too wide.

This is well understood, and irrelevant.

If we broke out CAP_SYS_ADMIN properly we'd have hundreds of
capabilities, and no one would be able to manage the capability
sets on anything. Just breaking out of CAP_SYS_ADMIN, especially
if the process is going to need other capabilities anyway, gains
you nothing.

>  If a process has
> CAP_SYS_ADMIN, it can do a lot of things and  the operation of forking a
> process with a specified pid isn't the most dangerous one in this case.
> Offten security policies don't allow to grant CAP_SYS_ADMIN to any
> third-party tools even in non-root user namespaces.
>

Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Andrei Vagin]

On Fri, May 22, 2020 at 09:40:37AM -0700, Casey Schaufler wrote:
> On 5/21/2020 10:53 PM, Adrian Reber wrote:
> > There are probably a few more things guarded by CAP_SYS_ADMIN required
> > to run checkpoint/restore as non-root,
> 
> If you need CAP_SYS_ADMIN anyway you're not gaining anything by
> separating out CAP_RESTORE.
> 
> >  but by applying this patch I can
> > already checkpoint and restore processes as non-root. As there are
> > already multiple workarounds I would prefer to do it correctly in the
> > kernel to avoid that CRIU users are starting to invent more workarounds.
> 
> You've presented a couple of really inappropriate implementations
> that would qualify as workarounds. But the other two are completely
> appropriate within the system security policy. They don't "get around"
> the problem, they use existing mechanisms as they are intended.
> 

With CAP_CHECKPOINT_RESTORE, we will need to use the same mechanisms.

The problem is that CAP_SYS_ADMIN is too wide. If a process has
CAP_SYS_ADMIN, it can do a lot of things and  the operation of forking a
process with a specified pid isn't the most dangerous one in this case.
Offten security policies don't allow to grant CAP_SYS_ADMIN to any
third-party tools even in non-root user namespaces.

Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Andrei Vagin]

On Fri, May 22, 2020 at 09:53:31AM +0200, Christian Brauner wrote:
> On Fri, May 22, 2020 at 07:53:50AM +0200, Adrian Reber wrote:
> > 
> > There are probably a few more things guarded by CAP_SYS_ADMIN required
> > to run checkpoint/restore as non-root, but by applying this patch I can
> > already checkpoint and restore processes as non-root. As there are
> > already multiple workarounds I would prefer to do it correctly in the
> > kernel to avoid that CRIU users are starting to invent more workarounds.
> 
> It sounds ok to me as long as this feature is guarded by any sensible
> capability. I don't want users to be able to randomly choose their pid
> without any capability required.
> 
> We've heard the plea for unprivileged checkpoint/restore through the
> grapevine and a few times about CAP_RESTORE at plumbers but it's one of
> those cases where nobody pushed for it so it's urgency was questionable.
> This is 5.9 material though and could you please add selftests?
> 
> It also seems you have future changes planned that would make certain
> things accessible via CAP_RESTORE that are currently guarded by other
> capabilities. Any specific things in mind? It might be worth knowing
> what we'd be getting ourselves into if you're planning on flipping
> switches in other places.

/proc/pid/map_files is one of the first candidate what we need to think
about. CRIU opens files from /proc/pid/map_files to dump file mappings,
shared memory mappings, memfd files.

Right now, it is impossible to open these files without CAP_SYS_ADMIN in
the root user-namespace (proc_map_files_get_link).

Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Casey Schaufler]

On 5/21/2020 10:53 PM, Adrian Reber wrote:
> This enables CRIU to checkpoint and restore a process as non-root.

I know it sounds pedantic, but could you spell out CRIU once?
While I know that everyone who cares either knows or can guess
what you're talking about, it may be a mystery to some of the
newer kernel developers.

> Over the last years CRIU upstream has been asked a couple of time if it
> is possible to checkpoint and restore a process as non-root. The answer
> usually was: 'almost'.
>
> The main blocker to restore a process was that selecting the PID of the
> restored process, which is necessary for CRIU, is guarded by CAP_SYS_ADMIN.

What are the other blockers? Are you going to suggest additional new
capabilities to clear them?

> In the last two years the questions about checkpoint/restore as non-root
> have increased and especially in the last few months we have seen
> multiple people inventing workarounds.

Giving a process CAP_SYS_ADMIN is a non-root solution.

> The use-cases so far and their workarounds:
>
>  * Checkpoint/Restore in an HPC environment in combination with
>a resource manager distributing jobs. Users are always running
>as non root, but there was the desire to provide a way to
>checkpoint and restore long running jobs.
>Workaround: setuid wrapper to start CRIU as root as non-root
>
> https://github.com/FredHutch/slurm-examples/blob/master/checkpointer/lib/checkpointer/checkpointer-suid.c

This is a classic and well understood mechanism for dealing with
this kind of situation. You could have checkpointer-filecap-sys_admin.c
instead, if you want to reduce use of the super-user.

> * Another use case to checkpoint/restore processes as non-root
>uses as workaround a non privileged process which cycles through
>PIDs by calling fork() as fast as possible with a rate of
>100,000 pids/s instead of writing to ns_last_pid
>https://github.com/twosigma/set_ns_last_pid

Oh dear.

>  * Fast Java startup using checkpoint/restore.
>We have been in contact with JVM developers who are integrating
>CRIU into a JVM to decrease the startup time.
>Workaround so far: patch out CAP_SYS_ADMIN checks in the kernel

That's not a workaround, it's a policy violation.
Bad JVM! No biscuit!

>  * Container migration as non root. There are people already
>using CRIU to migrate containers as non-root. The solution
>there is to run it in a user namespace. So if you are able
>to carefully setup your environment with the namespaces
>it is already possible to restore a container/process as non-root.

This is exactly the kind of situation that user namespaces are
supposed to address.

>Unfortunately it is not always possible to setup an environment
>in such a way and for easier access to non-root based container
>migration this patch is also required.

If a user namespace solution is impossible or (more likely) too
expensive, there's always the checkpointer-filecap-sys_admin option.

> There are probably a few more things guarded by CAP_SYS_ADMIN required
> to run checkpoint/restore as non-root,

If you need CAP_SYS_ADMIN anyway you're not gaining anything by
separating out CAP_RESTORE.

>  but by applying this patch I can
> already checkpoint and restore processes as non-root. As there are
> already multiple workarounds I would prefer to do it correctly in the
> kernel to avoid that CRIU users are starting to invent more workarounds.

You've presented a couple of really inappropriate implementations
that would qualify as workarounds. But the other two are completely
appropriate within the system security policy. They don't "get around"
the problem, they use existing mechanisms as they are intended.



> I have used the following tests to verify that this change works as
> expected by setting the new capability CAP_RESTORE on the two resulting
> test binaries:
>
> $ cat ns_last_pid.c
>  // 
> http://efiop-notes.blogspot.com/2014/06/how-to-set-pid-using-nslastpid.html
>  #include 
>  #include 
>  #include 
>  #include 
>  #include 
>  #include 
>
> int main(int argc, char *argv[])
> {
>   pid_t pid, new_pid;
>   char buf[32];
>   int fd;
>
>   if (argc != 2)
>   return 1;
>
>   printf("Opening ns_last_pid...\n");
>   fd = open("/proc/sys/kernel/ns_last_pid", O_RDWR | O_CREAT, 0644);
>   if (fd < 0) {
>   perror("Cannot open ns_last_pid");
>   return 1;
>   }
>
>   printf("Locking ns_last_pid...\n");
>   if (flock(fd, LOCK_EX)) {
>   close(fd);
>   printf("Cannot lock ns_last_pid\n");
>   return 1;
>   }
>
>   pid = atoi(argv[1]);
>   snprintf(buf, sizeof(buf), "%d", pid - 1);
>   printf("Writing pid-1 to ns_last_pid...\n");
>   if (write(fd, buf, strlen(buf)) != strlen(buf)) {
>   printf("Cannot write to buf\n");
>   return 1;
>   }
>
>   printf("Forking...\n");
>   

Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Christian Brauner]

On Fri, May 22, 2020 at 07:53:50AM +0200, Adrian Reber wrote:
> This enables CRIU to checkpoint and restore a process as non-root.
> 
> Over the last years CRIU upstream has been asked a couple of time if it
> is possible to checkpoint and restore a process as non-root. The answer
> usually was: 'almost'.
> 
> The main blocker to restore a process was that selecting the PID of the
> restored process, which is necessary for CRIU, is guarded by CAP_SYS_ADMIN.
> 
> In the last two years the questions about checkpoint/restore as non-root
> have increased and especially in the last few months we have seen
> multiple people inventing workarounds.
> 
> The use-cases so far and their workarounds:
> 
>  * Checkpoint/Restore in an HPC environment in combination with
>a resource manager distributing jobs. Users are always running
>as non root, but there was the desire to provide a way to
>checkpoint and restore long running jobs.
>Workaround: setuid wrapper to start CRIU as root as non-root
>
> https://github.com/FredHutch/slurm-examples/blob/master/checkpointer/lib/checkpointer/checkpointer-suid.c
>  * Another use case to checkpoint/restore processes as non-root
>uses as workaround a non privileged process which cycles through
>PIDs by calling fork() as fast as possible with a rate of
>100,000 pids/s instead of writing to ns_last_pid
>https://github.com/twosigma/set_ns_last_pid
>  * Fast Java startup using checkpoint/restore.
>We have been in contact with JVM developers who are integrating
>CRIU into a JVM to decrease the startup time.
>Workaround so far: patch out CAP_SYS_ADMIN checks in the kernel
>  * Container migration as non root. There are people already
>using CRIU to migrate containers as non-root. The solution
>there is to run it in a user namespace. So if you are able
>to carefully setup your environment with the namespaces
>it is already possible to restore a container/process as non-root.
>Unfortunately it is not always possible to setup an environment
>in such a way and for easier access to non-root based container
>migration this patch is also required.
> 
> There are probably a few more things guarded by CAP_SYS_ADMIN required
> to run checkpoint/restore as non-root, but by applying this patch I can
> already checkpoint and restore processes as non-root. As there are
> already multiple workarounds I would prefer to do it correctly in the
> kernel to avoid that CRIU users are starting to invent more workarounds.
> 
> I have used the following tests to verify that this change works as
> expected by setting the new capability CAP_RESTORE on the two resulting
> test binaries:
> 
> $ cat ns_last_pid.c
>  // 
> http://efiop-notes.blogspot.com/2014/06/how-to-set-pid-using-nslastpid.html
>  #include 
>  #include 
>  #include 
>  #include 
>  #include 
>  #include 
> 
> int main(int argc, char *argv[])
> {
>   pid_t pid, new_pid;
>   char buf[32];
>   int fd;
> 
>   if (argc != 2)
>   return 1;
> 
>   printf("Opening ns_last_pid...\n");
>   fd = open("/proc/sys/kernel/ns_last_pid", O_RDWR | O_CREAT, 0644);
>   if (fd < 0) {
>   perror("Cannot open ns_last_pid");
>   return 1;
>   }
> 
>   printf("Locking ns_last_pid...\n");
>   if (flock(fd, LOCK_EX)) {
>   close(fd);
>   printf("Cannot lock ns_last_pid\n");
>   return 1;
>   }
> 
>   pid = atoi(argv[1]);
>   snprintf(buf, sizeof(buf), "%d", pid - 1);
>   printf("Writing pid-1 to ns_last_pid...\n");
>   if (write(fd, buf, strlen(buf)) != strlen(buf)) {
>   printf("Cannot write to buf\n");
>   return 1;
>   }
> 
>   printf("Forking...\n");
>   new_pid = fork();
>   if (new_pid == 0) {
>   printf("I am the child!\n");
>   exit(0);
>   } else if (new_pid == pid)
>   printf("I am the parent. My child got the pid %d!\n", new_pid);
>   else
>   printf("pid (%d) does not match expected pid (%d)\n", new_pid, 
> pid);
> 
>   printf("Cleaning up...\n");
>   if (flock(fd, LOCK_UN))
>   printf("Cannot unlock\n");
>   close(fd);
>   return 0;
> }
> $ id -u; /home/libcap/ns_last_pid 30
> 1001
> Opening ns_last_pid...
> Locking ns_last_pid...
> Writing pid-1 to ns_last_pid...
> Forking...
> I am the parent. My child got the pid 30!
> I am the child!
> Cleaning up...
> 
> For the clone3() based approach:
> $ cat clone3_set_tid.c
>  #define _GNU_SOURCE
>  #include 
>  #include 
>  #include 
>  #include 
>  #include 
>  #include 
>  #include 
>  #include 
>  #include 
> 
>  #define ptr_to_u64(ptr) ((__u64)((uintptr_t)(ptr)))
> 
> int main(int argc, char *argv[])
> {
>   struct clone_args c_args = { };
>   pid_t pid, new_pid;
> 
>   if (argc != 2)
>   return 1;
> 
>   pid = atoi(argv[1]);
>   c_args.set_tid = 

Re: [PATCH] capabilities: Introduce CAP_RESTORE

[Christian Brauner]

On Fri, May 22, 2020 at 07:53:50AM +0200, Adrian Reber wrote:
> This enables CRIU to checkpoint and restore a process as non-root.
> 
> Over the last years CRIU upstream has been asked a couple of time if it
> is possible to checkpoint and restore a process as non-root. The answer
> usually was: 'almost'.
> 
> The main blocker to restore a process was that selecting the PID of the
> restored process, which is necessary for CRIU, is guarded by CAP_SYS_ADMIN.
> 
> In the last two years the questions about checkpoint/restore as non-root
> have increased and especially in the last few months we have seen
> multiple people inventing workarounds.
> 
> The use-cases so far and their workarounds:
> 
>  * Checkpoint/Restore in an HPC environment in combination with
>a resource manager distributing jobs. Users are always running
>as non root, but there was the desire to provide a way to
>checkpoint and restore long running jobs.
>Workaround: setuid wrapper to start CRIU as root as non-root
>
> https://github.com/FredHutch/slurm-examples/blob/master/checkpointer/lib/checkpointer/checkpointer-suid.c
>  * Another use case to checkpoint/restore processes as non-root
>uses as workaround a non privileged process which cycles through
>PIDs by calling fork() as fast as possible with a rate of
>100,000 pids/s instead of writing to ns_last_pid
>https://github.com/twosigma/set_ns_last_pid
>  * Fast Java startup using checkpoint/restore.
>We have been in contact with JVM developers who are integrating
>CRIU into a JVM to decrease the startup time.
>Workaround so far: patch out CAP_SYS_ADMIN checks in the kernel
>  * Container migration as non root. There are people already
>using CRIU to migrate containers as non-root. The solution
>there is to run it in a user namespace. So if you are able
>to carefully setup your environment with the namespaces
>it is already possible to restore a container/process as non-root.
>Unfortunately it is not always possible to setup an environment
>in such a way and for easier access to non-root based container
>migration this patch is also required.
> 
> There are probably a few more things guarded by CAP_SYS_ADMIN required
> to run checkpoint/restore as non-root, but by applying this patch I can
> already checkpoint and restore processes as non-root. As there are
> already multiple workarounds I would prefer to do it correctly in the
> kernel to avoid that CRIU users are starting to invent more workarounds.

It sounds ok to me as long as this feature is guarded by any sensible
capability. I don't want users to be able to randomly choose their pid
without any capability required.

We've heard the plea for unprivileged checkpoint/restore through the
grapevine and a few times about CAP_RESTORE at plumbers but it's one of
those cases where nobody pushed for it so it's urgency was questionable.
This is 5.9 material though and could you please add selftests?

It also seems you have future changes planned that would make certain
things accessible via CAP_RESTORE that are currently guarded by other
capabilities. Any specific things in mind? It might be worth knowing
what we'd be getting ourselves into if you're planning on flipping
switches in other places.

Christian