RE: [PATCH v5 0/3] close_range()

2020-06-07 Thread David Laight
From: Christian Brauner
> Sent: 04 June 2020 00:24
..
> -struct files_struct *dup_fd(struct files_struct *oldf, int *errorp)
> +struct files_struct *dup_fd(struct files_struct *oldf, unsigned int max_fds, 
> int *errorp)

Shouldn't this get changed to use ERR_PTR() etc?

..
> -int __close_range(struct files_struct *files, unsigned fd, unsigned max_fd)
> +int __close_range(unsigned fd, unsigned max_fd, unsigned int flags)

If the lowest fd that had ever has CLOEXEC (or CLOFORK) set were
remembered a flag could be passed in to say 'only close the fd
with CLOEXEC set'.

Given that CLOEXEC is almost never cleared, and is typically set
on all but a few fd the 'optimisation' of the bitmap is
probably a pessimisation.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, 
UK
Registration No: 1397386 (Wales)


Re: [PATCH v5 0/3] close_range()

2020-06-03 Thread Christian Brauner
On Wed, Jun 03, 2020 at 05:13:36PM -0700, Linus Torvalds wrote:
> On Wed, Jun 3, 2020 at 4:24 PM Christian Brauner
>  wrote:
> >
> > Ok, here's what I have. Does the below look somewhat sane?
> 
> Probably. Needs lots of testing. But this one looks wrong:

Right, there's a patch for a test-suite for the new flag too using
CLONE_FILES to create a shared fdtable and the proceeds to close all
(or subsets of) fds:

https://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git/commit/?h=close_range=498e7e844fe6e3f3306b2cd1b5e926e1cd394b99

I've been running that in an endless loop for a while.

> 
> > +int __close_range(unsigned fd, unsigned max_fd, unsigned int flags)
> >  {
> > +   if ((max_fd + 1) >= cur_max)
> > +   max_unshare_fds = fd;
> 
> A normal value for "close everything starting at X" would have a
> max_fd value of ~0.

Ugh, obvious braino from my side. This should just be:

if (max_fd >= cur_max)
max_unshare_fds = fd;

> 
> So "max_fd+1" would overflow to 0, and then this would never trigger.
> 
> Other than that it looks what what I imagine my feverdreams were about.

Thanks!
Christian


Re: [PATCH v5 0/3] close_range()

2020-06-03 Thread Linus Torvalds
On Wed, Jun 3, 2020 at 4:24 PM Christian Brauner
 wrote:
>
> Ok, here's what I have. Does the below look somewhat sane?

Probably. Needs lots of testing. But this one looks wrong:

> +int __close_range(unsigned fd, unsigned max_fd, unsigned int flags)
>  {
> +   if ((max_fd + 1) >= cur_max)
> +   max_unshare_fds = fd;

A normal value for "close everything starting at X" would have a
max_fd value of ~0.

So "max_fd+1" would overflow to 0, and then this would never trigger.

Other than that it looks what what I imagine my feverdreams were about.

  Linus


Re: [PATCH v5 0/3] close_range()

2020-06-03 Thread Christian Brauner
On Tue, Jun 02, 2020 at 05:08:22PM -0700, Linus Torvalds wrote:
> On Tue, Jun 2, 2020 at 4:33 PM Christian Brauner
>  wrote:
> > >
> > > And maybe this _did_ get mentioned last time, and I just don't find
> > > it. I also don't see anything like that in the patches, although the
> > > flags argument is there.
> >
> > I spent some good time digging and I couldn't find this mentioned
> > anywhere so maybe it just never got sent to the list?
> 
> It's entirely possible that it was just a private musing, and you
> re-opening this issue just resurrected the thought.
> 
> I'm not sure how simple it would be to implement, but looking at it it
> shouldn't be problematic to add a "max_fd" argument to unshare_fd()
> and dup_fd().
> 
> Although the range for unsharing is obviously reversed, so I'd suggest
> not trying to make "dup_fd()" take the exact range into account.
> 
> More like just making __close_range() do basically something like
> 
> rcu_read_lock();
> cur_max = files_fdtable(files)->max_fds;
> rcu_read_unlock();
> 
> if (flags & CLOSE_RANGE_UNSHARE) {
> unsigned int max_unshare_fd = ~0u;
> if (cur_max >= max_fd)
> max_unshare_fd = fd;
> unshare_fd(max_unsgare_fd);
> }
> 
> .. do the rest of __close_range() here ..
> 
> and all that "max_unsgare_fd" would do would be to limit the top end
> of the file descriptor table unsharing: we'd still do the exact range
> handling in __close_range() itself.
> 
> Because teaching unshare_fd() and dup_fd() about anything more complex
> than the above doesn't sound worth it, but adding a way to just avoid
> the unnecessary copy of any high file descriptors sounds simple
> enough.

Ok, here's what I have. (I think in your example above cur_max and
max_fd are switched or I might have missed your point completely.) I was
a little in doubt whether capping dup_fd() between NR_OPEN_DEFAULT and
open_files was a sane thing to do but I think it is. Torture testing
this with a proper test-suite and with all debugging options enabled
didn't yet find any obvious issues. Does the below look somewhat sane?:

>From 4ee3fdac02f3cd70e31669e35d3f494913f3fd3f Mon Sep 17 00:00:00 2001
From: Christian Brauner 
Date: Wed, 3 Jun 2020 21:48:55 +0200
Subject: [PATCH 1/2] close_range: add CLOSE_RANGE_UNSHARE

One of the use-cases of close_range() is to drop file descriptors just before
execve(). This would usually be expressed in the sequence:

unshare(CLONE_FILES);
close_range(3, ~0U);

as pointed out by Linus it might be desirable to have this be a part of
close_range() itself under a new flag CLOSE_RANGE_UNSHARE.

This expands {dup,unshare)_fd() to take a max_fds argument that indicates the
maximum number of file descriptors to copy from the old struct files. When the
user requests that all file descriptors are supposed to be closed via
close_range(min, max) then we can cap via unshare_fd(min) and hence don't need
to do any of the heavy fput() work for everything above min.

The patch makes it so that if CLOSE_RANGE_UNSHARE is requested and we do in
fact currently share our file descriptor table we create a new private copy.
We then close all fds in the requested range and finally after we're done we
install the new fd table.

Suggested-by: Linus Torvalds 
Signed-off-by: Christian Brauner 
---
 fs/file.c| 65 
 fs/open.c|  5 +--
 include/linux/fdtable.h  |  8 ++--
 include/uapi/linux/close_range.h |  9 +
 kernel/fork.c| 11 +++---
 5 files changed, 79 insertions(+), 19 deletions(-)
 create mode 100644 include/uapi/linux/close_range.h

diff --git a/fs/file.c b/fs/file.c
index e260bfe687d1..718356ed6682 100644
--- a/fs/file.c
+++ b/fs/file.c
@@ -19,6 +19,7 @@
 #include 
 #include 
 #include 
+#include 
 
 unsigned int sysctl_nr_open __read_mostly = 1024*1024;
 unsigned int sysctl_nr_open_min = BITS_PER_LONG;
@@ -265,12 +266,22 @@ static unsigned int count_open_files(struct fdtable *fdt)
return i;
 }
 
+static unsigned int sane_fdtable_size(struct fdtable *fdt, unsigned int 
max_fds)
+{
+   unsigned int count;
+
+   count = count_open_files(fdt);
+   if (max_fds < NR_OPEN_DEFAULT)
+   max_fds = NR_OPEN_DEFAULT;
+   return min(count, max_fds);
+}
+
 /*
  * Allocate a new files structure and copy contents from the
  * passed in files structure.
  * errorp will be valid only when the returned files_struct is NULL.
  */
-struct files_struct *dup_fd(struct files_struct *oldf, int *errorp)
+struct files_struct *dup_fd(struct files_struct *oldf, unsigned int max_fds, 
int *errorp)
 {
struct files_struct *newf;
struct file **old_fds, **new_fds;
@@ -297,7 +308,7 @@ struct files_struct *dup_fd(struct files_struct *oldf, int 
*errorp)
 
spin_lock(>file_lock);
old_fdt = files_fdtable(oldf);
-   open_files = 

Re: [PATCH v5 0/3] close_range()

2020-06-02 Thread Linus Torvalds
On Tue, Jun 2, 2020 at 4:33 PM Christian Brauner
 wrote:
> >
> > And maybe this _did_ get mentioned last time, and I just don't find
> > it. I also don't see anything like that in the patches, although the
> > flags argument is there.
>
> I spent some good time digging and I couldn't find this mentioned
> anywhere so maybe it just never got sent to the list?

It's entirely possible that it was just a private musing, and you
re-opening this issue just resurrected the thought.

I'm not sure how simple it would be to implement, but looking at it it
shouldn't be problematic to add a "max_fd" argument to unshare_fd()
and dup_fd().

Although the range for unsharing is obviously reversed, so I'd suggest
not trying to make "dup_fd()" take the exact range into account.

More like just making __close_range() do basically something like

rcu_read_lock();
cur_max = files_fdtable(files)->max_fds;
rcu_read_unlock();

if (flags & CLOSE_RANGE_UNSHARE) {
unsigned int max_unshare_fd = ~0u;
if (cur_max >= max_fd)
max_unshare_fd = fd;
unshare_fd(max_unsgare_fd);
}

.. do the rest of __close_range() here ..

and all that "max_unsgare_fd" would do would be to limit the top end
of the file descriptor table unsharing: we'd still do the exact range
handling in __close_range() itself.

Because teaching unshare_fd() and dup_fd() about anything more complex
than the above doesn't sound worth it, but adding a way to just avoid
the unnecessary copy of any high file descriptors sounds simple
enough.

But I haven't thought deeply about this. I might have missed something.

Linus


Re: [PATCH v5 0/3] close_range()

2020-06-02 Thread Christian Brauner
On Tue, Jun 02, 2020 at 02:03:09PM -0700, Linus Torvalds wrote:
> On Tue, Jun 2, 2020 at 1:42 PM Christian Brauner
>  wrote:
> >
> > This is a resend of the close_range() syscall, as discussed in [1]. There 
> > weren't any outstanding
> > discussions anymore and this was in mergeable shape. I simply hadn't gotten 
> > around to moving this
> > into my for-next the last few cycles and then forgot about it. Thanks to 
> > Kyle and the Python people,
> > and others for consistenly reminding me before every merge window and mea 
> > culpa for not moving on
> > this sooner. I plan on moving this into for-next after v5.8-rc1 has been 
> > released and targeting the
> > v5.9 merge window.
> 
> Btw, I did have one reaction that I can't find in the original thread,
> which probably means that it got lost.
> 
> If one of the designed uses for this is for dropping file descriptors
> just before execve(), it's possible that we'd want to have the option
> to say "unshare my fd array" as part of close_range().
> 
> Yes, yes, you can do
> 
> unshare(CLONE_FILES);
> close_range(3,~0u);
> 
> to do it as two operations (and you had that as the example typical
> use), but it would actually be better to be able to do
> 
> close_range(3, ~0ul, CLOSE_RANGE_UNSHARE);
> 
> instead. Because otherwise we just waste time copying the file
> descriptors first in the unshare, and then closing them after.. Double
> the work..
> 
> And maybe this _did_ get mentioned last time, and I just don't find
> it. I also don't see anything like that in the patches, although the
> flags argument is there.

I spent some good time digging and I couldn't find this mentioned
anywhere so maybe it just never got sent to the list?
It sounds pretty useful, so yeah let me add a patch for this tomorrow.

Christian


Re: [PATCH v5 0/3] close_range()

2020-06-02 Thread Linus Torvalds
On Tue, Jun 2, 2020 at 1:42 PM Christian Brauner
 wrote:
>
> This is a resend of the close_range() syscall, as discussed in [1]. There 
> weren't any outstanding
> discussions anymore and this was in mergeable shape. I simply hadn't gotten 
> around to moving this
> into my for-next the last few cycles and then forgot about it. Thanks to Kyle 
> and the Python people,
> and others for consistenly reminding me before every merge window and mea 
> culpa for not moving on
> this sooner. I plan on moving this into for-next after v5.8-rc1 has been 
> released and targeting the
> v5.9 merge window.

Btw, I did have one reaction that I can't find in the original thread,
which probably means that it got lost.

If one of the designed uses for this is for dropping file descriptors
just before execve(), it's possible that we'd want to have the option
to say "unshare my fd array" as part of close_range().

Yes, yes, you can do

unshare(CLONE_FILES);
close_range(3,~0u);

to do it as two operations (and you had that as the example typical
use), but it would actually be better to be able to do

close_range(3, ~0ul, CLOSE_RANGE_UNSHARE);

instead. Because otherwise we just waste time copying the file
descriptors first in the unshare, and then closing them after.. Double
the work..

And maybe this _did_ get mentioned last time, and I just don't find
it. I also don't see anything like that in the patches, although the
flags argument is there.

Linus