RE: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

From: Al Viro > Sent: 04 November 2015 16:28 > On Wed, Nov 04, 2015 at 03:54:09PM +, David Laight wrote: > > > Sigh... The kernel has no idea when other threads are done with "all > > > io activities using that fd" - it can wait for them to leave the > > > kernel mode, but there's fuck-all it

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Fri, Nov 06, 2015 at 03:07:30PM +, David Laight wrote: > > Oh, _lovely_. So instead of continuation of that write(2) going down > > the throat of something opened by unrelated thread, it (starting from a > > pretty arbitrary point) will go into the descriptor the closing thread > > passed

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Wed, Nov 04, 2015 at 03:54:09PM +, David Laight wrote: > > Sigh... The kernel has no idea when other threads are done with "all > > io activities using that fd" - it can wait for them to leave the > > kernel mode, but there's fuck-all it can do about e.g. a userland > > loop doing write()

RE: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

From: Al Viro > Sent: 30 October 2015 21:10 > On Fri, Oct 30, 2015 at 05:43:21PM +, David Laight wrote: > > > ISTM that the correct call should be listen(fd, 0); > > Although that doesn't help a thread stuck in recvmsg() for a datagram. > > > > It is also tempting to think that close(fd)

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Mon, 2015-11-02 at 00:24 +, Al Viro wrote: > This ought to be a bit cleaner. Eric, could you test the variant below on > your > setup? Sure ! 5 runs of : lpaa24:~# taskset ff0ff ./opensock -t 16 -n 1000 -l 10 total = 4386311 total = 4560402 total = 4437309 total = 4516227 total =

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Sun, Nov 1, 2015 at 4:24 PM, Al Viro wrote: > > This ought to be a bit cleaner. Eric, could you test the variant below on > your > setup? I'd love to see the numbers, but at the same time I really can't say I love your patch. I've merged my own two patches for now

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Sat, Oct 31, 2015 at 08:29:29PM +, Al Viro wrote: > On Sat, Oct 31, 2015 at 12:54:50PM -0700, Linus Torvalds wrote: > > On Sat, Oct 31, 2015 at 12:34 PM, Al Viro wrote: > > > > > > ... and here's the current variant of mine. > > > > Ugh. I really liked how simple

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Sun, Nov 01, 2015 at 06:14:43PM -0800, Eric Dumazet wrote: > On Mon, 2015-11-02 at 00:24 +, Al Viro wrote: > > > This ought to be a bit cleaner. Eric, could you test the variant below on > > your > > setup? > > Sure ! > > 5 runs of : > lpaa24:~# taskset ff0ff ./opensock -t 16 -n

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Fri, Oct 30, 2015 at 04:52:41PM -0700, Linus Torvalds wrote: > I really suspect this patch is "good enough" in reality, and I would > *much* rather do something like this than add a new non-POSIX flag > that people have to update their binaries for. I agree with Eric that > *some* people will

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Sat, Oct 31, 2015 at 12:54:50PM -0700, Linus Torvalds wrote: > On Sat, Oct 31, 2015 at 12:34 PM, Al Viro wrote: > > > > ... and here's the current variant of mine. > > Ugh. I really liked how simple mine ended up being. Yours is definitely not. Note that it's not the

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Sat, Oct 31, 2015 at 12:34 PM, Al Viro wrote: > > ... and here's the current variant of mine. Ugh. I really liked how simple mine ended up being. Yours is definitely not. And based on the profiles from Eric, finding the fd is no longer the problem even with my

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Sat, 2015-10-31 at 12:54 -0700, Linus Torvalds wrote: > On Sat, Oct 31, 2015 at 12:34 PM, Al Viro wrote: > > > > ... and here's the current variant of mine. > > Ugh. I really liked how simple mine ended up being. Yours is definitely not. > > And based on the profiles

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Sat, Oct 31, 2015 at 1:45 PM, Eric Dumazet wrote: > 13.84% opensock [kernel.kallsyms][k] queued_spin_lock_slowpath > | > --- queued_spin_lock_slowpath > | > |--99.97%--

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Sat, Oct 31, 2015 at 02:23:31PM -0700, Linus Torvalds wrote: > The other stuff we probably can't do all that much about. Unless we > decide to go for some complicated lockless optimistic file descriptor > allocation scheme with retry-on-failure instead of locks. Which I'm > sure is possible,

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Fri, 2015-10-30 at 16:52 -0700, Linus Torvalds wrote: sequential allocations... > > I don't think it would matter in real life, since I don't really think > you have lots of fd's with strictly sequential behavior. > > That said, the trivial "open lots of fds" benchmark would show it, so > I

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Sat, 2015-10-31 at 14:23 -0700, Linus Torvalds wrote: > Mind testing something really stupid, and making the __clear_bit() in > __clear_close_on_exec() conditiona, something like this: > > static inline void __clear_close_on_exec(int fd, struct fdtable *fdt) > { > -

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Fri, Oct 30, 2015 at 2:23 PM, Linus Torvalds wrote: > On Fri, Oct 30, 2015 at 2:02 PM, Al Viro wrote: >> >> Your variant has 1:64 ratio; obviously better than now, but we can actually >> do 1:bits-per-cacheline quite easily. > > Ok, but

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Fri, Oct 30, 2015 at 2:02 PM, Al Viro wrote: > > Your variant has 1:64 ratio; obviously better than now, but we can actually > do 1:bits-per-cacheline quite easily. Ok, but in that case you end up needing a counter for each cacheline too (to count how many bits, in

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Fri, Oct 30, 2015 at 3:33 PM, Al Viro wrote: > On Fri, Oct 30, 2015 at 02:50:46PM -0700, Linus Torvalds wrote: > >> Anyway. This is a pretty simple patch, and I actually think that we >> could just get rid of the "next_fd" logic entirely with this. That >> would make

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Fri, 2015-10-30 at 14:50 -0700, Linus Torvalds wrote: > On Fri, Oct 30, 2015 at 2:23 PM, Linus Torvalds > wrote: > > On Fri, Oct 30, 2015 at 2:02 PM, Al Viro wrote: > >> > >> Your variant has 1:64 ratio; obviously better than now, but we

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Fri, Oct 30, 2015 at 04:52:41PM -0700, Linus Torvalds wrote: > I really suspect this patch is "good enough" in reality, and I would > *much* rather do something like this than add a new non-POSIX flag > that people have to update their binaries for. I agree with Eric that > *some* people will

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Thu, Oct 29, 2015 at 5:35 AM, Eric Dumazet wrote: > > Well, I already tested the O_FD_FASTALLOC thing, and I can tell you > find_next_zero_bit() is nowhere to be found in kernel profiles anymore. > It also lowers time we hold the fd array spinlock while doing fd alloc.

RE: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

From: David Holland > Sent: 29 October 2015 14:59 > On Tue, Oct 27, 2015 at 10:52:46AM +, Alan Burlison wrote: > > >But in general, this is basically a problem with the application: the file > > >descriptor space is shared between threads and having one thread sniping > > >at open files,

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Fri, Oct 30, 2015 at 10:18:12AM -0700, Linus Torvalds wrote: > I do wonder if we couldn't just speed up the bitmap allocator by an > order of magnitude. It would be nicer to be able to make existing > loads faster without any new "don't bother with POSIX semantics" flag. > > We could, for

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Fri, Oct 30, 2015 at 05:43:21PM +, David Laight wrote: > ISTM that the correct call should be listen(fd, 0); > Although that doesn't help a thread stuck in recvmsg() for a datagram. > > It is also tempting to think that close(fd) should sleep until all > io activities using that fd have

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Fri, Oct 30, 2015 at 02:50:46PM -0700, Linus Torvalds wrote: > Anyway. This is a pretty simple patch, and I actually think that we > could just get rid of the "next_fd" logic entirely with this. That > would make this *patch* be more complicated, but it would make the > resulting *code* be

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Thu, 2015-10-29 at 05:35 -0700, Eric Dumazet wrote: > Current kernel : > > 64.98% [kernel] [k] queued_spin_lock_slowpath > 14.88% opensock [.] memset// this part simulates user land > actual work ;) > 11.15% [kernel] [k]

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Thu, 2015-10-29 at 04:16 +, Al Viro wrote: > Have you tried to experiment with that in userland? I mean, emulate that > thing in normal userland code, count the cacheline accesses and drive it > with the use patterns collected from actual applications. Sure. > > I can sit down and play

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On 29/10/2015 14:58, David Holland wrote: ISTM that the best way to do this is to post a signal to the thread so accept bails with EINTR, at which point it can check to see if it's supposed to be exiting. Yes, you could use pthread_kill, but that would require keeping a list of the tids of

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Thu, Oct 29, 2015 at 03:18:40PM +, Alan Burlison wrote: > On 29/10/2015 14:58, David Holland wrote: > >ISTM that the best way to do this is to post a signal to the thread so > >accept bails with EINTR, at which point it can check to see if it's > >supposed to be exiting. > > Yes, you

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Tue, Oct 27, 2015 at 10:52:46AM +, Alan Burlison wrote: > >But in general, this is basically a problem with the application: the file > >descriptor space is shared between threads and having one thread sniping > >at open files, you do have a problem and whatever the kernel does in that

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On 29/10/2015 16:01, David Holland wrote: Hardly; it moves the burden of doing stupid things to the application. If as you said the goal is to shut down all threads cleanly, then it doesn't need to keep track in detail anyway; it can just post SIGTERM to every thread, or SIGUSR1 if SIGTERM is

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

[off list] On 29/10/2015 17:07, Al Viro wrote: Could the esteemed sir possibly be ars^H^H^Hprevailed upon to quote the exact place in POSIX that requires such behaviour? If that's the way the conversation is going to go, sorry, no. -- Alan Burlison -- -- To unsubscribe from this list: send

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Thu, Oct 29, 2015 at 04:15:33PM +, Alan Burlison wrote: > There was an attempt to interpret POSIX that way, with which I still > disagree. If a FD is closed or reassigned then any current pending > operations on it should be terminated. Could the esteemed sir possibly be

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Thu, Oct 29, 2015 at 04:15:33PM +, Alan Burlison wrote: > >close(2) as specified by POSIX doesn't prohibit this weird revoke-like > >behavior, but there's nothing in there that mandates it either. (I > >thought this discussion had already clarified that.) > > There was an attempt to

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

From: Alan Burlison Date: Thu, 29 Oct 2015 17:12:44 + > On 29/10/2015 17:07, Al Viro wrote: > >> Could the esteemed sir possibly be ars^H^H^Hprevailed upon to quote >> the exact >> place in POSIX that requires such behaviour? > > If that's the way the conversation

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

From: Al Viro Date: Thu, 29 Oct 2015 17:07:48 + > _IF_ you are doing that kind of "close a descriptor under other threads" > thing, you need to inform the potentially affected threads anyway, and > you'd better not rely on them being currently in kernel mode. +1 --

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Wed, 2015-10-28 at 12:35 +, Al Viro wrote: > [Linus and Dave added, Solaris and NetBSD folks dropped from Cc] > > On Tue, Oct 27, 2015 at 05:13:56PM -0700, Eric Dumazet wrote: > > On Tue, 2015-10-27 at 23:17 +, Al Viro wrote: > > > > > * [Linux-specific aside] our __alloc_fd() can

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

[Linus and Dave added, Solaris and NetBSD folks dropped from Cc] On Tue, Oct 27, 2015 at 05:13:56PM -0700, Eric Dumazet wrote: > On Tue, 2015-10-27 at 23:17 +, Al Viro wrote: > > > * [Linux-specific aside] our __alloc_fd() can degrade quite badly > > with some use patterns. The

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Wed, 2015-10-28 at 06:24 -0700, Eric Dumazet wrote: > Before I take a deep look at your suggestion, are you sure plain use of > include/linux/percpu-refcount.h infra is not possible for struct cred ? BTW, I am not convinced we need to spend so much energy and per-cpu memory for struct cred

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On 27/10/2015 23:17, Al Viro wrote: Frankly, as far as I'm concerned, the bottom line is * there are two variants of semantics in that area and there's not much that could be done about that. Yes, that seems to be the case. * POSIX is vague enough for both variants to comply

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Thu, 2015-10-29 at 00:15 +, Al Viro wrote: > On Wed, Oct 28, 2015 at 04:08:29PM -0700, Eric Dumazet wrote: > > > > Except for legacy stuff and stdin/stdout/stderr games, I really doubt > > > > lot of applications absolutely rely on the POSIX thing... > > > > > > We obviously can't turn

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Wed, Oct 28, 2015 at 08:29:41PM -0700, Eric Dumazet wrote: > But this is an optimization : If you do not use the initial dup2(), the > fd array can be automatically expanded if needed (all slots are in use) Whee... > No locking change. files->file_lock is still taken. > > We only want to

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Wed, 2015-10-28 at 21:13 +, Al Viro wrote: > On Wed, Oct 28, 2015 at 07:47:57AM -0700, Eric Dumazet wrote: > > On Wed, 2015-10-28 at 06:24 -0700, Eric Dumazet wrote: > > > > > Before I take a deep look at your suggestion, are you sure plain use of > > > include/linux/percpu-refcount.h

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Wed, Oct 28, 2015 at 07:47:57AM -0700, Eric Dumazet wrote: > On Wed, 2015-10-28 at 06:24 -0700, Eric Dumazet wrote: > > > Before I take a deep look at your suggestion, are you sure plain use of > > include/linux/percpu-refcount.h infra is not possible for struct cred ? > > BTW, I am not

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Wed, Oct 28, 2015 at 02:44:28PM -0700, Eric Dumazet wrote: > Well, all this complexity goes away with a O_FD_FASTALLOC / > SOCK_FD_FASTALLOC bit in various fd allocations, which specifically > tells the kernel we do not care getting the lowest possible fd as POSIX > mandates. ... which won't

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Wed, 2015-10-28 at 22:33 +, Al Viro wrote: > On Wed, Oct 28, 2015 at 02:44:28PM -0700, Eric Dumazet wrote: > > > Well, all this complexity goes away with a O_FD_FASTALLOC / > > SOCK_FD_FASTALLOC bit in various fd allocations, which specifically > > tells the kernel we do not care getting

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Wed, Oct 28, 2015 at 04:08:29PM -0700, Eric Dumazet wrote: > > > Except for legacy stuff and stdin/stdout/stderr games, I really doubt > > > lot of applications absolutely rely on the POSIX thing... > > > > We obviously can't turn that into default behaviour, though. BTW, what > >

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

>And no, I'm not fond of such irregular ways to pass file descriptors, but >we can't kill ioctl(2) with all weirdness hiding behind it, more's the pity... Yeah, there are a number of calls which supposed work on one but have a second argument which is also a file descriptor; mostly part of

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On 27/10/2015 09:08, casper@oracle.com wrote: Generally I wouldn't see that as a problem, but in the case of a socket blocking on accept indefinitely, I do see it as a problem especially as the thread actually wants to stop listening. But in general, this is basically a problem with the

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

From: Alan Burlison Date: Tue, 27 Oct 2015 10:52:46 + > an implicit shutdown() it would mean that well-written applications > that handled the scoping, sharing and reuse of FDs properly could just > call close() and have it work the same way across *NIX platforms.

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On 27/10/2015 12:01, Eric Dumazet wrote: Are non multi threaded applications considered well written ? listener = socket(...); bind(listener, ...); listen(fd, 1); Loop 1 10 if (fork() == 0) do_accept(listener) Now if a child does a close(listener), or is killed, you propose that

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Tue, 2015-10-27 at 12:27 +, Alan Burlison wrote: > On 27/10/2015 12:01, Eric Dumazet wrote: > > > Are non multi threaded applications considered well written ? > > > > listener = socket(...); > > bind(listener, ...); > > listen(fd, 1); > > Loop 1 10 > >if (fork() == 0) > >

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Tue, 2015-10-27 at 10:52 +, Alan Burlison wrote: > Unfortunately Hadoop isn't the only thing that pulls the shutdown() > trick, so I don't think there's a simple fix for this, as discussed > earlier in the thread. Having said that, if close() on Linux also did an > implicit shutdown()

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On 27/10/2015 14:39, David Miller wrote: You will never be able to assume it is available everywhere under Linux. Ever. This is the fundamental issue that you seem to completely not understand. If that was true in general then Linux would be dead and there would be no point ever adding any

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

From: Alan Burlison Date: Tue, 27 Oct 2015 14:39:56 + > On 27/10/2015 14:39, David Miller wrote: > >> Making this worse is that there isn't going to be a straightforward >> nor reliable way to test for the presence of this at run time. > > I attached a test case

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On 27/10/2015 15:04, David Miller wrote: I said "run time". Like when your final code runs on a target system. Not at build tree configure time. Yes, you could do it then as well. -- Alan Burlison -- -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Tue, 2015-10-27 at 23:17 +, Al Viro wrote: > * [Linux-specific aside] our __alloc_fd() can degrade quite badly > with some use patterns. The cacheline pingpong in the bitmap is probably > inevitable, unless we accept considerably heavier memory footprint, > but we also have a case

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Tue, Oct 27, 2015 at 10:52:46AM +, Alan Burlison wrote: > Unfortunately Hadoop isn't the only thing that pulls the shutdown() > trick, so I don't think there's a simple fix for this, as discussed > earlier in the thread. Having said that, if close() on Linux also > did an implicit

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

From: Alan Burlison Date: Tue, 27 Oct 2015 14:13:31 + > Ideally there'd be a single way of doing this that worked > cross-platform, at the moment there isn't. And yes, even if such a > mechanism was available now it would be some time before it could be > assumed to

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

From: Alan Burlison Date: Tue, 27 Oct 2015 13:37:26 + > If you took that argument to it's logical extreme they you'd never > make any changes that made changes to existing behaviour, and that's > patently not the case. You know exactly what I mean, and what you're

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On 27/10/2015 13:42, David Miller wrote: This semantic would only exist after Linux version X.Y.Z and vendor kernels that decided to backport the feature. Ergo, this application would ironically be non-portable on Linux machines. Yes, that's true enough, until nobody was using the old

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On 27/10/2015 13:59, David Miller wrote: From: Alan Burlison Date: Tue, 27 Oct 2015 13:37:26 + If you took that argument to it's logical extreme they you'd never make any changes that made changes to existing behaviour, and that's patently not the case. You

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On 23/10/2015 17:00, Eric Dumazet wrote: Ermm, you *really* want me to submit a patch removing 'Conforms to POSIX.1-2001' from *every* Linux manpage? Only on the pages you think there is an error that matters. If there's consensus that the current shutdown(), dup2(), close() and accept()

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Fri, 2015-10-23 at 09:00 -0700, Eric Dumazet wrote: > on the pages you think there is an error that matters. > > > > > > Have you tested the patch I sent ? > > > > The AF_UNIX poll one? No, I don't have the means to do so, and in any > > case that's not a POSIX issue, just a plain bug. I'm

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On 23/10/2015 15:21, Eric Dumazet wrote: I claim nothing. If you believe a man page should be fixed, please send a patch to man page maintainer. Ermm, you *really* want me to submit a patch removing 'Conforms to POSIX.1-2001' from *every* Linux manpage? Have you tested the patch I sent ?

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Fri, 2015-10-23 at 16:46 +0100, Alan Burlison wrote: > On 23/10/2015 15:21, Eric Dumazet wrote: > > > I claim nothing. If you believe a man page should be fixed, please send > > a patch to man page maintainer. > > Ermm, you *really* want me to submit a patch removing 'Conforms to >

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Fri, 2015-10-23 at 15:20 +0200, casper@oracle.com wrote: > This point was not about a 10M sockets server but in general... > Hey, we do not design the OS only for handling desktop workloads. If netstat does not work on this typical server workload, then it is a joke. > Solaris has had

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Fri, 2015-10-23 at 15:20 +0200, casper@oracle.com wrote: > > >Yet another POSIX deficiency. > > > >When a server deals with 10,000,000+ socks, we absolutely do not care of > >this requirement. > > > >O(log(n)) is still crazy if it involves O(log(n)) cache misses. > > You miss the fire

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Fri, 2015-10-23 at 11:52 +0200, casper@oracle.com wrote: > > >Ho-hum... It could even be made lockless in fast path; the problems I see > >are > > * descriptor-to-file lookup becomes unsafe in a lot of locking > >conditions. Sure, most of that happens on the entry to some syscall,

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

>Yet another POSIX deficiency. > >When a server deals with 10,000,000+ socks, we absolutely do not care of >this requirement. > >O(log(n)) is still crazy if it involves O(log(n)) cache misses. You miss the fire point of the algorithm; you *always* find an available file descriptor in O(log(N))

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On 23/10/2015 14:02, Eric Dumazet wrote: Other data is used to make sure that it only takes O(log(n)) to find the lowest available file descriptor entry. (Where n, I think, is the returned descriptor) Yet another POSIX deficiency. When a server deals with 10,000,000+ socks, we absolutely do

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

>Ho-hum... It could even be made lockless in fast path; the problems I see >are > * descriptor-to-file lookup becomes unsafe in a lot of locking >conditions. Sure, most of that happens on the entry to some syscall, with >very light locking environment, but... auditing every sodding ioctl

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On 23/10/2015 17:19, Eric Dumazet wrote: The AF_UNIX poll one? No, I don't have the means to do so, and in any case that's not a POSIX issue, just a plain bug. I'm happy to log a bug if that helps. BTW, there is no kernel bug here. POSIX poll() man page says : POLLOUT Normal data may be

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Thu, Oct 22, 2015 at 09:50:10PM +0200, casper@oracle.com wrote: > >Sigh... It completely fails to mention descriptor-passing. Which > > a) is relevant to what "last close" means and > > b) had been there for nearly the third of a century. > > Why is that different? These clearly

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Fri, 2015-10-23 at 17:40 +0100, Alan Burlison wrote: > On 23/10/2015 17:19, Eric Dumazet wrote: > > >>> The AF_UNIX poll one? No, I don't have the means to do so, and in any > >>> case that's not a POSIX issue, just a plain bug. I'm happy to log a bug > >>> if that helps. > > > > BTW, there is

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Fri, Oct 23, 2015 at 11:52:34AM +0200, casper@oracle.com wrote: > > > >Ho-hum... It could even be made lockless in fast path; the problems I see > >are > > * descriptor-to-file lookup becomes unsafe in a lot of locking > >conditions. Sure, most of that happens on the entry to some

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Fri, 2015-10-23 at 14:35 +0100, Alan Burlison wrote: > If you think it's a POSIX deficiency then logging a DR is probably the > correct way of addressing that. And as I've said it's fine to decide > that you don't care about what POSIX says on the subject but you can't > simultaneously

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

>On Thu, Oct 22, 2015 at 08:34:19AM +0200, casper@oracle.com wrote: >> >> >> >And I'm really curious about the things Solaris would do with dup2() there. >> >Does it take into account the possibility of new accept() coming just as >> >dup2() is trying to terminate the ongoing ones? Is

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

From: Al Viro >Except that in this case "correctness" is the matter of rather obscure and >ill-documented areas in POSIX. Don't get me wrong - this semantics isn't >inherently bad, but it's nowhere near being an absolute requirement. It would more fruitful to have

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Thu, Oct 22, 2015 at 05:44:58AM +0100, Al Viro wrote: > Except that in this case "correctness" is the matter of rather obscure and > ill-documented areas in POSIX. Don't get me wrong - this semantics isn't > inherently bad, but it's nowhere near being an absolute requirement. PS: in

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

>It's been said that the current mechanisms in Linux & some BSD variants >can be subject to races, and the behaviour exhibited doesn't conform to >POSIX, for example requiring the use of shutdown() on unconnected >sockets because close() doesn't kick off other threads accept()ing on >the same

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On 22/10/2015 18:05, Al Viro wrote: Oh, for... Right in this thread an example of complete BS has been quoted from POSIX close(2). The part about closing a file when the last descriptor gets closed. _Nothing_ is POSIX-compliant in that respect (nor should it be). That's not exactly what it

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

>On Thu, Oct 22, 2015 at 08:24:51PM +0200, casper@oracle.com wrote: > >> The external behaviour atomic; you cannot distinguish the order >> between the closing of the original file (and waking up other threads >> waiting for a record lock) or changing the file referenced by that newfd. >> >>

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Thu, Oct 22, 2015 at 06:39:34PM +0100, Alan Burlison wrote: > On 22/10/2015 18:05, Al Viro wrote: > > >Oh, for... Right in this thread an example of complete BS has been quoted > >from POSIX close(2). The part about closing a file when the last descriptor > >gets closed. _Nothing_ is

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Thu, Oct 22, 2015 at 08:24:51PM +0200, casper@oracle.com wrote: > The external behaviour atomic; you cannot distinguish the order > between the closing of the original file (and waking up other threads > waiting for a record lock) or changing the file referenced by that newfd. > > But

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Thu, Oct 22, 2015 at 08:34:19AM +0200, casper@oracle.com wrote: > > > >And I'm really curious about the things Solaris would do with dup2() there. > >Does it take into account the possibility of new accept() coming just as > >dup2() is trying to terminate the ongoing ones? Is there a

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

From: Al Viro >On Thu, Oct 22, 2015 at 06:39:34PM +0100, Alan Burlison wrote: >> On 22/10/2015 18:05, Al Viro wrote: >> >> >Oh, for... Right in this thread an example of complete BS has been quoted >> >from POSIX close(2). The part about closing a file when the last

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On 22/10/2015 07:51, casper@oracle.com wrote: It would more fruitful to have such a discussion in one of the OpenGroup mailing lists; people gathered there have a lot of experience and it is also possible to fix the standard when it turns out that it indeed as vague as you claim it is (I

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On 22/10/2015 05:44, Al Viro wrote: It's been said that the current mechanisms in Linux & some BSD variants can be subject to races You do realize that it goes for the entire area? And the races found in this thread are in the BSD variant that tries to do something similar to what you guys

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Thu, 2015-10-22 at 08:15 +0200, casper@oracle.com wrote: > >It's been said that the current mechanisms in Linux & some BSD variants > >can be subject to races, and the behaviour exhibited doesn't conform to > >POSIX, for example requiring the use of shutdown() on unconnected > >sockets

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On 22/10/2015 12:30, Eric Dumazet wrote: We absolutely do not _want_ to do this just so that linux becomes slower to the point Solaris can compete, or you guys can avoid some work. Sentiments such as that really have no place in a discussion that's been focussed primarily on the behaviour of

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Thu, 2015-10-22 at 12:58 +0100, Alan Burlison wrote: > On 22/10/2015 12:30, Eric Dumazet wrote: > > > We absolutely do not _want_ to do this just so that linux becomes slower > > to the point Solaris can compete, or you guys can avoid some work. > > Sentiments such as that really have no

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Thu, Oct 22, 2015 at 09:51:05PM +0200, casper@oracle.com wrote: > > >On Thu, Oct 22, 2015 at 08:24:51PM +0200, casper@oracle.com wrote: > > > >> The external behaviour atomic; you cannot distinguish the order > >> between the closing of the original file (and waking up other threads >

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

From: Eric Dumazet Date: Thu, 22 Oct 2015 05:10:58 -0700 > I am a major linux networking developper and wont accept linux is > hijacked by guys who never contributed to it, just so it meets their > unreasonable expectations. > > We absolutely care about performance. And

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On 22/10/2015 13:10, Eric Dumazet wrote: Sentiments such as that really have no place in a discussion that's been focussed primarily on the behaviour of interfaces, albeit with digressions into the potential performance impacts. The discussion has been cordial and I for one appreciate Al Viro's

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

>And I'm really curious about the things Solaris would do with dup2() there. >Does it take into account the possibility of new accept() coming just as >dup2() is trying to terminate the ongoing ones? Is there a window when >descriptor-to-file lookups would fail? Looks like a race/deadlock

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On Thu, Oct 22, 2015 at 02:14:42PM +0100, Alan Burlison wrote: > The issues I hit were in the context of application porting, where > the APIs in question are covered by POSIX. The Linux manpages for > open(), close(), socket(), dup2() and shutdown() all claim > POSIX.1-2001 conformance. If

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

From: Alan Burlison Date: Wed, 21 Oct 2015 15:38:51 +0100 > While this algorithm is pretty expensive, it is not often invoked. I bet it can be easily intentionally invoked, by a malicious entity no less. -- To unsubscribe from this list: send the line "unsubscribe

Re: [Bug 106241] New: shutdown(3)/close(3) behaviour is incorrect for sockets in accept(3)

On 22/10/2015 02:29, David Miller wrote: From: Al Viro Date: Wed, 21 Oct 2015 19:51:04 +0100 Sure, but the upkeep of data structures it would need is there whether you actually end up triggering it or not. Both in memory footprint and in cacheline pingpong... +1

  1   2   >