Re: ipc,sem: sysv semaphore scalability

2013-05-06 Thread Jörn Engel
On Sat, 4 May 2013 20:12:45 +0200, Borislav Petkov wrote: > On Sat, May 04, 2013 at 11:55:58AM -0400, Jörn Engel wrote: > > Blockconsole currently lives here: > > https://git.kernel.org/cgit/linux/kernel/git/joern/bcon2.git/ > > Tja, if only that were upstream... Linus has a pull request. If he

Re: ipc,sem: sysv semaphore scalability

2013-05-04 Thread Borislav Petkov
On Sat, May 04, 2013 at 11:55:58AM -0400, Jörn Engel wrote: > Blockconsole currently lives here: > https://git.kernel.org/cgit/linux/kernel/git/joern/bcon2.git/ Tja, if only that were upstream... -- Regards/Gruss, Boris. Sent from a fat crate under my desk. Formatting is fine. -- -- To unsu

Re: ipc,sem: sysv semaphore scalability

2013-05-04 Thread Jörn Engel
On Sat, 23 March 2013 10:19:16 +0700, Emmanuel Benisty wrote: > > I could reproduce it but could you please let me know what would be > the right tools I should use to catch the original oops? > This is what I got but I doubt it will be helpful: > http://i.imgur.com/Mewi1hC.jpg You could use eith

Re: ipc,sem: sysv semaphore scalability

2013-05-03 Thread Peter Hurley
On 03/29/2013 03:01 PM, Dave Jones wrote: On Tue, Mar 26, 2013 at 12:43:09PM -0700, Andrew Morton wrote: > On Tue, 26 Mar 2013 15:28:52 -0400 Dave Jones wrote: > > > On Thu, Mar 21, 2013 at 02:10:58PM -0700, Andrew Morton wrote: > > > > > Whichever way we go, we should get a wiggle on

Re: ipc,sem: sysv semaphore scalability

2013-04-16 Thread Andrew Morton
On Tue, 26 Mar 2013 11:35:33 -0700 Andrew Morton wrote: > Do we need the locking at all? What does it actually do? > > sem_lock_and_putref(sma); > if (sma->sem_perm.deleted) { > sem_unlock(sma, -1); >

Re: ipc,sem: sysv semaphore scalability

2013-04-02 Thread Dave Jones
On Tue, Apr 02, 2013 at 03:53:01PM -0400, Sasha Levin wrote: > On 04/02/2013 01:52 PM, Linus Torvalds wrote: > > On Tue, Apr 2, 2013 at 9:08 AM, Sasha Levin wrote: > >> > >> By just playing with the 'msgsz' parameter with MSG_COPY set. > > > > Hmm. Looking closer, I suspect you're testing w

Re: ipc,sem: sysv semaphore scalability

2013-04-02 Thread Sasha Levin
On 04/02/2013 01:52 PM, Linus Torvalds wrote: > On Tue, Apr 2, 2013 at 9:08 AM, Sasha Levin wrote: >> >> By just playing with the 'msgsz' parameter with MSG_COPY set. > > Hmm. Looking closer, I suspect you're testing without commit > 88b9e456b164 ("ipc: don't allocate a copy larger than max"). Th

Re: ipc,sem: sysv semaphore scalability

2013-04-02 Thread Linus Torvalds
On Tue, Apr 2, 2013 at 9:08 AM, Sasha Levin wrote: > > By just playing with the 'msgsz' parameter with MSG_COPY set. Hmm. Looking closer, I suspect you're testing without commit 88b9e456b164 ("ipc: don't allocate a copy larger than max"). That should limit the size passed in to prepare_copy -> lo

Re: ipc,sem: sysv semaphore scalability

2013-04-02 Thread Linus Torvalds
On Tue, Apr 2, 2013 at 9:08 AM, Sasha Levin wrote: > > If you guys are already looking at this, the conversions between size_t, > long and int in the do_msgrcv/load_msg/alloc_msg code are a mess. You could > trigger anything from: Good catch. Let's just change the "(long)bufsz < 0" into "bufsz >

Re: ipc,sem: sysv semaphore scalability

2013-04-02 Thread Sasha Levin
On 03/29/2013 03:36 PM, Peter Hurley wrote: > On Fri, 2013-03-29 at 12:26 -0700, Linus Torvalds wrote: >> On Fri, Mar 29, 2013 at 12:06 PM, Dave Jones wrote: >>> >>> Here's an oops I just hit.. >>> >>> BUG: unable to handle kernel NULL pointer dereference at 000f >>> IP: [] testmsg.isr

Re: ipc,sem: sysv semaphore scalability

2013-04-01 Thread Stanislav Kinsbursky
29.03.2013 22:43, Linus Torvalds пишет: On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones wrote: > >Now that I have that reverted, I'm not seeing msgrcv traces any more, but >I've started seeing this.. > >general protection fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC Do you have CONFIG_CHECKPOINT_R

Re: ipc,sem: sysv semaphore scalability

2013-03-31 Thread Linus Torvalds
On Sun, Mar 31, 2013 at 6:45 AM, Rik van Riel wrote: > > Should we use "semid" here, like Linus suggested, instead of "un->semid"? As Davidlohr noted, in linux-next the rcu read-lock is held over the whole thing, so no, un->semid should be stable once "un" has been re-looked-up under the semaphor

Re: ipc,sem: sysv semaphore scalability

2013-03-31 Thread Emmanuel Benisty
Hi Davidlohr, On Sun, Mar 31, 2013 at 12:01 PM, Davidlohr Bueso wrote: > Specially dropping the rcu read lock before the continue statement > (sorry for not mentioning this in the last email). I was missing this indeed, thanks. Still the same issues however... I'll do some more testing on the sa

Re: ipc,sem: sysv semaphore scalability

2013-03-31 Thread Rik van Riel
On 03/31/2013 01:01 AM, Davidlohr Bueso wrote: diff --git a/ipc/sem.c b/ipc/sem.c index f257afe..74cedfe 100644 --- a/ipc/sem.c +++ b/ipc/sem.c @@ -1867,8 +1867,7 @@ void exit_sem(struct task_struct *tsk) struct sem_array *sma; struct sem_undo *un;

Re: ipc,sem: sysv semaphore scalability

2013-03-30 Thread Davidlohr Bueso
On Sat, 2013-03-30 at 11:33 +0700, Emmanuel Benisty wrote: > On Sat, Mar 30, 2013 at 10:46 AM, Linus Torvalds > wrote: > > On Fri, Mar 29, 2013 at 8:02 PM, Emmanuel Benisty > > wrote: > >> > >> Then I start building a random package and the problems start. They > >> may also happen without compi

Re: ipc,sem: sysv semaphore scalability

2013-03-30 Thread Emmanuel Benisty
Hi Linus, On Sun, Mar 31, 2013 at 12:22 AM, Linus Torvalds wrote: > On Fri, Mar 29, 2013 at 10:57 PM, Emmanuel Benisty > wrote: >> On Sat, Mar 30, 2013 at 12:10 PM, Linus Torvalds >>> >>> This came from the gcc build? >> >> yes, very early in the build process, IIRC this line was repeated a >>

Re: ipc,sem: sysv semaphore scalability

2013-03-30 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 10:57 PM, Emmanuel Benisty wrote: > On Sat, Mar 30, 2013 at 12:10 PM, Linus Torvalds >> >> This came from the gcc build? > > yes, very early in the build process, IIRC this line was repeated a > few times and the build just stalled. Ok, we're bringing out the crazy hacks n

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Emmanuel Benisty
On Sat, Mar 30, 2013 at 12:10 PM, Linus Torvalds wrote: >> Another shot in >> the dark, I had this weird message when trying to build gcc: >> semop(2): encountered an error: Identifier removed > > This came from the gcc build? yes, very early in the build process, IIRC this line was repeated a fe

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 9:33 PM, Emmanuel Benisty wrote: > > I just tried the 7 original patches + the 2 one liners from -next + > modified Linus' patch (attached) .. that patch looks fine. > on the top of 3.9-rc4 using > PREEMPT_NONE and after moving sem_lock(sma, NULL, -1) as explained > above

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Emmanuel Benisty
On Sat, Mar 30, 2013 at 10:46 AM, Linus Torvalds wrote: > On Fri, Mar 29, 2013 at 8:02 PM, Emmanuel Benisty wrote: >> >> Then I start building a random package and the problems start. They >> may also happen without compiling but this seems to trigger the bug >> quite quickly. > > I suspect it's

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 8:02 PM, Emmanuel Benisty wrote: > > Then I start building a random package and the problems start. They > may also happen without compiling but this seems to trigger the bug > quite quickly. I suspect it's about preemption, and the build just results in enough scheduling

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Emmanuel Benisty
Hi Davidlohr, On Sat, Mar 30, 2013 at 9:08 AM, Davidlohr Bueso wrote: > Not sure which one liner you refer to, but, if you haven't already done > so, please try with these fixes (queued in linux-next): > > http://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=a9cead0347283f3e

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Davidlohr Bueso
On Fri, 2013-03-29 at 19:09 -0700, Linus Torvalds wrote: > On Fri, Mar 29, 2013 at 6:36 PM, Emmanuel Benisty wrote: > > > > I had to slightly modify the patch since it wouldn't match the changes > > introduced by 7-7-ipc-sem-fine-grained-locking-for-semtimedop.patch, > > hope that was the right th

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 6:36 PM, Emmanuel Benisty wrote: > > I had to slightly modify the patch since it wouldn't match the changes > introduced by 7-7-ipc-sem-fine-grained-locking-for-semtimedop.patch, > hope that was the right thing to do. So, what I tried was: original 7 > patches + the one lin

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Davidlohr Bueso
On Sat, 2013-03-30 at 08:36 +0700, Emmanuel Benisty wrote: > Hi Linus, > > On Sat, Mar 30, 2013 at 6:16 AM, Linus Torvalds > wrote: > > Emmanuel, can you try the attached patch? I think it applies cleanly > > on top of the scalability series too without any changes, but I didn't > > check if the

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Emmanuel Benisty
Hi Linus, On Sat, Mar 30, 2013 at 6:16 AM, Linus Torvalds wrote: > Emmanuel, can you try the attached patch? I think it applies cleanly > on top of the scalability series too without any changes, but I didn't > check if the patches perhaps changed some of the naming or something. I had to slight

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 2:12 PM, Linus Torvalds wrote: > > I dunno. I'm still not sure this is triggerable, but it looks bad. But > both the semaphore case and the msg cases seem to be solvable by > moving the unlock down, and shm seem to have no getref/putref users to > race with, so this (whites

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 1:41 PM, Linus Torvalds wrote: > > The alternative would be to make sure the thing is always locked (and > in a rcu-read-safe region) before putref/getref. The only place (apart > from the initial allocation, which doesn't matter, because nothing can > see if itf that path

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones wrote: > > Now that I have that reverted, I'm not seeing msgrcv traces any more, but > I've started seeing this.. > > general protection fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC > RIP: 0010:[] [] free_msg+0x2b/0x40 > Call Trace: > [] freeque+0xcf/0

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 12:33 PM, Peter Hurley wrote: > On Fri, 2013-03-29 at 11:43 -0700, Linus Torvalds wrote: >> I think I foud at least one bug in the MSG_COPY stuff: it leaks the >> "copy" allocation if >> >> mode == SEARCH_LESSEQUAL >> >> but maybe I'm misreading it. > > Yes, you're misr

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Peter Hurley
On Fri, 2013-03-29 at 12:26 -0700, Linus Torvalds wrote: > On Fri, Mar 29, 2013 at 12:06 PM, Dave Jones wrote: > > > > Here's an oops I just hit.. > > > > BUG: unable to handle kernel NULL pointer dereference at 000f > > IP: [] testmsg.isra.5+0x1a/0x60 > > Btw, looking at the code lea

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Peter Hurley
On Fri, 2013-03-29 at 11:43 -0700, Linus Torvalds wrote: > I think I foud at least one bug in the MSG_COPY stuff: it leaks the > "copy" allocation if > > mode == SEARCH_LESSEQUAL > > but maybe I'm misreading it. Yes, you're misreading it. copy_msg() returns the 'copy' address when copying is

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 12:06 PM, Dave Jones wrote: > > Here's an oops I just hit.. > > BUG: unable to handle kernel NULL pointer dereference at 000f > IP: [] testmsg.isra.5+0x1a/0x60 Btw, looking at the code leading up to this, what the f*ck is wrong with the IPC stuff? It's using t

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 12:06 PM, Dave Jones wrote: > > Btw, something that's really bothering me is just how much bogus > 'follow-on' spew we have lately. I'm not sure what changed, but it > seems to have gotten worse. .. we have many more sanity checks, and they tend to trigger in the case of u

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Dave Jones
On Fri, Mar 29, 2013 at 11:43:25AM -0700, Linus Torvalds wrote: > On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones wrote: > > > > Now that I have that reverted, I'm not seeing msgrcv traces any more, but > > I've started seeing this.. > > > > general protection fault: [#1] PREEMPT SMP DEBUG_

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Dave Jones
On Tue, Mar 26, 2013 at 12:43:09PM -0700, Andrew Morton wrote: > On Tue, 26 Mar 2013 15:28:52 -0400 Dave Jones wrote: > > > On Thu, Mar 21, 2013 at 02:10:58PM -0700, Andrew Morton wrote: > > > > > Whichever way we go, we should get a wiggle on - this has been hanging > > > around for too

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones wrote: > > Now that I have that reverted, I'm not seeing msgrcv traces any more, but > I've started seeing this.. > > general protection fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC Do you have CONFIG_CHECKPOINT_RESTORE enabled? Does it go away if you d

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 11:04 AM, Dave Jones wrote: > > mainline. Your current tree. Ok, that's what I thought you were generally testing, just wanted to verify. The subject kind of implied otherwise.. Linus -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Dave Jones
On Fri, Mar 29, 2013 at 11:00:27AM -0700, Linus Torvalds wrote: > On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones wrote: > > > > Now that I have that reverted, I'm not seeing msgrcv traces any more, but > > I've started seeing this.. > > > > general protection fault: [#1] PREEMPT SMP DEBUG_

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Linus Torvalds
On Fri, Mar 29, 2013 at 9:17 AM, Dave Jones wrote: > > Now that I have that reverted, I'm not seeing msgrcv traces any more, but > I've started seeing this.. > > general protection fault: [#1] PREEMPT SMP DEBUG_PAGEALLOC > > Looks like seg was already kfree'd. Just to clarify: is this you te

Re: ipc,sem: sysv semaphore scalability

2013-03-29 Thread Dave Jones
On Tue, Mar 26, 2013 at 12:43:09PM -0700, Andrew Morton wrote: > On Tue, 26 Mar 2013 15:28:52 -0400 Dave Jones wrote: > > > On Thu, Mar 21, 2013 at 02:10:58PM -0700, Andrew Morton wrote: > > > > > Whichever way we go, we should get a wiggle on - this has been hanging > > > around for too

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Andrew Morton
On Tue, 26 Mar 2013 15:28:52 -0400 Dave Jones wrote: > On Thu, Mar 21, 2013 at 02:10:58PM -0700, Andrew Morton wrote: > > > Whichever way we go, we should get a wiggle on - this has been hanging > > around for too long. Dave, do you have time to determine whether > > reverting 88b9e456b16497

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Dave Jones
On Thu, Mar 21, 2013 at 02:10:58PM -0700, Andrew Morton wrote: > Whichever way we go, we should get a wiggle on - this has been hanging > around for too long. Dave, do you have time to determine whether > reverting 88b9e456b1649722673ff ("ipc: don't allocate a copy larger > than max") fixes t

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Andrew Morton
On Tue, 26 Mar 2013 10:59:27 -0700 Davidlohr Bueso wrote: > On Mon, 2013-03-25 at 20:47 +0700, Emmanuel Benisty wrote: > > On Mon, Mar 25, 2013 at 12:10 AM, Linus Torvalds > > wrote: > > > And you never see this problem without Rik's patches? > > > > No, never. > > > > > Could you bisect > >

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Rik van Riel
On 03/26/2013 02:07 PM, Sasha Levin wrote: On 03/26/2013 01:51 PM, Davidlohr Bueso wrote: On Tue, 2013-03-26 at 13:33 -0400, Sasha Levin wrote: On 03/20/2013 03:55 PM, Rik van Riel wrote: This series makes the sysv semaphore code more scalable, by reducing the time the semaphore lock is held,

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Rik van Riel
On 03/26/2013 01:59 PM, Davidlohr Bueso wrote: From: Davidlohr Bueso Subject: [PATCH] ipc, sem: prevent possible deadlock In semctl_main(), when cmd == GETALL, we're locking sma->sem_perm.lock (through sem_lock_and_putref), yet after the conditional, we lock it again. Unlock sma right after ex

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Sasha Levin
On 03/20/2013 03:55 PM, Rik van Riel wrote: > This series makes the sysv semaphore code more scalable, > by reducing the time the semaphore lock is held, and making > the locking more scalable for semaphore arrays with multiple > semaphores. Hi Rik, Another issue that came up is: [ 96.347341]

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Sasha Levin
On 03/26/2013 01:51 PM, Davidlohr Bueso wrote: > On Tue, 2013-03-26 at 13:33 -0400, Sasha Levin wrote: >> On 03/20/2013 03:55 PM, Rik van Riel wrote: >>> This series makes the sysv semaphore code more scalable, >>> by reducing the time the semaphore lock is held, and making >>> the locking more sca

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Davidlohr Bueso
On Mon, 2013-03-25 at 20:47 +0700, Emmanuel Benisty wrote: > On Mon, Mar 25, 2013 at 12:10 AM, Linus Torvalds > wrote: > > And you never see this problem without Rik's patches? > > No, never. > > > Could you bisect > > *which* patch it starts with? Are the first four ones ok (the moving > > of t

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Paul E. McKenney
On Tue, Mar 26, 2013 at 01:33:07PM -0400, Sasha Levin wrote: > On 03/20/2013 03:55 PM, Rik van Riel wrote: > > This series makes the sysv semaphore code more scalable, > > by reducing the time the semaphore lock is held, and making > > the locking more scalable for semaphore arrays with multiple >

Re: ipc,sem: sysv semaphore scalability

2013-03-26 Thread Davidlohr Bueso
On Tue, 2013-03-26 at 13:33 -0400, Sasha Levin wrote: > On 03/20/2013 03:55 PM, Rik van Riel wrote: > > This series makes the sysv semaphore code more scalable, > > by reducing the time the semaphore lock is held, and making > > the locking more scalable for semaphore arrays with multiple > > semap

Re: ipc,sem: sysv semaphore scalability

2013-03-25 Thread Sasha Levin
On 03/20/2013 03:55 PM, Rik van Riel wrote: > Include lkml in the CC: this time... *sigh* > ---8<--- > > This series makes the sysv semaphore code more scalable, > by reducing the time the semaphore lock is held, and making > the locking more scalable for semaphore arrays with multiple > semaphore

Re: ipc,sem: sysv semaphore scalability

2013-03-25 Thread Emmanuel Benisty
On Mon, Mar 25, 2013 at 10:53 PM, Rik van Riel wrote: > On 03/25/2013 11:20 AM, Emmanuel Benisty wrote: >> >> On Mon, Mar 25, 2013 at 9:03 PM, Rik van Riel wrote: > > With the first four patches only, I got some X server freeze (just > tried once). Could you try bo

Re: ipc,sem: sysv semaphore scalability

2013-03-25 Thread Rik van Riel
On 03/25/2013 11:20 AM, Emmanuel Benisty wrote: On Mon, Mar 25, 2013 at 9:03 PM, Rik van Riel wrote: With the first four patches only, I got some X server freeze (just tried once). Could you try booting with panic=1 so the kernel panics on the first oops? Sorry that should be "oops=panic"

Re: ipc,sem: sysv semaphore scalability

2013-03-25 Thread Emmanuel Benisty
On Mon, Mar 25, 2013 at 9:03 PM, Rik van Riel wrote: >>> With the first four patches only, I got some X server freeze (just >>> tried once). >> >> >> Could you try booting with panic=1 so the kernel panics on the first >> oops? > > > Sorry that should be "oops=panic" > > >> Maybe that way (if we a

Re: ipc,sem: sysv semaphore scalability

2013-03-25 Thread Emmanuel Benisty
On Mon, Mar 25, 2013 at 9:01 PM, Rik van Riel wrote: > On 03/25/2013 09:47 AM, Emmanuel Benisty wrote: >> >> On Mon, Mar 25, 2013 at 12:10 AM, Linus Torvalds >> wrote: >>> >>> And you never see this problem without Rik's patches? >> >> >> No, never. >> >>> Could you bisect >>> *which* patch it st

Re: ipc,sem: sysv semaphore scalability

2013-03-25 Thread Rik van Riel
On 03/25/2013 10:00 AM, Rik van Riel wrote: On 03/25/2013 09:47 AM, Emmanuel Benisty wrote: On Mon, Mar 25, 2013 at 12:10 AM, Linus Torvalds wrote: And you never see this problem without Rik's patches? No, never. Could you bisect *which* patch it starts with? Are the first four ones ok (th

Re: ipc,sem: sysv semaphore scalability

2013-03-25 Thread Rik van Riel
On 03/25/2013 09:47 AM, Emmanuel Benisty wrote: On Mon, Mar 25, 2013 at 12:10 AM, Linus Torvalds wrote: And you never see this problem without Rik's patches? No, never. Could you bisect *which* patch it starts with? Are the first four ones ok (the moving of the locking around, but without t

Re: ipc,sem: sysv semaphore scalability

2013-03-25 Thread Rik van Riel
On 03/25/2013 09:47 AM, Emmanuel Benisty wrote: On Mon, Mar 25, 2013 at 12:10 AM, Linus Torvalds wrote: And you never see this problem without Rik's patches? No, never. Could you bisect *which* patch it starts with? Are the first four ones ok (the moving of the locking around, but without t

Re: ipc,sem: sysv semaphore scalability

2013-03-25 Thread Emmanuel Benisty
On Mon, Mar 25, 2013 at 12:10 AM, Linus Torvalds wrote: > And you never see this problem without Rik's patches? No, never. > Could you bisect > *which* patch it starts with? Are the first four ones ok (the moving > of the locking around, but without the fine-grained ones), for > example? With t

Re: ipc,sem: sysv semaphore scalability

2013-03-24 Thread Linus Torvalds
On Sun, Mar 24, 2013 at 6:46 AM, Emmanuel Benisty wrote: > > Thanks Linus. I hope I got this right, here's the result (3.9-rc4, 7+1 > patches): http://i.imgur.com/BebGZxV.jpg Ok, that's *slightly* more informative, but not much. At least now we see the actual page fault information, and see what

Re: ipc,sem: sysv semaphore scalability

2013-03-24 Thread Emmanuel Benisty
On Sun, Mar 24, 2013 at 2:45 AM, Linus Torvalds wrote: > On Fri, Mar 22, 2013 at 8:19 PM, Emmanuel Benisty wrote: >> >> I could reproduce it but could you please let me know what would be >> the right tools I should use to catch the original oops? >> This is what I got but I doubt it will be help

Re: ipc,sem: sysv semaphore scalability

2013-03-23 Thread Linus Torvalds
On Fri, Mar 22, 2013 at 8:19 PM, Emmanuel Benisty wrote: > > I could reproduce it but could you please let me know what would be > the right tools I should use to catch the original oops? > This is what I got but I doubt it will be helpful: > http://i.imgur.com/Mewi1hC.jpg In this case, I think t

Re: ipc,sem: sysv semaphore scalability

2013-03-22 Thread Emmanuel Benisty
Hi Linus, On Fri, Mar 22, 2013 at 10:37 PM, Linus Torvalds wrote: > On Fri, Mar 22, 2013 at 4:04 AM, Emmanuel Benisty wrote: >> >> I was trying your patchset and my machine died while building a >> package. I could reproduce the bug the (only) two times I tried. >> There's a poor quality picture

Re: ipc,sem: sysv semaphore scalability

2013-03-22 Thread Davidlohr Bueso
On Wed, 2013-03-20 at 15:55 -0400, Rik van Riel wrote: > Include lkml in the CC: this time... *sigh* > ---8<--- > > This series makes the sysv semaphore code more scalable, > by reducing the time the semaphore lock is held, and making > the locking more scalable for semaphore arrays with multiple

Re: ipc,sem: sysv semaphore scalability

2013-03-22 Thread Linus Torvalds
On Fri, Mar 22, 2013 at 4:04 AM, Emmanuel Benisty wrote: > > I was trying your patchset and my machine died while building a > package. I could reproduce the bug the (only) two times I tried. > There's a poor quality picture here: http://i.imgur.com/MuYuyQC.jpg Hmm. The original oops may well hav

Re: ipc,sem: sysv semaphore scalability

2013-03-22 Thread Emmanuel Benisty
Hi Rik, On Thu, Mar 21, 2013 at 2:55 AM, Rik van Riel wrote: > This series makes the sysv semaphore code more scalable, > by reducing the time the semaphore lock is held, and making > the locking more scalable for semaphore arrays with multiple > semaphores. I was trying your patchset and my mac

Re: ipc,sem: sysv semaphore scalability

2013-03-22 Thread Mike Galbraith
On Wed, 2013-03-20 at 15:55 -0400, Rik van Riel wrote: > Include lkml in the CC: this time... *sigh* > ---8<--- > > This series makes the sysv semaphore code more scalable, > by reducing the time the semaphore lock is held, and making > the locking more scalable for semaphore arrays with multiple

Re: ipc,sem: sysv semaphore scalability

2013-03-21 Thread Rik van Riel
On 03/21/2013 09:23 PM, Linus Torvalds wrote: On Thu, Mar 21, 2013 at 6:12 PM, Davidlohr Bueso wrote: ipc lock contention: 100 users: 8,74% (vanilla)3.17% (v3 patchset) 400 users: 21,86% (vanilla)5.23% (v3 patchset) 800 users 84,35% (vanilla)7.39% (v3 patchset) Ok, I'd call

Re: ipc,sem: sysv semaphore scalability

2013-03-21 Thread Rik van Riel
On 03/21/2013 06:01 PM, Andrew Morton wrote: On Thu, 21 Mar 2013 17:50:05 -0400 Peter Hurley wrote: On Thu, 2013-03-21 at 14:10 -0700, Andrew Morton wrote: On Wed, 20 Mar 2013 15:55:30 -0400 Rik van Riel wrote: This series makes the sysv semaphore code more scalable, by reducing the time

Re: ipc,sem: sysv semaphore scalability

2013-03-21 Thread Linus Torvalds
On Thu, Mar 21, 2013 at 6:12 PM, Davidlohr Bueso wrote: > > ipc lock contention: > 100 users: 8,74% (vanilla)3.17% (v3 patchset) > 400 users: 21,86% (vanilla)5.23% (v3 patchset) > 800 users 84,35% (vanilla)7.39% (v3 patchset) Ok, I'd call that pretty much "solved". Sure, it's sti

Re: ipc,sem: sysv semaphore scalability

2013-03-21 Thread Davidlohr Bueso
On Wed, 2013-03-20 at 15:55 -0400, Rik van Riel wrote: > Include lkml in the CC: this time... *sigh* > ---8<--- > > This series makes the sysv semaphore code more scalable, > by reducing the time the semaphore lock is held, and making > the locking more scalable for semaphore arrays with multiple

Re: ipc,sem: sysv semaphore scalability

2013-03-21 Thread Andrew Morton
On Thu, 21 Mar 2013 17:50:05 -0400 Peter Hurley wrote: > On Thu, 2013-03-21 at 14:10 -0700, Andrew Morton wrote: > > On Wed, 20 Mar 2013 15:55:30 -0400 Rik van Riel wrote: > > > > > This series makes the sysv semaphore code more scalable, > > > by reducing the time the semaphore lock is held,

Re: ipc,sem: sysv semaphore scalability

2013-03-21 Thread Peter Hurley
On Thu, 2013-03-21 at 14:10 -0700, Andrew Morton wrote: > On Wed, 20 Mar 2013 15:55:30 -0400 Rik van Riel wrote: > > > This series makes the sysv semaphore code more scalable, > > by reducing the time the semaphore lock is held, and making > > the locking more scalable for semaphore arrays with m

Re: ipc,sem: sysv semaphore scalability

2013-03-21 Thread Peter Hurley
On Thu, 2013-03-21 at 14:10 -0700, Andrew Morton wrote: > On Wed, 20 Mar 2013 15:55:30 -0400 Rik van Riel wrote: > > > This series makes the sysv semaphore code more scalable, > > by reducing the time the semaphore lock is held, and making > > the locking more scalable for semaphore arrays with m

Re: ipc,sem: sysv semaphore scalability

2013-03-21 Thread Andrew Morton
On Wed, 20 Mar 2013 15:55:30 -0400 Rik van Riel wrote: > This series makes the sysv semaphore code more scalable, > by reducing the time the semaphore lock is held, and making > the locking more scalable for semaphore arrays with multiple > semaphores. > > The first four patches were written by

Re: ipc,sem: sysv semaphore scalability

2013-03-20 Thread Davidlohr Bueso
On Wed, 2013-03-20 at 13:49 -0700, Linus Torvalds wrote: > On Wed, Mar 20, 2013 at 12:55 PM, Rik van Riel wrote: > > > > This series makes the sysv semaphore code more scalable, > > by reducing the time the semaphore lock is held, and making > > the locking more scalable for semaphore arrays with

Re: ipc,sem: sysv semaphore scalability

2013-03-20 Thread Linus Torvalds
On Wed, Mar 20, 2013 at 1:49 PM, Linus Torvalds wrote: > > It *would* be lovely to see this run with the actual Swingbench > numbers. The microbenchmark always looked much nicer. Do the > additional multi-semaphore scalability patches on top of Davidlohr's > patches help with the swingbench issue,

Re: ipc,sem: sysv semaphore scalability

2013-03-20 Thread Linus Torvalds
On Wed, Mar 20, 2013 at 12:55 PM, Rik van Riel wrote: > > This series makes the sysv semaphore code more scalable, > by reducing the time the semaphore lock is held, and making > the locking more scalable for semaphore arrays with multiple > semaphores. The series looks sane to me, and I like how

ipc,sem: sysv semaphore scalability

2013-03-20 Thread Rik van Riel
Include lkml in the CC: this time... *sigh* ---8<--- This series makes the sysv semaphore code more scalable, by reducing the time the semaphore lock is held, and making the locking more scalable for semaphore arrays with multiple semaphores. The first four patches were written by Davidlohr Buess