Re: PROBLEM: Concurrency issue in sem_lock

2015-10-10 Thread Manfred Spraul 

Hi,

On 10/09/2015 10:24 AM, Felix Hübner wrote:

Hi all,

I have just reported a concurrency issue in the implementation of
sem_lock, see https://bugzilla.kernel.org/show_bug.cgi?id=105651

[...]
# P0 does spin_lock(>lock); in line 336.

spin_lock(>lock);

[...]

# P2 performs rest of semtimedop, increments complex_count and ends up
in line 1961 and starts to sleep.

return -1;
}
That is the problem: semtimedop() increments complex_count - thus 
sem_wait_array() returns without a spin_unlock_wait() loop - but P0 
already owns spin_lock(>lock).


How do we want to fix it?
- revert my patch (simplify code, but slower for one corner case)
- add the missing sem_wait_array (more complex, but also better for 
complex semops).


what do you think?

(patch untested)

--
Manfred
>From 0ce84d118e2ee7ebc98ad4a8cfd23f04ad45115c Mon Sep 17 00:00:00 2001
From: Manfred Spraul 
Date: Sat, 10 Oct 2015 08:37:22 +0200
Subject: [PATCH] ipc/sem.c: Alternative for fixing Concurrency bug

Two ideas for fixing the bug found by Felix:
- Revert my initial patch.
	Problem: Significant slowdown for application that use large sem
	arrays and complex operations: Every semop() does a loop
	with spin_lock() on all semaphores.

- Add another sem_wait_array() that catches operations that are in
  the middle of sem_lock().

What do you think? Is it worth to optimize for complex ops?

Reported-by: fel...@informatik.uni-bremen.de
Signed-off-by: Manfred Spraul 
---
 ipc/sem.c | 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/ipc/sem.c b/ipc/sem.c
index b471e5a..9a55cfb 100644
--- a/ipc/sem.c
+++ b/ipc/sem.c
@@ -1936,9 +1936,16 @@ SYSCALL_DEFINE4(semtimedop, int, semid, struct sembuf __user *, tsops,
 			list_add_tail(, >pending_const);
 		}
 	} else {
-		if (!sma->complex_count)
+		if (!sma->complex_count) {
 			merge_queues(sma);
 
+			/*
+			 * squeeze out any simple operations that are in the middle
+			 * of sem_lock()
+			 */
+			sem_wait_array(sma);
+		}
+
 		if (alter)
 			list_add_tail(, >pending_alter);
 		else
-- 
2.4.3

Re: PROBLEM: Concurrency issue in sem_lock

2015-10-10 Thread Manfred Spraul 

Hi,

On 10/09/2015 10:24 AM, Felix Hübner wrote:

Hi all,

I have just reported a concurrency issue in the implementation of
sem_lock, see https://bugzilla.kernel.org/show_bug.cgi?id=105651

[...]
# P0 does spin_lock(>lock); in line 336.

spin_lock(>lock);

[...]

# P2 performs rest of semtimedop, increments complex_count and ends up
in line 1961 and starts to sleep.

return -1;
}
That is the problem: semtimedop() increments complex_count - thus 
sem_wait_array() returns without a spin_unlock_wait() loop - but P0 
already owns spin_lock(>lock).


How do we want to fix it?
- revert my patch (simplify code, but slower for one corner case)
- add the missing sem_wait_array (more complex, but also better for 
complex semops).


what do you think?

(patch untested)

--
Manfred
>From 0ce84d118e2ee7ebc98ad4a8cfd23f04ad45115c Mon Sep 17 00:00:00 2001
From: Manfred Spraul 
Date: Sat, 10 Oct 2015 08:37:22 +0200
Subject: [PATCH] ipc/sem.c: Alternative for fixing Concurrency bug

Two ideas for fixing the bug found by Felix:
- Revert my initial patch.
	Problem: Significant slowdown for application that use large sem
	arrays and complex operations: Every semop() does a loop
	with spin_lock() on all semaphores.

- Add another sem_wait_array() that catches operations that are in
  the middle of sem_lock().

What do you think? Is it worth to optimize for complex ops?

Reported-by: fel...@informatik.uni-bremen.de
Signed-off-by: Manfred Spraul 
---
 ipc/sem.c | 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/ipc/sem.c b/ipc/sem.c
index b471e5a..9a55cfb 100644
--- a/ipc/sem.c
+++ b/ipc/sem.c
@@ -1936,9 +1936,16 @@ SYSCALL_DEFINE4(semtimedop, int, semid, struct sembuf __user *, tsops,
 			list_add_tail(, >pending_const);
 		}
 	} else {
-		if (!sma->complex_count)
+		if (!sma->complex_count) {
 			merge_queues(sma);
 
+			/*
+			 * squeeze out any simple operations that are in the middle
+			 * of sem_lock()
+			 */
+			sem_wait_array(sma);
+		}
+
 		if (alter)
 			list_add_tail(, >pending_alter);
 		else
-- 
2.4.3

PROBLEM: Concurrency issue in sem_lock

2015-10-09 Thread Felix Hübner 
Hi all,

I have just reported a concurrency issue in the implementation of
sem_lock, see https://bugzilla.kernel.org/show_bug.cgi?id=105651

Please set me to CC for answers/comments.


[1.] One line summary of the problem:

Concurrency issue in sem_lock

[2.] Full description of the problem/report:

In line 376 (file ipc/sem.c) sem_wait_array is called with pre condition
sma->complex_count!=0 resulting in an immediate return.

The following scenario is possible, allowing P0 and P1 to be in the
critical section (section between sem_lock und sem_unlock) at the same time:

P0 is a process performing up with semtimedop on single semaphore 1.
P1 is a process performing up with semtimedop on single semaphore 1.
P2 is a process performing down with semtimedop on semaphores 1 and 2.

start condition:

sma->complex_count=0

All line references apply to the file ipc/sem.c.

# P0 runs sem_lock up to line 332: if(sma->complex_count==0) evaluates
to true

static inline int sem_lock(struct sem_array *sma, struct sembuf *sops,
  int nsops)
{
struct sem *sem;

if (nsops != 1) {
...
}
sem = sma->sem_base + sops->sem_num;

if (sma->complex_count == 0) {

# P1 runs sem_lock up to line 329.

static inline int sem_lock(struct sem_array *sma, struct sembuf *sops,
  int nsops)
{
struct sem *sem;

if (nsops != 1) {
...
}

sem = sma->sem_base + sops->sem_num;

# P2 runs sem_lock to (including) line 310.

static inline int sem_lock(struct sem_array *sma, struct sembuf *sops,
  int nsops)
{
struct sem *sem;

if (nsops != 1) {
ipc_lock_object(>sem_perm);

sem_wait_array(sma);

# P0 does spin_lock(>lock); in line 336.

spin_lock(>lock);

# P2 performs rest of semtimedop, increments complex_count and ends up
in line 1961 and starts to sleep.

return -1;
}

  ##semtimedop:
locknum = sem_lock(sma, sops, nsops);

...

error = perform_atomic_semop(sma, );
if (error == 0) {
...
}
if (error <= 0)
...

if (nsops == 1) {
...
} else {
if (!sma->complex_count)
merge_queues(sma);

if (alter)
list_add_tail(, >pending_alter);
else
list_add_tail(, >pending_const);

sma->complex_count++;
}

queue.status = -EINTR;
queue.sleeper = current;

sleep_again:
__set_current_state(TASK_INTERRUPTIBLE);
sem_unlock(sma, locknum);
rcu_read_unlock();

if (timeout)
jiffies_left = schedule_timeout(jiffies_left);
else
schedule();

# P1 evaluates if(sem->complex_count==0) in line 331 and ends up in line
361.

if (sma->complex_count == 0) {
...
}

# P0 evaluates if(!spin_is_locked(>sem_perm.lock)) in line 339 and
ends up after line 346.

if (!spin_is_locked(>sem_perm.lock)) { 
ipc_smp_acquire__after_spin_is_unlocked();


# P1 performs ipc_lock_object(>sem_perm); line 362; it then
evaluates if(sma->complex_count==0) and executes the else statement,
calls sem_wait_array in line 376, which returns erroneously.

ipc_lock_object(>sem_perm);

if (sma->complex_count == 0) {
...
} else {
sem_wait_array(sma);

static void sem_wait_array(struct sem_array*sma)
{
int i;
struct sem *sem;

if (sma->complex_count)  {
return;
}   

return -1;
}

# P1 runs the rest of semtimedop and wakes P2 up in do_smart_update
(line 1911), update_queue and unlink_queue respectively. This reduces
complex_count.

locknum = sem_lock(sma, sops, nsops);
...
...
error = perform_atomic_semop(sma, );
if (error == 0) {
if (alter)
do_smart_update(sma, sops, nsops, 1, );

# P0 now evaluates if(sma->complex_count==0) in line 353 and is able to
enter the critical section, while P1 is still in do_smart_update().

if (sma->complex_count == 0) {
/* fast path successful! */
return sops->sem_num;
}   


Thanks to Manfred Spraul for his comments on the problem description.


[3.] Keywords (i.e., modules, networking, kernel):
IPC, sem_lock
[4.] Kernel

PROBLEM: Concurrency issue in sem_lock

2015-10-09 Thread Felix Hübner 
Hi all,

I have just reported a concurrency issue in the implementation of
sem_lock, see https://bugzilla.kernel.org/show_bug.cgi?id=105651

Please set me to CC for answers/comments.


[1.] One line summary of the problem:

Concurrency issue in sem_lock

[2.] Full description of the problem/report:

In line 376 (file ipc/sem.c) sem_wait_array is called with pre condition
sma->complex_count!=0 resulting in an immediate return.

The following scenario is possible, allowing P0 and P1 to be in the
critical section (section between sem_lock und sem_unlock) at the same time:

P0 is a process performing up with semtimedop on single semaphore 1.
P1 is a process performing up with semtimedop on single semaphore 1.
P2 is a process performing down with semtimedop on semaphores 1 and 2.

start condition:

sma->complex_count=0

All line references apply to the file ipc/sem.c.

# P0 runs sem_lock up to line 332: if(sma->complex_count==0) evaluates
to true

static inline int sem_lock(struct sem_array *sma, struct sembuf *sops,
  int nsops)
{
struct sem *sem;

if (nsops != 1) {
...
}
sem = sma->sem_base + sops->sem_num;

if (sma->complex_count == 0) {

# P1 runs sem_lock up to line 329.

static inline int sem_lock(struct sem_array *sma, struct sembuf *sops,
  int nsops)
{
struct sem *sem;

if (nsops != 1) {
...
}

sem = sma->sem_base + sops->sem_num;

# P2 runs sem_lock to (including) line 310.

static inline int sem_lock(struct sem_array *sma, struct sembuf *sops,
  int nsops)
{
struct sem *sem;

if (nsops != 1) {
ipc_lock_object(>sem_perm);

sem_wait_array(sma);

# P0 does spin_lock(>lock); in line 336.

spin_lock(>lock);

# P2 performs rest of semtimedop, increments complex_count and ends up
in line 1961 and starts to sleep.

return -1;
}

  ##semtimedop:
locknum = sem_lock(sma, sops, nsops);

...

error = perform_atomic_semop(sma, );
if (error == 0) {
...
}
if (error <= 0)
...

if (nsops == 1) {
...
} else {
if (!sma->complex_count)
merge_queues(sma);

if (alter)
list_add_tail(, >pending_alter);
else
list_add_tail(, >pending_const);

sma->complex_count++;
}

queue.status = -EINTR;
queue.sleeper = current;

sleep_again:
__set_current_state(TASK_INTERRUPTIBLE);
sem_unlock(sma, locknum);
rcu_read_unlock();

if (timeout)
jiffies_left = schedule_timeout(jiffies_left);
else
schedule();

# P1 evaluates if(sem->complex_count==0) in line 331 and ends up in line
361.

if (sma->complex_count == 0) {
...
}

# P0 evaluates if(!spin_is_locked(>sem_perm.lock)) in line 339 and
ends up after line 346.

if (!spin_is_locked(>sem_perm.lock)) { 
ipc_smp_acquire__after_spin_is_unlocked();


# P1 performs ipc_lock_object(>sem_perm); line 362; it then
evaluates if(sma->complex_count==0) and executes the else statement,
calls sem_wait_array in line 376, which returns erroneously.

ipc_lock_object(>sem_perm);

if (sma->complex_count == 0) {
...
} else {
sem_wait_array(sma);

static void sem_wait_array(struct sem_array*sma)
{
int i;
struct sem *sem;

if (sma->complex_count)  {
return;
}   

return -1;
}

# P1 runs the rest of semtimedop and wakes P2 up in do_smart_update
(line 1911), update_queue and unlink_queue respectively. This reduces
complex_count.

locknum = sem_lock(sma, sops, nsops);
...
...
error = perform_atomic_semop(sma, );
if (error == 0) {
if (alter)
do_smart_update(sma, sops, nsops, 1, );

# P0 now evaluates if(sma->complex_count==0) in line 353 and is able to
enter the critical section, while P1 is still in do_smart_update().

if (sma->complex_count == 0) {
/* fast path successful! */
return sops->sem_num;
}   


Thanks to Manfred Spraul for his comments on the problem description.


[3.] Keywords (i.e., modules, networking, kernel):
IPC, sem_lock
[4.] Kernel

PROBLEM: Concurrency issue in sem_lock

2015-10-09 Thread Felix Hübner 
Hi all,

I have just reported a concurrency issue in the implementation of
sem_lock, see https://bugzilla.kernel.org/show_bug.cgi?id=105651

Please set me to CC for answers/comments.


[1.] One line summary of the problem:

Concurrency issue in sem_lock

[2.] Full description of the problem/report:

In line 376 (file ipc/sem.c) sem_wait_array is called with pre condition
sma->complex_count!=0 resulting in an immediate return.

The following scenario is possible, allowing P0 and P1 to be in the
critical section (section between sem_lock und sem_unlock) at the same time:

P0 is a process performing up with semtimedop on single semaphore 1.
P1 is a process performing up with semtimedop on single semaphore 1.
P2 is a process performing down with semtimedop on semaphores 1 and 2.

start condition:

sma->complex_count=0

All line references apply to the file ipc/sem.c.

# P0 runs sem_lock up to line 332: if(sma->complex_count==0) evaluates
to true

static inline int sem_lock(struct sem_array *sma, struct sembuf *sops,
  int nsops)
{
struct sem *sem;

if (nsops != 1) {
...
}
sem = sma->sem_base + sops->sem_num;

if (sma->complex_count == 0) {

# P1 runs sem_lock up to line 329.

static inline int sem_lock(struct sem_array *sma, struct sembuf *sops,
  int nsops)
{
struct sem *sem;

if (nsops != 1) {
...
}

sem = sma->sem_base + sops->sem_num;

# P2 runs sem_lock to (including) line 310.

static inline int sem_lock(struct sem_array *sma, struct sembuf *sops,
  int nsops)
{
struct sem *sem;

if (nsops != 1) {
ipc_lock_object(>sem_perm);

sem_wait_array(sma);

# P0 does spin_lock(>lock); in line 336.

spin_lock(>lock);

# P2 performs rest of semtimedop, increments complex_count and ends up
in line 1961 and starts to sleep.

return -1;
}

  ##semtimedop:
locknum = sem_lock(sma, sops, nsops);

...

error = perform_atomic_semop(sma, );
if (error == 0) {
...
}
if (error <= 0)
...

if (nsops == 1) {
...
} else {
if (!sma->complex_count)
merge_queues(sma);

if (alter)
list_add_tail(, >pending_alter);
else
list_add_tail(, >pending_const);

sma->complex_count++;
}

queue.status = -EINTR;
queue.sleeper = current;

sleep_again:
__set_current_state(TASK_INTERRUPTIBLE);
sem_unlock(sma, locknum);
rcu_read_unlock();

if (timeout)
jiffies_left = schedule_timeout(jiffies_left);
else
schedule();

# P1 evaluates if(sem->complex_count==0) in line 331 and ends up in line
361.

if (sma->complex_count == 0) {
...
}

# P0 evaluates if(!spin_is_locked(>sem_perm.lock)) in line 339 and
ends up after line 346.

if (!spin_is_locked(>sem_perm.lock)) { 
ipc_smp_acquire__after_spin_is_unlocked();


# P1 performs ipc_lock_object(>sem_perm); line 362; it then
evaluates if(sma->complex_count==0) and executes the else statement,
calls sem_wait_array in line 376, which returns erroneously.

ipc_lock_object(>sem_perm);

if (sma->complex_count == 0) {
...
} else {
sem_wait_array(sma);

static void sem_wait_array(struct sem_array*sma)
{
int i;
struct sem *sem;

if (sma->complex_count)  {
return;
}   

return -1;
}

# P1 runs the rest of semtimedop and wakes P2 up in do_smart_update
(line 1911), update_queue and unlink_queue respectively. This reduces
complex_count.

locknum = sem_lock(sma, sops, nsops);
...
...
error = perform_atomic_semop(sma, );
if (error == 0) {
if (alter)
do_smart_update(sma, sops, nsops, 1, );

# P0 now evaluates if(sma->complex_count==0) in line 353 and is able to
enter the critical section, while P1 is still in do_smart_update().

if (sma->complex_count == 0) {
/* fast path successful! */
return sops->sem_num;
}   


Thanks to Manfred Spraul for his comments on the problem description.


[3.] Keywords (i.e., modules, networking, kernel):
IPC, sem_lock
[4.] Kernel

PROBLEM: Concurrency issue in sem_lock

2015-10-09 Thread Felix Hübner 
Hi all,

I have just reported a concurrency issue in the implementation of
sem_lock, see https://bugzilla.kernel.org/show_bug.cgi?id=105651

Please set me to CC for answers/comments.


[1.] One line summary of the problem:

Concurrency issue in sem_lock

[2.] Full description of the problem/report:

In line 376 (file ipc/sem.c) sem_wait_array is called with pre condition
sma->complex_count!=0 resulting in an immediate return.

The following scenario is possible, allowing P0 and P1 to be in the
critical section (section between sem_lock und sem_unlock) at the same time:

P0 is a process performing up with semtimedop on single semaphore 1.
P1 is a process performing up with semtimedop on single semaphore 1.
P2 is a process performing down with semtimedop on semaphores 1 and 2.

start condition:

sma->complex_count=0

All line references apply to the file ipc/sem.c.

# P0 runs sem_lock up to line 332: if(sma->complex_count==0) evaluates
to true

static inline int sem_lock(struct sem_array *sma, struct sembuf *sops,
  int nsops)
{
struct sem *sem;

if (nsops != 1) {
...
}
sem = sma->sem_base + sops->sem_num;

if (sma->complex_count == 0) {

# P1 runs sem_lock up to line 329.

static inline int sem_lock(struct sem_array *sma, struct sembuf *sops,
  int nsops)
{
struct sem *sem;

if (nsops != 1) {
...
}

sem = sma->sem_base + sops->sem_num;

# P2 runs sem_lock to (including) line 310.

static inline int sem_lock(struct sem_array *sma, struct sembuf *sops,
  int nsops)
{
struct sem *sem;

if (nsops != 1) {
ipc_lock_object(>sem_perm);

sem_wait_array(sma);

# P0 does spin_lock(>lock); in line 336.

spin_lock(>lock);

# P2 performs rest of semtimedop, increments complex_count and ends up
in line 1961 and starts to sleep.

return -1;
}

  ##semtimedop:
locknum = sem_lock(sma, sops, nsops);

...

error = perform_atomic_semop(sma, );
if (error == 0) {
...
}
if (error <= 0)
...

if (nsops == 1) {
...
} else {
if (!sma->complex_count)
merge_queues(sma);

if (alter)
list_add_tail(, >pending_alter);
else
list_add_tail(, >pending_const);

sma->complex_count++;
}

queue.status = -EINTR;
queue.sleeper = current;

sleep_again:
__set_current_state(TASK_INTERRUPTIBLE);
sem_unlock(sma, locknum);
rcu_read_unlock();

if (timeout)
jiffies_left = schedule_timeout(jiffies_left);
else
schedule();

# P1 evaluates if(sem->complex_count==0) in line 331 and ends up in line
361.

if (sma->complex_count == 0) {
...
}

# P0 evaluates if(!spin_is_locked(>sem_perm.lock)) in line 339 and
ends up after line 346.

if (!spin_is_locked(>sem_perm.lock)) { 
ipc_smp_acquire__after_spin_is_unlocked();


# P1 performs ipc_lock_object(>sem_perm); line 362; it then
evaluates if(sma->complex_count==0) and executes the else statement,
calls sem_wait_array in line 376, which returns erroneously.

ipc_lock_object(>sem_perm);

if (sma->complex_count == 0) {
...
} else {
sem_wait_array(sma);

static void sem_wait_array(struct sem_array*sma)
{
int i;
struct sem *sem;

if (sma->complex_count)  {
return;
}   

return -1;
}

# P1 runs the rest of semtimedop and wakes P2 up in do_smart_update
(line 1911), update_queue and unlink_queue respectively. This reduces
complex_count.

locknum = sem_lock(sma, sops, nsops);
...
...
error = perform_atomic_semop(sma, );
if (error == 0) {
if (alter)
do_smart_update(sma, sops, nsops, 1, );

# P0 now evaluates if(sma->complex_count==0) in line 353 and is able to
enter the critical section, while P1 is still in do_smart_update().

if (sma->complex_count == 0) {
/* fast path successful! */
return sops->sem_num;
}   


Thanks to Manfred Spraul for his comments on the problem description.


[3.] Keywords (i.e., modules, networking, kernel):
IPC, sem_lock
[4.] Kernel