Re: Are SysV semaphores thread-safe on CURRENT?

[Brian Smith]

On Mon, 18 Nov 2002 22:05:34 -0800, Terry Lambert wrote:

Use mmap of a backing-store file, and then use file locking to
do record locking in the shared memory segment.

Ok, I did this, and it actually works considerably better than
the SysV shared memory.  However flock() has the same problem
as the SysV semaphores, where they block the entire process,
allowing the same deadlock situation to occur.  Has this flock()
behavior changed in CURRENT?  

It seems like this behavior is much more likely to change than
the SysV code.

Thanks!

Brian Smith


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message

Re: Are SysV semaphores thread-safe on CURRENT?

[Daniel Eischen]

On Sat, 30 Nov 2002, Brian Smith wrote:

 On Mon, 18 Nov 2002 22:05:34 -0800, Terry Lambert wrote:
 
 Use mmap of a backing-store file, and then use file locking to
 do record locking in the shared memory segment.
 
 Ok, I did this, and it actually works considerably better than
 the SysV shared memory.  However flock() has the same problem
 as the SysV semaphores, where they block the entire process,
 allowing the same deadlock situation to occur.  Has this flock()
 behavior changed in CURRENT?  

No, libc_r doesn't properly handle flock.  Usually, all syscalls
that take file descriptors as arguments honor the non-blocking
mode of the file if set.  I guess flock(2) doesn't and has its
own option to the operation argument (LOCK_NB).

I hacked libc_r to periodically check (every 100msecs) the
flock.  See if this fixes things:

-- 
Dan Eischen

Index: lib/libc_r/uthread/uthread_flock.c
===
RCS file: /opt/d/CVS/src/lib/libc_r/uthread/uthread_flock.c,v
retrieving revision 1.10
diff -u -r1.10 uthread_flock.c
--- lib/libc_r/uthread/uthread_flock.c  10 Apr 2001 04:19:20 - 1.10
+++ lib/libc_r/uthread/uthread_flock.c  30 Nov 2002 16:23:59 -
@@ -32,6 +32,7 @@
  * $FreeBSD: src/lib/libc_r/uthread/uthread_flock.c,v 1.10 2001/04/10 04:19:20 
deischen Exp $
  */
 #include sys/file.h
+#include errno.h
 #include pthread.h
 #include pthread_private.h
 
@@ -40,10 +41,40 @@
 int
 _flock(int fd, int operation)
 {
-   int ret;
+   struct pthread *curthread;
+   struct timespec ts;
+   int ret;
 
if ((ret = _FD_LOCK(fd, FD_RDWR, NULL)) == 0) {
-   ret = __sys_flock(fd, operation);
+   if ((operation  LOCK_NB) != 0) {
+   ret = __sys_flock(fd, operation);
+   }
+   else {
+   curthread = _get_curthread();
+   ts.tv_sec = 0;
+   ts.tv_nsec = 1; /* 100msecs */
+   while (((ret = __sys_flock(fd, operation | LOCK_NB)) == 0)
+   || (errno == EWOULDBLOCK)) {
+   curthread-data.fd.fd = fd;
+   _thread_kern_set_timeout(ts);
+
+   /* Reset the interrupted operation flag: */
+   curthread-interrupted = 0;
+
+   _thread_kern_sched_state(PS_SLEEP_WAIT,
+   __FILE__, __LINE__);
+
+   /*
+* Check if the operation was interrupted
+* by a signal
+*/
+   if (curthread-interrupted) {
+   errno = EINTR;
+   ret = -1;
+   break;
+   }
+   }
+   }
_FD_UNLOCK(fd, FD_RDWR);
}
return (ret);



To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message

Re: Are SysV semaphores thread-safe on CURRENT?

[Daniel Eischen]

On Sat, 30 Nov 2002, Daniel Eischen wrote:
 On Sat, 30 Nov 2002, Brian Smith wrote:
 
  On Mon, 18 Nov 2002 22:05:34 -0800, Terry Lambert wrote:
  
  Use mmap of a backing-store file, and then use file locking to
  do record locking in the shared memory segment.
  
  Ok, I did this, and it actually works considerably better than
  the SysV shared memory.  However flock() has the same problem
  as the SysV semaphores, where they block the entire process,
  allowing the same deadlock situation to occur.  Has this flock()
  behavior changed in CURRENT?  
 
 No, libc_r doesn't properly handle flock.  Usually, all syscalls
 that take file descriptors as arguments honor the non-blocking
 mode of the file if set.  I guess flock(2) doesn't and has its
 own option to the operation argument (LOCK_NB).
 
 I hacked libc_r to periodically check (every 100msecs) the
 flock.  See if this fixes things:

Oops, I had the wrong operators.  See below.

 -- 
 Dan Eischen
 
 Index: lib/libc_r/uthread/uthread_flock.c
 ===
 RCS file: /opt/d/CVS/src/lib/libc_r/uthread/uthread_flock.c,v
 retrieving revision 1.10
 diff -u -r1.10 uthread_flock.c
 --- lib/libc_r/uthread/uthread_flock.c10 Apr 2001 04:19:20 - 1.10
 +++ lib/libc_r/uthread/uthread_flock.c30 Nov 2002 16:23:59 -
 @@ -32,6 +32,7 @@
   * $FreeBSD: src/lib/libc_r/uthread/uthread_flock.c,v 1.10 2001/04/10 04:19:20 
deischen Exp $
   */
  #include sys/file.h
 +#include errno.h
  #include pthread.h
  #include pthread_private.h
  
 @@ -40,10 +41,40 @@
  int
  _flock(int fd, int operation)
  {
 - int ret;
 + struct pthread *curthread;
 + struct timespec ts;
 + int ret;
  
   if ((ret = _FD_LOCK(fd, FD_RDWR, NULL)) == 0) {
 - ret = __sys_flock(fd, operation);
 + if ((operation  LOCK_NB) != 0) {
 + ret = __sys_flock(fd, operation);
 + }
 + else {
 + curthread = _get_curthread();
 + ts.tv_sec = 0;
 + ts.tv_nsec = 1; /* 100msecs */
 + while (((ret = __sys_flock(fd, operation | LOCK_NB)) == 0)
 + || (errno == EWOULDBLOCK)) {

The above two lines should be:

while (((ret = __sys_flock(fd, operation | LOCK_NB)) != 0)
 (errno == EWOULDBLOCK)) {

 + curthread-data.fd.fd = fd;
 + _thread_kern_set_timeout(ts);
 +
 + /* Reset the interrupted operation flag: */
 + curthread-interrupted = 0;
 +
 + _thread_kern_sched_state(PS_SLEEP_WAIT,
 + __FILE__, __LINE__);
 +
 + /*
 +  * Check if the operation was interrupted
 +  * by a signal
 +  */
 + if (curthread-interrupted) {
 + errno = EINTR;
 + ret = -1;
 + break;
 + }
 + }
 + }
   _FD_UNLOCK(fd, FD_RDWR);
   }
   return (ret);
 
 
 
 To Unsubscribe: send mail to [EMAIL PROTECTED]
 with unsubscribe freebsd-current in the body of the message
 


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message

Re: Are SysV semaphores thread-safe on CURRENT?

[Terry Lambert]

Brian Smith wrote:
 On Mon, 18 Nov 2002 22:05:34 -0800, Terry Lambert wrote:
 Use mmap of a backing-store file, and then use file locking to
 do record locking in the shared memory segment.
 
 Ok, I did this, and it actually works considerably better than
 the SysV shared memory.  However flock() has the same problem
 as the SysV semaphores, where they block the entire process,
 allowing the same deadlock situation to occur.  Has this flock()
 behavior changed in CURRENT?
 
 It seems like this behavior is much more likely to change than
 the SysV code.

Do you mean flock(), or do you mean fcntl(fs, F_SETLKW, ...)?

If you are using range locks, then you mean fcntl().

That's unfortunate: there's an easy way to convert blocking
file locks into non-blocking, plus a context switch.  I
thought th threads library already did this for you in the
fcntl() wrapper, in /usr/src/lib/libc_r/uthread/uthread_fcntl.c,
but apparently it doesn't.  8-(.


The easy way to do this is to convert the blocking request into
a non-blocking request, with a retry; e.g., where you have a
call to:

err = fcntl( fd, F_SETLKW, flock);

Replace it with:

while( ( err = fcntl( fs, F_SETLK, flock)) == -1  errno == EAGAIN) {
sleep( 1);  /* use nanosleep(), if 1 second is too big */
}

This will cause the processor to be yielded to other threads for
as long as the lock can't be acquired, an acquisititon will be
retried until it succeeds (effectively, blocking only that thread
in sleep()).  The difference between F_SETLKW and F_SETLK is why
I suggested the approach in the first place (FWIW).

The cost of doing this is that blocking requests will not be
serviced in FIFO order, as they would if F_SETLKW were being
used.  This may get expensive if you have a highly contended
resource, because you are effectively implementing a low cost
polling to obtain the lock.  The answer to this is that you
are not supposed to use semaphores for highly contended resources,
or if you do, use a spin-lock before you use the semaphore, so
you can fail early at reduced expense.

Probably making the above code into an line function and/or
actually modifiying the _fcntl() implementation in the threads
library is the way to go.


Worse comes to worse, I can give you a kernel patch so that an
fcntl() to assert a blocking lock on a non-blocking fd returns
the EWOULDBLOCK error, with a patch against _fcntl() similar to
the code in _read().

I didn't do that this time, because I don't know how much code
really depends on a lock assert on a non-blocking fd blocking
anyway, and no matter how you slice it, it's still going to
have the same non-FIFO ordering, unless I implemented a FIFO
ordered request queue, as well (it'd have to, to be correct).

-- Terry

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message

Re: Are SysV semaphores thread-safe on CURRENT?

[Terry Lambert]

Daniel Eischen wrote:
  No, libc_r doesn't properly handle flock.  Usually, all syscalls
  that take file descriptors as arguments honor the non-blocking
  mode of the file if set.  I guess flock(2) doesn't and has its
  own option to the operation argument (LOCK_NB).
 
  I hacked libc_r to periodically check (every 100msecs) the
  flock.  See if this fixes things:

Same thing I suggested, only I think he was really using fcntl(),
not flock()?

My patch wasn't integral to the library (it was more of a hack),
and my default time was 1S, not 100uS.

Same non-FIFO request ordering, too.  8-(.

I guess the real question is what is an fcntl()/flock() supposed
to do on a blocking call against a non-blocking fd?  I could not
tell, so I punted.

-- Terry

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message

Are SysV semaphores thread-safe on CURRENT?

[Brian Smith]

I've been porting an application from Linux to FreeBSD using STABLE, and it appears 
from looking at the 
semaphore code that the SysV semaphores are not thread safe on STABLE.  I don't have 
CURRENT source 
code to look at currently.  Does anyone know if they are thread-safe in CURRENT?

Thanks!

Brian Smith



To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message

Re: Are SysV semaphores thread-safe on CURRENT?

[Daniel Eischen]

On Mon, 18 Nov 2002, Brian Smith wrote:

 I've been porting an application from Linux to FreeBSD using STABLE, and it appears 
from looking at the 
 semaphore code that the SysV semaphores are not thread safe on STABLE.  I don't have 
CURRENT source 
 code to look at currently.  Does anyone know if they are thread-safe in CURRENT?

[ I assume you mean semop, semctl, not sema_* or sem_* ]

Sure SysV semaphores are thread-safe.  When a thread blocks on
one, the entire process blocks (no threads run).  You won't
get any safer than that ;-)

-- 
Dan Eischen


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message

Re: Are SysV semaphores thread-safe on CURRENT?

[Brian Smith]

On Mon, 18 Nov 2002 21:33:38 -0500 (EST), Daniel Eischen wrote:

[ I assume you mean semop, semctl, not sema_* or sem_* ]

Yes ... semop() specifically is causing the problems...

Sure SysV semaphores are thread-safe.  When a thread blocks on
one, the entire process blocks (no threads run).  You won't
get any safer than that ;-)

Yikes that isn't good.  Is that only in STABLE?  or does CURRENT
do that as well?  I guess I'll have to protect the semop() call 
with a pthread mutex to prevent two threads locking a single 
semaphore by the same process (creating a deadlock situation).

Is this the recommended method of preventing these problems?

(the SysV semaphore is protecting shared memory accessed by
multiple processes).

Thanks for the info... it explains alot!

Brian Smith




To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message

Re: Are SysV semaphores thread-safe on CURRENT?

[Garrett Wollman]

On Mon, 18 Nov 2002 22:31:28 -0600, Brian Smith [EMAIL PROTECTED] said:

 Is this the recommended method of preventing these problems?

The recommended method of preventing these problems generally is to
use POSIX semaphores (or other POSIX synchronization mechanisms
appropriate to threaded programs.  However, the code to implement
process-shared POSIX semaphores is still experimental.

The problem with System V semaphores is that there is no way to poll
their status while still polling the process's file descriptors.  This
could be fixed by introducing a new kqueue filter, but the semantics
of System V semaphores are difficult to emulate.

Depending on the semantics you require, a semaphore may be implemented
using a pipe or fifo, or by using file or record locking.

-GAWollman


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message

Re: Are SysV semaphores thread-safe on CURRENT?

[Terry Lambert]

Brian Smith wrote:
 Sure SysV semaphores are thread-safe.  When a thread blocks on
 one, the entire process blocks (no threads run).  You won't
 get any safer than that ;-)
 
 Yikes that isn't good.  Is that only in STABLE?  or does CURRENT
 do that as well?  I guess I'll have to protect the semop() call
 with a pthread mutex to prevent two threads locking a single
 semaphore by the same process (creating a deadlock situation).
 
 Is this the recommended method of preventing these problems?

Yeah: don't make blocking system calls for which there are no
asynchronous equivalents.  Use the POSIX interfaces for use
by pthreads, instead.


 (the SysV semaphore is protecting shared memory accessed by
 multiple processes).
 
 Thanks for the info... it explains alot!

Use mmap of a backing-store file, and then use file locking to
do record locking in the shared memory segment.

-- Terry

To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message