Thanks you David -- excellent explanation. Frank
On Dec 31, 2009, at 5:51 PM, David Holland wrote: > On Thu, Dec 31, 2009 at 01:52:48PM -0600, Frank Zerangue wrote: >> Help request -- Mutex(9) indicates that mutex replaces the spl(9) system. > > Here are some general (non-NetBSD-specific) answers based on > underlying principles that will hopefully explain the situation better. > >> (1) When writing an interrupt handler, should the handler acquire a >> spin mutex before modifying some IO that may be accessed also by a >> LWP? > > Yes. In general, in a multiprocessor kernel, the interrupt handler > will not necessarily run on the same CPU that has been posting > requests to the device... and in general more than one CPU may be > doing that. It is prohibitively expensive (as well as generally > undesirable) to disable interrupts for all CPUs at once. Therefore, > disabling interrupts only affects the current CPU, so in order to keep > everything from being tied in knots, you need one or more locks. And > because you can't sleep in an interrupt handler, these must be > spinlocks. > > Spinlocks that are used from interrupt handlers must themselves also > disable interrupts. Otherwise, if a thread holding the spinlock is > interrupted by an interrupt handler that tries to acquire the same > spinlock, you get a deadlock. > > For this reason, in essentially all multiprocessor systems, the > spinlocks that you use for mutual exclusion in interrupt code disable > and re-enable interrupts for you. In NetBSD each mutex can have an > interrupt level associated with it; if that interrupt level is not > IPL_NONE, acquiring the mutex raises the current interrupt level and > releasing the mutex lowers the current interrupt level. > > And in turn, in such systems one generally never sees or uses the > splfoo() functions except in code that hasn't yet been > multiprocessorized. > > Currently in NetBSD the interrupt level is only lowered to zero and > only when all spinlocks have been released, instead of any time the > necessary interrupt level drops. This is to avoid complexity when > spinlocks are not released in order, and it mostly makes no practical > difference. (It isn't a good idea to embed e.g. a small block using an > IPL_HIGH mutex inside a large block using a lower-interrupt-level > mutex. But then again, it also isn't a good idea to have a large block > disabling interrupts or using a spinlock anyway.) > > NetBSD also has "soft interrupts" that have more process context than > ordinary interrupt handlers; instead of borrowing the context of > whatever's running when the interrupt arrives, they run on dedicated > kernel threads; this means they can sleep to acquire mutexes. AIUI, > the intended design is that most hard interrupts will do as little > work as possible and trigger a soft interrupt to do the rest; this > reduces the number of mutexes used from real interrupt handlers and > reduces the overall amount of spinning. I'm not entirely up on the > exact details at the moment and hopefully someone else will clarify if > there are questions. > >> (2) What happens when the interrupt handler cannot acquire the >> mutex? Will the LWP that holds it ever be able to run again? > > Define "cannot acquire". If the mutex is held by some thread on > another processor, the interrupt handler will spin until it's > released. If it's never released, the interrupt handler will spin > forever, but that's not supposed to happen. > > If the mutex is held by some thread on the same processor, then you're > deadlocked. This is why you have to disable interrupts before > attempting to get a spinlock that's used from an interrupt handler. > (Note: the exact details depend somewhat on the implementation. But > you don't want to be writing code that pushes the boundaries.) > >> (3) Will a LWP that holds a spin mutex be pre-empted by the scheduler? > > In general, that depends on the interrupt level associated with the > spin mutex. > > -- > David A. Holland > [email protected]
