On Thu, Dec 31, 2009 at 01:52:48PM -0600, Frank Zerangue wrote: > Help request -- Mutex(9) indicates that mutex replaces the spl(9) system.
Here are some general (non-NetBSD-specific) answers based on underlying principles that will hopefully explain the situation better. > (1) When writing an interrupt handler, should the handler acquire a > spin mutex before modifying some IO that may be accessed also by a > LWP? Yes. In general, in a multiprocessor kernel, the interrupt handler will not necessarily run on the same CPU that has been posting requests to the device... and in general more than one CPU may be doing that. It is prohibitively expensive (as well as generally undesirable) to disable interrupts for all CPUs at once. Therefore, disabling interrupts only affects the current CPU, so in order to keep everything from being tied in knots, you need one or more locks. And because you can't sleep in an interrupt handler, these must be spinlocks. Spinlocks that are used from interrupt handlers must themselves also disable interrupts. Otherwise, if a thread holding the spinlock is interrupted by an interrupt handler that tries to acquire the same spinlock, you get a deadlock. For this reason, in essentially all multiprocessor systems, the spinlocks that you use for mutual exclusion in interrupt code disable and re-enable interrupts for you. In NetBSD each mutex can have an interrupt level associated with it; if that interrupt level is not IPL_NONE, acquiring the mutex raises the current interrupt level and releasing the mutex lowers the current interrupt level. And in turn, in such systems one generally never sees or uses the splfoo() functions except in code that hasn't yet been multiprocessorized. Currently in NetBSD the interrupt level is only lowered to zero and only when all spinlocks have been released, instead of any time the necessary interrupt level drops. This is to avoid complexity when spinlocks are not released in order, and it mostly makes no practical difference. (It isn't a good idea to embed e.g. a small block using an IPL_HIGH mutex inside a large block using a lower-interrupt-level mutex. But then again, it also isn't a good idea to have a large block disabling interrupts or using a spinlock anyway.) NetBSD also has "soft interrupts" that have more process context than ordinary interrupt handlers; instead of borrowing the context of whatever's running when the interrupt arrives, they run on dedicated kernel threads; this means they can sleep to acquire mutexes. AIUI, the intended design is that most hard interrupts will do as little work as possible and trigger a soft interrupt to do the rest; this reduces the number of mutexes used from real interrupt handlers and reduces the overall amount of spinning. I'm not entirely up on the exact details at the moment and hopefully someone else will clarify if there are questions. > (2) What happens when the interrupt handler cannot acquire the > mutex? Will the LWP that holds it ever be able to run again? Define "cannot acquire". If the mutex is held by some thread on another processor, the interrupt handler will spin until it's released. If it's never released, the interrupt handler will spin forever, but that's not supposed to happen. If the mutex is held by some thread on the same processor, then you're deadlocked. This is why you have to disable interrupts before attempting to get a spinlock that's used from an interrupt handler. (Note: the exact details depend somewhat on the implementation. But you don't want to be writing code that pushes the boundaries.) > (3) Will a LWP that holds a spin mutex be pre-empted by the scheduler? In general, that depends on the interrupt level associated with the spin mutex. -- David A. Holland [email protected]
