[Linux-streams] Re: lis_spin_lock_irq and lis_spin_lock_irqsave

[Dave Grothe]

If you need to exclude your interrupt routine's execution with execution of your service routine, or a section thereof, you need to use the "irq" form of the spin locks everywhere.

If you only need to exclude multiple instances of service routines, say from simultaneously accessing some global resource, use the "non-irq" versions of the spin locks.

LiS makes its spin locks nestable.  Thus, the same thread can do two or more "lock" operations followed by the same number of "unlock" operations without any blocking.  If a service routine "locks" a spin lock, without irq protection, and then an interrupt occurs there can be two possible outcomes, neither desireable:

1. If the kernel runs the ISR on the same stack as the service routine then the ISR will gate through the spin lock and access critical data "simultaneously" with the service procedure.
2. If the kernel runs the ISR on a different stack the ISR will spin on the lock.  If the service routine's thread can still run this will all sort itself out.  If not, say if the kernel temporarily halts execution of the interrupted thread, there will be a deadlock.

Using the "irq" form of the lock in the service routine prevents this.

In Solaris they run ISRs on the interrupted stack in nested fashion until the ISR attempts to acquire a mutex AND would have to wait (because it is already owned).  Then they switch the ISR execution to a separate stack with its own LWP thread structure and execute the "waiting" code in that thread.  The interrupted thread is then allowed to run again.

This means that everything in Solaris is just an ordinary spin lock with no need to disable interrupts.  If the Linux kernel ran ISRs on their own threads then there would also be no need for the "irq" form of the spin locks.  But the Linux kernel does not work that way.

This means that you probably want to use the locks to exclude fairly fine grained sections of code in Linux since the service procedure needs to hold an "irqsave" lock to exclude ISR execution.

On the other hand, the service routine only needs to use the "irq" form of spin locks to protect structures that are shared with the ISRs.  Structures shared with other service routines can be protected using "non-irq" forms of the spin locks.

Hope this helps.

-- Dave

Matthew Gierlach wrote:

Hi Dave:

        I think I need some further clarification concerning the use
        of these LiS spin lock mechanisms. I do hope you can help.

        From what I read in the LiS DKI, I have the following understanding
        (whether it be correct or incorrect) about the LiS spin lock
        mechanisms. Please correct me where I'm wrong: Only use
        lis_spin_lock_irqsave/unlock_irqrestore in the STREAMs
        driver ISR. lis_spin_lock_irq/unlock_irq should not be
        used in ISRs.

        Both lis_spin_lock_irqsave/unlock_irqrestore and
        lis_spin_lock_irq/unlock_irq can be used in STREAMs
        routines to serialize the modification of critical data
        elements between the STREAMs routines and between the
        STREAMs routines and the ISR. What's the rationale for
        using one pair over the other in the STREAMs service
        routines? Both types disable interrupts - we're not
        accustomed to that being a common practice in service
        routines. One typically only disables interrupts in ISRs.

        We're running on Linux SMP, and we need the mutual exclusion
        serialization of CPUs in the ISRs via elevated, or masked,
        interrupts, but we're not accustomed to having to mask interrupts
        in the STREAMs service routines in order to serialize service
        routines with the ISRs; that's what the spin lock is supposed
        to do - let the routine owning the lock run while delaying the
        routine that wants the lock until it's free - correct?

        I ran an experiment where lis_spin_lock(&common_driver_lock) was
        used in all the STREAMs service routines and
        lis_spin_lock_irqsave/unlock_irqrestore was used in the ISR. The
        effect was nearly immediate frozen system once the driver was
        invoked and interrupts occured. Utilizing lis_spin_lock_irq/unlock_irq
        in the STREAMs service routines prevents the lock-up. Are both
        observed behaviors to be expected?

        The paradigm we're used to is Solaris mutex locks for both UP and
        SMP systems. In Solaris, we use

                mutex_enter(&common_driver_lock)

        in both our ISRs and our STREAMs routines. We would like further
        guidance and advice on the use of the lis_spin_lock mechanisms
        beyond that provided in the DKI documentation. We need to be
        very clear about the proper mechanisms and their usage for SMP
        systems. Is there a Linux STREAMs HW driver for SMP systems
        example we could review that clarifies the use of the various
        mechanisms?

        Thanks, Matt Gierlach