So now that we have a mechanism to install interrupt handlers, it is
time to look at our mutexes again. To prevent lock ordering problems
with the kernel lock, we need to make sure we block all interrupts
that can grab the kernel lock. The simplest way to achieve this is to
make sure mutexes always raise the ipl to the highest level that has
interrupts that grab the kernel lock. On amd64 that currently is
IPL_AUDIO, but this will soon be IPL_VM or IPL_TTY. In theory this
increases interrupt latency. In practice it seems that this increase
is just noise compared to the latency we get due to kernel lock
contention.
I'd like to keep mutexes as lightweight as possible, and preferably
not add any code to the mtx_enter() and mtx_leave(). So the idea is
to adjust the ipl for the mutex when it gets initialized. The diff
below implements this strategy for amd64.
An earlier version of this diff has been tested by a couple of
devlopers already. IIRC mikeb@ even did some network benchmarking
with it and didn't notice any decreased performance. There are ways
to make this a bit smarter, but let's leave that until we've gained a
little bit more experience with running code that doesn't grab the
kernel lock. So I'd like to move forward with this, doing at least
i386 and sparc64 as soon as we we're done bikeshedding over this amd64
version ;).
ok?
Index: amd64/mutex.S
===================================================================
RCS file: /cvs/src/sys/arch/amd64/amd64/mutex.S,v
retrieving revision 1.8
diff -u -p -r1.8 mutex.S
--- amd64/mutex.S 24 Sep 2010 13:21:30 -0000 1.8
+++ amd64/mutex.S 17 May 2013 17:51:56 -0000
@@ -38,7 +38,7 @@
* Yeah, we don't really need to implement mtx_init here, but let's keep
* all the functions in the same place.
*/
-ENTRY(mtx_init)
+ENTRY(__mtx_init)
movl %esi, MTX_WANTIPL(%rdi)
movl $0, MTX_OLDIPL(%rdi)
movq $0, MTX_OWNER(%rdi)
Index: include/mutex.h
===================================================================
RCS file: /cvs/src/sys/arch/amd64/include/mutex.h,v
retrieving revision 1.4
diff -u -p -r1.4 mutex.h
--- include/mutex.h 23 Mar 2011 16:54:34 -0000 1.4
+++ include/mutex.h 17 May 2013 17:51:56 -0000
@@ -33,7 +33,17 @@ struct mutex {
__volatile void *mtx_owner;
};
-#define MUTEX_INITIALIZER(ipl) { (ipl), 0, NULL }
+#ifdef MULTIPROCESSOR
+#define __MUTEX_IPL(ipl) \
+ (((ipl) > IPL_NONE && (ipl) < IPL_AUDIO) ? IPL_AUDIO : (ipl))
+#else
+#define __MUTEX_IPL(ipl) (ipl)
+#endif
+
+#define MUTEX_INITIALIZER(ipl) { __MUTEX_IPL((ipl)), 0, NULL }
+
+void __mtx_init(struct mutex *, int);
+#define mtx_init(mtx, ipl) __mtx_init((mtx), __MUTEX_IPL((ipl)))
#define MUTEX_ASSERT_LOCKED(mtx) do { \
if ((mtx)->mtx_owner != curcpu()) \
