Hi Andrej,
I'd use lockstat (-kIW) or the dtrace profile provider to
initially get a handle on where your cycles are going. e.g.
something like:
dtrace -n 'profile-200us...@[cpu, arg0] = count();}
END{printa("%u:%a...@u\n", @);}'
Should give you a very quick profile indexed by cpu id. This
probably won't tell you exactly what you are seeing but it should
help with where to look next. You should also probably run
lockstat -A to see if you have contention/hold time problems on
locks.
Rgds,
Jon
Andrej Podzimek wrote:
Hello,
I ran into some problems with dynamic task queue performance. So I ran a
benchmark inside the kernel. It creates a task queue like this:
taskq_create_sysdc(
"blabla", /* name */
512, /* nthreads */
72, /* minalloc */
INT_MAX, /* maxalloc */
my_kernel_process, /* proc */
80, /* dc */
TASKQ_DYNAMIC | TASKQ_PREPOPULATE /* flags */
);
The benchmark starts 8 kernel LWPs. (BTW, it runs on an 8-thread Intel
Core i7.) Each of these LWPs enqueues a million callbacks like this:
static void
callback(uint32_t *counter) {
atomic_dec_32(counter);
}
There are multiple counters and pointers to those counters are
distributed evenly among the callbacks.
Here comes a simple thought:
* Let's assume one CPU can run one billion instructions per second.
* Let's assume one callback (with all the overhead) could cost ten
thousand instructions.
* Then each CPU could process 100000 callbacks per second on an
otherwise idle system...
Now the reality:
* I booted onnv_144 (DEBUG kernel), started the benchmark and thought it
would take just seconds.
* After 10 *minutes*, I started mdb to see what was going on. :-(
* All the 8 benchmarking LWPs were *sleeping* in taskq_dispatch().
* All the taskq threads I looked at were sleeping as well, at least at
the moment of observation.
* By looking at the tq_executed counter periodically, I found out that
only about 110 tasks ran per second.
* The CPU was spending 90% of time in the kernel, which doesn't look
like deep sleeping.
* The counters were decremented as expected, but it took ages...
What could be wrong? Where is the bottleneck?
On one hand, most task queue threads appear to be sleeping, waiting for
a job. The 8 threads producing the callbacks also appear to be sleeping.
On the other hand, all CPUs spend more than 90% of time in the kernel...
There must be a bottleneck somewhere. What could I try? Fewer task queue
threads? Or a non-DEBUG kernel? How could this be diagnosed on a running
system? (I can provide a full 'halt -d' dump.)
I know that task queues are not designed for this type of "workload".
But performing 110 atomic decrements per second on an 8-thread Nehalem
CPU is just far below what I would expect.
Any thoughts or hints would be very helpful. :-)
I think this could be a memory-related issue, but I'm not sure about it.
I reduced the number of callbacks 1000 times and rerun the benchmark.
Based on an estimate of 100 callbacks per second, it should have taken
about 80 seconds. But this time the pathological case did not happen.
The benchmark completed in a fraction of a second.
But there's still something wrong with this theory: When I looked at the
memory statistics during the benchmark with the original high number of
callbacks, there was *no* evidence of memory pressure:
::memstat
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 633467 2474 31%
ZFS File Data 40890 159 2%
Anon 21287 83 1%
Exec and libs 1259 4 0%
Page cache 4564 17 0%
Free (cachelist) 4148 16 0%
Free (freelist) 1370935 5355 66%
Total 2076550 8111
Physical 2076549 8111
Those 2.5 gigabytes of kernel memory were allocated by the benchmark, so
there is nothing inexplicable. Page allocation can sleep when available
memory < throttlefree, but it is obvious that this did not happen.
Perhaps the kernel has another memory allocation throttling mechanism
unknown to me...
Andrej
------------------------------------------------------------------------
_______________________________________________
on-discuss mailing list
on-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/on-discuss
--
<http://www.oracle.com>
Jon Anderson
Tel: ++44 (0) 1252 421 868
Mob: ++44 (0) 7747 180 910
ORACLE RPE Systems
SPARC House (UK)
"Please consider your environmental responsibility before
printing this email."
_______________________________________________
on-discuss mailing list
on-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/on-discuss
