On 05.09.2013 15:40, Alexander Motin wrote:
Some may remember that not so long ago I complained about high lock
congestion on pbuf_mtx. At that time switching the mutex to padalign
reduced the problem. But now after improving scalability in CAM and GEOM
and doing more then half million IOPS on 32-core system I again heavily
hit that problem -- hwpmc shows about 30% of CPU time spent on that
mutex spinning and another 30% of time spent on attempt of threads to go
to sleep on that mutex and getting more collisions there.

Trying to mitigate that I've made a patch
(http://people.freebsd.org/~mav/pcpu_pbuf.patch) to split single queue
of pbufs into several. That definitely cost some amount of KVA and
memory, but on my tests it fixes problem redically, removing any
measurable congestion there. The patch is not complete and don't even
boot on i386 now, but I would like to hear opinions about the approach,
or may be some better propositions.

On kib@ proposition I've tried to reimplement that patch using vmem(9). Code indeed looks much better (at least looked before workarounds):
http://people.freebsd.org/~mav/pbuf_vmem.patch
and it works fast, but I have found number of problems:
- now we have only 256 (or even less) pbufs and UMA used by vmem for quick caches tend to allocate up to 256 items per CPU and never release them back. I've partially workarounded that by passing fake MAGIC_SIZE value to vmem and down to UMA as size to make initial bucket sizes smaller, but that is a hack and not always sufficient since size may grow on congestion and again never shrink back. - UMA panics with "uma_zalloc: Bucket pointer mangled." if I am giving vmem zero as valid pointer. I've workarounded that by adding an offset to the value, but I think that assertion in UMA should be removed if we are going to use it for abstract values now.

Another patch I've made
(http://people.freebsd.org/~mav/si_threadcount.patch) removes lock
acquisition from dev_relthread() by using atomics for reference
counting. That fixes another congestion I see. This patch looks fine to
me and the only congestion I see after that is on HBA driver locks, but
may be I am missing something?

--
Alexander Motin
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"

Reply via email to