Btw, I already considered avoiding hyper threads and running all server threads on the same socket (to avoid cross socket latencies and better cache usage), but it does not seem to be a hardware related problem.
On Mon, Feb 9, 2015 at 4:29 PM, Saman Barghi <sama...@gmail.com> wrote: > Thanks for you response, find my response below: > > On Mon, Feb 9, 2015 at 1:39 PM, dormando <dorma...@rydia.net> wrote: > >> > I am running some tests using memached 1.4.22 over an Intel Xeon E5 (4 >> sockets with 8 core each, 2 Hyper threads per core, and 4 NUMA nodes) and >> > running Ubuntu trusty. I compiled memcached with gcc-4.8.2 with default >> CFLAGS and configuration options. >> > >> > The problem is whenever I start memcached with odd number of server >> threads (3,5,7,9,11,..) everything is ok, and all threads are engaging in >> > processing requests, the status of all threads are "Running". However, >> if I start the server with even number of threads (2,4,6,8,..), half of the >> > threads are always in sleep mode and do not engage in servicing >> clients. This is related to memached, as memaslap, for example, is running >> with no >> > such pattern. I ran the exact test on an AMD Opteron and things are ok >> with memached. So my question is: is there any specific tuning required for >> > Intel machines? Is there any specific flag or some part of the code >> that might cause worker threads to not engage? >> > >> > >> > Thanks, >> > Saman >> >> That is pretty weird. I've not run it on a quad socket but plenty of intel >> machines without problem. Modern ones too. >> > > I see, I am not sure why it happens cause everything is very straight > forward with memcached. > > >> How many clients are you telling memslap to use? Can you try >> https://github.com/dormando/mc-crusher quickly? (run loadconf/similar to >> load some values, then a different one to hammer it). >> > > I fire memaslap with the same number of threads as memcached, and with > concurrency 20 per thread, so enough to keep server threads busy. > > It seems that using mc-crusher, all threads are engaged when loading and > with mget_test. So does it mean there is something fishy with memaslap? > > >> Connections are dispersed via thread.c:dispatch_conn_new() >> >> int tid = (last_thread + 1) % settings.num_threads; >> >> LIBEVENT_THREAD *thread = threads + tid; >> >> last_thread = tid; >> >> which is pretty simple at the base. >> > > Right, I printed out 'last_thread' to make sure nothing funny is > happening, but it's perfect round robin. > > >> If you can gdb up can you dump the per-thread stats structures? that will >> show definitively if those threads ever get work or not. > > > > I ran memcached with -t 8 and client side is > > memaslap -s localhost -T 8 -c 160 -t 1m > > and below find the gdb output for per thread stats structure. It seems > that every other thread is not doing anything!? Also I can confirm that all > memaspal threads are consuming 100% of the cores they are running on, and > again with odd number of server threads this does not happen. You can see > the kind of results I get when running memcached and increase the number of > threads on that machine. It is not consistent at all! > > Thanks, > Saman > > > (gdb) print threads[0].stats > $21 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = > 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = > 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, get_cmds = 0, > get_misses = 0, > touch_cmds = 0, touch_misses = 0, delete_misses = 0, incr_misses = 0, > decr_misses = 0, cas_misses = 0, bytes_read = 0, bytes_written = 0, > flush_cmds = 0, conn_yields = 0, auth_cmds = 0, auth_errors = 0, slab_stats > = {{set_cmds = 0, > get_hits = 0, touch_hits = 0, delete_hits = 0, cas_hits = 0, > cas_badval = 0, incr_hits = 0, decr_hits = 0} <repeats 201 times>}} > (gdb) print > threads[1].stats > > $22 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = > 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = > 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, get_cmds = > 4466788, > get_misses = 3863038, touch_cmds = 0, touch_misses = 0, delete_misses = > 0, incr_misses = 0, decr_misses = 0, cas_misses = 0, bytes_read = > 861116495, bytes_written = 693448306, flush_cmds = 0, conn_yields = 0, > auth_cmds = 0, > auth_errors = 0, slab_stats = {{set_cmds = 0, get_hits = 0, touch_hits = > 0, delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits > = 0} <repeats 12 times>, {set_cmds = 496327, get_hits = 603750, touch_hits > = 0, > delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, > decr_hits = 0}, {set_cmds = 0, get_hits = 0, touch_hits = 0, delete_hits = > 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits = 0} <repeats 188 > times>}} > (gdb) print > threads[2].stats > > $23 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = > 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = > 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, get_cmds = 0, > get_misses = 0, > touch_cmds = 0, touch_misses = 0, delete_misses = 0, incr_misses = 0, > decr_misses = 0, cas_misses = 0, bytes_read = 0, bytes_written = 0, > flush_cmds = 0, conn_yields = 0, auth_cmds = 0, auth_errors = 0, slab_stats > = {{set_cmds = 0, > get_hits = 0, touch_hits = 0, delete_hits = 0, cas_hits = 0, > cas_badval = 0, incr_hits = 0, decr_hits = 0} <repeats 201 times>}} > (gdb) print > threads[3].stats > > $24 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = > 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = > 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, get_cmds = > 4120462, > get_misses = 3550471, touch_cmds = 0, touch_misses = 0, delete_misses = > 0, incr_misses = 0, decr_misses = 0, cas_misses = 0, bytes_read = > 794355485, bytes_written = 654105157, flush_cmds = 0, conn_yields = 0, > auth_cmds = 0, > auth_errors = 0, slab_stats = {{set_cmds = 0, get_hits = 0, touch_hits = > 0, delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits > = 0} <repeats 12 times>, {set_cmds = 457849, get_hits = 569991, touch_hits > = 0, > delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, > decr_hits = 0}, {set_cmds = 0, get_hits = 0, touch_hits = 0, delete_hits = > 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits = 0} <repeats 188 > times>}} > (gdb) print threads[4].stats > $25 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = > 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = > 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, get_cmds = 0, > get_misses = 0, > touch_cmds = 0, touch_misses = 0, delete_misses = 0, incr_misses = 0, > decr_misses = 0, cas_misses = 0, bytes_read = 0, bytes_written = 0, > flush_cmds = 0, conn_yields = 0, auth_cmds = 0, auth_errors = 0, slab_stats > = {{set_cmds = 0, > get_hits = 0, touch_hits = 0, delete_hits = 0, cas_hits = 0, > cas_badval = 0, incr_hits = 0, decr_hits = 0} <repeats 201 times>}} > (gdb) print threads[5].stats > $26 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = > 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = > 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, get_cmds = > 4038230, > get_misses = 3493086, touch_cmds = 0, touch_misses = 0, delete_misses = > 0, incr_misses = 0, decr_misses = 0, cas_misses = 0, bytes_read = > 778500650, bytes_written = 626164950, flush_cmds = 0, conn_yields = 0, > auth_cmds = 0, > auth_errors = 0, slab_stats = {{set_cmds = 0, get_hits = 0, touch_hits = > 0, delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits > = 0} <repeats 12 times>, {set_cmds = 448710, get_hits = 545144, touch_hits > = 0, > delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, > decr_hits = 0}, {set_cmds = 0, get_hits = 0, touch_hits = 0, delete_hits = > 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits = 0} <repeats 188 > times>}} > (gdb) print threads[6].stats > $27 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = > 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = > 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, get_cmds = 0, > get_misses = 0, > touch_cmds = 0, touch_misses = 0, delete_misses = 0, incr_misses = 0, > decr_misses = 0, cas_misses = 0, bytes_read = 0, bytes_written = 0, > flush_cmds = 0, conn_yields = 0, auth_cmds = 0, auth_errors = 0, slab_stats > = {{set_cmds = 0, > get_hits = 0, touch_hits = 0, delete_hits = 0, cas_hits = 0, > cas_badval = 0, incr_hits = 0, decr_hits = 0} <repeats 201 times>}} > (gdb) print threads[7].stats > $28 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = > 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = > 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, get_cmds = > 4472436, > get_misses = 3868324, touch_cmds = 0, touch_misses = 0, delete_misses = > 0, incr_misses = 0, decr_misses = 0, cas_misses = 0, bytes_read = > 862203585, bytes_written = 693881564, flush_cmds = 0, conn_yields = 0, > auth_cmds = 0, > auth_errors = 0, slab_stats = {{set_cmds = 0, get_hits = 0, touch_hits = > 0, delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits > = 0} <repeats 12 times>, {set_cmds = 496953, get_hits = 604112, touch_hits > = 0, > delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, > decr_hits = 0}, {set_cmds = 0, get_hits = 0, touch_hits = 0, delete_hits = > 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits = 0} <repeats 188 > times>}} > > > -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to memcached+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
