Thanks for you response, find my response below: On Mon, Feb 9, 2015 at 1:39 PM, dormando <dorma...@rydia.net> wrote:
> > I am running some tests using memached 1.4.22 over an Intel Xeon E5 (4 > sockets with 8 core each, 2 Hyper threads per core, and 4 NUMA nodes) and > > running Ubuntu trusty. I compiled memcached with gcc-4.8.2 with default > CFLAGS and configuration options. > > > > The problem is whenever I start memcached with odd number of server > threads (3,5,7,9,11,..) everything is ok, and all threads are engaging in > > processing requests, the status of all threads are "Running". However, > if I start the server with even number of threads (2,4,6,8,..), half of the > > threads are always in sleep mode and do not engage in servicing clients. > This is related to memached, as memaslap, for example, is running with no > > such pattern. I ran the exact test on an AMD Opteron and things are ok > with memached. So my question is: is there any specific tuning required for > > Intel machines? Is there any specific flag or some part of the code that > might cause worker threads to not engage? > > > > > > Thanks, > > Saman > > That is pretty weird. I've not run it on a quad socket but plenty of intel > machines without problem. Modern ones too. > I see, I am not sure why it happens cause everything is very straight forward with memcached. > How many clients are you telling memslap to use? Can you try > https://github.com/dormando/mc-crusher quickly? (run loadconf/similar to > load some values, then a different one to hammer it). > I fire memaslap with the same number of threads as memcached, and with concurrency 20 per thread, so enough to keep server threads busy. It seems that using mc-crusher, all threads are engaged when loading and with mget_test. So does it mean there is something fishy with memaslap? > Connections are dispersed via thread.c:dispatch_conn_new() > > int tid = (last_thread + 1) % settings.num_threads; > > LIBEVENT_THREAD *thread = threads + tid; > > last_thread = tid; > > which is pretty simple at the base. > Right, I printed out 'last_thread' to make sure nothing funny is happening, but it's perfect round robin. > If you can gdb up can you dump the per-thread stats structures? that will > show definitively if those threads ever get work or not. I ran memcached with -t 8 and client side is memaslap -s localhost -T 8 -c 160 -t 1m and below find the gdb output for per thread stats structure. It seems that every other thread is not doing anything!? Also I can confirm that all memaspal threads are consuming 100% of the cores they are running on, and again with odd number of server threads this does not happen. You can see the kind of results I get when running memcached and increase the number of threads on that machine. It is not consistent at all! Thanks, Saman (gdb) print threads[0].stats $21 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, get_cmds = 0, get_misses = 0, touch_cmds = 0, touch_misses = 0, delete_misses = 0, incr_misses = 0, decr_misses = 0, cas_misses = 0, bytes_read = 0, bytes_written = 0, flush_cmds = 0, conn_yields = 0, auth_cmds = 0, auth_errors = 0, slab_stats = {{set_cmds = 0, get_hits = 0, touch_hits = 0, delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits = 0} <repeats 201 times>}} (gdb) print threads[1].stats $22 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, get_cmds = 4466788, get_misses = 3863038, touch_cmds = 0, touch_misses = 0, delete_misses = 0, incr_misses = 0, decr_misses = 0, cas_misses = 0, bytes_read = 861116495, bytes_written = 693448306, flush_cmds = 0, conn_yields = 0, auth_cmds = 0, auth_errors = 0, slab_stats = {{set_cmds = 0, get_hits = 0, touch_hits = 0, delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits = 0} <repeats 12 times>, {set_cmds = 496327, get_hits = 603750, touch_hits = 0, delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits = 0}, {set_cmds = 0, get_hits = 0, touch_hits = 0, delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits = 0} <repeats 188 times>}} (gdb) print threads[2].stats $23 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, get_cmds = 0, get_misses = 0, touch_cmds = 0, touch_misses = 0, delete_misses = 0, incr_misses = 0, decr_misses = 0, cas_misses = 0, bytes_read = 0, bytes_written = 0, flush_cmds = 0, conn_yields = 0, auth_cmds = 0, auth_errors = 0, slab_stats = {{set_cmds = 0, get_hits = 0, touch_hits = 0, delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits = 0} <repeats 201 times>}} (gdb) print threads[3].stats $24 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, get_cmds = 4120462, get_misses = 3550471, touch_cmds = 0, touch_misses = 0, delete_misses = 0, incr_misses = 0, decr_misses = 0, cas_misses = 0, bytes_read = 794355485, bytes_written = 654105157, flush_cmds = 0, conn_yields = 0, auth_cmds = 0, auth_errors = 0, slab_stats = {{set_cmds = 0, get_hits = 0, touch_hits = 0, delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits = 0} <repeats 12 times>, {set_cmds = 457849, get_hits = 569991, touch_hits = 0, delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits = 0}, {set_cmds = 0, get_hits = 0, touch_hits = 0, delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits = 0} <repeats 188 times>}} (gdb) print threads[4].stats $25 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, get_cmds = 0, get_misses = 0, touch_cmds = 0, touch_misses = 0, delete_misses = 0, incr_misses = 0, decr_misses = 0, cas_misses = 0, bytes_read = 0, bytes_written = 0, flush_cmds = 0, conn_yields = 0, auth_cmds = 0, auth_errors = 0, slab_stats = {{set_cmds = 0, get_hits = 0, touch_hits = 0, delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits = 0} <repeats 201 times>}} (gdb) print threads[5].stats $26 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, get_cmds = 4038230, get_misses = 3493086, touch_cmds = 0, touch_misses = 0, delete_misses = 0, incr_misses = 0, decr_misses = 0, cas_misses = 0, bytes_read = 778500650, bytes_written = 626164950, flush_cmds = 0, conn_yields = 0, auth_cmds = 0, auth_errors = 0, slab_stats = {{set_cmds = 0, get_hits = 0, touch_hits = 0, delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits = 0} <repeats 12 times>, {set_cmds = 448710, get_hits = 545144, touch_hits = 0, delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits = 0}, {set_cmds = 0, get_hits = 0, touch_hits = 0, delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits = 0} <repeats 188 times>}} (gdb) print threads[6].stats $27 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, get_cmds = 0, get_misses = 0, touch_cmds = 0, touch_misses = 0, delete_misses = 0, incr_misses = 0, decr_misses = 0, cas_misses = 0, bytes_read = 0, bytes_written = 0, flush_cmds = 0, conn_yields = 0, auth_cmds = 0, auth_errors = 0, slab_stats = {{set_cmds = 0, get_hits = 0, touch_hits = 0, delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits = 0} <repeats 201 times>}} (gdb) print threads[7].stats $28 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, get_cmds = 4472436, get_misses = 3868324, touch_cmds = 0, touch_misses = 0, delete_misses = 0, incr_misses = 0, decr_misses = 0, cas_misses = 0, bytes_read = 862203585, bytes_written = 693881564, flush_cmds = 0, conn_yields = 0, auth_cmds = 0, auth_errors = 0, slab_stats = {{set_cmds = 0, get_hits = 0, touch_hits = 0, delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits = 0} <repeats 12 times>, {set_cmds = 496953, get_hits = 604112, touch_hits = 0, delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits = 0}, {set_cmds = 0, get_hits = 0, touch_hits = 0, delete_hits = 0, cas_hits = 0, cas_badval = 0, incr_hits = 0, decr_hits = 0} <repeats 188 times>}} -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to memcached+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
