Patrick, are you sure the time is not spent GC'ing? Are you using icms (-XX:+UseConcMarkSweepGC)? Is your verbose gc logging on?
On Mon, Feb 21, 2011 at 9:23 PM, Patrick Santora <[email protected]> wrote: > Its just strange. Memcaced with verbose logging looks ok but the client > machines just take forever to get data. Like in the stats I don't see > anything out of the ordinary. The nic settings look ok too. Quite > frustrating... > > On Feb 21, 2011 11:51 AM, "Patrick Santora" <[email protected]> wrote: >> I will need to look at those further today. This weekend went a little >> haywire for me. :) >> On Feb 21, 2011 11:42 AM, "dormando" <[email protected]> wrote: >>> Have you walked through those links I gave you? You haven't mentioned >>> exactly what you're seeing and those links walk you through narrowing it >>> down a lot as well as listing a lot of things to look for. >>> >>> On Mon, 21 Feb 2011, Patrick Santora wrote: >>> >>>> Hrmm. Still having issues. Here is the latest stats dump. I also talked >> with my IT person and he mentioned the following setup, which does >>>> not look like an issue? >>>> NIC SETTINGS >>>> the servers should all be autonegotiating to 100/Full and we apply these >> additional kernel tuning parameters >>>> net.core.rmem_max = 16777216 >>>> net.core.wmem_max = 16777216 >>>> net.ipv4.tcp_rmem = 4096 87380 16777216 >>>> net.ipv4.tcp_wmem = 4096 65536 16777216 >>>> >>>> LATEST STATS >>>> STAT pid 1788 >>>> STAT uptime 44811 >>>> STAT time 1298311271 >>>> STAT version 1.4.5 >>>> STAT pointer_size 64 >>>> STAT rusage_user 178.875806 >>>> STAT rusage_system 763.939863 >>>> STAT curr_connections 811 >>>> STAT total_connections 2012 >>>> STAT connection_structures 813 >>>> STAT cmd_get 876886 >>>> STAT cmd_set 74747 >>>> STAT cmd_flush 0 >>>> STAT get_hits 858907 >>>> STAT get_misses 17979 >>>> STAT delete_misses 0 >>>> STAT delete_hits 2 >>>> STAT incr_misses 0 >>>> STAT incr_hits 0 >>>> STAT decr_misses 0 >>>> STAT decr_hits 0 >>>> STAT cas_misses 0 >>>> STAT cas_hits 0 >>>> STAT cas_badval 0 >>>> STAT auth_cmds 0 >>>> STAT auth_errors 0 >>>> STAT bytes_read 17426408671 >>>> STAT bytes_written 180479901035 >>>> STAT limit_maxbytes 536870912 >>>> STAT accepting_conns 1 >>>> STAT listen_disabled_num 0 >>>> STAT threads 4 >>>> STAT conn_yields 0 >>>> STAT bytes 3501518 >>>> STAT curr_items 3230 >>>> STAT total_items 74747 >>>> STAT evictions 0 >>>> STAT reclaimed 20950 >>>> END >>>> >>>> >>>> On Mon, Feb 21, 2011 at 8:12 AM, Patrick Santora <[email protected]> >> wrote: >>>> @Dustin >>>> Thanks, I will be disabling them to see if that helps. >>>> >>>> -Pat >>>> >>>> >>>> On Mon, Feb 21, 2011 at 5:59 AM, Dustin <[email protected]> wrote: >>>> >>>> On Feb 21, 12:31 am, Patrick Santora <[email protected]> wrote: >>>> > Heh. I had a funny feeling that was going to be the answer. I was >> curious >>>> > mostly because the Binary mode seemed to do quite a deal of good for >>>> > Facebook when it was used. I'm imagining that they cached images so >> binary >>>> > was a good idea, but for simple structures like json, it might not >>>> > make >> much >>>> > sense. So thought I would get some opinions :). >>>> >>>> binary protocol doesn't make much of a difference wrt what you're >>>> caching, but can help you optimize some access patterns with a >>>> sufficiently smart client. If you're concerned that it may be making >>>> things worse (it probably doesn't have a huge effect from what I'm >>>> hearing here), you can just try disabling it. >>>> >>>> >>>> >>>> >>>> >
