Yeah I have the debug switch on. Thanks for the recommendation on running it in two places. I will give that a shot. I saw that the set and get times will be 0 if there is anything wrong via the debug. I take it if that's the case then I should look into what the verbose logs give me? Is there anything specific I should look for as when I look at it I'm not seeing anything very unusual.
Thanks -Pat On Mon, Feb 21, 2011 at 8:02 PM, dormando <[email protected]> wrote: > Run two, and keep them running all the time, so you see log from > before/after. You can also enable the "debug" switch and have it log > everything. > > So yeah. run one on the client and one on an idle machine elsewhere. > > On Mon, 21 Feb 2011, Patrick Santora wrote: > > > > > Yeah. I will run it the next time the issue comes up. Does it matter if I > run the tester on the same box the clients on? It should not matter but > > thought ii would ask. > > > > Thanks! > > > > On Feb 21, 2011 6:25 PM, "dormando" <[email protected]> wrote: > > > Have you been running the connection tester tool while observing the > > > client slowdown? > > > > > > The tool is there so you can rule if your client is an issue or not, > ie; > > > if the tool never sees a blip but all/most/some of your clients are > seeing > > > blips, it's the client's fault. If the tool sees a blip, you can see > > > exactly where it's getting hung up and further narrow it down. > > > > > > On Mon, 21 Feb 2011, Patrick Santora wrote: > > > > > >> > > >> Its just strange. Memcaced with verbose logging looks ok but the > client machines just take forever to get data. Like in the stats I don't > > >> see anything out of the ordinary. The nic settings look ok too. Quite > frustrating... > > >> > > >> On Feb 21, 2011 11:51 AM, "Patrick Santora" <[email protected]> > wrote: > > >> > I will need to look at those further today. This weekend went a > little > > >> > haywire for me. :) > > >> > On Feb 21, 2011 11:42 AM, "dormando" <[email protected]> wrote: > > >> >> Have you walked through those links I gave you? You haven't > mentioned > > >> >> exactly what you're seeing and those links walk you through > narrowing it > > >> >> down a lot as well as listing a lot of things to look for. > > >> >> > > >> >> On Mon, 21 Feb 2011, Patrick Santora wrote: > > >> >> > > >> >>> Hrmm. Still having issues. Here is the latest stats dump. I also > talked > > >> > with my IT person and he mentioned the following setup, which does > > >> >>> not look like an issue? > > >> >>> NIC SETTINGS > > >> >>> the servers should all be autonegotiating to 100/Full and we apply > these > > >> > additional kernel tuning parameters > > >> >>> net.core.rmem_max = 16777216 > > >> >>> net.core.wmem_max = 16777216 > > >> >>> net.ipv4.tcp_rmem = 4096 87380 16777216 > > >> >>> net.ipv4.tcp_wmem = 4096 65536 16777216 > > >> >>> > > >> >>> LATEST STATS > > >> >>> STAT pid 1788 > > >> >>> STAT uptime 44811 > > >> >>> STAT time 1298311271 > > >> >>> STAT version 1.4.5 > > >> >>> STAT pointer_size 64 > > >> >>> STAT rusage_user 178.875806 > > >> >>> STAT rusage_system 763.939863 > > >> >>> STAT curr_connections 811 > > >> >>> STAT total_connections 2012 > > >> >>> STAT connection_structures 813 > > >> >>> STAT cmd_get 876886 > > >> >>> STAT cmd_set 74747 > > >> >>> STAT cmd_flush 0 > > >> >>> STAT get_hits 858907 > > >> >>> STAT get_misses 17979 > > >> >>> STAT delete_misses 0 > > >> >>> STAT delete_hits 2 > > >> >>> STAT incr_misses 0 > > >> >>> STAT incr_hits 0 > > >> >>> STAT decr_misses 0 > > >> >>> STAT decr_hits 0 > > >> >>> STAT cas_misses 0 > > >> >>> STAT cas_hits 0 > > >> >>> STAT cas_badval 0 > > >> >>> STAT auth_cmds 0 > > >> >>> STAT auth_errors 0 > > >> >>> STAT bytes_read 17426408671 > > >> >>> STAT bytes_written 180479901035 > > >> >>> STAT limit_maxbytes 536870912 > > >> >>> STAT accepting_conns 1 > > >> >>> STAT listen_disabled_num 0 > > >> >>> STAT threads 4 > > >> >>> STAT conn_yields 0 > > >> >>> STAT bytes 3501518 > > >> >>> STAT curr_items 3230 > > >> >>> STAT total_items 74747 > > >> >>> STAT evictions 0 > > >> >>> STAT reclaimed 20950 > > >> >>> END > > >> >>> > > >> >>> > > >> >>> On Mon, Feb 21, 2011 at 8:12 AM, Patrick Santora < > [email protected]> > > >> > wrote: > > >> >>> @Dustin > > >> >>> Thanks, I will be disabling them to see if that helps. > > >> >>> > > >> >>> -Pat > > >> >>> > > >> >>> > > >> >>> On Mon, Feb 21, 2011 at 5:59 AM, Dustin <[email protected]> > wrote: > > >> >>> > > >> >>> On Feb 21, 12:31 am, Patrick Santora <[email protected]> wrote: > > >> >>> > Heh. I had a funny feeling that was going to be the answer. I > was > > >> > curious > > >> >>> > mostly because the Binary mode seemed to do quite a deal of good > for > > >> >>> > Facebook when it was used. I'm imagining that they cached images > so > > >> > binary > > >> >>> > was a good idea, but for simple structures like json, it might > not make > > >> > much > > >> >>> > sense. So thought I would get some opinions :). > > >> >>> > > >> >>> binary protocol doesn't make much of a difference wrt what you're > > >> >>> caching, but can help you optimize some access patterns with a > > >> >>> sufficiently smart client. If you're concerned that it may be > making > > >> >>> things worse (it probably doesn't have a huge effect from what I'm > > >> >>> hearing here), you can just try disabling it. > > >> >>> > > >> >>> > > >> >>> > > >> >>> > > >> >>> > > >> > > >> > > > > >
