Hi Boris, I think I may have mislead you. It is not one or two keys that get corrupted, it seems that most (if not all) keys fetched return incorrect data. For example during one of these failures (just this morning), a session key (prefixed with session_) returned an array related to a customer record (prefixed with lab_), a key related to a customer return a string related to a translation, and a key related to a translation returned....
All heck breaks loose (seemingly) across all keys. A flush brings things back into the fold. Make sense? Thanks, Mike On Wednesday, November 19, 2014 2:22:50 PM UTC-4, Boris wrote: > > I can think of many ways to screw up an application in a way that you > describe. Simple programmer error can lead to this sort of behavior. I'd > just log every time you do a set for that key with value type you are > setting. > > On Wed, Nov 19, 2014 at 1:00 PM, <[email protected] <javascript:>> wrote: > >> Thanks Boris, >> >> I haven't really given that much thought. Out of curiosity, why do you >> think the issue might be on the client end? I ask, cause I really don't >> have a sense of what to look for on that end and wonder if you might have >> some suggestions. >> >> Best, >> >> Mike >> >> >> On Wednesday, November 19, 2014 12:46:16 PM UTC-4, Boris wrote: >>> >>> Hi Mike, this sounds to me more like a client/coding error rather than >>> memcached server. That's where I would focus first. >>> >>> Boris >>> >>> On Wed, Nov 19, 2014 at 11:41 AM, <[email protected]> wrote: >>> >>>> I just had another failure. After pulling down my apache web servers, >>>> and before restarting memcached I grabbed stats to see if they showed >>>> anything of interest: >>>> >>>> - All 3 servers were reporting for duty following a getServerStatus >>>> (PHP client call) >>>> - curr_connections were listed as 8 across all the instances (apache >>>> was down but cron jobs up, so that would have dropped things down >>>> considerably) >>>> - listen_disabled_num was listed as 0 across all the instances >>>> - accepting_conns was listed as 1 across all the instances >>>> - evictions listed as 0 >>>> - All items across all instances had an evicted and evicted_nonzero >>>> and evicted_time value of 0 >>>> - All slabs across all instances had a total_pages value of 1 >>>> - tailrepairs and outofmemory is listed with a value of 0 across all >>>> items in each instance >>>> - global hit rate is 0.9937 >>>> - get_hits is always* greater than cmd_set on a per slab basis. *One >>>> slab reported both values as equal >>>> >>>> >>>> As far as I can tell, memcache is reporting that the world is fine and >>>> dandy. Should I be enlarging scope of the search to look at OS related >>>> factors that could result in the client receiving bad data? None of the >>>> machines are dipping into swap. >>>> >>>> Thanks, >>>> >>>> Mike >>>> >>>> >>>> >>>> On Wednesday, November 19, 2014 9:35:19 AM UTC-4, [email protected] >>>> wrote: >>>>> >>>>> For what it is worth, I'm hesitant to upgrade memcached to the latest >>>>> version as a step to try and solve this issue. It seems to me that since >>>>> our installs have been running without issue for quite some time (close >>>>> to >>>>> a year), that there are other variables at play here. I just don't >>>>> understand the variables. ;) >>>>> >>>>> Thanks, >>>>> >>>>> Mike >>>>> >>>>> >>>>> On Tuesday, November 18, 2014 2:00:46 PM UTC-4, [email protected] >>>>> wrote: >>>>>> >>>>>> Hi There, >>>>>> >>>>>> I'm trying to diagnose a new problem with Memcache that seems to be >>>>>> happening with greater frequency. The issue has to do with memcache get >>>>>> requests returning incorrect responses (data from from other keys >>>>>> returned). Restarting or flushing the servers seems to resolve the >>>>>> issue. >>>>>> >>>>>> Do any memcache veterans have any suggestions of how I might dig into >>>>>> this issue? Stats that I might want to trace, log files to look at, >>>>>> etc? >>>>>> Does maybe this symptom fit the description of any known issues? >>>>>> >>>>>> I'm keeping a casual eye on on curr_connections, listen_disabled_num, >>>>>> accepting_conns, >>>>>> bytes, and limit_maxbytes (all show nothing unusual). I've verified >>>>>> that >>>>>> all servers and clients are set up in a consistent fashion. I'm not >>>>>> sure >>>>>> where to go from here to better understand the problem. >>>>>> >>>>>> >>>>>> If it helps, I'm running 1.4.13 (ubuntu 12.04 LTS) across 3 servers, >>>>>> connecting in with PHP Memcache 3.0.6 >>>>>> >>>>>> >>>>>> Tips? >>>>>> >>>>>> Mike >>>>>> >>>>>> >>>>>> >>>> >>>> >>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "memcached" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "memcached" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > > -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
