Status: Accepted
Owner: [email protected]
Labels: Type-Defect Priority-High
New issue 202 by [email protected]: TOP_KEYS feature fixes
http://code.google.com/p/memcached/issues/detail?id=202
I did some real basic bench testing against the TOP_KEYS feature while ...
working on the benchmark stuff.
The basic results:
Normal: 400,000 gets/sec
TOP_KEYS: 200,000 gets/sec
8 threads were used. increasing thread count didn't increase performance.
Appears to be stacking on locks and spending a lot of time in sprintf.
Arguments I've heard about this being okay:
- memcached is so fast that it doesn't matter if you cut the capacity in
half
- users desperately need this so it's worth cutting capacity in half
Arguments I have against it:
- there will be things we do in the future that will slow it down, and
ideally we won't be putting ourselves in a position where you enable all of
these features at once and end up halving the capacity several times.
Performance reduction via in-line features should be strictly limited to
features that cannot be implemented outside of the daemon (ie; via tcpdump
and a script/program).
That said, I'm all for shipping a script or C app with memcached for
doing "topkeys-like" quick analysis.
- I dunno, that's basically it.
Approaches for fixing the issue:
- I'd prefer to tear it out and develop it in a branch, then add it back in
during 1.6.1 if it's fixable, or relegate it to a module or engine
extension and leave it out.
- The feature samples all keys, and is enabled at start time via an
environment variable. The only user friendly bits about the whole thing is
the information it eventually gives you. Math and usability rules don't
back up why this feature is the way it is; It should be a sampling set
(defaulting to 0, changeable at runtime). One sample every 1,000+ commands
on a busy server is probably 10x as frequent as it needs to be to find
the "top keys".
- I *hate* this thing. This is maybe 1/3rd of the useful information you
can get out of a key stream, and it's not in any form that could be
extensible. If we distribute another fast C based app (possibly start from
perl or whatever), you can find "top keys" via snapshots (the data only
matters when you're looking at it), and you can discern patterns from top
related keys. The tools I use to track down keys are often customized to
look for common sections of keys to see if particular features are going
off-kilter.
So in short, either way you need a key-stream analysis method to actually
get useful information out of a running instance. Providing anything else
without a method of getting the full picture is just flatly missleading.
