I think it's clearer if we separate the requirements: (A) continuous monitoring of the whole cluster, in production. (B) troubleshooting a specific node, key, operation or client -- without impacting (A)
For (A) you want the most robust, scale-out measurement you can find that will not impact performance but still provide as much insight as possible. Packing random samples into XDR-encoded UDP datagrams (i.e. sFlow) is a good fit for this. Implemented carefully the overhead is roughly equivalent to adding one counter to the stats block. It's worth agreeing on a standard format because the cluster-wide analysis is necessarily "external" to memcached. There is no single node that can tell you the cluster-wide top-keys. Selecting sFlow as the format makes sense because the cluster also comprises networking, servers, hypervisors, web-servers etc. that can all export sFlow too (each providing their own perspective). This way the continuous monitoring can all be done by listening for standard UDP datagrams on a single UDP socket on a separate server (or servers). For (B), I think the "tap" toolkit looks promising. It complements sFlow well. A stream-oriented protocol with options and filters to allow you to nose into all the dark corners. Perfect. In other words. These two solutions are different in almost every way, and I think you want both. There's nothing stopping you from consuming random samples "internally" (locally on one node) as well. I just can't think of an every-day reason to do that. Any analysis where you want to filter/aggregate/sort/chop/threshold on the populations of keys can be done at the sFlow collector server -- either for a particular node or for the whole cluster. You can change the query without making any change at all to the nodes in the cluster. Doing the same thing as a "builtin" just seems redundant, risks hurting performance (yes TOPKEYS, I mean you!), and results in a config change on the node every time you change the query. I suspect that many memcached users would break out in a cold sweat just changing the sFlow sampling rate from 1-in-1000 to 1-in-1001, so anything more invasive than that is likely to be a tough sell. If your analysis occasionally calls for a local measurement that cannot be driven from the sFlow samples (e.g. because it needs to see the content of the value object, or has to see every transaction that matches a filter) then you can use "tap". I think you would still want as much as possible to be done "externally" in another process, though. The simpler and more robust the "tap" protocol is, the more likely it will be trusted and used. Neil P.S. Minor point: the current sFlow patch includes generic code that allows for multiple "agents", "samplers", "pollers" and "receivers" all with different sampling rates and polling intervals. In practice we only have one of each here, so the patch could be stripped down to a few hundred lines if we cut out all the layers of indirection and just assembled the UDP datagram inline. I'd be happy to do that if you think it would help. On Aug 7, 2011, at 6:11 PM, dormando wrote: >>> From: [email protected] [mailto:[email protected]] On >>> Behalf Of Peter Portante >>> Sent: Monday, 8 August 2011 10:49 AM >>> >>> How 'bout random sample request profiling? >> >> Profiling for monitoring and activity estimation purposes - isn't that the >> point of the sFlow set of patches mentioned a few times on list? > > The sFlow patches bother me as I'd prefer to be able to generate sFlow > events from a proper internal system, as opposed to the inverse. You > shouldn't have to be an sFlow consumer, and it's much more difficult to > vary the type of data you'd be ingesting (full headers, vs partial, vs > item bodies, etc). > > The internal statistical sampling would be the start, then come methods of > shipping it. You could send to listeners connected over a socket, or have > a plugin listen as an internal consumer to the samplings. The internal > consumer could provide "builtin" statistical summaries the same as an > external daemon could. Which could make everyone happy in this case. > > I like the sFlow stuff, I'm just at a loss for why it's so important that > everything be generated on top of sFlow. So far nobody's addressed my > specific arguments as listed above. > > -Dormando
