Hello,
To mighty memcached developers, and memcached users that do C/C++, let
me introduce XProbes, a framework for static user space probes.
A while ago DTrace probes were injected into memcached source code.
While usable on many systems (including Linux via dtrace-to-systemtap
converter), DTrace/SystemTap primary purpose is OS kernel tracing.
When used for user space applications, these tools provide measurable
performance penalty (DTrace uses trap instruction, SystemTap uses
either trap or system call, i.e., every probe hit requires a context
switch). Also, their support for user-defined types is very limited.
Consider this definition from memcached_dtrace.d:
/**
* Fired when a connection object is destroyed ("released back to
* the memory subsystem").
* @param ptr pointer to the connection object
*/
probe conn__destroy(void *ptr);
Note that struct conn is passed as void *, you are supposed to
manually cast its fields to basic types in the probe action code
(which by the way is to be written in yet another interpreted language
called D). SystemTap is no different: you write probes in a script
language that is translated to C, compiled and loaded into Linux
kernel. Interpreter is removed, but you still have to use some
scripting language will little correspondence to your application
code.
XProbes serves the same purposes as DTrace, SystemTap (and LTTng), but
focuses solely on user space applications, and provides efficient
probing with natural data access for them.
Here's an example. We will implement conn__release probe because
conn__destroy is hit very rarely. First, in memcached.c next to
DTrace macro call
MEMCACHED_CONN_RELEASE(c->sfd);
we inject
XPROBES_SITE(memcached_conn_release,
(const struct conn *),
(c));
(defined in <xprobes/site.h>), and link memcached with -lxprobes (I
actually did the example, tweaked memcached_LDADD and
memcached_debug_LDADD in top Makefile.am). Then we start memcached,
and it happily runs forever with no measurable performance penalty.
Later, when we want to peek at released connections, we write a
probe. Suppose I decided to count different connection types:
#include <xprobes/probe.h>
#include "memcached.h"
#include <string.h>
#include <stdio.h>
static int ascii = 0;
static int binary = 0;
static
void
release(const struct conn *c)
{
if (c->protocol == ascii_prot)
++ascii;
else
++binary;
}
static
void
command(const char *cmd, int (*out)(const char *msg))
{
if (strcmp(cmd, "protocol") == 0)
{
char buf[64];
sprintf(buf,
"ascii: %d\n"
"binary: %d\n",
ascii, binary);
out(buf);
}
else
{
out("unknown command ");
out(cmd);
}
}
XPROBES_MODULE(command, 0,
XPROBES_PROBE("*_conn_release", release, (const struct conn *)));
Note how naturally we can access struct conn fields and enum protocol
members. Now we compile the probe (in memcached source directory)
with
gcc -std=gnu99 -shared -fPIC -DPIC -O2 -DHAVE_CONFIG_H probe.c -o probe.so
(here and above we assume that XProbes is installed in standard
location, and we don't have to provide -I (and -L for -lxprobes
above)).
We all know what it costs to restart memcached, so let's attach the
probe to a running process:
bash$ xprobes `pidof memcached`
Connecting... done
xprobes>
First let's ensure that our probe site is there:
xprobes> sites
./memcached:
memcached_conn_release(const struct conn*)
xprobes>
Here it is. Now let's load the probe module:
xprobes> load /tmp/memcached/probe.so
command in progress, waiting...
loaded
xprobes>
NOTE: when you try this example on otherwise idle memcached, xprobes
will seem to hang on "command in progress, waiting...". This is
because whenever you see this message the action can't complete until
any of the probe sites (attached or detached) is actually hit. Here
we have only one site, in conn_close(), so we have to trigger that
function (connect with telnet, then disconnect). On a busy memcached
you won't notice the delay.
We can list probe modules to see that it is indeed there:
xprobes> modules
* /tmp/memcached/probe.so
xprobes>
(as asterisk before the name indicates that is is enabled), and we can
list module's probes:
xprobes> probes
/tmp/memcached/probe.so:
*_conn_release (const struct conn*) -> release
xprobes>
Repeating 'sites' command
xprobes> sites
./memcached:
* memcached_conn_release(const struct conn*) => *_conn_release -> release
[/tmp/memcached/probe.so]
xprobes>
we see that our probe has attached to the site in memcached server.
Now we can query the probe for what it has accumulated so far:
xprobes> command /tmp/memcached/probe.so protocol
command in progress, waiting...
ascii: 2
binary: 0
xprobes>
When we had enough fun, we can disable probe module, and unload it
(after some safety delay), 'help' command is your friend. Again,
don't forget to put some load onto memcached to keep hitting probe
sites.
To summarize, while XProbes serves the same purposes as DTrace,
SystemTap, and LTTng, its low overhead (few instructions for detached
probe site, simple shared library call for enabled probe site) and
native access to application data make it a tool of choice not only
for debugging and tracing the application during development, but also
for monitoring (and even extending!) live production systems.
More details on http://wiki.github.com/kroki/XProbes/
Play with it in your project, it's a lot of fun!
--
Tomash Brechko
