Upcoming is also interested in this sort of stuff. :)
-Gordon
john allspaw wrote:
Please do share the love. Lots of interest here in flickr-land.
-j
----- Original Message ----
From: Randy Wigginton <[EMAIL PROTECTED]>
To: [email protected]
Sent: Thursday, October 11, 2007 2:16:01 PM
Subject: MOM: Memcached Operations Monitoring
Hi All,
My company does something unusual with memcached that is extremely
valuable to us, and I'm wondering if others would find the code
useful. I apologize if this email is long, but what I'm proposing
requires some background explanation, as it is radically different
from the typical use of memcached. The ultimate question is whether
it is worthwhile to make the code available for others to use.
One problem with running a large site (hundreds of millions of hits
per hour) is keeping track of what is going on. At one point I
worked on the Ebay swat team, and I would get calls at 2 in the
morning from the operations center, wondering why a set of machines
was acting up. Without instrumentation, it is nearly impossible to
figure out. With the proper instrumentation, it is child's play.
Ebay has a VERY large system to track all activity on the site;
however, that system is VERY large and VERY expensive. Using
memcached, I've developed something that gives you 90% of the value
of Ebay's system for perhaps 1% of the cost.
I have modified memcached as well as the java client library; with
these modifications, and very few lines of code in the application, I
can tell precisely how many URLs and SQLs are executing on a
particular machine in any given minute or in any given hour. I can
tell you the average execution time, the maximum execution time, as
well as the number of failures. I can tell you which URLs were
expensive, which URLs invoked SQL statements, which urls failed most
often. Coupled with a small mysql database, I can give you more
operational statistics on our site than many larger sites have
available.
Just to reassure those who are assuming this must be a very expensive
use of memcached, I can say from experience that with a single
instance running on a linux box with a mere 10M of memory assigned,
we aggregate information on about 20 pools and several hundred
machines at a rate of 10-15K operations per second, and have never
gotten close to capacity.
Would anyone else be interested in this? Or is this too far off the
beaten path? It is mostly helpful for very busy sites. Thanks.
--randy
____________________________________________________________________________________
Boardwalk for $500? In 2007? Ha! Play Monopoly Here and Now (it's updated for
today's economy) at Yahoo! Games.
http://get.games.yahoo.com/proddesc?gamekey=monopolyherenow
