On Apr 30, 2009, at 6:52 PM, MARK CALLAGHAN wrote:
On Thu, Apr 30, 2009 at 6:35 PM, Clint Byrum <[email protected]> wrote:
On Apr 30, 2009, at 5:43 PM, MARK CALLAGHAN wrote:
Help me to understand what it is your proposing here..
* Connect to a server with a hint that you're readonly, and that
the last
change you made to the DB was at global transaction ID X
* Server informs you of its highest global transaction ID and you
decide
whether or not to stay connected?
This sounds like a job for a proxy. It would be *awesome* if this
were
always in the protocol, and the proxy would just improve things by
helping
to find an acceptable server w/o requiring the user to connect over
and
over.
Once that is working, adding additional constraints based on QoS and
server health should be easy.
I want a client to connect to a db server name, have all writes get
done on primary and have queries redirected to a slave when possible
where redirected queries are guaranteed to see any changes that client
committed on a primary.
I wrote my response to the proxying part below first, but then I re-
read this paragraph above, and recalled a problem I've had in the past.
"Have queries redirected to a slave when possible"
We had a system at (little, tiny) Adicio, much like, I'm sure, other
places have. The user's session gets marked as "dirty" when writes
happen. Basically for about 5 minutes after they change anything, they
are on the master. Then after 5 minutes, the slaves that are within 5
minutes are used.
The problem arises where the master gets *really* busy with writes,
and so the slaves get far, far, behind briefly. Our system has a
quorum that basically says "if less than half of the slaves are within
5 minutes, just send queries to them all." So in these instances,
users start seeing old data and getting confused until they change
things (at which point, they start seeing a lot of spinning
browsers). ;) We abandoned this mode of operation in favor of finding
instances where reads can always go to the slaves, or be pushed out to
memcached/memcachedb, and setting business expectations for realtime
to 'near-realtime'.
However I've heard of other systems that handle things differently.
When the slaves get too far behind, the reads *all* go to the master.
cowsay FAIL.
It strikes me that when one tries to be opportunistic about load
distribution, they might as well not do it. The time we really need
more capacity is when we're stressed. So it makes me think that a big
focus should be on making replication as fast and reliable as
possible. This, to me, means being able to break it up into smaller
pieces and not have it be single threaded or even single hosted
(slaves shard while master's don't.. crazy? well.. thats kind of what
memcached does).
I don't want the proxy to be a separate set of processes as I already
have enough of those to manage and failure of standalone proxy
processes would not be good for a highly available database service.
I think from a latency and complexity standpoint the server that you
sent the request to should get that result for you. If it works akin
to HTTP's 301/302, that might mean making a new connection or (better)
grabbing one out of a pool of connections.
This whole issue reminds me of mod_backhand. If you don't recall.. it
was (I say was because I don't think people use it anymore, but maybe
I'm wrong.. ;) a module for apache that would broadcast host stats and
then redirect, or proxy requests to other nodes based on those
stats.It was *really* effective at keeping resource utilization
distributed. This is where redirects would actually be handy, because
I know for us, mod_backhand failed us when we had too many individual
users accessing one resource, causing us to have to keep a lot of
proxy connections open on the mod_backhand machine itself. When we put
a load balancer in front of that, there was no more need for
mod_backhand.
Anyway, memcached rather than broadcast, wouldn't be a bad idea to
publish each slave's transaction ID.
I could totally see the replication slave process simply setting a
memcache item, something like 'transid_slave1', every time it commits
a transaction to that slave. The client libraries could then just keep
reasonably good track of that value and send queries directly without
the server protocol doing any more than giving back that transaction
id whenever it changes. This is where ranged key fetching would be
nice in memcached.. as you could just do something like memcachedb's
rget..
rget transid_ some_sane_endkey 0 0 100
Which would get up to 100 of them quickly and you could then pick the
best one. You could keep that list cached locally right up until your
thread changes its transactionid. Connections would still have to be
checked and discarded in case the information was inaccurate, or maybe
not, but either way, its basically just a hint as to which servers can
serve your request.
Maybe the master could manage all of this.. however.. I get a little
hesitant when we ask masters to keep track of slaves, knowing what a
mess server-id has been in the past. Seems much better to just let the
slaves keep track of themselves and make that information highly
available.
Still I see the point that if you let the users get away with ignoring
those hints, you're right back at "damnit use the slaves!". So, if all
of the logic above happens in a plugin in the master, the only thing
really required in the protocol is global transaction ID, and the
redirect response. However, I'm still troubled by "opportunistic load
distribution".. but .. my world is tiny, so maybe I just don't get
it. ;)
_______________________________________________
Mailing list: https://launchpad.net/~drizzle-discuss
Post to : [email protected]
Unsubscribe : https://launchpad.net/~drizzle-discuss
More help : https://help.launchpad.net/ListHelp
