Clint Morgan wrote:
So if we wrote all operations for a transaction first to ZooKeeper, we
still need something like a Distributed Transaction Manager to
orchestrate the commit process: Send BatchUpdates to each
RegionServer, ask them to commit, then commit or rollback based on
results from all participating RegionServers.
Yes.
Or is there some more
clever way to use ZooKeeper? Maybe encoding a commit protocol into the
Zookeeper nodes...
This page is interesting discussing how you can build various
cluster-wide primitives such as locks and two-phase commit using
zookeeper: http://zookeeper.wiki.sourceforge.net/ZooKeeperRecipes.
Still would need a transaction orchestrator of some sort.
Looks like google's datastore has a mechanism for keeping groups of
rows (entity groups) together on the same server (datastore node).
From
http://code.google.com/appengine/docs/datastore/keysandentitygroups.html:
"When the application creates an entity, it can assign another entity as
the parent of the new entity. Assigning a parent to a new entity puts
the new entity in the same entity group as the parent entity."
I think I need to sign up for app engine and use it to see if I can
figure how the above is done.
Then they allow transactions only on rows in the same group. This way
they don't have to worry about distributed transactions. Rather than
locking, they use optimistic concurrency control. This means they do
the transaction in a sandbox, then check for conflicts from other
transactions before committing.
We'd need to have HBASE-493 in place building any kind of OCC.
St.Ack
-clint
On Tue, May 27, 2008 at 2:13 PM, stack <[EMAIL PROTECTED]> wrote:
Clint Morgan wrote:
Zookeeper makes good sense for distributed locking to get isolation.
But we still need transaction start, commit, and rollback to get
atomicity. I think this properly belongs in hbase.
Since all clients are going via zookeeper anyways ('isolation'), maybe
it'd be better to just run the whole transaction management out of
zookeeper? Clients would open a transaction on zookeeper and put their
edits there so they were available for rollback and/or commit. If client
died midway, could ask zookeeper for outstanding transactions and pickup
whereever it'd left off. Otherwise, on success (or rollback), clean up
the transaction log.
Alternatively, all clients would have to go via the hbase master so it
could orchestrate row access. Master would need to hold outstanding
transactions somewhere either in an in-memory transactions catalog table
or itself over in zookeeper.
St.Ack
