[
https://issues.apache.org/jira/browse/HELIX-54?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721284#comment-13721284
]
dafu commented on HELIX-54:
---------------------------
Current helix admin api's are not atomic: i) no sync on invoking from multiple
processes, and ii) not handling zk session expiry (due to java gc).
To improve it, we have two options:
a) Making use of zookeeper transaction:
http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction()
This requires upgrade zookeeper jar to 3.4.0+ and rewrite all helix admin api's
using zookeeper#transaction()
b) Implement distributed lock
This is a more general solution but has some limitations. The basic idea is:
1) the 1st process creates a permanent znode representing holding the lock; all
other processes wait on the removal of the znode
2) the 1st process performs the helix operation
3) the 1st process removes the znode.
4) all other processes get notified and proceed
Note that we can't use an ephemeral znode, since if the process creating the
ephemeral znode experiences a long gc, the ephemeral node will be removed. some
other process will create a new ephemeral node and grabs the lock. at this
time, if the formal process recovers from gc, it would think it has the lock
also.
Based on distributed-lock, all helix admin api that needs sync on multiple
processes could use the following idiom:
DistributedLock lock = new DistributedLock(String namespace);
if (lock.tryLock(timeout-value, time-unit) // wait until locking successfully
or timeout
{
if (zookeeper content shows operation not successfully performed) {
// clean up if necessary
// call helix admin api
}
} finally {
lock.unlock();
}
if (zookeeper content shows operation not successfully performed) {
// fail permanently
}
a) Handling multi-process: the helix admin operation is sync on the lock, only
the 1st process can grab the lock and perform the admin operation.
b) Handling zk session expiry (due to java gc): if the 1st process holds the
lock and gets a zk session expiry, then all other processes will wait on the
lock until timeout. If zk session expiry recovers before timeout, the 1st
process will continue finish the helix operation, otherwise fails permanently.
Note that the distributed-lock solution is not fault-tolerant. if the 1st
process crashes or experience a long gc, the helix admin operation will fail
permanently. if this happens, we also need a way to clean orphan znodes.
> make helix-admin operations atomic
> ----------------------------------
>
> Key: HELIX-54
> URL: https://issues.apache.org/jira/browse/HELIX-54
> Project: Apache Helix
> Issue Type: Bug
> Affects Versions: 0.6.0-incubating
> Environment: make helix-admin operations e.g. addCluster()
> multi-process safe.
> Reporter: dafu
> Assignee: dafu
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
