[
https://issues.apache.org/jira/browse/HBASE-5494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HBASE-5494:
-------------------------------
Attachment: D2997.3.patch
avf requested code review of "[jira] [HBASE-5494] [89-fb] Table-level locks for
schema changing operations.".
Reviewers: Kannan, mbautin, Liyin, JIRA
Since concurrent modification (e.g., disabling and dropping a table under
creation) could leave a cluster in an inconsistent state, we need table-level
locks for schema changing operations.
A ZooKeeper-based distributed lock has been implemented that
attempts to create a persistent ZNode (one ZNode per entity being locked,
i.e.,
one per table) if one does not exist. Currently in case a master crashes while
holding the lock, the lock must be manually removed using the ZooKeeper
command
line (locks being stored in "/hbase/tableLock/").
The locks implemented are not fair or re-entrant. RecoverableZooKeeper is used
to correctly handle connection loss.
To test the locks, InjectionHandler and InjectionEvent have been introduced,
allowing for injection of arbitrary events, in this case adding delays during
schema changing operations as to induce a race condition.
Future work involves automatically deleting stale lock ZNodes upon server
recovery (providing the attempted operations are not resumed), adding metrics
around locks (e.g., list all locks held).
TEST PLAN
Since concurrent modification (e.g., disabling and dropping a table
under creation) could leave a cluster in an inconsistent state, we
need table-level locks for schema changing operations.
A ZooKeeper-based distributed lock has been implemented that attempts
to create a persistent ZNode (one ZNode per entity being locked, i.e.,
one per table) if one does not exist. Currently in case a master
crashes while holding the lock, the lock must be manually removed
using the ZooKeeper command line (locks being stored in
"/hbase/tableLock/").
The locks implemented are not fair or re-entrant. RecoverableZooKeeper
is used to correctly handle connection loss.
To test the locks, InjectionHandler and InjectionEvent have been
introduced, allowing for injection of arbitrary events, in this case
adding delays during schema changing operations as to induce a race
condition.
Future work involves automatically deleting stale lock ZNodes upon
server recovery (providing the attempted operations are not resumed),
adding metrics around locks (e.g., list all locks held).
REVISION DETAIL
https://reviews.facebook.net/D2997
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/TableLockTimeoutException.java
src/main/java/org/apache/hadoop/hbase/master/HMaster.java
src/main/java/org/apache/hadoop/hbase/master/TableLockManager.java
src/main/java/org/apache/hadoop/hbase/util/InjectionEvent.java
src/main/java/org/apache/hadoop/hbase/util/InjectionHandler.java
src/main/java/org/apache/hadoop/hbase/zookeeper/DistributedLock.java
src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWrapper.java
src/test/java/org/apache/hadoop/hbase/master/TestSchemaModificationLocks.java
src/test/java/org/apache/hadoop/hbase/util/DelayInducingInjectionHandler.java
src/test/java/org/apache/hadoop/hbase/zookeeper/TestDistributedLock.java
> Introduce a zk hosted table-wide read/write lock so only one table operation
> at a time
> --------------------------------------------------------------------------------------
>
> Key: HBASE-5494
> URL: https://issues.apache.org/jira/browse/HBASE-5494
> Project: HBase
> Issue Type: Improvement
> Reporter: stack
> Attachments: D2997.3.patch
>
>
> I saw this facility over in the accumulo code base.
> Currently we just try to sort out the mess when splits come in during an
> online schema edit; somehow we figure we can figure all possible region
> transition combinations and make the right call.
> We could try and narrow the number of combinations by taking out a zk table
> lock when doing table operations.
> For example, on split or merge, we could take a read-only lock meaning the
> table can't be disabled while these are running.
> We could then take a write only lock if we want to ensure the table doesn't
> change while disabling or enabling process is happening.
> Shouldn't be too hard to add.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
