[
https://issues.apache.org/jira/browse/HBASE-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397556#comment-13397556
]
nkeywal commented on HBASE-6156:
--------------------------------
Small status as of June.
* Improvements identified
Failure detection time: performed by ZK, with a timeout. With the default
value, we needed 90 seconds before starting to act on a software or hardware
issue.
Recovery time - server side: split in two parts: reassigning the regions of a
dead RS to a new RS, replaying the WAL. Must be as fast as possible.
Recovery time - client side: errors should be transparent for the user code.
On the client side, we must as well limit the time lost on errors to a minimum.
Planned rolling restart: just make this as fast and less disruptive as
possible
Other possible changes. detailed below.
* Status
Failure detection time: software crash - done
Done in HBASE-5844, HBASE-5926
Failure detection time: hardware issue - not started
1) as much as possible, it should be handled by ZooKeeper and not HBase, see
open Jira as ZOOKEEPER-702, ZOOKEEPER-922, ...
2) we need to make easy for a monitoring tool to tag a RS or Master as dead.
This way, specialized HW tools could point out dead RS. Jira to open.
Recovery time - Server: in progress
1) bulk assignment: To be retested, there are many just-closed JIRA on this
(HBASE-5998, HBASE-6109, HBASE-5970, ...). A lot of work by many people. There
are still possible improvements (HBASE-6058, ...)
2) Log replay: To be retested, there are many just-closed JIRA on this
(HBASE-6134, ...).
Recovery time - Client - done
1) The RS now returns the new RS to the client after a region move
(HBASE-5992, HBASE-5877)
2) Client retries sooner on errors (HBASE-5924).
3) In the future, it could be interesting to share the region location in ZK
with the client. It's not reasonable today as it could lead to have too many
connection to ZK. ZOOKEEPER-1147 is an open JIRA on this.
Planned rolling restart performances - in progress
Benefits from the modifications in the client mentioned above.
To do: analyze move performances to make it faster if possible.
Other possible changes
- Restart the server immediately on software crash: done in HBASE-5939
- Reuse the same assignment on software crash: not planned
- Use spare hardware to reuse the same assignment on hardware failure: not
planned
- Multiple RS for the same region (excluded in the initial document: hbase
architecture change previously discussed by Gary H./Andy P.): not planned
> Improve multiop performances in HTable#flushCommits
> ---------------------------------------------------
>
> Key: HBASE-6156
> URL: https://issues.apache.org/jira/browse/HBASE-6156
> Project: HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.96.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Minor
> Fix For: 0.96.0
>
>
> This code:
> {noformat}
> @Override
> public void flushCommits() throws IOException {
> try {
> Object[] results = new Object[writeBuffer.size()];
> try {
> this.connection.processBatch(writeBuffer, tableName, pool, results);
> } catch (InterruptedException e) {
> throw new IOException(e);
> } finally {
> // mutate list so that it is empty for complete success, or contains
> // only failed records results are returned in the same order as the
> // requests in list walk the list backwards, so we can remove from
> list
> // without impacting the indexes of earlier members
> for (int i = results.length - 1; i>=0; i--) {
> if (results[i] instanceof Result) {
> // successful Puts are removed from the list here.
> writeBuffer.remove(i);
> }
> }
> }
> } finally {
> if (clearBufferOnFail) {
> writeBuffer.clear();
> currentWriteBufferSize = 0;
> } else {
> // the write buffer was adjusted by processBatchOfPuts
> currentWriteBufferSize = 0;
> for (Put aPut : writeBuffer) {
> currentWriteBufferSize += aPut.heapSize();
> }
> }
> }
> }
> {noformat}
> Can be improved by:
> - not iterating on the list if clearBufferOnFail is set
> - not iterating the the list of there are no error
> - iterating on the list only once instead of two when we really have to.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
