Answers inline. > -----Original Message----- > From: Jun Li [mailto:[email protected]] > Sent: Wednesday, July 14, 2010 10:15 AM > To: [email protected] > Subject: write-ahead log and failure recovery for region servers > > Hi, > > I am in the middle of performance testing on my service using HBase as > the backend, and I have difficulty to explain the write and read > performance difference, which bring up the following questions. > > (1) write-ahead log > > In the default mode of HTable "put", does the write-ahead log get turn > on? If not, what is the API that allows me to turn on the write-ahead > log? Without the write-ahead logging turned on, is it true that when > the region server dies, all the data that has been written by the > client but not flushed to the persistent HDFS will get lost?
By default, WAL is on. > So far I have not seen any documentation on the HBase configuration > parameters to specify where the write-ahead log is store. Does it get > stored in a reliable shared file system that the system administrator > needs to provide, for example, a SAN? Or HBase uses the HDFS as the > reliable shared file system? Previously I read some articles saying > that HBase is not using (or did not use) HDFS for write-ahead logging, > as the operation of "append" provided from HDFS is not robust enough. The WAL is stored in HDFS. In previous HDFS versions, there was no append/sync support, so this meant that you would lose some (not all) edits under hard node failure. The next release of HBase (0.90 and 0.89 dev releases) are designed to work with HDFS versions that support appends (Hadoop 20 append branch, Cloudera CDHv3, etc). For more info on this, check out the slides on this post: http://hbaseblog.com/2010/07/04/hug11-hbase-0-90-preview-wrap-up/ > (2) Failure recovery > > Related to the write-ahead log problem that I brought up, could you > explain to me the current mechanism adopted in HBase, when a region > server fails and the client is making a request to fetch a row to this > particular region server? My understanding is that the master node > will detect that the region server fails and then starts to migrate > the data (including the write-ahead log) that is served by that region > to the other region and update the meta data of the master node after > the migration finishes. This is correct. The region will be opened by another server and the master will coordinate replaying the WAL log. > The concern that I have is that such migration takes quite some time, > depending on the data size of the failed region server. For large > size, it will take, say couple of minutes. During this time window, > will it be the case that all the clients that try to access data rows > on that failed region server will all fail? As a result, the HBase > client will need to have a retry window that matches the region data's > migration time. Basically, yes. Recovery can take some time. There are a number of active JIRAs related to performance improvements for log replay (distributed log replay, adding more concurrency, etc...) Search around JIRA if interested. > > Thank you very much for your information in advance! > > Jun Good luck! JG
