Thanks Flavio for the bookkeeper info. Good stuff. Well, I'm not sure if I want to introduce another complexity to HBase. So upcoming CDH3 will be my choice. But I didn't know about bookkeeper and I'll keep it for something else.
- Tatsuya On 05/17/2010, at 4:51 PM, Flavio Junqueira wrote: > Hi, Given the topic of this message, I'd like to point out that bookkeeper > (HBASE-2315) provides a strong durability guarantee. We sync writes to disk > on a quorum of machines. I don't think this feature is currently on the > roadmap of hbase, though. > > Thanks, > -Flavio > > On May 17, 2010, at 6:01 AM, Tatsuya Kawano wrote: > >> >> Hi, >> >> A few days ago, I had a discussion with other Japanese developers on >> hadoop-jp Google group. It was about HLog durability on the recent >> Hadoop releases (0.20.1, 0.20.2) I never looked at this issue closely >> until then as I was certain to use Hadoop 0.21 from the beginning. >> >> Someone showed us Todd's presentation at HUG March 2010, and we were >> all agreed that in order to solve this issue, we will need to use >> Hadoop trunk or Cloudera CDH3 including HDFS-200 and related patches. >> >> Then I came up with a couple of questions: >> >> 1. On Hadoop 0.20.x (without HDFS-200 patch), I must close HLog to >> make it's entries durable, right? While rolling HLog does this, how >> about region server failure? >> >> Someone in the discussion tried this senario. He killed (-9) a region >> server process after a few puts. The HLog was read by HMaster before >> it was closed. HMaster couldn't see any entry in the log and simply >> deleted it. So his lost some puts. >> >> Is this the expected behavior? He used Hadoop 0.20.1 and HBase 0.20.3. >> >> 2. On Hadoop trunk, I'd prefer not to hflush() every single put, but >> rely on un-flushed replicas on HDFS nodes, so I can avoid the >> performace penalty. Will this still durable? Will HMaster see un- >> flushed appends right after a region server failure? >> >> Thanks in advance, >> >> -- >> 河野 達也 >> Tatsuya Kawano (mr.) >> Tokyo, Japan >> >> >> >
