On Mon, Oct 11, 2010 at 5:12 PM, Sean Bigdatafun <[email protected]> wrote: > Can someone give me a detailed look at the HLog mechanism for 0.90 > durablity? > I recall that HBase committers claim that data will be truly durable in 0.90 > after the client gets 'ok' acknowledgement from server, while it was not > true in 0.20 (i.e., HBase may have the chance to lose the data even it says > 'ok, I got it' to clients). Can someone give me some detailed info? >
The issue was that HDFS did not support appends. It would only retain data that was in a file that had been properly closed. If a regionserver crash while we were appending to the WAL/HLog -- i.e. the crash would make it so the writer did not properly close the WAL/HLog -- we'd lose effectively all edits that had been applied to the WAL/HLog up to the time of the crash. > First, HLog can't seem to be flushed onto disk for each Put operation for > performance reason (i.e., there exists time window when the log of a certain > amout of hlog reside in the RegionServer's memory). If this understanding is > correct, how are we going to assure total durablity? > You can have total durability if you want. You will pay in slower performance -- as it is for any database. St.Ack
