Stack & Enis: Your responses clear my thoughts. Thanks.
On Wed, May 29, 2013 at 6:19 AM, Enis Söztutar <[email protected]> wrote: > Hi, > > HDFS has two interfaces for durability: hflush and hsync: > > Hflush() : Flush the data packet down the datanode pipeline. Wait for > ack’s. > Hsync() : Flush the data packet down the pipeline. Have datanodes execute > FSYNC equivalent. Wait for ack’s. > > There is some work on adding a Durability API in HBase: see HBASE-7801 and > HBASE-8375. > > However, as Stack mentioned, without HBASE-5954 is fixed, HBase right now > cannot make use of the hsync() API. I want to rebase the patch in > HBASE-5954, but it might take some more time. > > The good news is that although not perfect, hflush, which is current > default makes sure that the update is send to 3 replicas, so unless there > is a data center power failure or similar, the data will make into the > disks pretty quickly. > > Hope this helps. > Enis > > > > > On Tue, May 28, 2013 at 9:53 AM, Stack <[email protected]> wrote: > > > On Tue, May 28, 2013 at 7:09 AM, jingguo yao <[email protected]> > wrote: > > > > > Section 2.1.3 says that Hadoop 1.0.4 works with HBase-0.94.x [1]. And > > > Section 2.1.3.3 says that 1.0.4 has a working durable sync. But when I > > > check the source code of DFSClient.DFSOutputStream's sync method, I > > > finds the following javadoc: > > > > > > /** > > > * All data is written out to datanodes. It is not guaranteed > > > * that data has been flushed to persistent store on the > > > * datanode. Block allocations are persisted on namenode. > > > */ > > > > > > So it seems that sync does not support a durable sync. It contradicts > > > with [1]. > > > > > > Can anybody help me on this confusion? Thanks. > > > > > > > > This issue is probably the best source for the state of sync in hbase > (and > > hdfs): https://issues.apache.org/jira/browse/HBASE-5954 > > > > In short, the refguide is misleading -- let me fix -- as 1.0.4 indeed > has a > > sync but it is just a sync to the memory of three datanodes, not a true > > fsync out to disk. The above cited issue is tracking issues that our > Lars > > and other have contributed to HDFS to add fsync support. > > > > Yours, > > St.Ack > > > -- Jingguo
