this is internal to the dfsclient. this would explain why performance would suck with queue threshold of 1.
leave it up to Dhruba to explain the details. On Tue, Jan 12, 2010 at 9:16 PM, stack <st...@duboce.net> wrote: > On Tue, Jan 12, 2010 at 9:12 PM, stack <st...@duboce.net> wrote: > >> > any IO to a HDFS-file (appends, writes, etc) ae actually blocked on a >> > pending sync. "sync" in HDFS is a pretty heavyweight operation as it >> stands. >> >> i think this is likely to explain limited throughput with the default >> write queue threshold of 1. if the appends cannot make progress while >> one is waiting for the sync - then the write pipeline is going to be >> idle most of the time (with queue threshold of 1). >> >> i think it would be good to have the sync not block other writers on >> the file/pipeline. logically - it's not clear why it needs to (since >> the sync is just a wait for the completion as of some write >> transaction id - allowing new ones to be queued up subsequently). > > > Are you talking about internal to DFSClient Joydeep? Or some > synchronization block up in hlog? > > St.Ack >
