On Tue, Jan 12, 2010 at 9:12 PM, stack <st...@duboce.net> wrote: > > any IO to a HDFS-file (appends, writes, etc) ae actually blocked on a > > pending sync. "sync" in HDFS is a pretty heavyweight operation as it > stands. > > i think this is likely to explain limited throughput with the default > write queue threshold of 1. if the appends cannot make progress while > one is waiting for the sync - then the write pipeline is going to be > idle most of the time (with queue threshold of 1). > > i think it would be good to have the sync not block other writers on > the file/pipeline. logically - it's not clear why it needs to (since > the sync is just a wait for the completion as of some write > transaction id - allowing new ones to be queued up subsequently).
Are you talking about internal to DFSClient Joydeep? Or some synchronization block up in hlog? St.Ack
