I opened http://issues.apache.org/jira/browse/HDFS-895 for this one.
thanks, dhruba On Tue, Jan 12, 2010 at 9:41 PM, Joydeep Sarma <jsensa...@gmail.com> wrote: > this is internal to the dfsclient. this would explain why performance > would suck with queue threshold of 1. > > leave it up to Dhruba to explain the details. > > On Tue, Jan 12, 2010 at 9:16 PM, stack <st...@duboce.net> wrote: > > On Tue, Jan 12, 2010 at 9:12 PM, stack <st...@duboce.net> wrote: > > > >> > any IO to a HDFS-file (appends, writes, etc) ae actually blocked on a > >> > pending sync. "sync" in HDFS is a pretty heavyweight operation as it > >> stands. > >> > >> i think this is likely to explain limited throughput with the default > >> write queue threshold of 1. if the appends cannot make progress while > >> one is waiting for the sync - then the write pipeline is going to be > >> idle most of the time (with queue threshold of 1). > >> > >> i think it would be good to have the sync not block other writers on > >> the file/pipeline. logically - it's not clear why it needs to (since > >> the sync is just a wait for the completion as of some write > >> transaction id - allowing new ones to be queued up subsequently). > > > > > > Are you talking about internal to DFSClient Joydeep? Or some > > synchronization block up in hlog? > > > > St.Ack > > > -- Connect to me at http://www.facebook.com/dhruba
