Awesome, I will try to post a patch soon and will let you know as soon as I have the first version ready.
thanks, dhruba On Wed, Jan 13, 2010 at 10:40 AM, Jean-Daniel Cryans <jdcry...@apache.org>wrote: > I'll be happy to benchmark, we already have code to test the > multi-client hitting 1 region server case. > know > J-D > > On Wed, Jan 13, 2010 at 10:38 AM, Dhruba Borthakur <dhr...@gmail.com> > wrote: > > I will try to make a patch for it first. depending on the complexity of > the > > patch code, we can decide which release it can go in. > > > > thanks, > > dhruba > > > > On Wed, Jan 13, 2010 at 9:56 AM, Jean-Daniel Cryans <jdcry...@apache.org > >wrote: > > > >> That's great dhruba, I guess the sooner it could go in is 0.21.1? > >> > >> J-D > >> > >> On Wed, Jan 13, 2010 at 8:51 AM, Dhruba Borthakur <dhr...@gmail.com> > >> wrote: > >> > I opened http://issues.apache.org/jira/browse/HDFS-895 for this one. > >> > > >> > thanks, > >> > dhruba > >> > > >> > On Tue, Jan 12, 2010 at 9:41 PM, Joydeep Sarma <jsensa...@gmail.com> > >> wrote: > >> > > >> >> this is internal to the dfsclient. this would explain why performance > >> >> would suck with queue threshold of 1. > >> >> > >> >> leave it up to Dhruba to explain the details. > >> >> > >> >> On Tue, Jan 12, 2010 at 9:16 PM, stack <st...@duboce.net> wrote: > >> >> > On Tue, Jan 12, 2010 at 9:12 PM, stack <st...@duboce.net> wrote: > >> >> > > >> >> >> > any IO to a HDFS-file (appends, writes, etc) ae actually blocked > on > >> a > >> >> >> > pending sync. "sync" in HDFS is a pretty heavyweight operation > as > >> it > >> >> >> stands. > >> >> >> > >> >> >> i think this is likely to explain limited throughput with the > default > >> >> >> write queue threshold of 1. if the appends cannot make progress > while > >> >> >> one is waiting for the sync - then the write pipeline is going to > be > >> >> >> idle most of the time (with queue threshold of 1). > >> >> >> > >> >> >> i think it would be good to have the sync not block other writers > on > >> >> >> the file/pipeline. logically - it's not clear why it needs to > (since > >> >> >> the sync is just a wait for the completion as of some write > >> >> >> transaction id - allowing new ones to be queued up subsequently). > >> >> > > >> >> > > >> >> > Are you talking about internal to DFSClient Joydeep? Or some > >> >> > synchronization block up in hlog? > >> >> > > >> >> > St.Ack > >> >> > > >> >> > >> > > >> > > >> > > >> > -- > >> > Connect to me at http://www.facebook.com/dhruba > >> > > >> > > > > > > > > -- > > Connect to me at http://www.facebook.com/dhruba > > > -- Connect to me at http://www.facebook.com/dhruba
