(Below is a note from Joydeep. Something about Joydeeps' messages are requiring that I approve/disapprove them. For the message below, I his disapprove by mistake so am copying it here manually)
---------- Forwarded message ---------- From: Joydeep Sarma <jssa...@apache.org> To: hbase-dev@hadoop.apache.org, kan...@facebook.com, Dhruba Borthakur < dhr...@facebook.com> Date: Tue, 12 Jan 2010 15:39:05 -0800 Subject: Re: commit semantics btw - i followed up with Dhruba afterwards on this comment: > any IO to a HDFS-file (appends, writes, etc) ae actually blocked on a > pending sync. "sync" in HDFS is a pretty heavyweight operation as it stands. i think this is likely to explain limited throughput with the default write queue threshold of 1. if the appends cannot make progress while one is waiting for the sync - then the write pipeline is going to be idle most of the time (with queue threshold of 1). i think it would be good to have the sync not block other writers on the file/pipeline. logically - it's not clear why it needs to (since the sync is just a wait for the completion as of some write transaction id - allowing new ones to be queued up subsequently). Joydeep
