While I applaud these experiments, the next challenge is getting them in to a shipping Hadoop. I think it's a relative nonstarter if we require someone to patch in a bunch of patches that are/were being refused to be committed.
Keep on experimenting and collecting that evidence though! One day! -ryan On Sat, Dec 4, 2010 at 3:04 PM, Todd Lipcon <[email protected]> wrote: > On Sat, Dec 4, 2010 at 2:57 PM, Vladimir Rodionov > <[email protected]>wrote: > >> From my own experiments performance difference is huge even on >> sequential R/W operations (up to 300%) when you do local File I/O vs HDFS >> File I/O >> >> Overhead of HDFS I/O is substantial to say the least. >> >> > Much of this is from checksumming, though - turn off checksums and you > should see about a 2x improvement at least. > > -Todd > > >> Best regards, >> Vladimir Rodionov >> Principal Platform Engineer >> Carrier IQ, www.carrieriq.com >> e-mail: [email protected] >> >> ________________________________________ >> From: Todd Lipcon [[email protected]] >> Sent: Saturday, December 04, 2010 12:30 PM >> To: [email protected] >> Subject: Re: Local sockets >> >> Hi Leen, >> >> Check out HDFS-347 for more info on this. I hope to pick this back up in >> 2011 - in 2010 we mostly focused on stability above performance in HBase's >> interactions with HDFS. >> >> Thanks >> -Todd >> >> On Sat, Dec 4, 2010 at 12:28 PM, Leen Toelen <[email protected]> wrote: >> >> > Hi, >> > >> > has anyone tested the performance impact (when there is a hdfs >> > datanode and a hbase node on the same machine) of using unix domain >> > sockets communication or shared memory ipc using nio? I guess this >> > should make a difference on reads? >> > >> > Regards, >> > Leen >> > >> >> >> >> -- >> Todd Lipcon >> Software Engineer, Cloudera >> > > > > -- > Todd Lipcon > Software Engineer, Cloudera >
