Thanks Todd. Yes, the stress test is NN-only. The simulated datanodes (using MiniDFSCluster) don't read or write actual data, only log the metadata.
So, it sounds like the slowdown on the NN is to be expected, correct? The race condition I was experiencing before is no longer happening, so the benefit of correct locking has resulted in an acceptable slowdown on the namenode. Is that correct? Thanks, -Eric > -----Original Message----- > From: Todd Lipcon [mailto:t...@cloudera.com] > Sent: Friday, July 01, 2011 7:49 PM > To: hdfs-dev@hadoop.apache.org > Subject: Re: HDFS on trunk is now quite slow > > My guess is HDFS-988 caused the slowdown by coarsening some locking that > was > previously incorrect. Your stress test is NN-only (metadata ops), not an > I/O > benchmark, right? I/O should be faster in trunk than ever before. > > -Todd > > On Fri, Jul 1, 2011 at 8:23 AM, Eric Payne <eric.payne1...@yahoo.com> > wrote: > > > Hi gang, > > > > I ran some stress tests on the latest HDFS trunk yesterday, and the > > performance > > is a lot slower (sometimes 10 times slower) when compared with the HDFS > in > > MR-279. The HDFS in MR-279 is slightly behind trunk. The stability of > > HDFS trunk > > seems to be better than HDFS MR-279, but I'm not sure if the slowness is > > just > > avoiding the race contitions or if they are actually fixed in trunk. > > > > At this point, I'm not sure what is causing this performance disparity. > I > > notice > > that Block management has recently undergone significant changes in > trunk. > > It > > has some new locking and it is now in its own package. Could this be > part > > of the > > cause? > > > > Thanks, > > -Eric > > > > > -- > Todd Lipcon > Software Engineer, Cloudera
