Thanks for the reply. We ran the TestDFSIO benchmark on both the development and production and found the production to be better.The statistics are shown below.
But once we bring HBase into the picture things gets reversed :-( The count operation,map-reduces etc becomes less performing on the production box.We are using Pseudo Distribution mode in both the development and production servers for both hadoop and hbase. *DEVELOPMENT SERVER* 11/05/15 21:26:26 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write 11/05/15 21:26:26 INFO fs.TestDFSIO: Date & time: Sun May 15 21:26:26 EDT 2011 11/05/15 21:26:26 INFO fs.TestDFSIO: Number of files: 10 11/05/15 21:26:26 INFO fs.TestDFSIO: Total MBytes processed: 10000 11/05/15 21:26:26 INFO fs.TestDFSIO: Throughput mb/sec: 58.09495038691237 11/05/15 21:26:26 INFO fs.TestDFSIO: Average IO rate mb/sec: 59.699485778808594 11/05/15 21:26:26 INFO fs.TestDFSIO: IO rate std deviation: 10.54547265175703 11/05/15 21:26:26 INFO fs.TestDFSIO: Test exec time sec: 163.354 11/05/15 21:26:26 INFO fs.TestDFSIO: 11/05/15 21:28:44 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read 11/05/15 21:28:44 INFO fs.TestDFSIO: Date & time: Sun May 15 21:28:44 EDT 2011 11/05/15 21:28:44 INFO fs.TestDFSIO: Number of files: 10 11/05/15 21:28:44 INFO fs.TestDFSIO: Total MBytes processed: 10000 11/05/15 21:28:44 INFO fs.TestDFSIO: Throughput mb/sec: 682.4075337791729 11/05/15 21:28:44 INFO fs.TestDFSIO: Average IO rate mb/sec: 755.5845947265625 11/05/15 21:28:44 INFO fs.TestDFSIO: IO rate std deviation: 229.60029445080488 11/05/15 21:28:44 INFO fs.TestDFSIO: Test exec time sec: 63.896 11/05/15 21:28:44 INFO fs.TestDFSIO: *PRODUCTION SERVER* 5/16 01:00:43 INFO fs.TestDFSIO: ----- TestDFSIO ----- : *WRITE PERFORMANCE* 11/05/16 01:00:43 INFO fs.TestDFSIO: Date & time: Mon May 16 01:00:43 GMT+00:00 2011 11/05/16 01:00:43 INFO fs.TestDFSIO: Number of files: 10 11/05/16 01:00:43 INFO fs.TestDFSIO: Total MBytes processed: 10000 11/05/16 01:00:43 INFO fs.TestDFSIO: Throughput mb/sec: 69.25447557048375 11/05/16 01:00:43 INFO fs.TestDFSIO: Average IO rate mb/sec: 70.06581115722656 11/05/16 01:00:43 INFO fs.TestDFSIO: IO rate std deviation: 7.243961483443693 11/05/16 01:00:43 INFO fs.TestDFSIO: Test exec time sec: 126.896 5/16 01:25:01 INFO fs.TestDFSIO: ----- TestDFSIO ----- : *READ PERFORMANCE* 11/05/16 01:25:01 INFO fs.TestDFSIO: Date & time: Mon May 16 01:25:01 GMT+00:00 2011 11/05/16 01:25:01 INFO fs.TestDFSIO: Number of files: 10 11/05/16 01:25:01 INFO fs.TestDFSIO: Total MBytes processed: 10000 11/05/16 01:25:01 INFO fs.TestDFSIO: Throughput mb/sec: 1487.20999405116 11/05/16 01:25:01 INFO fs.TestDFSIO: Average IO rate mb/sec: 1525.230712890625 11/05/16 01:25:01 INFO fs.TestDFSIO: IO rate std deviation: 239.54492784268226 11/05/16 01:25:01 INFO fs.TestDFSIO: Test exec time sec: 51.117 On Mon, May 16, 2011 at 2:23 PM, Jean-Daniel Cryans <[email protected]>wrote: > You are giving us the mile high overview of the problem, pointing to a > specific culprit could be very time consuming. Instead, can you run > some system tests and make sure things work the way they should? Are > the disks strangely slow? Any switches acting up? > > Regarding your CPUs, counting is mostly IO bound so I don't see how > that would change anything (which is why I ask about disks and > network). > > J-D > > On Mon, May 16, 2011 at 10:33 AM, Himanish Kushary <[email protected]> > wrote: > > Hi, > > > > We are in the process of moving a small Hbase/Hadoop cluster from our > > development to production environment.Our development environment were > few > > intel desktops (8 cores CPU/8 Gigs RAM/7200 rpm disks) running centOS > while > > the production environment are blades with (24 cores AMD CPU/32 gigs > > RAM/15000 rpm disks) AMD architecture running centOS. > > > > Strangely the hbase performance seems to degrade after moving stuffs to > the > > production enviroment (suppoesed to have more horse power).We are using > the > > latest and default installation for cloudera version of hadoop and > hbase.No > > changes to memory or other parameter were done on both the environment. > > > > Any idea what could cause this.Could the AMD architecture be the > > cause.Pointers to things to look for to improve performance in the > > production cluster would be really appreciated. > > > > Note: We ran "count" from hbase shell on a huge table and found the > desktops > > to be performing much better. We are in the process of comparing > Map-Reduces > > presently. > > > > --------------------------- > > Thanks & Regards > > Himanish > > > -- Thanks & Regards Himanish
