I was curious because when a vendor (big storage company) presented they were offering a hadoop solution. They posted IOPS and I wasn't sure how they were determining this number....
On Tue, Oct 23, 2012 at 9:19 AM, Michael Segel <michael_se...@hotmail.com>wrote: > You have two issues. > > 1) You need to know the throughput in terms of data transfer between disks > and controller cards on the node. > > 2) The actual network throughput of having all of the nodes talking to one > another as fast as they can. This will let you see your real limitations in > the ToR Switch's fabric. > > Not sure why you really want to do this except to test the disk, disk > controller, and then networking infrastructure of your ToR and then your > backplane to connect multiple racks.... > > > HTH > > -Mike > > On Oct 23, 2012, at 7:47 AM, Ravi Prakash <ravi...@ymail.com> wrote: > > > Do you mean in a cluster being used by users, or as a benchmark to > measure the maximum? > > > > The JMX page <nn:port>/jmx provides some interesting stats, but I'm not > sure they have what you want. And I'm unaware of other tools which could. > > > > > > > > > > > > ________________________________ > > From: Rita <rmorgan...@gmail.com> > > To: common-user@hadoop.apache.org; Ravi Prakash <ravi...@ymail.com> > > Sent: Monday, October 22, 2012 6:46 PM > > Subject: Re: measuring iops > > > > Is it possible to know how many reads and writes are occurring thru the > > entire cluster in a consolidated manner -- this does not include > > replication factors. > > > > > > On Mon, Oct 22, 2012 at 10:28 AM, Ravi Prakash <ravi...@ymail.com> > wrote: > > > >> Hi Rita, > >> > >> SliveTest can help you measure the number of reads / writes / deletes / > ls > >> / appends per second your NameNode can handle. > >> > >> DFSIO can be used to help you measure the amount of throughput. > >> > >> Both these tests are actually very flexible and have a plethora of > options > >> to help you test different facets of performance. In my experience, you > >> actually have to be very careful and understand what the tests are doing > >> for the results to be sensible. > >> > >> HTH > >> Ravi > >> > >> > >> > >> > >> ________________________________ > >> From: Rita <rmorgan...@gmail.com> > >> To: "<common-user@hadoop.apache.org>" <common-user@hadoop.apache.org> > >> Sent: Monday, October 22, 2012 7:23 AM > >> Subject: Re: measuring iops > >> > >> Anyone? > >> > >> > >> On Sun, Oct 21, 2012 at 8:30 AM, Rita <rmorgan...@gmail.com> wrote: > >> > >>> Hi, > >>> > >>> Was curious if there was a method to measure the total number of IOPS > >> (I/O > >>> operations per second) on a HDFS cluster. > >>> > >>> > >>> > >>> -- > >>> --- Get your facts first, then you can distort them as you please.-- > >>> > >> > >> > >> > >> -- > >> --- Get your facts first, then you can distort them as you please.-- > >> > > > > > > > > -- > > --- Get your facts first, then you can distort them as you please.-- > > -- --- Get your facts first, then you can distort them as you please.--
