RC3 was promoted to 0.90 release, so I didn't expect much difference.
I also tried 0.90 build which also included newer patches such as HBASE-3455
and observed same result.
The cluster only runs one job at a time.
Client buffer size for our 0.20.6 cluster was set to 20MB and I didn't see
similar issue.
Here is output from top on one node:
top - 22:42:26 up 104 days, 20:38, 1 user, load average: 1.04, 1.34, 2.13
Tasks: 152 total, 1 running, 151 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.5%us, 0.2%sy, 0.0%ni, 98.8%id, 0.0%wa, 0.0%hi, 0.4%si,
0.0%st
Mem: 16440036k total, 15532616k used, 907420k free, 42472k buffers
Swap: 16777208k total, 5719488k used, 11057720k free, 4530368k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
15261 hadoop 25 0 4776m 1.0g 9312 S 2.0 6.7 133:27.04 java
16442 hadoop 21 0 1625m 760m 14m S 1.3 4.7 0:39.28
java
16453 hadoop 18 0 4690m 3.4g 9712 S 1.3 21.4 311:49.25 java
15261 DataNode
19442 map task
16453 HRegionServer
---------
[sjc1-hadoop3.sjc1:hadoop 1298]iostat
Linux 2.6.18-53.el5 (sjc1-hadoop3.sjc1.carrieriq.com) 01/30/2011
avg-cpu: %user %nice %system %iowait %steal %idle
4.56 0.00 0.48 0.60 0.00 94.35
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 6.15 403.62 411.58 3656841492 3728977064
sdb 7.64 131.10 303.53 1187817806 2750029496
sdc 7.38 396.23 387.12 3589886886 3507388096
sdd 6.35 403.99 399.77 3660260454 3621972144
dm-0 22.51 37.78 173.52 342296914 1572156872
dm-1 27.92 93.32 130.01 845519592 1177872624
On Sun, Jan 30, 2011 at 2:34 PM, Stack <[email protected]> wrote:
> On Sun, Jan 30, 2011 at 3:41 PM, Ted Yu <[email protected]> wrote:
> > Using 0.90 RC3 on our 7 node clsuter, the first time (after cluster
> start)
> > our map tasks ran quite fast writing to hbase (client side buffer size
> being
> > 17MB).
> >
>
> Can you come up on release 0.90.0 Ted?
>
> 17MB client buffer might be a little large.
>
>
> > If I run the same hadoop job soon after the first finishes, there is
> > noticeable slow down.
> >
>
> What do you see in your ganglia? Anything else running on cluster at
> the time? What are the machines doing? CPU, iowait? Swapping?
>
> Thanks,
> St.Ack
>
