How much should I give the region servers? That machine is already overallocated, by which I mean that the sum of the max heap sizes of all java processes running there is greater than the amount physical memory, which can lead to swapping. We have: Hadoop data node, Hadoop task tracker, ZooKeeper peer, and region server. The machine has 8G of physical memory. The region server currently has a max heap size of 4G. Should I increase to 6G? Should I decrease the block cache back down to 20% or even lower? Do we need to move to a 16G server?
Thanks, James On Tue, 2010-02-16 at 00:48 -0600, Dan Washusen wrote: > 32% IO on region server 3! Ouch! :) > > Increasing the block cache to 40% of VM memory without upping the total > available memory may only exacerbated the issue. I notice that region > server 2 was already using 3300mb of the 4000mb heap. By increasing the > block cache size to 40% you have now given the block cache 1600mb compared > to the previous 800mb... > > Can you give the region servers more memory? > > Cheers, > Dan > > On 16 February 2010 17:42, James Baldassari <ja...@dataxu.com> wrote: > > > On Tue, 2010-02-16 at 00:14 -0600, Stack wrote: > > > On Mon, Feb 15, 2010 at 10:05 PM, James Baldassari <ja...@dataxu.com> > > wrote: > > > > Applying HBASE-2180 isn't really an option at this > > > > time because we've been told to stick with the Cloudera distro. > > > > > > I'm sure the wouldn't mind (smile). Seems to about double throughput. > > > > Hmm, well I might be able to convince them ;) > > > > > > > > > > > > If I had to guess, I would say the performance issues start to happen > > > > around the time the region servers hit max heap size, which occurs > > > > within minutes of exposing the app to live traffic. Could GC be > > killing > > > > us? We use the concurrent collector as suggested. I saw on the > > > > performance page some mention of limiting the size of the new > > generation > > > > like -XX:NewSize=6m -XX:MaxNewSize=6m. Is that worth trying? > > > > > > Enable GC logging for a while? See hbase-env.sh. Uncomment this line: > > > > > > # export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails > > > XX:+PrintGCDateStamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log" > > > > I did uncomment that line, but I can't figure out where the gc-hbase.log > > is. It's not with the other logs. When starting HBase the GC output > > seems to be going to stdout rather than the file. Maybe a Cloudera > > thing. I'll do some digging. > > > > > > > > You are using recent JVM? 1.6.0_10 or greater? 1.6.0_18 might have > > issues. > > > > We're on 1.6.0_16 at the moment. > > > > > > > > Whats CPU and iowait or wa in top look like on these machines, > > > particularly the loaded machine? > > > > > > How many disks in the machines? > > > > I'll have to ask our ops guys about the disks. The high load has now > > switched from region server 1 to 3. I just saw in our logs that it took > > 139383.065 milliseconds to do 5000 gets, ~36 gets/second, ouch. Here > > are the highlights from top for each region server: > > > > Region Server 1: > > top - 01:39:41 up 4 days, 13:44, 4 users, load average: 1.89, 0.99, 1.19 > > Tasks: 194 total, 1 running, 193 sleeping, 0 stopped, 0 zombie > > Cpu(s): 15.6%us, 5.8%sy, 0.0%ni, 76.9%id, 0.0%wa, 0.1%hi, 1.6%si, > > 0.0%st > > Mem: 8166588k total, 8112812k used, 53776k free, 8832k buffers > > Swap: 1052248k total, 152k used, 1052096k free, 2831076k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 21961 hadoop 19 0 4830m 4.2g 10m S 114.3 53.6 37:26.58 java > > 21618 hadoop 21 0 4643m 578m 9804 S 66.1 7.3 19:06.89 java > > > > Region Server 2: > > top - 01:40:28 up 4 days, 13:43, 4 users, load average: 3.93, 2.17, 1.39 > > Tasks: 194 total, 1 running, 193 sleeping, 0 stopped, 0 zombie > > Cpu(s): 11.3%us, 3.1%sy, 0.0%ni, 83.4%id, 1.2%wa, 0.1%hi, 0.9%si, > > 0.0%st > > Mem: 8166588k total, 7971572k used, 195016k free, 34972k buffers > > Swap: 1052248k total, 152k used, 1052096k free, 2944712k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 15752 hadoop 18 0 4742m 4.1g 10m S 210.6 53.1 41:52.80 java > > 15405 hadoop 20 0 4660m 317m 9800 S 114.0 4.0 27:34.17 java > > > > Region Server 3: > > top - 01:40:35 up 2 days, 9:04, 4 users, load average: 10.15, 11.05, > > 11.79 > > Tasks: 195 total, 1 running, 194 sleeping, 0 stopped, 0 zombie > > Cpu(s): 28.7%us, 10.1%sy, 0.0%ni, 25.8%id, 32.9%wa, 0.1%hi, 2.4%si, > > 0.0%st > > Mem: 8166572k total, 8118592k used, 47980k free, 3264k buffers > > Swap: 1052248k total, 140k used, 1052108k free, 2099896k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 15636 hadoop 18 0 4806m 4.2g 10m S 206.9 53.3 87:48.81 java > > 15243 hadoop 18 0 4734m 1.3g 9800 S 117.6 16.7 63:46.52 java > > > > -James > > > > > > > > St>Ack > > > > > > > > > > > > > > > > > Here are the new region server stats along with load averages: > > > > > > > > Region Server 1: > > > > request=0.0, regions=16, stores=16, storefiles=33, > > storefileIndexSize=4, memstoreSize=1, compactionQueueSize=0, usedHeap=2891, > > maxHeap=4079, blockCacheSize=1403878072, blockCacheFree=307135816, > > blockCacheCount=21107, blockCacheHitRatio=84, fsReadLatency=0, > > fsWriteLatency=0, fsSyncLatency=0 > > > > Load Averages: 10.34, 10.58, 7.08 > > > > > > > > Region Server 2: > > > > request=0.0, regions=15, stores=16, storefiles=26, > > storefileIndexSize=3, memstoreSize=1, compactionQueueSize=0, usedHeap=3257, > > maxHeap=4079, blockCacheSize=661765368, blockCacheFree=193741576, > > blockCacheCount=9942, blockCacheHitRatio=77, fsReadLatency=0, > > fsWriteLatency=0, fsSyncLatency=0 > > > > Load Averages: 1.90, 1.23, 0.98 > > > > > > > > Region Server 3: > > > > request=0.0, regions=16, stores=16, storefiles=41, > > storefileIndexSize=4, memstoreSize=4, compactionQueueSize=0, usedHeap=1627, > > maxHeap=4079, blockCacheSize=665117184, blockCacheFree=190389760, > > blockCacheCount=9995, blockCacheHitRatio=70, fsReadLatency=0, > > fsWriteLatency=0, fsSyncLatency=0 > > > > Load Averages: 2.01, 3.56, 4.18 > > > > > > > > That first region server is getting hit much harder than the others. > > > > They're identical machines (8-core), and the distribution of keys > > should > > > > be fairly random, so I'm not sure why that would happen. Any other > > > > ideas or suggestions would be greatly appreciated. > > > > > > > > Thanks, > > > > James > > > > > > > > > > > > On Mon, 2010-02-15 at 21:51 -0600, Stack wrote: > > > >> Yeah, I was going to say that if your loading is mostly read, you can > > > >> probably go up from the 0.2 given over to cache. I like Dan's > > > >> suggestion of trying it first on one server, if you can. > > > >> > > > >> St.Ack > > > >> > > > >> On Mon, Feb 15, 2010 at 5:22 PM, Dan Washusen <d...@reactive.org> > > wrote: > > > >> > So roughly 72% of reads use the blocks held in the block cache... > > > >> > > > > >> > It would be interesting to see the difference between when it was > > working OK > > > >> > and now. Could you try increasing the memory allocated to one of > > the > > > >> > regions and also increasing the "hfile.block.cache.size" to say > > '0.4' on the > > > >> > same region? > > > >> > > > > >> > On 16 February 2010 11:54, James Baldassari <ja...@dataxu.com> > > wrote: > > > >> > > > > >> >> Hi Dan. Thanks for your suggestions. I am doing writes at the > > same > > > >> >> time as reads, but there are usually many more reads than writes. > > Here > > > >> >> are the stats for all three region servers: > > > >> >> > > > >> >> Region Server 1: > > > >> >> request=0.0, regions=15, stores=16, storefiles=34, > > storefileIndexSize=3, > > > >> >> memstoreSize=308, compactionQueueSize=0, usedHeap=3096, > > maxHeap=4079, > > > >> >> blockCacheSize=705474544, blockCacheFree=150032400, > > blockCacheCount=10606, > > > >> >> blockCacheHitRatio=76, fsReadLatency=0, fsWriteLatency=0, > > fsSyncLatency=0 > > > >> >> > > > >> >> Region Server 2: > > > >> >> request=0.0, regions=16, stores=16, storefiles=39, > > storefileIndexSize=4, > > > >> >> memstoreSize=225, compactionQueueSize=0, usedHeap=3380, > > maxHeap=4079, > > > >> >> blockCacheSize=643172800, blockCacheFree=212334144, > > blockCacheCount=9660, > > > >> >> blockCacheHitRatio=69, fsReadLatency=0, fsWriteLatency=0, > > fsSyncLatency=0 > > > >> >> > > > >> >> Region Server 3: > > > >> >> request=0.0, regions=13, stores=13, storefiles=31, > > storefileIndexSize=4, > > > >> >> memstoreSize=177, compactionQueueSize=0, usedHeap=1905, > > maxHeap=4079, > > > >> >> blockCacheSize=682848608, blockCacheFree=172658336, > > blockCacheCount=10262, > > > >> >> blockCacheHitRatio=72, fsReadLatency=0, fsWriteLatency=0, > > fsSyncLatency=0 > > > >> >> > > > >> >> The average blockCacheHitRatio is about 72. Is this too low? > > Anything > > > >> >> else I can check? > > > >> >> > > > >> >> -James > > > >> >> > > > >> >> > > > >> >> On Mon, 2010-02-15 at 18:16 -0600, Dan Washusen wrote: > > > >> >> > Maybe the block cache is thrashing? > > > >> >> > > > > >> >> > If you are regularly writing data to your tables then it's > > possible that > > > >> >> the > > > >> >> > block cache is no longer being effective. On the region server > > web UI > > > >> >> check > > > >> >> > the blockCacheHitRatio value. You want this value to be high (0 > > - 100). > > > >> >> If > > > >> >> > this value is low it means that HBase has to go to disk to fetch > > blocks > > > >> >> of > > > >> >> > data. You can control the amount of VM memory that HBase > > allocates to > > > >> >> the > > > >> >> > block cache using the "hfile.block.cache.size" property (default > > is 0.2 > > > >> >> > (20%)). > > > >> >> > > > > >> >> > Cheers, > > > >> >> > Dan > > > >> >> > > > > >> >> > On 16 February 2010 10:45, James Baldassari <ja...@dataxu.com> > > wrote: > > > >> >> > > > > >> >> > > Hi, > > > >> >> > > > > > >> >> > > Does anyone have any tips to share regarding optimization for > > random > > > >> >> > > read performance? For writes I've found that setting a large > > write > > > >> >> > > buffer and setting auto-flush to false on the client side > > significantly > > > >> >> > > improved put performance. Are there any similar easy tweaks to > > improve > > > >> >> > > random read performance? > > > >> >> > > > > > >> >> > > I'm using HBase 0.20.3 in a very read-heavy real-time system > > with 1 > > > >> >> > > master and 3 region servers. It was working ok for a while, > > but today > > > >> >> > > there was a severe degradation in read performance. Restarting > > Hadoop > > > >> >> > > and HBase didn't help, are there are no errors in the logs. > > Read > > > >> >> > > performance starts off around 1,000-2,000 gets/second but > > quickly > > > >> >> > > (within minutes) drops to around 100 gets/second. > > > >> >> > > > > > >> >> > > I've already looked at the performance tuning wiki page. On > > the server > > > >> >> > > side I've increased hbase.regionserver.handler.count from 10 to > > 100, > > > >> >> but > > > >> >> > > it didn't help. Maybe this is expected because I'm only using > > a single > > > >> >> > > client to do reads. I'm working on implementing a client pool > > now, but > > > >> >> > > I'm wondering if there are any other settings on the server or > > client > > > >> >> > > side that might improve things. > > > >> >> > > > > > >> >> > > Thanks, > > > >> >> > > James > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > >> >> > > > >> > > > > > > > > > > > > >