Hi there,
Our goal is to fully utilize the free RAM on each node/region server for
HBase. At the same time, we do not want to incur too much pressure from
GC (garbage collection). Based on Ted's sugguestion, we are trying to
using bucket cache.
However, we are not sure:
- The relation between XX:MaxDirectMemorySize and java heap size. Is
MaxDirectMemorySize part of java heap size ?
- The relation between XX:MaxDirectMemorySize and
hbase.bucketcache.size. Are they equal?
- How to adjust hbase.bucketcache.percentage.in.combinedcache?
Right now, we have the following configuration. Does it make sense?
- java heap size of each hbase region server to 12 GB
- -XX:MaxDirectMemorySize to be 6GB
- hbase-site.xml :
<property>
<name>hbase.offheapcache.percentage</name>
<value>0</value>
</property>
<property>
<name>hbase.bucketcache.ioengine</name>
<value>offheap</value>
</property>
<property>
<name>hbase.bucketcache.percentage.in.combinedcache</name>
<value>0.8</value>
</property>
<property>
<name>hbase.bucketcache.size</name>
<value>6144</value>
</property>
Thanks,
Jane
On 7/17/2014 3:05 PM, Ted Yu wrote:
Have you considered using BucketCache ?
Please read 9.6.4.1 under
http://hbase.apache.org/book.html#regionserver.arch
Note: remember to verify the config values against the hbase release you're
using.
Cheers
On Thu, Jul 17, 2014 at 2:53 PM, Jane Tao <[email protected]> wrote:
Hi Ted,
In my case, there is a 6 Node HBase cluster setup (running on Oracle BDA).
Each node has plenty of RAM (64GB) and CPU cores. Several articles seem to
suggest
that it is not a good idea to allocate too much RAM to region server's
heap setting.
If each region server has 10GB heap and there is only one region server
per node, then
I have 10x6=60GB for the whole HBase. This setting is good for ~100M rows
but starts
to incur lots of GC activities on region servers when loading billions of
rows.
Basically, I need a configuration that can fully utilize the free RAM on
each node for HBase.
Thanks,
Jane
On 7/16/2014 4:17 PM, Ted Yu wrote:
Jane:
Can you briefly describe the use case where multiple region servers are
needed on the same host ?
Cheers
On Wed, Jul 16, 2014 at 3:14 PM, Dhaval Shah <[email protected]
wrote:
Its certainly possible (atleast with command line) but probably very
messy. You will need to have different ports, different log files,
different pid files, possibly even different configs on the same machine.
Regards,
Dhaval
________________________________
From: Jane Tao <[email protected]>
To: [email protected]
Sent: Wednesday, 16 July 2014 6:06 PM
Subject: multiple region servers at one machine
Hi there,
Is it possible to run multiple region servers at one machine/node? If
this is possible, how to start multiple region servers with command
lines or cloudera manager?
Thanks,
Jane
--
--
--
