+1 on what Ram said abt why HM also need this setting in ur version.. HM starts a RS internally.. You can do one thing.. Use different hbase config xml for HM process other than what used for RS. May be we need some fix also as Ram said.. Specially when no regions are allowed in HM side.
In 0.98 also, u set this MaxDirectMemory config specifically? Or there this setting was not present at all in hbase-env.sh? Because by def there is no cap for the Java process for off heap memory usage. You have to give some value bigger than the bucket cache size for this MaxDirectMemory.. Some other areas of code also uses the off heap memory.. Like the RPC uses java nio socket communication which would internally need some off heap memory to be allocated.. Just saying for ur ref. -Anoop- On Tue, Sep 12, 2017 at 10:41 PM, ramkrishna vasudevan <ramkrishna.s.vasude...@gmail.com> wrote: > Why Master also accepts MaxDirectMemory config is probably because in 1.1 > onwards we treat HMaster as HregionServer but it does the region managment > also. Am not very sure in 1.1.2 is HMaster allowed to host regions ? If so > you need to configure MaxDirectMemory, if not probably we can see how we > can avoid it. We need to raise a JIRA for that. > > Coming to the size of MaxDirectMemory less than bucket cache - I am > wondering was there a bug previously? Because assume you need 25G offheap > bucket cache then atleast 25G MaxDirectMemory is a must. Ideally you may > need some delta more than 25G. > > 0.98 is obsolete now so its better we go with how 1.1.2 works. But if you > feel there is a documentation that could help I think it is better we > provide one so that users like you are not affected. > > Regards > Ram > > > > > On Tue, Sep 12, 2017 at 10:22 PM, Arul Ramachandran <arkup...@gmail.com> > wrote: > >> Thank you, Ram. >> >> >> So are you trying to use bucket cache feature in offheap mode with >> 1.1.2? >> >> Yes. >> >> >> So even in 0.98 you were using bucket cache in offheap mode? >> >> Yes, but it is a different hbase cluster and it run 0.98. The one I am >> trying to setup offheap cache is hbase 1.1.2 >> >> >> -arul >> >> >> On Tue, Sep 12, 2017 at 9:34 AM, ramkrishna vasudevan < >> ramkrishna.s.vasude...@gmail.com> wrote: >> >> > Hi >> > >> > So are you trying to use bucket cache feature in offheap mode with 1.1.2? >> > If so then it is needed that the MaxDirectMemory is greater than the >> > offheap bucket cache size. >> > >> > If you are not using in offheap mode then probably there is no need for >> > MaxDirectMemory to be greater than bucket cache size. >> > >> > >>in Hbase 0.98, I had to set -XX:MaxDirectMemorySize less than >> > hbase.bucket.cache.size >> > So even in 0.98 you were using bucket cache in offheap mode? >> > >> > Regards >> > Ram >> > >> > On Tue, Sep 12, 2017 at 9:40 PM, Ted Yu <yuzhih...@gmail.com> wrote: >> > >> > > Looks like the config you meant should be hbase.bucketcache.size >> > > >> > > As the refguide says: >> > > >> > > A float that EITHER represents a percentage of total heap memory size >> to >> > > give to the cache (if < 1.0) OR, it is the total capacity in megabytes >> of >> > > BucketCache. Default: 0.0 >> > > >> > > If you specify the size as capacity, -XX:MaxDirectMemorySize should be >> > > bigger than the capacity. >> > > >> > > For #2, did you encounter some error ? >> > > >> > > Cheers >> > > >> > > On Tue, Sep 12, 2017 at 8:52 AM, Arul Ramachandran <arkup...@gmail.com >> > >> > > wrote: >> > > >> > > > In HBase 1.1.2, I am setting up bucket cache. I set MaxDirectMemory >> > size >> > > > greater than hbase.bucket.cache.size - only then it would work. >> > > > >> > > > 1) Does HBASE_REGIONSERVER_OPTS -XX:MaxDirectMemorySize needs to be >> > > greater >> > > > than hbase.bucket.cache.size? >> > > > 2) It seems with hbase 1.1.2, HBASE_MASTER_OPTS also needs the >> > > > -XX:MaxDirectMemorySize setting? >> > > > >> > > > IIRC, in Hbase 0.98, I had to set -XX:MaxDirectMemorySize less than >> > > > hbase.bucket.cache.size --and-- I did not have to set >> > > > -XX:MaxDirectMemorySize for HBASE_MASTER_OPTS. >> > > > >> > > > >> > > > Thanks, >> > > > Arul >> > > > >> > > >> > >>
