bq. Use different hbase config xml for HM process other than what used for RS
In Ambari, there is concept of host group which should allow the above. Again, further discussion should be carried out in Ambari mailing list. On Tue, Sep 12, 2017 at 10:17 AM, Anoop John <[email protected]> wrote: > +1 on what Ram said abt why HM also need this setting in ur version.. > HM starts a RS internally.. You can do one thing.. Use different > hbase config xml for HM process other than what used for RS. May be > we need some fix also as Ram said.. Specially when no regions are > allowed in HM side. > > In 0.98 also, u set this MaxDirectMemory config specifically? Or > there this setting was not present at all in hbase-env.sh? Because by > def there is no cap for the Java process for off heap memory usage. > You have to give some value bigger than the bucket cache size for this > MaxDirectMemory.. Some other areas of code also uses the off heap > memory.. Like the RPC uses java nio socket communication which would > internally need some off heap memory to be allocated.. Just saying for > ur ref. > > -Anoop- > > On Tue, Sep 12, 2017 at 10:41 PM, ramkrishna vasudevan > <[email protected]> wrote: > > Why Master also accepts MaxDirectMemory config is probably because in 1.1 > > onwards we treat HMaster as HregionServer but it does the region > managment > > also. Am not very sure in 1.1.2 is HMaster allowed to host regions ? If > so > > you need to configure MaxDirectMemory, if not probably we can see how we > > can avoid it. We need to raise a JIRA for that. > > > > Coming to the size of MaxDirectMemory less than bucket cache - I am > > wondering was there a bug previously? Because assume you need 25G offheap > > bucket cache then atleast 25G MaxDirectMemory is a must. Ideally you may > > need some delta more than 25G. > > > > 0.98 is obsolete now so its better we go with how 1.1.2 works. But if you > > feel there is a documentation that could help I think it is better we > > provide one so that users like you are not affected. > > > > Regards > > Ram > > > > > > > > > > On Tue, Sep 12, 2017 at 10:22 PM, Arul Ramachandran <[email protected]> > > wrote: > > > >> Thank you, Ram. > >> > >> >> So are you trying to use bucket cache feature in offheap mode with > >> 1.1.2? > >> > >> Yes. > >> > >> >> So even in 0.98 you were using bucket cache in offheap mode? > >> > >> Yes, but it is a different hbase cluster and it run 0.98. The one I am > >> trying to setup offheap cache is hbase 1.1.2 > >> > >> > >> -arul > >> > >> > >> On Tue, Sep 12, 2017 at 9:34 AM, ramkrishna vasudevan < > >> [email protected]> wrote: > >> > >> > Hi > >> > > >> > So are you trying to use bucket cache feature in offheap mode with > 1.1.2? > >> > If so then it is needed that the MaxDirectMemory is greater than the > >> > offheap bucket cache size. > >> > > >> > If you are not using in offheap mode then probably there is no need > for > >> > MaxDirectMemory to be greater than bucket cache size. > >> > > >> > >>in Hbase 0.98, I had to set -XX:MaxDirectMemorySize less than > >> > hbase.bucket.cache.size > >> > So even in 0.98 you were using bucket cache in offheap mode? > >> > > >> > Regards > >> > Ram > >> > > >> > On Tue, Sep 12, 2017 at 9:40 PM, Ted Yu <[email protected]> wrote: > >> > > >> > > Looks like the config you meant should be hbase.bucketcache.size > >> > > > >> > > As the refguide says: > >> > > > >> > > A float that EITHER represents a percentage of total heap memory > size > >> to > >> > > give to the cache (if < 1.0) OR, it is the total capacity in > megabytes > >> of > >> > > BucketCache. Default: 0.0 > >> > > > >> > > If you specify the size as capacity, -XX:MaxDirectMemorySize should > be > >> > > bigger than the capacity. > >> > > > >> > > For #2, did you encounter some error ? > >> > > > >> > > Cheers > >> > > > >> > > On Tue, Sep 12, 2017 at 8:52 AM, Arul Ramachandran < > [email protected] > >> > > >> > > wrote: > >> > > > >> > > > In HBase 1.1.2, I am setting up bucket cache. I set > MaxDirectMemory > >> > size > >> > > > greater than hbase.bucket.cache.size - only then it would work. > >> > > > > >> > > > 1) Does HBASE_REGIONSERVER_OPTS -XX:MaxDirectMemorySize needs to > be > >> > > greater > >> > > > than hbase.bucket.cache.size? > >> > > > 2) It seems with hbase 1.1.2, HBASE_MASTER_OPTS also needs the > >> > > > -XX:MaxDirectMemorySize setting? > >> > > > > >> > > > IIRC, in Hbase 0.98, I had to set -XX:MaxDirectMemorySize less > than > >> > > > hbase.bucket.cache.size --and-- I did not have to set > >> > > > -XX:MaxDirectMemorySize for HBASE_MASTER_OPTS. > >> > > > > >> > > > > >> > > > Thanks, > >> > > > Arul > >> > > > > >> > > > >> > > >> >
