-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44573/#review122776
-----------------------------------------------------------




ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py (line 
604)
<https://reviews.apache.org/r/44573/#comment184871>

    We should not reserve half of direct memory.
    
    For HBase, the offheap memory is mostly used by the offheap block cache 
(also called bucket cache). The remaining is used by the HDFS offheap buffers 
on the client side. 
    
    2G is a good fixed amount to reserve for this.



ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py (line 
606)
<https://reviews.apache.org/r/44573/#comment184879>

    we already have bucketcache_offheap_memory which is the same thing as 
bucketcache_size. No need to do this computation. We can set
    hbase_bucketcache_size = bucketcache_offheap_memory


- enis


On March 9, 2016, 4:35 p.m., Dmitro Lisnichenko wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44573/
> -----------------------------------------------------------
> 
> (Updated March 9, 2016, 4:35 p.m.)
> 
> 
> Review request for Ambari, Arpit Gupta, enis, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-15355
>     https://issues.apache.org/jira/browse/AMBARI-15355
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> On one of the clusters i did EU from 2.2.x to 2.3.x.
> 
> During upgrade there were problems with HBase service checks for region 
> servers and thus upgrade is paused.
> 
> Region server start is failing with error
> 
> {code}
> 2016-03-03 19:55:31,203 ERROR [regionserver:16020] 
> regionserver.HRegionServer: Failed init
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:658)
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> at 
> org.apache.hadoop.hbase.util.ByteBufferArray.<init>(ByteBufferArray.java:65)
> at 
> org.apache.hadoop.hbase.io.hfile.bucket.ByteBufferIOEngine.<init>(ByteBufferIOEngine.java:47)
> at 
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:307)
> at 
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.<init>(BucketCache.java:217)
> at 
> org.apache.hadoop.hbase.io.hfile.CacheConfig.getBucketCache(CacheConfig.java:614)
> at org.apache.hadoop.hbase.io.hfile.CacheConfig.getL2(CacheConfig.java:553)
> at 
> org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(CacheConfig.java:637)
> at org.apache.hadoop.hbase.io.hfile.CacheConfig.<init>(CacheConfig.java:231)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1361)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:899)
> at java.lang.Thread.run(Thread.java:745)
> 2016-03-03 19:55:31,206 FATAL [regionserver:16020] 
> regionserver.RSRpcServices: Run out of memory; RSRpcServices will abort 
> itself immediately
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:658)
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> at 
> org.apache.hadoop.hbase.util.ByteBufferArray.<init>(ByteBufferArray.java:65)
> at 
> org.apache.hadoop.hbase.io.hfile.bucket.ByteBufferIOEngine.<init>(ByteBufferIOEngine.java:47)
> at 
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:307)
> at 
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.<init>(BucketCache.java:217)
> at 
> org.apache.hadoop.hbase.io.hfile.CacheConfig.getBucketCache(CacheConfig.java:614)
> at org.apache.hadoop.hbase.io.hfile.CacheConfig.getL2(CacheConfig.java:553)
> at 
> org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(CacheConfig.java:637)
> at org.apache.hadoop.hbase.io.hfile.CacheConfig.<init>(CacheConfig.java:231)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1361)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:899)
> at java.lang.Thread.run(Thread.java:745)
> 2016-03-03 19:55:35,138 INFO  [main] zookeeper.ZooKeeper: Client 
> environment:zookeeper.version=3.4.6-3485--1, built on 12/16/2015 02:35 GMT
> {code}
> 
> issue is not that hbase_max_direct_memory_size is not set, but the value 
> coming from "HBase off-heap MaxDirectMemorySize" which I assume comes as the 
> hbase_max_direct_memory_size templete variable is set to 12288, but the 
> bucket cache is set to 18G:
> hbase.bucketcache.size 18432
> Since bucket cache is an offheap cache, hbase_max_direct_memory_size should 
> be > hbase.bucketcache.size
> 
> This was seen on the following cluster: 
> https://s.c:8443/#/main/services/HBASE/configs
> 
> the bucket cache config (which is an offheap cache) is set to 18G, but the 
> hbase_max_direct_memory_size is set to 12G. hbase_max_direct_memory_size 
> should always be higher than the offheap cache size. Both are configured from 
> Ambari. it reproduces on nodes with a lot of RAM (>23 GB), stack advisor 
> performs recommendation of hbase_max_direct_memory_size
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py 
> f88eb0c 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 9638054 
> 
> Diff: https://reviews.apache.org/r/44573/diff/
> 
> 
> Testing
> -------
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Reply via email to