[jira] [Updated] (HBASE-10403) Simplify offheap cache configuration
[ https://issues.apache.org/jira/browse/HBASE-10403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-10403: -- Issue Type: Umbrella (was: Improvement) > Simplify offheap cache configuration > > > Key: HBASE-10403 > URL: https://issues.apache.org/jira/browse/HBASE-10403 > Project: HBase > Issue Type: Umbrella > Components: io >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-10403.0.patch > > > The BucketCache (HBASE-7404) is a very nice piece of functionality which is > hidden behind complex configuration. Enabling it currently requires manual > calculation of L1 cache. It'd be nice to make this easier to use and conform > better with the existing heap management tools we already have. > Turning it on currently requires explicitly setting LruBlockCache (L1) > instance size and IOEngine (L2) size, making sure that L1 size isn't too big > vs global memstore and total heap. This is further confused by > hbase.bucketcache.size accepting a percentage of total heap OR explicit size > in MB. Enabling SlabCache is slightly easier in that it just accepts whatever > LruBlockCache is provided. > Turning on BucketCache using off-heap mode could look like: > hbase-env.sh: > Set HBASE_REGIONSERVER_OPTS: > -Xmx5000m > -XX:MaxDirectMemorySize=15000m > hbase-site.xml: > - hbase.regionserver.global.memstore.size = 0.7 > - hbase.regionserver.onheap.blockcache.size = 0.1 > - hbase.regionserver.blockcache.impl = BucketCache > - hbase.bucketcache.ioengine = offheap > The result being a CombinedCache instance with 500m LruBlockCache + 15000m > ByteBufferIOEngine running in direct mode. > This example does a couple things (mostly for the admin): > - knows NOT to enable SlabCache > - s/hfile.block.cache.size/hbase.regionserver.onheap.blockcache.size/ > - maintains the validity of HBaseConfiguration's existing check that global > MemStore + LruBlockCache == 0.8 > - maps "BucketCache" into meaning "a CombinedCache instance with these > implementations for L1 and L2." > - Figures out appropriate values for hbase.bucketcache.size and > hbase.bucketcache.percentage.in.combinedcache -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-10403) Simplify offheap cache configuration
[ https://issues.apache.org/jira/browse/HBASE-10403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carter Shanklin updated HBASE-10403: Assignee: Nick Dimiduk (was: Carter Shanklin) > Simplify offheap cache configuration > > > Key: HBASE-10403 > URL: https://issues.apache.org/jira/browse/HBASE-10403 > Project: HBase > Issue Type: Improvement > Components: io >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-10403.0.patch > > > The BucketCache (HBASE-7404) is a very nice piece of functionality which is > hidden behind complex configuration. Enabling it currently requires manual > calculation of L1 cache. It'd be nice to make this easier to use and conform > better with the existing heap management tools we already have. > Turning it on currently requires explicitly setting LruBlockCache (L1) > instance size and IOEngine (L2) size, making sure that L1 size isn't too big > vs global memstore and total heap. This is further confused by > hbase.bucketcache.size accepting a percentage of total heap OR explicit size > in MB. Enabling SlabCache is slightly easier in that it just accepts whatever > LruBlockCache is provided. > Turning on BucketCache using off-heap mode could look like: > hbase-env.sh: > Set HBASE_REGIONSERVER_OPTS: > -Xmx5000m > -XX:MaxDirectMemorySize=15000m > hbase-site.xml: > - hbase.regionserver.global.memstore.size = 0.7 > - hbase.regionserver.onheap.blockcache.size = 0.1 > - hbase.regionserver.blockcache.impl = BucketCache > - hbase.bucketcache.ioengine = offheap > The result being a CombinedCache instance with 500m LruBlockCache + 15000m > ByteBufferIOEngine running in direct mode. > This example does a couple things (mostly for the admin): > - knows NOT to enable SlabCache > - s/hfile.block.cache.size/hbase.regionserver.onheap.blockcache.size/ > - maintains the validity of HBaseConfiguration's existing check that global > MemStore + LruBlockCache == 0.8 > - maps "BucketCache" into meaning "a CombinedCache instance with these > implementations for L1 and L2." > - Figures out appropriate values for hbase.bucketcache.size and > hbase.bucketcache.percentage.in.combinedcache -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10403) Simplify offheap cache configuration
[ https://issues.apache.org/jira/browse/HBASE-10403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-10403: -- Fix Version/s: (was: 0.99.0) 2.0.0 Issue Type: Improvement (was: Bug) I think that this is a 2.0.0 rather than a 1.0 effort at this stage. Marking the issue so. Even a less audacious change in configs could break existing cache setups unless done right. That said, block cache has had a bunch of doc added since, the options have been narrowed so less options to get lost in, and there is more visibility into what is deployed -- sizes, whether L1 and L2 and how they relate -- so this project less critical now. Moving out of 0.99. Simplify offheap cache configuration Key: HBASE-10403 URL: https://issues.apache.org/jira/browse/HBASE-10403 Project: HBase Issue Type: Improvement Components: io Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Minor Fix For: 2.0.0 Attachments: HBASE-10403.0.patch The BucketCache (HBASE-7404) is a very nice piece of functionality which is hidden behind complex configuration. Enabling it currently requires manual calculation of L1 cache. It'd be nice to make this easier to use and conform better with the existing heap management tools we already have. Turning it on currently requires explicitly setting LruBlockCache (L1) instance size and IOEngine (L2) size, making sure that L1 size isn't too big vs global memstore and total heap. This is further confused by hbase.bucketcache.size accepting a percentage of total heap OR explicit size in MB. Enabling SlabCache is slightly easier in that it just accepts whatever LruBlockCache is provided. Turning on BucketCache using off-heap mode could look like: hbase-env.sh: Set HBASE_REGIONSERVER_OPTS: -Xmx5000m -XX:MaxDirectMemorySize=15000m hbase-site.xml: - hbase.regionserver.global.memstore.size = 0.7 - hbase.regionserver.onheap.blockcache.size = 0.1 - hbase.regionserver.blockcache.impl = BucketCache - hbase.bucketcache.ioengine = offheap The result being a CombinedCache instance with 500m LruBlockCache + 15000m ByteBufferIOEngine running in direct mode. This example does a couple things (mostly for the admin): - knows NOT to enable SlabCache - s/hfile.block.cache.size/hbase.regionserver.onheap.blockcache.size/ - maintains the validity of HBaseConfiguration's existing check that global MemStore + LruBlockCache == 0.8 - maps BucketCache into meaning a CombinedCache instance with these implementations for L1 and L2. - Figures out appropriate values for hbase.bucketcache.size and hbase.bucketcache.percentage.in.combinedcache -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10403) Simplify offheap cache configuration
[ https://issues.apache.org/jira/browse/HBASE-10403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-10403: -- Fix Version/s: 0.99.0 Simplify offheap cache configuration Key: HBASE-10403 URL: https://issues.apache.org/jira/browse/HBASE-10403 Project: HBase Issue Type: Bug Components: io Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Minor Fix For: 0.99.0 Attachments: HBASE-10403.0.patch The BucketCache (HBASE-7404) is a very nice piece of functionality which is hidden behind complex configuration. Enabling it currently requires manual calculation of L1 cache. It'd be nice to make this easier to use and conform better with the existing heap management tools we already have. Turning it on currently requires explicitly setting LruBlockCache (L1) instance size and IOEngine (L2) size, making sure that L1 size isn't too big vs global memstore and total heap. This is further confused by hbase.bucketcache.size accepting a percentage of total heap OR explicit size in MB. Enabling SlabCache is slightly easier in that it just accepts whatever LruBlockCache is provided. Turning on BucketCache using off-heap mode could look like: hbase-env.sh: Set HBASE_REGIONSERVER_OPTS: -Xmx5000m -XX:MaxDirectMemorySize=15000m hbase-site.xml: - hbase.regionserver.global.memstore.size = 0.7 - hbase.regionserver.onheap.blockcache.size = 0.1 - hbase.regionserver.blockcache.impl = BucketCache - hbase.bucketcache.ioengine = offheap The result being a CombinedCache instance with 500m LruBlockCache + 15000m ByteBufferIOEngine running in direct mode. This example does a couple things (mostly for the admin): - knows NOT to enable SlabCache - s/hfile.block.cache.size/hbase.regionserver.onheap.blockcache.size/ - maintains the validity of HBaseConfiguration's existing check that global MemStore + LruBlockCache == 0.8 - maps BucketCache into meaning a CombinedCache instance with these implementations for L1 and L2. - Figures out appropriate values for hbase.bucketcache.size and hbase.bucketcache.percentage.in.combinedcache -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10403) Simplify offheap cache configuration
[ https://issues.apache.org/jira/browse/HBASE-10403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-10403: - Attachment: HBASE-10403.0.patch [~zjushch] you noticed :) Here's a sketch of the direction I'm moving. A couple observation: - the logic in CacheConfig#instantiateBlockCache is pretty convoluted, with the different implementations all configured (or partially so) by the same if block - CombinedBlockCache and DoubleBlockCache are actually strategies for implementing a 2-level cache. It would be nice, for instance, for a SlabCache user to take advantage of the CombinedBlockCache strategy. - A little cleanup here will let us experiment with other caching strategies. IE, I'd like to try a 3-level cache with different implementations for index blocks (L0), decompressed hot blocks (L1), and compressed cold/speculatively read blocks (L2). Simplify offheap cache configuration Key: HBASE-10403 URL: https://issues.apache.org/jira/browse/HBASE-10403 Project: HBase Issue Type: Bug Components: io Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Minor Attachments: HBASE-10403.0.patch The BucketCache (HBASE-7404) is a very nice piece of functionality which is hidden behind complex configuration. Enabling it currently requires manual calculation of L1 cache. It'd be nice to make this easier to use and conform better with the existing heap management tools we already have. Turning it on currently requires explicitly setting LruBlockCache (L1) instance size and IOEngine (L2) size, making sure that L1 size isn't too big vs global memstore and total heap. This is further confused by hbase.bucketcache.size accepting a percentage of total heap OR explicit size in MB. Enabling SlabCache is slightly easier in that it just accepts whatever LruBlockCache is provided. Turning on BucketCache using off-heap mode could look like: hbase-env.sh: Set HBASE_REGIONSERVER_OPTS: -Xmx5000m -XX:MaxDirectMemorySize=15000m hbase-site.xml: - hbase.regionserver.global.memstore.size = 0.7 - hbase.regionserver.onheap.blockcache.size = 0.1 - hbase.regionserver.blockcache.impl = BucketCache - hbase.bucketcache.ioengine = offheap The result being a CombinedCache instance with 500m LruBlockCache + 15000m ByteBufferIOEngine running in direct mode. This example does a couple things (mostly for the admin): - knows NOT to enable SlabCache - s/hfile.block.cache.size/hbase.regionserver.onheap.blockcache.size/ - maintains the validity of HBaseConfiguration's existing check that global MemStore + LruBlockCache == 0.8 - maps BucketCache into meaning a CombinedCache instance with these implementations for L1 and L2. - Figures out appropriate values for hbase.bucketcache.size and hbase.bucketcache.percentage.in.combinedcache -- This message was sent by Atlassian JIRA (v6.1.5#6160)
