[jira] [Commented] (HBASE-23296) Add CompositeBucketCache to support tiered BC

2020-10-19 Thread Anoop Sam John (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-23296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216675#comment-17216675
 ] 

Anoop Sam John commented on HBASE-23296:


[~javaman_chen]  Still interested in this contribution I believe. What you say 
abt above comment/suggestion? 

> Add CompositeBucketCache to support tiered BC
> -
>
> Key: HBASE-23296
> URL: https://issues.apache.org/jira/browse/HBASE-23296
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
>
> LruBlockCache is not suitable in the following scenarios:
> (1) cache size too large (will take too much heap memory, and 
> evictBlocksByHfileName is not so efficient, as HBASE-23277 mentioned)
> (2) block evicted frequently, especially cacheOnWrite & prefetchOnOpen are 
> enabled.
> Since block‘s data is reclaimed by GC, this may affect GC performance.
> So how about enabling a Bucket based L1 Cache.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (HBASE-23296) Add CompositeBucketCache to support tiered BC

2020-06-03 Thread Anoop Sam John (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-23296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124834#comment-17124834
 ] 

Anoop Sam John commented on HBASE-23296:


The way we can introduce that can be this way (I believe that is what mostly 
the patch is doing).  
1. We can have a combined cache as that of today - No change.
   There will be on heap LRU cache for which size is defined as % of Xmx. Then 
there can be Bucket Cache on either offheap or file. Here all index and bloom 
goes to on heap cache and data goes to bucket cache.
2. Allow 2 tier in Bucket Cache itself. I would recommend not using the term L1 
and L2 here (even in configs) so that it is clear what it does. The tier 1 can 
be off heap (will be mostly) and tier 2 will be file based. Now if this is 
configured, nothing will go to the on heap cache. In fact user should be able 
to config the on heap block cache size % as 0. Even if configured a value, 
there is no usage. We at least need to log that very clearly.  And here the 
index and bloom will go to tier 1 and data to tier 2.   I believe right now if 
we have block cache % as 0, we will not even create the bucket cache.  The RS 
can not have a bucket cache alone thing.  This we need to change in code.

This 2 types might be enough for this jira. But I would like to see a 3rd type 
where the tiered Bucket cache can be configured as L1, L2 way. (Like what we 
have in 1.x with CombinedCache)  The tier 1 will become L1 and tier 2 will be 
L2 cache for all types of blocks. 

> Add CompositeBucketCache to support tiered BC
> -
>
> Key: HBASE-23296
> URL: https://issues.apache.org/jira/browse/HBASE-23296
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
>
> LruBlockCache is not suitable in the following scenarios:
> (1) cache size too large (will take too much heap memory, and 
> evictBlocksByHfileName is not so efficient, as HBASE-23277 mentioned)
> (2) block evicted frequently, especially cacheOnWrite & prefetchOnOpen are 
> enabled.
> Since block‘s data is reclaimed by GC, this may affect GC performance.
> So how about enabling a Bucket based L1 Cache.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (HBASE-23296) Add CompositeBucketCache to support tiered BC

2020-06-03 Thread Guanghao Zhang (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-23296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124700#comment-17124700
 ] 

Guanghao Zhang commented on HBASE-23296:


{quote}Do you mean BucketCache with heap ioEngine? seems this is not supported 
now. If use LruBlockCache there may be some gc problems.
{quote}
I mean LruBlockCache. For combined cache, now we only support two tired cache, 
let's call it L1 and L2 cache. Now L1 cache must be on heap. And L2 cache may 
be offheap, file or others. Introduce a offheap L1 cache is good idea.

> Add CompositeBucketCache to support tiered BC
> -
>
> Key: HBASE-23296
> URL: https://issues.apache.org/jira/browse/HBASE-23296
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
>
> LruBlockCache is not suitable in the following scenarios:
> (1) cache size too large (will take too much heap memory, and 
> evictBlocksByHfileName is not so efficient, as HBASE-23277 mentioned)
> (2) block evicted frequently, especially cacheOnWrite & prefetchOnOpen are 
> enabled.
> Since block‘s data is reclaimed by GC, this may affect GC performance.
> So how about enabling a Bucket based L1 Cache.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (HBASE-23296) Add CompositeBucketCache to support tiered BC

2020-06-01 Thread chenxu (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-23296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17121511#comment-17121511
 ] 

chenxu commented on HBASE-23296:


bq. Why don't use bigger heap as L1, because gc problem?
Do you mean BucketCache with heap ioEngine? seems this is not supported now. If 
use LruBlockCache there may be some gc problems.

> Add CompositeBucketCache to support tiered BC
> -
>
> Key: HBASE-23296
> URL: https://issues.apache.org/jira/browse/HBASE-23296
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
>
> LruBlockCache is not suitable in the following scenarios:
> (1) cache size too large (will take too much heap memory, and 
> evictBlocksByHfileName is not so efficient, as HBASE-23277 mentioned)
> (2) block evicted frequently, especially cacheOnWrite & prefetchOnOpen are 
> enabled.
> Since block‘s data is reclaimed by GC, this may affect GC performance.
> So how about enabling a Bucket based L1 Cache.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (HBASE-23296) Add CompositeBucketCache to support tiered BC

2020-06-01 Thread Anoop Sam John (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-23296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120882#comment-17120882
 ] 

Anoop Sam John commented on HBASE-23296:


bq.Why don't use bigger heap as L1, because gc problem?
Yes as I understand.  Ya that is a valid concern.  As per the cache requirement 
on the cloud backed HBase is very large here, the index/bloom cache need looks 
to be 100+ GB.  A larger Xmx than this is a valid concern. (Even on G1GC). The 
off heap BC can give similar perf numbers as on heap LRU cache after the off 
heaping optimization.


> Add CompositeBucketCache to support tiered BC
> -
>
> Key: HBASE-23296
> URL: https://issues.apache.org/jira/browse/HBASE-23296
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
>
> LruBlockCache is not suitable in the following scenarios:
> (1) cache size too large (will take too much heap memory, and 
> evictBlocksByHfileName is not so efficient, as HBASE-23277 mentioned)
> (2) block evicted frequently, especially cacheOnWrite & prefetchOnOpen are 
> enabled.
> Since block‘s data is reclaimed by GC, this may affect GC performance.
> So how about enabling a Bucket based L1 Cache.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (HBASE-23296) Add CompositeBucketCache to support tiered BC

2020-06-01 Thread Guanghao Zhang (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-23296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120804#comment-17120804
 ] 

Guanghao Zhang commented on HBASE-23296:


{quote}yes, we need a bigger L1, it's not suitable to keep it on the heap
{quote}
So use offheap as L1 and file as L2? Why don't use heap as L1, because gc 
problem?

> Add CompositeBucketCache to support tiered BC
> -
>
> Key: HBASE-23296
> URL: https://issues.apache.org/jira/browse/HBASE-23296
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
>
> LruBlockCache is not suitable in the following scenarios:
> (1) cache size too large (will take too much heap memory, and 
> evictBlocksByHfileName is not so efficient, as HBASE-23277 mentioned)
> (2) block evicted frequently, especially cacheOnWrite & prefetchOnOpen are 
> enabled.
> Since block‘s data is reclaimed by GC, this may affect GC performance.
> So how about enabling a Bucket based L1 Cache.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (HBASE-23296) Add CompositeBucketCache to support tiered BC

2020-05-28 Thread chenxu (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-23296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118849#comment-17118849
 ] 

chenxu commented on HBASE-23296:


bq. So the key point is a bigger L1 cache to cache all index/bloom blocks and 
the heap size is not enough?
yes, we need a bigger L1, it's not suitable to keep it on the heap

> Add CompositeBucketCache to support tiered BC
> -
>
> Key: HBASE-23296
> URL: https://issues.apache.org/jira/browse/HBASE-23296
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
>
> LruBlockCache is not suitable in the following scenarios:
> (1) cache size too large (will take too much heap memory, and 
> evictBlocksByHfileName is not so efficient, as HBASE-23277 mentioned)
> (2) block evicted frequently, especially cacheOnWrite & prefetchOnOpen are 
> enabled.
> Since block‘s data is reclaimed by GC, this may affect GC performance.
> So how about enabling a Bucket based L1 Cache.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (HBASE-23296) Add CompositeBucketCache to support tiered BC

2020-05-27 Thread Guanghao Zhang (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-23296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118219#comment-17118219
 ] 

Guanghao Zhang commented on HBASE-23296:


{quote}bq.But if the index and bloom miss, we may need three seeks. So only 
index/bloom go to L1 may be suitable. The current patch do things like this.
{quote}
So the key point is a bigger L1 cache to cache all index/bloom blocks and the 
heap size is not enough?

> Add CompositeBucketCache to support tiered BC
> -
>
> Key: HBASE-23296
> URL: https://issues.apache.org/jira/browse/HBASE-23296
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
>
> LruBlockCache is not suitable in the following scenarios:
> (1) cache size too large (will take too much heap memory, and 
> evictBlocksByHfileName is not so efficient, as HBASE-23277 mentioned)
> (2) block evicted frequently, especially cacheOnWrite & prefetchOnOpen are 
> enabled.
> Since block‘s data is reclaimed by GC, this may affect GC performance.
> So how about enabling a Bucket based L1 Cache.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (HBASE-23296) Add CompositeBucketCache to support tiered BC

2020-05-26 Thread chenxu (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-23296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116670#comment-17116670
 ] 

chenxu commented on HBASE-23296:


{quote}So when user config L1 and L2 tiers at the BucketCache level, the on 
heap LRU cache will not get used at all?
{quote}
yes as you said.
{quote}We can say all these clearly in the doc pls.
{quote}
OK, will improve the doc soon

> Add CompositeBucketCache to support tiered BC
> -
>
> Key: HBASE-23296
> URL: https://issues.apache.org/jira/browse/HBASE-23296
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
>
> LruBlockCache is not suitable in the following scenarios:
> (1) cache size too large (will take too much heap memory, and 
> evictBlocksByHfileName is not so efficient, as HBASE-23277 mentioned)
> (2) block evicted frequently, especially cacheOnWrite & prefetchOnOpen are 
> enabled.
> Since block‘s data is reclaimed by GC, this may affect GC performance.
> So how about enabling a Bucket based L1 Cache.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (HBASE-23296) Add CompositeBucketCache to support tiered BC

2020-05-26 Thread Anoop Sam John (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-23296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116626#comment-17116626
 ] 

Anoop Sam John commented on HBASE-23296:


So when user config L1 and L2 tiers at the BucketCache level, the on heap LRU 
cache will not get used at all? Sorry I did not go in to ur patch.  We can say 
all these clearly in the doc pls.

> Add CompositeBucketCache to support tiered BC
> -
>
> Key: HBASE-23296
> URL: https://issues.apache.org/jira/browse/HBASE-23296
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
>
> LruBlockCache is not suitable in the following scenarios:
> (1) cache size too large (will take too much heap memory, and 
> evictBlocksByHfileName is not so efficient, as HBASE-23277 mentioned)
> (2) block evicted frequently, especially cacheOnWrite & prefetchOnOpen are 
> enabled.
> Since block‘s data is reclaimed by GC, this may affect GC performance.
> So how about enabling a Bucket based L1 Cache.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (HBASE-23296) Add CompositeBucketCache to support tiered BC

2020-05-26 Thread chenxu (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-23296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116573#comment-17116573
 ] 

chenxu commented on HBASE-23296:


Hi, [~anoop.hbase] [~ram_krish], thanks for your review about the doc.
{quote}Are we proposing doing this? Or just the above one?
{quote}
Currently, we implement the above only. Not sure if it’s valuable for the below 
diagram.
{quote}Or should we even allow this model to act like today's CombinedCache 
model?
{quote}
I’d like this model, in our case we want warmup all the index and blooms, even 
the data cache miss, just one seek is enough.
 But if the index and bloom miss, we may need three seeks. So only index/bloom 
go to L1 may be suitable. The current path do things like this.

> Add CompositeBucketCache to support tiered BC
> -
>
> Key: HBASE-23296
> URL: https://issues.apache.org/jira/browse/HBASE-23296
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
>
> LruBlockCache is not suitable in the following scenarios:
> (1) cache size too large (will take too much heap memory, and 
> evictBlocksByHfileName is not so efficient, as HBASE-23277 mentioned)
> (2) block evicted frequently, especially cacheOnWrite & prefetchOnOpen are 
> enabled.
> Since block‘s data is reclaimed by GC, this may affect GC performance.
> So how about enabling a Bucket based L1 Cache.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (HBASE-23296) Add CompositeBucketCache to support tiered BC

2020-05-26 Thread Anoop Sam John (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-23296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116498#comment-17116498
 ] 

Anoop Sam John commented on HBASE-23296:


Thanks [~javaman_chen]  Added some comments on doc.  This is regarding the 
generic approach.  I think later we can move the discussion to Jira for better 
visibility.

> Add CompositeBucketCache to support tiered BC
> -
>
> Key: HBASE-23296
> URL: https://issues.apache.org/jira/browse/HBASE-23296
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
>
> LruBlockCache is not suitable in the following scenarios:
> (1) cache size too large (will take too much heap memory, and 
> evictBlocksByHfileName is not so efficient, as HBASE-23277 mentioned)
> (2) block evicted frequently, especially cacheOnWrite & prefetchOnOpen are 
> enabled.
> Since block‘s data is reclaimed by GC, this may affect GC performance.
> So how about enabling a Bucket based L1 Cache.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (HBASE-23296) Add CompositeBucketCache to support tiered BC

2020-05-25 Thread chenxu (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-23296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116335#comment-17116335
 ] 

chenxu commented on HBASE-23296:


sorry,make it as public already [~anoop.hbase]

> Add CompositeBucketCache to support tiered BC
> -
>
> Key: HBASE-23296
> URL: https://issues.apache.org/jira/browse/HBASE-23296
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
>
> LruBlockCache is not suitable in the following scenarios:
> (1) cache size too large (will take too much heap memory, and 
> evictBlocksByHfileName is not so efficient, as HBASE-23277 mentioned)
> (2) block evicted frequently, especially cacheOnWrite & prefetchOnOpen are 
> enabled.
> Since block‘s data is reclaimed by GC, this may affect GC performance.
> So how about enabling a Bucket based L1 Cache.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (HBASE-23296) Add CompositeBucketCache to support tiered BC

2020-05-16 Thread Anoop Sam John (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-23296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109108#comment-17109108
 ] 

Anoop Sam John commented on HBASE-23296:


[~javaman_chen]   I dont have access to the Google doc.  Can u make it public 
pls?

> Add CompositeBucketCache to support tiered BC
> -
>
> Key: HBASE-23296
> URL: https://issues.apache.org/jira/browse/HBASE-23296
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
>
> LruBlockCache is not suitable in the following scenarios:
> (1) cache size too large (will take too much heap memory, and 
> evictBlocksByHfileName is not so efficient, as HBASE-23277 mentioned)
> (2) block evicted frequently, especially cacheOnWrite & prefetchOnOpen are 
> enabled.
> Since block‘s data is reclaimed by GC, this may affect GC performance.
> So how about enabling a Bucket based L1 Cache.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (HBASE-23296) Add CompositeBucketCache to support tiered BC

2019-11-22 Thread chenxu (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-23296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16980043#comment-16980043
 ] 

chenxu commented on HBASE-23296:


A initial patch and an simple design doc was attached.

> Add CompositeBucketCache to support tiered BC
> -
>
> Key: HBASE-23296
> URL: https://issues.apache.org/jira/browse/HBASE-23296
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
>
> LruBlockCache is not suitable in the following scenarios:
> (1) cache size too large (will take too much heap memory, and 
> evictBlocksByHfileName is not so efficient, as HBASE-23277 mentioned)
> (2) block evicted frequently, especially cacheOnWrite & prefetchOnOpen are 
> enabled.
> Since block‘s data is reclaimed by GC, this may affect GC performance.
> So how about enabling a Bucket based L1 Cache.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (HBASE-23296) Add CompositeBucketCache to support tiered BC

2019-11-18 Thread Anoop Sam John (Jira)


[ 
https://issues.apache.org/jira/browse/HBASE-23296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977089#comment-16977089
 ] 

Anoop Sam John commented on HBASE-23296:


This needs a high level design doc also.  Also marking it as new feature.

> Add CompositeBucketCache to support tiered BC
> -
>
> Key: HBASE-23296
> URL: https://issues.apache.org/jira/browse/HBASE-23296
> Project: HBase
>  Issue Type: Improvement
>  Components: BlockCache
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
>
> LruBlockCache is not suitable in the following scenarios:
> (1) cache size too large (will take too much heap memory, and 
> evictBlocksByHfileName is not so efficient, as HBASE-23277 mentioned)
> (2) block evicted frequently, especially cacheOnWrite & prefetchOnOpen are 
> enabled.
> Since block‘s data is reclaimed by GC, this may affect GC performance.
> So how about enabling a Bucket based L1 Cache.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)