[
https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15889489#comment-15889489
]
Hadoop QA commented on HBASE-16630:
-----------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s {color}
| {color:red} HBASE-16630 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12855290/HBASE-16630-v3-branch-1.patch
|
| JIRA Issue | HBASE-16630 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5893/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Fragmentation in long running Bucket Cache
> ------------------------------------------
>
> Key: HBASE-16630
> URL: https://issues.apache.org/jira/browse/HBASE-16630
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
> Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3
> Reporter: deepankar
> Assignee: deepankar
> Priority: Critical
> Attachments: 16630-v2-suggest.patch, 16630-v3-suggest.patch,
> HBASE-16630.patch, HBASE-16630-v2.patch, HBASE-16630-v3-branch-1.patch,
> HBASE-16630-v3-branch-1.X.patch, HBASE-16630-v3.patch
>
>
> As we are running bucket cache for a long time in our system, we are
> observing cases where some nodes after some time does not fully utilize the
> bucket cache, in some cases it is even worse in the sense they get stuck at a
> value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables
> are configured in-memory for simplicity sake).
> We took a heap dump and analyzed what is happening and saw that is classic
> case of fragmentation, current implementation of BucketCache (mainly
> BucketAllocator) relies on the logic that fullyFreeBuckets are available for
> switching/adjusting cache usage between different bucketSizes . But once a
> compaction / bulkload happens and the blocks are evicted from a bucket size ,
> these are usually evicted from random places of the buckets of a bucketSize
> and thus locking the number of buckets associated with a bucketSize and in
> the worst case of the fragmentation we have seen some bucketSizes with
> occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to
> share with the other bucketSize.
> Currently the existing eviction logic helps in the cases where cache used is
> more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also
> done, the eviction (freeSpace function) will not evict anything and the cache
> utilization will be stuck at that value without any allocations for other
> required sizes.
> The fix for this we came up with is simple that we do deFragmentation (
> compaction) of the bucketSize and thus increasing the occupancy ratio and
> also freeing up the buckets to be fullyFree, this logic itself is not
> complicated as the bucketAllocator takes care of packing the blocks in the
> buckets, we need evict and re-allocate the blocks for all the BucketSizes
> that dont fit the criteria.
> I am attaching an initial patch just to give an idea of what we are thinking
> and I'll improve it based on the comments from the community.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
