[
https://issues.apache.org/jira/browse/HBASE-15240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15140843#comment-15140843
]
stack commented on HBASE-15240:
-------------------------------
The 100000 upper limit per file has been in there since we first did reporting
on blockcache: "HBASE-4089 blockCache contents report"
Prefetch is too slow. Takes ages to warm a big cache.
There is an NPE the odd time here abouts. Need to investigate:
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
index 4ada262..834588e 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
@@ -1502,7 +1502,7 @@ public class HFileBlock implements Cacheable {
// guaranteed to use hdfs checksum verification.
boolean doVerificationThruHBaseChecksum =
streamWrapper.shouldUseHBaseChecksum();
FSDataInputStream is =
streamWrapper.getStream(doVerificationThruHBaseChecksum);
-
+ if (is == null) throw new NullPointerException(toString());
HFileBlock blk = readBlockDataInternal(is, offset,
onDiskSizeWithHeaderL,
uncompressedSize, pread,
{code}
Need below to be able to load a bunch into cache by prefetching:
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
index f9d8167..c6fceea 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
@@ -154,11 +154,11 @@ public class BucketCache implements BlockCache, HeapSize {
/** Cache access count (sequential ID) */
private final AtomicLong accessCount = new AtomicLong(0);
- private static final int DEFAULT_CACHE_WAIT_TIME = 50;
+ private static final int DEFAULT_CACHE_WAIT_TIME = 5000;
// Used in test now. If the flag is false and the cache speed is very fast,
// bucket cache will skip some blocks when caching. If the flag is true, we
// will wait blocks flushed to IOEngine for some time when caching
- boolean wait_when_cache = false;
+ boolean wait_when_cache = true;
private final BucketCacheStats cacheStats = new BucketCacheStats();
{code}
Need to think about it.
Also needed these conffigs to do more work serializing into bucketcache.
{code}
<name>hbase.ui.blockcache.by.file.max
</name>
<value>2147483640</value>
</property>
<property>
<name>hbase.bucketcache.writer.queuelength
</name>
<value>256</value>
</property>
<property>
<name>hbase.bucketcache.writer.threads
</name>
<value>16</value>
</property>
{code}
Finally, we ask that blocks be 4k in configs but we are writing out 4.2k -- the
header of following block too. On the other end, fetching a block seems to be
making for two seeks. Investigate
> Go Big BucketCache Fixes
> ------------------------
>
> Key: HBASE-15240
> URL: https://issues.apache.org/jira/browse/HBASE-15240
> Project: HBase
> Issue Type: Umbrella
> Components: BucketCache
> Reporter: stack
> Assignee: stack
>
> Umbrella issue to which we will attach issues that prevent bucketcache going
> big; there's a few.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
