[
https://issues.apache.org/jira/browse/HBASE-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15691636#comment-15691636
]
stack commented on HBASE-17072:
-------------------------------
Test fails because of the below part of the patch which tries to go to the
index to find block lengths seeking. If I instrument the test, I can see that
we seek one time less w/ the below in place. We are hitting cache one time
extra which is why the test fails. The patch also seems to change how we start
up our scan. Before patch we seek twice to point zero then twice to the
trailer. With the below in place we seek to the trailer twice, to zero twice,
and then back to the trailer. Not sure why.
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java
index 4887550..be8fc89 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java
@@ -1087,7 +1087,6 @@ public class HFileReaderImpl implements HFile.Reader,
Configurable {
/**
* Positions this scanner at the start of the file.
- *
* @return false if empty file; i.e. a call to next would return false and
* the current key and value are undefined.
* @throws IOException
@@ -1104,12 +1103,14 @@ public class HFileReaderImpl implements HFile.Reader,
Configurable {
}
long firstDataBlockOffset =
reader.getTrailer().getFirstDataBlockOffset();
- if (curBlock != null
- && curBlock.getOffset() == firstDataBlockOffset) {
+ if (curBlock != null && curBlock.getOffset() == firstDataBlockOffset) {
return processFirstDataBlock();
}
-
- readAndUpdateNewBlock(firstDataBlockOffset);
+ Cell firstKey = this.reader.getFirstKey();
+ HFileBlockIndex.BlockIndexReader indexReader =
reader.getDataBlockIndexReader();
+ BlockWithScanInfo blockWithScanInfo =
indexReader.loadDataBlockWithScanInfo(firstKey, curBlock,
+ cacheBlocks, pread, isCompaction, getEffectiveDataBlockEncoding());
+ updateCurrentBlock(blockWithScanInfo.getHFileBlock());
return true;
}
@@ -1119,16 +1120,6 @@ public class HFileReaderImpl implements HFile.Reader,
Configurable {
return true;
}
- protected void readAndUpdateNewBlock(long firstDataBlockOffset) throws
IOException,
- CorruptHFileException {
- HFileBlock newBlock = reader.readBlock(firstDataBlockOffset, -1,
cacheBlocks, pread,
- isCompaction, true, BlockType.DATA, getEffectiveDataBlockEncoding());
- if (newBlock.getOffset() < 0) {
- throw new IOException("Invalid block offset: " + newBlock.getOffset());
- }
- updateCurrentBlock(newBlock);
- }
-
protected int loadBlockAndSeekToKey(HFileBlock seekToBlock, Cell
nextIndexedKey,
boolean rewind, Cell key, boolean seekBefore) throws IOException {
if (this.curBlock == null
{code}
> CPU usage starts to climb up to 90-100% when using G1GC
> -------------------------------------------------------
>
> Key: HBASE-17072
> URL: https://issues.apache.org/jira/browse/HBASE-17072
> Project: HBase
> Issue Type: Bug
> Components: Performance, regionserver
> Affects Versions: 1.0.0, 2.0.0, 1.2.0
> Reporter: Eiichi Sato
> Attachments: HBASE-17072.master.001.patch,
> HBASE-17072.master.002.patch, disable-block-header-cache.patch,
> mat-threadlocals.png, mat-threads.png, metrics.png, slave1.svg, slave2.svg,
> slave3.svg, slave4.svg
>
>
> h5. Problem
> CPU usage of a region server in our CDH 5.4.5 cluster, at some point, starts
> to gradually get higher up to nearly 90-100% when using G1GC. We've also run
> into this problem on CDH 5.7.3 and CDH 5.8.2.
> In our production cluster, it normally takes a few weeks for this to happen
> after restarting a RS. We reproduced this on our test cluster and attached
> the results. Please note that, to make it easy to reproduce, we did some
> "anti-tuning" on a table when running tests.
> In metrics.png, soon after we started running some workloads against a test
> cluster (CDH 5.8.2) at about 7 p.m. CPU usage of the two RSs started to rise.
> Flame Graphs (slave1.svg to slave4.svg) are generated from jstack dumps of
> each RS process around 10:30 a.m. the next day.
> After investigating heapdumps from another occurrence on a test cluster
> running CDH 5.7.3, we found that the ThreadLocalMap contain a lot of
> contiguous entries of {{HFileBlock$PrefetchedHeader}} probably due to primary
> clustering. This caused more loops in
> {{ThreadLocalMap#expungeStaleEntries()}}, consuming a certain amount of CPU
> time. What is worse is that the method is called from RPC metrics code,
> which means even a small amount of per-RPC time soon adds up to a huge amount
> of CPU time.
> This is very similar to the issue in HBASE-16616, but we have many
> {{HFileBlock$PrefetchedHeader}} not only {{Counter$IndexHolder}} instances.
> Here are some OQL counts from Eclipse Memory Analyzer (MAT). This shows a
> number of ThreadLocal instances in the ThreadLocalMap of a single handler
> thread.
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name =
> "org.apache.hadoop.hbase.io.hfile.HFileBlock$PrefetchedHeader"
> #=> 10980 instances
> {code}
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name = "org.apache.hadoop.hbase.util.Counter$IndexHolder"
> #=> 2052 instances
> {code}
> Although as described in HBASE-16616 this somewhat seems to be an issue in
> G1GC side regarding weakly-reachable objects, we should keep ThreadLocal
> usage minimal and avoid creating an indefinite number (in this case, a number
> of HFiles) of ThreadLocal instances.
> HBASE-16146 removes ThreadLocals from the RPC metrics code. That may solve
> the issue (I just saw the patch, never tested it at all), but the
> {{HFileBlock$PrefetchedHeader}} are still there in the ThreadLocalMap, which
> may cause issues in the future again.
> h5. Our Solution
> We simply removed the whole {{HFileBlock$PrefetchedHeader}} caching and
> fortunately we didn't notice any performance degradation for our production
> workloads.
> Because the PrefetchedHeader caching uses ThreadLocal and because RPCs are
> handled randomly in any of the handlers, small Get or small Scan RPCs do not
> benefit from the caching (See HBASE-10676 and HBASE-11402 for the details).
> Probably, we need to see how well reads are saved by the caching for large
> Scan or Get RPCs and especially for compactions if we really remove the
> caching. It's probably better if we can remove ThreadLocals without breaking
> the current caching behavior.
> FWIW, I'm attaching the patch we applied. It's for CDH 5.4.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
