Enis Soztutar created HBASE-16288:
-------------------------------------
Summary: HFile intermediate block level indexes might recurse
forever creating multi TB files
Key: HBASE-16288
URL: https://issues.apache.org/jira/browse/HBASE-16288
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Mighty [~elserj] was debugging an opentsdb cluster where some region directory
ended up having 5TB+ files under <regiondir>/.tmp/
Further debugging and analysis, we were able to reproduce the problem locally
where we never we recursing in this code path for writing intermediate level
indices:
{code:title=HFileBlockIndex.java}
if (curInlineChunk != null) {
while (rootChunk.getRootSize() > maxChunkSize) {
rootChunk = writeIntermediateLevel(out, rootChunk);
numLevels += 1;
}
}
{code}
The problem happens if we end up with a very large rowKey (larger than
"hfile.index.block.max.size" being the first key in the block, then moving all
the way to the root-level index building. We will keep writing and building the
next level of intermediate level indices with a single very-large key. This can
happen in flush / compaction / region recovery causing cluster inoperability
due to ever-growing files.
Seems the issue was also reported earlier, with a temporary workaround:
https://github.com/OpenTSDB/opentsdb/issues/490
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
