[
https://issues.apache.org/jira/browse/HBASE-12866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14283447#comment-14283447
]
Andrew Purtell commented on HBASE-12866:
----------------------------------------
TestHFilePerformance is broken generally.
The tool also behaves really differently in 0.98 than in later branches. I
think to start we should sync the master version of this file back to 0.98.
This is master, see the exception at the bottom of the below ("Added a key not
lexically larger than previous."):
{noformat}
$ ./bin/hbase org.apache.hadoop.hbase.io.hfile.TestHFilePerformance
2015-01-19 21:50:32,664 WARN [main] util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
****************************** Sequence File *****************************
File Type: SequenceFile
Writing SequenceFile with codecName: none cipherName: none
2015-01-19 21:50:33 Started timing.
/home/apurtell/src/hbase/target/test-data/d966ea92-5aa3-4559-8cb3-ef886a718656/TestHFilePerformance/SequenceFile.Performance
2015-01-19 21:50:34 Stopped timing.
2015-01-19 21:50:34 Data written:
2015-01-19 21:50:34 rate = 72MB/s
2015-01-19 21:50:34 total = 52200000B
2015-01-19 21:50:34 File written:
2015-01-19 21:50:34 rate = 73MB/s
2015-01-19 21:50:34 total = 52560076B
+++++++
Reading file of type: SequenceFile
Input file size: 52560076
2015-01-19 21:50:34 Started timing.
2015-01-19 21:50:34 Compression codec class: none
2015-01-19 21:50:34 Stopped timing.
2015-01-19 21:50:34 Finished in 406ms
2015-01-19 21:50:34 Data read:
2015-01-19 21:50:34 rate = 122MB/s
2015-01-19 21:50:34 total = 52200000B
2015-01-19 21:50:34 File read:
2015-01-19 21:50:34 rate = 123MB/s
2015-01-19 21:50:34 total = 52560076B
----------------------
File Type: SequenceFile
Writing SequenceFile with codecName: gz cipherName: none
2015-01-19 21:50:34 Started timing.
/home/apurtell/src/hbase/target/test-data/d966ea92-5aa3-4559-8cb3-ef886a718656/TestHFilePerformance/SequenceFile.Performance
Skipping sequencefile gz: SequenceFile doesn't work with GzipCodec without
native-hadoop code!
****************************** HFile *****************************
File Type: HFile
Writing HFile with codecName: none cipherName: none
2015-01-19 21:50:34 Started timing.
/home/apurtell/src/hbase/target/test-data/d966ea92-5aa3-4559-8cb3-ef886a718656/TestHFilePerformance/HFile.Performance
HFile write method:
2015-01-19 21:50:34,634 INFO [main] hfile.CacheConfig: CacheConfig:disabled
2015-01-19 21:50:34,676 ERROR [main] util.AbstractHBaseTool: Error running
command-line tool
java.io.IOException: Added a key not lexically larger than previous. Current
cell =
8U\x86\xB4\x1F\xEF\xF2\x05\xB4\xE0Z\x00\x08#\xF7\x8B_\x8F\x5C\x02C\x9C\xE8\xF6zx\x1D\x90\xCB\xE6\xBF\x1A\xE7\xF2\xBC@\xA4\x97\x09\xA0l\x0E1I\x9B\xF0)i\xCAB\xD2\x03\xE5f\xBC\xC6\x96\xDE\x08\xFA\x01\x02\xA0\xFD.#0\xB0\x96J\xBB|D0
\xDE\x1C\xAD\x09\xBF\xD68\x1F\xFB\x94\xDA\xAF\xBB\x90\xC4\xED\x91\xA0a:\xD1\xDCKG\x03//LATEST_TIMESTAMP/Maximum/vlen=5120/seqid=0,
lastCell = `\xB4
\xBB8Q\xD9\xD4z\xCB\x93=\xBEp9\x9B\xF6\xC9-\xA3:\xF0\x1DO\xB7p\xE9\x8C\x03%\xF4\x1D>\xBA\xF8\x98m\xA7\x12\xC8+\xCDMUK\xF0\xB5@#\xC2\x9BbM\xE9\xEF\x9C/\x93\x1E\xFCX\x0F\x9A\xFB\x08\x1B\x12\xE1\x07\xB1\xE8\x05\xF2\xB4\xF5\xF0\xF1\xD0\x0C-\x0FbcFp\x92\x1CPXg\xFF
\xF6\xA83^\x98\xAF\x87%//LATEST_TIMESTAMP/Maximum/vlen=5120/seqid=0
at
org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:201)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:261)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterV3.append(HFileWriterV3.java:87)
at
org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.timeWrite(TestHFilePerformance.java:200)
at
org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.testRunComparisons(TestHFilePerformance.java:380)
at
org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.doWork(TestHFilePerformance.java:447)
at
org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.main(TestHFilePerformance.java:452)
{noformat}
Same result on branch-1:
{noformat}
****************************** HFile *****************************
File Type: HFile
Writing HFile with codecName: none cipherName: none
2015-01-19 21:53:46 Started timing.
/home/apurtell/src/hbase/target/test-data/37e20539-2afb-4f8a-8c95-15ee6b9ed7cd/TestHFilePerformance/HFile.Performance
HFile write method:
2015-01-19 21:53:46,953 INFO [main] hfile.CacheConfig: CacheConfig:disabled
2015-01-19 21:53:46,997 ERROR [main] util.AbstractHBaseTool: Error running
command-line tool
java.io.IOException: Added a key not lexically larger than previous. Current
cell =
8U\x86\xB4\x1F\xEF\xF2\x05\xB4\xE0Z\x00\x08#\xF7\x8B_\x8F\x5C\x02C\x9C\xE8\xF6zx\x1D\x90\xCB\xE6\xBF\x1A\xE7\xF2\xBC@\xA4\x97\x09\xA0l\x0E1I\x9B\xF0)i\xCAB\xD2\x03\xE5f\xBC\xC6\x96\xDE\x08\xFA\x01\x02\xA0\xFD.#0\xB0\x96J\xBB|D0
\xDE\x1C\xAD\x09\xBF\xD68\x1F\xFB\x94\xDA\xAF\xBB\x90\xC4\xED\x91\xA0a:\xD1\xDCKG\x03//LATEST_TIMESTAMP/Maximum/vlen=5120/seqid=0,
lastCell = `\xB4
\xBB8Q\xD9\xD4z\xCB\x93=\xBEp9\x9B\xF6\xC9-\xA3:\xF0\x1DO\xB7p\xE9\x8C\x03%\xF4\x1D>\xBA\xF8\x98m\xA7\x12\xC8+\xCDMUK\xF0\xB5@#\xC2\x9BbM\xE9\xEF\x9C/\x93\x1E\xFCX\x0F\x9A\xFB\x08\x1B\x12\xE1\x07\xB1\xE8\x05\xF2\xB4\xF5\xF0\xF1\xD0\x0C-\x0FbcFp\x92\x1CPXg\xFF
\xF6\xA83^\x98\xAF\x87%//LATEST_TIMESTAMP/Maximum/vlen=5120/seqid=0
at
org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:201)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:261)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterV3.append(HFileWriterV3.java:87)
at
org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.timeWrite(TestHFilePerformance.java:200)
at
org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.testRunComparisons(TestHFilePerformance.java:380)
at
org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.doWork(TestHFilePerformance.java:447)
at
org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.main(TestHFilePerformance.java:452)
{noformat}
Finally, 0.98 produces results as reported in the description of this issue.
{noformat}
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile
Trailer from file
/home/apurtell/src/hbase/target/test-data/7b0f81cc-fbf8-4b67-b1f5-de77c2ee3074/TestHFilePerformance/HFile.Performance
{noformat}
My guess is the configuration for the tool does not allow for HFileV3, but I
haven't dug in yet.
> TestHFilePerformance is broken
> ------------------------------
>
> Key: HBASE-12866
> URL: https://issues.apache.org/jira/browse/HBASE-12866
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.98.8
> Environment: Command
> bin/hbase org.apache.hadoop.hbase.io.hfile.TestHFilePerformance
> Failure observed for test with the following options: Read HFile with
> codecName: none cipherName: aes
> Reporter: Vikas Vishwakarma
> Assignee: Andrew Purtell
> Priority: Minor
> Fix For: 1.0.0, 2.0.0, 1.1.0, 0.98.11
>
>
> Command
> bin/hbase org.apache.hadoop.hbase.io.hfile.TestHFilePerformance
> File Type: HFile
> Writing HFile with codecName: none cipherName: aes
> 2015-01-15 16:54:51 Started timing.
> /home/vvishwakarma/vikas/projects/hbase-src-0.98.8/hbase-0.98.8/target/test-data/03d50949-0185-4fac-b072-957d2da6ae0e/TestHFilePerformanceHFile.Performance
> HFile write method:
> 2015-01-15 16:54:52 Stopped timing.
> 2015-01-15 16:54:52 Data written:
> 2015-01-15 16:54:52 rate = 66MB/s
> 2015-01-15 16:54:52 total = 52200000B
> 2015-01-15 16:54:52 File written:
> 2015-01-15 16:54:52 rate = 66MB/s
> 2015-01-15 16:54:52 total = 52303530B
> +++++++
> Reading file of type: HFile
> Input file size: 52303530
> 2015-01-15 16:54:52 Started timing.
> 2015-01-15 16:54:52,680 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool
> org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile
> Trailer from file
> /home/vvishwakarma/vikas/projects/hbase-src-0.98.8/hbase-0.98.8/target/test-data/03d50949-0185-4fac-b072-957d2da6ae0e/TestHFilePerformance/HFile.Performance
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:463)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.createReaderFromStream(HFile.java:517)
> at
> org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.timeReading(TestHFilePerformance.java:272)
> at
> org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.testRunComparisons(TestHFilePerformance.java:390)
> at
> org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.doWork(TestHFilePerformance.java:447)
> at
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.io.hfile.TestHFilePerformance.main(TestHFilePerformance.java:452)
> Caused by: java.io.IOException: Using no compression but
> onDiskSizeWithoutHeader=134, uncompressedSizeWithoutHeader=113,
> numChecksumbytes=4
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.assumeUncompressed(HFileBlock.java:561)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1589)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1408)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1248)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1256)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.<init>(HFileReaderV2.java:146)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:451)
> ... 7 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
