Author: nspiegelberg Date: Tue Oct 11 02:03:37 2011 New Revision: 1181378 URL: http://svn.apache.org/viewvc?rev=1181378&view=rev Log: HBASE-3006: Fix for severe read penalty on compressed column families
Summary: Details up in: https://issues.apache.org/jira/browse/HBASE-3006 Summary: Cuts down RPC calls to DFS by 5-10x when a HBase block cache miss happens, and it needs to go to DFS. Test Plan: Ran Pranav's schema search test, and for loading full index for a user: Averages dropped from 1-2 seconds to 100ms for my test runs. No outliers > 10 seconds. Previously we had many, and some which took 40-50 seconds. For 1-letter prefix match: the averages dropped from 25-50 ms to under 5 ms. [Caveat: In this run about 50% of the users were already warmed by the first test.] Running unit tests now. DiffCamp Revision: 158894 Reviewed By: nspiegelberg Commenters: pmalik CC: nspiegelberg, pmalik, jfan, kannan, tnovak, ruifang, hbase@lists Tasks: #383346: Clicking other folder takes forever then displays no messages when I have 100's of messages. Revert Plan: OK Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java Modified: hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java URL: http://svn.apache.org/viewvc/hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java?rev=1181378&r1=1181377&r2=1181378&view=diff ============================================================================== --- hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java (original) +++ hbase/branches/0.89/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java Tue Oct 11 02:03:37 2011 @@ -19,6 +19,7 @@ */ package org.apache.hadoop.hbase.io.hfile; +import java.io.BufferedInputStream; import java.io.Closeable; import java.io.DataInputStream; import java.io.DataOutputStream; @@ -1090,9 +1091,11 @@ public class HFile { // bunch of data w/o regard to whether decompressor is coming to end of a // decompression. InputStream is = this.compressAlgo.createDecompressionStream( - new BoundedRangeFileInputStream(this.istream, offset, compressedSize, - pread), - decompressor, 0); + new BufferedInputStream( + new BoundedRangeFileInputStream(this.istream, offset, compressedSize, + pread), + Math.min(65536, compressedSize)), + decompressor, 0); buf = ByteBuffer.allocate(decompressedSize); IOUtils.readFully(is, buf.array(), 0, buf.capacity()); is.close();
