[
https://issues.apache.org/jira/browse/HBASE-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HBASE-4686:
-------------------------------
Attachment: D87.4.patch
mbautin updated the revision "[jira] [HBASE-4686] [89-fb] Fix per-store metrics
aggregation
".
Reviewers: Liyin, JIRA
Added another load test in a unit test with encoding turned on. That found
some bugs similar to those that I observed in cluster testing, and I added a
smaller test reproducing the same bugs (TestEncodedSeekers). Fixed the bugs by
correctly restoring additional state when going to previous key/value
(previously, only the vanilla BufferedDataBlockEncoder.SeekerState was restored
but not algorithm-specific state). I also had to remove BitsetKeyDeltaEncoder
for now because I could not fix its encoded seeker yet (it seems to have some
more complicated bugs) but we are not planning to use that algorithm for now
Also, fixed the most recent comment by Ted and
TestHFileBlock.testBlockHeapSize failure on a 32-bit JVM (thanks to Ted for
pointing that one out as well).
REVISION DETAIL
https://reviews.facebook.net/D87
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
src/main/java/org/apache/hadoop/hbase/HConstants.java
src/main/java/org/apache/hadoop/hbase/KeyValue.java
src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java
src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
src/main/java/org/apache/hadoop/hbase/io/encoding/CompressionState.java
src/main/java/org/apache/hadoop/hbase/io/encoding/CopyKeyDataBlockEncoder.java
src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoding.java
src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java
src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java
src/main/java/org/apache/hadoop/hbase/io/encoding/EncoderBufferTooSmallException.java
src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java
src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java
src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoder.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileDataBlockEncoderImpl.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
src/main/java/org/apache/hadoop/hbase/io/hfile/NoOpDataBlockEncoder.java
src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaConfigured.java
src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java
src/main/ruby/hbase/admin.rb
src/test/java/org/apache/hadoop/hbase/BROKE_TODO_FIX_TestAcidGuarantees.java
src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java
src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java
src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java
src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java
src/test/java/org/apache/hadoop/hbase/io/encoding/RedundantKVGenerator.java
src/test/java/org/apache/hadoop/hbase/io/encoding/TestBufferedDataBlockEncoder.java
src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java
src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileDataBlockEncoder.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
src/test/java/org/apache/hadoop/hbase/regionserver/CreateRandomStoreFile.java
src/test/java/org/apache/hadoop/hbase/regionserver/DataBlockEncodingTool.java
src/test/java/org/apache/hadoop/hbase/regionserver/EncodedSeekPerformanceTest.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestSeekOptimizations.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java
src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
src/test/java/org/apache/hadoop/hbase/util/TestByteBufferUtils.java
src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadEncoded.java
src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java
src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
> [89-fb] Fix per-store metrics aggregation
> ------------------------------------------
>
> Key: HBASE-4686
> URL: https://issues.apache.org/jira/browse/HBASE-4686
> Project: HBase
> Issue Type: Bug
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Attachments: D87.1.patch, D87.2.patch, D87.3.patch, D87.4.patch,
> HBASE-4686-TestRegionServerMetics-and-Store-metric-a-20111027134023-cc718144.patch,
>
> HBASE-4686-jira-89-fb-Fix-per-store-metrics-aggregat-20111027152723-05bea421.patch
>
>
> In r1182034 per-Store metrics were broken, because the aggregation of
> StoreFile metrics over all stores in a region was replaced by overriding them
> every time. We saw these metrics drop by a factor of numRegions on a
> production cluster -- thanks to Kannan for noticing this! We need to fix the
> metrics and add a unit test to ensure regressions like this don't happen in
> the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
