[
https://issues.apache.org/jira/browse/HBASE-4608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192025#comment-13192025
]
[email protected] commented on HBASE-4608:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2740/
-----------------------------------------------------------
(Updated 2012-01-24 09:00:37.768707)
Review request for hbase, Eli Collins and Todd Lipcon.
Summary
-------
HLog compression. Has unit tests and a command line tool for
compressing/decompressing.
This addresses bug HBase-4608.
https://issues.apache.org/jira/browse/HBase-4608
Diffs (updated)
-----
CHANGES.txt 1d7238e
bin/hbase 350abef
bin/hbase-daemon.sh 5c42ac1
dev-support/findHangingTest.sh PRE-CREATION
pom.xml 6566a1c
src/docbkx/book.xml c67ca06
src/docbkx/configuration.xml 7fd90e7
src/docbkx/ops_mgt.xml f93c9f2
src/docbkx/performance.xml e61248f
src/docbkx/preface.xml 10fa755
src/docbkx/troubleshooting.xml 0b7c93a
src/docbkx/upgrading.xml c0642f5
src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon 24caabd
src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java 0477be8
src/main/java/org/apache/hadoop/hbase/HConstants.java 5120a3c
src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java 8ec5042
src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 6cdeec1
src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/Delete.java 51bbc63
src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 8cd9bd0
src/main/java/org/apache/hadoop/hbase/client/HConnection.java 0e78d96
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 852a810
src/main/java/org/apache/hadoop/hbase/client/HTable.java 839d79b
src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java 0bc9577
src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 4135e55
src/main/java/org/apache/hadoop/hbase/client/RowMutation.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java 9b568e3
src/main/java/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.java
0d4a9e4
src/main/java/org/apache/hadoop/hbase/coprocessor/AggregateImplementation.java
ba3414d
src/main/java/org/apache/hadoop/hbase/coprocessor/AggregateProtocol.java
f25ba11
src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
b47423c
src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 9002a0f
src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java 3ad6cd5
src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 07ddbca
src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 4327a44
src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java 39c73f5
src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
bd574b2
src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormat.java 3dcbf74
src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java e6f8a6e
src/main/java/org/apache/hadoop/hbase/master/HMaster.java cb2f084
src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java 89685bb
src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 3938fa7
src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 9de1784
src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java 667a8b1
src/main/java/org/apache/hadoop/hbase/master/handler/ClosedRegionHandler.java
2dfc3e7
src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java
2dd497b
src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java
493dcdb
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java fb4ec05
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3917d40
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java
18b6c13
src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java c840e7c
src/main/java/org/apache/hadoop/hbase/regionserver/OperationStatus.java
b6f7456
src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
7cee17c
src/main/java/org/apache/hadoop/hbase/regionserver/SplitRequest.java 41f5dff
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java b928731
src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java bd6f70d
src/main/java/org/apache/hadoop/hbase/regionserver/handler/CloseMetaHandler.java
e8e95ed
src/main/java/org/apache/hadoop/hbase/regionserver/handler/CloseRegionHandler.java
a25ca32
src/main/java/org/apache/hadoop/hbase/regionserver/handler/CloseRootHandler.java
fa38ad6
src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
490694c
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
97dd8e6
src/main/java/org/apache/hadoop/hbase/regionserver/wal/CompressedKeyValue.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/wal/CompressionContext.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/wal/Compressor.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 24407af
src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java f067221
src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
43bfba0
src/main/java/org/apache/hadoop/hbase/regionserver/wal/LRUDictionary.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
d9cd6de
src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogWriter.java
cbef70f
src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALDictionary.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java e1117ef
src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
7fe0ae5
src/main/java/org/apache/hadoop/hbase/rest/MultiRowResource.java 2ba6a0d
src/main/java/org/apache/hadoop/hbase/rest/RowResource.java dade6a8
src/main/java/org/apache/hadoop/hbase/rest/TableResource.java cc719bc
src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java f93c81d
src/main/java/org/apache/hadoop/hbase/rest/transform/Base64.java f991121
src/main/java/org/apache/hadoop/hbase/rest/transform/NullTransform.java
8492cc6
src/main/java/org/apache/hadoop/hbase/rest/transform/Transform.java 9f33bab
src/main/java/org/apache/hadoop/hbase/thrift/HThreadedSelectorServerArgs.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/thrift/TBoundedThreadPoolServer.java
690a57f
src/main/java/org/apache/hadoop/hbase/thrift/ThriftServer.java 3fa5d41
src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/thrift/generated/AlreadyExists.java
0479e31
src/main/java/org/apache/hadoop/hbase/thrift/generated/BatchMutation.java
be902c9
src/main/java/org/apache/hadoop/hbase/thrift/generated/ColumnDescriptor.java
04b42fe
src/main/java/org/apache/hadoop/hbase/thrift/generated/Hbase.java 9e31c61
src/main/java/org/apache/hadoop/hbase/thrift/generated/IOError.java 778e869
src/main/java/org/apache/hadoop/hbase/thrift/generated/IllegalArgument.java
9ae5340
src/main/java/org/apache/hadoop/hbase/thrift/generated/Mutation.java 7aa9bcd
src/main/java/org/apache/hadoop/hbase/thrift/generated/TCell.java ed420d4
src/main/java/org/apache/hadoop/hbase/thrift/generated/TRegionInfo.java
161dedc
src/main/java/org/apache/hadoop/hbase/thrift/generated/TRowResult.java
0f31e5e
src/main/java/org/apache/hadoop/hbase/thrift/generated/TScan.java 3b894db
src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumn.java 3e116e7
src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumnIncrement.java
8390015
src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumnValue.java
424a87b
src/main/java/org/apache/hadoop/hbase/thrift2/generated/TDelete.java 68b4f8e
src/main/java/org/apache/hadoop/hbase/thrift2/generated/TDeleteType.java
2abdee0
src/main/java/org/apache/hadoop/hbase/thrift2/generated/TGet.java b1a1a12
src/main/java/org/apache/hadoop/hbase/thrift2/generated/THBaseService.java
272a4a5
src/main/java/org/apache/hadoop/hbase/thrift2/generated/TIOError.java 283d430
src/main/java/org/apache/hadoop/hbase/thrift2/generated/TIllegalArgument.java
254fbe5
src/main/java/org/apache/hadoop/hbase/thrift2/generated/TIncrement.java
3cc82e9
src/main/java/org/apache/hadoop/hbase/thrift2/generated/TPut.java 97ab5dc
src/main/java/org/apache/hadoop/hbase/thrift2/generated/TResult.java 73c8340
src/main/java/org/apache/hadoop/hbase/thrift2/generated/TScan.java d76c355
src/main/java/org/apache/hadoop/hbase/thrift2/generated/TTimeRange.java
ad9fdc7
src/main/java/org/apache/hadoop/hbase/util/SoftValueSortedMap.java 11dfbef
src/main/java/org/apache/hadoop/hbase/util/Threads.java 6f81b62
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java 9b83840
src/main/resources/hbase-webapps/static/favicon.ico PRE-CREATION
src/main/resources/hbase-webapps/static/hbase_logo.png 03fa793
src/site/resources/images/favicon.ico 161bcf7
src/site/resources/images/hbase_logo.png 03fa793
src/site/resources/images/hbase_logo.svg PRE-CREATION
src/site/resources/images/hbase_logo_med.gif 36d3e3c
src/site/resources/images/hbase_small.gif 3275765
src/site/xdoc/index.xml 9157d6a
src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java dada051
src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java
80d69b4
src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java
c1a077f
src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java bb077d0
src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java ab80020
src/test/java/org/apache/hadoop/hbase/coprocessor/TestAggregateProtocol.java
0d38ac9
src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
5b64895
src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java
5e3e994
src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScan.java
46e1bee
src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
cc0f30f
src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java c359f4b
src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
a348f0c
src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java 32ad7e8
src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
42db18b
src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java
0a34371
src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java
64e61bb
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java 2d87567
src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java
853a35f
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLRUDictionary.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
6e89cc4
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
59910bf
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplayCompressed.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/rest/TestTransform.java 2e2ba4c
src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java 12247d0
src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServerCmdLine.java
477141f
src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java
0b45ac1
Diff: https://reviews.apache.org/r/2740/diff
Testing
-------
Thanks,
Li
> HLog Compression
> ----------------
>
> Key: HBASE-4608
> URL: https://issues.apache.org/jira/browse/HBASE-4608
> Project: HBase
> Issue Type: New Feature
> Reporter: Li Pi
> Assignee: Li Pi
> Attachments: 4608v1.txt, 4608v5.txt, 4608v6.txt, 4608v7.txt,
> 4608v8fixed.txt
>
>
> The current bottleneck to HBase write speed is replicating the WAL appends
> across different datanodes. We can speed up this process by compressing the
> HLog. Current plan involves using a dictionary to compress table name, region
> id, cf name, and possibly other bits of repeated data. Also, HLog format may
> be changed in other ways to produce a smaller HLog.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
