[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289774#comment-14289774
]
Hudson commented on HBASE-12728:
--------------------------------
FAILURE: Integrated in HBase-TRUNK #6049 (See
[https://builds.apache.org/job/HBase-TRUNK/6049/])
HBASE-12728 buffered writes substantially less useful after removal of
HTablePool (Solomon Duskis and Nick Dimiduk) (ndimiduk: rev
ab18158e6001a7f15a35679ca8fc7ff772f90e25)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestHTableWrapper.java
*
hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/PerformanceEvaluation.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java
*
hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestWithCellVisibilityLoadAndVerify.java
*
hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsReplication.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableOutputFormat.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionAdapter.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationChangingPeerRegionservers.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Connection.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientPushback.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Table.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
*
hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestLoadAndVerify.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
*
hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/BufferedMutatorExample.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestScannerWithBulkload.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
*
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionFavoredNodes.java
*
hbase-it/src/test/java/org/apache/hadoop/hbase/trace/IntegrationTestSendTraceRequests.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutator.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMaster.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableWrapper.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationWithTags.java
*
hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedListWithVisibility.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestRpcControllerFactory.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableOutputFormat.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestCloneSnapshotFromClient.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorParams.java
*
hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteTable.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreFlushSnapshotFromClient.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java
* hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RowResource.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableConfiguration.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java
*
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientNoCluster.java
> buffered writes substantially less useful after removal of HTablePool
> ---------------------------------------------------------------------
>
> Key: HBASE-12728
> URL: https://issues.apache.org/jira/browse/HBASE-12728
> Project: HBase
> Issue Type: Bug
> Components: hbase
> Affects Versions: 0.98.0
> Reporter: Aaron Beppu
> Assignee: Nick Dimiduk
> Priority: Blocker
> Fix For: 1.0.0, 2.0.0, 1.1.0
>
> Attachments: 12728.connection-owns-buffers.example.branch-1.0.patch,
> HBASE-12728-2.patch, HBASE-12728-3.patch, HBASE-12728-4.patch,
> HBASE-12728-5.patch, HBASE-12728-6.patch, HBASE-12728-6.patch,
> HBASE-12728.05-branch-1.0.patch, HBASE-12728.05-branch-1.patch,
> HBASE-12728.06-branch-1.0.patch, HBASE-12728.06-branch-1.patch,
> HBASE-12728.patch, bulk-mutator.patch
>
>
> In previous versions of HBase, when use of HTablePool was encouraged, HTable
> instances were long-lived in that pool, and for that reason, if autoFlush was
> set to false, the table instance could accumulate a full buffer of writes
> before a flush was triggered. Writes from the client to the cluster could
> then be substantially larger and less frequent than without buffering.
> However, when HTablePool was deprecated, the primary justification seems to
> have been that creating HTable instances is cheap, so long as the connection
> and executor service being passed to it are pre-provided. A use pattern was
> encouraged where users should create a new HTable instance for every
> operation, using an existing connection and executor service, and then close
> the table. In this pattern, buffered writes are substantially less useful;
> writes are as small and as frequent as they would have been with
> autoflush=true, except the synchronous write is moved from the operation
> itself to the table close call which immediately follows.
> More concretely :
> ```
> // Given these two helpers ...
> private HTableInterface getAutoFlushTable(String tableName) throws
> IOException {
> // (autoflush is true by default)
> return storedConnection.getTable(tableName, executorService);
> }
> private HTableInterface getBufferedTable(String tableName) throws IOException
> {
> HTableInterface table = getAutoFlushTable(tableName);
> table.setAutoFlush(false);
> return table;
> }
> // it's my contention that these two methods would behave almost identically,
> // except the first will hit a synchronous flush during the put call,
> and the second will
> // flush during the (hidden) close call on table.
> private void writeAutoFlushed(Put somePut) throws IOException {
> try (HTableInterface table = getAutoFlushTable(tableName)) {
> table.put(somePut); // will do synchronous flush
> }
> }
> private void writeBuffered(Put somePut) throws IOException {
> try (HTableInterface table = getBufferedTable(tableName)) {
> table.put(somePut);
> } // auto-close will trigger synchronous flush
> }
> ```
> For buffered writes to actually provide a performance benefit to users, one
> of two things must happen:
> - The writeBuffer itself shouldn't live, flush and die with the lifecycle of
> it's HTableInstance. If the writeBuffer were managed elsewhere and had a long
> lifespan, this could cease to be an issue. However, if the same writeBuffer
> is appended to by multiple tables, then some additional concurrency control
> will be needed around it.
> - Alternatively, there should be some pattern for having long-lived HTable
> instances. However, since HTable is not thread-safe, we'd need multiple
> instances, and a mechanism for leasing them out safely -- which sure sounds a
> lot like the old HTablePool to me.
> See discussion on mailing list here :
> http://mail-archives.apache.org/mod_mbox/hbase-user/201412.mbox/%3CCAPdJLkEzmUQZ_kvD%3D8mrxi4V%3DhCmUp3g9MUZsddD%2Bmon%2BAvNtg%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
