[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131128#comment-16131128 ] Hudson commented on HBASE-18500: FAILURE: Integrated in Jenkins build HBASE-14070.HLC #233 (See [https://builds.apache.org/job/HBASE-14070.HLC/233/]) HBASE-18500 Performance issue: Don't use BufferedMutator for HTable's (zghao: rev cabdbf181a0b44bd1f9c32aa67b8a2ee3b863758) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestNamespaceCommands.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Table.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestHTableWrapper.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/RegionAsTable.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableBuilderBase.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientPushback.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestServerBusyException.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableBuilder.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableWrapper.java * (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/constraint/TestConstraint.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMultiSlaveReplication.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java * (edit) hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0, 3.0.0 > > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, > HBASE-18500-v5.patch, HBASE-18500-v5.patch, HBASE-18500-v6.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For t
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123305#comment-16123305 ] Hudson commented on HBASE-18500: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3512 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3512/]) HBASE-18500 Performance issue: Don't use BufferedMutator for HTable's (zghao: rev cabdbf181a0b44bd1f9c32aa67b8a2ee3b863758) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Table.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableWrapper.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/RegionAsTable.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMultiSlaveReplication.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableBuilderBase.java * (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientPushback.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestNamespaceCommands.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableBuilder.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/constraint/TestConstraint.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestHTableWrapper.java * (edit) hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestServerBusyException.java > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0, 3.0.0 > > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, > HBASE-18500-v5.patch, HBASE-18500-v5.patch, HBASE-18500-v6.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of write
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123254#comment-16123254 ] Hudson commented on HBASE-18500: FAILURE: Integrated in Jenkins build HBase-2.0 #309 (See [https://builds.apache.org/job/HBase-2.0/309/]) HBASE-18500 Performance issue: Don't use BufferedMutator for HTable's (zghao: rev 0c16bb591b165050439cf945aca19fa708e0b350) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientPushback.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMultiSlaveReplication.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Table.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/RegionAsTable.java * (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableWrapper.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/constraint/TestConstraint.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * (edit) hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestNamespaceCommands.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableBuilder.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestHTableWrapper.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestServerBusyException.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableBuilderBase.java > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0, 3.0.0 > > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, > HBASE-18500-v5.patch, HBASE-18500-v5.patch, HBASE-18500-v6.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performan
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123122#comment-16123122 ] Guanghao Zhang commented on HBASE-18500: Ok. Let me fill new issues for the above problems. :-) > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0, 3.0.0 > > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, > HBASE-18500-v5.patch, HBASE-18500-v5.patch, HBASE-18500-v6.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. > Review: https://reviews.apache.org/r/61454/ > Copy the comments from [~chia7712]. Remove the BufferdMutator brings four > benefits. > 1. correct the metrics (see HBASE-18476) > 2. make HTable thread-safe (see HBASE-17368) > 3. reduce the latency > 4. get rid of some deprecated methods in Table -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123114#comment-16123114 ] Anoop Sam John commented on HBASE-18500: Do we have an issue for the RpcContext inconsistency? Will you raise [~zghaobac]? > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0, 3.0.0 > > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, > HBASE-18500-v5.patch, HBASE-18500-v5.patch, HBASE-18500-v6.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. > Review: https://reviews.apache.org/r/61454/ > Copy the comments from [~chia7712]. Remove the BufferdMutator brings four > benefits. > 1. correct the metrics (see HBASE-18476) > 2. make HTable thread-safe (see HBASE-17368) > 3. reduce the latency > 4. get rid of some deprecated methods in Table -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123079#comment-16123079 ] Chia-Ping Tsai commented on HBASE-18500: bq. suggest to also push the fix into branch-1. The HTableInterface has some methods related to BF, so it is difficult to remove the BF for branch-1. > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0, 3.0.0 > > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, > HBASE-18500-v5.patch, HBASE-18500-v5.patch, HBASE-18500-v6.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. > Review: https://reviews.apache.org/r/61454/ > Copy the comments from [~chia7712]. Remove the BufferdMutator brings four > benefits. > 1. correct the metrics (see HBASE-18476) > 2. make HTable thread-safe (see HBASE-17368) > 3. reduce the latency > 4. get rid of some deprecated methods in Table -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123075#comment-16123075 ] Yu Li commented on HBASE-18500: --- I'm afraid this is not only issue for master/branch-2 but also for branch-1 since BufferedMutator is also used there, and suggest to also push the fix into branch-1. Better if we could do some verification on branch-1 performance of course (smile). > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0, 3.0.0 > > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, > HBASE-18500-v5.patch, HBASE-18500-v5.patch, HBASE-18500-v6.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. > Review: https://reviews.apache.org/r/61454/ > Copy the comments from [~chia7712]. Remove the BufferdMutator brings four > benefits. > 1. correct the metrics (see HBASE-18476) > 2. make HTable thread-safe (see HBASE-17368) > 3. reduce the latency > 4. get rid of some deprecated methods in Table -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122961#comment-16122961 ] Hadoop QA commented on HBASE-18500: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 13s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 17 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 39s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 47s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 11s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 40s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 43s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 30m 28s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 25s{color} | {color:green} hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green}107m 57s{color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 54s{color} | {color:green} hbase-rest in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 46s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}182m 27s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:bdc94b1 | | JIRA Issue | HBASE-18500 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881386/HBASE-18500-v6.patch | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 7eaafc17be8c 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 8da77b4 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC3 | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/8028/testReport/ | | modules | C: hbase-client hbase-server hbase-rest U: . | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122763#comment-16122763 ] Guanghao Zhang commented on HBASE-18500: bq. We should address it via some other jira. As of here, just do as what u did in patch with a TODO to fix the issue. Will add a TODO when commit it. Thanks all for reviewing. > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, > HBASE-18500-v5.patch, HBASE-18500-v5.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. > Review: https://reviews.apache.org/r/61454/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122740#comment-16122740 ] Guanghao Zhang commented on HBASE-18500: Thanks for your explanation. bq. The table on which put was called is created via the RegionCP env and that uses a special Connection. Checked the code, I thought this was introduced by HBASE-9534. bq. For #1 we may have to use InheritableThreadLocal instead of ThreadLocal. Agreed. But I don't know if it has any other side-effects. bq. When doing a runAsUser, how/whether we should reset the Rpc context to have the new user. Do we have other scenario which need reset the Rpc context? If not, I thought we can only fixed this for this problem. We can introduce a new config or method to disable the "Short-Circuit Coprocessor HTable access" feature. The code may be: {code} User.runAsLoginUser(new PrivilegedExceptionAction() { @Override public Void run() throws Exception { // disable the "Short-Circuit Coprocessor HTable access" feature first // Then start a new rpc call AccessControlLists.addUserPermission(regionEnv.getConfiguration(), perm, regionEnv.getTable(AccessControlLists.ACL_TABLE_NAME), request.getMergeExistingPermissions()); return null; } }); {code} > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, > HBASE-18500-v5.patch, HBASE-18500-v5.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. > Review: https://reviews.apache.org/r/61454/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121917#comment-16121917 ] Anoop Sam John commented on HBASE-18500: For #1 we may have to use InheritableThreadLocal instead of ThreadLocal (?) > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, > HBASE-18500-v5.patch, HBASE-18500-v5.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. > Review: https://reviews.apache.org/r/61454/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121630#comment-16121630 ] Anoop Sam John commented on HBASE-18500: Thanks for the nice debug. Things are clear now. This Table.put() happens within the RS. We are already in AccessController. Means already there was an RPC happened to server (Actual grant call from client side). At RpcServer we have a ThreadLocal where we keep the CallContext and inside that the current RPC called user info is set. The table on which put was called is created via the RegionCP env and that uses a special Connection. The normal RPC channel will be by passed here means there would have no further contact on to the RpcServer. So the ThreadLocal is never getting reset. We ran the new put as a super user (User.runAsLoginUser where the login user is the user who started RS process) but still as per the RPC context it is the old user. When this patch changed the Table.put(Put) flow so that there is no more AsyncProcess involvement, the same thread executing the put. So the Rpc context ThreadLocal is still accessible and we see the user as old Rpc user. When AsyncProcess was used, the execute happen via another thread from pool and so old ThreadLocal variable is not accessible and so it looks as if no Rpc context and we were relying on the super user who starts the RS process. We can solve this particular issue in diff ways. But this investigation raises few Qs. 1. We provide CPs a way to make Table ops. The Table can be obtained from the env and may be the flow will get RPC bypassed if the calling region on same RS. In that case if the Table.put(Put) is called we will see the old RPC user in the second region call path also. But if it was AsyncProcess involved flow, we wont see. How/whether we should make possible to get the Rpc context in the second region flow also? To know the user. And we have to solve this incosistency based on API usage any way. 2. When doing a runAsUser, how/whether we should reset the Rpc context to have the new user. We should address it via some other jira. As of here, just do as what u did in patch with a TODO to fix the issue. > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, > HBASE-18500-v5.patch, HBASE-18500-v5.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. > Review: https://reviews.apache.org/r/61454/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121382#comment-16121382 ] Guanghao Zhang commented on HBASE-18500: AccessController.grant method will check permission first, then use runAsLoginUser to put a record to ACL table. {code} switch(request.getUserPermission().getPermission().getType()) { case Global : case Table : requirePermission(caller, "grant", perm.getTableName(), perm.getFamily(), perm.getQualifier(), Action.ADMIN); break; case Namespace : requireNamespacePermission(caller, "grant", perm.getNamespace(), Action.ADMIN); break; } User.runAsLoginUser(new PrivilegedExceptionAction() { @Override public Void run() throws Exception { AccessControlLists.addUserPermission(regionEnv.getConfiguration(), perm, regionEnv.getTable(AccessControlLists.ACL_TABLE_NAME), request.getMergeExistingPermissions()); return null; } }); {code} When put record to ACL, it will check permission, too. And it will get active user from Context first, then get the logged user. {code} private User getActiveUser(ObserverContext ctx) throws IOException { User user = ctx.getCaller(); if (user == null) { // for non-rpc handling, fallback to system user user = userProvider.getCurrent(); } return user; } {code} And when we use table.put(Put) to write record to ACL, I found it still print log in a same thread. Then it will use context user to require permission. When use put(List), it will use AsyncProcess and will run it in a new thread. Then the context user is null, it will use the logged user to require permission. I thought the problem is the runAsLoginUser method didn't change the context user when it run in a same thread. But I don't have a good idea to fix this. Any ideas about this? > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, > HBASE-18500-v5.patch, HBASE-18500-v5.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. > Review: https://reviews.apache.org/r/61454/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1612#comment-1612 ] Guanghao Zhang commented on HBASE-18500: bq. Could you leave above comment in the right place? Or file a issue. Let me dig it more. > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, > HBASE-18500-v5.patch, HBASE-18500-v5.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. > Review: https://reviews.apache.org/r/61454/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121102#comment-16121102 ] Chia-Ping Tsai commented on HBASE-18500: {quote} When grant user permission, it will put a record to ACL. The old implementation use BufferMutator, so it will send a multi request. And use AccessController.preBatchMutate to check permission when put a record. When HTable doesn't use BufferMutator, it will send a mutate request and use AccessController.prePut to check permission. So I change to use put(List), this will send multi request, too. Then it will same with before. I thought AccessController may have some bugs which need to dig more. But it is not related to this issue? {quote} Could you leave above comment in the right place? Or file a issue. {quote} The default retries number is 35. So it will wait a long time when there are exception in ut. I thought we should set this to the default HBaseTestingUtility config in the future. {quote} Got it. +1 > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, > HBASE-18500-v5.patch, HBASE-18500-v5.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. > Review: https://reviews.apache.org/r/61454/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121087#comment-16121087 ] Guanghao Zhang commented on HBASE-18500: bq. t.put(Collections.singletonList(p)); When grant user permission, it will put a record to ACL. The old implementation use BufferMutator, so it will send a multi request. And use AccessControlLists.preBatchMutate to check permission when put a record. When HTable doesn't use BufferMutator, it will send a mutate request and use AccessControlLists.prePut to check permission. So I change to use put(List), this will send multi request, too. Then it will same with before. I thought AccessControlLists may have some bugs which need to dig more. But it is not related to this issue? bq. UTIL.getConfiguration().setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 2); The default retries number is 35. So it will wait a long time when there are exception in ut. I thought we should set this to the default HBaseTestingUtility config in the future. > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, > HBASE-18500-v5.patch, HBASE-18500-v5.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. > Review: https://reviews.apache.org/r/61454/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121080#comment-16121080 ] ramkrishna.s.vasudevan commented on HBASE-18500: LGTM but for the above comments from [~chia7712]. If he is fine then we can go for commit. Release note is needed I believe? > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, > HBASE-18500-v5.patch, HBASE-18500-v5.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. > Review: https://reviews.apache.org/r/61454/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120975#comment-16120975 ] Chia-Ping Tsai commented on HBASE-18500: {code} try { - t.put(p); + t.put(Collections.singletonList(p)); } finally { t.close(); } {code} {code} +UTIL.getConfiguration().setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 2); {code} Why do we need these changes? > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, > HBASE-18500-v5.patch, HBASE-18500-v5.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. > Review: https://reviews.apache.org/r/61454/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120169#comment-16120169 ] Hadoop QA commented on HBASE-18500: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 17 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 35s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 39s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 16s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 42s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 20s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 31m 25s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 30s{color} | {color:green} hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green}134m 40s{color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 13s{color} | {color:green} hbase-rest in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 49s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}198m 48s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:bdc94b1 | | JIRA Issue | HBASE-18500 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881012/HBASE-18500-v5.patch | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux f37008a3612d 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / f314b59 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC3 | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/7999/testReport/ | | modules | C: hbase-client hbase-server hbase-rest U: . | | Console output | https://builds.apache.org/job/PreCommit-HBA
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119464#comment-16119464 ] Hadoop QA commented on HBASE-18500: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 17 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 3s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 33s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 27s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 49s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 10s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 30m 5s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 28s{color} | {color:green} hbase-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}109m 49s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 58s{color} | {color:green} hbase-rest in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 44s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}172m 14s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.security.token.TestZKSecretWatcher | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:bdc94b1 | | JIRA Issue | HBASE-18500 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12880952/HBASE-18500-v5.patch | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 7e5d37f0bc36 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / f314b59 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC3 | | unit | https://builds.apache.org/job/PreCommit-HBASE-Build/7996/artifact/patchprocess/patch-unit-hbase-server.txt | | Test
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117857#comment-16117857 ] Hadoop QA commented on HBASE-18500: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 17 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 30s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 1s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 23s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 17s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 46s{color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 43s{color} | {color:red} hbase-rest in master has 3 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 33m 26s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 28s{color} | {color:green} hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green}109m 1s{color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 59s{color} | {color:green} hbase-rest in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 45s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}176m 30s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:bdc94b1 | | JIRA Issue | HBASE-18500 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12880767/HBASE-18500-v5.patch | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 6ed0a3a3ed0e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 7e7461e | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC3 | | findbugs | https://builds.apache.org/job/PreCommit-HBASE-Build/7969/artifact/patchprocess/branch-findbugs-hbase-rest-warnings.html | | Test Results | https://builds.apache.org/job/Pre
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117087#comment-16117087 ] Chia-Ping Tsai commented on HBASE-18500: bq. Was it before 1.3 itself? Yes, see HBASE-2898 > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch, HBASE-18500-v4.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. > Review: https://reviews.apache.org/r/61454/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116603#comment-16116603 ] Hadoop QA commented on HBASE-18500: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 17 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 32s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 16s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 16s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 41s{color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 34s{color} | {color:red} hbase-rest in master has 3 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 31m 36s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 3s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 20s{color} | {color:red} hbase-client generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 37s{color} | {color:green} hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green}111m 31s{color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 2s{color} | {color:green} hbase-rest in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 45s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}174m 24s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:bdc94b1 | | JIRA Issue | HBASE-18500 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12880624/HBASE-18500-v4.patch | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 115500b98c50 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / fd76eb3 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC3 | | findbugs | https://builds.apache.org/job/PreCommit-HBASE-Build/7960/artifact/patchprocess/branch-findbugs-hbase-rest-warnings.html | | java
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116418#comment-16116418 ] Guanghao Zhang commented on HBASE-18500: Review: https://reviews.apache.org/r/61454/ > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch, HBASE-18500-v4.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. > Review: https://reviews.apache.org/r/61454/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116328#comment-16116328 ] Hadoop QA commented on HBASE-18500: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 16 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 27s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 14s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 42s{color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 34s{color} | {color:red} hbase-rest in master has 3 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 31m 22s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 1s{color} | {color:red} hbase-client generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 18s{color} | {color:red} hbase-client generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 33s{color} | {color:green} hbase-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}110m 23s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 1s{color} | {color:green} hbase-rest in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 44s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}172m 49s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hbase-client | | | Dead store to response in org.apache.hadoop.hbase.client.HTable$3.rpcCall() At HTable.java:org.apache.hadoop.hbase.client.HTable$3.rpcCall() At HTable.java:[line 585] | | | Unread public/protected field:At TableBuilderBase.java:[line 51] | | Failed junit tests | hadoop.hbase.constraint.TestConstraint | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:bdc94b1 | | JIRA Issue | HBASE-18500 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12880596/HBASE-18500-v3.patch | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 79fe3c627abd 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116256#comment-16116256 ] ramkrishna.s.vasudevan commented on HBASE-18500: When was the delete API changed for modifying the passed input user list? Was it before 1.3 itself? > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116189#comment-16116189 ] Chia-Ping Tsai commented on HBASE-18500: I skim the user@mail, and there is no recently wish about "re-submit only the failed puts". We can leave a TODO in Put(). Or add an new methods which do the same way with delete. > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116183#comment-16116183 ] Guanghao Zhang commented on HBASE-18500: bq. So now we will change the List that user has passed in? The delete(List) API already did this: remove the success records and keep the failed records.. The key point is to find a proper way to let user figure out the failed records. Maybe return the failed records list? > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116143#comment-16116143 ] Anoop Sam John commented on HBASE-18500: So now we will change the List that user has passed in? May be better if we avoid that.. > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116138#comment-16116138 ] Chia-Ping Tsai commented on HBASE-18500: bq. I thought we can do this incompatibility change in hbase 2.0? I feel the same way. We need to announce this change as the best as we can. > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116131#comment-16116131 ] Guanghao Zhang commented on HBASE-18500: bq. The broken compatibility is caused by that we did not modify the input list before. I thought we can do this incompatibility change in hbase 2.0? > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, > HBASE-18500-v3.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115706#comment-16115706 ] Chia-Ping Tsai commented on HBASE-18500: bq. How you mean It is better to make the behavior of Table#put consistent with Table#delete. What that means is we should remove the succeed PUTs from list. The broken compatibility is caused by that we did not modify the input list before. Personally, i prefer removed the succeed PUTs because that make Table#put more easier to exercise. We had good discussion in HBASE-13271. > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115550#comment-16115550 ] stack commented on HBASE-18500: --- +1 on the removals. They have done a proper deprecation cycle. This change should be marked incompatible though if only to have it show up in list of possibly breaking changes when someone searches. bq. We should check the results in order to find the failed PUTs, but the change breaks the operational compatibility... How you mean [~chia7712] (BTW, nice summary of benefits regards removal of BM from HT)? What if we created a BM in side in Puts and called its flush... then operation would be as it was before? Thanks. > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114559#comment-16114559 ] Hadoop QA commented on HBASE-18500: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 14 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 40s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 12s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 34s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 31s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 50s{color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 42s{color} | {color:red} hbase-rest in master has 3 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 38m 10s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 19s{color} | {color:red} hbase-client generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 22s{color} | {color:red} hbase-client generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 45s{color} | {color:green} hbase-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 49s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 54s{color} | {color:green} hbase-rest in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m 22s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}163m 3s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hbase-client | | | Dead store to response in org.apache.hadoop.hbase.client.HTable$3.rpcCall() At HTable.java:org.apache.hadoop.hbase.client.HTable$3.rpcCall() At HTable.java:[line 585] | | | Unread public/protected field:At TableBuilderBase.java:[line 51] | | Failed junit tests | hadoop.hbase.security.access.TestNamespaceCommands | | | hadoop.hbase.security.access.TestAccessController | | | hadoop.hbase.client.TestClientPushback | | | hadoop.hbase.client.TestHCM | | Timed out junit tests | org.apache.hadoop.hbase.master.procedure.TestEnableTableProcedure | | | org.apache.hadoop.hbase.master.procedure.TestDeleteTableProcedure | | | org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure | | | org.apache.hadoop.hbase.regionse
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114407#comment-16114407 ] Chia-Ping Tsai commented on HBASE-18500: {code} @Override public void put(final List puts) throws IOException { -getBufferedMutator().mutate(puts); -flushCommits(); +for (Put put : puts) { + validatePut(put); +} +Object[] results = new Object[puts.size()]; +try { + batch(puts, results, writeRpcTimeout); +} catch (InterruptedException e) { + throw (InterruptedIOException) new InterruptedIOException().initCause(e); +} {code} We should check the results in order to find the failed PUTs, but the change breaks the operational compatibility... > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114152#comment-16114152 ] Chia-Ping Tsai commented on HBASE-18500: bq. I thought this should be done in hbase-2.0 +1 Consider changing the type (bug -> task or improvement) > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114131#comment-16114131 ] Guanghao Zhang commented on HBASE-18500: bq. So this can be done in hbase-2.0 itself or in hbase-3.0? I thought this should be done in hbase-2.0. And for hbase-3.0, the blocking client implementation can be removed and wrapped by the async client (See HBASE-18334). > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114117#comment-16114117 ] ramkrishna.s.vasudevan commented on HBASE-18500: bq.I agree completely because it brings four benefits. +1 on seeing the above comment. So this can be done in hbase-2.0 itself or in hbase-3.0? > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113847#comment-16113847 ] Chia-Ping Tsai commented on HBASE-18500: bq. I thought we can remove the BufferedMutator from Table. I agree completely because it brings four benefits. # correct the metrics (see HBASE-18476) # make HTable thread-safe (see HBASE-17368) # reduce the latency # get rid of some deprecated methods in Table > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113836#comment-16113836 ] Guanghao Zhang commented on HBASE-18500: I thought we can remove the BufferedMutator from Table. Then Table is used for no buffered write and BufferedMutator used for the buffered write. [~anoop.hbase] [~ram_krish] [~chia7712] [~stack] What do you think about this? > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110583#comment-16110583 ] Guanghao Zhang commented on HBASE-18500: bq. This fix is applicable which all branches? In master we do like what is done in patch now? I thought this should be fixed in all branch-1, branch-2 and master. For hbase 3.0, we can remove the sync code and reimplement it by the async interface. > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110582#comment-16110582 ] Hadoop QA commented on HBASE-18500: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 33s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 28s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 47s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 25s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 38s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 31m 1s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 12s{color} | {color:red} hbase-client generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 38s{color} | {color:red} hbase-client in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}121m 4s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 45s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}175m 44s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hbase-client | | | Dead store to response in org.apache.hadoop.hbase.client.HTable$3.rpcCall() At HTable.java:org.apache.hadoop.hbase.client.HTable$3.rpcCall() At HTable.java:[line 615] | | Failed junit tests | hadoop.hbase.client.TestAsyncProcess | | | hadoop.hbase.security.access.TestNamespaceCommands | | | hadoop.hbase.client.TestServerBusyException | | | hadoop.hbase.client.TestFromClientSide | | | hadoop.hbase.client.TestFromClientSideWithCoprocessor | | | hadoop.hbase.client.TestClientPushback | | | hadoop.hbase.client.TestHCM | | | hadoop.hbase.security.access.TestAccessController | | | hadoop.hbase.constraint.TestConstraint | | Timed out junit tests | org.apache.hadoop.hbase.master.TestMasterMetricsWrapper | | | org.apache.hadoop.hbase.master.procedure.TestDisableTableProcedure | | | org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure | | | org.apache.hadoop.hbase.master.TestGetLastFlushedSequenceId | | | org.apache.hadoop.hbase.m
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110546#comment-16110546 ] Guanghao Zhang commented on HBASE-18500: bq. Can we get rid of BUFFER for Table? Agreed. Table is used for no buffered write and BufferedMutator for the buffered write. > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110538#comment-16110538 ] Chia-Ping Tsai commented on HBASE-18500: We have some related issues. HBASE-17368, HBASE-18334, and HBASE-18476 Can we get rid of *BUFFER* for Table? > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110472#comment-16110472 ] ramkrishna.s.vasudevan commented on HBASE-18500: I think YCSB works like this only in case we need to measure write performance like latency. It was having clientSidebuffering and if that is false it will call HTable.put() directly. Probably this patch is needed if YCSB needs to support the same with hbase-2.0. > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110459#comment-16110459 ] Abhishek Singh Chouhan commented on HBASE-18500: Similar to HBASE-18476 i think. > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110410#comment-16110410 ] Anoop Sam John commented on HBASE-18500: This fix is applicable which all branches? In master we do like what is done in patch now? > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method
[ https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110373#comment-16110373 ] Guanghao Zhang commented on HBASE-18500: {code} ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 --nomapred --autoFlush=True randomWrite 1 {code} Mean latency(us). || || Test1 || Test2 || Test3 || Test4 || Test5 || | w/o patch | 153.70 | 158.64 | 168.57 | 152.04 | 147.29 | | w/ v1 patch | 111.03 | 115.34 | 120.70 | 124.56 | 119.86 | 50th latency(us). || || Test1 || Test2 || Test3 || Test4 || Test5 || | w/o patch | 102.00 | 105.00 | 113.00 | 122.00 | 114.00 | | w/ v1 patch | 82.00 | 88.00 | 92.00 | 94.00 | 96.00 | 99th latency(us). || || Test1 || Test2 || Test3 || Test4 || Test5 || | w/o patch | 419.00 | 442.00 | 407.00 | 450.00 | 416.00 | | w/ v1 patch | 322.00 | 305.00 | 290.00 | 304.00 | 276.00 | > Performance issue: Don't use BufferedMutator for HTable's put method > > > Key: HBASE-18500 > URL: https://issues.apache.org/jira/browse/HBASE-18500 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-18500-v1.patch > > > Copied the test result from HBASE-17994. > Run start-hbase.sh in my local computer and use the default config to test > with PE tool. > {code} > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True randomWrite 1 > ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 > --nomapred --autoFlush=True asyncRandomWrite 1 > {code} > Mean latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 | > | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 | > 50th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 | > | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 | > 99th latency test result. > || || Test1 || Test2 || Test3 || Test4 || Test5 || > | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 | > | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 | > In our internal 0.98 branch, the PE test result shows the async write has the > almost same latency with the blocking write. But for master branch, the > result shows the async write has better latency than the blocking client. > Take a look about the code, I thought the difference is the BufferedMutator. > For master branch, HTable don't have a write buffer and all write request > will be flushed directly. And user can use BufferedMutator when user want to > perform client-side buffering of writes. For the performance issue > (autoFlush=True), I thought we can use rpc caller directly in HTable's put > method. Thanks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)