subject:"\[jira\] \[Commented\] \(HBASE\-18500\) Performance issue\: Don't use BufferedMutator for HTable's put method"

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-17 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131128#comment-16131128
 ] 

Hudson commented on HBASE-18500:


FAILURE: Integrated in Jenkins build HBASE-14070.HLC #233 (See 
[https://builds.apache.org/job/HBASE-14070.HLC/233/])
HBASE-18500 Performance issue: Don't use BufferedMutator for HTable's (zghao: 
rev cabdbf181a0b44bd1f9c32aa67b8a2ee3b863758)
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestNamespaceCommands.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Table.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestHTableWrapper.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/RegionAsTable.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableBuilderBase.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientPushback.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestServerBusyException.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableBuilder.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableWrapper.java
* (edit) 
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/constraint/TestConstraint.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMultiSlaveReplication.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
* (edit) 
hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java


> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, 
> HBASE-18500-v5.patch, HBASE-18500-v5.patch, HBASE-18500-v6.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-11 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123305#comment-16123305
 ] 

Hudson commented on HBASE-18500:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3512 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/3512/])
HBASE-18500 Performance issue: Don't use BufferedMutator for HTable's (zghao: 
rev cabdbf181a0b44bd1f9c32aa67b8a2ee3b863758)
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Table.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableWrapper.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/RegionAsTable.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMultiSlaveReplication.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableBuilderBase.java
* (edit) 
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientPushback.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestNamespaceCommands.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableBuilder.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/constraint/TestConstraint.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestHTableWrapper.java
* (edit) 
hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestServerBusyException.java


> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, 
> HBASE-18500-v5.patch, HBASE-18500-v5.patch, HBASE-18500-v6.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-11 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123254#comment-16123254
 ] 

Hudson commented on HBASE-18500:


FAILURE: Integrated in Jenkins build HBase-2.0 #309 (See 
[https://builds.apache.org/job/HBase-2.0/309/])
HBASE-18500 Performance issue: Don't use BufferedMutator for HTable's (zghao: 
rev 0c16bb591b165050439cf945aca19fa708e0b350)
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientPushback.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMultiSlaveReplication.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Table.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/RegionAsTable.java
* (edit) 
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableWrapper.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/constraint/TestConstraint.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* (edit) 
hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestNamespaceCommands.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableBuilder.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestHTableWrapper.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestServerBusyException.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableBuilderBase.java


> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, 
> HBASE-18500-v5.patch, HBASE-18500-v5.patch, HBASE-18500-v6.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
>

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-11 Thread Guanghao Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123122#comment-16123122
 ] 

Guanghao Zhang commented on HBASE-18500:


Ok. Let me fill new issues for the above problems. :-)

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, 
> HBASE-18500-v5.patch, HBASE-18500-v5.patch, HBASE-18500-v6.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/
> Copy the comments from [~chia7712]. Remove the BufferdMutator brings four 
> benefits.
> 1. correct the metrics (see HBASE-18476)
> 2. make HTable thread-safe (see HBASE-17368)
> 3. reduce the latency
> 4. get rid of some deprecated methods in Table



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-11 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123114#comment-16123114
 ] 

Anoop Sam John commented on HBASE-18500:


Do we have an issue for the RpcContext inconsistency? Will you raise 
[~zghaobac]?

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, 
> HBASE-18500-v5.patch, HBASE-18500-v5.patch, HBASE-18500-v6.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/
> Copy the comments from [~chia7712]. Remove the BufferdMutator brings four 
> benefits.
> 1. correct the metrics (see HBASE-18476)
> 2. make HTable thread-safe (see HBASE-17368)
> 3. reduce the latency
> 4. get rid of some deprecated methods in Table



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-11 Thread Chia-Ping Tsai (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123079#comment-16123079
 ] 

Chia-Ping Tsai commented on HBASE-18500:


bq. suggest to also push the fix into branch-1.
The HTableInterface has some methods related to BF, so it is difficult to 
remove the BF for branch-1.

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, 
> HBASE-18500-v5.patch, HBASE-18500-v5.patch, HBASE-18500-v6.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/
> Copy the comments from [~chia7712]. Remove the BufferdMutator brings four 
> benefits.
> 1. correct the metrics (see HBASE-18476)
> 2. make HTable thread-safe (see HBASE-17368)
> 3. reduce the latency
> 4. get rid of some deprecated methods in Table



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-11 Thread Yu Li (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123075#comment-16123075
 ] 

Yu Li commented on HBASE-18500:
---

I'm afraid this is not only issue for master/branch-2 but also for branch-1 
since BufferedMutator is also used there, and suggest to also push the fix into 
branch-1. Better if we could do some verification on branch-1 performance of 
course (smile).

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, 
> HBASE-18500-v5.patch, HBASE-18500-v5.patch, HBASE-18500-v6.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/
> Copy the comments from [~chia7712]. Remove the BufferdMutator brings four 
> benefits.
> 1. correct the metrics (see HBASE-18476)
> 2. make HTable thread-safe (see HBASE-17368)
> 3. reduce the latency
> 4. get rid of some deprecated methods in Table



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-11 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122961#comment-16122961
 ] 

Hadoop QA commented on HBASE-18500:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 17 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
6s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
30m 28s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
25s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}107m 
57s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
54s{color} | {color:green} hbase-rest in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
46s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}182m 27s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18500 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12881386/HBASE-18500-v6.patch |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 7eaafc17be8c 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 8da77b4 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/8028/testReport/ |
| modules | C: hbase-client hbase-server hbase-rest U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/8028/console |

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-10 Thread Guanghao Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122763#comment-16122763
 ] 

Guanghao Zhang commented on HBASE-18500:


bq. We should address it via some other jira. As of here, just do as what u did 
in patch with a TODO to fix the issue.
Will add a TODO when commit it.
Thanks all for reviewing.

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, 
> HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-10 Thread Guanghao Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122740#comment-16122740
 ] 

Guanghao Zhang commented on HBASE-18500:


Thanks for your explanation.
bq. The table on which put was called is created via the RegionCP env and that 
uses a special Connection.
Checked the code, I thought this was introduced by HBASE-9534.
bq. For #1 we may have to use InheritableThreadLocal instead of ThreadLocal.
Agreed. But I don't know if it has any other side-effects.
bq. When doing a runAsUser, how/whether we should reset the Rpc context to have 
the new user.
Do we have other scenario which need reset the Rpc context? If not, I thought 
we can only fixed this for this problem. We can introduce a new config or 
method to disable the "Short-Circuit Coprocessor HTable access" feature. The 
code may be:
{code}
User.runAsLoginUser(new PrivilegedExceptionAction() {
  @Override
  public Void run() throws Exception {
 // disable the "Short-Circuit Coprocessor HTable access" feature 
first
 // Then start a new rpc call
AccessControlLists.addUserPermission(regionEnv.getConfiguration(), 
perm,
  regionEnv.getTable(AccessControlLists.ACL_TABLE_NAME), 
request.getMergeExistingPermissions());
return null;
  }
});
{code}

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, 
> HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-10 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121917#comment-16121917
 ] 

Anoop Sam John commented on HBASE-18500:


For #1 we may have to use InheritableThreadLocal instead of ThreadLocal (?)

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, 
> HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-10 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121630#comment-16121630
 ] 

Anoop Sam John commented on HBASE-18500:


Thanks for the nice debug.  Things are clear now.
This Table.put() happens within the RS. We are already in AccessController.  
Means already there was an RPC happened to server (Actual grant call from 
client side). At RpcServer we have a ThreadLocal where we keep the CallContext 
and inside that the current RPC called user info is set. The table on which put 
was called is created via the RegionCP env and that uses a special Connection. 
The normal RPC channel will be by passed here means there would have no further 
contact on to the RpcServer. So the ThreadLocal is never getting reset.  We ran 
the new put as a super user (User.runAsLoginUser where the login user is the 
user who started RS process) but still as per the RPC context it is the old 
user. When this patch changed the Table.put(Put) flow so that there is no more 
AsyncProcess involvement, the same thread executing the put. So the Rpc context 
ThreadLocal is still accessible and we see the user as old Rpc user.  When 
AsyncProcess was used, the execute happen via another thread from pool and so 
old ThreadLocal variable is not accessible and so it looks as if no Rpc context 
and we were relying on the super user who starts the RS process. 
We can solve this particular issue in diff ways. But this investigation raises 
few Qs.
1. We provide CPs a way to make Table ops. The Table can be obtained from the 
env and may be the flow will get RPC bypassed if the calling region on same RS. 
In that case if the Table.put(Put) is called we will see the old RPC user in 
the second region call path also. But if it was AsyncProcess involved flow, we 
wont see.  How/whether we should make possible to get the Rpc context in the 
second region flow also? To know the user. And we have to solve this 
incosistency based on API usage any way.
2. When doing a runAsUser, how/whether we should reset the Rpc context to have 
the new user.
We should address it via some other jira.  As of here, just do as what u did in 
patch with a TODO to fix the issue.

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, 
> HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-10 Thread Guanghao Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121382#comment-16121382
 ] 

Guanghao Zhang commented on HBASE-18500:


AccessController.grant method will check permission first, then use 
runAsLoginUser to put a record to ACL table.
{code}
switch(request.getUserPermission().getPermission().getType()) {
  case Global :
  case Table :
requirePermission(caller, "grant", perm.getTableName(),
perm.getFamily(), perm.getQualifier(), Action.ADMIN);
break;
  case Namespace :
requireNamespacePermission(caller, "grant", perm.getNamespace(), 
Action.ADMIN);
   break;
}

User.runAsLoginUser(new PrivilegedExceptionAction() {
  @Override
  public Void run() throws Exception {
AccessControlLists.addUserPermission(regionEnv.getConfiguration(), 
perm,
  regionEnv.getTable(AccessControlLists.ACL_TABLE_NAME), 
request.getMergeExistingPermissions());
return null;
  }
});
{code}

When put record to ACL, it will check permission, too. And it will get active 
user from Context first, then get the logged user.
{code}
  private User getActiveUser(ObserverContext ctx) throws IOException {
User user = ctx.getCaller();
if (user == null) {
  // for non-rpc handling, fallback to system user
  user = userProvider.getCurrent();
}
return user;
  }
{code}

And when we use table.put(Put) to write record to ACL, I found it still print 
log in a same thread. Then it will use context user to require permission. When 
use put(List), it will use AsyncProcess and will run it in a new thread. 
Then the context user is null, it will use the logged user to require 
permission. I thought the problem is the runAsLoginUser method didn't change 
the context user when it run in a same thread. But I don't have a good idea to 
fix this. Any ideas about this?

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, 
> HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-10 Thread Guanghao Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1612#comment-1612
 ] 

Guanghao Zhang commented on HBASE-18500:


bq. Could you leave above comment in the right place? Or file a issue.
Let me dig it more.

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, 
> HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-09 Thread Chia-Ping Tsai (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121102#comment-16121102
 ] 

Chia-Ping Tsai commented on HBASE-18500:


{quote}
When grant user permission, it will put a record to ACL. The old implementation 
use BufferMutator, so it will send a multi request. And use 
AccessController.preBatchMutate to check permission when put a record. When 
HTable doesn't use BufferMutator, it will send a mutate request and use 
AccessController.prePut to check permission. So I change to use put(List), 
this will send multi request, too. Then it will same with before. I thought 
AccessController may have some bugs which need to dig more. But it is not 
related to this issue?
{quote}
Could you leave above comment in the right place? Or file a issue.

{quote}
The default retries number is 35. So it will wait a long time when there are 
exception in ut. I thought we should set this to the default 
HBaseTestingUtility config in the future.
{quote}
Got it.

+1


> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, 
> HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-09 Thread Guanghao Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121087#comment-16121087
 ] 

Guanghao Zhang commented on HBASE-18500:


bq. t.put(Collections.singletonList(p));
When grant user permission, it will put a record to ACL. The old implementation 
use BufferMutator, so it will send a multi request. And use 
AccessControlLists.preBatchMutate to check permission when put a record. When 
HTable doesn't use BufferMutator, it will send a mutate request and use 
AccessControlLists.prePut to check permission. So I change to use 
put(List), this will send multi request, too. Then it will same with 
before. I thought  AccessControlLists may have some bugs which need to dig 
more. But it is not related to this issue?
bq. UTIL.getConfiguration().setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 2);
The default retries number is 35. So it will wait a long time when there are 
exception in ut. I thought we should set this to the default 
HBaseTestingUtility config in the future.

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, 
> HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-09 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121080#comment-16121080
 ] 

ramkrishna.s.vasudevan commented on HBASE-18500:


LGTM but for the above comments from [~chia7712]. If he is fine then we can go 
for commit. Release note is needed I believe?

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, 
> HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-09 Thread Chia-Ping Tsai (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120975#comment-16120975
 ] 

Chia-Ping Tsai commented on HBASE-18500:


{code}
 try {
-  t.put(p);
+  t.put(Collections.singletonList(p));
 } finally {
   t.close();
 }
{code}
{code}
+UTIL.getConfiguration().setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 2);
{code}
Why do we need these changes?

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch, HBASE-18500-v4.patch, HBASE-18500-v5.patch, 
> HBASE-18500-v5.patch, HBASE-18500-v5.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-09 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120169#comment-16120169
 ] 

Hadoop QA commented on HBASE-18500:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 17 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
31m 25s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
30s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}134m 
40s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  4m 
13s{color} | {color:green} hbase-rest in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
49s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}198m 48s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18500 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12881012/HBASE-18500-v5.patch |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux f37008a3612d 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / f314b59 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC3 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/7999/testReport/ |
| modules | C: hbase-client hbase-server hbase-rest U: . |
| Console output |

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-09 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119464#comment-16119464
 ] 

Hadoop QA commented on HBASE-18500:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 17 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
 3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
30m  5s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
28s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}109m 49s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
58s{color} | {color:green} hbase-rest in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
44s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}172m 14s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.security.token.TestZKSecretWatcher |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18500 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12880952/HBASE-18500-v5.patch |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 7e5d37f0bc36 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / f314b59 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC3 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/7996/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results |

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-07 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117857#comment-16117857
 ] 

Hadoop QA commented on HBASE-18500:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 17 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
 1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
46s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
43s{color} | {color:red} hbase-rest in master has 3 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
33m 26s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
28s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}109m  
1s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
59s{color} | {color:green} hbase-rest in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
45s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}176m 30s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18500 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12880767/HBASE-18500-v5.patch |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 6ed0a3a3ed0e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 7e7461e |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC3 |
| findbugs | 
https://builds.apache.org/job/PreCommit-HBASE-Build/7969/artifact/patchprocess/branch-findbugs-hbase-rest-warnings.html
 |
|  Test Results |

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-07 Thread Chia-Ping Tsai (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117087#comment-16117087
 ] 

Chia-Ping Tsai commented on HBASE-18500:


bq. Was it before 1.3 itself?
Yes, see HBASE-2898

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch, HBASE-18500-v4.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-07 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116603#comment-16116603
 ] 

Hadoop QA commented on HBASE-18500:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 17 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
41s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
34s{color} | {color:red} hbase-rest in master has 3 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
5s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
31m 36s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m  
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
20s{color} | {color:red} hbase-client generated 1 new + 0 unchanged - 0 fixed = 
1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
37s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}111m 
31s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m  
2s{color} | {color:green} hbase-rest in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
45s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}174m 24s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18500 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12880624/HBASE-18500-v4.patch |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 115500b98c50 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / fd76eb3 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC3 |
| findbugs | 
https://builds.apache.org/job/PreCommit-HBASE-Build/7960/artifact/patchprocess/branch-findbugs-hbase-rest-warnings.html
 |
| javadoc |

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-07 Thread Guanghao Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116418#comment-16116418
 ] 

Guanghao Zhang commented on HBASE-18500:


Review: https://reviews.apache.org/r/61454/

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch, HBASE-18500-v4.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.
> Review: https://reviews.apache.org/r/61454/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-07 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116328#comment-16116328
 ] 

Hadoop QA commented on HBASE-18500:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 16 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
42s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
34s{color} | {color:red} hbase-rest in master has 3 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
4s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
31m 22s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m  
1s{color} | {color:red} hbase-client generated 2 new + 0 unchanged - 0 fixed = 
2 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
18s{color} | {color:red} hbase-client generated 1 new + 0 unchanged - 0 fixed = 
1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
33s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}110m 23s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m  
1s{color} | {color:green} hbase-rest in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
44s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}172m 49s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-client |
|  |  Dead store to response in 
org.apache.hadoop.hbase.client.HTable$3.rpcCall()  At 
HTable.java:org.apache.hadoop.hbase.client.HTable$3.rpcCall()  At 
HTable.java:[line 585] |
|  |  Unread public/protected field:At TableBuilderBase.java:[line 51] |
| Failed junit tests | hadoop.hbase.constraint.TestConstraint |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-18500 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12880596/HBASE-18500-v3.patch |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 79fe3c627abd 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-07 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116256#comment-16116256
 ] 

ramkrishna.s.vasudevan commented on HBASE-18500:


When was the delete API changed for modifying the passed input user list? Was 
it before 1.3 itself? 

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-07 Thread Chia-Ping Tsai (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116189#comment-16116189
 ] 

Chia-Ping Tsai commented on HBASE-18500:


I skim the user@mail, and there is no recently wish about "re-submit only the 
failed puts". We can leave a TODO in Put(). Or add an new methods which do the 
same way with delete.

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-07 Thread Guanghao Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116183#comment-16116183
 ] 

Guanghao Zhang commented on HBASE-18500:


bq. So now we will change the List that user has passed in? 
The delete(List) API already did this: remove the success records and 
keep the failed records.. The key point is to find a proper way to let user 
figure out the failed records. Maybe return the failed records list?

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-07 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116143#comment-16116143
 ] 

Anoop Sam John commented on HBASE-18500:


So now we will change the List that user has passed in?  May  be better if we 
avoid that..

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-07 Thread Chia-Ping Tsai (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116138#comment-16116138
 ] 

Chia-Ping Tsai commented on HBASE-18500:


bq. I thought we can do this incompatibility change in hbase 2.0?
I feel the same way. We need to announce this change as the best as we can.

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-07 Thread Guanghao Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116131#comment-16116131
 ] 

Guanghao Zhang commented on HBASE-18500:


bq. The broken compatibility is caused by that we did not modify the input list 
before.
I thought we can do this incompatibility change in hbase 2.0?

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch, 
> HBASE-18500-v3.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-06 Thread Chia-Ping Tsai (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115706#comment-16115706
 ] 

Chia-Ping Tsai commented on HBASE-18500:


bq. How you mean
It is better to make the behavior of Table#put consistent with Table#delete. 
What that means is we should remove the succeed PUTs from list. The broken 
compatibility is caused by that we did not modify the input list before. 
Personally, i prefer removed the succeed PUTs because that make Table#put more 
easier to exercise. We had good discussion in HBASE-13271.

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-05 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115550#comment-16115550
 ] 

stack commented on HBASE-18500:
---

+1 on the removals. They have done a proper deprecation cycle. This change 
should be marked incompatible though if only to have it show up in list of 
possibly breaking changes when someone searches.

bq. We should check the results in order to find the failed PUTs, but the 
change breaks the operational compatibility...

How you mean [~chia7712] (BTW, nice summary of benefits regards removal of BM 
from HT)?

What if we created a BM in side in Puts and called its flush... then operation 
would be as it was before?

Thanks.

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-04 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114559#comment-16114559
 ] 

Hadoop QA commented on HBASE-18500:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 14 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
50s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
42s{color} | {color:red} hbase-rest in master has 3 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
19s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
38m 10s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
19s{color} | {color:red} hbase-client generated 2 new + 0 unchanged - 0 fixed = 
2 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
22s{color} | {color:red} hbase-client generated 1 new + 0 unchanged - 0 fixed = 
1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
45s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 49s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
54s{color} | {color:green} hbase-rest in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  1m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m  3s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-client |
|  |  Dead store to response in 
org.apache.hadoop.hbase.client.HTable$3.rpcCall()  At 
HTable.java:org.apache.hadoop.hbase.client.HTable$3.rpcCall()  At 
HTable.java:[line 585] |
|  |  Unread public/protected field:At TableBuilderBase.java:[line 51] |
| Failed junit tests | hadoop.hbase.security.access.TestNamespaceCommands |
|   | hadoop.hbase.security.access.TestAccessController |
|   | hadoop.hbase.client.TestClientPushback |
|   | hadoop.hbase.client.TestHCM |
| Timed out junit tests | 
org.apache.hadoop.hbase.master.procedure.TestEnableTableProcedure |
|   | org.apache.hadoop.hbase.master.procedure.TestDeleteTableProcedure |
|   | org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure |
|   |

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-04 Thread Chia-Ping Tsai (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114407#comment-16114407
 ] 

Chia-Ping Tsai commented on HBASE-18500:


{code}
   @Override
   public void put(final List puts) throws IOException {
-getBufferedMutator().mutate(puts);
-flushCommits();
+for (Put put : puts) {
+  validatePut(put);
+}
+Object[] results = new Object[puts.size()];
+try {
+  batch(puts, results, writeRpcTimeout);
+} catch (InterruptedException e) {
+  throw (InterruptedIOException) new InterruptedIOException().initCause(e);
+}
{code}
We should check the results in order to find the failed PUTs, but the change 
breaks the operational compatibility... 

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch, HBASE-18500-v2.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-04 Thread Chia-Ping Tsai (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114152#comment-16114152
 ] 

Chia-Ping Tsai commented on HBASE-18500:


bq. I thought this should be done in hbase-2.0
+1

Consider changing the type (bug -> task or improvement)

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-04 Thread Guanghao Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114131#comment-16114131
 ] 

Guanghao Zhang commented on HBASE-18500:


bq. So this can be done in hbase-2.0 itself or in hbase-3.0?
I thought this should be done in hbase-2.0. And for hbase-3.0, the blocking 
client implementation can be removed and wrapped by the async client (See 
HBASE-18334). 

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-04 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114117#comment-16114117
 ] 

ramkrishna.s.vasudevan commented on HBASE-18500:


bq.I agree completely because it brings four benefits.
+1 on seeing the above comment. So this can be done in hbase-2.0 itself or in 
hbase-3.0?

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-03 Thread Chia-Ping Tsai (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113847#comment-16113847
 ] 

Chia-Ping Tsai commented on HBASE-18500:


bq. I thought we can remove the BufferedMutator from Table.
I agree completely because it brings four benefits.
# correct the metrics (see HBASE-18476)
# make HTable thread-safe (see HBASE-17368)
# reduce the latency
# get rid of some deprecated methods in Table

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-03 Thread Guanghao Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113836#comment-16113836
 ] 

Guanghao Zhang commented on HBASE-18500:


I thought we can remove the BufferedMutator from Table. Then Table is used for 
no buffered write and BufferedMutator used for the buffered write.
[~anoop.hbase] [~ram_krish] [~chia7712] [~stack] What do you think about this?


> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-02 Thread Guanghao Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110583#comment-16110583
 ] 

Guanghao Zhang commented on HBASE-18500:


bq. This fix is applicable which all branches? In master we do like what is 
done in patch now?
I thought this should be fixed in all branch-1, branch-2 and master. For hbase 
3.0,  we can remove the sync code and reimplement it by the async interface.

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-02 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110582#comment-16110582
 ] 

Hadoop QA commented on HBASE-18500:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
31m  1s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
12s{color} | {color:red} hbase-client generated 1 new + 0 unchanged - 0 fixed = 
1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  2m 38s{color} 
| {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}121m  4s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
45s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}175m 44s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-client |
|  |  Dead store to response in 
org.apache.hadoop.hbase.client.HTable$3.rpcCall()  At 
HTable.java:org.apache.hadoop.hbase.client.HTable$3.rpcCall()  At 
HTable.java:[line 615] |
| Failed junit tests | hadoop.hbase.client.TestAsyncProcess |
|   | hadoop.hbase.security.access.TestNamespaceCommands |
|   | hadoop.hbase.client.TestServerBusyException |
|   | hadoop.hbase.client.TestFromClientSide |
|   | hadoop.hbase.client.TestFromClientSideWithCoprocessor |
|   | hadoop.hbase.client.TestClientPushback |
|   | hadoop.hbase.client.TestHCM |
|   | hadoop.hbase.security.access.TestAccessController |
|   | hadoop.hbase.constraint.TestConstraint |
| Timed out junit tests | 
org.apache.hadoop.hbase.master.TestMasterMetricsWrapper |
|   | org.apache.hadoop.hbase.master.procedure.TestDisableTableProcedure |
|   | org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure |
|   | org.apache.hadoop.hbase.master.TestGetLastFlushedSequenceId |
|   |

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-02 Thread Guanghao Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110546#comment-16110546
 ] 

Guanghao Zhang commented on HBASE-18500:


bq. Can we get rid of BUFFER for Table?
Agreed. Table is used for no buffered write and BufferedMutator for the 
buffered write.

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-02 Thread Chia-Ping Tsai (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110538#comment-16110538
 ] 

Chia-Ping Tsai commented on HBASE-18500:


We have some related issues. HBASE-17368, HBASE-18334, and HBASE-18476
Can we get rid of *BUFFER* for Table?

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-02 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110472#comment-16110472
 ] 

ramkrishna.s.vasudevan commented on HBASE-18500:


I think YCSB works like this only in case we need to measure write performance 
like latency. It was having clientSidebuffering and if that is false it will 
call HTable.put() directly. Probably this patch is needed if YCSB needs to 
support the same with hbase-2.0.

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-02 Thread Abhishek Singh Chouhan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110459#comment-16110459
 ] 

Abhishek Singh Chouhan commented on HBASE-18500:


Similar to HBASE-18476 i think.

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-02 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110410#comment-16110410
 ] 

Anoop Sam John commented on HBASE-18500:


This fix is applicable which all branches?  In master we do like what is done 
in patch now?

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18500) Performance issue: Don't use BufferedMutator for HTable's put method

2017-08-02 Thread Guanghao Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-18500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110373#comment-16110373
 ] 

Guanghao Zhang commented on HBASE-18500:


{code}
./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
--nomapred --autoFlush=True randomWrite 1
{code}

Mean latency(us).
|| || Test1 || Test2 || Test3 || Test4 || Test5 ||
| w/o patch | 153.70 | 158.64 | 168.57 | 152.04 | 147.29 |
| w/ v1 patch | 111.03 | 115.34 | 120.70 | 124.56 | 119.86 |

50th latency(us).
|| || Test1 || Test2 || Test3 || Test4 || Test5 ||
| w/o patch | 102.00 | 105.00 | 113.00 | 122.00 | 114.00 |
| w/ v1 patch | 82.00 | 88.00 | 92.00 | 94.00 | 96.00 |

99th latency(us).
|| || Test1 || Test2 || Test3 || Test4 || Test5 ||
| w/o patch | 419.00 | 442.00 | 407.00 | 450.00 | 416.00 |
| w/ v1 patch | 322.00 | 305.00 | 290.00 | 304.00 | 276.00 |

> Performance issue: Don't use BufferedMutator for HTable's put method
> 
>
> Key: HBASE-18500
> URL: https://issues.apache.org/jira/browse/HBASE-18500
> Project: HBase
>  Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-18500-v1.patch
>
>
> Copied the test result from HBASE-17994.
> Run start-hbase.sh in my local computer and use the default config to test 
> with PE tool.
> {code}
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True randomWrite 1
> ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 
> --nomapred --autoFlush=True asyncRandomWrite 1
> {code}
> Mean latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 164.39 | 161.22 | 164.78 | 140.61 | 151.69 |
> | asyncRandomWrite | 122.29 | 125.58 | 122.23 | 113.18 | 123.02 |
> 50th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 130.00 | 125.00 | 123.00 | 112.00 | 121.00 |
> | asyncRandomWrite | 95.00 | 97.00 | 95.00 | 88.00 | 95.00 |
> 99th latency test result.
> || || Test1 || Test2 || Test3 || Test4 || Test5 ||
> | randomWrite | 600.00 | 600.00 | 650.00 | 404.00 | 425.00 |
> | asyncRandomWrite | 339.00 | 327.00 | 297.00 | 311.00 | 318.00 |
> In our internal 0.98 branch, the PE test result shows the async write has the 
> almost same latency with the blocking write. But for master branch, the 
> result shows the async write has better latency than the blocking client.  
> Take a look about the code, I thought the difference is the BufferedMutator. 
> For master branch, HTable don't have a write buffer and all write request 
> will be flushed directly. And user can use BufferedMutator when user want to 
> perform client-side buffering of writes. For the performance issue 
> (autoFlush=True), I thought we can use rpc caller directly in HTable's put 
> method. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

48 matches

Mail list logo