subject:"\[jira\] \[Commented\] \(HBASE\-5162\) Basic client pushback mechanism"

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2015-01-22 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14288762#comment-14288762
 ] 

Hudson commented on HBASE-5162:
---

SUCCESS: Integrated in HBase-1.0 #676 (See 
[https://builds.apache.org/job/HBase-1.0/676/])
HBASE-12730 Backport HBASE-5162 (Basic client pushback mechanism) to branch-1 
(apurtell: rev 06eef6dafb51eaeaea9b0566252bcab15be5729d)
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterConnection.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerStatisticTracker.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiAction.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Result.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ExponentialClientBackoffPolicy.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEditsReplaySink.java
* 
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientExponentialBackoff.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ServerStatistics.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* 
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestDelayingRunner.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/StatsTrackingRpcRetryingCaller.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/ResultStatsUtil.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
* hbase-protocol/src/main/protobuf/Client.proto
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientPushback.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerFactory.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/DelayingRunner.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionAdapter.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ClientBackoffPolicy.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ClientBackoffPolicyFactory.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RegionCoprocessorRpcChannel.java
* 
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java


 Basic client pushback mechanism
 ---

 Key: HBASE-5162
 URL: https://issues.apache.org/jira/browse/HBASE-5162
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jesse Yates
 Fix For: 2.0.0

 Attachments: 5162-addendum2.txt, hbase-5162-branch-1-v0.patch, 
 hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, 
 hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, 
 hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, 
 hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, 
 hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, 
 hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, 
 hbase-5162-trunk-v8.patch, java_HBASE-5162.patch


 The current blocking we do when we are close to some limits (memstores over 
 the multiplier factor, too many store files, global memstore memory) is bad, 
 too coarse and confusing. After hitting HBASE-5161, it really becomes obvious 
 that we need something better.
 I did a little brainstorm with Stack, we came up quickly with two solutions:
  - Send some exception to the client, like OverloadedException, that's thrown 
 when some situation happens like getting past the low memory barrier. It 
 would be thrown when the client gets a handler and does some check while 
 putting or deleting. The client would treat this a retryable exception but 
 ideally wouldn't check .META. for a new location. It could be fancy and have 
 multiple levels of pushback, like send the exception to 25% of the clients, 
 and then go up if the situation persists. Should be easy to implement but 
 we'll be using a lot more IO to send the payload over and over again (but at 
 least it wouldn't sit in the RS's memory).
  - Send a message alongside a successful put or delete to tell the client to 
 slow down a little, this way we don't have to do back and forth with the 
 payload between the client and the server.

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2015-01-22 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14288832#comment-14288832
 ] 

Hudson commented on HBASE-5162:
---

SUCCESS: Integrated in HBase-1.1 #99 (See 
[https://builds.apache.org/job/HBase-1.1/99/])
HBASE-12730 Backport HBASE-5162 (Basic client pushback mechanism) to branch-1 
(apurtell: rev 04a003d6a2b628bc551a55ae556c1b44485e3022)
* 
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestDelayingRunner.java
* hbase-protocol/src/main/protobuf/Client.proto
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/DelayingRunner.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiAction.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RegionCoprocessorRpcChannel.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ClientBackoffPolicyFactory.java
* 
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientExponentialBackoff.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/StatsTrackingRpcRetryingCaller.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ClientBackoffPolicy.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerFactory.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Result.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerStatisticTracker.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/ResultStatsUtil.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterConnection.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ExponentialClientBackoffPolicy.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEditsReplaySink.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ServerStatistics.java
* 
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionAdapter.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientPushback.java


 Basic client pushback mechanism
 ---

 Key: HBASE-5162
 URL: https://issues.apache.org/jira/browse/HBASE-5162
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jesse Yates
 Fix For: 2.0.0

 Attachments: 5162-addendum2.txt, hbase-5162-branch-1-v0.patch, 
 hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, 
 hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, 
 hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, 
 hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, 
 hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, 
 hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, 
 hbase-5162-trunk-v8.patch, java_HBASE-5162.patch


 The current blocking we do when we are close to some limits (memstores over 
 the multiplier factor, too many store files, global memstore memory) is bad, 
 too coarse and confusing. After hitting HBASE-5161, it really becomes obvious 
 that we need something better.
 I did a little brainstorm with Stack, we came up quickly with two solutions:
  - Send some exception to the client, like OverloadedException, that's thrown 
 when some situation happens like getting past the low memory barrier. It 
 would be thrown when the client gets a handler and does some check while 
 putting or deleting. The client would treat this a retryable exception but 
 ideally wouldn't check .META. for a new location. It could be fancy and have 
 multiple levels of pushback, like send the exception to 25% of the clients, 
 and then go up if the situation persists. Should be easy to implement but 
 we'll be using a lot more IO to send the payload over and over again (but at 
 least it wouldn't sit in the RS's memory).
  - Send a message alongside a successful put or delete to tell the client to 
 slow down a little, this way we don't have to do back and forth with the 
 payload between the client and the server.

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2015-01-19 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14283415#comment-14283415
 ] 

Hudson commented on HBASE-5162:
---

SUCCESS: Integrated in HBase-0.98 #805 (See 
[https://builds.apache.org/job/HBase-0.98/805/])
HBASE-12729 Backport HBASE-5162 (Basic client pushback mechanism) to 0.98 
(apurtell: rev 85e7270b6f8cfb593c1cba137241f2aad986a3e1)
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ClientBackoffPolicyFactory.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiAction.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* 
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientExponentialBackoff.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ExponentialClientBackoffPolicy.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientPushback.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerFactory.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
* 
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/DelayingRunner.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCaller.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEditsReplaySink.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Result.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ClientBackoffPolicy.java
* 
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestDelayingRunner.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/StatisticsHConnection.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ServerStatistics.java
* hbase-protocol/src/main/protobuf/Client.proto
* hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/StatsTrackingRpcRetryingCaller.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/ResultStatsUtil.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientSmallScanner.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerStatisticTracker.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java


 Basic client pushback mechanism
 ---

 Key: HBASE-5162
 URL: https://issues.apache.org/jira/browse/HBASE-5162
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jesse Yates
 Fix For: 2.0.0

 Attachments: 5162-addendum2.txt, hbase-5162-branch-1-v0.patch, 
 hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, 
 hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, 
 hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, 
 hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, 
 hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, 
 hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, 
 hbase-5162-trunk-v8.patch, java_HBASE-5162.patch


 The current blocking we do when we are close to some limits (memstores over 
 the multiplier factor, too many store files, global memstore memory) is bad, 
 too coarse and confusing. After hitting HBASE-5161, it really becomes obvious 
 that we need something better.
 I did a little brainstorm with Stack, we came up quickly with two solutions:
  - Send some exception to the client, like OverloadedException, that's thrown 
 when some situation happens like getting past the low memory barrier. It 
 would be thrown when the client gets a handler and does some check while 
 putting or deleting. The client would treat this a retryable exception but 
 ideally wouldn't check .META. for a new location. It could be fancy and have 
 multiple levels of pushback, like send the exception to 25% of the clients, 
 and then go up if the situation persists. Should be easy to implement but 
 we'll be using a lot more IO to send the payload over and over again (but at 
 least it wouldn't sit in the RS's memory).
  - Send a message alongside a successful put or

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2015-01-19 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14283549#comment-14283549
 ] 

Hudson commented on HBASE-5162:
---

SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #768 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/768/])
HBASE-12729 Backport HBASE-5162 (Basic client pushback mechanism) to 0.98 
(apurtell: rev 85e7270b6f8cfb593c1cba137241f2aad986a3e1)
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ExponentialClientBackoffPolicy.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ServerStatistics.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerFactory.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
* hbase-protocol/src/main/protobuf/Client.proto
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/StatisticsHConnection.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerStatisticTracker.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientPushback.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Result.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/DelayingRunner.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ClientBackoffPolicy.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiAction.java
* 
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestDelayingRunner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientSmallScanner.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEditsReplaySink.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ClientBackoffPolicyFactory.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/ResultStatsUtil.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/StatsTrackingRpcRetryingCaller.java
* 
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientExponentialBackoff.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCaller.java


 Basic client pushback mechanism
 ---

 Key: HBASE-5162
 URL: https://issues.apache.org/jira/browse/HBASE-5162
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jesse Yates
 Fix For: 2.0.0

 Attachments: 5162-addendum2.txt, hbase-5162-branch-1-v0.patch, 
 hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, 
 hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, 
 hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, 
 hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, 
 hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, 
 hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, 
 hbase-5162-trunk-v8.patch, java_HBASE-5162.patch


 The current blocking we do when we are close to some limits (memstores over 
 the multiplier factor, too many store files, global memstore memory) is bad, 
 too coarse and confusing. After hitting HBASE-5161, it really becomes obvious 
 that we need something better.
 I did a little brainstorm with Stack, we came up quickly with two solutions:
  - Send some exception to the client, like OverloadedException, that's thrown 
 when some situation happens like getting past the low memory barrier. It 
 would be thrown when the client gets a handler and does some check while 
 putting or deleting. The client would treat this a retryable exception but 
 ideally wouldn't check .META. for a new location. It could be fancy and have 
 multiple levels of pushback, like send the exception to 25% of the clients, 
 and then go up if the situation persists. Should be easy to implement but 
 we'll be using a lot more IO to send the payload over and over again (but at 
 least it wouldn't sit in the RS's memory).
  - Send a message

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2015-01-12 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14274201#comment-14274201
 ] 

Hudson commented on HBASE-5162:
---

FAILURE: Integrated in HBase-TRUNK #6012 (See 
[https://builds.apache.org/job/HBase-TRUNK/6012/])
HBASE-12838 After HBASE-5162 RSRpcServices accidentally applies mutations twice 
(apurtell: rev e52d8f85c48adc772dfe99e1f5cebb814e89c16c)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java


 Basic client pushback mechanism
 ---

 Key: HBASE-5162
 URL: https://issues.apache.org/jira/browse/HBASE-5162
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jesse Yates
 Fix For: 2.0.0

 Attachments: 5162-addendum2.txt, hbase-5162-branch-1-v0.patch, 
 hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, 
 hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, 
 hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, 
 hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, 
 hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, 
 hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, 
 hbase-5162-trunk-v8.patch, java_HBASE-5162.patch


 The current blocking we do when we are close to some limits (memstores over 
 the multiplier factor, too many store files, global memstore memory) is bad, 
 too coarse and confusing. After hitting HBASE-5161, it really becomes obvious 
 that we need something better.
 I did a little brainstorm with Stack, we came up quickly with two solutions:
  - Send some exception to the client, like OverloadedException, that's thrown 
 when some situation happens like getting past the low memory barrier. It 
 would be thrown when the client gets a handler and does some check while 
 putting or deleting. The client would treat this a retryable exception but 
 ideally wouldn't check .META. for a new location. It could be fancy and have 
 multiple levels of pushback, like send the exception to 25% of the clients, 
 and then go up if the situation persists. Should be easy to implement but 
 we'll be using a lot more IO to send the payload over and over again (but at 
 least it wouldn't sit in the RS's memory).
  - Send a message alongside a successful put or delete to tell the client to 
 slow down a little, this way we don't have to do back and forth with the 
 payload between the client and the server. It's a cleaner (I think) but more 
 involved solution.
 In every case the RS should do very obvious things to notify the operators of 
 this situation, through logs, web UI, metrics, etc.
 Other ideas?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-17 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249618#comment-14249618
 ] 

Hudson commented on HBASE-5162:
---

FAILURE: Integrated in HBase-TRUNK #5937 (See 
[https://builds.apache.org/job/HBase-TRUNK/5937/])
Revert HBASE-5162 Basic client pushback mechanism (Jesse Yates); ADDENDUM 
(apurtell: rev 6a042c381a04ccd4b82bc51720af7a043e15)
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerFactory.java
* hbase-client/pom.xml
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/StatsTrackingRpcRetryingCaller.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientPushback.java
Amend HBASE-5162 Basic client pushback mechanism (Jesse Yates) (apurtell: rev 
99a11390b4758c211af04af2ca0696ac6e3e0aeb)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientPushback.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/StatsTrackingRpcRetryingCaller.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerFactory.java


 Basic client pushback mechanism
 ---

 Key: HBASE-5162
 URL: https://issues.apache.org/jira/browse/HBASE-5162
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jesse Yates
 Fix For: 1.0.0, 2.0.0

 Attachments: hbase-5162-branch-1-v0.patch, 
 hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, 
 hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, 
 hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, 
 hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, 
 hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, 
 hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, 
 hbase-5162-trunk-v8.patch, java_HBASE-5162.patch


 The current blocking we do when we are close to some limits (memstores over 
 the multiplier factor, too many store files, global memstore memory) is bad, 
 too coarse and confusing. After hitting HBASE-5161, it really becomes obvious 
 that we need something better.
 I did a little brainstorm with Stack, we came up quickly with two solutions:
  - Send some exception to the client, like OverloadedException, that's thrown 
 when some situation happens like getting past the low memory barrier. It 
 would be thrown when the client gets a handler and does some check while 
 putting or deleting. The client would treat this a retryable exception but 
 ideally wouldn't check .META. for a new location. It could be fancy and have 
 multiple levels of pushback, like send the exception to 25% of the clients, 
 and then go up if the situation persists. Should be easy to implement but 
 we'll be using a lot more IO to send the payload over and over again (but at 
 least it wouldn't sit in the RS's memory).
  - Send a message alongside a successful put or delete to tell the client to 
 slow down a little, this way we don't have to do back and forth with the 
 payload between the client and the server. It's a cleaner (I think) but more 
 involved solution.
 In every case the RS should do very obvious things to notify the operators of 
 this situation, through logs, web UI, metrics, etc.
 Other ideas?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-17 Thread Andrew Purtell (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14250012#comment-14250012
 ] 

Andrew Purtell commented on HBASE-5162:
---

We need to determine why the test is failing in the Jenkins environment or 
revert. 

 Basic client pushback mechanism
 ---

 Key: HBASE-5162
 URL: https://issues.apache.org/jira/browse/HBASE-5162
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jesse Yates
 Fix For: 1.0.0, 2.0.0

 Attachments: hbase-5162-branch-1-v0.patch, 
 hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, 
 hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, 
 hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, 
 hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, 
 hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, 
 hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, 
 hbase-5162-trunk-v8.patch, java_HBASE-5162.patch


 The current blocking we do when we are close to some limits (memstores over 
 the multiplier factor, too many store files, global memstore memory) is bad, 
 too coarse and confusing. After hitting HBASE-5161, it really becomes obvious 
 that we need something better.
 I did a little brainstorm with Stack, we came up quickly with two solutions:
  - Send some exception to the client, like OverloadedException, that's thrown 
 when some situation happens like getting past the low memory barrier. It 
 would be thrown when the client gets a handler and does some check while 
 putting or deleting. The client would treat this a retryable exception but 
 ideally wouldn't check .META. for a new location. It could be fancy and have 
 multiple levels of pushback, like send the exception to 25% of the clients, 
 and then go up if the situation persists. Should be easy to implement but 
 we'll be using a lot more IO to send the payload over and over again (but at 
 least it wouldn't sit in the RS's memory).
  - Send a message alongside a successful put or delete to tell the client to 
 slow down a little, this way we don't have to do back and forth with the 
 payload between the client and the server. It's a cleaner (I think) but more 
 involved solution.
 In every case the RS should do very obvious things to notify the operators of 
 this situation, through logs, web UI, metrics, etc.
 Other ideas?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-17 Thread stack (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14250308#comment-14250308
]

stack commented on HBASE-5162:
--

Arg. Sorry about that [~apurtell] You seem to be doing a bunch of cleanup after
me lately. Let me be better

Basic client pushback mechanism
---

Key: HBASE-5162
URL: https://issues.apache.org/jira/browse/HBASE-5162
Project: HBase
Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jesse Yates
Fix For: 1.0.0, 2.0.0

Attachments: hbase-5162-branch-1-v0.patch,
hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch,
hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch,
hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch,
hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch,
hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch,
hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch,
hbase-5162-trunk-v8.patch, java_HBASE-5162.patch

The current blocking we do when we are close to some limits (memstores over
the multiplier factor, too many store files, global memstore memory) is bad,
too coarse and confusing. After hitting HBASE-5161, it really becomes obvious
that we need something better.
I did a little brainstorm with Stack, we came up quickly with two solutions:
- Send some exception to the client, like OverloadedException, that's thrown
when some situation happens like getting past the low memory barrier. It
would be thrown when the client gets a handler and does some check while
putting or deleting. The client would treat this a retryable exception but
ideally wouldn't check .META. for a new location. It could be fancy and have
multiple levels of pushback, like send the exception to 25% of the clients,
and then go up if the situation persists. Should be easy to implement but
we'll be using a lot more IO to send the payload over and over again (but at
least it wouldn't sit in the RS's memory).
- Send a message alongside a successful put or delete to tell the client to
slow down a little, this way we don't have to do back and forth with the
payload between the client and the server. It's a cleaner (I think) but more
involved solution.
In every case the RS should do very obvious things to notify the operators of
this situation, through logs, web UI, metrics, etc.
Other ideas?

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-17 Thread Dima Spivak (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14250381#comment-14250381
]

Dima Spivak commented on HBASE-5162:

Can someone explain to me where this magic number of 15% comes from? (And then
where your magic number of 11% comes from, [~tedyu]? :) ) I might well just be
dense here, but I'm not following what part of the setup of the test dictates a
memstore load to be expected.

Basic client pushback mechanism
---

Attachments: 5162-addendum2.txt, hbase-5162-branch-1-v0.patch,
hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch,
hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch,
hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch,
hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch,
hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch,
hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch,
hbase-5162-trunk-v8.patch, java_HBASE-5162.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-17 Thread Ted Yu (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14250425#comment-14250425
]

Ted Yu commented on HBASE-5162:
---

The test writes some data to client-pushback table and gets memstore load
through StatisticsTracker.
The load varies: 11 was the reading shown in build #5937.

Jesse would be able to give us more detail.

Basic client pushback mechanism
---

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-17 Thread Jesse Yates (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14250479#comment-14250479
]

Jesse Yates commented on HBASE-5162:

thanks all for helping out!

[~dimaspivak] the number is just what the memstore is currently at, load wise,
which is basically the current number of bytes/number of bytes after which it
flushes. A put is a certain number of bytes, so it should put a certain amount
of load on the memstore.

I think what is happening in Jenkins is something is hanging, so the memstore
is flushing before we expect. I think we can go ahead with Ted's patch, for the
meantime. An alternative would be to force-flush the table before doing the
write and then get the load - nothing else should be influencing it.

Basic client pushback mechanism
---

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-17 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14250488#comment-14250488
 ] 

Ted Yu commented on HBASE-5162:
---

Planning to commit the addendum soon - this JIRA is in 1.0 whose RC is coming 
out today.

 Basic client pushback mechanism
 ---

 Key: HBASE-5162
 URL: https://issues.apache.org/jira/browse/HBASE-5162
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jesse Yates
 Fix For: 1.0.0, 2.0.0

 Attachments: 5162-addendum2.txt, hbase-5162-branch-1-v0.patch, 
 hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, 
 hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, 
 hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, 
 hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, 
 hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, 
 hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, 
 hbase-5162-trunk-v8.patch, java_HBASE-5162.patch


 The current blocking we do when we are close to some limits (memstores over 
 the multiplier factor, too many store files, global memstore memory) is bad, 
 too coarse and confusing. After hitting HBASE-5161, it really becomes obvious 
 that we need something better.
 I did a little brainstorm with Stack, we came up quickly with two solutions:
  - Send some exception to the client, like OverloadedException, that's thrown 
 when some situation happens like getting past the low memory barrier. It 
 would be thrown when the client gets a handler and does some check while 
 putting or deleting. The client would treat this a retryable exception but 
 ideally wouldn't check .META. for a new location. It could be fancy and have 
 multiple levels of pushback, like send the exception to 25% of the clients, 
 and then go up if the situation persists. Should be easy to implement but 
 we'll be using a lot more IO to send the payload over and over again (but at 
 least it wouldn't sit in the RS's memory).
  - Send a message alongside a successful put or delete to tell the client to 
 slow down a little, this way we don't have to do back and forth with the 
 payload between the client and the server. It's a cleaner (I think) but more 
 involved solution.
 In every case the RS should do very obvious things to notify the operators of 
 this situation, through logs, web UI, metrics, etc.
 Other ideas?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-17 Thread Jesse Yates (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14250490#comment-14250490
]

Jesse Yates commented on HBASE-5162:

This jira isn't actually going into 1.0 - couldn't get [~enis]'s attention to
ok it going in. I'll move it out to 1.1 for now. Feel free to commit to master
Ted.

Basic client pushback mechanism
---

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-17 Thread Dima Spivak (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14250495#comment-14250495
 ] 

Dima Spivak commented on HBASE-5162:


I really don't think changing one arbitrary to another arbitrary number is the 
way to go here; the test will start failing in the future if we ever tweak the 
memstore and that number changes. I think Jesse's alternative solution, setting 
up the memstore in a known state during/before the test is the most reasonable 
thing to do.

 Basic client pushback mechanism
 ---

 Key: HBASE-5162
 URL: https://issues.apache.org/jira/browse/HBASE-5162
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jesse Yates
 Fix For: 2.0.0, 1.1.0

 Attachments: 5162-addendum2.txt, hbase-5162-branch-1-v0.patch, 
 hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, 
 hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, 
 hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, 
 hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, 
 hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, 
 hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, 
 hbase-5162-trunk-v8.patch, java_HBASE-5162.patch


 The current blocking we do when we are close to some limits (memstores over 
 the multiplier factor, too many store files, global memstore memory) is bad, 
 too coarse and confusing. After hitting HBASE-5161, it really becomes obvious 
 that we need something better.
 I did a little brainstorm with Stack, we came up quickly with two solutions:
  - Send some exception to the client, like OverloadedException, that's thrown 
 when some situation happens like getting past the low memory barrier. It 
 would be thrown when the client gets a handler and does some check while 
 putting or deleting. The client would treat this a retryable exception but 
 ideally wouldn't check .META. for a new location. It could be fancy and have 
 multiple levels of pushback, like send the exception to 25% of the clients, 
 and then go up if the situation persists. Should be easy to implement but 
 we'll be using a lot more IO to send the payload over and over again (but at 
 least it wouldn't sit in the RS's memory).
  - Send a message alongside a successful put or delete to tell the client to 
 slow down a little, this way we don't have to do back and forth with the 
 payload between the client and the server. It's a cleaner (I think) but more 
 involved solution.
 In every case the RS should do very obvious things to notify the operators of 
 this situation, through logs, web UI, metrics, etc.
 Other ideas?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-17 Thread Ted Yu (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14250499#comment-14250499
]

Ted Yu commented on HBASE-5162:
---

Integrated to master branch.

It is not clear when 1.1.0 release comes out. Should this JIRA be resolved and
a new JIRA opened for backporting to branch-1 ?

Basic client pushback mechanism
---

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-17 Thread Jesse Yates (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14250502#comment-14250502
 ] 

Jesse Yates commented on HBASE-5162:


We could. Was hoping to get the 1.0 backport in sooner rather than later, but 
either way works for me

 Basic client pushback mechanism
 ---

 Key: HBASE-5162
 URL: https://issues.apache.org/jira/browse/HBASE-5162
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jesse Yates
 Fix For: 2.0.0, 1.1.0

 Attachments: 5162-addendum2.txt, hbase-5162-branch-1-v0.patch, 
 hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, 
 hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, 
 hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, 
 hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, 
 hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, 
 hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, 
 hbase-5162-trunk-v8.patch, java_HBASE-5162.patch


 The current blocking we do when we are close to some limits (memstores over 
 the multiplier factor, too many store files, global memstore memory) is bad, 
 too coarse and confusing. After hitting HBASE-5161, it really becomes obvious 
 that we need something better.
 I did a little brainstorm with Stack, we came up quickly with two solutions:
  - Send some exception to the client, like OverloadedException, that's thrown 
 when some situation happens like getting past the low memory barrier. It 
 would be thrown when the client gets a handler and does some check while 
 putting or deleting. The client would treat this a retryable exception but 
 ideally wouldn't check .META. for a new location. It could be fancy and have 
 multiple levels of pushback, like send the exception to 25% of the clients, 
 and then go up if the situation persists. Should be easy to implement but 
 we'll be using a lot more IO to send the payload over and over again (but at 
 least it wouldn't sit in the RS's memory).
  - Send a message alongside a successful put or delete to tell the client to 
 slow down a little, this way we don't have to do back and forth with the 
 payload between the client and the server. It's a cleaner (I think) but more 
 involved solution.
 In every case the RS should do very obvious things to notify the operators of 
 this situation, through logs, web UI, metrics, etc.
 Other ideas?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-17 Thread Sean Busbey (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14250504#comment-14250504
]

Sean Busbey commented on HBASE-5162:

please make a backport ticket. that's what we've done for other things that
didn't make the 1.0 cut off.

Basic client pushback mechanism
---

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-17 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14250597#comment-14250597
]

Hadoop QA commented on HBASE-5162:
--

{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12687800/5162-addendum2.txt
against master branch at commit 422df8a3f430e1e41f971a644eec298f23decaf8.
ATTACHMENT ID: 12687800

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.

{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100

{color:green}+1 site{color}. The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12114//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12114//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12114//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12114//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12114//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12114//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12114//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12114//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12114//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12114//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12114//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12114//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12114//artifact/patchprocess/checkstyle-aggregate.html

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12114//console

This message is automatically generated.

Basic client pushback mechanism
---

Key: HBASE-5162
URL: https://issues.apache.org/jira/browse/HBASE-5162
Project: HBase
Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jesse Yates
Fix For: 2.0.0

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-17 Thread Andrew Purtell (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14250640#comment-14250640
]

Andrew Purtell commented on HBASE-5162:
---

[~jesse_yates], what do you think about filing another issue for follow on work
for the test... We could force flush as suggested above to set things at a
known state. Or we could maybe mock something.

Basic client pushback mechanism
---

Key: HBASE-5162
URL: https://issues.apache.org/jira/browse/HBASE-5162
Project: HBase
Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jesse Yates
Fix For: 2.0.0

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-17 Thread Jesse Yates (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14250645#comment-14250645
 ] 

Jesse Yates commented on HBASE-5162:


[~apurtell] filed and patch posted: HBASE-12703

 Basic client pushback mechanism
 ---

 Key: HBASE-5162
 URL: https://issues.apache.org/jira/browse/HBASE-5162
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jesse Yates
 Fix For: 2.0.0

 Attachments: 5162-addendum2.txt, hbase-5162-branch-1-v0.patch, 
 hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, 
 hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, 
 hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, 
 hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, 
 hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, 
 hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, 
 hbase-5162-trunk-v8.patch, java_HBASE-5162.patch


 The current blocking we do when we are close to some limits (memstores over 
 the multiplier factor, too many store files, global memstore memory) is bad, 
 too coarse and confusing. After hitting HBASE-5161, it really becomes obvious 
 that we need something better.
 I did a little brainstorm with Stack, we came up quickly with two solutions:
  - Send some exception to the client, like OverloadedException, that's thrown 
 when some situation happens like getting past the low memory barrier. It 
 would be thrown when the client gets a handler and does some check while 
 putting or deleting. The client would treat this a retryable exception but 
 ideally wouldn't check .META. for a new location. It could be fancy and have 
 multiple levels of pushback, like send the exception to 25% of the clients, 
 and then go up if the situation persists. Should be easy to implement but 
 we'll be using a lot more IO to send the payload over and over again (but at 
 least it wouldn't sit in the RS's memory).
  - Send a message alongside a successful put or delete to tell the client to 
 slow down a little, this way we don't have to do back and forth with the 
 payload between the client and the server. It's a cleaner (I think) but more 
 involved solution.
 In every case the RS should do very obvious things to notify the operators of 
 this situation, through logs, web UI, metrics, etc.
 Other ideas?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-17 Thread Andrew Purtell (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14250646#comment-14250646
]

Andrew Purtell commented on HBASE-5162:
---

Ok, oops. Thanks.

Basic client pushback mechanism
---

Key: HBASE-5162
URL: https://issues.apache.org/jira/browse/HBASE-5162
Project: HBase
Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jesse Yates
Fix For: 2.0.0

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-17 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14250683#comment-14250683
 ] 

Hudson commented on HBASE-5162:
---

SUCCESS: Integrated in HBase-TRUNK #5939 (See 
[https://builds.apache.org/job/HBase-TRUNK/5939/])
HBASE-5162 Addendum to stabilize TestClientPushback (tedyu: rev 
6aa8b3727c75bf8578599ec35afcaad245f13e30)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientPushback.java


 Basic client pushback mechanism
 ---

 Key: HBASE-5162
 URL: https://issues.apache.org/jira/browse/HBASE-5162
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jesse Yates
 Fix For: 2.0.0

 Attachments: 5162-addendum2.txt, hbase-5162-branch-1-v0.patch, 
 hbase-5162-trunk-addendum.patch, hbase-5162-trunk-v0.patch, 
 hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, 
 hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, 
 hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, 
 hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, 
 hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, 
 hbase-5162-trunk-v8.patch, java_HBASE-5162.patch


 The current blocking we do when we are close to some limits (memstores over 
 the multiplier factor, too many store files, global memstore memory) is bad, 
 too coarse and confusing. After hitting HBASE-5161, it really becomes obvious 
 that we need something better.
 I did a little brainstorm with Stack, we came up quickly with two solutions:
  - Send some exception to the client, like OverloadedException, that's thrown 
 when some situation happens like getting past the low memory barrier. It 
 would be thrown when the client gets a handler and does some check while 
 putting or deleting. The client would treat this a retryable exception but 
 ideally wouldn't check .META. for a new location. It could be fancy and have 
 multiple levels of pushback, like send the exception to 25% of the clients, 
 and then go up if the situation persists. Should be easy to implement but 
 we'll be using a lot more IO to send the payload over and over again (but at 
 least it wouldn't sit in the RS's memory).
  - Send a message alongside a successful put or delete to tell the client to 
 slow down a little, this way we don't have to do back and forth with the 
 payload between the client and the server. It's a cleaner (I think) but more 
 involved solution.
 In every case the RS should do very obvious things to notify the operators of 
 this situation, through logs, web UI, metrics, etc.
 Other ideas?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-16 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248738#comment-14248738
]

Hadoop QA commented on HBASE-5162:
--

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12687543/hbase-5162-trunk-v12-committed.patch
against master branch at commit a411227b0ebf78b4ee8ae7179e162b54734e77de.
ATTACHMENT ID: 12687543

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 19 new
or modified tests.

{color:red}-1 patch{color}. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12100//console

This message is automatically generated.

Basic client pushback mechanism
---

Attachments: hbase-5162-trunk-v0.patch, hbase-5162-trunk-v1.patch,
hbase-5162-trunk-v10.patch, hbase-5162-trunk-v11.patch,
hbase-5162-trunk-v12-committed.patch, hbase-5162-trunk-v2.patch,
hbase-5162-trunk-v3.patch, hbase-5162-trunk-v4.patch,
hbase-5162-trunk-v5.patch, hbase-5162-trunk-v6.patch,
hbase-5162-trunk-v7.patch, hbase-5162-trunk-v8.patch, java_HBASE-5162.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-16 Thread Andrew Purtell (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248780#comment-14248780
]

Andrew Purtell commented on HBASE-5162:
---

I have a feeling [~enis] won't take this at this late time for 1.0. What if we
open a branch (soon) for 1.1.

I think we can look at this for 0.98 if we don't break non-private interfaces
and we can get it back into 0.98 from master via any 1.x.

Basic client pushback mechanism
---

Attachments: hbase-5162-branch-1-v0.patch, hbase-5162-trunk-v0.patch,
hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch,
hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch,
hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch,
hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch,
hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch,
hbase-5162-trunk-v8.patch, java_HBASE-5162.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-16 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248946#comment-14248946
 ] 

Hudson commented on HBASE-5162:
---

FAILURE: Integrated in HBase-TRUNK #5930 (See 
[https://builds.apache.org/job/HBase-TRUNK/5930/])
HBASE-5162 Basic client pushback mechanism (jyates: rev 
a411227b0ebf78b4ee8ae7179e162b54734e77de)
* 
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientExponentialBackoff.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/DelayingRunner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEditsReplaySink.java
* hbase-protocol/src/main/protobuf/Client.proto
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerFactory.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiAction.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ExponentialClientBackoffPolicy.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCaller.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ClientBackoffPolicyFactory.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/StatsTrackingRpcRetryingCaller.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Result.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ClientBackoffPolicy.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerImpl.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerStatisticTracker.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterConnection.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientPushback.java
* 
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/ResultStatsUtil.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* 
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestFastFailWithoutTestUtil.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestReplicasClient.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionAdapter.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/backoff/ServerStatistics.java
* 
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RegionCoprocessorRpcChannel.java


 Basic client pushback mechanism
 ---

 Key: HBASE-5162
 URL: https://issues.apache.org/jira/browse/HBASE-5162
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jesse Yates
 Fix For: 1.0.0, 2.0.0

 Attachments: hbase-5162-branch-1-v0.patch, hbase-5162-trunk-v0.patch, 
 hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, 
 hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, 
 hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, 
 hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, 
 hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, 
 hbase-5162-trunk-v8.patch, java_HBASE-5162.patch


 The current blocking we do when we are close to some limits (memstores over 
 the multiplier factor, too many store files, global memstore memory) is bad, 
 too coarse and confusing. After hitting HBASE-5161, it really becomes obvious 
 that we need something better.
 I did a little brainstorm with Stack, we came up quickly with two solutions:
  - Send some exception to the client, like OverloadedException, that's thrown 
 when some situation happens like getting past the low memory barrier. It 
 would be thrown when the client gets a handler and does some check while 
 putting or deleting. The client would treat this a retryable exception but 
 ideally wouldn't check .META. for a new location. It could be fancy and have 
 multiple levels of pushback, like send the exception to 25% of the clients, 
 and

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-16 Thread Jesse Yates (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248961#comment-14248961
]

Jesse Yates commented on HBASE-5162:

Hmmm, well that's an issue. Looking into the NPE now.

Basic client pushback mechanism
---

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-16 Thread Andrew Purtell (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248968#comment-14248968
]

Andrew Purtell commented on HBASE-5162:
---

Thanks. If it's a minor fix please commit an addendum. If it's more involved,
we can revert and commit again later.

Basic client pushback mechanism
---

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-16 Thread Nick Dimiduk (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249066#comment-14249066
]

Nick Dimiduk commented on HBASE-5162:
-

bq. What if we open a branch (soon) for 1.1.

Why wait for 1.1, any reason this couldn't go into 1.0.1? I mean, if it's okay
for 0.98.x, why not 1.0.x?

Basic client pushback mechanism
---

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-16 Thread Andrew Purtell (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249079#comment-14249079
 ] 

Andrew Purtell commented on HBASE-5162:
---

Either option sounds good IMHO :-)

 Basic client pushback mechanism
 ---

 Key: HBASE-5162
 URL: https://issues.apache.org/jira/browse/HBASE-5162
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Jesse Yates
 Fix For: 1.0.0, 2.0.0

 Attachments: hbase-5162-branch-1-v0.patch, hbase-5162-trunk-v0.patch, 
 hbase-5162-trunk-v1.patch, hbase-5162-trunk-v10.patch, 
 hbase-5162-trunk-v11.patch, hbase-5162-trunk-v12-committed.patch, 
 hbase-5162-trunk-v2.patch, hbase-5162-trunk-v3.patch, 
 hbase-5162-trunk-v4.patch, hbase-5162-trunk-v5.patch, 
 hbase-5162-trunk-v6.patch, hbase-5162-trunk-v7.patch, 
 hbase-5162-trunk-v8.patch, java_HBASE-5162.patch


 The current blocking we do when we are close to some limits (memstores over 
 the multiplier factor, too many store files, global memstore memory) is bad, 
 too coarse and confusing. After hitting HBASE-5161, it really becomes obvious 
 that we need something better.
 I did a little brainstorm with Stack, we came up quickly with two solutions:
  - Send some exception to the client, like OverloadedException, that's thrown 
 when some situation happens like getting past the low memory barrier. It 
 would be thrown when the client gets a handler and does some check while 
 putting or deleting. The client would treat this a retryable exception but 
 ideally wouldn't check .META. for a new location. It could be fancy and have 
 multiple levels of pushback, like send the exception to 25% of the clients, 
 and then go up if the situation persists. Should be easy to implement but 
 we'll be using a lot more IO to send the payload over and over again (but at 
 least it wouldn't sit in the RS's memory).
  - Send a message alongside a successful put or delete to tell the client to 
 slow down a little, this way we don't have to do back and forth with the 
 payload between the client and the server. It's a cleaner (I think) but more 
 involved solution.
 In every case the RS should do very obvious things to notify the operators of 
 this situation, through logs, web UI, metrics, etc.
 Other ideas?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-5162) Basic client pushback mechanism

2014-12-16 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249254#comment-14249254
]

Hadoop QA commented on HBASE-5162:
--

{color:green}+1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12687610/hbase-5162-trunk-addendum.patch
against master branch at commit 9e7f7211b95ef4e9c64f5e54411c7c9fa7eeb235.
ATTACHMENT ID: 12687610