[ 
https://issues.apache.org/jira/browse/HBASE-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13212781#comment-13212781
 ] 

jirapos...@reviews.apache.org commented on HBASE-5162:
------------------------------------------------------



bq.  On 2012-02-18 04:58:21, Lars Hofhansl wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/regionserver/StoreUtils.java, line 
7
bq.  > <https://reviews.apache.org/r/3930/diff/1/?file=75496#file75496line7>
bq.  >
bq.  >     General comment: Where are we on putting/documenting these things in 
hbase-defaults.xml?

That is already in there, but having the constants just reinforces the 
'correct' behavior. Though I think there are some situations where the code is 
different from defaults.xml


bq.  On 2012-02-18 04:58:21, Lars Hofhansl wrote:
bq.  > 
src/main/java/org/apache/hadoop/hbase/regionserver/MemstorePressureMonitor.java,
 line 124
bq.  > <https://reviews.apache.org/r/3930/diff/1/?file=75492#file75492line124>
bq.  >
bq.  >     Ah ok, this is where we gracefully delay the server thread a bit.
bq.  >     Seems this would need to be tweaked carefully to make it effective 
while not slowing normal operations.
bq.  >     
bq.  >     Should the serverPauseTime be somehow related to the amount of 
pressure.
bq.  >     I.e. wait a bit more if the pressure is higher?
bq.  >     Maybe the pausetime calculation should be part of the pluggable 
policy?
bq.  >     
bq.  >     Also in the jira there was some discussion about throwing 
(presumably retryable) exceptions back to the client is the pressure gets too 
high. That would slow the client, without consuming server resources (beyond 
multiple requests).

Definitely needs some knobs with the policy here and tuning.

It would be possible to tie the pause time to the amount of pressure (some sort 
of multiplier effect). Would have to think about the implications of that

Throwing the exception back the client here means you get a lot of bandwidth 
overhead as the client passes back a bunch of puts _again_, which is rough. 
Otherwise, we are *definitely* going to be messing with wire-compatibility in a 
functional (if not strictly RPC) sense.


bq.  On 2012-02-18 04:58:21, Lars Hofhansl wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/client/HTable.java, line 979
bq.  > <https://reviews.apache.org/r/3930/diff/1/?file=75487#file75487line979>
bq.  >
bq.  >     This will break backwards compatibility, right? (Not saying that's 
not ok, just calling it out)
bq.  >     
bq.  >     I'd almost rather have the client not know about this, until we 
reach a bad spot (in which case we can throw back retryable exceptions).

I don't think so. If the server isn't upgraded to the send monitoredResults, 
then this bit of code won't matter. If the client isn't upgraded, then it won't 
even consider if the MonitoredResult isn't just a Result.


bq.  On 2012-02-18 04:58:21, Lars Hofhansl wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/client/HTable.java, line 958
bq.  > <https://reviews.apache.org/r/3930/diff/1/?file=75487#file75487line958>
bq.  >
bq.  >     Was this a problem before? Or only now becaue of the background 
thread?

Just b/c of the background thread.


bq.  On 2012-02-18 04:58:21, Lars Hofhansl wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/client/HTable.java, line 792
bq.  > <https://reviews.apache.org/r/3930/diff/1/?file=75487#file75487line792>
bq.  >
bq.  >     If sleepTime is 0 (for example from NoServerBackoffPolicy), we 
should probably not create the thread and flush right here.
bq.  >     
bq.  >     (But as I said in the comment above, I'd probably not bother with 
this extra thread to begin with :) )

+1 fixing in next version, assuming the client stuff is kept


bq.  On 2012-02-18 04:58:21, Lars Hofhansl wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/client/HTable.java, line 791
bq.  > <https://reviews.apache.org/r/3930/diff/1/?file=75487#file75487line791>
bq.  >
bq.  >     What if the flusher is not null? Should we re-calculate the wait 
time?

We are temporarily growing the write buffer, so if we get another write, I was 
thinking we would just let it accumulate too - it will get flushed soon enough. 
Only corner case here is if the size grows beyond the allowed, in which case 
there will be a forced flush.


bq.  On 2012-02-18 04:58:21, Lars Hofhansl wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/client/HTable.java, line 790
bq.  > <https://reviews.apache.org/r/3930/diff/1/?file=75487#file75487line790>
bq.  >
bq.  >     I find this a bit dubious.
bq.  >     This won't actually slow the client thread down, but just accumulate 
more data and reduce the number of RPCs. In the end it might lead to more load 
on the server, because we can deliver more puts as with fewer but larger 
batches.
bq.  >     
bq.  >     I'd rather just rely on the server sleeping the thread for a bit (as 
you do later).

The question here is what we consider the 'client'. To me, the client was the 
client-side HTable, not the using application. So yes, this will not slow the 
using application, but that was the intention...keeping HBase seeming as 
responsive, but just letting the messaging layer handle the queuing and 
buffering.

Technically, this will actually slow the rate of writes _to the server_, which 
is what we really are worrying about. Essentially, this implementation is such 
that we temporarily increase batch size so the server has time to finish its 
compactions and flushing. 

The server can handle the larger batch sizes (except for extreme cases) since 
the assumption has to be clientMemory << serverMemory. 

This is just experimental :) I'll need to do some stress testing to actually 
see the effects


- Jesse


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3930/#review5209
-----------------------------------------------------------


On 2012-02-16 20:45:50, Jesse Yates wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3930/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2012-02-16 20:45:50)
bq.  
bq.  
bq.  Review request for hbase, Michael Stack, Jean-Daniel Cryans, and Lars 
Hofhansl.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  Under heavy write load, HBase will create a saw-tooth pattern in accepting 
writes. This is due to the I/O in minor compactions not being able to keep up 
with the write load. Specifically, the memstore is attempting to flush while we 
are attempting to do a minor compaction, leading to blocking _all_ writes. 
Instead, we need to have the option of graceful degradation mechanism.
bq.  
bq.  This patch supports both a short-term,adjustable server-side write 
blocking as well as client-side back-off to help alleviate temporary memstore 
pressure.
bq.  
bq.  
bq.  This addresses bug HBASE-5162.
bq.      https://issues.apache.org/jira/browse/HBASE-5162
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    src/main/java/org/apache/hadoop/hbase/HConstants.java 763fe89 
bq.    src/main/java/org/apache/hadoop/hbase/client/BackoffPolicy.java 
PRE-CREATION 
bq.    src/main/java/org/apache/hadoop/hbase/client/HTable.java 57605e6 
bq.    src/main/java/org/apache/hadoop/hbase/client/MonitoredResult.java 
PRE-CREATION 
bq.    src/main/java/org/apache/hadoop/hbase/client/NoServerBackoffPolicy.java 
PRE-CREATION 
bq.    src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 25cb31d 
bq.    src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 
7d7be3c 
bq.    
src/main/java/org/apache/hadoop/hbase/regionserver/MemstorePressureMonitor.java 
PRE-CREATION 
bq.    src/main/java/org/apache/hadoop/hbase/regionserver/OperationStatus.java 
1b94ab5 
bq.    src/main/java/org/apache/hadoop/hbase/regionserver/PressureMonitor.java 
PRE-CREATION 
bq.    
src/main/java/org/apache/hadoop/hbase/regionserver/SimpleMemStorePressureMonitor.java
 PRE-CREATION 
bq.    src/main/java/org/apache/hadoop/hbase/regionserver/StoreUtils.java 
PRE-CREATION 
bq.    src/test/java/org/apache/hadoop/hbase/client/TestClientBackOff.java 
PRE-CREATION 
bq.    
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStorePressureMonitor.java
 PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/3930/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jesse
bq.  
bq.


                
> Basic client pushback mechanism
> -------------------------------
>
>                 Key: HBASE-5162
>                 URL: https://issues.apache.org/jira/browse/HBASE-5162
>             Project: HBase
>          Issue Type: New Feature
>    Affects Versions: 0.92.0
>            Reporter: Jean-Daniel Cryans
>             Fix For: 0.94.0
>
>         Attachments: java_HBASE-5162.patch
>
>
> The current blocking we do when we are close to some limits (memstores over 
> the multiplier factor, too many store files, global memstore memory) is bad, 
> too coarse and confusing. After hitting HBASE-5161, it really becomes obvious 
> that we need something better.
> I did a little brainstorm with Stack, we came up quickly with two solutions:
>  - Send some exception to the client, like OverloadedException, that's thrown 
> when some situation happens like getting past the low memory barrier. It 
> would be thrown when the client gets a handler and does some check while 
> putting or deleting. The client would treat this a retryable exception but 
> ideally wouldn't check .META. for a new location. It could be fancy and have 
> multiple levels of pushback, like send the exception to 25% of the clients, 
> and then go up if the situation persists. Should be "easy" to implement but 
> we'll be using a lot more IO to send the payload over and over again (but at 
> least it wouldn't sit in the RS's memory).
>  - Send a message alongside a successful put or delete to tell the client to 
> slow down a little, this way we don't have to do back and forth with the 
> payload between the client and the server. It's a cleaner (I think) but more 
> involved solution.
> In every case the RS should do very obvious things to notify the operators of 
> this situation, through logs, web UI, metrics, etc.
> Other ideas?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Reply via email to