[jira] [Updated] (HBASE-5757) TableInputFormat should handle as many errors as possible

[Jan Lukavsky (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Lukavsky updated HBASE-5757:


Attachment: HBASE-5757.patch

Attaching patch including modified tests (pass on my box) and counter in the 
new API.

> TableInputFormat should handle as many errors as possible
> -
>
> Key: HBASE-5757
> URL: https://issues.apache.org/jira/browse/HBASE-5757
> Project: HBase
>  Issue Type: Bug
>  Components: mapred, mapreduce
>Affects Versions: 0.90.6
>Reporter: Jan Lukavsky
> Attachments: HBASE-5757.patch, HBASE-5757.patch
>
>
> Prior to HBASE-4196 there was different handling of IOExceptions thrown from 
> scanner in mapred and mapreduce API. The patch to HBASE-4196 unified this 
> handling so that if exception is caught a reconnect is attempted (without 
> bothering the mapred client). After that, HBASE-4269 changed this behavior 
> back, but in both mapred and mapreduce APIs. The question is, is there any 
> reason not to handle all errors that the input format can handle? In other 
> words, why not try to reissue the request after *any* IOException? I see the 
> following disadvantages of current approach
>  * the client may see exceptions like LeaseException and 
> ScannerTimeoutException if he fails to process all fetched data in timeout
>  * to avoid ScannerTimeoutException the client must raise 
> hbase.regionserver.lease.period
>  * timeouts for tasks is aready configured in mapred.task.timeout, so this 
> seems to me a bit redundant, because typically one needs to update both these 
> parameters
>  * I don't see any possibility to get rid of LeaseException (this is 
> configured on server side)
> I think all of these issues would be gone, if the DoNotRetryIOException would 
> not be rethrown. -On the other hand, handling errors in InputFormat has 
> disadvantage, that it may hide from the user some inefficiency. Eg. if I have 
> very big scanner.caching, and I manage to process only a few rows in timeout, 
> I will end up with single row being fetched many times (and will not be 
> explicitly notified about this). Could we solve this problem by adding some 
> counter to the InputFormat?-

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5757) TableInputFormat should handle as many errors as possible

[Jan Lukavsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13275793#comment-13275793
 ] 

Jan Lukavsky commented on HBASE-5757:
-

{quote}Note that we've been able to can set scanner caching on each individual 
scan in since 0.20 (HBASE-1759) – setting it for that job may be more 
'correct'. {quote}

We are setting different caching for different jobs, the problem is that the 
rows may take different time to process (based on job) and this cannot be told 
in advance. Currently, it is only possible to set the caching for the whole 
job, but even if it was possible to change the caching *during* the job, we 
would not know that we need to do it before we will get the 
ScannerTimeoutException. So handling this error in the TableInputFormat seems 
right solution to me.

> TableInputFormat should handle as many errors as possible
> -
>
> Key: HBASE-5757
> URL: https://issues.apache.org/jira/browse/HBASE-5757
> Project: HBase
>  Issue Type: Bug
>  Components: mapred, mapreduce
>Affects Versions: 0.90.6
>Reporter: Jan Lukavsky
> Attachments: HBASE-5757.patch, HBASE-5757.patch
>
>
> Prior to HBASE-4196 there was different handling of IOExceptions thrown from 
> scanner in mapred and mapreduce API. The patch to HBASE-4196 unified this 
> handling so that if exception is caught a reconnect is attempted (without 
> bothering the mapred client). After that, HBASE-4269 changed this behavior 
> back, but in both mapred and mapreduce APIs. The question is, is there any 
> reason not to handle all errors that the input format can handle? In other 
> words, why not try to reissue the request after *any* IOException? I see the 
> following disadvantages of current approach
>  * the client may see exceptions like LeaseException and 
> ScannerTimeoutException if he fails to process all fetched data in timeout
>  * to avoid ScannerTimeoutException the client must raise 
> hbase.regionserver.lease.period
>  * timeouts for tasks is aready configured in mapred.task.timeout, so this 
> seems to me a bit redundant, because typically one needs to update both these 
> parameters
>  * I don't see any possibility to get rid of LeaseException (this is 
> configured on server side)
> I think all of these issues would be gone, if the DoNotRetryIOException would 
> not be rethrown. -On the other hand, handling errors in InputFormat has 
> disadvantage, that it may hide from the user some inefficiency. Eg. if I have 
> very big scanner.caching, and I manage to process only a few rows in timeout, 
> I will end up with single row being fetched many times (and will not be 
> explicitly notified about this). Could we solve this problem by adding some 
> counter to the InputFormat?-

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5539) asynchbase PerformanceEvaluation

[Benoit Sigoure (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13275887#comment-13275887
 ] 

Benoit Sigoure commented on HBASE-5539:
---

Yeah I have preliminary results at http://goo.gl/mZAcK – it shows that 
asynchbase can be quite a bit faster than {{HTable}}, surprisingly perhaps 
especially for read-heavy workloads, as well as for write-heavy workloads with 
many threads, where {{HTable}} suffers from really poor concurrency.

BTW, asynchbase 1.3.0 has been released, so the patch I attached originally to 
this issue needs to be updated to change the dependency to be on that version.  
I'll post a new patch soon, unless someone beats me to it.

> asynchbase PerformanceEvaluation
> 
>
> Key: HBASE-5539
> URL: https://issues.apache.org/jira/browse/HBASE-5539
> Project: HBase
>  Issue Type: New Feature
>  Components: performance
>Reporter: Benoit Sigoure
>Assignee: Benoit Sigoure
>Priority: Minor
>  Labels: benchmark
> Attachments: 0001-asynchbase-PerformanceEvaluation.patch
>
>
> I plugged [asynchbase|https://github.com/stumbleupon/asynchbase] into 
> {{PerformanceEvaluation}}.  This enables testing asynchbase from 
> {{PerformanceEvaluation}} and comparing its performance to {{HTable}}.  Also 
> asynchbase doesn't come with any benchmark, so it was good that I was able to 
> plug it into {{PerformanceEvaluation}} relatively easily.
> I am in the processing of collecting results on a dev cluster running 0.92.1 
> and will publish them once they're ready.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5974) Scanner retry behavior with RPC timeout on next() seems incorrect

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-5974:
--

Attachment: HBASE-5974_0.94.patch

Patch for 94. 1st version.. there are some points to discuss which will be done 
in following comments

> Scanner retry behavior with RPC timeout on next() seems incorrect
> -
>
> Key: HBASE-5974
> URL: https://issues.apache.org/jira/browse/HBASE-5974
> Project: HBase
>  Issue Type: Bug
>  Components: client, regionserver
>Affects Versions: 0.90.7, 0.92.1, 0.94.0, 0.96.0
>Reporter: Todd Lipcon
>Priority: Critical
> Attachments: HBASE-5974_0.94.patch
>
>
> I'm seeing the following behavior:
> - set RPC timeout to a short value
> - call next() for some batch of rows, big enough so the client times out 
> before the result is returned
> - the HConnectionManager stuff will retry the next() call to the same server. 
> At this point, one of two things can happen: 1) the previous next() call will 
> still be processing, in which case you get a LeaseException, because it was 
> removed from the map during the processing, or 2) the next() call will 
> succeed but skip the prior batch of rows.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5974) Scanner retry behavior with RPC timeout on next() seems incorrect

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13275904#comment-13275904
 ] 

Anoop Sam John commented on HBASE-5974:
---

{code}
+if (-1 != cookie && cookie != rswc.cookie) {
+  this.scanners.remove(scannerName);   
+  this.leases.cancelLease(scannerName);
+  throw new ScannerCookieOutOfOrderException("Cookie from client and 
cookie at server do not match");
+}
{code}
When the cookie is out of order, we remove the scanner and cancel Lease.[Not 
waiting for the lease timeout] This is in same line as what happens with NSRE. 
One question which arise in my mind is do we need to contact the CP hooks of 
pre and post scanner close here?
Now in normal close() of scanner from client side and lease timeout based 
close, we call CP hooks. But what abt the scanner getting removed from RS when 
the NSRE happens. [Same applicable to ScannerCookieOutOfOrderException case too]

> Scanner retry behavior with RPC timeout on next() seems incorrect
> -
>
> Key: HBASE-5974
> URL: https://issues.apache.org/jira/browse/HBASE-5974
> Project: HBase
>  Issue Type: Bug
>  Components: client, regionserver
>Affects Versions: 0.90.7, 0.92.1, 0.94.0, 0.96.0
>Reporter: Todd Lipcon
>Priority: Critical
> Attachments: HBASE-5974_0.94.patch
>
>
> I'm seeing the following behavior:
> - set RPC timeout to a short value
> - call next() for some batch of rows, big enough so the client times out 
> before the result is returned
> - the HConnectionManager stuff will retry the next() call to the same server. 
> At this point, one of two things can happen: 1) the previous next() call will 
> still be processing, in which case you get a LeaseException, because it was 
> removed from the map during the processing, or 2) the next() call will 
> succeed but skip the prior batch of rows.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5970) Improve the AssignmentManager#updateTimer and speed up handling opened event

[Zhihong Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5970:
--

Status: Patch Available  (was: Open)

> Improve the AssignmentManager#updateTimer and speed up handling opened event
> 
>
> Key: HBASE-5970
> URL: https://issues.apache.org/jira/browse/HBASE-5970
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Reporter: chunhui shen
>Assignee: chunhui shen
> Attachments: HBASE-5970.patch, HBASE-5970v2.patch, HBASE-5970v3.patch
>
>
> We found handing opened event very slow in the environment with lots of 
> regions.
> The problem is the slow AssignmentManager#updateTimer.
> We do the test for bulk assigning 10w (i.e. 100k) regions, the whole process 
> of bulk assigning took 1 hours.
> 2012-05-06 20:31:49,201 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 10 
> region(s) round-robin across 5 server(s)
> 2012-05-06 21:26:32,103 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done
> I think we could do the improvement for the AssignmentManager#updateTimer: 
> Make a thread do this work.
> After the improvement, it took only 4.5mins
> 2012-05-07 11:03:36,581 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 10 
> region(s) across 5 server(s), retainAssignment=true 
> 2012-05-07 11:07:57,073 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5920) New Compactions Logic can silently prevent user-initiated compactions from occurring

[Derek Wollenstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Derek Wollenstein updated HBASE-5920:
-

Status: Patch Available  (was: Open)

This patch makes the following changes:
1) Trace-level logging whenever a compaction is requested
2) Debug-level logging whenever compaction is changed
3) If a user requests a major compaction, this will stay a major compaction 
even if it violated max files (easy to take this part out)
  3a) If a user-initiates a max compaction that requires too many files to be 
compacted, this will log an error.  
4) Migrates utility functions from HBaseTestCase (Deprecated?) to 
HBaseTestingUtility to ease testing compaction behavior in TestCompaction

> New Compactions Logic can silently prevent user-initiated compactions from 
> occurring
> 
>
> Key: HBASE-5920
> URL: https://issues.apache.org/jira/browse/HBASE-5920
> Project: HBase
>  Issue Type: Bug
>  Components: client, regionserver
>Affects Versions: 0.92.1
>Reporter: Derek Wollenstein
>Priority: Minor
>  Labels: compaction
> Attachments: HBASE-5920.patch
>
>
> There seem to be some tuning settings in which manually triggered major 
> compactions will do nothing, including loggic
> From Store.java in the function
>   List compactSelection(List candidates)
> When a user manually triggers a compaction, this follows the same logic as a 
> normal compaction check.  when a user manually triggers a major compaction, 
> something similar happens.  Putting this all together:
> 1. If a user triggers a major compaction, this is checked against a max files 
> threshold (hbase.hstore.compaction.max). If the number of storefiles to 
> compact is > max files, then we downgrade to a minor compaction
> 2. If we are in a minor compaction, we do the following checks:
>a. If the file is less than a minimum size 
> (hbase.hstore.compaction.min.size) we automatically include it
>b. Otherwise, we check how the size compares to the next largest size.  
> based on hbase.hstore.compaction.ratio.  
>   c. If the number of files included is less than a minimum count 
> (hbase.hstore.compaction.min) then don't compact.
> In many of the exit strategies, we aren't seeing an error message.
> The net-net of this is that if we have a mix of very large and very small 
> files, we may end up having too many files to do a major compact, but too few 
> files to do a minor compact.
> I'm trying to go through and see if I'm understanding things correctly, but 
> this seems like the bug
> To put it another way
> 2012-05-02 20:09:36,389 DEBUG 
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Large Compaction 
> requested: 
> regionName=str,44594594594594592,1334939064521.f7aed25b55d4d7988af763bede9ce74e.,
>  store
> Name=c, fileCount=15, fileSize=1.5g (20.2k, 362.5m, 155.3k, 3.0m, 30.7k, 
> 361.2m, 6.9m, 4.7m, 14.7k, 363.4m, 30.9m, 3.2m, 7.3k, 362.9m, 23.5m), 
> priority=-9, time=3175046817624398; Because: Recursive enqueue; 
> compaction_queue=(59:0), split_queue=0
> When we had a minimum compaction size of 128M, and default settings for 
> hbase.hstore.compaction.min,hbase.hstore.compaction.max,hbase.hstore.compaction.ratio,
>  we were not getting a compaction to run even if we ran
> major_compact 
> 'str,44594594594594592,1334939064521.f7aed25b55d4d7988af763bede9ce74e.' from 
> the ruby shell.  Note that we had many tiny regions (20k, 155k, 3m, 30k,..) 
> and several large regions (362.5m,361.2m,363.4m,362.9m).  I think the bimodal 
> nature of the sizes prevented us from doing a compaction.
> I'm not 100% sure where this errored out because when I manually triggered a 
> compaction, I did not see
> '  // if we don't have enough files to compact, just wait 
>   if (filesToCompact.size() < this.minFilesToCompact) {  
> if (LOG.isDebugEnabled()) {  
>   LOG.debug("Skipped compaction of " + this.storeNameStr 
> + ".  Only " + (end - start) + " file(s) of size "   
> + StringUtils.humanReadableInt(totalSize)
> + " have met compaction criteria."); 
> }
> ' 
> being printed in the logs (and I know DEBUG logging was enabled because I saw 
> this elsewhere).  
> I'd be happy with better error messages when we decide not to compact for 
> user enabled compactions.
> I'd also like to see some override that says "user triggered major compaction 
> always occurs", but maybe that's a bad idea for other reasons.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, 

[jira] [Updated] (HBASE-5920) New Compactions Logic can silently prevent user-initiated compactions from occurring

[Derek Wollenstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Derek Wollenstein updated HBASE-5920:
-

Attachment: HBASE-5920.patch

I'm not sure if this is being done correctly, but I've provided a patch that 
seems to do the right thing here.  It includes a large change area because of 
the refactoring of HBaseTestCase utilities into HBaseTestingUtility.

> New Compactions Logic can silently prevent user-initiated compactions from 
> occurring
> 
>
> Key: HBASE-5920
> URL: https://issues.apache.org/jira/browse/HBASE-5920
> Project: HBase
>  Issue Type: Bug
>  Components: client, regionserver
>Affects Versions: 0.92.1
>Reporter: Derek Wollenstein
>Priority: Minor
>  Labels: compaction
> Attachments: HBASE-5920.patch
>
>
> There seem to be some tuning settings in which manually triggered major 
> compactions will do nothing, including loggic
> From Store.java in the function
>   List compactSelection(List candidates)
> When a user manually triggers a compaction, this follows the same logic as a 
> normal compaction check.  when a user manually triggers a major compaction, 
> something similar happens.  Putting this all together:
> 1. If a user triggers a major compaction, this is checked against a max files 
> threshold (hbase.hstore.compaction.max). If the number of storefiles to 
> compact is > max files, then we downgrade to a minor compaction
> 2. If we are in a minor compaction, we do the following checks:
>a. If the file is less than a minimum size 
> (hbase.hstore.compaction.min.size) we automatically include it
>b. Otherwise, we check how the size compares to the next largest size.  
> based on hbase.hstore.compaction.ratio.  
>   c. If the number of files included is less than a minimum count 
> (hbase.hstore.compaction.min) then don't compact.
> In many of the exit strategies, we aren't seeing an error message.
> The net-net of this is that if we have a mix of very large and very small 
> files, we may end up having too many files to do a major compact, but too few 
> files to do a minor compact.
> I'm trying to go through and see if I'm understanding things correctly, but 
> this seems like the bug
> To put it another way
> 2012-05-02 20:09:36,389 DEBUG 
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Large Compaction 
> requested: 
> regionName=str,44594594594594592,1334939064521.f7aed25b55d4d7988af763bede9ce74e.,
>  store
> Name=c, fileCount=15, fileSize=1.5g (20.2k, 362.5m, 155.3k, 3.0m, 30.7k, 
> 361.2m, 6.9m, 4.7m, 14.7k, 363.4m, 30.9m, 3.2m, 7.3k, 362.9m, 23.5m), 
> priority=-9, time=3175046817624398; Because: Recursive enqueue; 
> compaction_queue=(59:0), split_queue=0
> When we had a minimum compaction size of 128M, and default settings for 
> hbase.hstore.compaction.min,hbase.hstore.compaction.max,hbase.hstore.compaction.ratio,
>  we were not getting a compaction to run even if we ran
> major_compact 
> 'str,44594594594594592,1334939064521.f7aed25b55d4d7988af763bede9ce74e.' from 
> the ruby shell.  Note that we had many tiny regions (20k, 155k, 3m, 30k,..) 
> and several large regions (362.5m,361.2m,363.4m,362.9m).  I think the bimodal 
> nature of the sizes prevented us from doing a compaction.
> I'm not 100% sure where this errored out because when I manually triggered a 
> compaction, I did not see
> '  // if we don't have enough files to compact, just wait 
>   if (filesToCompact.size() < this.minFilesToCompact) {  
> if (LOG.isDebugEnabled()) {  
>   LOG.debug("Skipped compaction of " + this.storeNameStr 
> + ".  Only " + (end - start) + " file(s) of size "   
> + StringUtils.humanReadableInt(totalSize)
> + " have met compaction criteria."); 
> }
> ' 
> being printed in the logs (and I know DEBUG logging was enabled because I saw 
> this elsewhere).  
> I'd be happy with better error messages when we decide not to compact for 
> user enabled compactions.
> I'd also like to see some override that says "user triggered major compaction 
> always occurs", but maybe that's a bad idea for other reasons.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5979) Non-pread DFSInputStreams should be associated with scanners, not HFile.Readers

[Kannan Muthukkaruppan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13275996#comment-13275996
 ] 

Kannan Muthukkaruppan commented on HBASE-5979:
--

Todd: Nice catch! Your suggestion makes sense.

> Non-pread DFSInputStreams should be associated with scanners, not 
> HFile.Readers
> ---
>
> Key: HBASE-5979
> URL: https://issues.apache.org/jira/browse/HBASE-5979
> Project: HBase
>  Issue Type: Improvement
>  Components: performance, regionserver
>Reporter: Todd Lipcon
>
> Currently, every HFile.Reader has a single DFSInputStream, which it uses to 
> service all gets and scans. For gets, we use the positional read API (aka 
> "pread") and for scans we use a synchronized block to seek, then read. The 
> advantage of pread is that it doesn't hold any locks, so multiple gets can 
> proceed at the same time. The advantage of seek+read for scans is that the 
> datanode starts to send the entire rest of the HDFS block, rather than just 
> the single hfile block necessary. So, in a single thread, pread is faster for 
> gets, and seek+read is faster for scans since you get a strong pipelining 
> effect.
> However, in a multi-threaded case where there are multiple scans (including 
> scans which are actually part of compactions), the seek+read strategy falls 
> apart, since only one scanner may be reading at a time. Additionally, a large 
> amount of wasted IO is generated on the datanode side, and we get none of the 
> earlier-mentioned advantages.
> In one test, I switched scans to always use pread, and saw a 5x improvement 
> in throughput of the YCSB scan-only workload, since it previously was 
> completely blocked by contention on the DFSIS lock.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5546) Master assigns region in the original region server when opening region failed

[Ashutosh Jindal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Jindal updated HBASE-5546:
---

Attachment: hbase-5546_2.patch

Submitted updated patch for 0.96. Please review and provide your 
suggestion/comments.

> Master assigns region in the original region server when opening region 
> failed  
> 
>
> Key: HBASE-5546
> URL: https://issues.apache.org/jira/browse/HBASE-5546
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.0
>Reporter: gaojinchao
>Assignee: Ashutosh Jindal
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: hbase-5546.patch, hbase-5546_1.patch, hbase-5546_2.patch
>
>
> Master assigns region in the original region server when 
> RS_ZK_REGION_FAILED_OPEN envent was coming.
> Maybe we should choose other region server.
> [2012-03-07 10:14:21,750] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:14:31,826] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:14:41,903] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:14:51,975] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:02,056] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:12,167] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:22,231] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:32,303] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:42,375] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:52,447] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:16:02,528] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:16:12,600] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:16:22,676] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5958) Replace ByteString.copyFrom with new ByteString()

[Jimmy Xiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276001#comment-13276001
 ] 

Jimmy Xiang commented on HBASE-5958:


By the way, is it an option to use reflection to access the private 
constructor? If so, I can have a wrap method to use the private constructor, or 
the original copyFrom if the private constructor is not accessible. Reflection 
has overhead of course.

> Replace ByteString.copyFrom with new ByteString()
> -
>
> Key: HBASE-5958
> URL: https://issues.apache.org/jira/browse/HBASE-5958
> Project: HBase
>  Issue Type: Improvement
>  Components: performance
>Affects Versions: 0.96.0
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
>
> ByteString.copyFrom makes a copy of a byte array in case it is changed in 
> other thread.
> In most case, we don't need to worry about that.  We should avoid copying the 
> bytes
> for performance issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5959) Add other load balancers

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-5959:
-

Attachment: HBASE-5959-6.patch

Not sure why Phabricator is not attaching a good patch.  Here is a patch 
version of what is up on reviews.facebook.net.

Attaching so that hadoopQA can take a crack at it.

> Add other load balancers
> 
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
>  Issue Type: New Feature
>  Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, 
> HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, 
> HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch
>
>
> Now that balancers are pluggable we should give some options.b

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5979) Non-pread DFSInputStreams should be associated with scanners, not HFile.Readers

[Todd Lipcon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276018#comment-13276018
 ] 

Todd Lipcon commented on HBASE-5979:


Kannan: I tried to work on this a bit in my spare time, but didn't get very 
far. So if FB folks have cycles to work on it, that would be awesome!

I think one route is to do like I suggested above and have the 
StoreFileScanners hold a DFSInputStream. Another option would be to make a 
wrapper FileSystem (or FSReader) which pools a few streams. Then change the 
scanners to always issue positional reads, and have the wrapper code look for 
any stream which is already seeked to the right position (or just before the 
right position). The advantage of this technique is that we'd end up getting 
the same sequential read benefit, even if the user was issuing normal get() 
calls in ascending row order.

> Non-pread DFSInputStreams should be associated with scanners, not 
> HFile.Readers
> ---
>
> Key: HBASE-5979
> URL: https://issues.apache.org/jira/browse/HBASE-5979
> Project: HBase
>  Issue Type: Improvement
>  Components: performance, regionserver
>Reporter: Todd Lipcon
>
> Currently, every HFile.Reader has a single DFSInputStream, which it uses to 
> service all gets and scans. For gets, we use the positional read API (aka 
> "pread") and for scans we use a synchronized block to seek, then read. The 
> advantage of pread is that it doesn't hold any locks, so multiple gets can 
> proceed at the same time. The advantage of seek+read for scans is that the 
> datanode starts to send the entire rest of the HDFS block, rather than just 
> the single hfile block necessary. So, in a single thread, pread is faster for 
> gets, and seek+read is faster for scans since you get a strong pipelining 
> effect.
> However, in a multi-threaded case where there are multiple scans (including 
> scans which are actually part of compactions), the seek+read strategy falls 
> apart, since only one scanner may be reading at a time. Additionally, a large 
> amount of wasted IO is generated on the datanode side, and we get none of the 
> earlier-mentioned advantages.
> In one test, I switched scans to always use pread, and saw a 5x improvement 
> in throughput of the YCSB scan-only workload, since it previously was 
> completely blocked by contention on the DFSIS lock.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5958) Replace ByteString.copyFrom with new ByteString()

[Todd Lipcon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276021#comment-13276021
 ] 

Todd Lipcon commented on HBASE-5958:


Makes me nervous to reach in and use the private constructor... do you have 
some benchmarks that show that there's a noticeable speedup by doing so?

> Replace ByteString.copyFrom with new ByteString()
> -
>
> Key: HBASE-5958
> URL: https://issues.apache.org/jira/browse/HBASE-5958
> Project: HBase
>  Issue Type: Improvement
>  Components: performance
>Affects Versions: 0.96.0
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
>
> ByteString.copyFrom makes a copy of a byte array in case it is changed in 
> other thread.
> In most case, we don't need to worry about that.  We should avoid copying the 
> bytes
> for performance issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5920) New Compactions Logic can silently prevent user-initiated compactions from occurring

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276030#comment-13276030
 ] 

Hadoop QA commented on HBASE-5920:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527464/HBASE-5920.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 26 new or modified tests.

-1 patch.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1871//console

This message is automatically generated.

> New Compactions Logic can silently prevent user-initiated compactions from 
> occurring
> 
>
> Key: HBASE-5920
> URL: https://issues.apache.org/jira/browse/HBASE-5920
> Project: HBase
>  Issue Type: Bug
>  Components: client, regionserver
>Affects Versions: 0.92.1
>Reporter: Derek Wollenstein
>Priority: Minor
>  Labels: compaction
> Attachments: HBASE-5920.patch
>
>
> There seem to be some tuning settings in which manually triggered major 
> compactions will do nothing, including loggic
> From Store.java in the function
>   List compactSelection(List candidates)
> When a user manually triggers a compaction, this follows the same logic as a 
> normal compaction check.  when a user manually triggers a major compaction, 
> something similar happens.  Putting this all together:
> 1. If a user triggers a major compaction, this is checked against a max files 
> threshold (hbase.hstore.compaction.max). If the number of storefiles to 
> compact is > max files, then we downgrade to a minor compaction
> 2. If we are in a minor compaction, we do the following checks:
>a. If the file is less than a minimum size 
> (hbase.hstore.compaction.min.size) we automatically include it
>b. Otherwise, we check how the size compares to the next largest size.  
> based on hbase.hstore.compaction.ratio.  
>   c. If the number of files included is less than a minimum count 
> (hbase.hstore.compaction.min) then don't compact.
> In many of the exit strategies, we aren't seeing an error message.
> The net-net of this is that if we have a mix of very large and very small 
> files, we may end up having too many files to do a major compact, but too few 
> files to do a minor compact.
> I'm trying to go through and see if I'm understanding things correctly, but 
> this seems like the bug
> To put it another way
> 2012-05-02 20:09:36,389 DEBUG 
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Large Compaction 
> requested: 
> regionName=str,44594594594594592,1334939064521.f7aed25b55d4d7988af763bede9ce74e.,
>  store
> Name=c, fileCount=15, fileSize=1.5g (20.2k, 362.5m, 155.3k, 3.0m, 30.7k, 
> 361.2m, 6.9m, 4.7m, 14.7k, 363.4m, 30.9m, 3.2m, 7.3k, 362.9m, 23.5m), 
> priority=-9, time=3175046817624398; Because: Recursive enqueue; 
> compaction_queue=(59:0), split_queue=0
> When we had a minimum compaction size of 128M, and default settings for 
> hbase.hstore.compaction.min,hbase.hstore.compaction.max,hbase.hstore.compaction.ratio,
>  we were not getting a compaction to run even if we ran
> major_compact 
> 'str,44594594594594592,1334939064521.f7aed25b55d4d7988af763bede9ce74e.' from 
> the ruby shell.  Note that we had many tiny regions (20k, 155k, 3m, 30k,..) 
> and several large regions (362.5m,361.2m,363.4m,362.9m).  I think the bimodal 
> nature of the sizes prevented us from doing a compaction.
> I'm not 100% sure where this errored out because when I manually triggered a 
> compaction, I did not see
> '  // if we don't have enough files to compact, just wait 
>   if (filesToCompact.size() < this.minFilesToCompact) {  
> if (LOG.isDebugEnabled()) {  
>   LOG.debug("Skipped compaction of " + this.storeNameStr 
> + ".  Only " + (end - start) + " file(s) of size "   
> + StringUtils.humanReadableInt(totalSize)
> + " have met compaction criteria."); 
> }
> ' 
> being printed in the logs (and I know DEBUG logging was enabled because I saw 
> this elsewhere).  
> I'd be happy with better error messages when we decide not to compact for 
> user enabled compactions.
> I'd also like to see some override that says "user triggered major compaction 
> always occurs", but maybe that's a bad idea for other reasons.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
htt

[jira] [Commented] (HBASE-5958) Replace ByteString.copyFrom with new ByteString()

[Jimmy Xiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276032#comment-13276032
 ] 

Jimmy Xiang commented on HBASE-5958:


I don't have a benchmark yet.  Based on profiling, copyFrom and toByteArray are 
some hotspots.

> Replace ByteString.copyFrom with new ByteString()
> -
>
> Key: HBASE-5958
> URL: https://issues.apache.org/jira/browse/HBASE-5958
> Project: HBase
>  Issue Type: Improvement
>  Components: performance
>Affects Versions: 0.96.0
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.96.0
>
>
> ByteString.copyFrom makes a copy of a byte array in case it is changed in 
> other thread.
> In most case, we don't need to worry about that.  We should avoid copying the 
> bytes
> for performance issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6000) Cleanup where we keep .proto files

[stack (JIRA)]

stack created HBASE-6000:


 Summary: Cleanup where we keep .proto files
 Key: HBASE-6000
 URL: https://issues.apache.org/jira/browse/HBASE-6000
 Project: HBase
  Issue Type: Bug
Reporter: stack


I see Andrew for his pb work over in rest has .protos files under 
src/main/resources.  We should unify where these files live.  The recently 
added .protos place them under src/main/protobuf  Its confusing.

The thift idl files are here under resources too.

Seems like we should move src/main/protobuf under src/resources to be 
consistent.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5999) AggregationClient throws an exception when startRow is set and stopRow is not set in scan object.

[Zhihong Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276045#comment-13276045
 ] 

Zhihong Yu commented on HBASE-5999:
---

I think we can relax the original constraint by checking scan.hasFilter()
If a filter is present, there is no need to set stopRow.

> AggregationClient throws an exception when startRow is set and stopRow is not 
> set in scan object.
> -
>
> Key: HBASE-5999
> URL: https://issues.apache.org/jira/browse/HBASE-5999
> Project: HBase
>  Issue Type: Bug
>  Components: client, coprocessors
>Affects Versions: 0.92.0, 0.92.1
>Reporter: Anil Gupta
>Priority: Minor
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> AggregationClient throws an exception when the startRow is set and stopRow is 
> not set in scan object. AggregationClient should not throw the exception in 
> this case because the user might want to scan the entire table starting from 
> the startRow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6001) Upgrade slf4j to 1.6.1

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HBASE-6001:
--

 Summary: Upgrade slf4j to 1.6.1
 Key: HBASE-6001
 URL: https://issues.apache.org/jira/browse/HBASE-6001
 Project: HBase
  Issue Type: Task
Reporter: Jimmy Xiang


We need to upgrade slf4j to 1.6.1 since other hadoop components use 1.6.1 now.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6000) Cleanup where we keep .proto files

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276053#comment-13276053
 ] 

Andrew Purtell commented on HBASE-6000:
---

+1

I propose this convention:

- IDL files should be under src/resources/, by package plus protobuf/, e.g. 
src/resources/org/apache/hadoop/hbase/rest/protobuf/

- Generated package names should go into the source tree with the suffix 
'.generated', e.g. src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated


> Cleanup where we keep .proto files
> --
>
> Key: HBASE-6000
> URL: https://issues.apache.org/jira/browse/HBASE-6000
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>
> I see Andrew for his pb work over in rest has .protos files under 
> src/main/resources.  We should unify where these files live.  The recently 
> added .protos place them under src/main/protobuf  Its confusing.
> The thift idl files are here under resources too.
> Seems like we should move src/main/protobuf under src/resources to be 
> consistent.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6002) Possible chance of resource leak in HlogSplitter

[Chinna Rao Lalam (JIRA)]

Chinna Rao Lalam created HBASE-6002:
---

 Summary: Possible chance of resource leak in HlogSplitter
 Key: HBASE-6002
 URL: https://issues.apache.org/jira/browse/HBASE-6002
 Project: HBase
  Issue Type: Bug
  Components: wal
Affects Versions: 0.94.0
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam


In HLogSplitter.splitLogFileToTemp-Reader(in) is not closed and in finally 
block in loop while closing the writers(wap.w) if any exception comes other 
writers won't close.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6002) Possible chance of resource leak in HlogSplitter

[Chinna Rao Lalam (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276057#comment-13276057
 ] 

Chinna Rao Lalam commented on HBASE-6002:
-

Its good to close these writers in seperate finally block if any writer is not 
closed because of exception it will be closed here.

> Possible chance of resource leak in HlogSplitter
> 
>
> Key: HBASE-6002
> URL: https://issues.apache.org/jira/browse/HBASE-6002
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 0.94.0
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Attachments: HBASE-6002.patch
>
>
> In HLogSplitter.splitLogFileToTemp-Reader(in) is not closed and in finally 
> block in loop while closing the writers(wap.w) if any exception comes other 
> writers won't close.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6002) Possible chance of resource leak in HlogSplitter

[Chinna Rao Lalam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinna Rao Lalam updated HBASE-6002:


Attachment: HBASE-6002.patch

> Possible chance of resource leak in HlogSplitter
> 
>
> Key: HBASE-6002
> URL: https://issues.apache.org/jira/browse/HBASE-6002
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 0.94.0
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Attachments: HBASE-6002.patch
>
>
> In HLogSplitter.splitLogFileToTemp-Reader(in) is not closed and in finally 
> block in loop while closing the writers(wap.w) if any exception comes other 
> writers won't close.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6000) Cleanup where we keep .proto files

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276058#comment-13276058
 ] 

Andrew Purtell commented on HBASE-6000:
---

I don't think we should consider protobuf IDL as source. In my opinion it 
should not be necessary to have protoc installed to build HBase, just like it's 
not necessary to have the Thrift compiler available. However, I will grant 
someone with an opposing view that now Hadoop core requires protoc on the path 
on the build host. (That is annoying because nothing keeps its version in sync 
with the protobuf jar that Hadoop is pulling down.)

> Cleanup where we keep .proto files
> --
>
> Key: HBASE-6000
> URL: https://issues.apache.org/jira/browse/HBASE-6000
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>
> I see Andrew for his pb work over in rest has .protos files under 
> src/main/resources.  We should unify where these files live.  The recently 
> added .protos place them under src/main/protobuf  Its confusing.
> The thift idl files are here under resources too.
> Seems like we should move src/main/protobuf under src/resources to be 
> consistent.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276059#comment-13276059
 ] 

Lars Hofhansl commented on HBASE-5954:
--

This is only partially true. It is true for the sync packet stuff that I added, 
because this does not require closing the block.

If the block needs to be closed (which causes it to be fsync'ed) it is done 
after the ack from the downstream DN and before the ack to the upstream DN. 
Here in that case the fsyncs are serial. Looking at the code, that part seems 
hard to change.

Good news is: HLog files are smaller than a DFS block, so for HBase we never 
run into the 2nd issue.
Semi bad news: HFiles also need to fsync'ed at least on block close, so here 
we'd see the issue. But since HFiles are written asynchronously it should be OK.


> Allow proper fsync support for HBase
> 
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
>  Issue Type: Improvement
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk.txt, 
> hbase-hdfs-744.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6001) Upgrade slf4j to 1.6.1

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276064#comment-13276064
 ] 

Andrew Purtell commented on HBASE-6001:
---

I guess we need per build profile SLF4J version properties. If there are 
multiple versions of the SLF4J jar on the classpath, it looks bad, the shell 
dumps a warning about multiple bindings after the first command. 

> Upgrade slf4j to 1.6.1
> --
>
> Key: HBASE-6001
> URL: https://issues.apache.org/jira/browse/HBASE-6001
> Project: HBase
>  Issue Type: Task
>Reporter: Jimmy Xiang
>
> We need to upgrade slf4j to 1.6.1 since other hadoop components use 1.6.1 now.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5867) Improve Compaction Throttle Default

[Nicolas Spiegelberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Spiegelberg updated HBASE-5867:
---

   Resolution: Fixed
Fix Version/s: 0.96.0
   Status: Resolved  (was: Patch Available)

> Improve Compaction Throttle Default
> ---
>
> Key: HBASE-5867
> URL: https://issues.apache.org/jira/browse/HBASE-5867
> Project: HBase
>  Issue Type: Improvement
>Reporter: Nicolas Spiegelberg
>Assignee: Nicolas Spiegelberg
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: D2943.1.patch, HBASE-5867-trunk.patch
>
>
> We recently had a production issue where our compactions fell behind because 
> our compaction throttle was improperly tuned and accidentally upgraded all 
> compactions to the large pool.  The default from HBASE-3877 makes 1 bad 
> assumption: the default number of flushed files in a compaction.  Currently 
> the algorithm is:
> throttleSize ~= flushSize * 2
> This assumes that the basic compaction utilizes 3 files and that all 3 files 
> are compressed.  In this case, "hbase.hstore.compaction.min" == 6 && the 
> values were not very compressible.  Both conditions should be taken into 
> consideration.  As a default, it is less damaging for the large thread to be 
> slightly higher than it needs to be versus having everything accidentally 
> promoted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5546) Master assigns region in the original region server when opening region failed

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-5546:
--

Status: Open  (was: Patch Available)

> Master assigns region in the original region server when opening region 
> failed  
> 
>
> Key: HBASE-5546
> URL: https://issues.apache.org/jira/browse/HBASE-5546
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.0
>Reporter: gaojinchao
>Assignee: Ashutosh Jindal
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: hbase-5546.patch, hbase-5546_1.patch, hbase-5546_2.patch
>
>
> Master assigns region in the original region server when 
> RS_ZK_REGION_FAILED_OPEN envent was coming.
> Maybe we should choose other region server.
> [2012-03-07 10:14:21,750] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:14:31,826] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:14:41,903] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:14:51,975] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:02,056] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:12,167] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:22,231] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:32,303] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:42,375] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:52,447] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:16:02,528] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:16:12,600] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:16:22,676] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5546) Master assigns region in the original region server when opening region failed

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-5546:
--

Status: Patch Available  (was: Open)

> Master assigns region in the original region server when opening region 
> failed  
> 
>
> Key: HBASE-5546
> URL: https://issues.apache.org/jira/browse/HBASE-5546
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.0
>Reporter: gaojinchao
>Assignee: Ashutosh Jindal
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: hbase-5546.patch, hbase-5546_1.patch, hbase-5546_2.patch
>
>
> Master assigns region in the original region server when 
> RS_ZK_REGION_FAILED_OPEN envent was coming.
> Maybe we should choose other region server.
> [2012-03-07 10:14:21,750] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:14:31,826] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:14:41,903] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:14:51,975] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:02,056] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:12,167] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:22,231] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:32,303] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:42,375] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:52,447] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:16:02,528] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:16:12,600] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:16:22,676] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5959) Add other load balancers

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-5959:
---

Attachment: HBASE-5959.D3189.4.patch

eclark updated the revision "HBASE-5959 [jira] Add other load balancers".
Reviewers: JIRA

  Make TestRegionRebalancing test both DefaultLoadBalancer and the newer 
stochastic one.


REVISION DETAIL
  https://reviews.facebook.net/D3189

AFFECTED FILES
  pom.xml
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java
  src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
  src/main/java/org/apache/hadoop/hbase/master/RegionPlan.java
  src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
  src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java
  src/main/java/org/apache/hadoop/hbase/master/balancer/DefaultLoadBalancer.java
  src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java
  src/main/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerFactory.java
  src/main/java/org/apache/hadoop/hbase/master/LoadBalancerFactory.java
  
src/main/java/org/apache/hadoop/hbase/master/balancer/RegionInfoComparator.java
  src/main/java/org/apache/hadoop/hbase/master/ServerAndLoad.java
  
src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java
  src/main/java/org/apache/hadoop/hbase/master/balancer/ServerAndLoad.java
  
src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
  src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java
  src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
  src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java
  src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
  
src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java
  
src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java
  
src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java

To: JIRA, eclark


> Add other load balancers
> 
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
>  Issue Type: New Feature
>  Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, 
> HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, 
> HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch, 
> HBASE-5959.D3189.4.patch
>
>
> Now that balancers are pluggable we should give some options.b

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5546) Master assigns region in the original region server when opening region failed

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276077#comment-13276077
 ] 

Hadoop QA commented on HBASE-5546:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527466/hbase-5546_2.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 31 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1870//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1870//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1870//console

This message is automatically generated.

> Master assigns region in the original region server when opening region 
> failed  
> 
>
> Key: HBASE-5546
> URL: https://issues.apache.org/jira/browse/HBASE-5546
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.0
>Reporter: gaojinchao
>Assignee: Ashutosh Jindal
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: hbase-5546.patch, hbase-5546_1.patch, hbase-5546_2.patch
>
>
> Master assigns region in the original region server when 
> RS_ZK_REGION_FAILED_OPEN envent was coming.
> Maybe we should choose other region server.
> [2012-03-07 10:14:21,750] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:14:31,826] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:14:41,903] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:14:51,975] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:02,056] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:12,167] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:22,231] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:32,303] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:42,375] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:52,447] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:16:02,528] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:16:12,600] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 

[jira] [Commented] (HBASE-5999) AggregationClient throws an exception when startRow is set and stopRow is not set in scan object.

[Anil Gupta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276082#comment-13276082
 ] 

Anil Gupta commented on HBASE-5999:
---

Hi Zhihong,

I have got the fix for it and I discussed my fix with Ted Yu and he seemed ok 
with that. I just need to create a patch. Please refer to user mailing list 
discussion between me and Ted.
Essentially this is the change required in validateParameter method in 
AggregationClient class:
if (scan == null || (Bytes.equals(scan.getStartRow(), scan.getStopRow()) && 
!Bytes.equals(scan.getStartRow(), HConstants.EMPTY_START_ROW)) || 
(Bytes.compareTo(scan.getStartRow(), scan.getStopRow()) > 0 && 
!Bytes.equals(scan.getStopRow(), HConstants.EMPTY_END_ROW) ))

!Bytes.equals(scan.getStopRow(), HConstants.EMPTY_END_ROW) is the additional 
condition required to handle this case. I have tested my code and it works 
fine. Please let me know if this solution is not right. 

Thanks,
Anil Gupta


> AggregationClient throws an exception when startRow is set and stopRow is not 
> set in scan object.
> -
>
> Key: HBASE-5999
> URL: https://issues.apache.org/jira/browse/HBASE-5999
> Project: HBase
>  Issue Type: Bug
>  Components: client, coprocessors
>Affects Versions: 0.92.0, 0.92.1
>Reporter: Anil Gupta
>Priority: Minor
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> AggregationClient throws an exception when the startRow is set and stopRow is 
> not set in scan object. AggregationClient should not throw the exception in 
> this case because the user might want to scan the entire table starting from 
> the startRow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5970) Improve the AssignmentManager#updateTimer and speed up handling opened event

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276090#comment-13276090
 ] 

Hadoop QA commented on HBASE-5970:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12526715/HBASE-5970v3.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 31 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
 

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1872//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1872//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1872//console

This message is automatically generated.

> Improve the AssignmentManager#updateTimer and speed up handling opened event
> 
>
> Key: HBASE-5970
> URL: https://issues.apache.org/jira/browse/HBASE-5970
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Reporter: chunhui shen
>Assignee: chunhui shen
> Attachments: HBASE-5970.patch, HBASE-5970v2.patch, HBASE-5970v3.patch
>
>
> We found handing opened event very slow in the environment with lots of 
> regions.
> The problem is the slow AssignmentManager#updateTimer.
> We do the test for bulk assigning 10w (i.e. 100k) regions, the whole process 
> of bulk assigning took 1 hours.
> 2012-05-06 20:31:49,201 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 10 
> region(s) round-robin across 5 server(s)
> 2012-05-06 21:26:32,103 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done
> I think we could do the improvement for the AssignmentManager#updateTimer: 
> Make a thread do this work.
> After the improvement, it took only 4.5mins
> 2012-05-07 11:03:36,581 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 10 
> region(s) across 5 server(s), retainAssignment=true 
> 2012-05-07 11:07:57,073 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5959) Add other load balancers

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276092#comment-13276092
 ] 

Hadoop QA commented on HBASE-5959:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12527489/HBASE-5959.D3189.4.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 16 new or modified tests.

-1 patch.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1873//console

This message is automatically generated.

> Add other load balancers
> 
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
>  Issue Type: New Feature
>  Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, 
> HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, 
> HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch, 
> HBASE-5959.D3189.4.patch
>
>
> Now that balancers are pluggable we should give some options.b

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5959) Add other load balancers

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276093#comment-13276093
 ] 

Hadoop QA commented on HBASE-5959:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12527489/HBASE-5959.D3189.4.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 16 new or modified tests.

-1 patch.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1874//console

This message is automatically generated.

> Add other load balancers
> 
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
>  Issue Type: New Feature
>  Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, 
> HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, 
> HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, HBASE-5959.D3189.3.patch, 
> HBASE-5959.D3189.4.patch
>
>
> Now that balancers are pluggable we should give some options.b

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5987) HFileBlockIndex improvement

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-5987:
---

Attachment: D3237.1.patch

Liyin requested code review of "[jira][89-fb] [HBASE-5987] HFileBlockIndex 
improvement".
Reviewers: Kannan, mbautin

  Recently we find out a performance problem that it is quite slow when 
multiple requests are reading the same block of data or index.

  From the profiling, one of the causes is the IdLock contention which has been 
addressed in HBASE-5898.
  Another issue is that the HFileScanner will keep asking the HFileBlockIndex 
about the data block location for each target key value during the scan 
process(reSeekTo), even though the target key value has already been in the 
current data block. This issue will cause certain index block very HOT, 
especially when it is a sequential scan.


  To solve this issue, we propose to lookahead for one more block index so that 
the HFileScanner would know the start key value of next data block. So if the 
target key value for the scan(reSeekTo) is "smaller" than that start kv of next 
data block, it means the target key value has a very high possibility in the 
current data block (if not in current data block, then the start kv of next 
data block should be returned. Indexing on the start key has some defects here) 
and it shall NOT query the HFileBlockIndex in this case. On the contrary, if 
the target key value is "bigger", then it shall query the HFileBlockIndex. This 
improvement shall help to reduce the hotness of HFileBlockIndex and avoid some 
unnecessary IdLock Contention or Index Block Cache lookup.

TEST PLAN
  Unit tests

REVISION DETAIL
  https://reviews.facebook.net/D3237

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/HConstants.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockInfo.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
  src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java

MANAGE HERALD DIFFERENTIAL RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/7329/

To: Kannan, mbautin, Liyin
Cc: JIRA, todd


> HFileBlockIndex improvement
> ---
>
> Key: HBASE-5987
> URL: https://issues.apache.org/jira/browse/HBASE-5987
> Project: HBase
>  Issue Type: Improvement
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Attachments: D3237.1.patch, 
> screen_shot_of_sequential_scan_profiling.png
>
>
> Recently we find out a performance problem that it is quite slow when 
> multiple requests are reading the same block of data or index. 
> From the profiling, one of the causes is the IdLock contention which has been 
> addressed in HBASE-5898. 
> Another issue is that the HFileScanner will keep asking the HFileBlockIndex 
> about the data block location for each target key value during the scan 
> process(reSeekTo), even though the target key value has already been in the 
> current data block. This issue will cause certain index block very HOT, 
> especially when it is a sequential scan.
> To solve this issue, we propose the following solutions:
> First, we propose to lookahead for one more block index so that the 
> HFileScanner would know the start key value of next data block. So if the 
> target key value for the scan(reSeekTo) is "smaller" than that start kv of 
> next data block, it means the target key value has a very high possibility in 
> the current data block (if not in current data block, then the start kv of 
> next data block should be returned. +Indexing on the start key has some 
> defects here+) and it shall NOT query the HFileBlockIndex in this case. On 
> the contrary, if the target key value is "bigger", then it shall query the 
> HFileBlockIndex. This improvement shall help to reduce the hotness of 
> HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block 
> Cache lookup.
> Secondary, we propose to push this idea a little further that the 
> HFileBlockIndex shall index on the last key value of each data block instead 
> of indexing on the start key value. The motivation is to solve the HBASE-4443 
> issue (avoid seeking to "previous" block when key you are interested in is 
> the first one of a block) as well as +the defects mentioned above+.
> For example, if the target key value is "smaller" than the start key value of 
> the data block N. There is no way for sure the target key value is in the 
> data block N or N-1. So it has to seek from data block N-1. However, if the 
> block index is based on the last key value fo

[jira] [Updated] (HBASE-5935) Add Region-level PB-based calls to HMasterInterface

[Gregory Chanan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gregory Chanan updated HBASE-5935:
--

Attachment: HBASE-5935-v3.patch

Patch for latest version of trunk.

> Add Region-level PB-based calls to HMasterInterface
> ---
>
> Key: HBASE-5935
> URL: https://issues.apache.org/jira/browse/HBASE-5935
> Project: HBase
>  Issue Type: Task
>  Components: ipc, master, migration
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Fix For: 0.96.0
>
> Attachments: HBASE-5935-v2.patch, HBASE-5935-v2.patch, 
> HBASE-5935-v3.patch
>
>
> This should be a subtask of HBASE-5445, but since that is a subtask, I can't 
> also make this a subtask (apparently).
> This is for converting the region-level calls, i.e.:
> moveRegion
> assignRegion
> unassignRegion

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5959) Add other load balancers

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-5959:
-

Attachment: HBASE-5959-7.patch

Patch version again.

> Add other load balancers
> 
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
>  Issue Type: New Feature
>  Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, 
> HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, 
> HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, 
> HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch
>
>
> Now that balancers are pluggable we should give some options.b

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5935) Add Region-level PB-based calls to HMasterInterface

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276176#comment-13276176
 ] 

Hadoop QA commented on HBASE-5935:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527495/HBASE-5935-v3.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 6 new or modified tests.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 32 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1875//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1875//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1875//console

This message is automatically generated.

> Add Region-level PB-based calls to HMasterInterface
> ---
>
> Key: HBASE-5935
> URL: https://issues.apache.org/jira/browse/HBASE-5935
> Project: HBase
>  Issue Type: Task
>  Components: ipc, master, migration
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Fix For: 0.96.0
>
> Attachments: HBASE-5935-v2.patch, HBASE-5935-v2.patch, 
> HBASE-5935-v3.patch
>
>
> This should be a subtask of HBASE-5445, but since that is a subtask, I can't 
> also make this a subtask (apparently).
> This is for converting the region-level calls, i.e.:
> moveRegion
> assignRegion
> unassignRegion

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5959) Add other load balancers

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276191#comment-13276191
 ] 

Hadoop QA commented on HBASE-5959:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527497/HBASE-5959-7.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 16 new or modified tests.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 33 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1876//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1876//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1876//console

This message is automatically generated.

> Add other load balancers
> 
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
>  Issue Type: New Feature
>  Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, 
> HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, 
> HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, 
> HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch
>
>
> Now that balancers are pluggable we should give some options.b

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5987) HFileBlockIndex improvement

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276211#comment-13276211
 ] 

Phabricator commented on HBASE-5987:


tedyu has commented on the revision "[jira][89-fb] [HBASE-5987] HFileBlockIndex 
improvement".

INLINE COMMENTS
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java:411 'is to 
keep' -> 'keeps'
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java:415 'it 
means it' -> 'it means that'
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java:205 
Please add javadoc for the last three parameters
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java:208 Can 
this method be named getDataBlockInfo() ?
  For 'seekTo', I think DataBlock would be the target, not DataBlockInfo.
  See comment below w.r.t. naming of DataBlockInfo
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java:196 
'other attributes' -> 'additional attributes' ?
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java:293 'Only 
' can be removed.
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockInfo.java:2 No year, 
please.
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java:306 Can 
we use builder pattern to fill out nextIndexedKey ?
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockInfo.java:26 Would 
HFileBlockWithInfo be a better name ?
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java:480 Should 
this be '< 0' ?
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java:2 
Please remove year.
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java:44 
Please add test category.

REVISION DETAIL
  https://reviews.facebook.net/D3237

To: Kannan, mbautin, Liyin
Cc: JIRA, todd, tedyu


> HFileBlockIndex improvement
> ---
>
> Key: HBASE-5987
> URL: https://issues.apache.org/jira/browse/HBASE-5987
> Project: HBase
>  Issue Type: Improvement
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Attachments: D3237.1.patch, 
> screen_shot_of_sequential_scan_profiling.png
>
>
> Recently we find out a performance problem that it is quite slow when 
> multiple requests are reading the same block of data or index. 
> From the profiling, one of the causes is the IdLock contention which has been 
> addressed in HBASE-5898. 
> Another issue is that the HFileScanner will keep asking the HFileBlockIndex 
> about the data block location for each target key value during the scan 
> process(reSeekTo), even though the target key value has already been in the 
> current data block. This issue will cause certain index block very HOT, 
> especially when it is a sequential scan.
> To solve this issue, we propose the following solutions:
> First, we propose to lookahead for one more block index so that the 
> HFileScanner would know the start key value of next data block. So if the 
> target key value for the scan(reSeekTo) is "smaller" than that start kv of 
> next data block, it means the target key value has a very high possibility in 
> the current data block (if not in current data block, then the start kv of 
> next data block should be returned. +Indexing on the start key has some 
> defects here+) and it shall NOT query the HFileBlockIndex in this case. On 
> the contrary, if the target key value is "bigger", then it shall query the 
> HFileBlockIndex. This improvement shall help to reduce the hotness of 
> HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block 
> Cache lookup.
> Secondary, we propose to push this idea a little further that the 
> HFileBlockIndex shall index on the last key value of each data block instead 
> of indexing on the start key value. The motivation is to solve the HBASE-4443 
> issue (avoid seeking to "previous" block when key you are interested in is 
> the first one of a block) as well as +the defects mentioned above+.
> For example, if the target key value is "smaller" than the start key value of 
> the data block N. There is no way for sure the target key value is in the 
> data block N or N-1. So it has to seek from data block N-1. However, if the 
> block index is based on the last key value for each data block and the target 
> key value is beween the last key value of data block N-1 and data block N, 
> then the target key value is supposed be data block N for sure. 
> As long as HBase only supports the forward scan, the last key value makes 
> more sense to be indexed on than the start key value. 
> Thanks Kannan and Mikhail for the insightful discussions and suggestions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!defaul

[jira] [Created] (HBASE-6003) [refguide] performance.xml - fixed 2 links to Case Studies that were ???

[Doug Meil (JIRA)]

Doug Meil created HBASE-6003:


 Summary: [refguide] performance.xml - fixed 2 links to Case 
Studies that were ??? 
 Key: HBASE-6003
 URL: https://issues.apache.org/jira/browse/HBASE-6003
 Project: HBase
  Issue Type: Bug
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor


There were 2 links in the performance chapter that were ??? links that should 
have been going to Case Studies.

Corrected them so that they weren't dead links.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5998) Bulk assignment: regionserver optimization by using a temporary cache for table descriptors when receveing an open regions request

[Zhihong Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276213#comment-13276213
 ] 

Zhihong Yu commented on HBASE-5998:
---

Patch looks good.

> Bulk assignment: regionserver optimization by using a temporary cache for 
> table descriptors when receveing an open regions request
> --
>
> Key: HBASE-5998
> URL: https://issues.apache.org/jira/browse/HBASE-5998
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 0.96.0
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5998.v2.patch, 5998.v3.patch
>
>
> During the assignment, on the regionserver, before creating the handlers we 
> load the table description. Even if there is a cache, we check the timestamps 
> for each region, while it's not necessary. The test below is just with one 
> node, with more nodes the benefit will improve. By limiting the time spent in 
> HRegion#openRegion we increase the parallelization during cluster startup, as 
> the master is using a pool of threads to call the RS.
> -- Without the fix
> 2012-05-14 11:40:52,501 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 1193 
> region(s) to localhost,11003,1336988444043
> 2012-05-14 11:41:09,947 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done for 
> localhost,11003,1336988444043
> -- With the fix
> 2012-05-14 11:34:40,444 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 1193 
> region(s) to localhost,11003,1336988444043
> 2012-05-14 11:34:40,929 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done for 
> localhost,11003,1336988065948

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5959) Add other load balancers

[Zhihong Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276215#comment-13276215
 ] 

Zhihong Yu commented on HBASE-5959:
---

@Elliot:
Were all review comments addressed ?

Feel free to enumerate the ones you have different opinion with.

> Add other load balancers
> 
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
>  Issue Type: New Feature
>  Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, 
> HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, 
> HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, 
> HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch
>
>
> Now that balancers are pluggable we should give some options.b

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6004) Adding more logging to help debugging MR job

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HBASE-6004:
--

 Summary: Adding more logging to help debugging MR job
 Key: HBASE-6004
 URL: https://issues.apache.org/jira/browse/HBASE-6004
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.94.0, 0.96.0
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
 Fix For: 0.96.0, 0.94.0


MR job sometime fails because scanner expired. In this case, it will be helpful 
to know the last successful row, the ip of the region sever, and so on.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6003) [refguide] performance.xml - fixed 2 links to Case Studies that were ???

[Doug Meil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-6003:
-

Attachment: performance_hbase_6003.xml.patch

> [refguide] performance.xml - fixed 2 links to Case Studies that were ??? 
> -
>
> Key: HBASE-6003
> URL: https://issues.apache.org/jira/browse/HBASE-6003
> Project: HBase
>  Issue Type: Bug
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: performance_hbase_6003.xml.patch
>
>
> There were 2 links in the performance chapter that were ??? links that should 
> have been going to Case Studies.
> Corrected them so that they weren't dead links.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5970) Improve the AssignmentManager#updateTimer and speed up handling opened event

[Zhihong Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihong Yu updated HBASE-5970:
--

Attachment: 5970v3.patch

Re-attaching patch v3.

> Improve the AssignmentManager#updateTimer and speed up handling opened event
> 
>
> Key: HBASE-5970
> URL: https://issues.apache.org/jira/browse/HBASE-5970
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Reporter: chunhui shen
>Assignee: chunhui shen
> Attachments: 5970v3.patch, HBASE-5970.patch, HBASE-5970v2.patch, 
> HBASE-5970v3.patch
>
>
> We found handing opened event very slow in the environment with lots of 
> regions.
> The problem is the slow AssignmentManager#updateTimer.
> We do the test for bulk assigning 10w (i.e. 100k) regions, the whole process 
> of bulk assigning took 1 hours.
> 2012-05-06 20:31:49,201 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 10 
> region(s) round-robin across 5 server(s)
> 2012-05-06 21:26:32,103 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done
> I think we could do the improvement for the AssignmentManager#updateTimer: 
> Make a thread do this work.
> After the improvement, it took only 4.5mins
> 2012-05-07 11:03:36,581 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 10 
> region(s) across 5 server(s), retainAssignment=true 
> 2012-05-07 11:07:57,073 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6003) [refguide] performance.xml - fixed 2 links to Case Studies that were ???

[Doug Meil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-6003:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> [refguide] performance.xml - fixed 2 links to Case Studies that were ??? 
> -
>
> Key: HBASE-6003
> URL: https://issues.apache.org/jira/browse/HBASE-6003
> Project: HBase
>  Issue Type: Bug
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: performance_hbase_6003.xml.patch
>
>
> There were 2 links in the performance chapter that were ??? links that should 
> have been going to Case Studies.
> Corrected them so that they weren't dead links.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6003) [refguide] performance.xml - fixed 2 links to Case Studies that were ???

[Doug Meil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Meil updated HBASE-6003:
-

Status: Patch Available  (was: Open)

> [refguide] performance.xml - fixed 2 links to Case Studies that were ??? 
> -
>
> Key: HBASE-6003
> URL: https://issues.apache.org/jira/browse/HBASE-6003
> Project: HBase
>  Issue Type: Bug
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: performance_hbase_6003.xml.patch
>
>
> There were 2 links in the performance chapter that were ??? links that should 
> have been going to Case Studies.
> Corrected them so that they weren't dead links.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5546) Master assigns region in the original region server when opening region failed

[Zhihong Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276223#comment-13276223
 ] 

Zhihong Yu commented on HBASE-5546:
---

I think the getRegionPlan() call makes clearRegionPlan() unnecessary because 
forceNewPlan parameter is true.

> Master assigns region in the original region server when opening region 
> failed  
> 
>
> Key: HBASE-5546
> URL: https://issues.apache.org/jira/browse/HBASE-5546
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.0
>Reporter: gaojinchao
>Assignee: Ashutosh Jindal
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: hbase-5546.patch, hbase-5546_1.patch, hbase-5546_2.patch
>
>
> Master assigns region in the original region server when 
> RS_ZK_REGION_FAILED_OPEN envent was coming.
> Maybe we should choose other region server.
> [2012-03-07 10:14:21,750] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:14:31,826] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:14:41,903] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:14:51,975] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:02,056] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:12,167] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:22,231] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:32,303] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:42,375] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:15:52,447] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:16:02,528] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:16:12,600] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053
> [2012-03-07 10:16:22,676] [DEBUG] [main-EventThread] 
> [org.apache.hadoop.hbase.master.AssignmentManager 553] Handling 
> transition=RS_ZK_REGION_FAILED_OPEN, server=158-1-130-11,20020,1331108408232, 
> region=c70e98bdca98a0657a56436741523053

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HBASE-6001) Upgrade slf4j to 1.6.1

[Jimmy Xiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang reassigned HBASE-6001:
--

Assignee: Jimmy Xiang

> Upgrade slf4j to 1.6.1
> --
>
> Key: HBASE-6001
> URL: https://issues.apache.org/jira/browse/HBASE-6001
> Project: HBase
>  Issue Type: Task
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Attachments: hbase-6001.patch
>
>
> We need to upgrade slf4j to 1.6.1 since other hadoop components use 1.6.1 now.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6001) Upgrade slf4j to 1.6.1

[Jimmy Xiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-6001:
---

Attachment: hbase-6001.patch

> Upgrade slf4j to 1.6.1
> --
>
> Key: HBASE-6001
> URL: https://issues.apache.org/jira/browse/HBASE-6001
> Project: HBase
>  Issue Type: Task
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Attachments: hbase-6001.patch
>
>
> We need to upgrade slf4j to 1.6.1 since other hadoop components use 1.6.1 now.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6002) Possible chance of resource leak in HlogSplitter

[Zhihong Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276229#comment-13276229
 ] 

Zhihong Yu commented on HBASE-6002:
---

{code}
+  try {
+wap.w.close();
+  } catch (IOException e) {
+LOG.debug("Exception while closing the writer :", e);
{code}
The above may produce considerable logs. Can boolean flag, closed, be added to 
WriterAndPath so that we don't try to close an already closed writer ?

> Possible chance of resource leak in HlogSplitter
> 
>
> Key: HBASE-6002
> URL: https://issues.apache.org/jira/browse/HBASE-6002
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 0.94.0
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Attachments: HBASE-6002.patch
>
>
> In HLogSplitter.splitLogFileToTemp-Reader(in) is not closed and in finally 
> block in loop while closing the writers(wap.w) if any exception comes other 
> writers won't close.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6001) Upgrade slf4j to 1.6.1

[Jimmy Xiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-6001:
---

Status: Patch Available  (was: Open)

Hadoop 1.0.2 uses 1.4.3 slf4j.  Others use 1.6.1.

> Upgrade slf4j to 1.6.1
> --
>
> Key: HBASE-6001
> URL: https://issues.apache.org/jira/browse/HBASE-6001
> Project: HBase
>  Issue Type: Task
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Attachments: hbase-6001.patch
>
>
> We need to upgrade slf4j to 1.6.1 since other hadoop components use 1.6.1 now.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5959) Add other load balancers

[Elliott Clark (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276254#comment-13276254
 ] 

Elliott Clark commented on HBASE-5959:
--

And I didn't move the getTopBlockLocations into a helper as this is already a 
big enough patch and I wanted to keep the impact to a minimum.  If this gets 
accepted i'll be filing tickets to add region load cost function (region load 
seems broken right now i'm looking into it), and to refactor 
getTopBlockLocations.

Assuming the highest cost or the lowest cost for things has no functional 
difference.  However when/if region load cost functions are added assuming the 
worst would not work, so I think it's better to start now with always assuming 
the best cost unless it can be proven otherwise.

Other than that yes I believe all of the issues were addressed by what is up on 
phabricator.

> Add other load balancers
> 
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
>  Issue Type: New Feature
>  Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, 
> HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, 
> HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, 
> HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch
>
>
> Now that balancers are pluggable we should give some options.b

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6005) Broken Links on Homepages

[Elliott Clark (JIRA)]

Elliott Clark created HBASE-6005:


 Summary: Broken Links on Homepages
 Key: HBASE-6005
 URL: https://issues.apache.org/jira/browse/HBASE-6005
 Project: HBase
  Issue Type: Task
  Components: documentation
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Minor


I ran w3c's link checker on the homepage and there a few broken links.

I'll start getting a patch to fix the links that were broken.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5959) Add other load balancers

[Zhihong Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276257#comment-13276257
 ] 

Zhihong Yu commented on HBASE-5959:
---

Allow me some time to comment on phabricator.

Meanwhile, do you have performance numbers for stochastic balancer on a real 
cluster ? e.g. how long it takes to balance N regions, etc.

> Add other load balancers
> 
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
>  Issue Type: New Feature
>  Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, 
> HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, 
> HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, 
> HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch
>
>
> Now that balancers are pluggable we should give some options.b

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5453:
-

Attachment: 5453v3.txt

Trying against hadoopqa.

Moves over the five listed files to pb:

.tableinfo, .regioninfo, reference, hbase.id and hbase.version

Will post on pb w/ a better commit note now.

> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> --
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Todd Lipcon
>Assignee: stack
> Attachments: 5453.txt, 5453v2.txt, 5453v3.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5998) Bulk assignment: regionserver optimization by using a temporary cache for table descriptors when receveing an open regions request

[Zhihong Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276265#comment-13276265
 ] 

Zhihong Yu commented on HBASE-5998:
---

Integrated to trunk.

Thanks for the patch, N.

Thanks for the review, Ramkrishna.

> Bulk assignment: regionserver optimization by using a temporary cache for 
> table descriptors when receveing an open regions request
> --
>
> Key: HBASE-5998
> URL: https://issues.apache.org/jira/browse/HBASE-5998
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 0.96.0
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5998.v2.patch, 5998.v3.patch
>
>
> During the assignment, on the regionserver, before creating the handlers we 
> load the table description. Even if there is a cache, we check the timestamps 
> for each region, while it's not necessary. The test below is just with one 
> node, with more nodes the benefit will improve. By limiting the time spent in 
> HRegion#openRegion we increase the parallelization during cluster startup, as 
> the master is using a pool of threads to call the RS.
> -- Without the fix
> 2012-05-14 11:40:52,501 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 1193 
> region(s) to localhost,11003,1336988444043
> 2012-05-14 11:41:09,947 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done for 
> localhost,11003,1336988444043
> -- With the fix
> 2012-05-14 11:34:40,444 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 1193 
> region(s) to localhost,11003,1336988444043
> 2012-05-14 11:34:40,929 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done for 
> localhost,11003,1336988065948

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276269#comment-13276269
 ] 

Hadoop QA commented on HBASE-5453:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527528/5453v3.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 24 new or modified tests.

-1 patch.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1879//console

This message is automatically generated.

> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> --
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Todd Lipcon
>Assignee: stack
> Attachments: 5453.txt, 5453v2.txt, 5453v3.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5959) Add other load balancers

[Elliott Clark (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276271#comment-13276271
 ] 

Elliott Clark commented on HBASE-5959:
--

hbase(main):004:0> status
13 servers, 0 dead, 3.6154 average load

hbase(main):005:0> balancer
true



0 row(s) in 3.3460 seconds

hbase(main):006:0> balancer
true



0 row(s) in 3.3530 seconds

hbase(main):007:0> balancer
true



0 row(s) in 3.3160 seconds

hbase(main):008:0> balancer
true



0 row(s) in 3.3850 seconds

hbase(main):009:0> balancer
true



0 row(s) in 3.3580 seconds

hbase(main):010:0> balancer
true



0 row(s) in 3.3190 seconds


The maxSteps config is nice as it places an upper bounds on the time a large 
cluster will run the rebalance function.  It also allows people to slowly 
balance their cluster meaning that fewer regions will be moved at a time as the 
cluster approaches the final solution.


> Add other load balancers
> 
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
>  Issue Type: New Feature
>  Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, 
> HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, 
> HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, 
> HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch
>
>
> Now that balancers are pluggable we should give some options.b

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5970) Improve the AssignmentManager#updateTimer and speed up handling opened event

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276272#comment-13276272
 ] 

Hadoop QA commented on HBASE-5970:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527518/5970v3.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 31 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1877//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1877//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1877//console

This message is automatically generated.

> Improve the AssignmentManager#updateTimer and speed up handling opened event
> 
>
> Key: HBASE-5970
> URL: https://issues.apache.org/jira/browse/HBASE-5970
> Project: HBase
>  Issue Type: Improvement
>  Components: master
>Reporter: chunhui shen
>Assignee: chunhui shen
> Attachments: 5970v3.patch, HBASE-5970.patch, HBASE-5970v2.patch, 
> HBASE-5970v3.patch
>
>
> We found handing opened event very slow in the environment with lots of 
> regions.
> The problem is the slow AssignmentManager#updateTimer.
> We do the test for bulk assigning 10w (i.e. 100k) regions, the whole process 
> of bulk assigning took 1 hours.
> 2012-05-06 20:31:49,201 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 10 
> region(s) round-robin across 5 server(s)
> 2012-05-06 21:26:32,103 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done
> I think we could do the improvement for the AssignmentManager#updateTimer: 
> Make a thread do this work.
> After the improvement, it took only 4.5mins
> 2012-05-07 11:03:36,581 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 10 
> region(s) across 5 server(s), retainAssignment=true 
> 2012-05-07 11:07:57,073 INFO 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276276#comment-13276276
 ] 

jirapos...@reviews.apache.org commented on HBASE-5453:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/
---

Review request for hbase.


Summary
---

A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
  New  class to hold clusterid in.
M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
  Make it so can do pb serialization.  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
  Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
  Make it so can do pb serialization.  Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
  ClusterId under ZK got renamed as ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
  Hide the Reference#Range enums.  Don't let them out of this class.
  Make it so can do pb serialization.
M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
  Use new methods on Reference for getting top and bottom.
M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
  ClusterId under zk has been renamed ZKClusterId.
  Use new ClusterId class too.
M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
  Use new clusterid class.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
  Move the RegionInfo convertion up into HRegionInfo instead of here.
  Added generic toDelimitedByteArray helper.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
  Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
  Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
  Use new utility writing out .regioninfo files.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
  Formatting.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
  Range in Reference is no longer public.
  Range in Reference is no longer public.
M 
b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
M 
b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
  ClusterId got renamed ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
  Use new serialization utlity in HTD.
M  b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
  Generic method for writing dot file content.
M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
  Reference#Range is not public any more
M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
  Deprecated getHRegionInfo, etc.
D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
  Rename
A b/src/main/protobuf/ClusterId.proto
  Added file for ClusterId only since its written to fs and to zk.
A b/src/main/protobuf/FS.proto
  Protos for fs files.
M b/src/main/protobuf/ZooKeeper.proto
  Moved ClusterId out to own proto file
M b/src/main/protobuf/hbase.proto
  Added TableSchema and ColumnFamilySchema


This addresses bug hbase-5453.
https://issues.apache.org/jira/browse/hbase-5453


Diffs
-

  src/main/java/org/apache/hadoop/hbase/ClusterId.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 5862f15 
  src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 8d83ff3 
  src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java af89e3e 
  src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af 
  src/main/java/org/apache/hadoop/hbase/io/Reference.java 6360059 
  src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 
9e4ada9 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 947ec5f 
  src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 5052878 
  src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java ccc964e 
  src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java dabfbab 
  src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 45cb6cf 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java 
PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 
058c006 
  src/main/java/org/apache/hadoop/hba

[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5453:
-

Status: Open  (was: Patch Available)

> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> --
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Todd Lipcon
>Assignee: stack
> Attachments: 5453.txt, 5453v2.txt, 5453v3.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6004) Adding more logging to help debugging MR job

[Jimmy Xiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-6004:
---

Attachment: hbase-6004.patch

> Adding more logging to help debugging MR job
> 
>
> Key: HBASE-6004
> URL: https://issues.apache.org/jira/browse/HBASE-6004
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.94.0, 0.96.0
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Fix For: 0.94.0, 0.96.0
>
> Attachments: hbase-6004.patch
>
>
> MR job sometime fails because scanner expired. In this case, it will be 
> helpful to know the last successful row, the ip of the region sever, and so 
> on.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5959) Add other load balancers

[Zhihong Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276280#comment-13276280
 ] 

Zhihong Yu commented on HBASE-5959:
---

I was looking for output in master log similar to the following:
{code}
LOG.info("Done. Calculated a load balance in " + (endTime-startTime) + "ms. 
" +
"Moving " + totalNumMoved + " regions off of " +
serversOverloaded + " overloaded servers onto " +
serversUnderloaded + " less loaded servers. regions to return:" +
regionsToReturn.size());
{code}
w.r.t. maxSteps, is it possible to deduce the maximum time allowable in one 
round of stochastic balancing ?
Currently we have (in HMaster.balance()):
{code}
int maximumBalanceTime = getBalancerCutoffTime();
long cutoffTime = System.currentTimeMillis() + maximumBalanceTime;
{code}
I am thinking of the following:
{code}
int maxPlanGenerationTime = maximumBalanceTime *
// the maximum percentage for planning region rebalance
getConfiguration().getInt("hbase.balancer.max.planning.percent", 30) / 100;
...
List partialPlans = 
this.balancer.balanceCluster(assignments,
this.assignmentManager.getTableInitRegions(), 
maxPlanGenerationTime);
{code}

> Add other load balancers
> 
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
>  Issue Type: New Feature
>  Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, 
> HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, 
> HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, 
> HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch
>
>
> Now that balancers are pluggable we should give some options.b

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6001) Upgrade slf4j to 1.6.1

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276282#comment-13276282
 ] 

Hadoop QA commented on HBASE-6001:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527521/hbase-6001.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 hadoop23.  The patch compiles against the hadoop 0.23.x profile.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

-1 findbugs.  The patch appears to introduce 31 new Findbugs (version 
1.3.9) warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

 -1 core tests.  The patch failed these unit tests:
   org.apache.hadoop.hbase.coprocessor.TestClassLoading
  
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1878//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1878//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/1878//console

This message is automatically generated.

> Upgrade slf4j to 1.6.1
> --
>
> Key: HBASE-6001
> URL: https://issues.apache.org/jira/browse/HBASE-6001
> Project: HBase
>  Issue Type: Task
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Attachments: hbase-6001.patch
>
>
> We need to upgrade slf4j to 1.6.1 since other hadoop components use 1.6.1 now.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (HBASE-5959) Add other load balancers

[Zhihong Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276280#comment-13276280
 ] 

Zhihong Yu edited comment on HBASE-5959 at 5/15/12 10:21 PM:
-

I was looking for output in master log similar to the following:
{code}
LOG.info("Done. Calculated a load balance in " + (endTime-startTime) + "ms. 
" +
"Moving " + totalNumMoved + " regions off of " +
serversOverloaded + " overloaded servers onto " +
serversUnderloaded + " less loaded servers. regions to return:" +
regionsToReturn.size());
{code}
w.r.t. maxSteps, is it possible to deduce the maximum time allowable in one 
round of stochastic balancing ?
Currently we have (in HMaster.balance()):
{code}
int maximumBalanceTime = getBalancerCutoffTime();
long cutoffTime = System.currentTimeMillis() + maximumBalanceTime;
{code}
I am thinking of the following:
{code}
int maxPlanGenerationTime = maximumBalanceTime *
// the maximum percentage for planning region rebalance
getConfiguration().getInt("hbase.balancer.max.planning.percent", 30) / 100;
...
List partialPlans = 
this.balancer.balanceCluster(assignments,
maxPlanGenerationTime);
{code}

  was (Author: zhi...@ebaysf.com):
I was looking for output in master log similar to the following:
{code}
LOG.info("Done. Calculated a load balance in " + (endTime-startTime) + "ms. 
" +
"Moving " + totalNumMoved + " regions off of " +
serversOverloaded + " overloaded servers onto " +
serversUnderloaded + " less loaded servers. regions to return:" +
regionsToReturn.size());
{code}
w.r.t. maxSteps, is it possible to deduce the maximum time allowable in one 
round of stochastic balancing ?
Currently we have (in HMaster.balance()):
{code}
int maximumBalanceTime = getBalancerCutoffTime();
long cutoffTime = System.currentTimeMillis() + maximumBalanceTime;
{code}
I am thinking of the following:
{code}
int maxPlanGenerationTime = maximumBalanceTime *
// the maximum percentage for planning region rebalance
getConfiguration().getInt("hbase.balancer.max.planning.percent", 30) / 100;
...
List partialPlans = 
this.balancer.balanceCluster(assignments,
this.assignmentManager.getTableInitRegions(), 
maxPlanGenerationTime);
{code}
  
> Add other load balancers
> 
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
>  Issue Type: New Feature
>  Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, 
> HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, 
> HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, 
> HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch
>
>
> Now that balancers are pluggable we should give some options.b

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6004) Adding more logging to help debugging MR job

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276287#comment-13276287
 ] 

jirapos...@reviews.apache.org commented on HBASE-6004:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5131/
---

Review request for hbase.


Summary
---

Added some logging for MR debugging in case scanner times out.  The logging is 
disabled by default.
It will be helpful to know how much time spent in the scanner and, how much in 
the mapper task.
In case scanner issue, it is helpful to know the region server id, last 
successful rows and so on.


This addresses bug HBASE-6004.
https://issues.apache.org/jira/browse/HBASE-6004


Diffs
-

  src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 46b1c56 
  src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReaderImpl.java 
42569fb 
  src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 
1c8a393 

Diff: https://reviews.apache.org/r/5131/diff


Testing
---


Thanks,

Jimmy



> Adding more logging to help debugging MR job
> 
>
> Key: HBASE-6004
> URL: https://issues.apache.org/jira/browse/HBASE-6004
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.94.0, 0.96.0
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Fix For: 0.94.0, 0.96.0
>
> Attachments: hbase-6004.patch
>
>
> MR job sometime fails because scanner expired. In this case, it will be 
> helpful to know the last successful row, the ip of the region sever, and so 
> on.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5959) Add other load balancers

[Elliott Clark (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276291#comment-13276291
 ] 

Elliott Clark commented on HBASE-5959:
--

The new balancer doesn't have that logging. So there's no timing in the logs.  
I can add that if there are people that need that info in that format.

No since we iterate through all servers there is no given amount of time.  It 
could be that an iteration could:

* hit a cold hfile location cache
* choose not to move regions.
* or do a move.

All of those things will take a different amount of time; so we can't really 
estimate the amount of time an iteration will take.  Hence we would have to go 
back to the timer every iteration through.  That would be very costly.

> Add other load balancers
> 
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
>  Issue Type: New Feature
>  Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, 
> HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, 
> HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, 
> HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch
>
>
> Now that balancers are pluggable we should give some options.b

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6005) Broken Links on Homepages

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-6005:
-

Attachment: HBASE-6005-0.patch

Should get most of them.

Where would the new release list be?

> Broken Links on Homepages
> -
>
> Key: HBASE-6005
> URL: https://issues.apache.org/jira/browse/HBASE-6005
> Project: HBase
>  Issue Type: Task
>  Components: documentation
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Minor
> Attachments: HBASE-6005-0.patch
>
>
> I ran w3c's link checker on the homepage and there a few broken links.
> I'll start getting a patch to fix the links that were broken.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5959) Add other load balancers

[Zhihong Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276298#comment-13276298
 ] 

Zhihong Yu commented on HBASE-5959:
---

bq. need that info in that format.
Please add such information. Of particular interest is the timing information 
of the first several balancing rounds when block location cache is cold.

bq. we can't really estimate the amount of time an iteration will take.
Agreed.

bq. Hence we would have to go back to the timer every iteration through. That 
would be very costly.
Since the maximum time allowable in one iteration is passed to balancer, why 
would using local timer to gauge elapsed time be expensive ?

> Add other load balancers
> 
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
>  Issue Type: New Feature
>  Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, 
> HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, 
> HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, 
> HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch
>
>
> Now that balancers are pluggable we should give some options.b

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6006) TestDelayedRpc is flaky

[Devaraj Das (JIRA)]

Devaraj Das created HBASE-6006:
--

 Summary: TestDelayedRpc is flaky
 Key: HBASE-6006
 URL: https://issues.apache.org/jira/browse/HBASE-6006
 Project: HBase
  Issue Type: Bug
  Components: test
Reporter: Devaraj Das


I noticed that TestDelayedRpc is flaky - testTooManyDelayedRpcs fails quite 
often in my setup. Seems to me that the test needs to be fixed for it to not 
depend on the (unpredictable) thread scheduling order in the RPC server/clients 
that the test creates.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5998) Bulk assignment: regionserver optimization by using a temporary cache for table descriptors when receveing an open regions request

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276300#comment-13276300
 ] 

Hudson commented on HBASE-5998:
---

Integrated in HBase-TRUNK #2887 (See 
[https://builds.apache.org/job/HBase-TRUNK/2887/])
HBASE-5998 Bulk assignment: regionserver optimization by using a temporary 
cache for table descriptors when receveing an open regions request (N Keywal) 
(Revision 1338916)

 Result = SUCCESS
tedyu : 
Files : 
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java


> Bulk assignment: regionserver optimization by using a temporary cache for 
> table descriptors when receveing an open regions request
> --
>
> Key: HBASE-5998
> URL: https://issues.apache.org/jira/browse/HBASE-5998
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 0.96.0
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5998.v2.patch, 5998.v3.patch
>
>
> During the assignment, on the regionserver, before creating the handlers we 
> load the table description. Even if there is a cache, we check the timestamps 
> for each region, while it's not necessary. The test below is just with one 
> node, with more nodes the benefit will improve. By limiting the time spent in 
> HRegion#openRegion we increase the parallelization during cluster startup, as 
> the master is using a pool of threads to call the RS.
> -- Without the fix
> 2012-05-14 11:40:52,501 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 1193 
> region(s) to localhost,11003,1336988444043
> 2012-05-14 11:41:09,947 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done for 
> localhost,11003,1336988444043
> -- With the fix
> 2012-05-14 11:34:40,444 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 1193 
> region(s) to localhost,11003,1336988444043
> 2012-05-14 11:34:40,929 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done for 
> localhost,11003,1336988065948

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6003) [refguide] performance.xml - fixed 2 links to Case Studies that were ???

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276301#comment-13276301
 ] 

Hudson commented on HBASE-6003:
---

Integrated in HBase-TRUNK #2887 (See 
[https://builds.apache.org/job/HBase-TRUNK/2887/])
hbase-6003.  fixed 2 links in Performance.xml that should have been going 
to case-studies chapter. (Revision 1338899)

 Result = SUCCESS

> [refguide] performance.xml - fixed 2 links to Case Studies that were ??? 
> -
>
> Key: HBASE-6003
> URL: https://issues.apache.org/jira/browse/HBASE-6003
> Project: HBase
>  Issue Type: Bug
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: performance_hbase_6003.xml.patch
>
>
> There were 2 links in the performance chapter that were ??? links that should 
> have been going to Case Studies.
> Corrected them so that they weren't dead links.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5959) Add other load balancers

[Elliott Clark (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276317#comment-13276317
 ] 

Elliott Clark commented on HBASE-5959:
--

We would have to check sys time every loop through and that would require 
nanotime to have any real accuracy; nannotime is very expensive.

There is currently no soft realtime requirement.  It doesn't seem like adding 
that would really be worth the extra complexity until there is a big need or 
lots of people asking for it.  Also Stack didn't want me to change any of the 
public interfaces in this issue.  If it becomes a big thing we can file an 
issue and add soft realtime.

> Add other load balancers
> 
>
> Key: HBASE-5959
> URL: https://issues.apache.org/jira/browse/HBASE-5959
> Project: HBase
>  Issue Type: New Feature
>  Components: master
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-5959-0.patch, HBASE-5959-1.patch, 
> HBASE-5959-2.patch, HBASE-5959-3.patch, HBASE-5959-6.patch, 
> HBASE-5959-7.patch, HBASE-5959.D3189.1.patch, HBASE-5959.D3189.2.patch, 
> HBASE-5959.D3189.3.patch, HBASE-5959.D3189.4.patch
>
>
> Now that balancers are pluggable we should give some options.b

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276320#comment-13276320
 ] 

jirapos...@reviews.apache.org commented on HBASE-5453:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/#review7916
---


Do we have a story for when we can remove the writable stuff?


src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java


Looks like we are stepping on each other's toes a bit; I defined my own 
protos for this and HTableDescriptor (see the review request for HBASE-5445).  
Your definitions are pretty close though; I'll convert mine over.



src/main/java/org/apache/hadoop/hbase/HRegionInfo.java


"not be what ou want" -> "not be what you want"



src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java


Did you give any thought to storing the actual data in a TableSchema rather 
than converting?  Perhaps it's not worth it because:
1) we still need to maintain the writable for now, so would require 
rewriting that part, which is a waste
2) might not perform well, because we have to call copyFrom a bunch of 
times.



src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java


Nice.  I wrote up similar things for HTableDescriptor/HColumnDescriptor; I 
can just get rid of them and use your convert functions.



src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


Are we sure this assumption is valid (that if the file lengths are the 
same, the files are the same format)?

We can't check if there is a pb-prefix at the start of file or something? 



src/main/java/org/apache/hadoop/hbase/util/FSUtils.java


Same here, is this assumption valid?



src/main/protobuf/FS.proto


extra whitespace



src/main/protobuf/hbase.proto


extra whitespace


- Gregory


On 2012-05-15 22:14:17, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5130/
bq.  ---
bq.  
bq.  (Updated 2012-05-15 22:14:17)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
bq.New  class to hold clusterid in.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
bq.Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
bq.Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq.ClusterId under ZK got renamed as ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
bq.Hide the Reference#Range enums.  Don't let them out of this class.
bq.Make it so can do pb serialization.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
bq.Use new methods on Reference for getting top and bottom.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq.ClusterId under zk has been renamed ZKClusterId.
bq.Use new ClusterId class too.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
bq.Use new clusterid class.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq.Move the RegionInfo convertion up into HRegionInfo instead of here.
bq.Added generic toDelimitedByteArray helper.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
bq.Use HRegionInfo convertions instead.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq.Use new utility writing out .regioninfo files.
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
bq.Formatting.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
bq.  M b/src/main/java/org/apache/hadoop/hbase/regionserver/St

[jira] [Commented] (HBASE-6004) Adding more logging to help debugging MR job

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276324#comment-13276324
 ] 

jirapos...@reviews.apache.org commented on HBASE-6004:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5131/#review7918
---



src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java


We control logging via log4j.properties files. Here also new configuration 
in another file to switch some additional logging on and off. Would it make 
more sense to create a logger class e.g. ScannerCallable.ActivityLog that logs 
at TRACE level and update log4j.properties with 
log4j.logger.org.apache.hadoop.hbase.client.ScannerCallable.ActivityLog = TRACE 
(default is INFO, i.e. disabled). Just a thought. We did something like this 
for security audit logging.



src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java


Maybe the client should just log this?



src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java


The new message seems more helpful than the old.



src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java


We have other configuration variables that start with 'hbase.mapreduce.*'


- Andrew


On 2012-05-15 22:24:40, Jimmy Xiang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5131/
bq.  ---
bq.  
bq.  (Updated 2012-05-15 22:24:40)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Added some logging for MR debugging in case scanner times out.  The 
logging is disabled by default.
bq.  It will be helpful to know how much time spent in the scanner and, how 
much in the mapper task.
bq.  In case scanner issue, it is helpful to know the region server id, last 
successful rows and so on.
bq.  
bq.  
bq.  This addresses bug HBASE-6004.
bq.  https://issues.apache.org/jira/browse/HBASE-6004
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 
46b1c56 
bq.src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReaderImpl.java 
42569fb 
bq.
src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 
1c8a393 
bq.  
bq.  Diff: https://reviews.apache.org/r/5131/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jimmy
bq.  
bq.



> Adding more logging to help debugging MR job
> 
>
> Key: HBASE-6004
> URL: https://issues.apache.org/jira/browse/HBASE-6004
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.94.0, 0.96.0
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Fix For: 0.94.0, 0.96.0
>
> Attachments: hbase-6004.patch
>
>
> MR job sometime fails because scanner expired. In this case, it will be 
> helpful to know the last successful row, the ip of the region sever, and so 
> on.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-5453:
-

Attachment: 5453v6.txt

Stashing v6 here for the moment.  Does not address yet Gregory's comments.   
Working on that now in a v7.

> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> --
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
>  Issue Type: Sub-task
>  Components: ipc, master, migration, regionserver
>Reporter: Todd Lipcon
>Assignee: stack
> Attachments: 5453.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference files, HFile meta fields, etc) to PB

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276330#comment-13276330
 ] 

jirapos...@reviews.apache.org commented on HBASE-5453:
--



bq.  On 2012-05-15 23:08:35, Gregory Chanan wrote:
bq.  > Do we have a story for when we can remove the writable stuff?

No.  Smile.  Being optimistic.  Have to start somewhere.  No harm marking this 
stuff deprecated in  meantime as heads-up.

I think we won't be able to migrate over these old classes.  I'm thinking we'll 
have to do new ones and deprecate the old given they inherit from Writable.  
It'll be a bit of a pain though given some of these classes come out in our 
public API.

But we have to start somewhere.


bq.  On 2012-05-15 23:08:35, Gregory Chanan wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java, line 1017
bq.  > 

bq.  >
bq.  > Looks like we are stepping on each other's toes a bit; I defined my 
own protos for this and HTableDescriptor (see the review request for 
HBASE-5445).  Your definitions are pretty close though; I'll convert mine over.

Well, Andrew beat us both to it over in his REST pb stuff.  We need to 
reconcile his w/ ours too


bq.  On 2012-05-15 23:08:35, Gregory Chanan wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 954
bq.  > 
bq.  >
bq.  > "not be what ou want" -> "not be what you want"

Fixed


bq.  On 2012-05-15 23:08:35, Gregory Chanan wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java, line 1240
bq.  > 

bq.  >
bq.  > Did you give any thought to storing the actual data in a TableSchema 
rather than converting?  Perhaps it's not worth it because:
bq.  > 1) we still need to maintain the writable for now, so would require 
rewriting that part, which is a waste
bq.  > 2) might not perform well, because we have to call copyFrom a bunch 
of times.

Yeah.  Thought about it but we want Writable still converting old 
serializations.  Mostly I reasoned that these classes are rarely, relatively 
serialized, and performance is not important when we do want to pass these 
classes serialized.  Yeah, it'd be a pain converting.


bq.  On 2012-05-15 23:08:35, Gregory Chanan wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 748
bq.  > 
bq.  >
bq.  > Are we sure this assumption is valid (that if the file lengths are 
the same, the files are the same format)?
bq.  > 
bq.  > We can't check if there is a pb-prefix at the start of file or 
something?

This is my way of figuring if I need to convert the file.  There is an off 
chance that file could be same length though serialized with Writables.  Good 
one.  Let me fix that.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/#review7916
---


On 2012-05-15 22:14:17, Michael Stack wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5130/
bq.  ---
bq.  
bq.  (Updated 2012-05-15 22:14:17)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
bq.New  class to hold clusterid in.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
bq.Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
bq.Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
bq.Make it so can do pb serialization.  Deprecated Writable serialization.
bq.  M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq.ClusterId under ZK got renamed as ZKClusterId
bq.  M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
bq.Hide the Reference#Range enums.  Don't let them out of this class.
bq.Make it so can do pb serialization.
bq.  M 
b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
bq.Use new methods on Reference for getting top and bottom.
bq.  M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq.ClusterId under zk has been renamed ZKClusterId.
bq.Use new ClusterId class too.
bq.  M b/src/main/java/org/

[jira] [Created] (HBASE-6007) Make getTableRegions return an empty list if the table does not exist

[Mikhail Bautin (JIRA)]

Mikhail Bautin created HBASE-6007:
-

 Summary: Make getTableRegions return an empty list if the table 
does not exist
 Key: HBASE-6007
 URL: https://issues.apache.org/jira/browse/HBASE-6007
 Project: HBase
  Issue Type: Bug
Reporter: Mikhail Bautin
Priority: Minor
 Attachments: D3243.1.patch

Making the getTableRegions Thrift API method handle TableNotFoundException and 
return an empty list in that case. Without this the behavior is dependent on 
whether an HTable object is present in the thread-local cache in case a table 
was deleted.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6007) Make getTableRegions return an empty list if the table does not exist

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HBASE-6007:
---

Attachment: D3243.1.patch

mbautin requested code review of "[jira] [HBASE-6007] [89-fb] Make 
getTableRegions return an empty list if the table does not exist".
Reviewers: Kannan, Liyin, JIRA

  Making the getTableRegions Thrift API method handle TableNotFoundException 
and return an empty list in that case. Without this the behavior is dependent 
on whether an HTable object is present in the thread-local cache in case a 
table was deleted.

  I will add a JIRA number when JIRA is back.


TEST PLAN
  Unit tests

REVISION DETAIL
  https://reviews.facebook.net/D3243

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java

To: JIRA


> Make getTableRegions return an empty list if the table does not exist
> -
>
> Key: HBASE-6007
> URL: https://issues.apache.org/jira/browse/HBASE-6007
> Project: HBase
>  Issue Type: Bug
>Reporter: Mikhail Bautin
>Priority: Minor
> Attachments: D3243.1.patch
>
>
> Making the getTableRegions Thrift API method handle TableNotFoundException 
> and return an empty list in that case. Without this the behavior is dependent 
> on whether an HTable object is present in the thread-local cache in case a 
> table was deleted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276333#comment-13276333
 ] 

Lars Hofhansl commented on HBASE-5954:
--

Finally some numbers for "PerformanceEvaluation --nomapred --rows=10 
--presplit=3 randomWrite 10"
Without WAL/HFile sync: ~18s
With WAL sync: ~34s

With WAL sync on, I see a constant ~70mb/s write load. Without WAL sync I see a 
few spikes of far higher IO load.

This is all on a single machine with HBase in fully distributed mode on top of 
a pseudo distributed HDFS.

Note that my patch does not yet do HFile sync'ing correctly.

> Allow proper fsync support for HBase
> 
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
>  Issue Type: Improvement
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk.txt, 
> hbase-hdfs-744.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6004) Adding more logging to help debugging MR job

[jirapos...@reviews.apache.org (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276341#comment-13276341
 ] 

jirapos...@reviews.apache.org commented on HBASE-6004:
--



bq.  On 2012-05-15 23:11:04, Andrew Purtell wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java, line 
164
bq.  > 
bq.  >
bq.  > Maybe the client should just log this?

Client sometimes just gets a RetriesExhaustedException.


bq.  On 2012-05-15 23:11:04, Andrew Purtell wrote:
bq.  > 
src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java, 
line 51
bq.  > 
bq.  >
bq.  > We have other configuration variables that start with 
'hbase.mapreduce.*'

Will change to mapreduce.


bq.  On 2012-05-15 23:11:04, Andrew Purtell wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java, line 
54
bq.  > 
bq.  >
bq.  > We control logging via log4j.properties files. Here also new 
configuration in another file to switch some additional logging on and off. 
Would it make more sense to create a logger class e.g. 
ScannerCallable.ActivityLog that logs at TRACE level and update 
log4j.properties with 
log4j.logger.org.apache.hadoop.hbase.client.ScannerCallable.ActivityLog = TRACE 
(default is INFO, i.e. disabled). Just a thought. We did something like this 
for security audit logging.

It is a good idea.  Because of other parameters introduced, it's better to put 
them in the same place for now.


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5131/#review7918
---


On 2012-05-15 22:24:40, Jimmy Xiang wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5131/
bq.  ---
bq.  
bq.  (Updated 2012-05-15 22:24:40)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  Added some logging for MR debugging in case scanner times out.  The 
logging is disabled by default.
bq.  It will be helpful to know how much time spent in the scanner and, how 
much in the mapper task.
bq.  In case scanner issue, it is helpful to know the region server id, last 
successful rows and so on.
bq.  
bq.  
bq.  This addresses bug HBASE-6004.
bq.  https://issues.apache.org/jira/browse/HBASE-6004
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 
46b1c56 
bq.src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReaderImpl.java 
42569fb 
bq.
src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 
1c8a393 
bq.  
bq.  Diff: https://reviews.apache.org/r/5131/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jimmy
bq.  
bq.



> Adding more logging to help debugging MR job
> 
>
> Key: HBASE-6004
> URL: https://issues.apache.org/jira/browse/HBASE-6004
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.94.0, 0.96.0
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Fix For: 0.94.0, 0.96.0
>
> Attachments: hbase-6004.patch
>
>
> MR job sometime fails because scanner expired. In this case, it will be 
> helpful to know the last successful row, the ip of the region sever, and so 
> on.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6008) copy_tables_desc.rb has in proper imports

[Jonathan Hsieh (JIRA)]

Jonathan Hsieh created HBASE-6008:
-

 Summary: copy_tables_desc.rb has in proper imports
 Key: HBASE-6008
 URL: https://issues.apache.org/jira/browse/HBASE-6008
 Project: HBase
  Issue Type: Bug
Reporter: Jonathan Hsieh


This script, which seems to be part of replication does not work out of the box 
against 0.90, 0.92, 0.94, or trunk because of this line:

{code}
import org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper
{code}

This file is not last appeared in the 0.89 branch, and actually is not needed 
at all for the script to function.  

The script is of dubious use -- we may want to consider removing it instead of 
fixing it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6008) copy_tables_desc.rb has in proper imports

[Jonathan Hsieh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-6008:
--

Affects Version/s: 0.96.0
   0.90.0
   0.92.0
   0.94.0

> copy_tables_desc.rb has in proper imports
> -
>
> Key: HBASE-6008
> URL: https://issues.apache.org/jira/browse/HBASE-6008
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.0, 0.92.0, 0.94.0, 0.96.0
>Reporter: Jonathan Hsieh
>
> This script, which seems to be part of replication does not work out of the 
> box against 0.90, 0.92, 0.94, or trunk because of this line:
> {code}
> import org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper
> {code}
> This file is not last appeared in the 0.89 branch, and actually is not needed 
> at all for the script to function.  
> The script is of dubious use -- we may want to consider removing it instead 
> of fixing it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6008) copy_tables_desc.rb imports non-existant ZooKeeperWatcher

[Jonathan Hsieh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-6008:
--

Summary: copy_tables_desc.rb imports non-existant ZooKeeperWatcher  (was: 
copy_tables_desc.rb has in proper imports)

> copy_tables_desc.rb imports non-existant ZooKeeperWatcher
> -
>
> Key: HBASE-6008
> URL: https://issues.apache.org/jira/browse/HBASE-6008
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.90.0, 0.92.0, 0.94.0, 0.96.0
>Reporter: Jonathan Hsieh
>
> This script, which seems to be part of replication does not work out of the 
> box against 0.90, 0.92, 0.94, or trunk because of this line:
> {code}
> import org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper
> {code}
> This file is not last appeared in the 0.89 branch, and actually is not needed 
> at all for the script to function.  
> The script is of dubious use -- we may want to consider removing it instead 
> of fixing it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6003) [refguide] performance.xml - fixed 2 links to Case Studies that were ???

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276345#comment-13276345
 ] 

Hudson commented on HBASE-6003:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #5 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/5/])
hbase-6003.  fixed 2 links in Performance.xml that should have been going 
to case-studies chapter. (Revision 1338899)

 Result = FAILURE

> [refguide] performance.xml - fixed 2 links to Case Studies that were ??? 
> -
>
> Key: HBASE-6003
> URL: https://issues.apache.org/jira/browse/HBASE-6003
> Project: HBase
>  Issue Type: Bug
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: performance_hbase_6003.xml.patch
>
>
> There were 2 links in the performance chapter that were ??? links that should 
> have been going to Case Studies.
> Corrected them so that they weren't dead links.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5998) Bulk assignment: regionserver optimization by using a temporary cache for table descriptors when receveing an open regions request

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276344#comment-13276344
 ] 

Hudson commented on HBASE-5998:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #5 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/5/])
HBASE-5998 Bulk assignment: regionserver optimization by using a temporary 
cache for table descriptors when receveing an open regions request (N Keywal) 
(Revision 1338916)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java


> Bulk assignment: regionserver optimization by using a temporary cache for 
> table descriptors when receveing an open regions request
> --
>
> Key: HBASE-5998
> URL: https://issues.apache.org/jira/browse/HBASE-5998
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 0.96.0
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: 5998.v2.patch, 5998.v3.patch
>
>
> During the assignment, on the regionserver, before creating the handlers we 
> load the table description. Even if there is a cache, we check the timestamps 
> for each region, while it's not necessary. The test below is just with one 
> node, with more nodes the benefit will improve. By limiting the time spent in 
> HRegion#openRegion we increase the parallelization during cluster startup, as 
> the master is using a pool of threads to call the RS.
> -- Without the fix
> 2012-05-14 11:40:52,501 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 1193 
> region(s) to localhost,11003,1336988444043
> 2012-05-14 11:41:09,947 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done for 
> localhost,11003,1336988444043
> -- With the fix
> 2012-05-14 11:34:40,444 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 1193 
> region(s) to localhost,11003,1336988444043
> 2012-05-14 11:34:40,929 DEBUG 
> org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done for 
> localhost,11003,1336988065948

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-3877) Determine Proper Defaults for Compaction ThreadPools

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-3877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276346#comment-13276346
 ] 

Hudson commented on HBASE-3877:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #5 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/5/])
[HBASE-5867] Improve Compaction Throttle Default

Summary:
We recently had a production issue where our compactions fell
behind because our compaction throttle was improperly tuned and
accidentally upgraded all compactions to the large pool. The default
from HBASE-3877 makes 1 bad assumption: the default number of flushed
files in a compaction. MinFilesToCompact should be taken into
consideration. As a default, it is less damaging for the large thread
to be slightly higher than it needs to be and only get timed-majors
versus having everything accidentally promoted.

Test Plan:  - mvn test

Reviewers: JIRA, Kannan, Liyin
Reviewed By: Kannan
CC: stack

Differential Revision: https://reviews.facebook.net/D2943 (Revision 1338809)

 Result = FAILURE
nspiegelberg : 
Files : 
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java


> Determine Proper Defaults for Compaction ThreadPools
> 
>
> Key: HBASE-3877
> URL: https://issues.apache.org/jira/browse/HBASE-3877
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 0.92.0
>Reporter: Nicolas Spiegelberg
>Assignee: Nicolas Spiegelberg
>Priority: Trivial
>  Labels: compaction
>
> With the introduction of HBASE-1476, we now have multithreaded compactions + 
> 2 different ThreadPools for large and small compactions.  However, this is 
> disabled by default until we can determine a proper default throttle point.  
> Opening this JIRA to log all discussion on how to select a good default for 
> this case.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5867) Improve Compaction Throttle Default

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276347#comment-13276347
 ] 

Hudson commented on HBASE-5867:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #5 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/5/])
[HBASE-5867] Improve Compaction Throttle Default

Summary:
We recently had a production issue where our compactions fell
behind because our compaction throttle was improperly tuned and
accidentally upgraded all compactions to the large pool. The default
from HBASE-3877 makes 1 bad assumption: the default number of flushed
files in a compaction. MinFilesToCompact should be taken into
consideration. As a default, it is less damaging for the large thread
to be slightly higher than it needs to be and only get timed-majors
versus having everything accidentally promoted.

Test Plan:  - mvn test

Reviewers: JIRA, Kannan, Liyin
Reviewed By: Kannan
CC: stack

Differential Revision: https://reviews.facebook.net/D2943 (Revision 1338809)

 Result = FAILURE
nspiegelberg : 
Files : 
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java


> Improve Compaction Throttle Default
> ---
>
> Key: HBASE-5867
> URL: https://issues.apache.org/jira/browse/HBASE-5867
> Project: HBase
>  Issue Type: Improvement
>Reporter: Nicolas Spiegelberg
>Assignee: Nicolas Spiegelberg
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: D2943.1.patch, HBASE-5867-trunk.patch
>
>
> We recently had a production issue where our compactions fell behind because 
> our compaction throttle was improperly tuned and accidentally upgraded all 
> compactions to the large pool.  The default from HBASE-3877 makes 1 bad 
> assumption: the default number of flushed files in a compaction.  Currently 
> the algorithm is:
> throttleSize ~= flushSize * 2
> This assumes that the basic compaction utilizes 3 files and that all 3 files 
> are compressed.  In this case, "hbase.hstore.compaction.min" == 6 && the 
> values were not very compressible.  Both conditions should be taken into 
> consideration.  As a default, it is less damaging for the large thread to be 
> slightly higher than it needs to be versus having everything accidentally 
> promoted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276348#comment-13276348
 ] 

Lars Hofhansl commented on HBASE-5954:
--

Ok... Now with HFile sync.
With HFile sync: ~20s
Both WAL and HFile sync: ~35s

With HFile sync enabled, I seek occasional IO spikes on top of the constant WAL 
IO.

> Allow proper fsync support for HBase
> 
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
>  Issue Type: Improvement
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk.txt, 
> hbase-hdfs-744.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5954) Allow proper fsync support for HBase

[Lars Hofhansl (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-5954:
-

Attachment: 5954-trunk-hdfs-trunk-v3.txt

Patch that works correctly for WAL and HFile sync. (This is the one I used for 
above performance tests)

> Allow proper fsync support for HBase
> 
>
> Key: HBASE-5954
> URL: https://issues.apache.org/jira/browse/HBASE-5954
> Project: HBase
>  Issue Type: Improvement
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Attachments: 5954-trunk-hdfs-trunk-v2.txt, 
> 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5987) HFileBlockIndex improvement

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276352#comment-13276352
 ] 

Phabricator commented on HBASE-5987:


mbautin has commented on the revision "[jira][89-fb] [HBASE-5987] 
HFileBlockIndex improvement".

  Mostly discussed offline with Liyin. Comments are inline.

INLINE COMMENTS
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockInfo.java:26 I would 
suggest that we reflect the fact that the block is being scanned in the class 
name. Perhaps BlockWithScanInfo.
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java:224 Add a 
space before -
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java:435 This 
description is misleading. i is not really "the ith indexed key".

  I would say "the position of the index key to retrieve, starting at 0".
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java:480 
Discussed with Liyin offline. "< 1" seems fine, but there should definitely be 
a comment describing what happens in case compared == 0 (when the key being 
searched is the same as the first key of the next block). In that case we are 
relying on loadBlockAndSeekToKey positioning the scanner just before the key we 
are interested in, and on StoreFileScanner calling HFileScanner.next() to bring 
us to the first key we are interested in, potentially in the next block.

  Also, in case nextIndexedKey == NO_NEXT_INDEXED_KEY, we should do the same 
thing as if compared < 1 (also discussed with Liyin offline). Therefore, the 
overall condition should be along the lines of:

if (this.nextIndexedKey != null &&
(this.nextIndexedKey == NO_NEXT_INDEXED_KEY ||
 reader.getComparator().compare(key, offset, length,
 nextIndexedKey, 0, nextIndexedKey.length) < 1)) {
  ...
}

  src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java:205 You probably 
don't want to enhance the deprecated HBaseTestCase class. Instead, try to add 
new functionality to HBaseTestingUtility.
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java:44 
@tedyu: we don't categorize our tests in 89-fb. This diff will be ported to 
trunk separately.
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java:45 
Please don't write new tests inheriting from HBaseTestCase. Use 
HBaseTestingUtility and HTestConst instead.
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java:82 
schemMetricSnapShot -> schemaMetricSnapshot
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java:91 
Shouldn't this be the following?

while (s.next(results) || !results.isEmpty()) {
  results.clear();
}
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java:103 
Why are we assigning the return value of verifyDataAndIndexBlockRead to this 
variable? It is not used anywhere.
  src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java:107 
Are we using the return value anywhere?

REVISION DETAIL
  https://reviews.facebook.net/D3237

To: Kannan, mbautin, Liyin
Cc: JIRA, todd, tedyu


> HFileBlockIndex improvement
> ---
>
> Key: HBASE-5987
> URL: https://issues.apache.org/jira/browse/HBASE-5987
> Project: HBase
>  Issue Type: Improvement
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Attachments: D3237.1.patch, 
> screen_shot_of_sequential_scan_profiling.png
>
>
> Recently we find out a performance problem that it is quite slow when 
> multiple requests are reading the same block of data or index. 
> From the profiling, one of the causes is the IdLock contention which has been 
> addressed in HBASE-5898. 
> Another issue is that the HFileScanner will keep asking the HFileBlockIndex 
> about the data block location for each target key value during the scan 
> process(reSeekTo), even though the target key value has already been in the 
> current data block. This issue will cause certain index block very HOT, 
> especially when it is a sequential scan.
> To solve this issue, we propose the following solutions:
> First, we propose to lookahead for one more block index so that the 
> HFileScanner would know the start key value of next data block. So if the 
> target key value for the scan(reSeekTo) is "smaller" than that start kv of 
> next data block, it means the target key value has a very high possibility in 
> the current data block (if not in current data block, then the start kv of 
> next data block should be returned. +Indexing on the start key has some 
> defects here+) and it shall NOT query the HFileBlockIndex in this case. On 
> the contrary, if the target key value is "bigger", then it shall query the 
> HFileBlockIndex. This improvement shall help to reduce the hotness of 
> HFileBlockIndex and av

[jira] [Commented] (HBASE-5987) HFileBlockIndex improvement

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276355#comment-13276355
 ] 

Phabricator commented on HBASE-5987:


Liyin has commented on the revision "[jira][89-fb] [HBASE-5987] HFileBlockIndex 
improvement".

INLINE COMMENTS
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java:480 I 
missed this NO_NEXT_INDEXED_KEY ! Thanks Mikhail for this insightful comment !

REVISION DETAIL
  https://reviews.facebook.net/D3237

To: Kannan, mbautin, Liyin
Cc: JIRA, todd, tedyu


> HFileBlockIndex improvement
> ---
>
> Key: HBASE-5987
> URL: https://issues.apache.org/jira/browse/HBASE-5987
> Project: HBase
>  Issue Type: Improvement
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Attachments: D3237.1.patch, 
> screen_shot_of_sequential_scan_profiling.png
>
>
> Recently we find out a performance problem that it is quite slow when 
> multiple requests are reading the same block of data or index. 
> From the profiling, one of the causes is the IdLock contention which has been 
> addressed in HBASE-5898. 
> Another issue is that the HFileScanner will keep asking the HFileBlockIndex 
> about the data block location for each target key value during the scan 
> process(reSeekTo), even though the target key value has already been in the 
> current data block. This issue will cause certain index block very HOT, 
> especially when it is a sequential scan.
> To solve this issue, we propose the following solutions:
> First, we propose to lookahead for one more block index so that the 
> HFileScanner would know the start key value of next data block. So if the 
> target key value for the scan(reSeekTo) is "smaller" than that start kv of 
> next data block, it means the target key value has a very high possibility in 
> the current data block (if not in current data block, then the start kv of 
> next data block should be returned. +Indexing on the start key has some 
> defects here+) and it shall NOT query the HFileBlockIndex in this case. On 
> the contrary, if the target key value is "bigger", then it shall query the 
> HFileBlockIndex. This improvement shall help to reduce the hotness of 
> HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block 
> Cache lookup.
> Secondary, we propose to push this idea a little further that the 
> HFileBlockIndex shall index on the last key value of each data block instead 
> of indexing on the start key value. The motivation is to solve the HBASE-4443 
> issue (avoid seeking to "previous" block when key you are interested in is 
> the first one of a block) as well as +the defects mentioned above+.
> For example, if the target key value is "smaller" than the start key value of 
> the data block N. There is no way for sure the target key value is in the 
> data block N or N-1. So it has to seek from data block N-1. However, if the 
> block index is based on the last key value for each data block and the target 
> key value is beween the last key value of data block N-1 and data block N, 
> then the target key value is supposed be data block N for sure. 
> As long as HBase only supports the forward scan, the last key value makes 
> more sense to be indexed on than the start key value. 
> Thanks Kannan and Mikhail for the insightful discussions and suggestions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6009) Changes for HBASE-5209 are technically incompatible

[David S. Wang (JIRA)]

David S. Wang created HBASE-6009:


 Summary: Changes for HBASE-5209 are technically incompatible
 Key: HBASE-6009
 URL: https://issues.apache.org/jira/browse/HBASE-6009
 Project: HBase
  Issue Type: Bug
  Components: master
Affects Versions: 0.94.0, 0.92.1
Reporter: David S. Wang


The additions to add backup masters to ClusterStatus are technically 
incompatible between clients and servers.  Older clients will basically not 
read the extra bits that the newer server pushes for the backup masters, thus 
screwing up the serialization for the next blob in the pipe.

For the Writable, we can add a total size field for ClusterStatus at the 
beginning, or we can have start and end markers.  I can make a patch for either 
approach; interested in whatever folks have to suggest.  Would be good to get 
this in soon to limit the damage to 0.92.1 (don't know if we can get this in in 
time for 0.94.0).

Either change will make us forward-compatible starting with when the change 
goes in, but will not fix the backwards incompatibility, which we will have to 
mark with a release note as there have already been releases with this change.

Hopefully we can do this in a cleaner way when wire compat rolls around in 0.96.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-6010) Security audit logger configuration for log4j

[Andrew Purtell (JIRA)]

Andrew Purtell created HBASE-6010:
-

 Summary: Security audit logger configuration for log4j
 Key: HBASE-6010
 URL: https://issues.apache.org/jira/browse/HBASE-6010
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.92.2, 0.96.0, 0.94.1
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor


Set up a logger for security audit messages just as Hadoop core does.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-6010) Security audit logger configuration for log4j

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-6010:
--

Attachment: 6010.patch

> Security audit logger configuration for log4j
> -
>
> Key: HBASE-6010
> URL: https://issues.apache.org/jira/browse/HBASE-6010
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 0.92.2, 0.96.0, 0.94.1
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Attachments: 6010.patch
>
>
> Set up a logger for security audit messages just as Hadoop core does.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6000) Cleanup where we keep .proto files

[Todd Lipcon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-6000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276366#comment-13276366
 ] 

Todd Lipcon commented on HBASE-6000:


I don't think IDL belongs in "resources" because there's no reason for it to 
end up in the compiled jars. resources are for things that need to be available 
to the classloader at runtime.

> Cleanup where we keep .proto files
> --
>
> Key: HBASE-6000
> URL: https://issues.apache.org/jira/browse/HBASE-6000
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>
> I see Andrew for his pb work over in rest has .protos files under 
> src/main/resources.  We should unify where these files live.  The recently 
> added .protos place them under src/main/protobuf  Its confusing.
> The thift idl files are here under resources too.
> Seems like we should move src/main/protobuf under src/resources to be 
> consistent.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-5927) SSH and DisableTableHandler happening together does not clear the znode of the region and RIT map.

[rajeshbabu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

rajeshbabu updated HBASE-5927:
--

Attachment: HBASE-5927_trunk_2.patch

> SSH and DisableTableHandler happening together does not clear the znode of 
> the region and RIT map.
> --
>
> Key: HBASE-5927
> URL: https://issues.apache.org/jira/browse/HBASE-5927
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 0.92.1, 0.96.0, 0.94.1
>Reporter: Jieshan Bean
>Assignee: Jieshan Bean
> Fix For: 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-5927_94.patch, HBASE-5927_trunk.patch, 
> HBASE-5927_trunk_2.patch, TestCaseForReProduce.txt
>
>
> A possible exception: If the related regionserver was just killed(But HMaster 
> has not perceived that), then we will get a local exception "Connection reset 
> by peer". If this region belongs to a disabling table. what will happen?
> ServerShutdownHandler will remove this region from AM#regions. So this region 
> is still existing in RIT. TimeoutMonitor will take care of it after it got 
> timeout. Then invoke unassign again. Since this region has been removed from 
> AM#regions, it will return directly due to the below code:
> {code}
> synchronized (this.regions) {
>   // Check if this region is currently assigned
>   if (!regions.containsKey(region)) {
> LOG.debug("Attempted to unassign region " +
>   region.getRegionNameAsString() + " but it is not " +
>   "currently assigned anywhere");
> return;
>   }
> }
> {code}
> Then it leads to an end-less loop.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira