[jira] [Commented] (HBASE-15086) Fix findbugs complaint so yetus reports more green
[
https://issues.apache.org/jira/browse/HBASE-15086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106434#comment-15106434
]
Hadoop QA commented on HBASE-15086:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
31s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} branch-1 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 7s
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 6s
{color} | {color:green} branch-1 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 13s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 22s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 33s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 44s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 56s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 8s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 20s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m 31s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 42s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.7.1. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 108m 39s
{color} | {color:green} root in the patch passed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 6s {color}
| {color:red} root in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 20s
{color} | {color:red} Patch generated 24 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 233m 24s {color}
|
[jira] [Updated] (HBASE-15097) When the scan operation covered two regions,sometimes the final results have duplicated rows.
[
https://issues.apache.org/jira/browse/HBASE-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenrongwei updated HBASE-15097:
Attachment: HBASE-15097-v003.patch
Add an unit test case and HBASE-15125 should be merged first, else this patch
will cause a unit test case failure.
> When the scan operation covered two regions,sometimes the final results have
> duplicated rows.
> -
>
> Key: HBASE-15097
> URL: https://issues.apache.org/jira/browse/HBASE-15097
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: centos 6.5
> hbase 1.1.2
>Reporter: chenrongwei
>Assignee: chenrongwei
> Attachments: HBASE-15097-v001.patch, HBASE-15097-v002.patch,
> HBASE-15097-v003.patch, output.log, rowkey.txt, snapshot2016-01-13 pm
> 8.42.37.png
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> When the scan operation‘s start key and end key covered two regions,the first
> region returned the rows which were beyond of its' end key.So,this finally
> leads to duplicated rows in the results.
> To avoid this problem,we should add a judgment before setting the variable
> "stopRow" in the class of HRegion,like follow:
> if (Bytes.equals(scan.getStopRow(), HConstants.EMPTY_END_ROW) &&
> !scan.isGetScan()) {
> this.stopRow = null;
> } else {
> if (Bytes.compareTo(scan.getStopRow(),
> this.getRegionInfo().getEndKey()) >= 0) {
> this.stopRow = this.getRegionInfo().getEndKey();
> } else {
> this.stopRow = scan.getStopRow();
> }
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15125) HBaseFsck's adoptHdfsOrphan function create region with wrong end key boundary.
[
https://issues.apache.org/jira/browse/HBASE-15125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106455#comment-15106455
]
Hadoop QA commented on HBASE-15125:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 10s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 5m 17s
{color} | {color:red} Patch generated 6 new checkstyle issues in hbase-server
(total was 89, now 51). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 45s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 28s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 5s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 52s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 31s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 12s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 12m 3s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 13m 43s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 15m 23s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit
[jira] [Commented] (HBASE-15101) Leaked References to StoreFile.Reader after HBASE-13082
[ https://issues.apache.org/jira/browse/HBASE-15101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106464#comment-15106464 ] ramkrishna.s.vasudevan commented on HBASE-15101: Can you try coming up with a test case to see this problem is reproducible as an FT. Try to start 2 region servers and after some data insertion ensure there is a major compaction and count the number of files after the compacted discharger chore runs. Also in your experimental set up are you seeing that none of the compacted files are are getting finalized or few of them are getting finalized and few are getting missed out? If nothing is getting finalized then it would be great to check the status of those files in by adding some debug logs in HStore.closeAndArchiveStoreFiles() (the method name am not very sure but something like this was added which does the finalization part). > Leaked References to StoreFile.Reader after HBASE-13082 > --- > > Key: HBASE-15101 > URL: https://issues.apache.org/jira/browse/HBASE-15101 > Project: HBase > Issue Type: Bug > Components: HFile, io >Affects Versions: 2.0.0 >Reporter: deepankar >Assignee: deepankar > Attachments: HBASE-15101-v1.patch, HBASE-15101-v2.patch, > HBASE-15101-v3.patch, HBASE-15101.patch > > > We observed this production that after a region server dies there are huge > number of hfiles in that region for the region server running the version > with HBASE-13082, In the doc it is given that it is expected to happen, but > we found a one place where scanners are not being closed. If the scanners are > not closed their references are not decremented and that is leading to the > issue of huge number of store files not being finalized > All I was able to find is in the selectScannersFrom, where we discard some of > the scanners and we are not closing them. I am attaching a patch for that. > Also to avoid these issues should the files that are done be logged and > finalized (moved to archive) as a part of region close operation. This will > solve any leaks that can happen and does not cause any dire consequences? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14533) Thrift client gets "AsyncProcess: Failed to get region location .... closed"
[
https://issues.apache.org/jira/browse/HBASE-14533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106576#comment-15106576
]
Will Hardman commented on HBASE-14533:
--
Hi all,
(My first JIRA post)
I can confirm that I have an application which experiences exactly the same
error and stack trace as Pankaj's, above.
Application is a Python client connecting to HBase on HDP 2.3 using the Thrift
bindings from the Happybase library. Error is thrown after 10 - 15 mins of
near-continuous mini-batch insertions into HBase. The error occurs regardless
of whether the Thrift connection is closed and reopened in between mini-batches.
I am happy to provide more information if it helps, since this is a repeatable
error.
With regards,
Will
> Thrift client gets "AsyncProcess: Failed to get region location closed"
>
>
> Key: HBASE-14533
> URL: https://issues.apache.org/jira/browse/HBASE-14533
> Project: HBase
> Issue Type: Bug
> Components: REST, Thrift
>Affects Versions: 1.0.0
>Reporter: stack
>Assignee: stack
> Attachments: 14533.test.patch, 14533v2.branch-1.patch, test.patch
>
>
> An internal python client has been getting below stack trace since
> HBASE-134347
> {code}
> 2015-09-30 11:27:31,670 runnerERROR : scheduler
> executor error
> 2015-09-30 11:27:31,674 runnerERROR : Traceback (most
> recent call last):
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsRtiFetcher-0.1-py2.6.egg/cops_rti/fetcher/runner.py",
> line 82, in run
> fetch_list = self.__scheduler_executor.run()
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsRtiFetcher-0.1-py2.6.egg/cops_rti/fetcher/scheduler.py",
> line 35, in run
> with self.__fetch_db_dao.get_scanner() as scanner:
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsHbaseCommon-f796bf2929be11c26536c3e8f3e9c0b0ecb382b3-py2.6.egg/cops/hbase/common/hbase_dao.py",
> line 57, in get_scanner
> caching=caching, field_filter_list=field_filter_list)
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsHbaseCommon-f796bf2929be11c26536c3e8f3e9c0b0ecb382b3-py2.6.egg/cops/hbase/common/hbase_client_template.py",
> line 104, in get_entity_scanner
> self.__fix_cfs(self.__filter_columns(field_filter_list)), caching)
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsHbaseCommon-f796bf2929be11c26536c3e8f3e9c0b0ecb382b3-py2.6.egg/cops/hbase/common/hbase_entity_scanner.py",
> line 81, in open
> self.__scanner_id = client.scannerOpenWithScan(table_name, scan)
> File
> "/opt/cops/cops-related-ticket-info-fetcher/.crepo/cops-hbase-common/ext-py/hbase/Hbase.py",
> line 1494, in scannerOpenWithScan
> return self.recv_scannerOpenWithScan()
> File
> "/opt/cops/cops-related-ticket-info-fetcher/.crepo/cops-hbase-common/ext-py/hbase/Hbase.py",
> line 1518, in recv_scannerOpenWithScan
> raise result.io
> IOError:
> IOError(message="org.apache.hadoop.hbase.client.RetriesExhaustedException:
> Can't get the location\n\tat
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:308)\n\tat
>
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:149)\n\tat
>
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:57)\n\tat
>
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:293)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:268)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:140)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:135)\n\tat
> org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888)\n\tat
> org.apache.hadoop.hbase.thrift.ThriftServerRunner$HBaseHandler.scannerOpenWithScan(ThriftServerRunner.java:1446)\n\tat
> sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)\n\tat
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
> java.lang.reflect.Method.invoke(Method.java:606)\n\tat
> org.apache.hadoop.hbase.thrift.HbaseHandlerMetricsProxy.invoke(HbaseHandlerMetricsProxy.java:67)\n\tat
> com.sun.proxy.$Proxy14.scannerOpenWithScan(Unknown Source)\n\tat
>
[jira] [Commented] (HBASE-14533) Thrift client gets "AsyncProcess: Failed to get region location .... closed"
[
https://issues.apache.org/jira/browse/HBASE-14533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106580#comment-15106580
]
Will Hardman commented on HBASE-14533:
--
Also worth noting that in my case the work-around suggested by stack above (to
let connections be cached "forever") does not work in this case. The only
work-around I have is to restart the Thrift server via an SSH command as part
of the exception-handling code...)
> Thrift client gets "AsyncProcess: Failed to get region location closed"
>
>
> Key: HBASE-14533
> URL: https://issues.apache.org/jira/browse/HBASE-14533
> Project: HBase
> Issue Type: Bug
> Components: REST, Thrift
>Affects Versions: 1.0.0
>Reporter: stack
>Assignee: stack
> Attachments: 14533.test.patch, 14533v2.branch-1.patch, test.patch
>
>
> An internal python client has been getting below stack trace since
> HBASE-134347
> {code}
> 2015-09-30 11:27:31,670 runnerERROR : scheduler
> executor error
> 2015-09-30 11:27:31,674 runnerERROR : Traceback (most
> recent call last):
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsRtiFetcher-0.1-py2.6.egg/cops_rti/fetcher/runner.py",
> line 82, in run
> fetch_list = self.__scheduler_executor.run()
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsRtiFetcher-0.1-py2.6.egg/cops_rti/fetcher/scheduler.py",
> line 35, in run
> with self.__fetch_db_dao.get_scanner() as scanner:
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsHbaseCommon-f796bf2929be11c26536c3e8f3e9c0b0ecb382b3-py2.6.egg/cops/hbase/common/hbase_dao.py",
> line 57, in get_scanner
> caching=caching, field_filter_list=field_filter_list)
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsHbaseCommon-f796bf2929be11c26536c3e8f3e9c0b0ecb382b3-py2.6.egg/cops/hbase/common/hbase_client_template.py",
> line 104, in get_entity_scanner
> self.__fix_cfs(self.__filter_columns(field_filter_list)), caching)
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsHbaseCommon-f796bf2929be11c26536c3e8f3e9c0b0ecb382b3-py2.6.egg/cops/hbase/common/hbase_entity_scanner.py",
> line 81, in open
> self.__scanner_id = client.scannerOpenWithScan(table_name, scan)
> File
> "/opt/cops/cops-related-ticket-info-fetcher/.crepo/cops-hbase-common/ext-py/hbase/Hbase.py",
> line 1494, in scannerOpenWithScan
> return self.recv_scannerOpenWithScan()
> File
> "/opt/cops/cops-related-ticket-info-fetcher/.crepo/cops-hbase-common/ext-py/hbase/Hbase.py",
> line 1518, in recv_scannerOpenWithScan
> raise result.io
> IOError:
> IOError(message="org.apache.hadoop.hbase.client.RetriesExhaustedException:
> Can't get the location\n\tat
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:308)\n\tat
>
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:149)\n\tat
>
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:57)\n\tat
>
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:293)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:268)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:140)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:135)\n\tat
> org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888)\n\tat
> org.apache.hadoop.hbase.thrift.ThriftServerRunner$HBaseHandler.scannerOpenWithScan(ThriftServerRunner.java:1446)\n\tat
> sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)\n\tat
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
> java.lang.reflect.Method.invoke(Method.java:606)\n\tat
> org.apache.hadoop.hbase.thrift.HbaseHandlerMetricsProxy.invoke(HbaseHandlerMetricsProxy.java:67)\n\tat
> com.sun.proxy.$Proxy14.scannerOpenWithScan(Unknown Source)\n\tat
> org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$scannerOpenWithScan.getResult(Hbase.java:4609)\n\tat
>
> org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$scannerOpenWithScan.getResult(Hbase.java:4593)\n\tat
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)\n\tat
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)\n\tat
>
[jira] [Commented] (HBASE-15091) Forward-port to 1.2 HBASE-15031 "Fix merge of MVCC and SequenceID performance regression in branch-1.0"
[
https://issues.apache.org/jira/browse/HBASE-15091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106583#comment-15106583
]
Hadoop QA commented on HBASE-15091:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
29s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 21s
{color} | {color:green} branch-1.2 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s
{color} | {color:green} branch-1.2 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
46s {color} | {color:green} branch-1.2 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 8s
{color} | {color:red} hbase-client in branch-1.2 has 7 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 54s
{color} | {color:red} hbase-common in branch-1.2 has 1 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 12s
{color} | {color:red} hbase-server in branch-1.2 has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 16s
{color} | {color:green} branch-1.2 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s
{color} | {color:green} branch-1.2 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 24s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 21s
{color} | {color:red} Patch generated 1 new checkstyle issues in hbase-server
(total was 110, now 109). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 47s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 5s
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 25s
{color} | {color:green} hbase-common in the patch passed with JDK v1.8.0.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 115m 41s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 44s
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 11s
{color} | {color:green} hbase-common in the patch passed with JDK v1.7.0_79.
{color} |
| {color:red}-1{color} |
[jira] [Commented] (HBASE-15126) HBaseFsck's checkRegionBoundaries function set the 'storesFirstKey' was incorrect.
[
https://issues.apache.org/jira/browse/HBASE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106415#comment-15106415
]
Hadoop QA commented on HBASE-15126:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 51s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 1s
{color} | {color:red} Patch generated 6 new checkstyle issues in hbase-server
(total was 89, now 51). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
21m 49s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 104m 38s
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 107m 6s
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 255m 47s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12782987/HBASE-15126-v003.patch
|
| JIRA Issue | HBASE-15126 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / eb17f74 |
|
[jira] [Commented] (HBASE-15101) Leaked References to StoreFile.Reader after HBASE-13082
[ https://issues.apache.org/jira/browse/HBASE-15101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106413#comment-15106413 ] deepankar commented on HBASE-15101: --- I thought before HBASE-13082, when a compaction starts and before it completes the files are present in .tmp directory (of the region folder) and finalized once it completes giving a very small window (after moving in the files from .tmp and moving out files from RegionServer) where there could be that all files are present. This is not the case after HBASE-13082 because both the set of files are present in the folder for a longer period of time and if there is any leak in the reference counting then all the files co exist and it can lead to a region size explosion . This is what exactly happened with us, without this patch we were running one regionserver with HBASE-13082 and almost all the regions on that server had all the files from the time of begining of that regionserver and movement of region to that server (movement rarely happens). The worst is we force major compact regions daily and that lead to the region data getting repeated over 7 times and In panic when we shutdown (gracefully) this server it lead to other regionservers that hosted these regions keep on compacting the whole next day (as each of them contained 5-7x the data of normal region). So then when applied this patch and hosted only two regions on this experimental regionserver for 2 days, and the samething repeated and when again we shutdown (again gracefully) the regionserver all the files did remain in the directory and it did lead to longer compaction next time. If we can come up with patch after leak may I could take a stab testing again, I will also go through the close() to see if I am missing any thing. Thanks > Leaked References to StoreFile.Reader after HBASE-13082 > --- > > Key: HBASE-15101 > URL: https://issues.apache.org/jira/browse/HBASE-15101 > Project: HBase > Issue Type: Bug > Components: HFile, io >Affects Versions: 2.0.0 >Reporter: deepankar >Assignee: deepankar > Attachments: HBASE-15101-v1.patch, HBASE-15101-v2.patch, > HBASE-15101-v3.patch, HBASE-15101.patch > > > We observed this production that after a region server dies there are huge > number of hfiles in that region for the region server running the version > with HBASE-13082, In the doc it is given that it is expected to happen, but > we found a one place where scanners are not being closed. If the scanners are > not closed their references are not decremented and that is leading to the > issue of huge number of store files not being finalized > All I was able to find is in the selectScannersFrom, where we discard some of > the scanners and we are not closing them. I am attaching a patch for that. > Also to avoid these issues should the files that are done be logged and > finalized (moved to archive) as a part of region close operation. This will > solve any leaks that can happen and does not cause any dire consequences? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14533) Thrift client gets "AsyncProcess: Failed to get region location .... closed"
[
https://issues.apache.org/jira/browse/HBASE-14533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106617#comment-15106617
]
Pankaj Kumar commented on HBASE-14533:
--
My issue got resolved after merging HBASE-14196 patch.
> Thrift client gets "AsyncProcess: Failed to get region location closed"
>
>
> Key: HBASE-14533
> URL: https://issues.apache.org/jira/browse/HBASE-14533
> Project: HBase
> Issue Type: Bug
> Components: REST, Thrift
>Affects Versions: 1.0.0
>Reporter: stack
>Assignee: stack
> Attachments: 14533.test.patch, 14533v2.branch-1.patch, test.patch
>
>
> An internal python client has been getting below stack trace since
> HBASE-134347
> {code}
> 2015-09-30 11:27:31,670 runnerERROR : scheduler
> executor error
> 2015-09-30 11:27:31,674 runnerERROR : Traceback (most
> recent call last):
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsRtiFetcher-0.1-py2.6.egg/cops_rti/fetcher/runner.py",
> line 82, in run
> fetch_list = self.__scheduler_executor.run()
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsRtiFetcher-0.1-py2.6.egg/cops_rti/fetcher/scheduler.py",
> line 35, in run
> with self.__fetch_db_dao.get_scanner() as scanner:
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsHbaseCommon-f796bf2929be11c26536c3e8f3e9c0b0ecb382b3-py2.6.egg/cops/hbase/common/hbase_dao.py",
> line 57, in get_scanner
> caching=caching, field_filter_list=field_filter_list)
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsHbaseCommon-f796bf2929be11c26536c3e8f3e9c0b0ecb382b3-py2.6.egg/cops/hbase/common/hbase_client_template.py",
> line 104, in get_entity_scanner
> self.__fix_cfs(self.__filter_columns(field_filter_list)), caching)
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsHbaseCommon-f796bf2929be11c26536c3e8f3e9c0b0ecb382b3-py2.6.egg/cops/hbase/common/hbase_entity_scanner.py",
> line 81, in open
> self.__scanner_id = client.scannerOpenWithScan(table_name, scan)
> File
> "/opt/cops/cops-related-ticket-info-fetcher/.crepo/cops-hbase-common/ext-py/hbase/Hbase.py",
> line 1494, in scannerOpenWithScan
> return self.recv_scannerOpenWithScan()
> File
> "/opt/cops/cops-related-ticket-info-fetcher/.crepo/cops-hbase-common/ext-py/hbase/Hbase.py",
> line 1518, in recv_scannerOpenWithScan
> raise result.io
> IOError:
> IOError(message="org.apache.hadoop.hbase.client.RetriesExhaustedException:
> Can't get the location\n\tat
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:308)\n\tat
>
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:149)\n\tat
>
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:57)\n\tat
>
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:293)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:268)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:140)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:135)\n\tat
> org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888)\n\tat
> org.apache.hadoop.hbase.thrift.ThriftServerRunner$HBaseHandler.scannerOpenWithScan(ThriftServerRunner.java:1446)\n\tat
> sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)\n\tat
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
> java.lang.reflect.Method.invoke(Method.java:606)\n\tat
> org.apache.hadoop.hbase.thrift.HbaseHandlerMetricsProxy.invoke(HbaseHandlerMetricsProxy.java:67)\n\tat
> com.sun.proxy.$Proxy14.scannerOpenWithScan(Unknown Source)\n\tat
> org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$scannerOpenWithScan.getResult(Hbase.java:4609)\n\tat
>
> org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$scannerOpenWithScan.getResult(Hbase.java:4593)\n\tat
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)\n\tat
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)\n\tat
> org.apache.hadoop.hbase.thrift.ThriftServerRunner$3.process(ThriftServerRunner.java:502)\n\tat
>
>
[jira] [Commented] (HBASE-14533) Thrift client gets "AsyncProcess: Failed to get region location .... closed"
[
https://issues.apache.org/jira/browse/HBASE-14533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106628#comment-15106628
]
Will Hardman commented on HBASE-14533:
--
Great news!
Sadly I have no idea how to do that so I'm going to have to wait for an update
to HBase to be released.
Will
> Thrift client gets "AsyncProcess: Failed to get region location closed"
>
>
> Key: HBASE-14533
> URL: https://issues.apache.org/jira/browse/HBASE-14533
> Project: HBase
> Issue Type: Bug
> Components: REST, Thrift
>Affects Versions: 1.0.0
>Reporter: stack
>Assignee: stack
> Attachments: 14533.test.patch, 14533v2.branch-1.patch, test.patch
>
>
> An internal python client has been getting below stack trace since
> HBASE-134347
> {code}
> 2015-09-30 11:27:31,670 runnerERROR : scheduler
> executor error
> 2015-09-30 11:27:31,674 runnerERROR : Traceback (most
> recent call last):
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsRtiFetcher-0.1-py2.6.egg/cops_rti/fetcher/runner.py",
> line 82, in run
> fetch_list = self.__scheduler_executor.run()
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsRtiFetcher-0.1-py2.6.egg/cops_rti/fetcher/scheduler.py",
> line 35, in run
> with self.__fetch_db_dao.get_scanner() as scanner:
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsHbaseCommon-f796bf2929be11c26536c3e8f3e9c0b0ecb382b3-py2.6.egg/cops/hbase/common/hbase_dao.py",
> line 57, in get_scanner
> caching=caching, field_filter_list=field_filter_list)
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsHbaseCommon-f796bf2929be11c26536c3e8f3e9c0b0ecb382b3-py2.6.egg/cops/hbase/common/hbase_client_template.py",
> line 104, in get_entity_scanner
> self.__fix_cfs(self.__filter_columns(field_filter_list)), caching)
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsHbaseCommon-f796bf2929be11c26536c3e8f3e9c0b0ecb382b3-py2.6.egg/cops/hbase/common/hbase_entity_scanner.py",
> line 81, in open
> self.__scanner_id = client.scannerOpenWithScan(table_name, scan)
> File
> "/opt/cops/cops-related-ticket-info-fetcher/.crepo/cops-hbase-common/ext-py/hbase/Hbase.py",
> line 1494, in scannerOpenWithScan
> return self.recv_scannerOpenWithScan()
> File
> "/opt/cops/cops-related-ticket-info-fetcher/.crepo/cops-hbase-common/ext-py/hbase/Hbase.py",
> line 1518, in recv_scannerOpenWithScan
> raise result.io
> IOError:
> IOError(message="org.apache.hadoop.hbase.client.RetriesExhaustedException:
> Can't get the location\n\tat
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:308)\n\tat
>
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:149)\n\tat
>
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:57)\n\tat
>
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:293)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:268)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:140)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:135)\n\tat
> org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888)\n\tat
> org.apache.hadoop.hbase.thrift.ThriftServerRunner$HBaseHandler.scannerOpenWithScan(ThriftServerRunner.java:1446)\n\tat
> sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)\n\tat
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
> java.lang.reflect.Method.invoke(Method.java:606)\n\tat
> org.apache.hadoop.hbase.thrift.HbaseHandlerMetricsProxy.invoke(HbaseHandlerMetricsProxy.java:67)\n\tat
> com.sun.proxy.$Proxy14.scannerOpenWithScan(Unknown Source)\n\tat
> org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$scannerOpenWithScan.getResult(Hbase.java:4609)\n\tat
>
> org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$scannerOpenWithScan.getResult(Hbase.java:4593)\n\tat
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)\n\tat
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)\n\tat
> org.apache.hadoop.hbase.thrift.ThriftServerRunner$3.process(ThriftServerRunner.java:502)\n\tat
>
>
[jira] [Updated] (HBASE-15121) ConnectionImplementation#locateRegionInMeta() issue when master is restarted
[
https://issues.apache.org/jira/browse/HBASE-15121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samir Ahmic updated HBASE-15121:
Attachment: HBASE-15121-v0.patch
Here is patch.
> ConnectionImplementation#locateRegionInMeta() issue when master is restarted
>
>
> Key: HBASE-15121
> URL: https://issues.apache.org/jira/browse/HBASE-15121
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
> Fix For: 2.0.0
>
> Attachments: HBASE-15121-v0.patch
>
>
> I notice this issue while i was running
> IntegrationTestMTTR#testRestartMaster() test was failing on put operation.
> Here is sequence of events from logs leading to failed put operation:
> Master restart
> {code}
> INFO [pool-5-thread-1] util.Shell: Executing full command [/usr/bin/ssh
> hnode2 "sudo -u hbase ps aux | grep proc_master | grep -v grep | tr -s ' ' |
> cut -d ' ' -f2 | xargs kill -s SIGKILL"]
> {code}
> Client trying to locate region for row=70efdf2ec9b086079795c442636b55fb-17
> (this is additional logging inspecting metaKey which is used to search
> hbase:meta )
> {code}
> 2016-01-15 10:26:05,169 INFO [HBaseWriterThread_9]
> client.ConnectionImplementation: metaKey inspection:
> table=IntegrationTestMTTRLoadTestTool row=
> 70efdf2ec9b086079795c442636b55fb-17 metaKey=
> IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99
> {code}
> Client throwing TableNotFoundException (hbase:meta scan returned null)
> {code}
> 2016-01-15 10:32:58,154 INFO [HBaseWriterThread_5]
> client.ConnectionImplementation: regionInfo result is null:
> HBaseWriterThread_5 throwing TableNotFoundException logging details
> table=IntegrationTestMTTRLoadTestTool row=70efdf2ec9b086079795c442636b55fb-17
> metaKey=IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99
> 2016-01-15 10:32:58,154 ERROR [HBaseWriterThread_5] client.AsyncProcess:
> Failed to get region location
> org.apache.hadoop.hbase.TableNotFoundException:
> IntegrationTestMTTRLoadTestTool
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569)
> at
> org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146)
> at
> org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111)
> {code}
> And as result we have failed insert operation:
> {code}
> 2016-01-15 10:32:58,179 ERROR [HBaseWriterThread_5] util.MultiThreadedWriter:
> Failed to insert: 17 after 60046ms; region information: cached:
> region=IntegrationTestMTTRLoadTestTool,6660,1452849956427.05b437185a9437f178726a55a29a79b7.,
> hostname=hnode4,16020,1452776418437, seqNum=5; cache is up to date; errors:
> exception from null for 70efdf2ec9b086079795c442636b55fb-17
> org.apache.hadoop.hbase.TableNotFoundException:
> IntegrationTestMTTRLoadTestTool
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569)
> at
> org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146)
> at
> org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111)
> {code}
> leading to test failing:
> {code}
> Failed
[jira] [Updated] (HBASE-15121) ConnectionImplementation#locateRegionInMeta() issue when master is restarted
[
https://issues.apache.org/jira/browse/HBASE-15121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samir Ahmic updated HBASE-15121:
Status: Patch Available (was: Open)
> ConnectionImplementation#locateRegionInMeta() issue when master is restarted
>
>
> Key: HBASE-15121
> URL: https://issues.apache.org/jira/browse/HBASE-15121
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
> Fix For: 2.0.0
>
> Attachments: HBASE-15121-v0.patch
>
>
> I notice this issue while i was running
> IntegrationTestMTTR#testRestartMaster() test was failing on put operation.
> Here is sequence of events from logs leading to failed put operation:
> Master restart
> {code}
> INFO [pool-5-thread-1] util.Shell: Executing full command [/usr/bin/ssh
> hnode2 "sudo -u hbase ps aux | grep proc_master | grep -v grep | tr -s ' ' |
> cut -d ' ' -f2 | xargs kill -s SIGKILL"]
> {code}
> Client trying to locate region for row=70efdf2ec9b086079795c442636b55fb-17
> (this is additional logging inspecting metaKey which is used to search
> hbase:meta )
> {code}
> 2016-01-15 10:26:05,169 INFO [HBaseWriterThread_9]
> client.ConnectionImplementation: metaKey inspection:
> table=IntegrationTestMTTRLoadTestTool row=
> 70efdf2ec9b086079795c442636b55fb-17 metaKey=
> IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99
> {code}
> Client throwing TableNotFoundException (hbase:meta scan returned null)
> {code}
> 2016-01-15 10:32:58,154 INFO [HBaseWriterThread_5]
> client.ConnectionImplementation: regionInfo result is null:
> HBaseWriterThread_5 throwing TableNotFoundException logging details
> table=IntegrationTestMTTRLoadTestTool row=70efdf2ec9b086079795c442636b55fb-17
> metaKey=IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99
> 2016-01-15 10:32:58,154 ERROR [HBaseWriterThread_5] client.AsyncProcess:
> Failed to get region location
> org.apache.hadoop.hbase.TableNotFoundException:
> IntegrationTestMTTRLoadTestTool
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569)
> at
> org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146)
> at
> org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111)
> {code}
> And as result we have failed insert operation:
> {code}
> 2016-01-15 10:32:58,179 ERROR [HBaseWriterThread_5] util.MultiThreadedWriter:
> Failed to insert: 17 after 60046ms; region information: cached:
> region=IntegrationTestMTTRLoadTestTool,6660,1452849956427.05b437185a9437f178726a55a29a79b7.,
> hostname=hnode4,16020,1452776418437, seqNum=5; cache is up to date; errors:
> exception from null for 70efdf2ec9b086079795c442636b55fb-17
> org.apache.hadoop.hbase.TableNotFoundException:
> IntegrationTestMTTRLoadTestTool
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569)
> at
> org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146)
> at
> org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111)
> {code}
> leading to test failing:
> {code}
> Failed to write
[jira] [Commented] (HBASE-14533) Thrift client gets "AsyncProcess: Failed to get region location .... closed"
[
https://issues.apache.org/jira/browse/HBASE-14533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106642#comment-15106642
]
Pankaj Kumar commented on HBASE-14533:
--
Which HBase version are you using?
> Thrift client gets "AsyncProcess: Failed to get region location closed"
>
>
> Key: HBASE-14533
> URL: https://issues.apache.org/jira/browse/HBASE-14533
> Project: HBase
> Issue Type: Bug
> Components: REST, Thrift
>Affects Versions: 1.0.0
>Reporter: stack
>Assignee: stack
> Attachments: 14533.test.patch, 14533v2.branch-1.patch, test.patch
>
>
> An internal python client has been getting below stack trace since
> HBASE-134347
> {code}
> 2015-09-30 11:27:31,670 runnerERROR : scheduler
> executor error
> 2015-09-30 11:27:31,674 runnerERROR : Traceback (most
> recent call last):
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsRtiFetcher-0.1-py2.6.egg/cops_rti/fetcher/runner.py",
> line 82, in run
> fetch_list = self.__scheduler_executor.run()
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsRtiFetcher-0.1-py2.6.egg/cops_rti/fetcher/scheduler.py",
> line 35, in run
> with self.__fetch_db_dao.get_scanner() as scanner:
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsHbaseCommon-f796bf2929be11c26536c3e8f3e9c0b0ecb382b3-py2.6.egg/cops/hbase/common/hbase_dao.py",
> line 57, in get_scanner
> caching=caching, field_filter_list=field_filter_list)
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsHbaseCommon-f796bf2929be11c26536c3e8f3e9c0b0ecb382b3-py2.6.egg/cops/hbase/common/hbase_client_template.py",
> line 104, in get_entity_scanner
> self.__fix_cfs(self.__filter_columns(field_filter_list)), caching)
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsHbaseCommon-f796bf2929be11c26536c3e8f3e9c0b0ecb382b3-py2.6.egg/cops/hbase/common/hbase_entity_scanner.py",
> line 81, in open
> self.__scanner_id = client.scannerOpenWithScan(table_name, scan)
> File
> "/opt/cops/cops-related-ticket-info-fetcher/.crepo/cops-hbase-common/ext-py/hbase/Hbase.py",
> line 1494, in scannerOpenWithScan
> return self.recv_scannerOpenWithScan()
> File
> "/opt/cops/cops-related-ticket-info-fetcher/.crepo/cops-hbase-common/ext-py/hbase/Hbase.py",
> line 1518, in recv_scannerOpenWithScan
> raise result.io
> IOError:
> IOError(message="org.apache.hadoop.hbase.client.RetriesExhaustedException:
> Can't get the location\n\tat
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:308)\n\tat
>
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:149)\n\tat
>
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:57)\n\tat
>
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:293)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:268)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:140)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:135)\n\tat
> org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888)\n\tat
> org.apache.hadoop.hbase.thrift.ThriftServerRunner$HBaseHandler.scannerOpenWithScan(ThriftServerRunner.java:1446)\n\tat
> sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)\n\tat
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
> java.lang.reflect.Method.invoke(Method.java:606)\n\tat
> org.apache.hadoop.hbase.thrift.HbaseHandlerMetricsProxy.invoke(HbaseHandlerMetricsProxy.java:67)\n\tat
> com.sun.proxy.$Proxy14.scannerOpenWithScan(Unknown Source)\n\tat
> org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$scannerOpenWithScan.getResult(Hbase.java:4609)\n\tat
>
> org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$scannerOpenWithScan.getResult(Hbase.java:4593)\n\tat
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)\n\tat
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)\n\tat
> org.apache.hadoop.hbase.thrift.ThriftServerRunner$3.process(ThriftServerRunner.java:502)\n\tat
>
> org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer$ClientConnnection.run(TBoundedThreadPoolServer.java:289)\n\tat
>
>
[jira] [Created] (HBASE-15129) Set default value for hbase.fs.tmp.dir rather than fully depend on hbase-default.xml
Yu Li created HBASE-15129:
-
Summary: Set default value for hbase.fs.tmp.dir rather than fully
depend on hbase-default.xml
Key: HBASE-15129
URL: https://issues.apache.org/jira/browse/HBASE-15129
Project: HBase
Issue Type: Bug
Reporter: Yu Li
Assignee: Yu Li
One of our users has observed below error when our cluster upgrades from
0.98.12 to 1.1.2:
{noformat}
java.lang.IllegalArgumentException: Can not create a Path from a null string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:122)
at org.apache.hadoop.fs.Path.(Path.java:134)
at org.apache.hadoop.fs.Path.(Path.java:88)
at
org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configurePartitioner(HFileOutputFormat2.java:639)
at
org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalLoad(HFileOutputFormat2.java:489)
{noformat}
And checking the application code:
{code}
private Job createJob(Configuration jobConf) throws IOException {
String tableName = jobConf.get(TABLE_NAME_KEY);
Job job = new Job(jobConf);
Configuration tableConf = HBaseConfiguration.create();
HTable table = new HTable(tableConf, tableName);
HFileOutputFormat2.configureIncrementalLoad(job, table);
return job;
}
{code}
We could see the user has his own hbase-independent job configuration, so our
current code in HFileOutputFormat2:
{code}
Path partitionsPath = new Path(conf.get("hbase.fs.tmp.dir"), "partitions_" +
UUID.randomUUID());
{code}
will return a null string and cause the above exception.
We propose to add default value in the code rather than depend on
hbase-default.xml in this JIRA.
This is an improvement for HBASE-13625
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15082) Fix merge of MVCC and SequenceID performance regression
[
https://issues.apache.org/jira/browse/HBASE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107592#comment-15107592
]
Hadoop QA commented on HBASE-15082:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 18 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 18s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 16s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 42s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 50s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 58s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 19s
{color} | {color:red} hbase-examples in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 57s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 6s
{color} | {color:red} Patch generated 2 new checkstyle issues in hbase-common
(total was 97, now 99). {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 14s
{color} | {color:red} Patch generated 53 new checkstyle issues in hbase-server
(total was 853, now 825). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
20m 18s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 55s
{color} | {color:red} hbase-common introduced 1 new FindBugs issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 3s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 5m 26s
{color} | {color:red} hbase-server-jdk1.7.0_79 with JDK v1.7.0_79 generated 2
new issues (was 1, now 3). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 7s
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 57s
{color} | {color:green} hbase-common in the patch passed with JDK v1.8.0.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 32s
{color} | {color:green} hbase-examples in the patch passed with JDK v1.8.0.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 119m 7s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0.
[jira] [Updated] (HBASE-15107) Procedure V2 - Procedure Queue with Regions
[ https://issues.apache.org/jira/browse/HBASE-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-15107: Attachment: HBASE-15107-v1.patch > Procedure V2 - Procedure Queue with Regions > --- > > Key: HBASE-15107 > URL: https://issues.apache.org/jira/browse/HBASE-15107 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-15107-v0.patch, HBASE-15107-v1.patch > > > Adds the "region locking" that will be used to perform assign/unassign, > split/merge operations. > An operation take the xlock on the regions is working on, all the other > procedures will be suspended (removed from the runnable queue) and resumed > (put back in the runnable queue) when the operation that has the lock on the > region is completed. > https://reviews.apache.org/r/42213/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15102) HeapMemoryTuner can "overtune" memstore size and suddenly drop it into blocking zone
[ https://issues.apache.org/jira/browse/HBASE-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-15102: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed everywhere. Thanks > HeapMemoryTuner can "overtune" memstore size and suddenly drop it into > blocking zone > > > Key: HBASE-15102 > URL: https://issues.apache.org/jira/browse/HBASE-15102 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15102-V0.patch, HBASE-15102-V1.patch, > HBASE-15102-V2.patch, HBASE-15102-V3.patch > > > DefaultHeapMemoryTuner often resets the maximum step size for tuning to 8% of > total heap size. Often, when the size of memstore is to be decreased while > tuning, the 8% tuning can suddenly drop the memstore size below the low water > mark of the previous memstore size (which could potentially be the used size > of the memstore) > This is problematic because suddenly it blocks all the updates by suddenly > causing a situation where memstore used size is above high water mark. This > has a very bad performance impact on an otherwise fine HBase cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14902) Revert some of the stringency recently introduced by checkstyle tightening
[ https://issues.apache.org/jira/browse/HBASE-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107601#comment-15107601 ] stack commented on HBASE-14902: --- Pushed the patch. Lets see if it breaks checkstyle again. > Revert some of the stringency recently introduced by checkstyle tightening > -- > > Key: HBASE-14902 > URL: https://issues.apache.org/jira/browse/HBASE-14902 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: 14902.part1.patch, 14902.patch, 14902.patch, braces.patch > > > I think we should undo some of the plugins that were recently added to > checkstyle. They are too much. > JavadocTagContinuationIndentationCheck is about adding indent if javadoc is > two lines or more (javadoc tool doesn't care) > NonEmptyAtclauseDescriptionCheck would have us add javadoc on each exception: > e.g. @throws IOException needs to have text added. > NeedBracesCheck has us undoing cases where an if fits all on one line (don't > want to start style wars but if short and fits on one line, I think its more > readable... but I could relent on this one ). > The first two at least should go. > You ok w/ that [~appy] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15130) Backport 0.98 Scan different TimeRange for each column family
churro morales created HBASE-15130: -- Summary: Backport 0.98 Scan different TimeRange for each column family Key: HBASE-15130 URL: https://issues.apache.org/jira/browse/HBASE-15130 Project: HBase Issue Type: Bug Affects Versions: 0.98.17 Reporter: churro morales Assignee: churro morales Fix For: 0.98.18 branch 98 version backport for HBASE-14355 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14865) Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
[
https://issues.apache.org/jira/browse/HBASE-14865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107406#comment-15107406
]
Appy commented on HBASE-14865:
--
Em, there is one attached already (HBASE-14865-branch-1.patch, 2nd Dec, sorry
for not adding patch version to the name). I was able to cleanly merge it with
latest branch-1 using git apply just before I posted the request. Can you
please give that patch a try. Thanks [~tedyu].
> Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
> ---
>
> Key: HBASE-14865
> URL: https://issues.apache.org/jira/browse/HBASE-14865
> Project: HBase
> Issue Type: Improvement
> Components: security
>Reporter: Appy
>Assignee: Appy
> Fix For: 2.0.0
>
> Attachments: 14865-master-v7.patch, HBASE-14865-branch-1.2.patch,
> HBASE-14865-branch-1.patch, HBASE-14865-branch-1.patch,
> HBASE-14865-master-v2.patch, HBASE-14865-master-v3.patch,
> HBASE-14865-master-v4.patch, HBASE-14865-master-v5.patch,
> HBASE-14865-master-v6.patch, HBASE-14865-master-v7.patch,
> HBASE-14865-master.patch
>
>
> Currently, we can set the value of hbase.rpc.protection to one of
> authentication/integrity/privacy. It is the used to set
> {{javax.security.sasl.qop}} in SaslUtil.java.
> The problem is, if a cluster wants to switch from one qop to another, it'll
> have to take a downtime. Rolling upgrade will create a situation where some
> nodes have old value and some have new, which'll prevent any communication
> between them. There will be similar issue when clients will try to connect.
> {{javax.security.sasl.qop}} can take in a list of QOP in preferences order.
> So a transition from qop1 to qop2 can be easily done like this
> "qop1" --> "qop2,qop1" --> rolling restart --> "qop2" --> rolling restart
> Need to change hbase.rpc.protection to accept a list too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15130) Backport 0.98 Scan different TimeRange for each column family
[
https://issues.apache.org/jira/browse/HBASE-15130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107612#comment-15107612
]
Hadoop QA commented on HBASE-15130:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
49s {color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 17s
{color} | {color:green} 0.98 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s
{color} | {color:green} 0.98 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s {color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s {color} | {color:green} 0.98 passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 56s
{color} | {color:red} hbase-client in 0.98 has 19 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 43s
{color} | {color:red} hbase-common in 0.98 has 7 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 47s
{color} | {color:red} hbase-server in 0.98 has 84 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 30s
{color} | {color:red} hbase-client in 0.98 failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 25s
{color} | {color:red} hbase-common in 0.98 failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 38s
{color} | {color:red} hbase-server in 0.98 failed with JDK v1.8.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} 0.98 passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 15s
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 20s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 36s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 36s {color} |
{color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 36s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 19s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_79.
{color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 19s {color} |
{color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 19s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 9s
{color} | {color:red} Patch generated 2 new checkstyle issues in hbase-client
(total was 12, now 14). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s
{color} | {color:red} The patch has 144 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 44s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 2s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc
[jira] [Updated] (HBASE-15130) Backport 0.98 Scan different TimeRange for each column family
[ https://issues.apache.org/jira/browse/HBASE-15130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] churro morales updated HBASE-15130: --- Attachment: HBASE-15130-0.98.patch > Backport 0.98 Scan different TimeRange for each column family > -- > > Key: HBASE-15130 > URL: https://issues.apache.org/jira/browse/HBASE-15130 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.17 >Reporter: churro morales >Assignee: churro morales > Fix For: 0.98.18 > > Attachments: HBASE-15130-0.98.patch > > > branch 98 version backport for HBASE-14355 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15130) Backport 0.98 Scan different TimeRange for each column family
[ https://issues.apache.org/jira/browse/HBASE-15130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] churro morales updated HBASE-15130: --- Status: Patch Available (was: Open) > Backport 0.98 Scan different TimeRange for each column family > -- > > Key: HBASE-15130 > URL: https://issues.apache.org/jira/browse/HBASE-15130 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.17 >Reporter: churro morales >Assignee: churro morales > Fix For: 0.98.18 > > Attachments: HBASE-15130-0.98.patch > > > branch 98 version backport for HBASE-14355 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14872) Scan different timeRange per column family doesn't percolate down to the memstore
[ https://issues.apache.org/jira/browse/HBASE-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107376#comment-15107376 ] churro morales commented on HBASE-14872: moving the backport to its own jira HBASE-15130 > Scan different timeRange per column family doesn't percolate down to the > memstore > -- > > Key: HBASE-14872 > URL: https://issues.apache.org/jira/browse/HBASE-14872 > Project: HBase > Issue Type: Bug > Components: Client, regionserver, Scanners >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 1.2.0, 0.98.18 > > Attachments: HBASE-14872-0.98-v1.patch, HBASE-14872-0.98.patch, > HBASE-14872-v1.patch, HBASE-14872.patch > > > HBASE-14355 The scan different time range for column family feature was not > applied to the memstore it was only done for the store files. This breaks > the contract. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14865) Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
[
https://issues.apache.org/jira/browse/HBASE-14865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-14865:
---
Resolution: Fixed
Fix Version/s: 1.3.0
Status: Resolved (was: Patch Available)
> Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
> ---
>
> Key: HBASE-14865
> URL: https://issues.apache.org/jira/browse/HBASE-14865
> Project: HBase
> Issue Type: Improvement
> Components: security
>Reporter: Appy
>Assignee: Appy
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 14865-master-v7.patch, HBASE-14865-branch-1.2.patch,
> HBASE-14865-branch-1.patch, HBASE-14865-branch-1.patch,
> HBASE-14865-master-v2.patch, HBASE-14865-master-v3.patch,
> HBASE-14865-master-v4.patch, HBASE-14865-master-v5.patch,
> HBASE-14865-master-v6.patch, HBASE-14865-master-v7.patch,
> HBASE-14865-master.patch
>
>
> Currently, we can set the value of hbase.rpc.protection to one of
> authentication/integrity/privacy. It is the used to set
> {{javax.security.sasl.qop}} in SaslUtil.java.
> The problem is, if a cluster wants to switch from one qop to another, it'll
> have to take a downtime. Rolling upgrade will create a situation where some
> nodes have old value and some have new, which'll prevent any communication
> between them. There will be similar issue when clients will try to connect.
> {{javax.security.sasl.qop}} can take in a list of QOP in preferences order.
> So a transition from qop1 to qop2 can be easily done like this
> "qop1" --> "qop2,qop1" --> rolling restart --> "qop2" --> rolling restart
> Need to change hbase.rpc.protection to accept a list too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15106) Procedure V2 - Procedure Queue pass Procedure for better debuggability
[ https://issues.apache.org/jira/browse/HBASE-15106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-15106: Resolution: Fixed Fix Version/s: 1.2.0 Status: Resolved (was: Patch Available) > Procedure V2 - Procedure Queue pass Procedure for better debuggability > -- > > Key: HBASE-15106 > URL: https://issues.apache.org/jira/browse/HBASE-15106 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.3.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15106-v0.patch, HBASE-15106-v1.patch > > > Changes the various acquire/release methods to take the Procedure as argument. > That allows better debuggability. > (The patch it is just a refactor, it does not introduce any new thing) > https://reviews.apache.org/r/42271/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14524) Short-circuit comparison of rows in CellComparator
[
https://issues.apache.org/jira/browse/HBASE-14524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107437#comment-15107437
]
Lars Francke commented on HBASE-14524:
--
[~anoop.hbase] Thanks a lot for checking up on this, reviewing and committing!
> Short-circuit comparison of rows in CellComparator
> --
>
> Key: HBASE-14524
> URL: https://issues.apache.org/jira/browse/HBASE-14524
> Project: HBase
> Issue Type: Improvement
> Components: Performance, Scanners
>Reporter: Lars Francke
>Assignee: Lars Francke
> Fix For: 2.0.0
>
> Attachments: HBASE-14524.patch, HBASE-14524.patch, HBASE-14524.patch
>
>
> {{CellComparator#compareRows}} compares two rows. For Scans and Gets these
> objects can sometimes be exactly the same object.
> I see this (without fully understanding everything that's going on) for the
> first cell in each row.
> In these circumstances I think it makes sense to short-circuit the comparison
> (not do a byte comparison and class cast checks) and bail out early if the
> objects are identical.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14902) Revert some of the stringency recently introduced by checkstyle tightening
[
https://issues.apache.org/jira/browse/HBASE-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107571#comment-15107571
]
Hadoop QA commented on HBASE-14902:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 7s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 48s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s
{color} | {color:green} hbase-checkstyle in the patch passed with JDK v1.8.0.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 10s
{color} | {color:green} hbase-checkstyle in the patch passed with JDK
v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 56s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12783160/braces.patch |
| JIRA Issue | HBASE-14902 |
| Optional Tests | asflicense javac javadoc unit xml |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 318814d |
| JDK v1.7.0_79 Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/189/testReport/ |
| modules | C: hbase-checkstyle U: hbase-checkstyle |
| Max memory used | 191MB |
| Powered by | Apache Yetus 0.1.0 http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/189/console |
This message was automatically generated.
> Revert some of the stringency recently introduced by checkstyle tightening
> --
>
> Key: HBASE-14902
> URL: https://issues.apache.org/jira/browse/HBASE-14902
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments: 14902.part1.patch, 14902.patch, 14902.patch, braces.patch
>
>
> I think we should undo some of the plugins that were recently added to
> checkstyle. They are too much.
> JavadocTagContinuationIndentationCheck is about adding indent if javadoc is
> two lines or more (javadoc tool doesn't care)
> NonEmptyAtclauseDescriptionCheck would have us add javadoc on each exception:
> e.g. @throws IOException needs to have text added.
> NeedBracesCheck has us undoing cases where an if fits all on one line (don't
> want to start style wars but if short and fits on one line, I think its more
>
[jira] [Updated] (HBASE-15091) Forward-port to 1.2 HBASE-15031 "Fix merge of MVCC and SequenceID performance regression in branch-1.0"
[ https://issues.apache.org/jira/browse/HBASE-15091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15091: -- Attachment: HBASE-15091.png Here is diagram of slow vs fast using v9 of patch attached here using the included IncrementPerformanceTest. Difference is not dramatic. Diagram shows 'fast path' doing 50% more ops. Here is output from the two runs: slow path 2016-01-19 10:54:58,675 INFO [main] hbase.IncrementPerformanceTest: 75th=13.71780375, 95th=18.15133694997, 99th=22.9188527901 real1m51.259s user1m31.020s sys 0m39.621s 'fast' path (with hbase.increment.fast.but.narrow.consistency set to true in config) 2016-01-19 10:45:29,390 INFO [main] hbase.IncrementPerformanceTest: 75th=8.748498, 95th=11.46681715, 99th=18.06752105085 real1m10.096s user1m32.368s sys 0m38.271s > Forward-port to 1.2 HBASE-15031 "Fix merge of MVCC and SequenceID performance > regression in branch-1.0" > --- > > Key: HBASE-15091 > URL: https://issues.apache.org/jira/browse/HBASE-15091 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 1.2.0 > > Attachments: 15091v2.branch-1.2.patch, 15091v3.branch-1.2.patch, > 15091v4.branch-1.2.patch, 15091v5.branch-1.2.patch, 15091v6.branch-1.2.patch, > 15091v6.branch-1.patch, 15091v7.branch-1.2.patch, 15091v8.branch-1.2.patch, > 15091v9.branch-1.2.patch, HBASE-15091-branch-1.2.patch, > HBASE-15091-branch-1.2_v1.patch, HBASE-15091.png, > HBASE-15091.v1.branch-1.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14865) Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
[
https://issues.apache.org/jira/browse/HBASE-14865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107353#comment-15107353
]
Ted Yu commented on HBASE-14865:
Would be more than happy to.
Can you attach patch for branch-1 ?
{code}
-rw-r--r-- 1 tyu staff 13396 Jan 19 12:13
./hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestHBaseSaslRpcClient.java.rej
-rw-r--r-- 1 tyu staff 9254 Jan 19 12:13
./hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestSecureRPC.java.rej
{code}
> Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
> ---
>
> Key: HBASE-14865
> URL: https://issues.apache.org/jira/browse/HBASE-14865
> Project: HBase
> Issue Type: Improvement
> Components: security
>Reporter: Appy
>Assignee: Appy
> Fix For: 2.0.0
>
> Attachments: 14865-master-v7.patch, HBASE-14865-branch-1.2.patch,
> HBASE-14865-branch-1.patch, HBASE-14865-branch-1.patch,
> HBASE-14865-master-v2.patch, HBASE-14865-master-v3.patch,
> HBASE-14865-master-v4.patch, HBASE-14865-master-v5.patch,
> HBASE-14865-master-v6.patch, HBASE-14865-master-v7.patch,
> HBASE-14865-master.patch
>
>
> Currently, we can set the value of hbase.rpc.protection to one of
> authentication/integrity/privacy. It is the used to set
> {{javax.security.sasl.qop}} in SaslUtil.java.
> The problem is, if a cluster wants to switch from one qop to another, it'll
> have to take a downtime. Rolling upgrade will create a situation where some
> nodes have old value and some have new, which'll prevent any communication
> between them. There will be similar issue when clients will try to connect.
> {{javax.security.sasl.qop}} can take in a list of QOP in preferences order.
> So a transition from qop1 to qop2 can be easily done like this
> "qop1" --> "qop2,qop1" --> rolling restart --> "qop2" --> rolling restart
> Need to change hbase.rpc.protection to accept a list too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15126) HBaseFsck's checkRegionBoundaries function set the 'storesFirstKey' was incorrect.
[ https://issues.apache.org/jira/browse/HBASE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107114#comment-15107114 ] Ted Yu commented on HBASE-15126: Almost every line of HBaseFsck.java appeared in the patch. Please reformat the patch so that only relevant changes show up. > HBaseFsck's checkRegionBoundaries function set the 'storesFirstKey' was > incorrect. > --- > > Key: HBASE-15126 > URL: https://issues.apache.org/jira/browse/HBASE-15126 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 2.0.0 >Reporter: chenrongwei >Assignee: chenrongwei > Attachments: HBASE-15126-v001.patch, HBASE-15126-v002.patch, > HBASE-15126-v003.patch > > > HBaseFsck's checkRegionBoundaries function set the > 'currentRegionBoundariesInformation.storesFirstKey' was incorrect.I think it > should be set like below, > currentRegionBoundariesInformation.storesFirstKey = keyOnly(storeFirstKey); > but current the 'currentRegionBoundariesInformation.storesFirstKey ' is just > set to 'storeFirstKey',which will cause to > comparator.compare(currentRegionBoundariesInformation.storesFirstKey, > currentRegionBoundariesInformation.metaFirstKey) get a wrong > result. Because it just compared the rowkey's length. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14030) HBase Backup/Restore Phase 1
[ https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-14030: -- Attachment: HBASE-14030-v30.patch > HBase Backup/Restore Phase 1 > > > Key: HBASE-14030 > URL: https://issues.apache.org/jira/browse/HBASE-14030 > Project: HBase > Issue Type: Umbrella >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, > HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, > HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch, > HBASE-14030-v17.patch, HBASE-14030-v18.patch, HBASE-14030-v2.patch, > HBASE-14030-v20.patch, HBASE-14030-v21.patch, HBASE-14030-v22.patch, > HBASE-14030-v23.patch, HBASE-14030-v24.patch, HBASE-14030-v25.patch, > HBASE-14030-v26.patch, HBASE-14030-v27.patch, HBASE-14030-v28.patch, > HBASE-14030-v3.patch, HBASE-14030-v30.patch, HBASE-14030-v4.patch, > HBASE-14030-v5.patch, HBASE-14030-v6.patch, HBASE-14030-v7.patch, > HBASE-14030-v8.patch > > > This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design > doc for the phase description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14030) HBase Backup/Restore Phase 1
[ https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107121#comment-15107121 ] Vladimir Rodionov commented on HBASE-14030: --- Put the latest patch (v30) on review board: https://reviews.apache.org/r/36591/ > HBase Backup/Restore Phase 1 > > > Key: HBASE-14030 > URL: https://issues.apache.org/jira/browse/HBASE-14030 > Project: HBase > Issue Type: Umbrella >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, > HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, > HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch, > HBASE-14030-v17.patch, HBASE-14030-v18.patch, HBASE-14030-v2.patch, > HBASE-14030-v20.patch, HBASE-14030-v21.patch, HBASE-14030-v22.patch, > HBASE-14030-v23.patch, HBASE-14030-v24.patch, HBASE-14030-v25.patch, > HBASE-14030-v26.patch, HBASE-14030-v27.patch, HBASE-14030-v28.patch, > HBASE-14030-v3.patch, HBASE-14030-v4.patch, HBASE-14030-v5.patch, > HBASE-14030-v6.patch, HBASE-14030-v7.patch, HBASE-14030-v8.patch > > > This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design > doc for the phase description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15091) Forward-port to 1.2 HBASE-15031 "Fix merge of MVCC and SequenceID performance regression in branch-1.0"
[
https://issues.apache.org/jira/browse/HBASE-15091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-15091:
--
Release Note:
Increments can be 10x slower (or more) when there is high concurrency since
HBase 1.0.0 (HBASE-8763).
This 'fix' adds back a fast increment but speed is achieved by relaxing
row-level consistency for Increments (only). The default remains the old, slow,
consistent Increment behavior.
Set "hbase.increment.fast.but.narrow.consistency" to true in hbase-site.xml to
enable 'fast' increments and then rolling restart your cluster. This is a
setting the server-side needs to read.
Intermixing fast increment with other Mutations will give indeterminate
results; e.g. a Put and Increment against the same Cell will not always give
you the result you expect. Fast Increments are consistent unto themselves. A
Get with {@link IsolationLevel#READ_UNCOMMITTED} will return the latest
increment value or an Increment of an amount zero will do the same (beware
doing Get on a cell that has not been incremented yet -- this will return no
results).
The difference between fastAndNarrowConsistencyIncrement and
slowButConsistentIncrement is that the former holds the row lock until the WAL
sync completes; this allows us to reason that there are no other writers afoot
when we read the current increment value. In this case we do not need to wait
on mvcc reads to catch up to writes before we proceed with the read of the
current Increment value, the root of the slowdown seen in HBASE-14460. The
fast-path also does not wait on mvcc to complete before returning to the client
(but the write has been synced and put into memstore before we return).
Also adds a simple performance test tool that will run against existing
cluster. It expects the table to be already created (by default it expects the
table 'tableName' with a column family 'columnFamilyName'):
{code}
$ ./bin/hbase org.apache.hadoop.hbase.IncrementPerformanceTest
{code]
Configure it by passing -D options. Here are the set below:
2015-12-23 19:33:36,941 INFO [main] hbase.IncrementPerformanceTest: Running
test with hbase.zookeeper.quorum=localhost, tableName=tableName,
columnFamilyName=columnFamilyName, threadCount=80, incrementCount=1
... so to set the tableName pass -DtableName=SOME_TABLENAME
Here is an example use of the test tool:
{code}
$ time ./bin/hbase --config ~/conf_hbase
org.apache.hadoop.hbase.IncrementPerformanceTest -DincrementCount=5
{code}
> Forward-port to 1.2 HBASE-15031 "Fix merge of MVCC and SequenceID performance
> regression in branch-1.0"
> ---
>
> Key: HBASE-15091
> URL: https://issues.apache.org/jira/browse/HBASE-15091
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 1.2.0
>
> Attachments: 15091v2.branch-1.2.patch, 15091v3.branch-1.2.patch,
> 15091v4.branch-1.2.patch, 15091v5.branch-1.2.patch, 15091v6.branch-1.2.patch,
> 15091v6.branch-1.patch, 15091v7.branch-1.2.patch, 15091v8.branch-1.2.patch,
> 15091v9.branch-1.2.patch, HBASE-15091-branch-1.2.patch,
> HBASE-15091-branch-1.2_v1.patch, HBASE-15091.png,
> HBASE-15091.v1.branch-1.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15102) HeapMemoryTuner can "overtune" memstore size and suddenly drop it into blocking zone
[ https://issues.apache.org/jira/browse/HBASE-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashu Pachauri updated HBASE-15102: -- Attachment: HBASE-15102-V3.patch V3: Fixed checkstyle issues except the indentation ones (mainly because checkstyle reports no errors when I run it locally, not sure why). > HeapMemoryTuner can "overtune" memstore size and suddenly drop it into > blocking zone > > > Key: HBASE-15102 > URL: https://issues.apache.org/jira/browse/HBASE-15102 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15102-V0.patch, HBASE-15102-V1.patch, > HBASE-15102-V2.patch, HBASE-15102-V3.patch > > > DefaultHeapMemoryTuner often resets the maximum step size for tuning to 8% of > total heap size. Often, when the size of memstore is to be decreased while > tuning, the 8% tuning can suddenly drop the memstore size below the low water > mark of the previous memstore size (which could potentially be the used size > of the memstore) > This is problematic because suddenly it blocks all the updates by suddenly > causing a situation where memstore used size is above high water mark. This > has a very bad performance impact on an otherwise fine HBase cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14865) Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
[
https://issues.apache.org/jira/browse/HBASE-14865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107295#comment-15107295
]
Appy commented on HBASE-14865:
--
[~tedyu] Please commit it to branch-1 too. Thanks.
> Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
> ---
>
> Key: HBASE-14865
> URL: https://issues.apache.org/jira/browse/HBASE-14865
> Project: HBase
> Issue Type: Improvement
> Components: security
>Reporter: Appy
>Assignee: Appy
> Fix For: 2.0.0
>
> Attachments: 14865-master-v7.patch, HBASE-14865-branch-1.2.patch,
> HBASE-14865-branch-1.patch, HBASE-14865-branch-1.patch,
> HBASE-14865-master-v2.patch, HBASE-14865-master-v3.patch,
> HBASE-14865-master-v4.patch, HBASE-14865-master-v5.patch,
> HBASE-14865-master-v6.patch, HBASE-14865-master-v7.patch,
> HBASE-14865-master.patch
>
>
> Currently, we can set the value of hbase.rpc.protection to one of
> authentication/integrity/privacy. It is the used to set
> {{javax.security.sasl.qop}} in SaslUtil.java.
> The problem is, if a cluster wants to switch from one qop to another, it'll
> have to take a downtime. Rolling upgrade will create a situation where some
> nodes have old value and some have new, which'll prevent any communication
> between them. There will be similar issue when clients will try to connect.
> {{javax.security.sasl.qop}} can take in a list of QOP in preferences order.
> So a transition from qop1 to qop2 can be easily done like this
> "qop1" --> "qop2,qop1" --> rolling restart --> "qop2" --> rolling restart
> Need to change hbase.rpc.protection to accept a list too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15111) "hbase version" should write to stdout
[
https://issues.apache.org/jira/browse/HBASE-15111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107355#comment-15107355
]
Matteo Bertozzi commented on HBASE-15111:
-
+1
> "hbase version" should write to stdout
> --
>
> Key: HBASE-15111
> URL: https://issues.apache.org/jira/browse/HBASE-15111
> Project: HBase
> Issue Type: Improvement
> Components: util
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: hbase-15111-v1.patch
>
>
> Calling {{hbase version}} currently outputs the version info by writing to
> {{LOG.info}}. This means, if you change the default log level settings, you
> may get no output at all on the command line.
> Since {{VersionInfo.main()}} is being called, it should really just output
> straight to stdout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15128) Disable region splits and merges in HBCK
[ https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107248#comment-15107248 ] Enis Soztutar commented on HBASE-15128: --- bq. It is not only for hbck. Snapshots and bulk load will benefit from disabling splits as well. Agreed, a long running snapshot fails if there is a split or merge that happen concurrently. > Disable region splits and merges in HBCK > > > Key: HBASE-15128 > URL: https://issues.apache.org/jira/browse/HBASE-15128 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0 > > > In large clusters where region splits are frequent, and HBCK runs take > longer, the concurrent splits cause further problems in HBCK since HBCK > assumes a static state for the region partition map. We have just seen a case > where HBCK undo's a concurrently splitting region causing number of > inconsistencies to go up. > We can have a mode in master where splits and merges are disabled like the > balancer and catalog janitor switches. Master will reject the split requests > if regionservers decide to split. This switch can be turned on / off by the > admins and also automatically by HBCK while it is running (similar to > balancer switch being disabled by HBCK). > HBCK should also disable the Catalog Janitor just in case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15125) HBaseFsck's adoptHdfsOrphan function create region with wrong end key boundary.
[ https://issues.apache.org/jira/browse/HBASE-15125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107118#comment-15107118 ] Ted Yu commented on HBASE-15125: Please reformat patch to keep relevant changes only. > HBaseFsck's adoptHdfsOrphan function create region with wrong end key > boundary. > --- > > Key: HBASE-15125 > URL: https://issues.apache.org/jira/browse/HBASE-15125 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 2.0.0 >Reporter: chenrongwei >Assignee: chenrongwei > Attachments: HBASE-15125-V001.patch, HBASE-15125-v002.patch > > > There is a bug in HBaseFsck's adoptHdfsOrphan function.At the last of this > function will create a region,which want to cover all the orphan regions.But > the end key of this new region was set incorrectly.Correct region's boundary > should be [startKey,endKey),but this function create a region with boundary > of [startKey,endKey],this bug will leads to scan operation omit some data. > I think we should create the region like bellow, > // create new region on hdfs. move data into place. > HRegionInfo hri = new HRegionInfo(template.getTableName(), > orphanRegionRange.getFirst(), > Bytes.add(orphanRegionRange.getSecond(), new byte[1])); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15093) Replication can report incorrect size of log queue for the global source when multiwal is enabled
[ https://issues.apache.org/jira/browse/HBASE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashu Pachauri updated HBASE-15093: -- Status: Patch Available (was: Open) > Replication can report incorrect size of log queue for the global source when > multiwal is enabled > - > > Key: HBASE-15093 > URL: https://issues.apache.org/jira/browse/HBASE-15093 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.2.0, 1.2.1 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri >Priority: Minor > Attachments: HBASE-15093-V0.patch > > > Replication can report incorrect size for the size of log queue for the > global source when multiwal is enabled. This happens because the method > MetricsSource#setSizeofLogQueue performs non-trivial operations in a > multithreaded world, even though it is not synchronized. > We can simply divide MetricsSource#setSizeofLogQueue into > MetricsSource#incrSizeofLogQueue and MetricsSource#decrSizeofLogQueue. Not > sure why we are currently directly setting the size instead of > incrementing/decrementing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15122) Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
[
https://issues.apache.org/jira/browse/HBASE-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107074#comment-15107074
]
Hadoop QA commented on HBASE-15122:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 48s
{color} | {color:red} hbase-server in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 8 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 62 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 38s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 5s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 34s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 3s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 33s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m 3s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 33s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 12m 4s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 13m 34s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} |
[jira] [Updated] (HBASE-13960) HConnection stuck with UnknownHostException
[
https://issues.apache.org/jira/browse/HBASE-13960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-13960:
---
Status: Patch Available (was: Open)
> HConnection stuck with UnknownHostException
>
>
> Key: HBASE-13960
> URL: https://issues.apache.org/jira/browse/HBASE-13960
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 0.98.8
>Reporter: Kurt Young
> Attachments: HBASE-13960-0.98-v1.patch, HBASE-13960-update.patch,
> HBASE-13960-v2.patch
>
>
> when put/get from hbase, if we meet a temporary dns failure causes resolve
> RS's host, the error will never recovered. put/get will failed with
> UnknownHostException forever.
> I checked the code, and the reason maybe:
> 1. when RegionServerCallable or MultiServerCallable prepare(), it gets a
> ClientService.BlockingInterface stub from Hconnection
> 2. In HConnectionImplementation::getClient, it caches the stub with a
> BlockingRpcChannelImplementation
> 3. In BlockingRpcChannelImplementation(),
> this.isa = new InetSocketAddress(sn.getHostname(), sn.getPort()); If we
> meet a temporary dns failure then the "address" in isa will be null.
> 4. then we launch the real rpc call, the following stack is:
> Caused by: java.net.UnknownHostException: unknown host: xxx.host2
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.(RpcClient.java:385)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.createConnection(RpcClient.java:351)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1523)
> at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1435)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1654)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1712)
> Besides, i noticed there is a protection in RpcClient:
> if (remoteId.getAddress().isUnresolved()) {
> throw new UnknownHostException("unknown host: " +
> remoteId.getAddress().getHostName());
> }
> shouldn't we do something when this situation occurred?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14902) Revert some of the stringency recently introduced by checkstyle tightening
[ https://issues.apache.org/jira/browse/HBASE-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14902: -- Attachment: braces.patch Here is no-braces, statements on a single line... but as a patch I don't think it gets triggered Locally it seems to do fine but probably need to commit to see if this still breaks checkstyle. Let me try this patch first. > Revert some of the stringency recently introduced by checkstyle tightening > -- > > Key: HBASE-14902 > URL: https://issues.apache.org/jira/browse/HBASE-14902 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: 14902.part1.patch, 14902.patch, 14902.patch, braces.patch > > > I think we should undo some of the plugins that were recently added to > checkstyle. They are too much. > JavadocTagContinuationIndentationCheck is about adding indent if javadoc is > two lines or more (javadoc tool doesn't care) > NonEmptyAtclauseDescriptionCheck would have us add javadoc on each exception: > e.g. @throws IOException needs to have text added. > NeedBracesCheck has us undoing cases where an if fits all on one line (don't > want to start style wars but if short and fits on one line, I think its more > readable... but I could relent on this one ). > The first two at least should go. > You ok w/ that [~appy] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14902) Revert some of the stringency recently introduced by checkstyle tightening
[ https://issues.apache.org/jira/browse/HBASE-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14902: -- Status: Patch Available (was: Reopened) > Revert some of the stringency recently introduced by checkstyle tightening > -- > > Key: HBASE-14902 > URL: https://issues.apache.org/jira/browse/HBASE-14902 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: 14902.part1.patch, 14902.patch, 14902.patch, braces.patch > > > I think we should undo some of the plugins that were recently added to > checkstyle. They are too much. > JavadocTagContinuationIndentationCheck is about adding indent if javadoc is > two lines or more (javadoc tool doesn't care) > NonEmptyAtclauseDescriptionCheck would have us add javadoc on each exception: > e.g. @throws IOException needs to have text added. > NeedBracesCheck has us undoing cases where an if fits all on one line (don't > want to start style wars but if short and fits on one line, I think its more > readable... but I could relent on this one ). > The first two at least should go. > You ok w/ that [~appy] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14902) Revert some of the stringency recently introduced by checkstyle tightening
[ https://issues.apache.org/jira/browse/HBASE-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14902: -- Release Note: Changes the checkstyle so that on a continuation line for javadoc, instead of default four spaces, instead now it is two spaces. Also one line statements as in if (true) x =1; now pass checkstyle. (was: Changes the checkstyle so that on a continuation line for javadoc, instead of default four spaces, instead not it is two spaces. Also one line statements as in if (true) x =1; now pass checkstyle.) > Revert some of the stringency recently introduced by checkstyle tightening > -- > > Key: HBASE-14902 > URL: https://issues.apache.org/jira/browse/HBASE-14902 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: 14902.part1.patch, 14902.patch, 14902.patch, braces.patch > > > I think we should undo some of the plugins that were recently added to > checkstyle. They are too much. > JavadocTagContinuationIndentationCheck is about adding indent if javadoc is > two lines or more (javadoc tool doesn't care) > NonEmptyAtclauseDescriptionCheck would have us add javadoc on each exception: > e.g. @throws IOException needs to have text added. > NeedBracesCheck has us undoing cases where an if fits all on one line (don't > want to start style wars but if short and fits on one line, I think its more > readable... but I could relent on this one ). > The first two at least should go. > You ok w/ that [~appy] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14676) HBaseTestCase clean out: Purge Incommon Interface and Table and Region implementations
[ https://issues.apache.org/jira/browse/HBASE-14676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14676: -- Attachment: connection_for_region_as_table.patch Parking patch here. Was trying to make a Connection that would give back RegionAsTable... Got stuck on how to bring up Region. Need too many parameters. In particular I want the WAL to work but bunch of setup involved... could add it all as config... TODO (I wanted this to do perf testing against Region via YCSB). > HBaseTestCase clean out: Purge Incommon Interface and Table and Region > implementations > -- > > Key: HBASE-14676 > URL: https://issues.apache.org/jira/browse/HBASE-14676 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: 14676.patch, 14676v2.txt, 14676v3.txt, 14676v4.txt, > 14676v4.txt, RegionAdmin.java, connection_for_region_as_table.patch > > > As part of the hollowing out of the old HBaseTestCase in preparation for > removal, this patch purges the old Incommon trick that made it so you could > pass an Interface to a method to do loading or testing and the implementation > could be an HTable or a Region. Instead, replace it with a RegionTable, a > Regoin that has the Table Interface as a decoration so you can use same > loading code and same test both on top of the Region or on other side of > network via HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14877) maven archetype: client application
[ https://issues.apache.org/jira/browse/HBASE-14877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107741#comment-15107741 ] Daniel Vimont commented on HBASE-14877: --- Just wanted to check: Is there anything else *I* need to do to shepherd this patch forward from this point? (If it simply needs to be reviewed by others -- I realize you're all very busy, so please just get to it when you can.) I wanted to be sure nobody's waiting on *me* to do something. > maven archetype: client application > --- > > Key: HBASE-14877 > URL: https://issues.apache.org/jira/browse/HBASE-14877 > Project: HBase > Issue Type: Sub-task > Components: build, Usability >Affects Versions: 2.0.0 >Reporter: Nick Dimiduk >Assignee: Daniel Vimont > Labels: archetype, beginner, maven > Attachments: HBASE-14877-v2.patch, HBASE-14877-v3.patch, > HBASE-14877.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15131) Turn on multi-WAL by default
Enis Soztutar created HBASE-15131: - Summary: Turn on multi-WAL by default Key: HBASE-15131 URL: https://issues.apache.org/jira/browse/HBASE-15131 Project: HBase Issue Type: New Feature Reporter: Enis Soztutar Fix For: 2.0.0, 1.3.0 Something to discuss for 2.0 or even 1.3 or 1.4. Should we turn on multi-WAL by default now that it has seen some production use. Most of the known issues has been fixed I believe for replication, metrics etc. See HBASE-14457. Using bounded provider with default 4 (assuming 12 disks). Should we also look at the number of disks from datanode dirs and auto-configure? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14865) Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
[
https://issues.apache.org/jira/browse/HBASE-14865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107792#comment-15107792
]
Hudson commented on HBASE-14865:
FAILURE: Integrated in HBase-1.3 #501 (See
[https://builds.apache.org/job/HBase-1.3/501/])
HBASE-14865 Support passing multiple QOPs to SaslClient/Server via (tedyu: rev
e76a2e4e6d91deee250d180b75b890f743da4bf0)
* hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestSecureRPC.java
*
hbase-client/src/test/java/org/apache/hadoop/hbase/security/TestHBaseSaslRpcClient.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestAsyncSecureIPC.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/security/AbstractTestSecureIPC.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/security/SaslClientHandler.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestSecureIPC.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/security/SaslUtil.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcChannel.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
* hbase-client/src/test/java/org/apache/hadoop/hbase/security/TestSaslUtil.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestHBaseSaslRpcClient.java
> Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
> ---
>
> Key: HBASE-14865
> URL: https://issues.apache.org/jira/browse/HBASE-14865
> Project: HBase
> Issue Type: Improvement
> Components: security
>Reporter: Appy
>Assignee: Appy
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 14865-master-v7.patch, HBASE-14865-branch-1.2.patch,
> HBASE-14865-branch-1.patch, HBASE-14865-branch-1.patch,
> HBASE-14865-master-v2.patch, HBASE-14865-master-v3.patch,
> HBASE-14865-master-v4.patch, HBASE-14865-master-v5.patch,
> HBASE-14865-master-v6.patch, HBASE-14865-master-v7.patch,
> HBASE-14865-master.patch
>
>
> Currently, we can set the value of hbase.rpc.protection to one of
> authentication/integrity/privacy. It is the used to set
> {{javax.security.sasl.qop}} in SaslUtil.java.
> The problem is, if a cluster wants to switch from one qop to another, it'll
> have to take a downtime. Rolling upgrade will create a situation where some
> nodes have old value and some have new, which'll prevent any communication
> between them. There will be similar issue when clients will try to connect.
> {{javax.security.sasl.qop}} can take in a list of QOP in preferences order.
> So a transition from qop1 to qop2 can be easily done like this
> "qop1" --> "qop2,qop1" --> rolling restart --> "qop2" --> rolling restart
> Need to change hbase.rpc.protection to accept a list too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15102) HeapMemoryTuner can "overtune" memstore size and suddenly drop it into blocking zone
[ https://issues.apache.org/jira/browse/HBASE-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107790#comment-15107790 ] Hudson commented on HBASE-15102: FAILURE: Integrated in HBase-1.3 #501 (See [https://builds.apache.org/job/HBase-1.3/501/]) HBASE-15102 Fix HeapMemoryTuner overtuning memstore (eclark: rev a61f3ecc8b3dad256b7ec5258e1ec4aa811cb606) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHeapMemoryManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultHeapMemoryTuner.java > HeapMemoryTuner can "overtune" memstore size and suddenly drop it into > blocking zone > > > Key: HBASE-15102 > URL: https://issues.apache.org/jira/browse/HBASE-15102 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15102-V0.patch, HBASE-15102-V1.patch, > HBASE-15102-V2.patch, HBASE-15102-V3.patch > > > DefaultHeapMemoryTuner often resets the maximum step size for tuning to 8% of > total heap size. Often, when the size of memstore is to be decreased while > tuning, the 8% tuning can suddenly drop the memstore size below the low water > mark of the previous memstore size (which could potentially be the used size > of the memstore) > This is problematic because suddenly it blocks all the updates by suddenly > causing a situation where memstore used size is above high water mark. This > has a very bad performance impact on an otherwise fine HBase cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15093) Replication can report incorrect size of log queue for the global source when multiwal is enabled
[
https://issues.apache.org/jira/browse/HBASE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107814#comment-15107814
]
Hadoop QA commented on HBASE-15093:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 47s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 10s
{color} | {color:red} hbase-hadoop2-compat in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 18s
{color} | {color:red} Patch generated 2 new checkstyle issues in hbase-server
(total was 12, now 14). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
20m 31s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s
{color} | {color:green} hbase-hadoop2-compat in the patch passed with JDK
v1.8.0. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s
{color} | {color:green} hbase-hadoop-compat in the patch passed with JDK
v1.8.0. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 47s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 33s
{color} | {color:green} hbase-hadoop2-compat in the patch passed with JDK
v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s
{color} | {color:green} hbase-hadoop-compat in the patch passed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 59s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} |
[jira] [Commented] (HBASE-13960) HConnection stuck with UnknownHostException
[
https://issues.apache.org/jira/browse/HBASE-13960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107823#comment-15107823
]
Hadoop QA commented on HBASE-13960:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 52s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
20m 23s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 3s
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 53s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_79.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 37s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 251m 28s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0 Failed junit tests | hadoop.hbase.master.TestMasterNoCluster |
| JDK v1.7.0_79 Failed junit tests |
hadoop.hbase.client.TestBlockEvictionFromClient |
| | hadoop.hbase.master.TestMasterNoCluster |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
[jira] [Commented] (HBASE-14058) Stabilizing default heap memory tuner
[ https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107757#comment-15107757 ] Hudson commented on HBASE-14058: SUCCESS: Integrated in HBase-1.2-IT #401 (See [https://builds.apache.org/job/HBase-1.2-IT/401/]) HBASE-14058 Stabilizing default heap memory tuner (eclark: rev e738e69f8cc59581a454207483aca42e7f314396) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultHeapMemoryTuner.java > Stabilizing default heap memory tuner > - > > Key: HBASE-14058 > URL: https://issues.apache.org/jira/browse/HBASE-14058 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Abhilash >Assignee: Abhilash > Fix For: 2.0.0, 1.3.0 > > Attachments: 0001-Stabilizing-default-heap-memory-tuner.patch, > HBASE-14058-v1.patch, HBASE-14058.patch, after_modifications.png, > before_modifications.png > > > The memory tuner works well in general cases but when we have a work load > that is both read heavy as well as write heavy the tuner does too many > tuning. We should try to control the number of tuner operation and stabilize > it. The main problem was that the tuner thinks it is in steady state even if > it sees just one neutral tuner period thus does too many tuning operations > and too many reverts that too with large step sizes(step size was set to > maximum even after one neutral period). So to stop this I have thought of > these steps: > 1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically > ~62% periods will lie outside this range, which means 62% of the data points > are considered either high or low which is too much. Use μ + δ*0.8 and μ - > δ*0.8 instead. On expectations it will decrease number of tuner operations > per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values > will be considered to be high and 31% will be considered to be low (2*0.31 * > 0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22% > will be high(2*0.22*0.22 ~ 0.10). > 2) Defining proper steady state by looking at past few periods(it is equal to > hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last > tuner operation. We say tuner is in steady state when last few tuner periods > were NEUTRAL. We keep decreasing step size unless it is extremely low. Then > leave system in that state for some time. > 3) Rather then decreasing step size only while reverting, decrease the > magnitude of step size whenever we are trying to revert tuning done in last > few periods(sum the changes of last few periods and compare to current step) > rather than just looking at last period. When its magnitude gets too low then > make tuner steps NEUTRAL(no operation). This will cause step size to > continuously decrease unless we reach steady state. After that tuning process > will restart (tuner step size rests again when we reach steady state). > 4) The tuning done in last few periods will be decaying sum of past tuner > steps with sign. This parameter will be positive for increase in memstore and > negative for increase in block cache. Rather than using arithmetic mean we > use this to give more priority to recent tuner steps. > Please see the attachments. One represents the size of memstore(green) and > size of block cache(blue) adjusted by tuner without these modification and > other with the above modifications. The x-axis is time axis and y-axis is the > fraction of heap memory available to memstore and block cache at that time(it > always sums up to 80%). I configured min/max ranges for both components to > 0.1 and 0.7 respectively(so in the plots the y-axis min and max is 0.1 and > 0.7). In both cases the tuner tries to distribute memory by giving ~15% to > memstore and ~65% to block cache. But the modified one does it much more > smoothly. > I got these results from YCSB test. The test was doing approximately 5000 > inserts and 500 reads per second (for one region server). The results can be > further fine tuned and number of tuner operation can be reduced with these > changes in configuration. > For more fine tuning: > a) lower max step size (suggested = 4%) > b) lower min step size ( default if also fine ) > To further decrease frequency of tuning operations: > c) increase the number of lookup periods ( in the tests it was just 10, > default is 60 ) > d) increase tuner period ( in the tests it was just 20 secs, default is > 60secs) > I used smaller tuner period/ number of look up periods to get more data > points. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15106) Procedure V2 - Procedure Queue pass Procedure for better debuggability
[ https://issues.apache.org/jira/browse/HBASE-15106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107759#comment-15107759 ] Hudson commented on HBASE-15106: SUCCESS: Integrated in HBase-1.2-IT #401 (See [https://builds.apache.org/job/HBase-1.2-IT/401/]) HBASE-15106 Procedure v2 - Procedure Queue pass Procedure for better (matteo.bertozzi: rev 4fb7babad94d60f739667812802bbde5e8065c5e) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteColumnFamilyProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/AddColumnFamilyProcedure.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterProcedureScheduler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureScheduler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/TruncateTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyColumnFamilyProcedure.java > Procedure V2 - Procedure Queue pass Procedure for better debuggability > -- > > Key: HBASE-15106 > URL: https://issues.apache.org/jira/browse/HBASE-15106 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.3.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15106-v0.patch, HBASE-15106-v1.patch > > > Changes the various acquire/release methods to take the Procedure as argument. > That allows better debuggability. > (The patch it is just a refactor, it does not introduce any new thing) > https://reviews.apache.org/r/42271/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14221) Reduce the number of time row comparison is done in a Scan
[
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-14221:
--
Fix Version/s: (was: 1.0.4)
1.0.3
> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3
>
> Attachments: 14221-0.98-takeALook.txt, HBASE-14221-branch-1.patch,
> HBASE-14221.patch, HBASE-14221_1.patch, HBASE-14221_1.patch,
> HBASE-14221_6.patch, HBASE-14221_9.patch, withmatchingRowspatch.png,
> withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling with the PE tool found this.
> Currently we do row comparisons in 3 places in a simple Scan case.
> 1) ScanQueryMatcher
> {code}
>int ret = this.rowComparator.compareRows(curCell, cell);
> if (!this.isReversed) {
> if (ret <= -1) {
> return MatchCode.DONE;
> } else if (ret >= 1) {
> // could optimize this, if necessary?
> // Could also be called SEEK_TO_CURRENT_ROW, but this
> // should be rare/never happens.
> return MatchCode.SEEK_NEXT_ROW;
> }
> } else {
> if (ret <= -1) {
> return MatchCode.SEEK_NEXT_ROW;
> } else if (ret >= 1) {
> return MatchCode.DONE;
> }
> }
> {code}
> 2) In StoreScanner next() while starting to scan the row
> {code}
> if (!scannerContext.hasAnyLimit(LimitScope.BETWEEN_CELLS) ||
> matcher.curCell == null ||
> isNewRow || !CellUtil.matchingRow(peeked, matcher.curCell)) {
> this.countPerRow = 0;
> matcher.setToNewRow(peeked);
> }
> {code}
> Particularly to see if we are in a new row.
> 3) In HRegion
> {code}
> scannerContext.setKeepProgress(true);
> heap.next(results, scannerContext);
> scannerContext.setKeepProgress(tmpKeepProgress);
> nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> {code}
> Here again there are cases where we need to careful for a MultiCF case. Was
> trying to solve this for the MultiCF case but is having lot of cases to
> solve. But atleast for a single CF case I think these comparison can be
> reduced.
> So for a single CF case in the SQM we are able to find if we have crossed a
> row using the code pasted above in SQM. That comparison is definitely needed.
> Now in case of a single CF the HRegion is going to have only one element in
> the heap and so the 3rd comparison can surely be avoided if the
> StoreScanner.next() was over due to MatchCode.DONE caused by SQM.
> Coming to the 2nd compareRows that we do in StoreScanner. next() - even that
> can be avoided if we know that the previous next() call was over due to a new
> row. Doing all this I found that the compareRows in the profiler which was
> 19% got reduced to 13%. Initially we can solve for single CF case which can
> be extended to MultiCF cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15133) Data loss after compaction when a row has more than Integer.MAX_VALUE columns
Toshihiro Suzuki created HBASE-15133: Summary: Data loss after compaction when a row has more than Integer.MAX_VALUE columns Key: HBASE-15133 URL: https://issues.apache.org/jira/browse/HBASE-15133 Project: HBase Issue Type: Bug Components: Compaction Reporter: Toshihiro Suzuki We have lost the data in our development environment when a row has more than Integer.MAX_VALUE columns after compaction. I think the reason is type of StoreScanner's countPerRow is int. https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java#L67 After changing the type to long, it seems to be fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14058) Stabilizing default heap memory tuner
[ https://issues.apache.org/jira/browse/HBASE-14058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107903#comment-15107903 ] Hudson commented on HBASE-14058: FAILURE: Integrated in HBase-1.2 #512 (See [https://builds.apache.org/job/HBase-1.2/512/]) HBASE-14058 Stabilizing default heap memory tuner (eclark: rev e738e69f8cc59581a454207483aca42e7f314396) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultHeapMemoryTuner.java > Stabilizing default heap memory tuner > - > > Key: HBASE-14058 > URL: https://issues.apache.org/jira/browse/HBASE-14058 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Abhilash >Assignee: Abhilash > Fix For: 2.0.0, 1.3.0 > > Attachments: 0001-Stabilizing-default-heap-memory-tuner.patch, > HBASE-14058-v1.patch, HBASE-14058.patch, after_modifications.png, > before_modifications.png > > > The memory tuner works well in general cases but when we have a work load > that is both read heavy as well as write heavy the tuner does too many > tuning. We should try to control the number of tuner operation and stabilize > it. The main problem was that the tuner thinks it is in steady state even if > it sees just one neutral tuner period thus does too many tuning operations > and too many reverts that too with large step sizes(step size was set to > maximum even after one neutral period). So to stop this I have thought of > these steps: > 1) The division created by μ + δ/2 and μ - δ/2 is too small. Statistically > ~62% periods will lie outside this range, which means 62% of the data points > are considered either high or low which is too much. Use μ + δ*0.8 and μ - > δ*0.8 instead. On expectations it will decrease number of tuner operations > per 100 periods from 19 to just 10. If we use δ/2 then 31% of data values > will be considered to be high and 31% will be considered to be low (2*0.31 * > 0.31 = 0.19), on the other hand if we use δ*0.8 then 22% will be low and 22% > will be high(2*0.22*0.22 ~ 0.10). > 2) Defining proper steady state by looking at past few periods(it is equal to > hbase.regionserver.heapmemory.autotuner.lookup.periods) rather than just last > tuner operation. We say tuner is in steady state when last few tuner periods > were NEUTRAL. We keep decreasing step size unless it is extremely low. Then > leave system in that state for some time. > 3) Rather then decreasing step size only while reverting, decrease the > magnitude of step size whenever we are trying to revert tuning done in last > few periods(sum the changes of last few periods and compare to current step) > rather than just looking at last period. When its magnitude gets too low then > make tuner steps NEUTRAL(no operation). This will cause step size to > continuously decrease unless we reach steady state. After that tuning process > will restart (tuner step size rests again when we reach steady state). > 4) The tuning done in last few periods will be decaying sum of past tuner > steps with sign. This parameter will be positive for increase in memstore and > negative for increase in block cache. Rather than using arithmetic mean we > use this to give more priority to recent tuner steps. > Please see the attachments. One represents the size of memstore(green) and > size of block cache(blue) adjusted by tuner without these modification and > other with the above modifications. The x-axis is time axis and y-axis is the > fraction of heap memory available to memstore and block cache at that time(it > always sums up to 80%). I configured min/max ranges for both components to > 0.1 and 0.7 respectively(so in the plots the y-axis min and max is 0.1 and > 0.7). In both cases the tuner tries to distribute memory by giving ~15% to > memstore and ~65% to block cache. But the modified one does it much more > smoothly. > I got these results from YCSB test. The test was doing approximately 5000 > inserts and 500 reads per second (for one region server). The results can be > further fine tuned and number of tuner operation can be reduced with these > changes in configuration. > For more fine tuning: > a) lower max step size (suggested = 4%) > b) lower min step size ( default if also fine ) > To further decrease frequency of tuning operations: > c) increase the number of lookup periods ( in the tests it was just 10, > default is 60 ) > d) increase tuner period ( in the tests it was just 20 secs, default is > 60secs) > I used smaller tuner period/ number of look up periods to get more data > points. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15118) Fix findbugs complaint in hbase-server
[ https://issues.apache.org/jira/browse/HBASE-15118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107760#comment-15107760 ] Hudson commented on HBASE-15118: SUCCESS: Integrated in HBase-1.2-IT #401 (See [https://builds.apache.org/job/HBase-1.2-IT/401/]) HBASE-15118 Fix findbugs complaint in hbase-server; ADDENDUM TO FIX (stack: rev b9450687ce5a6ad524e7a9bca351bef43622f527) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java > Fix findbugs complaint in hbase-server > -- > > Key: HBASE-15118 > URL: https://issues.apache.org/jira/browse/HBASE-15118 > Project: HBase > Issue Type: Sub-task > Components: build >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15118.patch, 15118v2.patch, 15118v3.patch, 15118v4.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15102) HeapMemoryTuner can "overtune" memstore size and suddenly drop it into blocking zone
[ https://issues.apache.org/jira/browse/HBASE-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107758#comment-15107758 ] Hudson commented on HBASE-15102: SUCCESS: Integrated in HBase-1.2-IT #401 (See [https://builds.apache.org/job/HBase-1.2-IT/401/]) HBASE-15102 Fix HeapMemoryTuner overtuning memstore (eclark: rev e5953694750bd64f26a4bed927d095755b7dd105) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultHeapMemoryTuner.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHeapMemoryManager.java > HeapMemoryTuner can "overtune" memstore size and suddenly drop it into > blocking zone > > > Key: HBASE-15102 > URL: https://issues.apache.org/jira/browse/HBASE-15102 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15102-V0.patch, HBASE-15102-V1.patch, > HBASE-15102-V2.patch, HBASE-15102-V3.patch > > > DefaultHeapMemoryTuner often resets the maximum step size for tuning to 8% of > total heap size. Often, when the size of memstore is to be decreased while > tuning, the 8% tuning can suddenly drop the memstore size below the low water > mark of the previous memstore size (which could potentially be the used size > of the memstore) > This is problematic because suddenly it blocks all the updates by suddenly > causing a situation where memstore used size is above high water mark. This > has a very bad performance impact on an otherwise fine HBase cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15107) Procedure V2 - Procedure Queue with Regions
[
https://issues.apache.org/jira/browse/HBASE-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107794#comment-15107794
]
Hadoop QA commented on HBASE-15107:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 49s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 34s
{color} | {color:red} Patch generated 2 new checkstyle issues in hbase-server
(total was 22, now 24). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
21m 2s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 35s
{color} | {color:green} hbase-procedure in the patch passed with JDK v1.8.0.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 84m 12s
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 46s
{color} | {color:green} hbase-procedure in the patch passed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 82m 27s
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 218m 17s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12783170/HBASE-15107-v1.patch |
| JIRA Issue | HBASE-15107 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
[jira] [Commented] (HBASE-15093) Replication can report incorrect size of log queue for the global source when multiwal is enabled
[ https://issues.apache.org/jira/browse/HBASE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107836#comment-15107836 ] Enis Soztutar commented on HBASE-15093: --- The patch looks good to me. Is this something you've observed or theoretical? > Replication can report incorrect size of log queue for the global source when > multiwal is enabled > - > > Key: HBASE-15093 > URL: https://issues.apache.org/jira/browse/HBASE-15093 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.2.0, 1.2.1 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri >Priority: Minor > Attachments: HBASE-15093-V0.patch > > > Replication can report incorrect size for the size of log queue for the > global source when multiwal is enabled. This happens because the method > MetricsSource#setSizeofLogQueue performs non-trivial operations in a > multithreaded world, even though it is not synchronized. > We can simply divide MetricsSource#setSizeofLogQueue into > MetricsSource#incrSizeofLogQueue and MetricsSource#decrSizeofLogQueue. Not > sure why we are currently directly setting the size instead of > incrementing/decrementing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15126) HBaseFsck's checkRegionBoundaries function set the 'storesFirstKey' was incorrect.
[ https://issues.apache.org/jira/browse/HBASE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenrongwei updated HBASE-15126: Attachment: HBASE-15126-v004.patch Thanks for your reminder,the patch already had been reformatted. > HBaseFsck's checkRegionBoundaries function set the 'storesFirstKey' was > incorrect. > --- > > Key: HBASE-15126 > URL: https://issues.apache.org/jira/browse/HBASE-15126 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 2.0.0 >Reporter: chenrongwei >Assignee: chenrongwei > Attachments: HBASE-15126-v001.patch, HBASE-15126-v002.patch, > HBASE-15126-v003.patch, HBASE-15126-v004.patch > > > HBaseFsck's checkRegionBoundaries function set the > 'currentRegionBoundariesInformation.storesFirstKey' was incorrect.I think it > should be set like below, > currentRegionBoundariesInformation.storesFirstKey = keyOnly(storeFirstKey); > but current the 'currentRegionBoundariesInformation.storesFirstKey ' is just > set to 'storeFirstKey',which will cause to > comparator.compare(currentRegionBoundariesInformation.storesFirstKey, > currentRegionBoundariesInformation.metaFirstKey) get a wrong > result. Because it just compared the rowkey's length. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15093) Replication can report incorrect size of log queue for the global source when multiwal is enabled
[ https://issues.apache.org/jira/browse/HBASE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107888#comment-15107888 ] Enis Soztutar commented on HBASE-15093: --- bq. I saw this happening on a real production cluster. The size of global replication log queue is almost never sum of all queues and some times goes negative. Alright, was just curious. The failed tests seems relevant though. > Replication can report incorrect size of log queue for the global source when > multiwal is enabled > - > > Key: HBASE-15093 > URL: https://issues.apache.org/jira/browse/HBASE-15093 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.2.0, 1.2.1 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri >Priority: Minor > Attachments: HBASE-15093-V0.patch > > > Replication can report incorrect size for the size of log queue for the > global source when multiwal is enabled. This happens because the method > MetricsSource#setSizeofLogQueue performs non-trivial operations in a > multithreaded world, even though it is not synchronized. > We can simply divide MetricsSource#setSizeofLogQueue into > MetricsSource#incrSizeofLogQueue and MetricsSource#decrSizeofLogQueue. Not > sure why we are currently directly setting the size instead of > incrementing/decrementing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14865) Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
[
https://issues.apache.org/jira/browse/HBASE-14865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107914#comment-15107914
]
Hudson commented on HBASE-14865:
SUCCESS: Integrated in HBase-1.3-IT #444 (See
[https://builds.apache.org/job/HBase-1.3-IT/444/])
HBASE-14865 Support passing multiple QOPs to SaslClient/Server via (tedyu: rev
e76a2e4e6d91deee250d180b75b890f743da4bf0)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestAsyncSecureIPC.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestSecureIPC.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestHBaseSaslRpcClient.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcChannel.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestSecureRPC.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/security/AbstractTestSecureIPC.java
*
hbase-client/src/test/java/org/apache/hadoop/hbase/security/TestHBaseSaslRpcClient.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/security/SaslClientHandler.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/security/SaslUtil.java
* hbase-client/src/test/java/org/apache/hadoop/hbase/security/TestSaslUtil.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
> Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
> ---
>
> Key: HBASE-14865
> URL: https://issues.apache.org/jira/browse/HBASE-14865
> Project: HBase
> Issue Type: Improvement
> Components: security
>Reporter: Appy
>Assignee: Appy
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 14865-master-v7.patch, HBASE-14865-branch-1.2.patch,
> HBASE-14865-branch-1.patch, HBASE-14865-branch-1.patch,
> HBASE-14865-master-v2.patch, HBASE-14865-master-v3.patch,
> HBASE-14865-master-v4.patch, HBASE-14865-master-v5.patch,
> HBASE-14865-master-v6.patch, HBASE-14865-master-v7.patch,
> HBASE-14865-master.patch
>
>
> Currently, we can set the value of hbase.rpc.protection to one of
> authentication/integrity/privacy. It is the used to set
> {{javax.security.sasl.qop}} in SaslUtil.java.
> The problem is, if a cluster wants to switch from one qop to another, it'll
> have to take a downtime. Rolling upgrade will create a situation where some
> nodes have old value and some have new, which'll prevent any communication
> between them. There will be similar issue when clients will try to connect.
> {{javax.security.sasl.qop}} can take in a list of QOP in preferences order.
> So a transition from qop1 to qop2 can be easily done like this
> "qop1" --> "qop2,qop1" --> rolling restart --> "qop2" --> rolling restart
> Need to change hbase.rpc.protection to accept a list too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15106) Procedure V2 - Procedure Queue pass Procedure for better debuggability
[ https://issues.apache.org/jira/browse/HBASE-15106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107913#comment-15107913 ] Hudson commented on HBASE-15106: SUCCESS: Integrated in HBase-1.3-IT #444 (See [https://builds.apache.org/job/HBase-1.3-IT/444/]) HBASE-15106 Procedure v2 - Procedure Queue pass Procedure for better (matteo.bertozzi: rev 7d7a2d87127917b8b98a3e9fd308d1e853bbb8c0) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/AddColumnFamilyProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyColumnFamilyProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteColumnFamilyProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteNamespaceProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateNamespaceProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterProcedureScheduler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureScheduler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/TruncateTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyNamespaceProcedure.java > Procedure V2 - Procedure Queue pass Procedure for better debuggability > -- > > Key: HBASE-15106 > URL: https://issues.apache.org/jira/browse/HBASE-15106 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.3.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15106-v0.patch, HBASE-15106-v1.patch > > > Changes the various acquire/release methods to take the Procedure as argument. > That allows better debuggability. > (The patch it is just a refactor, it does not introduce any new thing) > https://reviews.apache.org/r/42271/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15102) HeapMemoryTuner can "overtune" memstore size and suddenly drop it into blocking zone
[ https://issues.apache.org/jira/browse/HBASE-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107912#comment-15107912 ] Hudson commented on HBASE-15102: SUCCESS: Integrated in HBase-1.3-IT #444 (See [https://builds.apache.org/job/HBase-1.3-IT/444/]) HBASE-15102 Fix HeapMemoryTuner overtuning memstore (eclark: rev a61f3ecc8b3dad256b7ec5258e1ec4aa811cb606) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHeapMemoryManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultHeapMemoryTuner.java > HeapMemoryTuner can "overtune" memstore size and suddenly drop it into > blocking zone > > > Key: HBASE-15102 > URL: https://issues.apache.org/jira/browse/HBASE-15102 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15102-V0.patch, HBASE-15102-V1.patch, > HBASE-15102-V2.patch, HBASE-15102-V3.patch > > > DefaultHeapMemoryTuner often resets the maximum step size for tuning to 8% of > total heap size. Often, when the size of memstore is to be decreased while > tuning, the 8% tuning can suddenly drop the memstore size below the low water > mark of the previous memstore size (which could potentially be the used size > of the memstore) > This is problematic because suddenly it blocks all the updates by suddenly > causing a situation where memstore used size is above high water mark. This > has a very bad performance impact on an otherwise fine HBase cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15106) Procedure V2 - Procedure Queue pass Procedure for better debuggability
[ https://issues.apache.org/jira/browse/HBASE-15106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107791#comment-15107791 ] Hudson commented on HBASE-15106: FAILURE: Integrated in HBase-1.3 #501 (See [https://builds.apache.org/job/HBase-1.3/501/]) HBASE-15106 Procedure v2 - Procedure Queue pass Procedure for better (matteo.bertozzi: rev 7d7a2d87127917b8b98a3e9fd308d1e853bbb8c0) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterProcedureScheduler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyNamespaceProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteColumnFamilyProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyColumnFamilyProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateNamespaceProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteNamespaceProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureScheduler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/AddColumnFamilyProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/TruncateTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java > Procedure V2 - Procedure Queue pass Procedure for better debuggability > -- > > Key: HBASE-15106 > URL: https://issues.apache.org/jira/browse/HBASE-15106 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.3.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15106-v0.patch, HBASE-15106-v1.patch > > > Changes the various acquire/release methods to take the Procedure as argument. > That allows better debuggability. > (The patch it is just a refactor, it does not introduce any new thing) > https://reviews.apache.org/r/42271/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14457) Umbrella: Improve Multiple WAL for production usage
[ https://issues.apache.org/jira/browse/HBASE-14457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107843#comment-15107843 ] Enis Soztutar commented on HBASE-14457: --- See HBASE-15131. > Umbrella: Improve Multiple WAL for production usage > --- > > Key: HBASE-14457 > URL: https://issues.apache.org/jira/browse/HBASE-14457 > Project: HBase > Issue Type: Umbrella >Reporter: Yu Li >Assignee: Yu Li > Fix For: 2.0.0, 1.3.0 > > Attachments: Action in Multiple WAL.pdf, Action in Multiple WAL.pdf > > > HBASE-5699 proposed the idea to run with multiple WAL in regionserver and did > a great initial work there, but when trying to use it in our production > cluster, we still found several issues to resolve, like tracking multiple WAL > paths in replication (HBASE-6617), fixing UT with multiwal provider > (HBASE-14411), introducing a namespace-based strategy for > RegionGroupingProvider (HBASE-14456), etc. This is an umbrella including(but > not limited of) all these works and efforts to make multiple wal ready for > production usage and give user a clear picture about it. > Besides the developing works done, I'd also like to share some scenarios and > testing/online data in this JIRA about our usage/performance of multiple wal, > to(hopefully) help people better judge whether to enable multiple wal or not > in their own cluster and what they might gain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15097) When the scan operation covered two regions,sometimes the final results have duplicated rows.
[
https://issues.apache.org/jira/browse/HBASE-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenrongwei updated HBASE-15097:
Attachment: HBASE-15097-v004.patch
Reformatted the patch.
> When the scan operation covered two regions,sometimes the final results have
> duplicated rows.
> -
>
> Key: HBASE-15097
> URL: https://issues.apache.org/jira/browse/HBASE-15097
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: centos 6.5
> hbase 1.1.2
>Reporter: chenrongwei
>Assignee: chenrongwei
> Attachments: HBASE-15097-v001.patch, HBASE-15097-v002.patch,
> HBASE-15097-v003.patch, HBASE-15097-v004.patch, output.log, rowkey.txt,
> snapshot2016-01-13 pm 8.42.37.png
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> When the scan operation‘s start key and end key covered two regions,the first
> region returned the rows which were beyond of its' end key.So,this finally
> leads to duplicated rows in the results.
> To avoid this problem,we should add a judgment before setting the variable
> "stopRow" in the class of HRegion,like follow:
> if (Bytes.equals(scan.getStopRow(), HConstants.EMPTY_END_ROW) &&
> !scan.isGetScan()) {
> this.stopRow = null;
> } else {
> if (Bytes.compareTo(scan.getStopRow(),
> this.getRegionInfo().getEndKey()) >= 0) {
> this.stopRow = this.getRegionInfo().getEndKey();
> } else {
> this.stopRow = scan.getStopRow();
> }
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15132) Master region merge RPC should authorize user request
Ted Yu created HBASE-15132: -- Summary: Master region merge RPC should authorize user request Key: HBASE-15132 URL: https://issues.apache.org/jira/browse/HBASE-15132 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu The normal flow for region merge is: 1. client sends a master RPC for dispatch merge regions 2. master moves the regions to the same regionserver 3. master calls mergeRegions RPC on the regionserver. For user initiated region merge, MasterRpcServices#dispatchMergingRegions() is called by HBaseAdmin. There is no coprocessor invocation in step 1. Step 3 is carried out in the "hbase" user context. This leaves potential security hole - any user without proper authorization can merge regions of any table. Thanks to Enis who spotted this flaw first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15093) Replication can report incorrect size of log queue for the global source when multiwal is enabled
[ https://issues.apache.org/jira/browse/HBASE-15093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107881#comment-15107881 ] Ashu Pachauri commented on HBASE-15093: --- I saw this happening on a real production cluster. The size of global replication log queue is almost never sum of all queues and some times goes negative. > Replication can report incorrect size of log queue for the global source when > multiwal is enabled > - > > Key: HBASE-15093 > URL: https://issues.apache.org/jira/browse/HBASE-15093 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.2.0, 1.2.1 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri >Priority: Minor > Attachments: HBASE-15093-V0.patch > > > Replication can report incorrect size for the size of log queue for the > global source when multiwal is enabled. This happens because the method > MetricsSource#setSizeofLogQueue performs non-trivial operations in a > multithreaded world, even though it is not synchronized. > We can simply divide MetricsSource#setSizeofLogQueue into > MetricsSource#incrSizeofLogQueue and MetricsSource#decrSizeofLogQueue. Not > sure why we are currently directly setting the size instead of > incrementing/decrementing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15100) Master WALProcs still never clean up
[
https://issues.apache.org/jira/browse/HBASE-15100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-15100:
Priority: Blocker (was: Critical)
> Master WALProcs still never clean up
>
>
> Key: HBASE-15100
> URL: https://issues.apache.org/jira/browse/HBASE-15100
> Project: HBase
> Issue Type: Bug
> Components: master, proc-v2
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Matteo Bertozzi
>Priority: Blocker
> Attachments: HBASE-15100-v0.patch, TestWalProcedure.java
>
>
> {code}
> bin/hdfs dfs -ls /hbase/MasterProcWALs | wc -l
> 218631
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15073) Finer grained control over normalization actions for RegionNormalizer
[
https://issues.apache.org/jira/browse/HBASE-15073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107767#comment-15107767
]
Ted Yu commented on HBASE-15073:
I am doing more homework and regathering my thoughts so that I can objectively
evaluate this initiative.
This involves understanding particular use case (AMS e.g.) more deeply.
Hopefully I can come back with better answer to review comments.
> Finer grained control over normalization actions for RegionNormalizer
> -
>
> Key: HBASE-15073
> URL: https://issues.apache.org/jira/browse/HBASE-15073
> Project: HBase
> Issue Type: Task
> Components: regionserver
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15073-v1.txt, 15073-v2.txt, 15073-v2.txt, 15073-v3.txt,
> 15073-v4.txt, 15073-v5.txt
>
>
> Currently both region split and merge actions are carried out during
> normalization for underlying table.
> However, for certain use case(s) (see
> https://issues.apache.org/jira/browse/HBASE-13103?focusedCommentId=14366255=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14366255),
> it would be desirable to perform only one type of action.
> There is one boolean flag, keyed by NORMALIZATION_ENABLED_KEY, per table that
> enables normalization.
> To provide finer grained control, we have several options:
> 1. introduce another per table flag to indicate which type(s) of actions are
> allowed ("N" for disabled, "S" for split only, "M" for merge only and "MS"
> for both split and merge)
> 2. introduce another global flag to indicate which type(s) of actions are
> allowed
> 3. modify the meaning of existing flag keyed by NORMALIZATION_ENABLED_KEY so
> that it indicates type(s) of actions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15132) Master region merge RPC should authorize user request
[ https://issues.apache.org/jira/browse/HBASE-15132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107873#comment-15107873 ] Ted Yu commented on HBASE-15132: The preMerge() hook is currently executed on region server. One solution is to enhance MasterObserver with preMerge() hook. MasterRpcServices#dispatchMergingRegions() can execute preMerge() hook in the context of request user. Comment is welcome. > Master region merge RPC should authorize user request > - > > Key: HBASE-15132 > URL: https://issues.apache.org/jira/browse/HBASE-15132 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > > The normal flow for region merge is: > 1. client sends a master RPC for dispatch merge regions > 2. master moves the regions to the same regionserver > 3. master calls mergeRegions RPC on the regionserver. > For user initiated region merge, MasterRpcServices#dispatchMergingRegions() > is called by HBaseAdmin. > There is no coprocessor invocation in step 1. > Step 3 is carried out in the "hbase" user context. > This leaves potential security hole - any user without proper authorization > can merge regions of any table. > Thanks to Enis who spotted this flaw first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15102) HeapMemoryTuner can "overtune" memstore size and suddenly drop it into blocking zone
[ https://issues.apache.org/jira/browse/HBASE-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107904#comment-15107904 ] Hudson commented on HBASE-15102: FAILURE: Integrated in HBase-1.2 #512 (See [https://builds.apache.org/job/HBase-1.2/512/]) HBASE-15102 Fix HeapMemoryTuner overtuning memstore (eclark: rev e5953694750bd64f26a4bed927d095755b7dd105) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultHeapMemoryTuner.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHeapMemoryManager.java > HeapMemoryTuner can "overtune" memstore size and suddenly drop it into > blocking zone > > > Key: HBASE-15102 > URL: https://issues.apache.org/jira/browse/HBASE-15102 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15102-V0.patch, HBASE-15102-V1.patch, > HBASE-15102-V2.patch, HBASE-15102-V3.patch > > > DefaultHeapMemoryTuner often resets the maximum step size for tuning to 8% of > total heap size. Often, when the size of memstore is to be decreased while > tuning, the 8% tuning can suddenly drop the memstore size below the low water > mark of the previous memstore size (which could potentially be the used size > of the memstore) > This is problematic because suddenly it blocks all the updates by suddenly > causing a situation where memstore used size is above high water mark. This > has a very bad performance impact on an otherwise fine HBase cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15118) Fix findbugs complaint in hbase-server
[ https://issues.apache.org/jira/browse/HBASE-15118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107906#comment-15107906 ] Hudson commented on HBASE-15118: FAILURE: Integrated in HBase-1.2 #512 (See [https://builds.apache.org/job/HBase-1.2/512/]) HBASE-15118 Fix findbugs complaint in hbase-server; ADDENDUM TO FIX (stack: rev b9450687ce5a6ad524e7a9bca351bef43622f527) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java > Fix findbugs complaint in hbase-server > -- > > Key: HBASE-15118 > URL: https://issues.apache.org/jira/browse/HBASE-15118 > Project: HBase > Issue Type: Sub-task > Components: build >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15118.patch, 15118v2.patch, 15118v3.patch, 15118v4.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15106) Procedure V2 - Procedure Queue pass Procedure for better debuggability
[ https://issues.apache.org/jira/browse/HBASE-15106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107905#comment-15107905 ] Hudson commented on HBASE-15106: FAILURE: Integrated in HBase-1.2 #512 (See [https://builds.apache.org/job/HBase-1.2/512/]) HBASE-15106 Procedure v2 - Procedure Queue pass Procedure for better (matteo.bertozzi: rev 4fb7babad94d60f739667812802bbde5e8065c5e) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyColumnFamilyProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/CreateTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/AddColumnFamilyProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/TruncateTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteColumnFamilyProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureScheduler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterProcedureScheduler.java > Procedure V2 - Procedure Queue pass Procedure for better debuggability > -- > > Key: HBASE-15106 > URL: https://issues.apache.org/jira/browse/HBASE-15106 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.3.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15106-v0.patch, HBASE-15106-v1.patch > > > Changes the various acquire/release methods to take the Procedure as argument. > That allows better debuggability. > (The patch it is just a refactor, it does not introduce any new thing) > https://reviews.apache.org/r/42271/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15073) Finer grained control over normalization actions for RegionNormalizer
[
https://issues.apache.org/jira/browse/HBASE-15073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107901#comment-15107901
]
Mikhail Antonov commented on HBASE-15073:
-
I don't know how many folks are running branch-1.2 builds under any realistic
workloads, so I guess until 1.2 is out and people start to actually use it, we
are unlikely to see reports from live clusters with highlights of shortcomings
of current impl. So I think at the moment the implemented (and proposed)
heuristics are assumptions, not battle-tested. Which kind of makes me wish to
make it on by default in 1.2..
> Finer grained control over normalization actions for RegionNormalizer
> -
>
> Key: HBASE-15073
> URL: https://issues.apache.org/jira/browse/HBASE-15073
> Project: HBase
> Issue Type: Task
> Components: regionserver
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15073-v1.txt, 15073-v2.txt, 15073-v2.txt, 15073-v3.txt,
> 15073-v4.txt, 15073-v5.txt
>
>
> Currently both region split and merge actions are carried out during
> normalization for underlying table.
> However, for certain use case(s) (see
> https://issues.apache.org/jira/browse/HBASE-13103?focusedCommentId=14366255=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14366255),
> it would be desirable to perform only one type of action.
> There is one boolean flag, keyed by NORMALIZATION_ENABLED_KEY, per table that
> enables normalization.
> To provide finer grained control, we have several options:
> 1. introduce another per table flag to indicate which type(s) of actions are
> allowed ("N" for disabled, "S" for split only, "M" for merge only and "MS"
> for both split and merge)
> 2. introduce another global flag to indicate which type(s) of actions are
> allowed
> 3. modify the meaning of existing flag keyed by NORMALIZATION_ENABLED_KEY so
> that it indicates type(s) of actions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15125) HBaseFsck's adoptHdfsOrphan function create region with wrong end key boundary.
[ https://issues.apache.org/jira/browse/HBASE-15125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenrongwei updated HBASE-15125: Attachment: HBASE-15125-v003.patch Thanks for your reminder,it's done now. > HBaseFsck's adoptHdfsOrphan function create region with wrong end key > boundary. > --- > > Key: HBASE-15125 > URL: https://issues.apache.org/jira/browse/HBASE-15125 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 2.0.0 >Reporter: chenrongwei >Assignee: chenrongwei > Attachments: HBASE-15125-V001.patch, HBASE-15125-v002.patch, > HBASE-15125-v003.patch > > > There is a bug in HBaseFsck's adoptHdfsOrphan function.At the last of this > function will create a region,which want to cover all the orphan regions.But > the end key of this new region was set incorrectly.Correct region's boundary > should be [startKey,endKey),but this function create a region with boundary > of [startKey,endKey],this bug will leads to scan operation omit some data. > I think we should create the region like bellow, > // create new region on hdfs. move data into place. > HRegionInfo hri = new HRegionInfo(template.getTableName(), > orphanRegionRange.getFirst(), > Bytes.add(orphanRegionRange.getSecond(), new byte[1])); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15131) Turn on multi-WAL by default
[ https://issues.apache.org/jira/browse/HBASE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107862#comment-15107862 ] Sean Busbey commented on HBASE-15131: - I'd like to see some perf numbers on a shipped version of HBase before we make this the default. IIRC, the benchmarks from HBASE-14457 used a custom backport on top of 0.98. I have some numbers from a small test cluster, but I'm not sure how useful such a small scale is. > Turn on multi-WAL by default > > > Key: HBASE-15131 > URL: https://issues.apache.org/jira/browse/HBASE-15131 > Project: HBase > Issue Type: New Feature > Components: wal >Reporter: Enis Soztutar > Fix For: 2.0.0, 1.3.0 > > > Something to discuss for 2.0 or even 1.3 or 1.4. Should we turn on multi-WAL > by default now that it has seen some production use. > Most of the known issues has been fixed I believe for replication, metrics > etc. See HBASE-14457. > Using bounded provider with default 4 (assuming 12 disks). Should we also > look at the number of disks from datanode dirs and auto-configure? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15131) Turn on multi-WAL by default
[ https://issues.apache.org/jira/browse/HBASE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-15131: Component/s: wal > Turn on multi-WAL by default > > > Key: HBASE-15131 > URL: https://issues.apache.org/jira/browse/HBASE-15131 > Project: HBase > Issue Type: New Feature > Components: wal >Reporter: Enis Soztutar > Fix For: 2.0.0, 1.3.0 > > > Something to discuss for 2.0 or even 1.3 or 1.4. Should we turn on multi-WAL > by default now that it has seen some production use. > Most of the known issues has been fixed I believe for replication, metrics > etc. See HBASE-14457. > Using bounded provider with default 4 (assuming 12 disks). Should we also > look at the number of disks from datanode dirs and auto-configure? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15133) Data loss after compaction when a row has more than Integer.MAX_VALUE columns
[ https://issues.apache.org/jira/browse/HBASE-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Toshihiro Suzuki updated HBASE-15133: - Attachment: master.patch I attached the patch. > Data loss after compaction when a row has more than Integer.MAX_VALUE columns > - > > Key: HBASE-15133 > URL: https://issues.apache.org/jira/browse/HBASE-15133 > Project: HBase > Issue Type: Bug > Components: Compaction >Reporter: Toshihiro Suzuki > Attachments: master.patch > > > We have lost the data in our development environment when a row has more than > Integer.MAX_VALUE columns after compaction. > I think the reason is type of StoreScanner's countPerRow is int. > https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java#L67 > After changing the type to long, it seems to be fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15131) Turn on multi-WAL by default
[ https://issues.apache.org/jira/browse/HBASE-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107923#comment-15107923 ] Yu Li commented on HBASE-15131: --- bq. IIRC, the benchmarks from HBASE-14457 used a custom backport on top of 0.98. You are right [~busbey]. But recently we're preparing to upgrade our cluster to 1.1.2 and have just done a comparison test between 0.98.12 and 1.1.2 with 4 WALs, the result shows a matchable performance, JFYI. And [~eclark] may have more experience of using multiple WAL in facebook. [~eclark] could you share with us? Many thanks :-) > Turn on multi-WAL by default > > > Key: HBASE-15131 > URL: https://issues.apache.org/jira/browse/HBASE-15131 > Project: HBase > Issue Type: New Feature > Components: wal >Reporter: Enis Soztutar > Fix For: 2.0.0, 1.3.0 > > > Something to discuss for 2.0 or even 1.3 or 1.4. Should we turn on multi-WAL > by default now that it has seen some production use. > Most of the known issues has been fixed I believe for replication, metrics > etc. See HBASE-14457. > Using bounded provider with default 4 (assuming 12 disks). Should we also > look at the number of disks from datanode dirs and auto-configure? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-15106) Procedure V2 - Procedure Queue pass Procedure for better debuggability
[ https://issues.apache.org/jira/browse/HBASE-15106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biao Chen reassigned HBASE-15106: - Assignee: Biao Chen (was: Matteo Bertozzi) > Procedure V2 - Procedure Queue pass Procedure for better debuggability > -- > > Key: HBASE-15106 > URL: https://issues.apache.org/jira/browse/HBASE-15106 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.3.0 >Reporter: Matteo Bertozzi >Assignee: Biao Chen > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-15106-v0.patch, HBASE-15106-v1.patch > > > Changes the various acquire/release methods to take the Procedure as argument. > That allows better debuggability. > (The patch it is just a refactor, it does not introduce any new thing) > https://reviews.apache.org/r/42271/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14902) Revert some of the stringency recently introduced by checkstyle tightening
[ https://issues.apache.org/jira/browse/HBASE-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107937#comment-15107937 ] Hudson commented on HBASE-14902: FAILURE: Integrated in HBase-Trunk_matrix #643 (See [https://builds.apache.org/job/HBase-Trunk_matrix/643/]) HBASE-14902 Revert some of the stringency recently introduced by (stack: rev 2c0394f078158ef668e75b74f589a7da59ff9e0e) * hbase-checkstyle/src/main/resources/hbase/checkstyle.xml > Revert some of the stringency recently introduced by checkstyle tightening > -- > > Key: HBASE-14902 > URL: https://issues.apache.org/jira/browse/HBASE-14902 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: 14902.part1.patch, 14902.patch, 14902.patch, braces.patch > > > I think we should undo some of the plugins that were recently added to > checkstyle. They are too much. > JavadocTagContinuationIndentationCheck is about adding indent if javadoc is > two lines or more (javadoc tool doesn't care) > NonEmptyAtclauseDescriptionCheck would have us add javadoc on each exception: > e.g. @throws IOException needs to have text added. > NeedBracesCheck has us undoing cases where an if fits all on one line (don't > want to start style wars but if short and fits on one line, I think its more > readable... but I could relent on this one ). > The first two at least should go. > You ok w/ that [~appy] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13960) HConnection stuck with UnknownHostException
[
https://issues.apache.org/jira/browse/HBASE-13960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107970#comment-15107970
]
Yu Li commented on HBASE-13960:
---
The findbugs issue is not introduced by this JIRA, but the ut failure is caused
by the changes here. Will update the patch to fix it.
> HConnection stuck with UnknownHostException
>
>
> Key: HBASE-13960
> URL: https://issues.apache.org/jira/browse/HBASE-13960
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 0.98.8
>Reporter: Kurt Young
> Attachments: HBASE-13960-0.98-v1.patch, HBASE-13960-update.patch,
> HBASE-13960-v2.patch
>
>
> when put/get from hbase, if we meet a temporary dns failure causes resolve
> RS's host, the error will never recovered. put/get will failed with
> UnknownHostException forever.
> I checked the code, and the reason maybe:
> 1. when RegionServerCallable or MultiServerCallable prepare(), it gets a
> ClientService.BlockingInterface stub from Hconnection
> 2. In HConnectionImplementation::getClient, it caches the stub with a
> BlockingRpcChannelImplementation
> 3. In BlockingRpcChannelImplementation(),
> this.isa = new InetSocketAddress(sn.getHostname(), sn.getPort()); If we
> meet a temporary dns failure then the "address" in isa will be null.
> 4. then we launch the real rpc call, the following stack is:
> Caused by: java.net.UnknownHostException: unknown host: xxx.host2
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.(RpcClient.java:385)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.createConnection(RpcClient.java:351)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1523)
> at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1435)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1654)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1712)
> Besides, i noticed there is a protection in RpcClient:
> if (remoteId.getAddress().isUnresolved()) {
> throw new UnknownHostException("unknown host: " +
> remoteId.getAddress().getHostName());
> }
> shouldn't we do something when this situation occurred?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15101) Leaked References to StoreFile.Reader after HBASE-13082
[ https://issues.apache.org/jira/browse/HBASE-15101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15101: --- Priority: Critical (was: Major) > Leaked References to StoreFile.Reader after HBASE-13082 > --- > > Key: HBASE-15101 > URL: https://issues.apache.org/jira/browse/HBASE-15101 > Project: HBase > Issue Type: Bug > Components: HFile, io >Affects Versions: 2.0.0 >Reporter: deepankar >Assignee: deepankar >Priority: Critical > Attachments: HBASE-15101-v1.patch, HBASE-15101-v2.patch, > HBASE-15101-v3.patch, HBASE-15101.patch > > > We observed this production that after a region server dies there are huge > number of hfiles in that region for the region server running the version > with HBASE-13082, In the doc it is given that it is expected to happen, but > we found a one place where scanners are not being closed. If the scanners are > not closed their references are not decremented and that is leading to the > issue of huge number of store files not being finalized > All I was able to find is in the selectScannersFrom, where we discard some of > the scanners and we are not closing them. I am attaching a patch for that. > Also to avoid these issues should the files that are done be logged and > finalized (moved to archive) as a part of region close operation. This will > solve any leaks that can happen and does not cause any dire consequences? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15101) Leaked References to StoreFile.Reader after HBASE-13082
[ https://issues.apache.org/jira/browse/HBASE-15101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15108007#comment-15108007 ] ramkrishna.s.vasudevan commented on HBASE-15101: Will commit this patch to trunk and wait for further analysis in subsequent JIRAs? [~dvdreddy] - What do you say? > Leaked References to StoreFile.Reader after HBASE-13082 > --- > > Key: HBASE-15101 > URL: https://issues.apache.org/jira/browse/HBASE-15101 > Project: HBase > Issue Type: Bug > Components: HFile, io >Affects Versions: 2.0.0 >Reporter: deepankar >Assignee: deepankar >Priority: Critical > Attachments: HBASE-15101-v1.patch, HBASE-15101-v2.patch, > HBASE-15101-v3.patch, HBASE-15101.patch > > > We observed this production that after a region server dies there are huge > number of hfiles in that region for the region server running the version > with HBASE-13082, In the doc it is given that it is expected to happen, but > we found a one place where scanners are not being closed. If the scanners are > not closed their references are not decremented and that is leading to the > issue of huge number of store files not being finalized > All I was able to find is in the selectScannersFrom, where we discard some of > the scanners and we are not closing them. I am attaching a patch for that. > Also to avoid these issues should the files that are done be logged and > finalized (moved to archive) as a part of region close operation. This will > solve any leaks that can happen and does not cause any dire consequences? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15101) Leaked References to StoreFile.Reader after HBASE-13082
[ https://issues.apache.org/jira/browse/HBASE-15101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] deepankar updated HBASE-15101: -- Attachment: HBASE-15101-v4.patch > Leaked References to StoreFile.Reader after HBASE-13082 > --- > > Key: HBASE-15101 > URL: https://issues.apache.org/jira/browse/HBASE-15101 > Project: HBase > Issue Type: Bug > Components: HFile, io >Affects Versions: 2.0.0 >Reporter: deepankar >Assignee: deepankar >Priority: Critical > Attachments: HBASE-15101-v1.patch, HBASE-15101-v2.patch, > HBASE-15101-v3.patch, HBASE-15101-v4.patch, HBASE-15101.patch > > > We observed this production that after a region server dies there are huge > number of hfiles in that region for the region server running the version > with HBASE-13082, In the doc it is given that it is expected to happen, but > we found a one place where scanners are not being closed. If the scanners are > not closed their references are not decremented and that is leading to the > issue of huge number of store files not being finalized > All I was able to find is in the selectScannersFrom, where we discard some of > the scanners and we are not closing them. I am attaching a patch for that. > Also to avoid these issues should the files that are done be logged and > finalized (moved to archive) as a part of region close operation. This will > solve any leaks that can happen and does not cause any dire consequences? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15101) Leaked References to StoreFile.Reader after HBASE-13082
[ https://issues.apache.org/jira/browse/HBASE-15101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15108017#comment-15108017 ] deepankar commented on HBASE-15101: --- Attached patch with close calls also. > Leaked References to StoreFile.Reader after HBASE-13082 > --- > > Key: HBASE-15101 > URL: https://issues.apache.org/jira/browse/HBASE-15101 > Project: HBase > Issue Type: Bug > Components: HFile, io >Affects Versions: 2.0.0 >Reporter: deepankar >Assignee: deepankar >Priority: Critical > Attachments: HBASE-15101-v1.patch, HBASE-15101-v2.patch, > HBASE-15101-v3.patch, HBASE-15101-v4.patch, HBASE-15101.patch > > > We observed this production that after a region server dies there are huge > number of hfiles in that region for the region server running the version > with HBASE-13082, In the doc it is given that it is expected to happen, but > we found a one place where scanners are not being closed. If the scanners are > not closed their references are not decremented and that is leading to the > issue of huge number of store files not being finalized > All I was able to find is in the selectScannersFrom, where we discard some of > the scanners and we are not closing them. I am attaching a patch for that. > Also to avoid these issues should the files that are done be logged and > finalized (moved to archive) as a part of region close operation. This will > solve any leaks that can happen and does not cause any dire consequences? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15111) "hbase version" should write to stdout
[
https://issues.apache.org/jira/browse/HBASE-15111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15108045#comment-15108045
]
Hudson commented on HBASE-15111:
FAILURE: Integrated in HBase-Trunk_matrix #644 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/644/])
HBASE-15111 hbase version should write to stdout (busbey: rev
df36178062dfa5146c3e8bb14d5a15c46f222119)
* hbase-common/src/main/java/org/apache/hadoop/hbase/util/VersionInfo.java
> "hbase version" should write to stdout
> --
>
> Key: HBASE-15111
> URL: https://issues.apache.org/jira/browse/HBASE-15111
> Project: HBase
> Issue Type: Improvement
> Components: util
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Trivial
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: hbase-15111-v1.patch
>
>
> Calling {{hbase version}} currently outputs the version info by writing to
> {{LOG.info}}. This means, if you change the default log level settings, you
> may get no output at all on the command line.
> Since {{VersionInfo.main()}} is being called, it should really just output
> straight to stdout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15101) Leaked References to StoreFile.Reader after HBASE-13082
[ https://issues.apache.org/jira/browse/HBASE-15101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15108061#comment-15108061 ] ramkrishna.s.vasudevan commented on HBASE-15101: bq. I think we need some cleanup in these area. In my previous patches with HBASE-13082 I added a TODO to clean it up. We can check that once and do the need ful cleanup. > Leaked References to StoreFile.Reader after HBASE-13082 > --- > > Key: HBASE-15101 > URL: https://issues.apache.org/jira/browse/HBASE-15101 > Project: HBase > Issue Type: Bug > Components: HFile, io >Affects Versions: 2.0.0 >Reporter: deepankar >Assignee: deepankar >Priority: Critical > Attachments: HBASE-15101-v1.patch, HBASE-15101-v2.patch, > HBASE-15101-v3.patch, HBASE-15101-v4.patch, HBASE-15101.patch > > > We observed this production that after a region server dies there are huge > number of hfiles in that region for the region server running the version > with HBASE-13082, In the doc it is given that it is expected to happen, but > we found a one place where scanners are not being closed. If the scanners are > not closed their references are not decremented and that is leading to the > issue of huge number of store files not being finalized > All I was able to find is in the selectScannersFrom, where we discard some of > the scanners and we are not closing them. I am attaching a patch for that. > Also to avoid these issues should the files that are done be logged and > finalized (moved to archive) as a part of region close operation. This will > solve any leaks that can happen and does not cause any dire consequences? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15101) Leaked References to StoreFile.Reader after HBASE-13082
[ https://issues.apache.org/jira/browse/HBASE-15101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15108063#comment-15108063 ] ramkrishna.s.vasudevan commented on HBASE-15101: bq.Adding those extra close(false) might not be really needed Ya no problem. It is only for consistency with other code changes. Will definitely not impact this JIRA. But the other changes where the unused scanners are getting closed may be helpful. > Leaked References to StoreFile.Reader after HBASE-13082 > --- > > Key: HBASE-15101 > URL: https://issues.apache.org/jira/browse/HBASE-15101 > Project: HBase > Issue Type: Bug > Components: HFile, io >Affects Versions: 2.0.0 >Reporter: deepankar >Assignee: deepankar >Priority: Critical > Attachments: HBASE-15101-v1.patch, HBASE-15101-v2.patch, > HBASE-15101-v3.patch, HBASE-15101-v4.patch, HBASE-15101.patch > > > We observed this production that after a region server dies there are huge > number of hfiles in that region for the region server running the version > with HBASE-13082, In the doc it is given that it is expected to happen, but > we found a one place where scanners are not being closed. If the scanners are > not closed their references are not decremented and that is leading to the > issue of huge number of store files not being finalized > All I was able to find is in the selectScannersFrom, where we discard some of > the scanners and we are not closing them. I am attaching a patch for that. > Also to avoid these issues should the files that are done be logged and > finalized (moved to archive) as a part of region close operation. This will > solve any leaks that can happen and does not cause any dire consequences? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14872) Scan different timeRange per column family doesn't percolate down to the memstore
[ https://issues.apache.org/jira/browse/HBASE-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15108066#comment-15108066 ] Sean Busbey commented on HBASE-14872: - so is this fine to close for 1.2+? > Scan different timeRange per column family doesn't percolate down to the > memstore > -- > > Key: HBASE-14872 > URL: https://issues.apache.org/jira/browse/HBASE-14872 > Project: HBase > Issue Type: Bug > Components: Client, regionserver, Scanners >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 1.2.0, 0.98.18 > > Attachments: HBASE-14872-0.98-v1.patch, HBASE-14872-0.98.patch, > HBASE-14872-v1.patch, HBASE-14872.patch > > > HBASE-14355 The scan different time range for column family feature was not > applied to the memstore it was only done for the store files. This breaks > the contract. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15130) Backport 0.98 Scan different TimeRange for each column family
[ https://issues.apache.org/jira/browse/HBASE-15130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-15130: Component/s: Scanners regionserver Client > Backport 0.98 Scan different TimeRange for each column family > -- > > Key: HBASE-15130 > URL: https://issues.apache.org/jira/browse/HBASE-15130 > Project: HBase > Issue Type: Bug > Components: Client, regionserver, Scanners >Affects Versions: 0.98.17 >Reporter: churro morales >Assignee: churro morales > Fix For: 0.98.18 > > Attachments: HBASE-15130-0.98.patch > > > branch 98 version backport for HBASE-14355 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15101) Leaked References to StoreFile.Reader after HBASE-13082
[ https://issues.apache.org/jira/browse/HBASE-15101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15108080#comment-15108080 ] ramkrishna.s.vasudevan commented on HBASE-15101: Will discuss and further check with [~dvdreddy] if there is still leak and will raise further JIRAs to track down the issue if it still exists in their branch. > Leaked References to StoreFile.Reader after HBASE-13082 > --- > > Key: HBASE-15101 > URL: https://issues.apache.org/jira/browse/HBASE-15101 > Project: HBase > Issue Type: Bug > Components: HFile, io >Affects Versions: 2.0.0 >Reporter: deepankar >Assignee: deepankar >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-15101-v1.patch, HBASE-15101-v2.patch, > HBASE-15101-v3.patch, HBASE-15101-v4.patch, HBASE-15101.patch > > > We observed this production that after a region server dies there are huge > number of hfiles in that region for the region server running the version > with HBASE-13082, In the doc it is given that it is expected to happen, but > we found a one place where scanners are not being closed. If the scanners are > not closed their references are not decremented and that is leading to the > issue of huge number of store files not being finalized > All I was able to find is in the selectScannersFrom, where we discard some of > the scanners and we are not closing them. I am attaching a patch for that. > Also to avoid these issues should the files that are done be logged and > finalized (moved to archive) as a part of region close operation. This will > solve any leaks that can happen and does not cause any dire consequences? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15098) Normalizer switch in configuration is not used
[
https://issues.apache.org/jira/browse/HBASE-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-15098:
Status: In Progress (was: Patch Available)
any update here? [~larsgeorge]?
> Normalizer switch in configuration is not used
> --
>
> Key: HBASE-15098
> URL: https://issues.apache.org/jira/browse/HBASE-15098
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 1.2.0
>Reporter: Lars George
>Priority: Blocker
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-15098.v1.patch
>
>
> The newly added global switch to enable the new normalizer functionality is
> never used apparently, meaning it is always on. The {{hbase-default.xml}} has
> this:
> {noformat}
>
> hbase.normalizer.enabled
> false
> If set to true, Master will try to keep region size
> within each table approximately the same.
>
> {noformat}
> But only a test class uses it to set the switch to "true". We should
> implement a proper {{if}} statement that checks this value and properly
> disables the feature cluster wide if not wanted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14970) Backport HBASE-13082 and its sub-jira to branch-1
[
https://issues.apache.org/jira/browse/HBASE-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15108114#comment-15108114
]
Hadoop QA commented on HBASE-14970:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 3s {color}
| {color:red} HBASE-14970 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/latest/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12783292/HBASE-14970_branch-1_6.patch
|
| JIRA Issue | HBASE-14970 |
| Powered by | Apache Yetus 0.1.0 http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/202/console |
This message was automatically generated.
> Backport HBASE-13082 and its sub-jira to branch-1
> -
>
> Key: HBASE-14970
> URL: https://issues.apache.org/jira/browse/HBASE-14970
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-13082-branch-1.patch, HBASE-14970_branch-1.patch,
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1.patch,
> HBASE-14970_branch-1_1.patch, HBASE-14970_branch-1_2.patch,
> HBASE-14970_branch-1_4.patch, HBASE-14970_branch-1_5.patch,
> HBASE-14970_branch-1_6.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14872) Scan different timeRange per column family doesn't percolate down to the memstore
[ https://issues.apache.org/jira/browse/HBASE-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14872: Resolution: Fixed Fix Version/s: (was: 0.98.18) Status: Resolved (was: Patch Available) > Scan different timeRange per column family doesn't percolate down to the > memstore > -- > > Key: HBASE-14872 > URL: https://issues.apache.org/jira/browse/HBASE-14872 > Project: HBase > Issue Type: Bug > Components: Client, regionserver, Scanners >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 1.2.0 > > Attachments: HBASE-14872-0.98-v1.patch, HBASE-14872-0.98.patch, > HBASE-14872-v1.patch, HBASE-14872.patch > > > HBASE-14355 The scan different time range for column family feature was not > applied to the memstore it was only done for the store files. This breaks > the contract. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
