[
https://issues.apache.org/jira/browse/HBASE-14533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15173832#comment-15173832
]
Hadoop QA commented on HBASE-14533:
-----------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
33s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s
{color} | {color:green} branch-1 passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} branch-1 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
52s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 1s
{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} branch-1 passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 37s
{color} | {color:green} branch-1 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 19s
{color} | {color:red} Patch generated 1 new checkstyle issues in hbase-thrift
(total was 13, now 14). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 4m
57s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 34s
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 14s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 53s {color}
| {color:red} hbase-thrift in the patch failed with JDK v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 42s
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_95.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 36s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 50s {color}
| {color:red} hbase-thrift in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 270m 20s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests | hadoop.hbase.mapreduce.TestImportExport |
| JDK v1.8.0_72 Timed out junit tests |
org.apache.hadoop.hbase.thrift.TestThriftServer |
| JDK v1.7.0_95 Failed junit tests | hadoop.hbase.mapreduce.TestImportExport |
| JDK v1.7.0_95 Timed out junit tests |
org.apache.hadoop.hbase.thrift.TestThriftServer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-03-01 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12765884/14533v2.branch-1.patch
|
| JIRA Issue | HBASE-14533 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux b49af80a5089 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision | branch-1 / a9dd9c7 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/775/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/775/artifact/patchprocess/diff-checkstyle-hbase-thrift.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/775/artifact/patchprocess/patch-unit-hbase-server-jdk1.8.0_72.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/775/artifact/patchprocess/patch-unit-hbase-thrift-jdk1.8.0_72.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/775/artifact/patchprocess/patch-unit-hbase-server-jdk1.7.0_95.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/775/artifact/patchprocess/patch-unit-hbase-thrift-jdk1.7.0_95.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/775/artifact/patchprocess/patch-unit-hbase-server-jdk1.8.0_72.txt
https://builds.apache.org/job/PreCommit-HBASE-Build/775/artifact/patchprocess/patch-unit-hbase-thrift-jdk1.8.0_72.txt
https://builds.apache.org/job/PreCommit-HBASE-Build/775/artifact/patchprocess/patch-unit-hbase-server-jdk1.7.0_95.txt
https://builds.apache.org/job/PreCommit-HBASE-Build/775/artifact/patchprocess/patch-unit-hbase-thrift-jdk1.7.0_95.txt
|
| JDK v1.7.0_95 Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/775/testReport/ |
| modules | C: hbase-client hbase-server hbase-thrift U: . |
| Max memory used | 127MB |
| Powered by | Apache Yetus 0.1.0 http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/775/console |
This message was automatically generated.
> Thrift client gets "AsyncProcess: Failed to get region location .... closed"
> ----------------------------------------------------------------------------
>
> Key: HBASE-14533
> URL: https://issues.apache.org/jira/browse/HBASE-14533
> Project: HBase
> Issue Type: Bug
> Components: REST, Thrift
> Affects Versions: 1.0.0
> Reporter: stack
> Assignee: stack
> Attachments: 14533.test.patch, 14533v2.branch-1.patch, test.patch
>
>
> An internal python client has been getting below stack trace since
> HBASE-134347
> {code}
> 2015-09-30 11:27:31,670 runner ERROR : scheduler
> executor error
> 2015-09-30 11:27:31,674 runner ERROR : Traceback (most
> recent call last):
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsRtiFetcher-0.1-py2.6.egg/cops_rti/fetcher/runner.py",
> line 82, in run
> fetch_list = self.__scheduler_executor.run()
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsRtiFetcher-0.1-py2.6.egg/cops_rti/fetcher/scheduler.py",
> line 35, in run
> with self.__fetch_db_dao.get_scanner() as scanner:
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsHbaseCommon-f796bf2929be11c26536c3e8f3e9c0b0ecb382b3-py2.6.egg/cops/hbase/common/hbase_dao.py",
> line 57, in get_scanner
> caching=caching, field_filter_list=field_filter_list)
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsHbaseCommon-f796bf2929be11c26536c3e8f3e9c0b0ecb382b3-py2.6.egg/cops/hbase/common/hbase_client_template.py",
> line 104, in get_entity_scanner
> self.__fix_cfs(self.__filter_columns(field_filter_list)), caching)
> File
> "/opt/cops/cops-related-ticket-info-fetcher/fetcher/.virtenv/lib/python2.6/site-packages/CopsHbaseCommon-f796bf2929be11c26536c3e8f3e9c0b0ecb382b3-py2.6.egg/cops/hbase/common/hbase_entity_scanner.py",
> line 81, in open
> self.__scanner_id = client.scannerOpenWithScan(table_name, scan)
> File
> "/opt/cops/cops-related-ticket-info-fetcher/.crepo/cops-hbase-common/ext-py/hbase/Hbase.py",
> line 1494, in scannerOpenWithScan
> return self.recv_scannerOpenWithScan()
> File
> "/opt/cops/cops-related-ticket-info-fetcher/.crepo/cops-hbase-common/ext-py/hbase/Hbase.py",
> line 1518, in recv_scannerOpenWithScan
> raise result.io
> IOError:
> IOError(message="org.apache.hadoop.hbase.client.RetriesExhaustedException:
> Can't get the location\n\tat
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:308)\n\tat
>
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:149)\n\tat
>
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:57)\n\tat
>
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:293)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:268)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:140)\n\tat
>
> org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:135)\n\tat
> org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888)\n\tat
> org.apache.hadoop.hbase.thrift.ThriftServerRunner$HBaseHandler.scannerOpenWithScan(ThriftServerRunner.java:1446)\n\tat
> sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)\n\tat
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
> java.lang.reflect.Method.invoke(Method.java:606)\n\tat
> org.apache.hadoop.hbase.thrift.HbaseHandlerMetricsProxy.invoke(HbaseHandlerMetricsProxy.java:67)\n\tat
> com.sun.proxy.$Proxy14.scannerOpenWithScan(Unknown Source)\n\tat
> org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$scannerOpenWithScan.getResult(Hbase.java:4609)\n\tat
>
> org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$scannerOpenWithScan.getResult(Hbase.java:4593)\n\tat
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)\n\tat
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)\n\tat
> org.apache.hadoop.hbase.thrift.ThriftServerRunner$3.process(ThriftServerRunner.java:502)\n\tat
>
> org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer$ClientConnnection.run(TBoundedThreadPoolServer.java:289)\n\tat
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat
> java.lang.Thread.run(Thread.java:745)\nCaused by: java.io.IOException:
> hconnection-0xa8e1bf9 closed\n\tat
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1117)\n\tat
>
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:299)\n\t...
> 23 more\n")
> {code}
> On the thrift server side we see this:
> {code}
> 2015-09-30 07:22:59,427 ERROR org.apache.hadoop.hbase.client.AsyncProcess:
> Failed to get region location
> java.io.IOException: hconnection-0x4142991e closed
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1117)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:369)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:320)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
> at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1496)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1107)
> at
> org.apache.hadoop.hbase.thrift.ThriftServerRunner$HBaseHandler.mutateRowTs(ThriftServerRunner.java:1256)
> at
> org.apache.hadoop.hbase.thrift.ThriftServerRunner$HBaseHandler.mutateRow(ThriftServerRunner.java:1209)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hbase.thrift.HbaseHandlerMetricsProxy.invoke(HbaseHandlerMetricsProxy.java:67)
> at com.sun.proxy.$Proxy14.mutateRow(Unknown Source)
> at
> org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$mutateRow.getResult(Hbase.java:4334)
> at
> org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$mutateRow.getResult(Hbase.java:4318)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hbase.thrift.ThriftServerRunner$3.process(ThriftServerRunner.java:502)
> at
> org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer$ClientConnnection.run(TBoundedThreadPoolServer.java:289)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> HBASE-13437 has us actual execute a close on timeout -- before we'd mark
> connection closed but would never call close on it.
> A background chore is going around stamping Connections in the
> ConnectionCache as 'closed' if they have not been used in ten minutes. The
> 'close' can come in at any time..... In particular between the point at which
> we get the table/connection and when we go to use it: i.e. flush puts. It is
> at the flush puts point that we get the above 'AsyncProcess: Failed to get
> region location' (It is not a failure to find region location but rather our
> noticing that the connection has been closed).
> Attempts at reproducing this issue locally letting the Connection timeout can
> generate the above exception if a certain dance is done but it is hard to do;
> I am not reproducing the actual usage by the aforementioned client.
> Next steps would be setting up python client talking via thrift and then try
> using connection after it has been evicted from the connection cache. Another
> thing to try is a pool of connections on the python side...connections are
> identified by user and table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
