[
https://issues.apache.org/jira/browse/HBASE-12219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14193727#comment-14193727
]
Dima Spivak commented on HBASE-12219:
-------------------------------------
I disabled forked process timeouts and reran TestAdmin before [~stack]'s
commit, which revealed this:
{code}
testTruncateTablePreservingSplits(org.apache.hadoop.hbase.client.TestAdmin)
Time elapsed: 242.311 sec <<< ERROR!
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=7, exceptions:
Sat Nov 01 22:54:33 PDT 2014, null, java.net.SocketTimeoutException:
callTimeout=60000, callDuration=291372: row '' on table
'testTruncateTablePreservingSplits
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:261)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:199)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:56)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:196)
at
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:287)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:267)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:139)
at
org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:134)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:789)
at
org.apache.hadoop.hbase.HBaseTestingUtility.countRows(HBaseTestingUtility.java:1894)
at
org.apache.hadoop.hbase.client.TestAdmin.testTruncateTable(TestAdmin.java:393)
at
org.apache.hadoop.hbase.client.TestAdmin.testTruncateTablePreservingSplits(TestAdmin.java:369)
Caused by: java.net.SocketTimeoutException: callTimeout=60000,
callDuration=291372: row '' on table 'testTruncateTablePreservingSplits
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:155)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:294)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:275)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get
the location
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:299)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:136)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:121)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:294)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:275)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.client.NoServerForRegionException: No server
address listed in hbase:meta for region
testTruncateTablePreservingSplits,,1414907431375.c73f9cd53b251842620244ea108213d5.
containing row
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1227)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1093)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1064)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:288)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:136)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:121)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:294)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:275)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
testTruncateTable(org.apache.hadoop.hbase.client.TestAdmin) Time elapsed:
242.6 sec <<< ERROR!
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=7, exceptions:
Sat Nov 01 23:02:52 PDT 2014, null, java.net.SocketTimeoutException:
callTimeout=60000, callDuration=292098: row '' on table 'testTruncateTable
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:261)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:199)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:56)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:196)
at
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:287)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:267)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:139)
at
org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:134)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:789)
at
org.apache.hadoop.hbase.HBaseTestingUtility.countRows(HBaseTestingUtility.java:1894)
at
org.apache.hadoop.hbase.client.TestAdmin.testTruncateTable(TestAdmin.java:393)
at
org.apache.hadoop.hbase.client.TestAdmin.testTruncateTable(TestAdmin.java:364)
Caused by: java.net.SocketTimeoutException: callTimeout=60000,
callDuration=292098: row '' on table 'testTruncateTable
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:155)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:294)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:275)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get
the location
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:299)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:136)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:121)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:294)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:275)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.client.NoServerForRegionException: No server
address listed in hbase:meta for region
testTruncateTable,,1414907930959.4e6ea2189874de8fc4b408137fa5d38f. containing
row
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1227)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1093)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1064)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:288)
at
org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:136)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:121)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:294)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:275)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
I also ran {{jstack -F}} on the process; let me know if you want me to put that
up on pastebin, [~esteban].
> Cache more efficiently getAll() and get() in FSTableDescriptors
> ---------------------------------------------------------------
>
> Key: HBASE-12219
> URL: https://issues.apache.org/jira/browse/HBASE-12219
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.94.24, 0.99.1, 0.98.6.1
> Reporter: Esteban Gutierrez
> Assignee: Esteban Gutierrez
> Labels: scalability
> Fix For: 2.0.0, 0.98.8, 0.99.2
>
> Attachments: HBASE-12219-0.98.patch, HBASE-12219-0.98.v1.patch,
> HBASE-12219-0.99.addendum.patch, HBASE-12219-0.99.patch,
> HBASE-12219-v1.patch, HBASE-12219-v1.patch, HBASE-12219.v0.txt,
> HBASE-12219.v2.patch, HBASE-12219.v3.patch, list.png
>
>
> Currently table descriptors and tables are cached once they are accessed for
> the first time. Next calls to the master only require a trip to HDFS to
> lookup the modified time in order to reload the table descriptors if
> modified. However in clusters with a large number of tables or concurrent
> clients and this can be too aggressive to HDFS and the master causing
> contention to process other requests. A simple solution is to have a TTL
> based cached for FSTableDescriptors#getAll() and
> FSTableDescriptors#TableDescriptorAndModtime() that can allow the master to
> process those calls faster without causing contention without having to
> perform a trip to HDFS for every call. to listtables() or getTableDescriptor()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
