[
https://issues.apache.org/jira/browse/PHOENIX-6571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Stavrov updated PHOENIX-6571:
-------------------------------------
Description:
We often have connection hangs after OOM. Hangs often occur on locks.
I made a test case locally that reproduces this problem, but I don't know Java,
so I have a complex bundle (perl -> dbd_jdbc -> hbase/phoenix) to use phoenix,
so because of this i am not showing a minimal example to reproduce this
behavior.
Here is the error log shown by dbd_jdbc: [https://pastebin.com/sHcH2iVq]
After the OOM happened, all connections to dbd_jdbc hang and very often have
such a stack trace:
{noformat}
"Thread-18" #197 prio=5 os_prio=0 cpu=12.95ms elapsed=1021.08s
tid=0x00007ffaa4124800 nid=0x1c84 in Object.wait() [0x00007ff9c0ecb000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait([email protected]/Native Method)
- waiting on <no object reference available>
at java.lang.Object.wait([email protected]/Object.java:462)
at
java.util.concurrent.TimeUnit.timedWait([email protected]/TimeUnit.java:408)
at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForSpecificCompletedTask(ResultBoundedCompletionService.java:258)
- waiting to re-lock in wait() <0x00000000ffdb05a0> (a
[Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForFirstSuccessfullyCompletedTask(ResultBoundedCompletionService.java:214)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:228)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
at
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:275)
at
org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436)
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:310)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1341)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1230)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1214)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1171)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:1003)
at
org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:80)
at
org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:519)
at
org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:735)
at
org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:705)
at
org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1808)
at
org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1763)
at
org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1743)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1769)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1749)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1996)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:657)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:545)
at
org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:772)
at
org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:442)
at
org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:227)
at
org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:205)
at org.apache.phoenix.util.ParseNodeUtil.rewrite(ParseNodeUtil.java:177)
at
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:489)
at
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:462)
at
org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:304)
at
org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:293)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:292)
at
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:285)
at
org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:176)
at
org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:183)
at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:744)
at com.vizdom.dbd.jdbc.Connection.run(Connection.java:233)
at java.lang.Thread.run([email protected]/Thread.java:829)
{noformat}
But sometimes after such exception
{noformat}
Exception in thread "IPC Client (55674532) connection to
bca736ff9b7d/172.17.0.2:35469 from root" java.lang.OutOfMemoryError: Java heap
space
21/10/08 09:19:53 [Thread-0-SendThread(127.0.0.1:2181)] WARN
server.ZooKeeperThread: Exception occurred from thread
Thread-0-SendThread(127.0.0.1:2181)
java.lang.OutOfMemoryError: Java heap space
{noformat}
we have such stacktrace:
{noformat}
Thread-15" #146 prio=5 os_prio=0 cpu=2.06ms elapsed=190.12s
tid=0x00007f130c110000 nid=0x1a1a in Object.wait() [0x00007f11eefe9000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait([email protected]/Native Method)
- waiting on <no object reference available>
at java.lang.Object.wait([email protected]/Object.java:328)
at
org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
- waiting to re-lock in wait() <0x00000000ff6e4d98> (a
org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn$Packet)
at
org.apache.phoenix.shaded.org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1210)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:629)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionState(MetaTableLocator.java:487)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionLocation(MetaTableLocator.java:168)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:607)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:588)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:561)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1266)
- locked <0x00000000f73ea510> (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1227)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:356)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
at
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:275)
at
org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436)
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:310)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1341)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1230)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1214)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1171)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:1003)
at
org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:80)
at
org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:519)
at
org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:735)
at
org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:705)
at
org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1808)
at
org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1763)
at
org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1743)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1769)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1749)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1996)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:657)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:545)
at
org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:772)
at
org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:442)
at
org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:227)
at
org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:205)
at org.apache.phoenix.util.ParseNodeUtil.rewrite(ParseNodeUtil.java:177)
at
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:489)
at
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:462)
at
org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:304)
at
org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:293)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:292)
at
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:285)
at
org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:176)
at
org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:183)
at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:744)
at com.vizdom.dbd.jdbc.Connection.run(Connection.java:233)
at java.lang.Thread.run([email protected]/Thread.java:829)
{noformat}
If it is difficult to reproduce this problem, then write to me here, I will try
to reproduce with a minimal example in Java.
was:
We often have connection hangs after OOM. Hangs often occur on locks.
I made a test case locally that reproduces this problem, but I don't know Java,
so I have a complex bundle (perl -> dbd_jdbc -> hbase/phoenix) to use phoenix,
so because of this i am not showing a minimal example to reproduce this
behavior.
Here is the error log shown by dbd_jdbc: https://pastebin.com/sHcH2iVq
After the OOM happened, all connections to dbd_jdbc hang and very often have
such a stack trace:
{noformat}
"Thread-18" #197 prio=5 os_prio=0 cpu=12.95ms elapsed=1021.08s
tid=0x00007ffaa4124800 nid=0x1c84 in Object.wait() [0x00007ff9c0ecb000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait([email protected]/Native Method)
- waiting on <no object reference available>
at java.lang.Object.wait([email protected]/Object.java:462)
at
java.util.concurrent.TimeUnit.timedWait([email protected]/TimeUnit.java:408)
at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForSpecificCompletedTask(ResultBoundedCompletionService.java:258)
- waiting to re-lock in wait() <0x00000000ffdb05a0> (a
[Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForFirstSuccessfullyCompletedTask(ResultBoundedCompletionService.java:214)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:228)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
at
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:275)
at
org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436)
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:310)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1341)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1230)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1214)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1171)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:1003)
at
org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:80)
at
org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:519)
at
org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:735)
at
org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:705)
at
org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1808)
at
org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1763)
at
org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1743)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1769)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1749)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1996)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:657)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:545)
at
org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:772)
at
org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:442)
at
org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:227)
at
org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:205)
at org.apache.phoenix.util.ParseNodeUtil.rewrite(ParseNodeUtil.java:177)
at
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:489)
at
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:462)
at
org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:304)
at
org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:293)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:292)
at
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:285)
at
org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:176)
at
org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:183)
at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:744)
at com.vizdom.dbd.jdbc.Connection.run(Connection.java:233)
at java.lang.Thread.run([email protected]/Thread.java:829)
{noformat}
But sometimes after such exception
{noformat}
Exception in thread "IPC Client (55674532) connection to
bca736ff9b7d/172.17.0.2:35469 from root" java.lang.OutOfMemoryError: Java heap
space
21/10/08 09:19:53 [Thread-0-SendThread(127.0.0.1:2181)] WARN
server.ZooKeeperThread: Exception occurred from thread
Thread-0-SendThread(127.0.0.1:2181)
java.lang.OutOfMemoryError: Java heap space
{noformat}
we have such stacktrace:
{noformat}
Thread-15" #146 prio=5 os_prio=0 cpu=2.06ms elapsed=190.12s
tid=0x00007f130c110000 nid=0x1a1a in Object.wait() [0x00007f11eefe9000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait([email protected]/Native Method)
- waiting on <no object reference available>
at java.lang.Object.wait([email protected]/Object.java:328)
at
org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
- waiting to re-lock in wait() <0x00000000ff6e4d98> (a
org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn$Packet)
at
org.apache.phoenix.shaded.org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1210)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:629)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionState(MetaTableLocator.java:487)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionLocation(MetaTableLocator.java:168)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:607)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:588)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:561)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1266)
- locked <0x00000000f73ea510> (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1227)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:356)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
at
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:275)
at
org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436)
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:310)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1341)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1230)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1214)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1171)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:1003)
at
org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:80)
at
org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:519)
at
org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:735)
at
org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:705)
at
org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1808)
at
org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1763)
at
org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1743)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1769)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1749)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1996)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:657)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:545)
at
org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:772)
at
org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:442)
at
org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:227)
at
org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:205)
at org.apache.phoenix.util.ParseNodeUtil.rewrite(ParseNodeUtil.java:177)
at
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:489)
at
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:462)
at
org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:304)
at
org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:293)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:292)
at
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:285)
at
org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:176)
at
org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:183)
at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:744)
at com.vizdom.dbd.jdbc.Connection.run(Connection.java:233)
at java.lang.Thread.run([email protected]/Thread.java:829)
{noformat}
If it is difficult to reproduce this problem, then write, I will try to
reproduce with a minimal example in Java.
> Hang of connections after OutOfMemoryError
> -------------------------------------------
>
> Key: PHOENIX-6571
> URL: https://issues.apache.org/jira/browse/PHOENIX-6571
> Project: Phoenix
> Issue Type: Bug
> Components: core
> Affects Versions: 4.16.1
> Reporter: Aleksey Stavrov
> Priority: Major
>
> We often have connection hangs after OOM. Hangs often occur on locks.
> I made a test case locally that reproduces this problem, but I don't know
> Java, so I have a complex bundle (perl -> dbd_jdbc -> hbase/phoenix) to use
> phoenix, so because of this i am not showing a minimal example to reproduce
> this behavior.
> Here is the error log shown by dbd_jdbc: [https://pastebin.com/sHcH2iVq]
> After the OOM happened, all connections to dbd_jdbc hang and very often have
> such a stack trace:
> {noformat}
> "Thread-18" #197 prio=5 os_prio=0 cpu=12.95ms elapsed=1021.08s
> tid=0x00007ffaa4124800 nid=0x1c84 in Object.wait() [0x00007ff9c0ecb000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>
> at java.lang.Object.wait([email protected]/Native Method)
> - waiting on <no object reference available>
> at java.lang.Object.wait([email protected]/Object.java:462)
> at
> java.util.concurrent.TimeUnit.timedWait([email protected]/TimeUnit.java:408)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForSpecificCompletedTask(ResultBoundedCompletionService.java:258)
> - waiting to re-lock in wait() <0x00000000ffdb05a0> (a
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
>
>
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForFirstSuccessfullyCompletedTask(ResultBoundedCompletionService.java:214)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:228)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:275)
> at
> org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436)
> at
> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:310)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1341)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1230)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1214)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1171)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:1003)
> at
> org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:80)
> at
> org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:519)
> at
> org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:735)
> at
> org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:705)
> at
> org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1808)
> at
> org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1763)
> at
> org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1743)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1769)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1749)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1996)
> at
> org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:657)
> at
> org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:545)
> at
> org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:772)
> at
> org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:442)
> at
> org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:227)
> at
> org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:205)
> at
> org.apache.phoenix.util.ParseNodeUtil.rewrite(ParseNodeUtil.java:177)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:489)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:462)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:304)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:293)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:292)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:285)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:176)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:183)
> at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:744)
> at com.vizdom.dbd.jdbc.Connection.run(Connection.java:233)
> at java.lang.Thread.run([email protected]/Thread.java:829)
> {noformat}
> But sometimes after such exception
> {noformat}
> Exception in thread "IPC Client (55674532) connection to
> bca736ff9b7d/172.17.0.2:35469 from root" java.lang.OutOfMemoryError: Java
> heap space
> 21/10/08 09:19:53 [Thread-0-SendThread(127.0.0.1:2181)] WARN
> server.ZooKeeperThread: Exception occurred from thread
> Thread-0-SendThread(127.0.0.1:2181)
> java.lang.OutOfMemoryError: Java heap space
> {noformat}
> we have such stacktrace:
> {noformat}
> Thread-15" #146 prio=5 os_prio=0 cpu=2.06ms elapsed=190.12s
> tid=0x00007f130c110000 nid=0x1a1a in Object.wait() [0x00007f11eefe9000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait([email protected]/Native Method)
> - waiting on <no object reference available>
>
>
>
> at java.lang.Object.wait([email protected]/Object.java:328)
> at
> org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
> - waiting to re-lock in wait() <0x00000000ff6e4d98> (a
> org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn$Packet)
> at
> org.apache.phoenix.shaded.org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1210)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:629)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionState(MetaTableLocator.java:487)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionLocation(MetaTableLocator.java:168)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:607)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:588)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:561)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1266)
> - locked <0x00000000f73ea510> (a java.lang.Object)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1227)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:356)
>
>
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:275)
> at
> org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436)
> at
> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:310)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1341)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1230)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1214)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1171)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:1003)
> at
> org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:80)
> at
> org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:519)
> at
> org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:735)
> at
> org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:705)
> at
> org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1808)
> at
> org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1763)
> at
> org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1743)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1769)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1749)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1996)
> at
> org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:657)
> at
> org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:545)
> at
> org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:772)
> at
> org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:442)
> at
> org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:227)
> at
> org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:205)
> at
> org.apache.phoenix.util.ParseNodeUtil.rewrite(ParseNodeUtil.java:177)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:489)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:462)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:304)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:293)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:292)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:285)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:176)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:183)
> at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:744)
> at com.vizdom.dbd.jdbc.Connection.run(Connection.java:233)
> at java.lang.Thread.run([email protected]/Thread.java:829)
> {noformat}
> If it is difficult to reproduce this problem, then write to me here, I will
> try to reproduce with a minimal example in Java.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
