[
https://issues.apache.org/jira/browse/PHOENIX-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866584#comment-17866584
]
Viraj Jasani commented on PHOENIX-7363:
---------------------------------------
While our systems were always exposed to this issue, the reason why we were
protected is because every getTable() call used to take HBase row level
writeLock and hence block all other getTable() calls on the same table (with
key: tenant id + schema name + table name), but of course this is not a
scalable solution and hence PHOENIX-6066 attempted to fix it. Hence, now we are
exposed to this issue.
> All meta handlers on SYSTEM.CATALOG regionserver exhausted within CQSI#init
> ---------------------------------------------------------------------------
>
> Key: PHOENIX-7363
> URL: https://issues.apache.org/jira/browse/PHOENIX-7363
> Project: Phoenix
> Issue Type: Bug
> Reporter: Viraj Jasani
> Assignee: Viraj Jasani
> Priority: Major
> Fix For: 5.2.1, 5.3.0
>
>
> After allowing readLock for getTable API at the server side (PHOENIX-6066),
> it seems that under heavy load, all meta handler threads can get exhausted
> within ConnectionQueryServicesImpl initialization as part of any of the
> MetaDataEndpointImpl coproc operations. When the table details are not
> present in the cache, MetaDataEndpointImpl coproc can attempt to create new
> connection on the server side in order to scan SYSTEM.CATALOG. Under heavy
> load, several (all of) meta handlers – which are dedicated for all metadata
> (system table) operations – could attempt to create server side connection,
> which can further lead into creating new PhoenixConnection to execute CREATE
> TABLE DDL for SYSTEM.CATALOG in order to ensure that the System tables exist.
> {code:java}
> "RpcServer.Metadata.Fifo.handler=254,queue=20,port=60020" #927 daemon prio=5
> os_prio=0 tid=0x00007fd53f16a000 nid=0x473 waiting for monitor entry
> [0x00007fd4b1234000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:3547)
> - waiting to lock <0x000000047da00058> (a
> org.apache.phoenix.query.ConnectionQueryServicesImpl)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:3537)
> at
> org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:76)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:3537)
> at
> org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:272)
> at
> org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
> at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:229)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:208)
> at org.apache.phoenix.util.QueryUtil.getConnection(QueryUtil.java:433)
> at
> org.apache.phoenix.util.QueryUtil.getConnectionOnServer(QueryUtil.java:410)
> at
> org.apache.phoenix.util.QueryUtil.getConnectionOnServer(QueryUtil.java:391)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1499)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1075)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:1069)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:737)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:3599)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.addIndexToTable(MetaDataEndpointImpl.java:832)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1490)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1075)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:1069)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:737)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:3599)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:669)
> at
> org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService.callMethod(MetaDataProtos.java:19507)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7941)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2537)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2511)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45035)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:415)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82) {code}
> {code:java}
> "RpcServer.Metadata.Fifo.handler=142,queue=12,port=60020" #815 daemon prio=5
> os_prio=0 tid=0x00007fd53f07f000 nid=0x403 waiting on condition
> [0x00007fd4b8234000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000000047d400640> (a
> java.util.concurrent.FutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at
> org.apache.hadoop.hbase.client.HTable.lambda$coprocessorService$23(HTable.java:1150)
> at
> org.apache.hadoop.hbase.client.HTable$$Lambda$933/100908000.run(Unknown
> Source)
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187)
> at
> org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1128)
> at
> org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1114)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.checkClientServerCompatibility(ConnectionQueryServicesImpl.java:1855)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:1601)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:2201)
> at
> org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:3202)
> at
> org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:1028)
> at
> org.apache.phoenix.compile.CreateTableCompiler$CreateTableMutationPlan.execute(CreateTableCompiler.java:483)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:569)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:534)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:533)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:521)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:2235)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:3586)
> - locked <0x000000047da00058> (a
> org.apache.phoenix.query.ConnectionQueryServicesImpl)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:3537)
> at
> org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:76)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:3537)
> at
> org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:272)
> at
> org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
> at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:229)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:208)
> at org.apache.phoenix.util.QueryUtil.getConnection(QueryUtil.java:433)
> at
> org.apache.phoenix.util.QueryUtil.getConnectionOnServer(QueryUtil.java:410)
> at
> org.apache.phoenix.util.QueryUtil.getConnectionOnServer(QueryUtil.java:391)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1499)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1075)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:1069)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:737)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:3599)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.addIndexToTable(MetaDataEndpointImpl.java:832)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1490)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1075)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:1069)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:737)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:3599)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:669)
> at
> org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService.callMethod(MetaDataProtos.java:19507)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7941)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2537)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2511)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45035)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:415)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82) {code}
> The system tables existence check is not required for server side connections.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
