[
https://issues.apache.org/jira/browse/PHOENIX-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viraj Jasani updated PHOENIX-7363:
----------------------------------
Summary: Protect metadata cache updates for the given PTable (was: All
meta handlers on SYSTEM.CATALOG regionserver exhausted within CQSI#init)
> Protect metadata cache updates for the given PTable
> ---------------------------------------------------
>
> Key: PHOENIX-7363
> URL: https://issues.apache.org/jira/browse/PHOENIX-7363
> Project: Phoenix
> Issue Type: Bug
> Reporter: Viraj Jasani
> Assignee: Viraj Jasani
> Priority: Blocker
> Fix For: 5.2.1, 5.3.0
>
>
> *All meta handlers on SYSTEM.CATALOG regionserver exhausted within CQSI#init*
> After allowing readLock for getTable API at the server side (PHOENIX-6066),
> it seems that under heavy load, all meta handler threads can get exhausted
> within ConnectionQueryServicesImpl initialization as part of any of the
> MetaDataEndpointImpl coproc operations. When the table details are not
> present in the cache, MetaDataEndpointImpl coproc can attempt to create new
> connection on the server side in order to scan SYSTEM.CATALOG. Under heavy
> load, several (all of) meta handlers – which are dedicated for all metadata
> (system table) operations – could attempt to create server side connection,
> which can further lead into creating new PhoenixConnection to execute CREATE
> TABLE DDL for SYSTEM.CATALOG in order to ensure that the System tables exist.
> {code:java}
> "RpcServer.Metadata.Fifo.handler=254,queue=20,port=60020" #927 daemon prio=5
> os_prio=0 tid=0x00007fd53f16a000 nid=0x473 waiting for monitor entry
> [0x00007fd4b1234000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:3547)
> - waiting to lock <0x000000047da00058> (a
> org.apache.phoenix.query.ConnectionQueryServicesImpl)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:3537)
> at
> org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:76)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:3537)
> at
> org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:272)
> at
> org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
> at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:229)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:208)
> at org.apache.phoenix.util.QueryUtil.getConnection(QueryUtil.java:433)
> at
> org.apache.phoenix.util.QueryUtil.getConnectionOnServer(QueryUtil.java:410)
> at
> org.apache.phoenix.util.QueryUtil.getConnectionOnServer(QueryUtil.java:391)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1499)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1075)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:1069)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:737)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:3599)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.addIndexToTable(MetaDataEndpointImpl.java:832)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1490)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1075)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:1069)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:737)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:3599)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:669)
> at
> org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService.callMethod(MetaDataProtos.java:19507)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7941)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2537)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2511)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45035)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:415)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82) {code}
> {code:java}
> "RpcServer.Metadata.Fifo.handler=142,queue=12,port=60020" #815 daemon prio=5
> os_prio=0 tid=0x00007fd53f07f000 nid=0x403 waiting on condition
> [0x00007fd4b8234000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000000047d400640> (a
> java.util.concurrent.FutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at
> org.apache.hadoop.hbase.client.HTable.lambda$coprocessorService$23(HTable.java:1150)
> at
> org.apache.hadoop.hbase.client.HTable$$Lambda$933/100908000.run(Unknown
> Source)
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187)
> at
> org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1128)
> at
> org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1114)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.checkClientServerCompatibility(ConnectionQueryServicesImpl.java:1855)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:1601)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:2201)
> at
> org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:3202)
> at
> org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:1028)
> at
> org.apache.phoenix.compile.CreateTableCompiler$CreateTableMutationPlan.execute(CreateTableCompiler.java:483)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:569)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:534)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:533)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:521)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:2235)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:3586)
> - locked <0x000000047da00058> (a
> org.apache.phoenix.query.ConnectionQueryServicesImpl)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:3537)
> at
> org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:76)
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:3537)
> at
> org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:272)
> at
> org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
> at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:229)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:208)
> at org.apache.phoenix.util.QueryUtil.getConnection(QueryUtil.java:433)
> at
> org.apache.phoenix.util.QueryUtil.getConnectionOnServer(QueryUtil.java:410)
> at
> org.apache.phoenix.util.QueryUtil.getConnectionOnServer(QueryUtil.java:391)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1499)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1075)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:1069)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:737)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:3599)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.addIndexToTable(MetaDataEndpointImpl.java:832)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1490)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1075)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:1069)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:737)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:3599)
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:669)
> at
> org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService.callMethod(MetaDataProtos.java:19507)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7941)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2537)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2511)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45035)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:415)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82) {code}
>
> Phoenix server to server system table RPC calls are supposed to be using
> separate handler thread-pools (PHOENIX-6687). However, this is not correctly
> working because regardless of whether the HBase Connection is initiated by
> client or server, Phoenix only provides ClientRpcControllerFactory by
> default. We need to provide separate RpcControllerFactory during HBase
> Connection initialization done by Coprocessors that operate on regionservers.
> For Phoenix server creating CQSI connection, we do not need to check for
> existence of system tables as well as client-server version compatibility.
> This redundant RPC call can be avoided.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
