Jacob Isaac created PHOENIX-6687:
------------------------------------
Summary: The region server hosting the SYSTEM.CATALOG fails to
serve any metadata requests as handler pools are exhausted.
Key: PHOENIX-6687
URL: https://issues.apache.org/jira/browse/PHOENIX-6687
Project: Phoenix
Issue Type: Bug
Components: core
Affects Versions: 4.16.1, 5.2.0
Reporter: Jacob Isaac
Fix For: 4.17.0, 5.2.0
When SYSTEM.CATALOG region server is restarted and the server is experiencing
heavy metadata call volume.
The stack traces indicate that all the default handler pool threads are waiting
for the CQSI.init thread to finish initializing.
The CQSI.init thread itself cannot proceed since it cannot complete the second
RPC call
(org.apache.phoenix.query.ConnectionQueryServicesImpl.checkClientServerCompatibility)
due to thread starvation.
For e.g
The following
[code|https://github.com/apache/phoenix/blob/3cff97087d79b85e282fca4ac69ddf499fb1f40f/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java#L661]
turned the getTable(..) into needing an additional server-to-server RPC call
when initializing a PhoenixConnection (CQSI.init) for the first time on the
JVM.
It is well-known that server-to-server RPC calls are prone to deadlocking due
to thread pool exhaustion.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
