[
https://issues.apache.org/jira/browse/PHOENIX-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17568217#comment-17568217
]
ASF GitHub Bot commented on PHOENIX-6687:
-----------------------------------------
gjacoby126 commented on code in PR #1440:
URL: https://github.com/apache/phoenix/pull/1440#discussion_r923869765
##########
phoenix-core/src/main/java/org/apache/hadoop/hbase/ipc/controller/ServerSideRPCControllerFactory.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.ipc.controller;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.ipc.RpcControllerFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link RpcControllerFactory} that should only be used when
+ * making server-server remote RPCs to the region servers hosting Phoenix
SYSTEM tables.
Review Comment:
OK, so let me make sure I understand properly.
Direct operations on a system table (UPSERTs or queries) would use the
MetadataRpcController and coproc calls (such as CREATE and ALTER operations
where MetadataClient calls MetadataEndpointImpl operations) would use
ServerSideRPCController? And there's no way to consolidate the two in either
direction because of the mismatch between shaded and unshaded protobufs
disallows casting?
> The region server hosting the SYSTEM.CATALOG fails to serve any metadata
> requests as default handler pool threads are exhausted.
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: PHOENIX-6687
> URL: https://issues.apache.org/jira/browse/PHOENIX-6687
> Project: Phoenix
> Issue Type: Bug
> Components: core
> Affects Versions: 5.1.0, 5.1.1, 4.16.1, 5.2.0, 5.1.2
> Reporter: Jacob Isaac
> Assignee: Jacob Isaac
> Priority: Major
> Fix For: 4.17.0, 5.2.0
>
> Attachments: stacktraces.txt
>
>
> When SYSTEM.CATALOG region server is restarted and the server is experiencing
> heavy metadata call volume.
> The stack traces indicate that all the default handler pool threads are
> waiting for the CQSI.init thread to finish initializing.
> The CQSI.init thread itself cannot proceed since it cannot complete the
> second RPC call
> (org.apache.phoenix.query.ConnectionQueryServicesImpl.checkClientServerCompatibility)
> due to thread starvation.
> For e.g
> The following
> [code|https://github.com/apache/phoenix/blob/3cff97087d79b85e282fca4ac69ddf499fb1f40f/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java#L661]
> turned the getTable(..) into needing an additional server-to-server RPC call
> when initializing a PhoenixConnection (CQSI.init) for the first time on the
> JVM.
> It is well-known that server-to-server RPC calls are prone to deadlocking due
> to thread pool exhaustion.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
